# Improving Scene Graph Generation with Relation Words’ Debiasing in Vision-
Language Models

Yuxuan Wang  Xiaoyuan Liu
Nanyang Technological University

###### Abstract

Scene Graph Generation (SGG) provides basic language representation of visual
scenes, requiring models to grasp complex and diverse semantics between
various objects. However, this complexity and diversity in SGG also leads to
underrepresentation, where part of test triplets are rare or even unseen
during training, resulting in imprecise predictions. To tackle this, we
propose using the SGG models with pre-trained vision-language models (VLMs) to
enhance representation. However, due to the gap between the pre-training and
SGG, directly emsembling the pre-trained VLMs leads to severe biases across
relation words. Thus, we introduce LM Estimation to approximate the words’
distribution underlies in the pre-training language sets, and then use the
distribution for debiasing. After that, we ensemble VLMs with SGG models to
enhance representation. Considering that each model may represent better at
different samples, we use a certainty-aware indicator to score each sample and
dynamically adjust the ensemble weights. Our method effectively addresses the
words biases, enhances SGG’s representation, and achieve markable performance
enhancements. It is training-free and integrates well with existing SGG
models.

Improving Scene Graph Generation with Relation Words’ Debiasing in Vision-
Language Models

Yuxuan Wang and Xiaoyuan Liu Nanyang Technological University

##  1 Introduction

Scene Graph Generation (SGG) is a fundamental vision-language task that
attracts much attention. It bridges visual and language semantics and serves
various applications, from enhancing robotic contextual awareness to aiding
visually impaired individuals. One of the key challenges of SGG is to grasp
complex, high-level semantics to understand inter-object relationships in a
scene.

Existing research in SGG primarily focuses on refining model architectures and
training from scratch with datasets like Visual Genome Krishna et al. (2017)
or Open Images Kuznetsova et al. (2020). However, SGG tasks inherently face
another challenge of underrepresentation. Arising from the inherent
complexities of SGG, there exists exponential variability of triplets combined
by the subject, object, and relation. It is extremely challenging for a
training set to cover such prohibitive diversity. As a result, a part of test
samples are underrepresented in training, leading to poorer predictions. In a
severe case, some triplets appear in the test set are entirely unseen during
training.

Figure 1: Illustration of the underrepresentation issue in Visual Genome. We
take out the relation class “carrying" from the top-right imbalanced class
distribution. Various samples are presented with their training representation
level and confidence score for the true class. Lower scores correlate with
low-quality predictions.

In Figure 1, we highlight the relation “carrying” from Visual Genome, showing
samples and their confidence score of the true class from a baseline model’s
predictions. While the well-represented samples score higher, the unseen
triplets like “woman carrying towel" score low. Moreover, the “woman carrying
umbrella" score only $0.15$ due to the umbrella being closed, while its
counterpart with an opened umbrella score markably higher. Even the triplet is
seen in training, the closed “umbrella” is less-represented than the opened
one.

A straightforward solution to this issue is to expand the model’s knowledge by
integrating advanced vision-language models (VLMs) pre-trained on extensive
datasets Kim et al. (2021); Li et al. (2020, 2019); Qi et al. (2020); Yu et
al. (2022); Radford et al. (2021). The comprehensive knowledge of VLMs can
help to compensate for the underrepresented samples. Through the Masked
Language Modeling (MLM) prompt format like “woman is [MASK] towel", we can
directly obtain the relation prediction from the fill-in answers produced by
zero-shot VLMs, which entirely keep the knowledge gained in pre-training.
However, due to the disalignment in data distribution and objectives between
pre-training and SGG, directly inference the zero-shot models on SGG will
cause severe relation-word bias. This bias underlies in the complex pre-
training language set of VLMs, determined by the frequency of different
relation words. Specifically, The models tend to favor relation words that
were prevalent during pre-training. Existing methods towards this bias depend
on explicit training relations’ distribution, which is often unattainable for
pre-trained VLMs: (1) The pre-training data is often confidential. (2) Since
the pre-training objectives are incompatible with SGG, there is no explicit
relation label distribution in pre-training data.

To tackle this challenge, we introduce a novel approach named LM Estimation
based on constrained optimization. It seeks to undermine the proxy relation
words’ distribution for SGG within VLMs, given there is no direct label
correspondence between the pre-training tasks and SGG. Upon obtaining the
estimated distribution, we proceed with relation words debiasing via a direct
post-hoc logits adjustment. Our LM Estimation, as demonstrated by
comprehensive experiments, is proved to be exceedingly effective in mitigating
the bias for zero-shot VLMs. The debiased VLMs can then be freely used for
ensemble to enhance the representation of SGG models.

While ensembling, we observe that some samples are better represented by the
zero-shot VLM, while others align better with the SGG-trained model.
Therefore, we propose to dynamically ensemble the zero-shot VLMs with task-
specific models trained on SGG. We employ a certainty-aware indicator to score
each sample. For each sample, the certainty score assesses its representation
level in the pre-trained VLM and the SGG model, subsequently determining the
ensemble weights. Our contributions can be summarized as:

  * •

While existing methods primarily focuses on refining model architecture, we
are among the pioneers in addressing the underrepresentation issue in SGG
using pre-trained VLMs.

  * •

Towards the relation words bias underlying in the pre-training language set,
we propose our LM Estimation, a concise solution to estimate the attainable
words’ distribution in pre-training.

  * •

We introduce a plug-and-play method that dynamically ensemble the zero-shot
VLMs. Without further training, it minimizes the computational and memory
burdens. Our method effectively enhance the representation in SGG, resulting
in markable performance improvement.

##  2 Related Work

Scene Graph Generation (SGG) is a fundamental task for understanding the
relationships between objects in images. Various of innovations Tang et al.
(2019); Gu et al. (2019); Li et al. (2021); Lin et al. (2022a, 2020, b); Zheng
et al. (2023); Xu et al. (2017) have been made in supervised SGG from the
Visual Genome benchmark Krishna et al. (2017). A typical approach involves
using a Faster R-CNN Sun et al. (2018) to identify image regions as objects,
followed by predicting their interrelations with a specialized network that
considers their attributes and spatial context. Existing efforts Li et al.
(2021); Lin et al. (2022a, b); Zheng et al. (2023) mainly focus on enhancing
this prediction network. For instance, Lin et al. (2022b) introduced a
regularized unrolling approach, and Zheng et al. (2023) used a prototypical
network for improved representation. These models specially tailored for SGG
has achieved a superior performance.

Unbiased Learning in SGG has been a long-standing challenge. Started by Tang
et al. (2020), the debiasing methods Dong et al. (2022); Li et al. (2021); Yan
et al. (2020); Li et al. (2022b, a) seek to removing the relation label bias
stemming from the imbalanced relation class distribution. These works have
achieved more balanced performance across all relation classes. However, these
methods rely on the interfere during training and are not feasible to the
words’ bias in pre-trained VLMs.

Pre-trained Vision-Language models (VLMs) have been widely applied in diverse
vision-language tasks Su et al. (2019); Radford et al. (2021); Kim et al.
(2021); Li et al. (2020) and have achieved substantial performance
improvements with the vast knowledge base obtained during pre-training.
Recently works start to adapt the comprehensive pre-trained knowledge in VLMs
to relation recognition and scene graph generation He et al. (2022); Gao et
al. (2023); Li et al. (2023); Yu et al. (2023); Zhang et al. (2023); Zhao et
al. (2023). Through prompt-tuning, He et al. (2022) is the first employing
VLMs to open-vocabulary scene graph generation. Then more approaches Zhang et
al. (2023); Yu et al. (2023); Gao et al. (2023) are designed towards this
task. These works demonstrate the capability of VLMs on recognizing relation,
inspiring us to utilize VLMs to improve the SGG representation.

##  3 Methodology

Figure 2: Illustration of our proposed architecture. left: the visual-language
inputs processed from image regions $\mathbf{x}_{i,j}$ and object labels
$(z_{i},z_{j})$, either provided or predicted by Faster R-CNN detector.
middle: the fixed zero-shot VLM $f_{\text{zs}}$ and the trainable task-
specific models $f_{\text{sg}}$, which we use a fine-tuned VLM as example.
right: the relation label debias process and the certainty-aware ensemble.

In this section, we first introduce our architecture consisting of a pre-
trained VLM and a task-specific model trained on SGG (Section 3.2). We then
address the relation words’ bias in the VLM (Section 3.3). Finally, we propose
our ensemble strategy that harnesses the debiased zero-shot VLM to enhance the
representation in SGG (Section 3.4).

###  3.1 Setup

Given a sample $(\mathbf{x},\mathcal{G})$ from an SGG dataset
$\mathcal{D}_{\text{sg}}$, the image $\mathbf{x}$ is parsed into a scene graph
$\mathcal{G}=\\{\mathcal{V},\mathcal{E}\\}$, where $\mathcal{V}$ is the object
set and $\mathcal{E}$ is the relation set. Specifically, each object
$\mathbf{v}\in\mathcal{V}$ consists of a corresponding bounding box
$\mathbf{b}$ and a categorical label $z$ either from annotation or predicted
by a trained Faster R-CNN detector; each $\mathbf{e}_{i,j}\in\mathcal{E}$
denotes the relation for subject-object pair $\mathbf{v}_{i}$ and
$\mathbf{v}_{j}$, represented by a predicate label $y\in\mathcal{C}_{e}$. The
predicate relation space $\mathcal{C}_{e}=\\{0\\}\cup\mathcal{C}_{r}$ includes
one background class $0$, indicating no relation and $K$ relationships
$\mathcal{C}_{r}=[K]$. The objective is to learn a model $f$ that, given the
predicted objects $z_{i}$ and $z_{j}$ for each pair with their cropped image
region $\mathbf{x}_{i,j}$, produces logits $\mathbf{o}$ for all relation
$y\in\mathcal{C}_{e}$, i.e., $\mathbf{o}=f(z_{i},z_{j},\mathbf{x}_{i,j})$.

###  3.2 Method Overview

Figure 2 illustrates the overview of our model $f$ comprising two braches: a
fixed zero-shot VLM $f_{\text{zs}}$ and a trainable task-specific model
$f_{\text{sg}}$. Here we employs a SGG fine-tuned VLM as $f_{\text{sg}}$,
where we forward the image region $\mathbf{x}_{i,j}$ to the visual encoder and
use the prompt template “what is the relationship between the {$z_{i}$} and
the {$z_{j}$}?” as the text input. Then, a classifier head is added to the
[CLS] token to generate logits $\mathbf{o}_{\text{sg}}$ of all relations
$y\in\mathcal{C}_{e}$. The task-specific model $f_{\text{sg}}$ is trained on
$\mathcal{D}_{\text{sg}}$ via empirical risk minimization.

The zero-shot model, represented as $f_{\text{zs}}$, leverages pre-trained
knowledge to the SGG task without fine-tuning. By providing prompts to the
pre-trained VLMs in the form “{$z_{i}$} is [MASK] {$z_{j}$}”, one can derive
the predicted logits $\mathbf{o}_{\text{zs}}^{k}$ of $K$ relation from the
fill-in answers. In SGG, the background class is defined when a relation is
outside $\mathcal{C}_{r}$. Predicting background relation is challenging for
$f_{\text{zs}}$: In its pre-training phase, the model has not been exposed to
the specific definition of background.

To facilitate our analysis, we breakdown the logits of fine-tuned models into
$\mathbf{o}_{\text{sg}}^{0}$ and $\mathbf{o}_{\text{sg}}^{k}$ to represent
background class and $K$ relation classes, respectively. The inference process
of the two branches can be formulated by:

$\left\\{\begin{aligned}
\mathbf{o}_{\text{zs}}^{k}&=f_{\text{zs}}(z_{i},z_{j},\mathbf{x}_{i,j})\in\mathbb{R}^{K}\\\
[\mathbf{o}_{\text{sg}}^{0},\mathbf{o}_{\text{sg}}^{k}]&=f_{\text{sg}}(z_{i},z_{j},\mathbf{x}_{i,j})\in\mathbb{R}^{K+1},\end{aligned}\right.$
(1)

The logits predicted by the two branches reflect the label distribution of
their training sets, leading to potential relation label bias if the target
distribution differs. To address this, in Section 3.3, we obtain the debiased
logits $\mathbf{\hat{o}}_{\text{zs}}^{k}$ and
$\mathbf{\hat{o}}_{\text{sg}}^{k}$ through a post-hoc logits adjustments. To
obtain $\mathbf{\hat{o}}_{\text{zs}}^{k}$, we propose our Lagrange-Multiplier
Estimation (LM Estimation) method to approximate the relation words’
distribution in the unobservable pre-training language set, and then use it
for words’ debiasing.

Subsequently, to mitigate the underrepresentation issue, we ensemble the
debiased logits from each branch to yield the final enhanced prediction. To
optimally weight each model’s representation based on its reliability towards
each sample, we employ a certainty-aware indicator to dynamically adjust the
ensemble weights. Specifically, for each input tuple
$(z_{i},z_{j},\mathbf{x}_{i,j})$ and its associated unbiased predictions
$\mathbf{\hat{o}}_{\text{zs}}^{k}$ and $\mathbf{\hat{o}}_{\text{sg}}^{k}$, a
certainty score is derived, which acts as the proportional weighting factor
for the two branches. We provide a detailed description of the ensemble
strategy in Section 3.4.

###  3.3 Relation Words Debiasing

Problem Definition.  For each subject-object pair that has a non-background
relation, we denote its relation word or label as $r\in\mathcal{C}_{r}$. Given
the logits $\mathbf{o}^{k}$ of $K$ relation classes, the conditional
probability $P_{\text{tr}}$ of the training set $\mathcal{D}_{\text{tr}}$ is
estimated by:

$P_{\text{tr}}(r|z_{i},z_{j},\mathbf{x}_{i,j})=\text{softmax}(\mathbf{o}^{k})_{r}.$
(2)

In our task, the training set $\mathcal{D}_{\text{tr}}$ can be either the SGG
dataset $\mathcal{D}_{\text{sg}}$ or the pre-training set
$\mathcal{D}_{\text{pt}}$, on which the task-specific model $f_{\text{sg}}$
and the zero-shot model $f_{\text{zs}}$ are trained.

In the evaluation phase, our goal is to estimate the actual test probability
$P_{\text{te}}$ rather than $P_{\text{tr}}$. By Bayes’ Rule, we have
$P(r|z_{i},z_{j},\mathbf{x}_{i,j})\propto
P(z_{i},z_{j},\mathbf{x}_{i,j}|r)\cdot P(r)$, where
$P\in\\{P_{\text{tr}},P_{\text{te}}\\}$. The relation-conditional probability
$P(z_{i},z_{j},\mathbf{x}_{i,j}|r)$ can be assumed as the same in both
training and testing. By change of variables, we have:

$\frac{P_{\text{tr}}(r|z_{i},z_{j},\mathbf{x}_{i,j})}{P_{\text{tr}}(r)}=\frac{P_{\text{te}}(r|z_{i},z_{j},\mathbf{x}_{i,j})}{P_{\text{te}}(r)}$
(3)

In a case where training marginal distribution $P_{\text{tr}}(r)$ not equals
to the test distribution $P_{\text{te}}(r)$, known as label shift, results in
the model’s predicted probability
$P_{\text{tr}}(r|z_{i},z_{j},\mathbf{x}_{i,j})\neq
P_{\text{te}}(r|z_{i},z_{j},\mathbf{x}_{i,j})$, the actual test probability.

In our framework in Figure 2, $f_{\text{zs}}$ is trained on
$\mathcal{D}_{\text{pt}}$ and $f_{\text{sg}}$ on $\mathcal{D}_{\text{sg}}$.
Their respective training label distributions $P_{\text{tr}}(r)$ can be
denoted by $\mathbf{\pi_{\text{pt}}}\in\mathbb{R}^{K}$ and
$\mathbf{\pi_{\text{sg}}}\in\mathbb{R}^{K}$. When evaluated by Recall, the
test label distribution $P_{\text{te}}(r)$ matches the training label
distribution $\mathbf{\pi_{\text{sg}}}$. If evaluated by mean Recall,
$P_{\text{te}}(r)$ becomes a constant $1/K$ indicating a uniform distribution.
The $P_{\text{tr}}(r)$ and $P_{\text{te}}(r)$ in each case can be summarized
as follow:

${P_{\text{tr}}(r)}=\begin{cases}{\mathbf{\pi_{\text{sg}}}},&\text{if
}\mathbf{o}_{\text{sg}}^{k}\\\ {\mathbf{\pi_{\text{pt}}}},&\text{if
}\mathbf{o}_{\text{zs}}^{k}\end{cases},{P_{\text{te}}(r)}=\begin{cases}{\mathbf{\pi_{\text{sg}}}},&\text{Recall}\\\
\frac{1}{K},&\text{mRecall}\end{cases}$ (4)

From Equation 4, we see the $P_{\text{te}}(r)\neq P_{\text{tr}}(r)$ happens in
the following cases.

  * •

Using mean recall to assess the task-specific model $f_{\text{sg}}$ may
uncover label shift from $\mathcal{D}_{\text{sg}}$ to
$\mathcal{D}_{\text{te}}$. In this case, the test distribution
$P_{\text{te}}(r)=1/K$ should reflect a uniform relation classes’
distribution, which is different from the imbalanced relation class
distribution in SGG training set, leading to $P_{\text{sg}}(r)\neq
P_{\text{te}}(r)$.

  * •

The pre-training set $\mathcal{D}_{\text{pt}}$ is not a SGG training set. It
does not share the same label distribution with the SGG test set
$\mathcal{D}_{\text{sg}}$, leading to $P_{\text{pt}}(r)\neq P_{\text{te}}(r)$.
Since $\mathcal{D}_{\text{pt}}$ does not have a uniform relation distribution,
it also occurs when using mean Recall, where $P_{\text{te}}(r)=1/K$.

Post-hoc Logits Adjustments.  The first case, where $P_{\text{sg}}(r)\neq
P_{\text{te}}(r)$, is a long-existing issue with many effective approaches
proposed in SGG Tang et al. (2020); Dong et al. (2022); Li et al. (2021); Yan
et al. (2020); Li et al. (2022b, a). Nevertheless, existing methods are not
feasible to the second case. While these methods are designed to reduce the
bias in training stage, the pre-training stage of $f_{\text{zs}}$ is not
accessible. A straightforward debias method for trained models is the post-hoc
logit adjustment Menon et al. (2020). Denoting the raw prediction and the
test-fitted prediction logits as $\mathbf{o}^{k}$ and $\mathbf{\hat{o}}^{k}$,
we can recast Equation 2 into a logistic form:

$\mathbf{\hat{o}}^{k}(r)=\mathbf{o}^{k}(r)-\log P_{\text{tr}}(r)+\log
P_{\text{te}}(r)$ (5)

It suggests that for any test set, given the label distribution, the unbiased
prediction $\mathbf{\hat{o}}^{k}(r)$ can be obtained through an post-hoc
adjustment on the raw prediction logits $\mathbf{o}^{k}(r)$. Where the
adjustment value is determined by Equation 4.

In Equation 4, $\mathbf{\pi_{\text{sg}}}$ can be simply obtained by counting
the relation frequencies in $\mathcal{D}_{\text{sg}}$. However, the
$\mathbf{\pi_{\text{pt}}}$ is the words’ preference hidden in the VLMs, which
is unobservable and driven by pre-training tasks that are distinct with SGG.
Therefore, a method is needed to estimate the $\mathbf{\pi_{\text{pt}}}$ for
removing the relation words bias in $f_{\text{zs}}$.

Lagrange Multiplier Estimation.  To estimate $\mathbf{\pi_{\text{pt}}}$, we
proposed a novel method based on constrained optimization. We start by
gathering all pairs with relation labels $r\in\mathcal{C}_{r}$ from
$\mathcal{D}_{\text{sg}}$’s validation set. Consequently, our optimization
objective is set to solve a $\mathbf{\pi_{\text{pt}}}$ that minimize the
cross-entropy loss between the logits $\mathbf{\hat{o}}^{k}$ adjusted by
$\mathbf{\pi_{\text{pt}}}$ (following Equation 5) and the ground-truth labels
$r$. Given that the relation distribution of the collected data is also
$\mathbf{\pi_{\text{sg}}}$ (since from $\mathcal{D}_{\text{sg}}$ val set), we
designate the term $P_{\text{te}}(r)$ to be $\mathbf{\pi_{\text{sg}}}$ to make
sure the solved $P_{\text{tr}}(r)=\mathbf{\pi_{\text{pt}}}$.

Through this, we can estimate the $\mathbf{\pi_{\text{pt}}}$ by solving a
constrained optimization problem. The equality and inenquality constraints are
set to ensure the solved $\mathbf{\pi_{\text{pt}}}$ representing a valid label
probability distribution:

$\displaystyle\mathbf{\pi_{\text{pt}}}=\>\underset{\mathbf{\pi_{\text{pt}}}}{\mathrm{argmin}}\>R_{ce}(\mathbf{o}^{k}-\log\mathbf{\pi_{\text{pt}}}+\log\mathbf{\pi_{\text{sg}}},\>r),$
$\displaystyle s.t.\>\mathbf{\pi_{\text{pt}}}(r)\geq
0,\>\mathrm{for}\>r\in\mathcal{C}_{r},\>\sum_{r\in\mathcal{C}_{r}}{\mathbf{\pi_{\text{pt}}}(r)}=1$
(6)

where $R_{ce}$ represents for the empirical risk computed by cross-entropy
loss. The constrained optimization problem can be solved through Lagrange
Multiplier method by rewritting Section 3.3 to:

$\displaystyle\mathbf{\pi_{\text{pt}}}=\>\underset{\mathbf{\pi_{\text{pt}}}}{\mathrm{argmin}}$
$\displaystyle\underset{\lambda_{r}\geq
0,v}{\mathrm{max}}R_{ce}-\sum_{r}{\lambda_{r}\mathbf{\pi_{\text{pt}}}(r)}$
$\displaystyle+v(1-\sum_{r}{\mathbf{\pi_{\text{pt}}}(r)})$ (7)

After obtaining $\mathbf{\pi_{\text{pt}}}$ and $\mathbf{\pi_{\text{sg}}}$, we
can then use the post-hoc logits adjustments for relation words debiasing
following Equation 5 and Equation 4.

###  3.4 Certainty-aware Ensemble

Upon mitigating the relation words bias inside $f_{\text{zs}}$, we can
leverage it to address the underrepresentation in $f_{\text{sg}}$. From the
logits $\mathbf{\hat{o}}_{\text{zs}}^{k}$ and
$\mathbf{\hat{o}}_{\text{sg}}^{k}$, we can obtain the unbiased probabilities
predicted towards $r\in\mathcal{C}_{r}$. Here we adopt a calibration parameter
$\tau$ outlined in Kumar et al. (2022) to avoid over-confidence problem. From
the logits
$\>\mathbf{\hat{o}}^{k}\in\\{\mathbf{\hat{o}}_{\text{zs}}^{k},\mathbf{\hat{o}}_{\text{sg}}^{k}\\}$
predicted by $\\{f_{\text{zs}},f_{\text{sg}}\\}$, the probabilities are
computed by:

${P_{\text{te}}}(r|z_{i},z_{j},\mathbf{x}_{i,j})=\text{softmax}(\mathbf{\hat{o}}^{k}/\tau)_{r}$
(8)

where $P_{\text{te}}\in\\{\hat{P_{\text{zs}}},\hat{P_{\text{sg}}}\\}$
represents the debiased probabilities from
$\\{f_{\text{zs}},f_{\text{sg}}\\}$. Based the two probabilities, we propose a
certainty-aware ensemble strategy to further generate the representation-
enhanced prediction.

Ensemble Strategy.  Dynamic ensemble has been a well-established method for
addressing Out-of-Distribution challengesHendrycks and Gimpel (2016);
Hendrycks et al. (2019, 2021). Considering that each model may be better at
different samples, it becomes essential to compute a dynamic score for each
sample based on its certainty in the two models. This score subsequently
informs the proportional weighting of the two models during ensemble.

Inspired by Hendrycks and Gimpel (2016), we leverage the confidence score of
the probability
$P_{\text{te}}\in\\{\hat{P_{\text{zs}}},\hat{P_{\text{sg}}}\\}$ obtained in
Equation 8 to compute the certainty-aware ensemble weight $W_{\text{cer}}$:

$\text{conf}=\underset{r\in\mathcal{C}_{r}}{\mathrm{max}}{P_{\text{te}}}(r|z_{i},z_{j},\mathbf{x}_{i,j}),$
(9)
$W_{\text{cer}}\propto\>\text{sigmoid}(\alpha*(\text{conf}_{\text{sg}}-\text{conf}_{\text{zs}}))\\\
$ (10)

The weight is then used in our ensemble to determine each model’s proportion.
Based on the certainty-aware weight, we ensemble the two models following:

$\displaystyle P_{\text{ens}}(r$
$\displaystyle|z_{i},z_{j},\mathbf{x}_{i,j})=W_{\text{cer}}*\hat{P_{\text{sg}}}(r|z_{i},z_{j},\mathbf{x}_{i,j})$
$\displaystyle+(1-W_{\text{cer}})*\hat{P_{\text{zs}}}(r|z_{i},z_{j},\mathbf{x}_{i,j})$
(11)

to obtain the representation-improved probabilities on $\mathcal{C}_{r}$.
Since $f_{\text{zs}}$ does not have the ability to predict the “background"
label, we relying solely on $f_{\text{sg}}$ to compute its probability.
Denoting
$\mathbf{o}_{\text{sg}}=[\mathbf{o}_{\text{sg}}^{0},\mathbf{o}_{\text{sg}}^{k}]$
as the raw logits predicted by $f_{\text{sg}}$ before label debiasing, the
background and foreground probability can be computed by softmax function:

$\left\\{\begin{aligned} &P_{\text{sg}}(y\neq
0|z_{i},z_{j},\mathbf{x}_{i,j})=1-\text{sfmx}(\mathbf{o}_{\text{sg}})_{0}\\\
&P_{\text{sg}}(y=0|z_{i},z_{j},\mathbf{x}_{i,j})=\text{sfmx}(\mathbf{o}_{\text{sg}})_{0}\end{aligned}\right.$
(12)

Finally, we can obtain the unbiased ensembled probability
$P_{\text{ens}}(y|z_{i},z_{j},\mathbf{x}_{i,j})$ on $\mathcal{C}_{e}$ through:

$\displaystyle P_{\text{ens}}$
$\displaystyle(y|z_{i},z_{j},\mathbf{x}_{i,j})=[P_{\text{sg}}(y=0|z_{i},z_{j},\mathbf{x}_{i,j}),$
$\displaystyle P_{\text{sg}}$ $\displaystyle(y\neq
0|z_{i},z_{j},\mathbf{x}_{i,j})\cdot
P_{\text{ens}}(r|z_{i},z_{j},\mathbf{x}_{i,j})]$ (13)

which serves as the final representation-enhanced prediction of our proposed
framework.

Models | Predicate Classification | Scene Graph Classification | Models | Predicate Classification | Scene Graph Classification
---|---|---|---|---|---
R@50 | R@100 | R@50 | R@100 | mR@50 | mR@100 | mR@50 | mR@100
KERN | 65.8 | 67.6 | 36.7 | 37.4 | VTransE | 17.1 | 18.6 | 8.2 | 8.7
R-CAGCN | 66.6 | 68.3 | 38.3 | 39.0 | SG-CogTree | 28.4 | 31.0 | 15.7 | 16.7
GPS-Net | 66.9 | 68.8 | 39.2 | 40.1 | BGNN | 30.4 | 32.9 | 14.3 | 16.5
VTransE | 65.7 | 67.6 | 38.6 | 39.4 | PCPL | 35.2 | 37.8 | 18.6 | 19.6
VCTree | 66.4 | 68.1 | 38.1 | 38.8 | Motifs-Rwt | 33.7 | 36.1 | 17.7 | 19.1
MOTIFS | 66.0 | 67.9 | 39.1 | 39.9 | Motifs-GCL | 36.1 | 38.2 | 20.8 | 21.8
SGGNLS | 65.6 | 67.4 | 40.0 | 40.8 | VCTree-TDE | 25.4 | 28.7 | 12.2 | 14.0
RU-Net | 68.1 | 70.1 | 41.2 | 42.1 | VCTree-GCL | 37.1 | 39.1 | 22.5 | 23.5
PENET | 68.2 | 70.1 | 41.3 | 42.3 | PENET-Rwt | 38.8 | 40.7 | 22.2 | 23.5
Oscar ft | 65.7 | 67.6 | 40.3 | 41.3 | Oscar ft-la | 38.4 | 41.3 | 22.6 | 23.8
Oscar ft + Ours | 67.4(+1.8) | 69.3(+1.7) | 41.4(+1.1) | 42.3(+1.0) | Oscar ft-la + Ours | 39.4(+1.0) | 42.7(+1.4) | 23.4(+0.8) | 25.0(+1.2)
ViLT ft | 65.7 | 68.4 | 40.2 | 41.8 | ViLT ft-la | 40.5 | 44.5 | 22.5 | 24.3
ViLT ft + Ours | 66.7(+1.0) | 69.8(+1.4) | 41.2(+1.0) | 42.9(+1.1) | ViLT ft-la + Ours | 42.3(+1.8) | 46.5(+2.0) | 23.5(+1.0) | 25.5(+1.2)
PENET | 68.2 | 70.1 | 41.3 | 42.3 | PENET-Rwt | 38.8 | 40.7 | 22.2 | 23.5
PENET + Ours | 69.0(+0.8) | 71.1(+1.0) | 41.8(+0.5) | 42.9(+0.6) | PENET-Rwt + Ours | 39.9(+1.1) | 42.3(+1.6) | 23.0(+0.8) | 24.5(+1.0)

Table 1: The Recall (R@K) and mRecall (mR@K) results on Visual Genome dataset
comparing with previous models and debiasing methods. The performance and gain
of our method is below each corresponding baseline. ft: The model is fine-
tuned on Visual Genome. la: The prediction logits is debiased by logits
adjustment with $\mathbf{\pi_{\text{sg}}}$.

##  4 Experiment Settings

To demostrate the efficacy of our method, we conduct comprehensive experiments
on SGG. In Section 5.1, we show our significant performance improvement
through a comparative analysis with previous methods. Concurrently, Section
5.2 offers an ablation study, showing the contribution of individual
components in our design to the overall performance.

Datasets.  We tested on Visual Genome Krishna et al. (2017) and Open Images V6
Kuznetsova et al. (2020) featuring extensive object and predicate categories.
For Visual Genome, we adopted a split with 108,077 images focusing on the most
common 150 object and 50 predicate categories, allocating 70% for training and
30% for testing, alongside a validation set of 5,000 images. Open Images V6
comprises 126,368 training, 1,813 validation, and 5,322 test images. Detailed
results and implementation for Open Images are in the supplementary material.

Evaluation on Visual Genome.  For the Visual Genome dataset, we focus on two
key sub-tasks: Predicate Classification (PredCls) and Scene Graph
Classification (SGCls). We skip the Scene Graph Detection (SGDet) here and
present a brief result in supplementary, considering its substantial
computational demands when employing VLMs and limited relevance to our
method’s core objectives.

Implementation Details.  In our framework illustrated in Figure 2, we utilize
two prominent vision-language models, ViLT Kim et al. (2021) and Oscar Li et
al. (2020). We directly use their zero-shot models as $f_{\text{zs}}$. For the
task-specific branch $f_{\text{sg}}$, we employs three baseline models as
$f_{\text{sg}}$: (1) We fine-tune the ViLT and Oscar using PredCls data and
establish them as our first two baselines. (2) Then to show our methods’
compatibility with existing SGG models, we undertake PENET Zheng et al.
(2023), a cutting-edge method with superior performance, as our third
baseline.

During pre-training, both ViLT and Oscar employ two main paradigms: Masked
Language Modeling (MLM) and Visual Question Answering (VQA). In MLM, tokens in
a sentence can be replaced by [MASK], with the model predicting the original
token using visual and language prompts. In VQA, the model, given a question
and visual input, predicts an answer via an MLP classifier using the [CLS]
token. For our task, we use MLM for the fixed branch $f_{\text{zs}}$ with the
prompt “$z_{i}$ is [MASK] $z_{j}$.” and VQA for the first two $f_{\text{sg}}$,
where we introduce a MLP head with the query "[CLS] what is the relationship
between the $z_{i}$ and the $z_{j}$?". For ensemble, we employ three
combinations: "fine-tuned ViLT + zero-shot ViLT", "fine-tuned Oscar + zero-
shot Oscar", and "PENET + zero-shot ViLT".

##  5 Results and Analysis

###  5.1 Efficacy Analysis

To validate the efficacy of our proposed method, in this section, we compare
our method with recent studies through a detailed result analysis on Visual
Genome. We put the Open Image results to Supp. The Recall and mean Recall
results are in Table 1, showcasing a performance comparison with a range of
cutting-edge models and debiasing methods.

Baseline. We first evaluate our three $f_{\text{sg}}$ baselines. Specifically,
when the target is a balanced distribution (mRecall), the post-hoc logits
adjustment is employed to the fine-tuned VLM baselines following Equation 4.
The two fine-tuned VLMs, though without task-specific designs and debiased by
simple post-hoc logits adjustments, still outperform all the cutting-edge
combinations on mean Recall, achieving 41.3 and 44.5 on mR@100. In contrast,
the two VLMs baseline fall behind the task-specific models on Recall, scored
67.6 and 68.4 on R@100, while PENET takes the lead.

Improvements. Then we employ our certainty-aware ensemble to integrate
debiased zero-shot VLMs $f_{\text{zs}}$ into the baseline models. All the two
$f_{\text{zs}}$ are debiased by our LM Estimation. For each $f_{\text{sg}}$
baseline, we observed a notable boost in performance after applying our
ensemble (+1.4/+2.0/+1.6 on mR@100 and +1.7/+1.4/+1.0 on R@100), indicating
its effectiveness in SGG’s representation enhancement. Moreover, the
improvement based on PENET shows our method’s adaptability to existing SGG-
specialized models. On both mRecall and Recall, our method have achieved the
best performance (46.5 on mR@100 and 71.1 on R@100), while the improvement on
mean Recall is particularly striking and surpasses the gains observed on
Recall (+1.4/+2.0/+1.6 vs. +1.7/+1.4/+1.0).

Our results suggest that our debiased VLMs contribute to enhance the
representation, leading to markable improvement on overall performance.
Besides, we observe that the underrepresentation problem is more common on
tail relation classes in the imbalanced class distribution. As a result, the
enhancement on representation leads to a more significant improvement in mean
Recall than Recall.

Models | All mAcc | All Acc | Unseen mAcc | Unseen Acc
---|---|---|---|---
w/o | LM E. | w/o | LM E. | w/o | LM E. | w/o | LM E.
ViLT-ft | 46.53 | 68.92 | 14.98 | 17.72
ViLT-zs | 21.88 | 37.42 | 57.15 | 67.09 | 8.99 | 16.92 | 18.81 | 20.93
ViLT-ens | 46.86 | 48.70 | 68.95 | 70.75 | 15.66 | 20.07 | 20.01 | 21.73
Ens. Gain | +0.33 | +2.17 | +0.03 | +1.83 | +0.68 | +5.09 | +2.29 | +4.01
Oscar-ft | 41.99 | 67.16 | 13.85 | 18.01
Oscar-zs | 17.18 | 33.96 | 45.78 | 57.31 | 6.68 | 16.01 | 19.11 | 20.05
Oscar-ens | 42.02 | 44.28 | 67.77 | 69.03 | 14.83 | 19.56 | 20.97 | 22.08
Ens. Gain | +0.03 | +3.29 | +0.61 | +1.87 | +0.98 | +5.71 | +2.96 | +4.07

Table 2: Top-1 accuracy and class-wise mean accuracy of relation
classification on Visual Genome. All: The results for all triplets with non-
background relation labels in test set. Unseen: The results for those triplets
absent from training set. w/o: The zero-shot prediction of pre-trained model
without debiasing. LM E.: The zero-shot prediction after debiasing with our LM
Estimation. ens: the ensemble results of the fine-tuned model with
corresponding zero-shot VLMs. Ens. Gain: the performance gain of ensemble
compared with the fine-tuned baseline performance.

###  5.2 Ablation Study

In our ablation study on Visual Genome, we evaluate our method’s ability to
improve representation, particularly focusing on unseen triplets absent during
training. We also assess how our LM Estimation technique addresses the
relation words bias in zero-shot VLMs. Besides Recall and mean Recall, we
introduce top-1 accuracy and mean accuracy as new metrics. These new metrics
help directly measure performance in relation classification by excluding the
"background" class, providing a clearer view on mitigating relation words
bias.

We present our findings in Table 2, detailing two distinct splits: one
including all triplets and the other focusing on unseen triplets. For each
split, we focus on the two fine-tuned VLMs $f_{\text{sg}}$ and their zero-shot
models $f_{\text{zs}}$. For each $f_{\text{sg}}$, we ensemble it with the raw
and debiased $f_{\text{zs}}$ for comparison.

Figure 3: The relation label distributions on Visual Genome. The upper figure
illustrates the distribution across all classes, while the lower one shows the
probability distribution on some typical categories. Train Set: The class
distribution $\mathbf{\pi_{\text{sg}}}$ in training set. ViLT and Oscar: The
estimated distribution $\mathbf{\pi_{\text{pt}}}$ using LM Estimation in the
two pre-training stages.

Representation Improvement.  To validate that the overall improvement comes
from representation enhancement, we specifically examine the performance on
unseen triplets. These triplets $(z_{i},z_{j},r)$ are present in the test set
but absent from the training set, consisting the worst tail samples in
underrepresentation issue. In Table 2, the all-triplets accuracy of both zero-
shot models, even after relation words debiasing, remains lower than the
corresponding fine-tuned baseline. For all triplets, the debiased zero-shot
Oscar achieves 33.96 mean accuracy (mAcc) and 57.31 overall accuracy (Acc),
outperforming the fine-tuned version’s 41.99 mAcc and 67.16 Acc. However, this
trend is inverted for unseen triplets, where the debiased zero-shot model
performs better. On unseen triplets, the zero-shot Oscar achieves 16.01 mAcc
and 20.05 Acc while the fine-tuned Oscar merely score 13.85 mAcc and 18.01
Acc. This confirms our hypothesis that with the completely preserved pre-
training knowledge, the zero-shot models $f_{\text{zs}}$ are better on the
underrepresented samples than SGG-specific models $f_{\text{sg}}$.

Moreover, when looking at the performance gains of ensemble, we find that the
improvement is significantly higher for unseen triplets (+5.09/+4.01) than for
all triplets (+2.17/+1.83). Since unseen triplets represent the extreme case
of underrepresentation, their performance gain can confirm our representation
improvement.

Models | mR@20 | mR@50 | mR@100
---|---|---|---
ViLT-ft | 31.2 | 40.5 | 44.5
ViLT-ens (w/o) | 30.9(-0.3) | 40.5(+0.0) | 44.6(+0.1)
ViLT-ens (LM E.) | 32.3(+0.9) | 42.3(+1.8) | 46.5(+2.0)
Oscar-ft | 30.4 | 38.4 | 41.3
Oscar-ens (w/o) | 30.3(-0.1) | 38.5(+0.1) | 41.6(+0.3)
Oscar-ens (LM E.) | 31.2(+0.8) | 39.4(+1.0) | 42.7(+1.4)
Models | R@20 | R@50 | R@100
ViLT-ft | 57.1 | 65.7 | 68.4
ViLT-ens (w/o) | 56.9(-0.2) | 65.7(+0.0) | 68.8(+0.4)
ViLT-ens (LM E.) | 58.0(+0.9) | 66.7(+1.0) | 69.8(+1.4)
Oscar-ft | 59.1 | 65.7 | 67.6
Oscar-ens (w/o) | 59.2(+0.1) | 65.9(+0.2) | 67.9(+0.3)
Oscar-ens (LM E.) | 60.5(+1.4) | 67.4(+1.7) | 69.3(+1.7)

Table 3: The mean Recall and Recall ablation results on Visual Genome. w/o:
The raw zero-shot prediction of pre-trained model. LM E.: The zero-shot
prediction after class-label debiasing with our LM Estimation. ens: Ensemble
the fine-tuned VLMs and w/o or LM E. zero-shot prediction.

Relation Words Debiasing.  In Section 3.3, we introduce our LM Estimation
method, which estimate the pre-training label distribution and addresses the
relation words bias of zero-shot VLMs. But can we ensemble the zero-shot VLM
without relation words debiasing? Here we further explore the efficacy and
necessity of our debiasing method.

In Figure 3, we depict the solved relation words’ distributions of zero-shot
ViLT and Oscar, comparing them with the distribution in VG training set. The
upper chart in Figure 3 depicts the distributions across all relations. We
find that all the three distributions exhibit significant imbalance, while
there is a large difference in their predominant classes. Furthermore, we zoom
in the distribution of some typical relations in the lower chart. Here we see
a substantial distribution discrepancy between the three distribution. This
variation precludes the direct application of VLMs without debiasing,
indicating the necessity of our LM Estimation debiasing method.

In Table 2, we present the relation classification accuracies of the raw zero-
shot VLMs without debiasing. Then we compare their performance with that of
debiased zero-shot VLMs. Without debiasing, the accuracy of zero-shot models
are much lower compared with the corresponding fine-tuned models, either for
all triplets or unseen triplets. In contrast, after relation words debiasing,
the zero-shot VLMs are markably improved and outperform fine-tuned models on
unseen triplets. Through our debiasing, both zero-shot VLMs overcome the words
bias and align more closely with SGG data.

Furthermore, from the ensemble performance in Table 2, we notice that simply
ensembling the raw zero-shot VLMs without debiasing can hardly improve the
performance. Specifically, ensembling the raw zero-shot ViLT only achieves
+0.33/+0.03 gain on all triplets and +0.68/+2.29 on unseen triplets. In
contrast, ensembling the debiased $f_{\text{zs}}$ achieves a significantly
more pronounced improvement, where the debiased ViLT brings +2.17/+1.83 gain
on all triplets and +5.09/+4.01 on unseen triplets. To keep consistent with
previous setting, we present the results of our ablation study, assessed
through both Recall and mean Recall, in Table 3. Clearly, there’s a
substantial improvement in both mean Recall and Recall when ensembling with
our debiased zero-shot VLMs (the highlighted row in each group). Without
debiasing, the overall performance might even deteriorate compared to the
baseline (the middle row in each group). This starkly highlights the necessity
of our label-debiasing approach with the proxy pre-training label distribution
produced by LM Estimation.

##  6 Conclusion

In conclusion, our study has made significant strides in efficiently and
effectively integrate pre-trained VLMs to SGG. By introducing the novel LM
Estimation, we effectively mitigate the relation words bias inside pre-trained
VLMs’ pretraining language set, allowing their comprehensive knowledge to be
employed in SGG. Besides, our certainty-aware ensemble strategy, which
ensembles the zero-shot VLMs with SGG model, effectively addresses the
underrepresentation issue and demonstrates a significant improvement in SGG
performance. Our work contributes to the field of SGG, suggesting potential
pathways for reducing bias of VLMs and leverage them in more complex vision-
language tasks.

##  7 Limitation

Though our methods does not require any training, comparing with original
$f_{\text{sg}}$, our ensemble framework still adds computational cost from
$f_{\text{zs}}$’s inference. This inference can be costly in an extreme case
that one scene has too many objects to predict their relations. Besides, even
after we solve the word bias inside VLMs, the final ensemble performance
relies highly on the pre-training quality, which requires the $f_{\text{zs}}$
to be pre-trained on comprehensive data to improve SGG’s representation.

## References

  * Dong et al. (2022) Xingning Dong, Tian Gan, Xuemeng Song, Jianlong Wu, Yuan Cheng, and Liqiang Nie. 2022.  Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19427–19436.
  * Gao et al. (2023) Kaifeng Gao, Long Chen, Hanwang Zhang, Jun Xiao, and Qianru Sun. 2023.  Compositional prompt tuning with motion cues for open-vocabulary video relation detection.  _arXiv preprint arXiv:2302.00268_.
  * Gu et al. (2019) Jiuxiang Gu, Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, and Mingyang Ling. 2019.  Scene graph generation with external knowledge and image reconstruction.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1969–1978.
  * He et al. (2022) Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. 2022.  Towards open-vocabulary scene graph generation with prompt-based finetuning.  In _European Conference on Computer Vision_ , pages 56–73. Springer.
  * Hendrycks et al. (2019) Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. 2019.  Scaling out-of-distribution detection for real-world settings.  _arXiv preprint arXiv:1911.11132_.
  * Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. 2021.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8340–8349.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In _International Conference on Learning Representations_.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _International Conference on Machine Learning_ , pages 5583–5594. PMLR.
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ , 123:32–73.
  * Kumar et al. (2022) Ananya Kumar, Tengyu Ma, Percy Liang, and Aditi Raghunathan. 2022.  Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift.  In _Uncertainty in Artificial Intelligence_ , pages 1041–1051. PMLR.
  * Kuznetsova et al. (2020) Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  _International Journal of Computer Vision_ , 128(7):1956–1981.
  * Li et al. (2022a) Lin Li, Long Chen, Yifeng Huang, Zhimeng Zhang, Songyang Zhang, and Jun Xiao. 2022a.  The devil is in the labels: Noisy label correction for robust scene graph generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18869–18878.
  * Li et al. (2023) Lin Li, Jun Xiao, Guikun Chen, Jian Shao, Yueting Zhuang, and Long Chen. 2023.  Zero-shot visual relation detection via composite visual cues from large language models.  _arXiv preprint arXiv:2305.12476_.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019.  Visualbert: A simple and performant baseline for vision and language.  _arXiv preprint arXiv:1908.03557_.
  * Li et al. (2021) Rongjie Li, Songyang Zhang, Bo Wan, and Xuming He. 2021.  Bipartite graph network with adaptive message passing for unbiased scene graph generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11109–11119.
  * Li et al. (2022b) Wei Li, Haiwei Zhang, Qijie Bai, Guoqing Zhao, Ning Jiang, and Xiaojie Yuan. 2022b.  Ppdl: Predicate probability distribution based loss for unbiased scene graph generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19447–19456.
  * Li et al. (2020) Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16_ , pages 121–137. Springer.
  * Lin et al. (2020) Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. 2020.  Gps-net: Graph property sensing network for scene graph generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3746–3753.
  * Lin et al. (2022a) Xin Lin, Changxing Ding, Yibing Zhan, Zijian Li, and Dacheng Tao. 2022a.  Hl-net: Heterophily learning network for scene graph generation.  In _proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 19476–19485.
  * Lin et al. (2022b) Xin Lin, Changxing Ding, Jing Zhang, Yibing Zhan, and Dacheng Tao. 2022b.  Ru-net: Regularized unrolling network for scene graph generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19457–19466.
  * Menon et al. (2020) Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. 2020.  Long-tail learning via logit adjustment.  _arXiv preprint arXiv:2007.07314_.
  * Qi et al. (2020) Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. 2020.  Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data.  _arXiv preprint arXiv:2001.07966_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR.
  * Su et al. (2019) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019.  Vl-bert: Pre-training of generic visual-linguistic representations.  _arXiv preprint arXiv:1908.08530_.
  * Sun et al. (2018) Xudong Sun, Pengcheng Wu, and Steven CH Hoi. 2018.  Face detection using deep learning: An improved faster rcnn approach.  _Neurocomputing_ , 299:42–50.
  * Tang et al. (2020) Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. 2020.  Unbiased scene graph generation from biased training.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3716–3725.
  * Tang et al. (2019) Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. 2019.  Learning to compose dynamic tree structures for visual contexts.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6619–6628.
  * Xu et al. (2017) Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. 2017.  Scene graph generation by iterative message passing.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5410–5419.
  * Yan et al. (2020) Shaotian Yan, Chen Shen, Zhongming Jin, Jianqiang Huang, Rongxin Jiang, Yaowu Chen, and Xian-Sheng Hua. 2020.  Pcpl: Predicate-correlation perception learning for unbiased scene graph generation.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 265–273.
  * Yu et al. (2022) Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022.  Coca: Contrastive captioners are image-text foundation models.  _arXiv preprint arXiv:2205.01917_.
  * Yu et al. (2023) Qifan Yu, Juncheng Li, Yu Wu, Siliang Tang, Wei Ji, and Yueting Zhuang. 2023.  Visually-prompted language model for fine-grained scene graph generation in an open world.  _arXiv preprint arXiv:2303.13233_.
  * Zhang et al. (2023) Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang, Tao Mei, and Chang-Wen Chen. 2023.  Learning to generate language-supervised and open-vocabulary scene graph using pre-trained visual-semantic space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2915–2924.
  * Zhao et al. (2023) Long Zhao, Liangzhe Yuan, Boqing Gong, Yin Cui, Florian Schroff, Ming-Hsuan Yang, Hartwig Adam, and Ting Liu. 2023.  Unified visual relationship detection with vision and language models.  _arXiv preprint arXiv:2303.08998_.
  * Zheng et al. (2023) Chaofan Zheng, Xinyu Lyu, Lianli Gao, Bo Dai, and Jingkuan Song. 2023.  Prototype-based embedding network for scene graph generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22783–22792.