# TT-BLIP: Enhancing Fake News Detection Using BLIP and Tri-Transformer
††thanks:

Eunjee Choi  School of Electrical Engineering
Korea University
Seoul, South Korea
<EMAIL_ADDRESS>Jong-Kook Kim  School of Electrical Engineering
Korea University
Seoul, South Korea
<EMAIL_ADDRESS>
###### Abstract

Detecting fake news has received a lot of attention. Many previous methods
concatenate independently encoded unimodal data, ignoring the benefits of
integrated multimodal information. Also, the absence of specialized feature
extraction for text and images further limits these methods. This paper
introduces an end-to-end model called TT-BLIP that applies the bootstrapping
language-image pretraining for unified vision-language understanding and
generation (BLIP) for three types of information: BERT and BLIPTxt for text,
ResNet and BLIPImg for images, and bidirectional BLIP encoders for multimodal
information. The Multimodal Tri-Transformer fuses tri-modal features using
three types of multi-head attention mechanisms, ensuring integrated modalities
for enhanced representations and improved multimodal data analysis. The
experiments are performed using two fake news datasets, Weibo and Gossipcop.
The results indicate TT-BLIP outperforms the state-of-the-art models.

###### Index Terms:

multimodal fusion, vision-language pretraining, fake news detection

##  I Introduction

The expansion of digital platforms has altered news consumption by broadening
access and increasing exposure to misinformation, emphasizing the urgent need
for enhanced fake news detection. Social media advancements have evolved news
delivery from traditional text articles to multimedia narratives [1],
integrating images and videos. Images provide visual information [2] that
enhances the textual content and can attract more attention.

Fig. 1 shows two examples from the Weibo dataset. The first image and its text
are easily verified as true. However, the second image and text, if viewed
separately, may provide insufficient information to assess the authenticity of
the news content. By examining both the image and text together the
fabrication becomes clear, illustrating how combining image and textual
information improves fake news detection.

(a) Iris Grace, a British autistic girl with amazing painting talent.

(b) Eight party members and leading cadres were notified for disciplinary
issues.

Figure 1: Real news (a) and fake news (b) examples from the Weibo dataset

The evolution of fake news detection techniques has transitioned from
traditional approaches to more advanced deep learning methodologies.
Previously, [3],[4] primarily utilized text content to discern between fake
and real news. Building on this, Ma et al. [5] delved into the potential of
deep neural networks for representing tweets, emphasizing temporal-linguistic
attributes. Chen et al. [6] then augmented this approach by incorporating
attention mechanisms into RNN structures. Deep learning-based fake news
detection highlights notable enhancements in performance over their
conventional counterparts, attributing this to the superior feature extraction
capabilities of the newer methods. Jin et al. [2] adopted an end-to-end
approach by merging image, textual, and social context features, using
attention mechanisms to inform their predictions. Meanwhile, Wang et al. [7]
introduced an event discriminator that aimed to identify applicable features
across various events and thereby enhancing the model adaptability to novel
events. Existing models have a clear deficiency in specialized feature
extraction for both text and images and using cross modal attention has shown
its own set of challenges. These two main challenges made the effective
detection of fake news more difficult.

This paper introduces Tri-Transformer BLIP (TT-BLIP) shown in Fig. 2, which is
based on bootstrapping language-image pretraining for unified vision-language
understanding and generation [8] to address the challenges above. The proposed
model uses BLIP [8], BERT [9], and ResNet [10] for feature extraction. It is a
three-pathway model employing BERT and BLIPTxt for text feature extraction,
ResNet and BLIPImg for image feature extraction and a pair of bidirectional
BLIP encoders for image-text correlation information. The Multimodal Tri-
Transformer fuses features from text, image, and image-text modalities for
fake news detection. The text modality employs self multi-head attention for
internal analysis, while image and image-text modalities use cross-attention
with text queries, aligning visual content with textual context. The process
prioritizes text-driven analysis essential for evaluating multimodal fake news
data. After the attention, each modality’s output undergoes Multi-Layer
Perceptron (MLP) transformation, ensuring data uniformity. The processed
outputs from different modalities are then concatenated, leading to a unified
and comprehensive representation. Experiments are conducted using two
multimodal fake news datasets to assess the performance of TT-BLIP: Weibo [11]
and Gossipcop [12]. The results demonstrate that TT-BLIP is the state-of-the-
art model for fake news detection.

In summary, the main contributions of this paper are:

  * •

TT-BLIP is proposed, which uses the pre-trained BLIP model for feature
extraction to detect fake news.

  * •

A novel fusion mechanism is introduced, which is a Multimodal Tri-Transformer.
By fusing text, images, image-text features, the transformer captures the
semantic information of all three modalities.

  * •

The TT-BLIP is tested using two multimodal fake news, Weibo and Gossipcop, and
results show that TT-BLIP peforms better than the state-of-the-art multimodal
detection models.

##  II RELATED WORK

In the domain of Multimodal fake news detection, various strategies have
emerged that emphasize extracting features from both images and texts from
news articles. EANN [7] is designed to detect fake news from social media,
particularly those involving event-invariant features. MVAE [13] is an end-to-
end network developed for fake news detection that incorporates a bimodal
variational autoencoder and a binary classifier. Spotfake by Singhal et al.
[14] uses BERT [9] for textual feature extraction and VGG19 [15] for image
features to efficiently detect fake news. They later refined their approach in
Spotfake+ [16] to better identify fake news in full articles. SAFE [17]
explored the relationship between textual and image information in news
articles for fake news detection. CAFE [18] detects fake news from social
platforms by adapting to cross-modal ambiguities and analyzing uni-modal and
cross-modal features. LIIMR [19] selectively diminishes information from less
significant modalities, while emphasizing and extracting related data from the
dominant modality for each sample. MCAN [20] utilizes co-attention layers
within its three sub-networks to merge textual and image features. This
approach emphasizes capturing inter-dependencies across multimodal inputs for
fake news detection.

Some methods employ more information from datasets. DistilBert by Allein et
al. [21] utilize the latent representations of user-generated and shared
content to detect disinformation in online news articles. By correlating user
preferences and sharing behaviors, the model can efficiently differentiate
between genuine and fake news without relying on user profiling during
prediction. BDANN [22] tackles multimodal fake news detection on microblogging
platforms by combining features from two modalities and eliminating event
specific biases. FND-CLIP [23] applied the Contrastive Language-Image Pre-
Training (CLIP, [24]) vision-language pretraining model to measure the
correlation between images and texts and utilize different modalities for
decision-making. FND-CLIP used two unimodal encoders, two pair-wise CLIP
encoders, guides the network learning for various modalities, and adeptly
aggregates text, image, and fused features with a modality-wise attention
module.

Unlike previous models, TT-BLIP uses a three pathway model extracting features
from image, text, and image-text multimodal information and the TT-BLIP model
incorporates a Multimodal Tri-Transformer. This novel approach enables the
fusion of text, images, and image-text features, offering a more comprehensive
and effective method for utilizing multimodal information.

##  III METHOD

###  III-A Overview

In this paper, a new multimodal fake news detection model is proposed and Fig.
2 shows the overall architecture of TT-BLIP. TT-BLIP is composed of three
modules: feature extraction module, feature fusion module, and fake news
detector module. In the feature extraction module, image, text, and image-text
features are extracted from text articles and associated images. In the fusion
module, the MultiModal Tri-Transformer method is used to aggregate the output
of the feature extraction module. Finally, in the fake news detector, the
integrated features processed through the MultiModal Tri-Transformer are used
to predict whether the content is real or fake.

Figure 2: The architecture of the proposed TT-BLIP.

###  III-B Feature Extraction Layer

The feature extraction layer is comprised of text feature extraction layers,
image feature extraction layers, and image-text feature extraction layers. The
data used are denoted as $x_{\text{Img}}$ for images and $x_{\text{Txt}}$ for
text.

####  III-B1 Textual feature extractor

: BERT [9] and BLIPTxt [8] are used in parallel to capture a comprehensive
representation of the textual data.

  * •

BERT-based Feature Extraction: the pretrained BERT is used, bidirectional
transformer model used for feature extraction. From this, the feature
representation $z_{\text{t1}}=f_{\text{bert}}(x_{\text{Txt}})$ is obtained.
The strength of BERT is its bidirectional training that enables it to
understand words in their context and thus enhancing textual feature
extraction.

  * •

BLIPTxt-based Feature Extraction: BLIPTxt is used to derive
$z_{\text{t2}}=f_{\text{blip\\_T}}(x_{\text{Img}},x_{\text{Txt}})$. BLIP is
inherently designed to accept both text and images as inputs simultaneously.
However, to ensure that only textual information is considered, an image
tensor (dimension: $224\times 224\times 3$) filled with zeros is inserted for
the image data. This ensures that BLIP focuses solely on the textual data and
extracts relevant features accordingly.

####  III-B2 Image feature extractor

For the image data, ResNet [10] and BLIPImg are used in parallel.

  * •

ResNet-based Feature Extraction: the pretrained ResNet model is employed for
image feature extraction. The feature representation
$z_{\text{i1}}=f_{\text{resnet}}(x_{\text{Img}})$ is obtained. This choice
ensures that our model can effectively capture intricate patterns and features
from images.

  * •

BLIPImg-based Feature Extraction: the BLIPImg is used to derive
$z_{\text{i2}}=f_{\text{blip\\_i}}(x_{\text{Img}},x_{\text{Txt}})$, which
serves as an alternative approach for image feature extraction. BLIP is
designed to handle both text and images as input simultaneously. However, to
focus solely on extracting image features in this work a dummy text is input.
This process ensures that the textual aspect does not influence the extraction
of image features.

The two image and two textual feature representations are then integrated into
single image and textual representations, respectively. For image features,
this integrated representation is denoted as $z_{\text{i}}$, and for text
features, it is denoted as $z_{\text{t}}$. These integrated features are
achieved by concatenating the two respective feature information shown as
$z_{\text{i}}=[z_{i1};z_{i2}]$ for the image features and
$z_{\text{t}}=[z_{t1};z_{t2}]$ for text features. These are then used in
subsequent stages of the proposed model.

####  III-B3 Image-text feature extractor

: the pretrained BLIP model is used to extract image-text features, aiming to
combine image and textual information. This process is essential to create a
comprehensive representation that captures the interaction between image and
text data. The resulting Image-text feature representation integrates
information from both modalities and can be expressed as follows:
$z_{\text{m}}=f_{\text{blip}}(x_{\text{Img}},x_{\text{Txt}})$. Here,
$z_{\text{m}}$ represents the integrated feature representation.

###  III-C Feature Fusion Layer

In the feature fusion layer (Fig 3), the MultiModal Tri-Transformer is
introduced and this idea comes from MultiModal BiTransformer [25] that
augments BERT embeddings with image-centric embeddings, facilitating enhanced
representation learning. The Tri-Transformer uses three types of multi-
attention[26] mechanisms. It applies cross-modal attention between text and
both image-text and image modalities. This ensures the text modality, which is
crucial for the task, is emphasized more than the others while keeping the
image and image-text channels independent. For text alone, it employs self
multi-head attention to enhance textual analysis.

$\text{C\\_Attention}_{i-t}(Q,K,V)=\text{softmax}\left(\frac{Q_{t}K_{i}^{T}}{\sqrt{d_{h}}}\right)V_{i}$
(1)

$\text{C\\_Attention}_{m-t}(Q,K,V)=\text{softmax}\left(\frac{Q_{t}K_{m}^{T}}{\sqrt{d_{h}}}\right)V_{m}$
(2)

$\text{S\\_Attention}_{t}(Q,K,V)=\text{softmax}\left(\frac{Q_{t}K_{t}^{T}}{\sqrt{d_{h}}}\right)V_{t}$
(3)

Let $Q_{t}$ represent the query vector for text modality, used to extract
relevant information. The key vectors for the text, image, and image-text
modalities are denoted by $K_{t}$, $K_{i}$, and $K_{m}$, determining data
relevance. The value vectors for these modalities are represented by $V_{t}$,
$V_{i}$, and $V_{m}$, holding the actual data to be retrieved. The
$\text{softmax}(\cdot)$ function refers to the softmax function. The term
$d_{h}$ indicates the dimensionality of each modality, and $T$ stands for
transpose.

In the Transformer architecture, several attentions are computed in parallel,
with each attention’s output known as a head. The $num^{th}$ head is computed
as:

$\text{head}^{num}_{*}=\text{Attention}_{*}\left(Q_{t}W^{Q_{t}}_{num},K_{*}W^{K_{*}}_{num},V_{*}W^{V_{*}}_{num}\right)$
(4)

where $\mathbf{W}^{Qt}_{num}\in\mathbb{R}^{d_{t}\times d_{q}}$ is the weight
matrix of $Q_{t}$ when computing the head of the $num^{th}$ text modality;
$\mathbf{W}^{K*}_{num}\in\mathbb{R}^{d_{*}\times d_{k}}$ be the weight matrix
of $K_{*}$ when computing the head of the $num^{th}$ $*$ modality; and
$\mathbf{W}^{V*}_{num}\in\mathbb{R}^{d_{*}\times d_{v}}$ be the weight matrix
of $V_{*}$ when computing the head of the $num^{th}$ $*$ modality, where
$*\in\\{i,t,m\\}$.

Following this, all heads of the $*$ modalities are connected, denoted as
$Y_{*}$, as follows:

$\begin{split}Y_{*}&=\text{MultiHead}(Q_{t},K_{*},V_{*})\\\
&=\text{Concat}(\text{head}^{1}_{*},\text{head}^{2}_{*},\ldots,\text{head}^{n}_{*})\mathbf{W}_{O_{*}},\end{split}$
(5)

Where $\mathbf{W}{O_{*}}$ is the weight matrix multiplied after splicing the
head of $*$ modalities and $n$ denotes the number of self-attention heads
used. Thus, the text-based image representation $f_{it}$, the text-based
image-text representation $f_{mt}$, and text representation $f_{t}$ can be
obtained as follows:

$\displaystyle f_{at}$
$\displaystyle=\text{MultiHead}(Y_{i};\theta_{c-\text{att}}^{i}),$ (6)
$\displaystyle f_{vt}$
$\displaystyle=\text{MultiHead}(Y_{m},\theta_{c-\text{att}}^{m}),$ (7)
$\displaystyle f_{t}$
$\displaystyle=\text{MultiHead}(Y_{t},\theta_{\text{att}}^{t})$ (8)

where
$\theta_{\text{att}}^{i}=\\{\mathbf{W}_{Q}^{i},\mathbf{W}_{K}^{i},\mathbf{W}_{V}^{i},\mathbf{W}_{O}^{i}\\}$
and
$\theta_{\text{att}}^{m}=\\{\mathbf{W}_{Q}^{m},\mathbf{W}_{K}^{m},\mathbf{W}_{V}^{m},\mathbf{W}_{O}^{m}\\}$
represent the main hyperparameters required for the cross-attention module.
$\theta_{\text{att}}^{t}=\\{\mathbf{W}_{Q}^{t},\mathbf{W}_{K}^{t},\mathbf{W}_{V}^{t},\mathbf{W}_{O}^{t}\\}$
is used for the self-attention module.

After the multi-head attention step, each modality’s output is passed through
its own MLP (Multi Layer Perceptron) layer. The MLP layer perform linear
transformations on data, integrating features extracted from previous layers
to ensure a uniform format and facilitate the generation of the final output.
The processed outputs are finally concatenated into a unified tensor.

###  III-D Fake News Detector

A fake news detector is used to verify the authenticity of multimodal news
articles. The combined data from the fusion layer is sent to a classifier,
which comprises of three linear layers with ReLU activations, batch
normalization, and outputs binary classification for fake news detection. The
label set is denoted by $\mathcal{Y}$. Specifically, a news article that is
classified as ’fake’ is assigned a label of 1, and 0 otherwise. To assess the
difference between the predicted and actual labels, the cross-entropy loss is
used and is formulated as:

$L_{\text{cls}}=-\sum_{(p,y_{i})\in(\mathcal{P},\mathcal{Y})}\left[y\cdot\log(\hat{y}_{i})+(1-y)\cdot\log(1-\hat{y}_{i})\right]$
(9)

TABLE I: Experimental Results on Weibo and Gossipcop Datasets
’-’ symbol indicates that the results are not available from the original paper. Dataset | Method | Accuracy | Fake News | Real News
---|---|---|---|---
|  |  | Precision | Recall | F1 Score | Precision | Recall | F1 Score
Weibo |  EANN [7] | 0.827 | 0.847 | 0.812 | 0.829 | 0.807 | 0.843 | 0.825
MVAE [13] | 0.824 | 0.854 | 0.769 | 0.809 | 0.802 | 0.875 | 0.837
Spotfake [14] | 0.892 | 0.902 | 0.964 | 0.932 | 0.847 | 0.656 | 0.739
SAFE [17] | 0.762 | 0.831 | 0.724 | 0.774 | 0.695 | 0.811 | 0.748
BDANN [22] | 0.821 | 0.790 | 0.610 | 0.690 | 0.830 | 0.920 | 0.870
LIIMR [19] | 0.900 | 0.882 | 0.823 | 0.847 | 0.908 | 0.941 | 0.925
MCAN [20] | 0.899 | 0.913 | 0.889 | 0.901 | 0.884 | 0.909 | 0.897
CAFE [18] | 0.840 | 0.855 | 0.830 | 0.842 | 0.825 | 0.851 | 0.837
FND-CLIP [23] | 0.907 | 0.914 | 0.901 | 0.908 | 0.914 | 0.901 | 0.907
TT-BLIP(VGG) | 0.960 | 0.963 | 0.952 | 0.957 | 0.951 | 0.963 | 0.957
TT-BLIP(XLNet) | 0.957 | 0.963 | 0.952 | 0.957 | 0.951 | 0.963 | 0.957
TT-BLIP(ours) | 0.961 | 0.979 | 0.944 | 0.961 | 0.944 | 0.980 | 0.962
Gossipcop |  Spotfake+ [16] | 0.856 | - | - | - | - | - | -
LSTM-ATT [27] | 0.842 | - | - | - | 0.839 | 0.842 | 0.821
SAFE [17] | 0.838 | 0.758 | 0.558 | 0.643 | 0.857 | 0.937 | 0.895
DistilBert [21] | 0.857 | 0.805 | 0.527 | 0.637 | 0.866 | 0.960 | 0.911
CAFE [18] | 0.867 | 0.732 | 0.490 | 0.587 | 0.887 | 0.957 | 0.921
FND-CLIP [23] | 0.880 | 0.761 | 0.549 | 0.638 | 0.899 | 0.959 | 0.928
TT-BLIP(VGG) | 0.846 | 0.842 | 0.552 | 0.667 | 0.847 | 0.960 | 0.900
TT-BLIP(XLNet) | 0.865 | 0.655 | 0.792 | 0.717 | 0.875 | 0.933 | 0.903
TT-BLIP(ours) | 0.885 | 0.737 | 0.596 | 0.659 | 0.910 | 0.950 | 0.930

##  IV EXPERIMENTS AND RESULTS

###  IV-A Dataset

####  IV-A1 Weibo

The Weibo dataset was introduced by Jin et al. [11]. The dataset aggregates
authentic news from esteemed Chinese news outlets, with the Xinhua News Agency
being a prime example. The deceptive news articles were crawled and
authenticated by Weibo’s official rumor verification system from May 2012 to
January 2016. This system encourages general users to report potential
misinformation, which is then assessed by a panel of reputable users. The
experimental training set consists of 6,137 news articles, where there are
2,802 fake and 3,335 real news pieces, while the testing set comprises of 833
fake and 852 real news.

####  IV-A2 Gossipcop

Gossipcop dataset is used in this paper ([12]). There are 10010 news articles
in the training set, where 2036 are fake and 7974 are real news pieces. The
testing set consists of 545 fake and 2285 real news.

###  IV-B Experimental Settings

For text feature extraction, the pretrained BERT model which is based on
Chinese was used for the Weibo dataset, and the ”bert-base-uncased” model for
the Gossipcop dataset. In terms of image feature extraction, the input image
size is set to 224 × 224 and ResNet [10] was used. Lastly, image-text features
are extracted by pairing images and text and processing them with the
pretrained BLIP [8] model. Because the BLIP is pretrained using English text,
for the Weibo dataset the Google Translation API is utilized to translate
Chinese texts into English. The model is trained using the Adam optimizer with
default parameters, a learning rate of $1\times 10^{-3}$ and a batch size of
64.

###  IV-C Results and Analysis

TT-BLIP is compared to the state-of-the-art models which are presented in
Table I to validate the performance. The experiments are performed using the
Weibo and the Gossipcop datasets. For the evaluation, accuracy, precision,
recall, and F1 scores are calculated for both real and fake news. Several
strategies, such as EANN and Spotfake, use concatenation or attention
mechanisms for feature fusion but often face correlation deficiencies because
features remain in distinct semantic spaces. CAFE attempts to mitigate this
through cross-modal alignment, aligning texts and images within a unified
semantic space. Yet, its effectiveness is limited by small datasets and
general labels, resulting in persistent semantic gaps between textual and
image features. FND-CLIP uses two pre-trained CLIP encoders to extract the
representations from the image and text. TT-BLIP achieved the highest accuracy
of 96.1% and 88.5% in detecting fake news mainly due to the following reasons:
1) the TT-BLIP architecture combines ResNet and BLIPImg for image data and
BERT and BLIPTxt for text processing, while using bidirectional BLIP encoders
for correlation information. ResNet is used for extracting complex features
from images, aiding in news image analysis. BERT is used for understanding
language nuances, important for analyzing news text. Experiments with
alternatives like TT-BLIP(VGG) (employing VGG [15] instead of ResNet) and TT-
BLIP(XLNet) (using XLNet [28] instead of BERT) demonstrated the superiority of
the original TT-BLIP setup. 2) The MultiModal Tri-Transformer uses attention
mechanisms across multiple modalities, ensuring a comprehensive and integrated
representation of the data. This approach allows for a more comprehensive
understanding of context and improves feature extraction through multi-head
attention analyzing data from various perspectives. By focusing on important
text and merging features from all modalities, this architecture improves data
representation, essential for better classification accuracy.

(a) Early Fusion

(b) Late Fusion

(c) Hybrid Fusion

(d) Tensor Fusion

Figure 3: Architecture of different fusion strategies for Multimodal fake news
detection

###  IV-D Comparison of Fusion Methods

TABLE II: Comparative Analysis of Fusion Techniques for Fake News Detection on Weibo and Gossipcop Datasets Dataset | Method | Accuracy | Fake News | Real News
---|---|---|---|---
|  |  | Precision | Recall | F1 Score | Precision | Recall | F1 Score
Weibo |  Early fusion [29] | 0.792 | 0.828 | 0.742 | 0.783 | 0.761 | 0.843 | 0.800
Late fusion [29] | 0.806 | 0.840 | 0.761 | 0.799 | 0.777 | 0.852 | 0.813
Hybrid fusion [30] | 0.815 | 0.850 | 0.770 | 0.808 | 0.785 | 0.861 | 0.821
Tensor fusion [31] | 0.821 | 0.876 | 0.786 | 0.829 | 0.766 | 0.864 | 0.812
TT-BLIP | 0.961 | 0.979 | 0.944 | 0.961 | 0.944 | 0.980 | 0.962
Gossipcop |  Early fusion [29] | 0.634 | 0.277 | 0.560 | 0.370 | 0.861 | 0.651 | 0.742
Late fusion [29] | 0.713 | 0.244 | 0.233 | 0.238 | 0.819 | 0.828 | 0.823
Hybrid fusion [30] | 0.853 | 0.613 | 0.642 | 0.627 | 0.914 | 0.903 | 0.909
Tensor fusion [31] | 0.868 | 0.754 | 0.466 | 0.576 | 0.883 | 0.964 | 0.922
TT-BLIP | 0.885 | 0.737 | 0.596 | 0.659 | 0.910 | 0.950 | 0.930

(a) with text query

(b) w/o fusion

(c) with image-text query

(d) with image query

Figure 4: t-SNE visualization of extracted features from the Weibo test set
using TT-BLIP. Each color represents a distinct label grouping.

In Table II, the TT-BLIP model, utilizing a multi-head attention mechanism, is
compared against traditional fusion approaches such as early fusion [29], late
fusion [29], hybrid fusion [30], and tensor fusion [31] on the Weibo and
Gossipcop datasets. Early Fusion initially combines image, text, and image-
text features, then applies classification to this unified dataset. Late
Fusion aggregates results from different classifiers, each trained on separate
modalities like image features, text features, and image-text features. Hybrid
Fusion merges image, text, and image-text features by employing both early and
late fusion methods. Tensor Fusion combines image, text, and image-text
features using the outer product, enhancing interaction between modalities for
improved classification. The architectures of traditional fusion methods are
depicted in Fig. 3, demonstrating structural differences and highlighting the
innovation of the TT-BLIP model.

In Table II, TT-BLIP outperforms traditional methods significantly. For the
Weibo dataset, TT-BLIP significantly outperformed the traditional methods on
Weibo with an accuracy of 96.1% and achieved higher precision (97.9%) and F1
scores (96.1% and 96.2%) for fake news detection. Other methods did not
perform as well, with the highest accuracy among them being only 82.1%. On the
Gossipcop dataset, TT-BLIP was superior with 88.5% accuracy, better than
tensor fusion at 86.8%. In terms of precision and recall for fake news
detection, TT-BLIP performs competitively, though not leading, with figures at
73.7% and 59.6% respectively. This indicates that while TT-BLIP excels in
overall accuracy and F1 score, it encounters competition from methods like
Hybrid and Tensor Fusion in specific metrics.

In Fig. 4, t-SNE visualizations analyze features processed by TT-BLIP in
different settings within the MultiModal Tri-Transformer’s feature fusion
mechanism, using the test set of Weibo. The visualizations compare before and
after fusion, utilizing different types of queries: text, image-text, and
image. The dots with the same color depicts data with same label. In Fig. 4,
the visualizations show that before fusion (Fig. 4 (b)) data points are
indicating a lack of clear separation between fake and real news instances.
Applying TT-BLIP, there’s a improvement in clustering, the real and fake news
instances are now grouped more distinctly. This enhanced separation
facilitates a more effective identification of fake news by allowing the model
to better integrate the underlying patterns within the data. In Fig. 4, which
displays various query configurations, a comparative analysis of (a), (c), and
(d) shows differences in data clustering effectiveness. Specifically,
configurations employing image-text and image queries, as shown in Fig. 4 (c)
and (d), result in less defined separations between clusters. This outcome
indicates that methods relying solely on image data are less effective for
classification. The reduced clarity in clustering with image or image-text
queries emphasizes the significant role of text in creating a clear and
distinct feature space. Textual content offers crucial context, improving
accuracy in differentiating categories, particularly in tasks like fake news
detection. This comparison shows that incorporating textual information
improves the model’s classification efficacy.

The distinct clustering with text queries demonstrates the value of text in
improving the model’s capacity to distinguish between categories, essential
for tasks such as fake news detection. Fig. 4 confirms the benefit of
prioritizing text in multimodal feature fusion, yielding more distinct feature
representations and supporting better classification outcomes.

TABLE III: Ablation experimental results of TT-BLIP Dataset | ResNet | BLIPImg | BERT | BLIPTxt | BLIPImg,Txt | Fusion | Accuracy | F1 Score
---|---|---|---|---|---|---|---|---
Fake News | Real News
Weibo | ✓ | ✓ |  |  |  |  | 0.659 | 0.684 | 0.629
|  | ✓ | ✓ |  |  | 0.837 | 0.836 | 0.839
| ✓ |  | ✓ | ✓ | ✓ | 0.858 | 0.846 | 0.869
✓ |  | ✓ |  |  | ✓ | 0.912 | 0.911 | 0.912
|  |  |  | ✓ |  | 0.747 | 0.747 | 0.747
✓ | ✓ | ✓ | ✓ | ✓ |  | 0.852 | 0.853 | 0.851
| ✓ | ✓ | ✓ | ✓ | ✓ | 0.959 | 0.959 | 0.959
✓ | ✓ |  | ✓ | ✓ | ✓ | 0.936 | 0.937 | 0.935
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 0.961 | 0.961 | 0.962
Gossipcop | ✓ | ✓ |  |  |  |  | 0.798 | 0.861 | 0.632
|  | ✓ | ✓ |  |  | 0.846 | 0.892 | 0.733
| ✓ |  | ✓ | ✓ | ✓ | 0.846 | 0.895 | 0.714
✓ |  | ✓ |  |  | ✓ | 0.848 | 0.715 | 0.902
|  |  |  | ✓ |  | 0.837 | 0.746 | 0.879
✓ | ✓ | ✓ | ✓ | ✓ |  | 0.837 | 0.886 | 0.712
| ✓ | ✓ | ✓ | ✓ | ✓ | 0.856 | 0.717 | 0.903
✓ | ✓ |  | ✓ | ✓ | ✓ | 0.846 | 0.692 | 0.897
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 0.885 | 0.659 | 0.930

###  IV-E Ablation study

Some variants and components of the model were tested to identify the
importance of the proposed model. The results are shown in Table III. If the
fusion module was not used, the features were directly concatenated. The
effectiveness of each components is measured through two metrics: Accuracy and
F1 Score, with separate F1 Scores for Fake News and Real News. The Text-only
model, utilizing BERT and BLIPTxt for textual analysis, proved more effective
than the Image-only model which employed ResNet and BLIPImg for image-based
feature extraction. This indicates that textual features are more crucial in
the classification of fake news. Integrating BLIPImg, BLIPTxt, and BLIPImg,Txt
slightly outperformed the Text-only model, indicating the benefits of
incorporating image features and textual data. The performance of the
BLIPImg,Txt only model, which uses only BLIPImg,Txt combined feature without
any fusion, did not reach the accuracy levels of the integrating BLIPImg,
BLIPTxt, and BLIPImg,Txt method. This highlights the necessity for more
integrated usage of features. The removal of the ResNet module resulted in a
minor decline in performance, implying its lesser importance compared to other
components. In contrast, the exclusion of the BERT component led to a
significant reduction in effectiveness, emphasizing the critical role BERT
plays in text processing. Models lacking fusion methods were less effective
than TT-BLIP, underlining the importance of advanced fusion techniques. The
complete TT-BLIP model, integrating ResNet, BLIPImg, BERT, BLIPTxt,
BLIPImg,Txt, and fusion techniques, showed the highest accuracy in classifying
fake news.

##  V Conclusion

This study presented the Tri-Transformer BLIP (TT-BLIP), which uses the
bootstrapping language-image pretraining (BLIP), that is used for enhanced
vision-language understanding, and BERT for text feature extraction, BLIP and
ResNet for image feature extraction, BLIP for image-text feature extraction
and the Multimodal Tri-Transformer for the fusion of the feature from
different modalities. A key contribution is the three-pathway structure of TT-
BLIP, which processes text and images independently while also learning the
correlation of the two different modals and the Multimodal Tri-Transformer
that provides an efficient fusion mechanism to integrate information across
these three modalities. The TT-BLIP outperformed the previous state of the art
fake news detection models in accuracy by 5.4% for the Weibo dataset (90.7%
versus 96.1%) and 0.5% for the Gossipcop (88% versus 88.5%).

## References

  * [1] Sakshini Hangloo and Bhavna Arora,  “Combating multimodal fake news on social media: methods, datasets, and future perspective,”  Multimedia systems, vol. 28, no. 6, pp. 2391–2422, 2022.
  * [2] Zhiwei Jin, Juan Cao, Yongdong Zhang, Jianshe Zhou, and Qi Tian,  “Novel visual and statistical image features for microblogs news verification,”  IEEE transactions on multimedia, vol. 19, no. 3, pp. 598–608, 2016.
  * [3] Carlos Castillo, Marcelo Mendoza, and Barbara Poblete,  “Information credibility on twitter,”  in Proceedings of the 20th international conference on World wide web, 2011, pp. 675–684.
  * [4] Nadia K Conroy, Victoria L Rubin, and Yimin Chen,  “Automatic deception detection: Methods for finding fake news,”  Proceedings of the association for information science and technology, vol. 52, no. 1, pp. 1–4, 2015.
  * [5] Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J Jansen, Kam-Fai Wong, and Meeyoung Cha,  “Detecting rumors from microblogs with recurrent neural networks,”  2016\.
  * [6] Tong Chen, Xue Li, Hongzhi Yin, and Jun Zhang,  “Call attention to rumors: Deep attention based recurrent neural networks for early rumor detection,”  in Trends and Applications in Knowledge Discovery and Data Mining: PAKDD 2018 Workshops, BDASC, BDM, ML4Cyber, PAISI, DaMEMO, Melbourne, VIC, Australia, June 3, 2018, Revised Selected Papers 22. Springer, 2018, pp. 40–52.
  * [7] Yaqing Wang, Fenglong Ma, Zhiwei Jin, Ye Yuan, Guangxu Xun, Kishlay Jha, Lu Su, and Jing Gao,  “Eann: Event adversarial neural networks for multi-modal fake news detection,”  in Proceedings of the 24th acm sigkdd international conference on knowledge discovery & data mining, 2018, pp. 849–857.
  * [8] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi,  “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,”  in International Conference on Machine Learning. PMLR, 2022, pp. 12888–12900.
  * [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “Bert: Pre-training of deep bidirectional transformers for language understanding,”  arXiv preprint arXiv:1810.04805, 2018.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [11] Zhiwei Jin, Juan Cao, Han Guo, Yongdong Zhang, and Jiebo Luo,  “Multimodal fusion with recurrent neural networks for rumor detection on microblogs,”  in Proceedings of the 25th ACM international conference on Multimedia, 2017, pp. 795–816.
  * [12] Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu,  “Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media,”  Big data, vol. 8, no. 3, pp. 171–188, 2020.
  * [13] Dhruv Khattar, Jaipal Singh Goud, Manish Gupta, and Vasudeva Varma,  “Mvae: Multimodal variational autoencoder for fake news detection,”  in The world wide web conference, 2019, pp. 2915–2921.
  * [14] Shivangi Singhal, Rajiv Ratn Shah, Tanmoy Chakraborty, Ponnurangam Kumaraguru, and Shin’ichi Satoh,  “Spotfake: A multi-modal framework for fake news detection,”  in 2019 IEEE fifth international conference on multimedia big data (BigMM). IEEE, 2019, pp. 39–47.
  * [15] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * [16] Shivangi Singhal, Anubha Kabra, Mohit Sharma, Rajiv Ratn Shah, Tanmoy Chakraborty, and Ponnurangam Kumaraguru,  “Spotfake+: A multimodal framework for fake news detection via transfer learning (student abstract),”  in Proceedings of the AAAI conference on artificial intelligence, 2020, vol. 34, pp. 13915–13916.
  * [17] Xinyi Zhou, Jindi Wu, and Reza Zafarani,  “Safe: similarity-aware multi-modal fake news detection (2020),”  Preprint. arXiv, vol. 200304981, pp. 2, 2020.
  * [18] Yixuan Chen, Dongsheng Li, Peng Zhang, Jie Sui, Qin Lv, Lu Tun, and Li Shang,  “Cross-modal ambiguity learning for multimodal fake news detection,”  in Proceedings of the ACM Web Conference 2022, 2022, pp. 2897–2905.
  * [19] Shivangi Singhal, Tanisha Pandey, Saksham Mrig, Rajiv Ratn Shah, and Ponnurangam Kumaraguru,  “Leveraging intra and inter modality relationship for multimodal fake news detection,”  in Companion Proceedings of the Web Conference 2022, 2022, pp. 726–734.
  * [20] Yang Wu, Pengwei Zhan, Yunjian Zhang, Liming Wang, and Zhen Xu,  “Multimodal fusion with co-attention networks for fake news detection,”  in Findings of the association for computational linguistics: ACL-IJCNLP 2021, 2021, pp. 2560–2569.
  * [21] Liesbeth Allein, Marie-Francine Moens, and Domenico Perrotta,  “Like article, like audience: Enforcing multimodal correlations for disinformation detection,”  arXiv preprint arXiv:2108.13892, 2021.
  * [22] Tong Zhang, Di Wang, Huanhuan Chen, Zhiwei Zeng, Wei Guo, Chunyan Miao, and Lizhen Cui,  “Bdann: Bert-based domain adaptation neural network for multi-modal fake news detection,”  in 2020 international joint conference on neural networks (IJCNN). IEEE, 2020, pp. 1–8.
  * [23] Yangming Zhou, Yuzhou Yang, Qichao Ying, Zhenxing Qian, and Xinpeng Zhang,  “Multimodal fake news detection via clip-guided learning,”  in 2023 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2023, pp. 2825–2830.
  * [24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.,  “Learning transferable visual models from natural language supervision,”  in International conference on machine learning. PMLR, 2021, pp. 8748–8763.
  * [25] Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine,  “Supervised multimodal bitransformers for classifying images and text,”  arXiv preprint arXiv:1909.02950, 2019.
  * [26] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017.
  * [27] Jun Lin, Glenna Tremblay-Taylor, Guanyi Mou, Di You, and Kyumin Lee,  “Detecting fake news articles,”  in 2019 IEEE International Conference on Big Data (Big Data). IEEE, 2019, pp. 3021–3025.
  * [28] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le,  “Xlnet: Generalized autoregressive pretraining for language understanding,”  Advances in neural information processing systems, vol. 32, 2019.
  * [29] Cees GM Snoek, Marcel Worring, and Arnold WM Smeulders,  “Early versus late fusion in semantic video analysis,”  in Proceedings of the 13th annual ACM international conference on Multimedia, 2005, pp. 399–402.
  * [30] Zhiyong Wu, Lianhong Cai, and Helen Meng,  “Multi-level fusion of audio and visual features for speaker identification,”  in Advances in Biometrics: International Conference, ICB 2006, Hong Kong, China, January 5-7, 2006. Proceedings. Springer, 2005, pp. 493–499.
  * [31] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency,  “Tensor fusion network for multimodal sentiment analysis,”  arXiv preprint arXiv:1707.07250, 2017.


11institutetext: Department of Physics and Astronomy, Purdue University

# NCQ scaling of $f_{0}(980)$ elliptic flow in 200 GeV Au+Au collisions by
STAR and its constituent quark content

Jie Zhao (for the STAR collaboration) 11<EMAIL_ADDRESS>
###### Abstract

Searching for exotic state particles and studying their properties have
furthered our understanding of quantum chromodynamics (QCD). The $f_{0}$(980)
resonance is an exotic state with relatively high production rate in
relativistic heavy-ion collisions, decaying primarily into $\pi\pi$. Currently
the structure and quark content of the $f_{0}$(980) are unknown with several
predictions from theory being a $q\overline{q}$ state, a
$qq\overline{q}\overline{q}$ state, a $K\overline{K}$ molecule state, or a
gluonium state. We report the first $f_{0}$(980) elliptic flow ($v_{2}$)
measurement from 200 GeV Au+Au collisions at STAR. The transverse momentum
dependence of $v_{2}$ is examined and compared to those of other hadrons
(baryons and mesons). The empirical number of constituent quark (NCQ) scaling
is used to investigate the constituent quark content of $f_{0}$(980), which
may potentially address an important question in QCD.

##  1 Introduction

Searching for exotic state particles and studying their properties have
furthered our understanding of quantum chromodynamics (QCD). Currently the
structure and quark content of $f_{0}$(980) are unknown with several
predictions being a $q\overline{q}$ state, a $qq\overline{q}\overline{q}$
state, a $K\overline{K}$ molecule state, or a gluonium state Weinstein:1983gd
; Weinstein:1990gu ; Kleefeld:2001ds ; Baru:2003qq ; BESIII:2015rug ;
Achasov:2020aun . In contrast to the vector and tensor mesons, the
identification of the scalar mesons is a long-standing puzzle pdg:2020paz .
Previous preliminary experimental measurements Fachini:2004jx  on the yield of
$f_{0}$(980) at RHIC and theoretical calculation ExHIC:2010gcb  suggest that
it could be a $K\overline{K}$ stat. In this analysis, the empirical number of
constituent quark (NCQ) scaling Molnar:2003ff ; PHENIX:2003qra ; STAR:2003wqp
is used to investigate the constituent quark content of $f_{0}$(980)
Gu:2019oyz .

##  2 Experiment setup and data analysis

The data reported here are from Au+Au collisions at a nucleon-nucleon center-
of-mass energy of 200 GeV, collected by the STAR experiment Ackermann:2002ad
at Brookhaven National Laboratory in 2011, 2014 and 2016. A total of 2.4
billion minimum-bias (MB) events are selected for this analysis. The main
subsystem used for the data analysis is the Time Projection Chamber (TPC)
Anderson:2003ur  with 2$\pi$ azimuthal coverage at mid-rapidity. The TPC
$dE/dx$ is used to select $\pi^{\pm}$ candidate with $0.2<p_{T}<5.0$ GeV/$c$.

The $\pi^{+}\pi^{-}$ are used to reconstruct the $f_{0}(980)$. The
combinatorial background subtraction is based on the mixed-event technique and
the like-sign method STAR:2015tnn . The acceptance-corrected like-sign pairs
STAR:2015tnn ; STAR:2013pwb  are used to subtract the combinatorial background
after being normalized to unlike-sign pairs in the invariant mass ($m_{inv}$)
range beyond 1.5 GeV/$c^{2}$ . Figure 1 (left) shows the background subtracted
$\pi^{+}\pi^{-}$ invariant mass distribution. The resonance peaks are
parametrized with the relativistic Breit-Wigner function STAR:2002caw ;
STAR:2003vqj . The total fit function is given by:

$f(m_{inv})=\left(\sum_{X=f_{0},\rho^{0},f_{2}}\frac{A_{X}m_{inv}m_{X}\Gamma(X)}{(m_{inv}^{2}-m_{X}^{2})^{2}+m_{X}^{2}\Gamma(X)^{2}}\right)\times
PS+bg(m_{inv})$ (1)

where
$\Gamma(X)=\frac{\Gamma_{X}m_{X}}{m_{inv}}\left(\frac{m_{inv}^{2}-4m_{\pi}^{2}}{m_{X}^{2}-4m_{\pi}^{2}}\right)^{J+1/2}$
STAR:2002caw ; STAR:2003vqj ,
$PS=\frac{m_{inv}}{\sqrt{m_{inv}^{2}+p_{T}^{2}}}\exp\left(-\frac{\sqrt{m_{inv}^{2}+p_{T}^{2}}}{T}\right)$
is the phase space correction taking into account the $\pi\pi$ scattering
during the hadronic phase Shuryak:2002kd ; Kolb:2003bk ; Rapp:2003ar ;
STAR:2003vqj , and $bg(m_{inv})$ is a third order polynomial function to
describe the residual background. $m_{X}$ and $\Gamma_{X}$ are the mass and
width of the corresponding resonances. $\Gamma_{\rho^{0}}$ is set to 160 MeV,
and $m_{f_{2}}$ and $\Gamma_{f_{2}}$ are set according to the PDG values
pdg:2020paz . T is the kinetic freeze-out temperature, set to 120 MeV
Shuryak:2002kd . $A_{f_{0}}$, $A_{\rho^{0}}$, $A_{f_{2}}$, $m_{f_{0}}$,
$\Gamma_{f_{0}}$, and $m_{\rho^{0}}$ are free parameters.

The event-plane method Poskanzer:1998yz  is used to study the elliptic flow
($v_{2}$) of $f_{0}(980)$. The event-plane is reconstructed by all charged
particles in the TPC with pseudorapidity $|\eta|<1$ and transverse momentum
$0.2<p_{T}<5.0$ GeV/$c$. For each $\pi\pi$ pair, the two $\pi$ candidates are
removed from the event-plane reconstruction to avoid auto-correlation. The
event-plane resolution is calculated by the correlation between two randomly
divided sub-events from the full TPC Poskanzer:1998yz . Wide centrality bin
effect is corrected by weighting the event-plane resolution with the
$f_{0}(980)$ yield in each narrow centrality bin of 10% size STAR:2015gge .
Figure 1(right) shows the $f_{0}(980)$ yield as function of the azimuthal
angle difference between the $\pi\pi$ pair ($\phi$) and the event-plane
direction ($\Psi$) in an example $p_{\rm T}$ bin.

Figure 1: (Color online) (Left) The background subtracted $\pi^{+}\pi^{-}$
invariant mass distribution over the $p_{T}$ range of $0<p_{T}<5.0$ GeV/$c$ in
30–80% Au+Au collisions at $\sqrt{s_{NN}}$=200 GeV. The red line is the result
of fit. The pink, green, violet lines represent the resonance peaks of the
relativistic Breit-Wigner function. The solid blue line represents the
residual background using a third order polynomial function. (Right)
$f_{0}(980)$ yield as function of $\phi-\Psi$ in a given $p_{\rm T}$ bin.
Errors are statistical. The red line represents a fit
($\propto(1+2v_{2}^{obs}\rm cos(2\phi-2\Psi))$) to the data.

Figure 2 shows $f_{0}(980)$ $v_{2}$ as a function of $p_{\rm T}$ in 30-80%
centrality Au+Au collisions at $\sqrt{s_{NN}}$ = 200 GeV. Results are compared
with other identified particles: $\pi,K,p,K_{s}^{0},\Lambda,\Xi,\Omega,\phi$
STAR:2015gge . In the low $p_{\rm T}$ region, the $f_{0}(980)$ $v_{2}$ seems
to follow the mass ordering. In the higher $p_{\rm T}$ region, the
$f_{0}(980)$ $v_{2}$ seems closer to the baryon band.

Figure 2: (Color online) $f_{0}(980)$ $v_{2}$ as a function of $p_{\rm T}$ in
30-80% centrality Au+Au collisions at $\sqrt{s_{NN}}$ = 200 GeV. Statistical
uncertainties are shown by the vertical bars and systematic uncertainties are
shown by the caps. Results of other particles are taken from Ref. STAR:2015gge
.

Figure 3 shows the number of constituent quark ($n_{q}$) scaled $v_{2}$ as
function of the $n_{q}$ scaled $p_{\rm T}$ (left) and $m_{T}-m_{0}$ (right).
Here the $f_{0}(980)$ is assumed to have either 2 quarks or 4 quarks. The data
are compared to the fit of other particles STAR:2015gge  using a NCQ scaling
inspired function Dong:2004ve :

$f_{v_{2}}(n_{q})=\frac{an_{q}}{1+\exp(-((m_{T}-m_{0})/n_{q}-b)/c)}-dn_{q}.$
(2)

The 2-quarks (4-quarks) scaled $f_{0}(980)$ $v_{2}$ seems to deviate from the
fit, above (below) the fit by $\sim 1\sigma$ for the last one or two points at
high $(m_{T}-m_{0})/n_{q}$.

Figure 4 shows $f_{0}(980)$ $v_{2}$ as a function of $m_{T}-m_{0}$ with a fit
according to the function shown in Eq. 2. In the fit, only the $n_{q}$ of
$f_{0}(980)$ is treated as a free parameter and all other parameters are fixed
according to the fit in the right panel of the Fig. 3. This NCQ scaling fit of
the $f_{0}(980)$ $v_{2}$ yields $n_{q}=3.0\pm 0.7\mbox{ (stat)}\pm 0.5\mbox{
(syst)}$.

Figure 3: (Color online) $f_{0}(980)$ $v_{2}$ divided by $n_{q}$ as a function
of $p_{\rm T}$$/n_{q}$ (left) and $(m_{T}-m_{0})/n_{q}$ (right) in 30-80%
centrality Au+Au collisions at $\sqrt{s_{NN}}$ = 200 GeV. Results of other
particles are taken from Ref. STAR:2015gge . Black line in the right panel
represents a fit to results of other particles using a NCQ scaling inspired
function (Eq. 2).  Figure 4: (Color online) $f_{0}(980)$ $v_{2}$ as a function
of $m_{T}-m_{0}$ in 30-80% centrality Au+Au collisions at $\sqrt{s_{NN}}$ =
200 GeV. The blue curve represents the NCQ inspired fit (Eq. 2), where the
only free parameter is the $n_{q}$ of $f_{0}(980)$ and all other parameters
are fixed according to the fit in the right panel of the Fig. 3.

With the current uncertainty, our result is not able to determine whether
$f_{0}(980)$ is a $q\overline{q}$, $qq\overline{q}\overline{q}$,
$K\overline{K}$ molecule, gluonium state, or produced through $\pi\pi$
coalescence. It could also be given by some combined states as well. Future
measurements, e.g. the $f_{0}(980)$ yields, could also provide different
aspect to understand it.

##  3 Summary

Preliminary results on the $f_{0}(980)$ $v_{2}$ in 30-80% centrality Au+Au
collisions at $\sqrt{s_{\rm NN}}$ = 200 GeV are presented. In the low $p_{\rm
T}$ region ($p_{\rm T}$<2 GeV/$c$), the $f_{0}(980)$ $v_{2}$ seems to follow
the mass ordering. In the higher $p_{\rm T}$ region ($p_{\rm T}$>2 GeV/$c$),
the $f_{0}(980)$ $v_{2}$ seems closer to the baryon band. A NCQ scaling
inspired function was used to fit the $f_{0}(980)$ $v_{2}$. The extracted
quark content of $f_{0}(980)$ is $n_{q}=3.0\pm 0.7\mbox{ (stat)}\pm 0.5\mbox{
(syst)}$. More data are needed to understand whether $f_{0}(980)$ is a
$q\overline{q}$, $qq\overline{q}\overline{q}$, $K\overline{K}$ molecule,
gluonium state, or produced through $\pi\pi$ coalescence. Our study indicates
that heavy-ion collisions can be a useful place to examine the quark content
of scalar mesons. The isobar data taken in 2018 at RHIC and the 8-fold
increase in Au+Au data expected in 2023-2025 would provide more insights.

$\mathbb{Acknowledgments}$ This work was partly supported by the U.S.
Department of Energy (Grant No. de-sc0012910).

## References

  * [1] J.D. Weinstein, N. Isgur, Phys. Rev. D 27, 588 (1983)
  * [2] J.D. Weinstein, N. Isgur, Phys. Rev. D 41, 2236 (1990)
  * [3] F. Kleefeld, E. van Beveren, G. Rupp, M.D. Scadron, Phys. Rev. D 66, 034007 (2002)
  * [4] V. Baru et al., Phys. Lett. B 586, 53 (2004)
  * [5] M. Ablikim et al. (BESIII), Phys. Rev. D 92, 052003 (2015)
  * [6] N.N. Achasov et al., Phys. Rev. D 103, 014010 (2021)
  * [7] P.A. Zyla et al. (Particle Data Group), Prog. Theor. Exp. Phys. 083C01 (2020)
  * [8] P. Fachini, J. Phys. G 30, S735 (2004)
  * [9] S. Cho et al. (ExHIC), Phys. Rev. Lett. 106, 212001 (2011)
  * [10] D. Molnar, S.A. Voloshin, Phys. Rev. Lett. 91, 092301 (2003)
  * [11] S.S. Adler et al. (PHENIX), Phys. Rev. Lett. 91, 182301 (2003)
  * [12] J. Adams et al. (STAR), Phys. Rev. Lett. 92, 052302 (2004)
  * [13] A. Gu, T. Edmonds, J. Zhao, F. Wang, Phys. Rev. C 101, 024908 (2020)
  * [14] K.H. Ackermann et al. (STAR), Nucl. Instrum. Meth. A499, 624 (2003)
  * [15] M. Anderson et al., Nucl. Instrum. Meth. A499, 659 (2003)
  * [16] L. Adamczyk et al. (STAR), Phys. Rev. C 92, 024912 (2015)
  * [17] L. Adamczyk et al. (STAR), Phys. Rev. Lett. 113, 022301 (2014)
  * [18] C. Adler et al. (STAR), Phys. Rev. Lett. 89, 272302 (2002)
  * [19] J. Adams et al. (STAR), Phys. Rev. Lett. 92, 092301 (2004)
  * [20] E.V. Shuryak, G.E. Brown, Nucl. Phys. A 717, 322 (2003)
  * [21] P.F. Kolb, M. Prakash, Phys. Rev. C 67, 044902 (2003)
  * [22] R. Rapp, Nucl. Phys. A 725, 254 (2003)
  * [23] A.M. Poskanzer, S.A. Voloshin, Phys. Rev. C 58, 1671 (1998)
  * [24] L. Adamczyk et al. (STAR), Phys. Rev. Lett. 116, 062301 (2016)
  * [25] X. Dong, S. Esumi, P. Sorensen, N. Xu, Z. Xu, Phys. Lett. B 597, 328 (2004)


# Improving Zero Shot Learning Baselines with Commonsense Knowledge

Abhinaba Roy⊘, Deepanway Ghosal†, Erik Cambria⊘,
Navonil Majumder†, Rada Mihalcea⊓ , Soujanya Poria†

⊘Nanyang Technological University, Singapore
†Singapore University of Technology and Design, Singapore
⊓University of Michigan, USA
{abhinaba.roy<EMAIL_ADDRESS>
<EMAIL_ADDRESS>sporia<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Zero shot learning — the problem of training and testing on a completely
disjoint set of classes — relies greatly on its ability to transfer knowledge
from train classes to test classes. Traditionally semantic embeddings
consisting of human defined attributes (HA) or distributed word embeddings
(DWE) are used to facilitate this transfer by improving the association
between visual and semantic embeddings. In this paper, we take advantage of
explicit relations between nodes defined in ConceptNet, a commonsense
knowledge graph, to generate commonsense embeddings of the class labels by
using a graph convolution network-based autoencoder. Our experiments performed
on three standard benchmark datasets surpass the strong baselines when we fuse
our commonsense embeddings with existing semantic embeddings i.e. HA and DWE.

##  1 Introduction

How does one recognize an instance of an object they have never seen before?
Instinctively, the first step is to find any resemblance with familiar
objects. Take the example of Fig. 1 — with the information “ _Olinguito has
the shape of a Koala and the color of a Bear_ ”, we can establish shape- and
color-based associations between known animals (Bear and Koala) and the
unknown one (Olinguito). Inspired by such analogies, we explore how we can
discover and exploit such associations between classes in zero-shot learning
(ZSL) — the problem of learning a visual classifier for a class with no
training data.

In a traditional ZSL setup, a visual classifier trained on a certain set of
classes, namely seen classes, is expected to perform reasonably well on a
completely disjoint set of classes, namely unseen classes. The key idea in
dealing with such a scenario is to utilize knowledge from the seen classes to
characterize unseen classes Norouzi et al. (2013). To this end, researchers
often model the association between semantic embeddings of the classes and
visual embeddings of the images.

Figure 1: Animal in the middle is relatively unknown. However, we know the
ones on the left and the right.

A high association score indicates a strong candidate class for the given
image. During inference, each of the unseen class embeddings is paired with
the visual embedding of the given image and fed to the association function.
Typically, the class with the highest association score is selected as the
predicted class. In literature, usually two types of semantic embeddings are
used Xian et al. (2017) — human-defined attributes (HA) and distributed word
embeddings (DWE).

A number of approaches Zhang et al. (2017); Kodirov et al. (2017); Kumar Verma
et al. (2018) use human-defined visually identifiable attributes (HA) such as
_stripes_ , _long_neck_ , _furry_ , _spots_ to describe classes. But, these
approaches are rather expensive as human annotation is involved.

On the other hand, distributed word embeddings (DWE), such as word2vec
successfully model different semantic properties of a language as a result of
it being trained on a large-scale textual corpus Jiang et al. (2017); Kodirov
et al. (2015); Qiao et al. (2017). The primary objective of learning such word
embeddings is based on the distributional hypothesis that states that similar
words tend to appear in similar context. Using this hypothesis, pretrained
word embeddings encode semantic properties of words. Nowadays, such embeddings
are extensively used in many NLP tasks. However, these word embeddings are not
trained on explicit knowledge representations to directly provide knowledge
for an unseen word. These models can also struggle to find similarity between
the words that never or rarely appear in each other’s context (see Table 2).

Returning to the example in Fig. 1, we can observe that external knowledge
helps in finding an explicit relation between different classes by traversing
through the paths between them in the graph. To elucidate this, consider the
example in Fig. 2 — class ‘Aeroplane’ (seen class) is connected to class
‘Motorbike’ through ‘Motor racing’ (unseen class). In this example, the
relations (in red) that connect the nodes (in yellow) on the path from
_aeroplane_ to _motorbike_ are explicitly defined in ConceptNet Speer et al.
(2017). Such a relational knowledge-graph that contains relations among the
words can efficiently model associations that can be very beneficial to the
ZSL task by transferring knowledge from seen to unseen classes.

Figure 2: An example of commonsense relations between Aeroplane and Motorbike
in ConceptNet.

Another benefit of using knowledge graphs for ZSL is that it can provide
additional background information on the class labels, which in turn can
improve the semantic embeddings. An illustration in Fig. 2 shows the
background knowledge of _aeroplane_ being represented with the nodes in
magenta. For someone oblivious to an aeroplane, the shape and size of it is
still imaginable through the associated background commonsense knowledge. For
instance,
$aeroplane\xrightarrow{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}CapableOf}}\text{fly}$
or
$aeroplane\xrightarrow{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}HasContext}}\text{wing}$
are strong conceptual indicators to aeroplanes, but not to motorbikes.
Naturally, the semantic embeddings, created from such knowledge graph, will be
vastly different for an aeroplane as compared to that of a motorbike, leading
to stronger association between the representation of the input image in Fig.
2 and aeroplane than motorbike. We therefore hypothesize that the improved
semantic embeddings will lead to improved association of visual and semantic
embeddings, resulting to correct class prediction during inference. We achieve
this by fusing commonsense embeddings, extracted from ConceptNet, with the
human-defined attributes (HA) and distributed word embeddings (DWE).

_Commonsense knowledge_ broadly represents the large amount of small but well-
known facts that we often take for granted Andrich et al. (2009), and yet we
regularly use to understand our surrounding context or to interpret or
synthesize language. To leverage commonsense-knowledge, we employ ConceptNet
Speer et al. (2017), a large-scale knowledge graph that contains commonsense
knowledge. We posit that using such commonsense knowledge is beneficial for
our task, as compared to other knowledge graphs, such as WordNet which only
contain relations like _IsA_ , _Synonymy_ , _Hypernymy_.

We first train a global (covering all possible classes as nodes) graph
autoencoder on ConceptNet to learn inter-class associations. To this end, a
graph convolutional network (GCN) is employed for relation prediction in the
graph. The learnt node embeddings are used as the commonsense embeddings (CSE)
of the corresponding class names. Further, we fuse this CSE with the semantic
embeddings. We surmise that the obtained CSE encode associations between seen
and unseen classes in the ZSL setup. Additionally, CSE should also encode key
background information about the image class labels. To validate our claim, we
use the CSE of the class labels to induce external commonsense knowledge in
two different ZSL baselines. We observe noteworthy improvements in ZSL
performance throughout our experiments across three different benchmark ZSL
datasets.

The main contributions of this paper are –

  * •

We encode the commonsense knowledge in ConceptNet using a graph convolutional
network (GCN)-based autoencoder;

  * •

We improve association between visual and semantic embeddings by fusing
existing semantic embeddings (HA and DWE) with commonsense embeddings (CSE);

  * •

We demonstrate effectiveness of CSE through extensive experiments and
comparisons with two different ZSL baseline methods across three different
datasets.

The rest of this paper is organized as follows: §2 briefly reviews ZSL
approaches related to this work; §3 provides task definition and the details
of the two baselines that we use in this work; §4 describes our method; we
report findings of our experiments and related discussions in §5; finally, §6
concludes this paper.

##  2 Related Work

Early works on zero-shot learning use human defined attributes Farhadi et al.
(2009); Lampert et al. (2009); Jayaraman and Grauman (2014) to represent each
class as a vector that denotes presence/absence of attributes. Direct mapping
between image features and these class vectors is used to learn visual
classifiers.

Another line of work use distributed word embeddings (DWE)s such as word2vec
Mikolov et al. (2013b) to represent each category. These are consecutively
mapped to visual classifiers Frome et al. (2013); Norouzi et al. (2013);
Kodirov et al. (2015); Roy et al. (2018); Qiao et al. (2017). Frome et al.
(2013) use convolutional neural network (CNN) and a transformation layer to
train a mapping from image to word embeddings. Few recent works Kodirov et al.
(2015); Roy et al. (2018); Qiao et al. (2017) propose to reduce the gap
between CNN image features and DWEs. Combination of attributes and distributed
word embeddings are used in works such as Akata et al. (2013); Zhang et al.
(2017).

Our work is also related to recent approaches that use knowledge graphs to
distill knowledge explicitly from knowledge representations Salakhutdinov et
al. (2011); Deng et al. (2014); Kampffmeyer et al. (2019); Wang et al. (2018).
Many approaches exist in object recognition that use such approach.
Salakhutdinov et al. (2011) uses a hierarchical classification model that
allows rare objects to borrow statistical strength from related objects that
have many training examples.

Closest to our work is Wang et al. (2018). Although our approach is similar to
theirs, in that we both use Graph Convolution Network (GCN) to improve class
embeddings, the way we use GCN is vastly different.

Figure 3: The overall idea of our proposed framework.

##  3 Background

###  3.1 Task Definition

In a ZSL setup, we have a set of training instances (in this case, these are
images), $X_{train}$ belonging to a set of classes $\mathcal{Y}_{seen}$. We
test on a set of images, $X_{test}$ belonging to a set of classes
$\mathcal{Y}_{unseen}$. It is assumed that
$\mathcal{Y}_{seen}\cap\mathcal{Y}_{unseen}=\emptyset$. Ideally, for each
$x_{i}\in X_{train}$, we extract its visual embedding
$f(x_{i})\in\mathbb{R}^{m}$ by passing it through a visual feature extractor
such as a deep CNN (Simonyan and Zisserman, 2014). Each class
$y\in\mathcal{Y}_{seen\cup unseen}$ has a semantic embedding
$s_{y}\in\mathbb{R}^{N}$. $s_{y}$ can be human defined attributes of the image
class-label such as _stripes, furry_ or distributed word embeddings of the
image class (such as word2vec). In a standard ZSL training, we use
$\mathcal{Y}_{seen}$ class data to learn relation module relation between
$f(x);\forall x\in X_{train}$ and $s_{y};\forall y\in\mathcal{Y}_{train}$.
During testing, we use the same relation module to find the corresponding
$\widetilde{y}\in\mathcal{Y}_{unseen}$ for a test image $\widetilde{x}\in
X_{test}$.

###  3.2 Baseline Methods

We utilize the commonsense knowledge extracted from ConceptNet in two
different strong ZSL baselines: 1) Deep Embedding ZSL; 2) Relation Network.
Each of them consists of two main modules:

  * •

Semantic embedding module encodes the semantic embedding of the image class-
label embedding of the image. We refer to this module as the $\Theta$ network.

  * •

Relation module takes input from the semantic embedding module and visual
embeddings. It aims to model the association between the semantic embedding of
the image class-label and the visual/image embedding.

We describe the two baselines in more detail below.

####  3.2.1 Deep Embedding ZSL (DeZSL)

Following the work of Zhang et al. (2017), we jointly encode both image and
semantic representations into a unified embedding space.

##### Semantic embedding module:

The $m$-dimensional semantic embeddings of the class-label $y$, namely
$s_{y}$, is fed to the semantic embedding module named the $\Theta$ network,
composed of two fully-connected (FC) layers with ReLU activation, for
projection onto visual embedding space:

$\displaystyle\Theta_{DeZSL}$ $\displaystyle=FC^{Relu}_{b}(FC^{Relu}_{a})$ (1)

The output size of the $FC^{Relu}_{b}$ is kept equal to image feature
dimension.

Relation module: The relation module tries to minimize distance between the
image feature $f(x_{i})$ of an image $x_{i}$ and the corresponding class
semantic representation $\Theta(s_{y_{i}})$ ($y_{i}$ is the class-label of
$x_{i}$). Eq. 2 shows the related objective function.

$\displaystyle\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\|f(x_{i})-\Theta_{DeZSL}(s_{y_{i}})\|^{2}$
(2)

Inference: For an unseen image $\widetilde{x}$, we find the nearest of all
possible prototype $\Theta(s);\forall s\in\mathcal{Y}_{unseen}$ in visual
embedding space, denoted as:

$\displaystyle
y=\arg\min_{y}\mathcal{D}(f(\widetilde{x}),\Theta_{DeZSL}(s_{y}))$ (3)

Here, $\mathcal{D}$ is a distance function and $y\in\mathcal{Y}_{test}$.

####  3.2.2 Relation Network (RN)

We use a two-branch Relation Network following the work of Sung et al. (2018)
as our second baseline approach.

Semantic embedding module: It takes semantic representations of the image
class-label $s_{y}$ as inputs, passes it to $\Theta$ and produces a
representation which later is sent to the relation module to model the
relation and association between the image class-label and the image features.

$\Theta_{RN}=FC^{Relu}_{c}$ (4)

Finally we obtain $\Theta_{RN}(s_{y})$ where $y\in\mathcal{Y}_{train}$

Relation module: Relation module of the RN takes concatenated image and
semantic representations of the image class-label and passes them through a
non-linear transformation using $FC^{Relu}$ layer to model the association and
relation among these two different types of representations.

$\Gamma(s_{y},x)=FC^{Relu}_{d}(\Theta_{RN}(s_{y})\oplus f(x))$ (5)

where $x\in X_{train}$, $f(x)$ represents visual features and $\oplus$ is the
concatenation operation. In order to calculate a single score to measure the
relation and association between the image class-label embeddings and the
image features, $\Gamma(s_{y},x)$ is fed to a FC layer of output size M with
sigmoid activation where M is the number of unique image class-labels in the
training data.

$R(s_{y},x)=FC_{sigmoid}(\Gamma(s_{y},x))$ (6)

where $s_{y}$ is the set of all unique class-labels i.e.,
${s_{y_{1}},s_{y_{2}},s_{y_{3}},.....,s_{y_{M}}}$ in the training dataset.

Finally the model is trained using MSE loss using the following equation:

$\displaystyle\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{M}\|R(x_{i},s_{y_{j}})-\textbf{1}(s_{y_{i}}==s_{y_{j}})\|^{2}$
(7)

Inference: For an unseen image $\widetilde{x}\in X_{test}$, we build the
relation pairs with all the unique image class-labels and choose the pair with
the highest value of $R(s_{y},\widetilde{x})$ according to Eq. 6.

$y=\arg\max_{y}(R(s_{y},\widetilde{x}))$ (8)

where $y\in\mathcal{Y}_{test}$.

We incorporate our commonsense knowledge embedding with attribute/semantic
embeddings in each of the $\Theta$s to improve the existing semantic embedding
(human defined attribute or woprd2vec embedding) $s_{y}$ in our experiments.

##  4 Method

Our method integrates commonsense knowledge extracted from ConceptNet, and
leverages commonsense embeddings obtained from graph convolutional
autoencoders to improve ZSL. Formally, the method consists of two sequential
phases:

##### Phase 1:

This phase deals with training a class-aggregated sub-graph of ConceptNet. In
particular: a) We create a sub-graph of ConceptNet based on all seen-unseen
classes (§4.1), and b) We train a graph-convolutional autoencoder model to
learn CSE of the classes (§4.2).

##### Phase 2:

After the graph autoencoder model is trained, a) We extract CSE for each of
the classes from the graph (§4.3), and b) We feed the corresponding CSE into
our ZSL framework (§3.2.1,§3.2.2).

We define each of these steps in detail below:

###  4.1 Phase 1: a) Class-Aggregated Commonsense Graph Construction

We construct our class-aggregated graph from ConceptNet Speer et al. (2017).
First, we introduce the following notation: ConceptNet graph is represented as
a directed labeled graph $\mathcal{G=(V,E,R)}$, with concepts/nodes
$v_{i}\in\mathcal{V}$ and labeled edges $(v_{i},r_{ij},v_{j})\in\mathcal{E}$,
where $r_{ij}\in\mathcal{R}$ is the relation type of the edge between $v_{i}$
and $v_{j}$. The concepts in ConceptNet are unigram words or n-gram phrases.
For instance, a labelled edge from ConceptNet is (baking-oven, AtLocation,
kitchen), where baking-oven and kitchen are concept nodes and AtLocation is
the relation.

ConceptNet has approximately 34 million edges, from which we first extract a
subset of edges based on the following heuristic. The seen and unseen class
names are treated as the seeds that we use to filter ConceptNet into a sub-
graph. In particular, we extract all the triplets from $\mathcal{G}$ which are
within the vicinity of radius 2 of any of those seed class names, resulting in
a sub-graph $\mathcal{G^{\prime}}=(V^{\prime},E^{\prime},R^{\prime})$, with
approximately 500K edges.

###  4.2 Phase 1: b) Knowledge Graph Pre-training

To utilize $\mathcal{G^{\prime}}$ in our task, we first need to compute a
representation of its nodes. We do this by training a graph autoencoder model
to perform link prediction (a standard knowledge graph completion task). The
model takes as input an incomplete set of edges $\hat{\mathcal{E^{\prime}}}$
from $\mathcal{E^{\prime}}$ in $\mathcal{G^{\prime}}$ and then assign scores
to possible edges $(c_{1},r,c_{2})$, determining how likely are these edges to
be in $\mathcal{E^{\prime}}$.

Following Schlichtkrull et al. (2018), our graph autoencoder model has two
modules: a R-GCN entity encoder and a DistMult scoring decoder.

##### Encoder Module.

We employ the Relational Graph Convolutional Network (R-GCN) encoder from
Schlichtkrull et al. (2018) as our graph encoder network. The R-GCN
transformation encodes concepts in a multi-relational graph and can be
understood as a special case of a basic differentiable message passing Gilmer
et al. (2017). The power of this model comes from its ability to accumulate
relational evidence in multiple inference steps from the local neighbourhood
around a given concept. The neighborhood-based convolutional feature
transformation process always ensures that distinct classes are connected to
each other via latent concepts and influence each other to create enriched
class-aggregated feature vectors.

Specifically, our encoder module consists of two R-GCN encoders stacked upon
one another. This enables information flow from concepts that are within a
distance of two hops. Extending this by the rule of associativity
($a\rightarrow b\rightarrow c$ and $c\leftarrow d\leftarrow e$) the model can
capture a relatively long path relation ($a\rightarrow b\rightarrow
c\leftarrow d\leftarrow e$). An alternate way to make the model capable of
capturing long path relations would be to make the encoder module deeper.
However, stacking more layers results in over-smoothing as node features
eventually converge to the same value. Training deep GCN networks is thus very
difficult and still an open research problem.

The initial concept feature vector $g_{i}$ is initialized randomly and
thereafter transformed into the class-aggregated feature vector
$h_{i}\in\mathbb{R}^{d}$ using the two-step graph convolution process. The
transformation process is detailed below:

$\displaystyle
f(x_{i},l)=\sigma(\sum\limits_{r\in\mathcal{R}}\sum\limits_{j\in
N_{i}^{r}}\frac{1}{c_{i,r}}W_{r}^{(l)}x_{j}+W_{0}^{(l)}x_{i})$ (9)
$\displaystyle h_{i}=h_{i}^{(2)}=f(h_{i}^{(1)},2)\quad;\quad
h_{i}^{(1)}=f(g_{i}\,1)$

where, $N_{i}^{r}$ denotes the neighbouring concepts of concept $i$ under
relation $r\in\mathcal{R}$; $c_{i,r}$ is a problem specific normalization
constant which can be set in advance, such that, $c_{i,r}=|N_{i}^{r}|$, or can
be learned in a gradient-based learning setup. $\sigma$ is an activation
function such as ReLU, and $W_{r}^{(1/2)}$, $W_{0}^{(1/2)}$ are learnable
parameters of the transformation.

This stack of transformations (Eq. 9) effectively accumulates normalized sum
of the local neighbourhood i.e. the neighbourhood information for each concept
in the graph. The self connection ensures self-dependent feature
transformation.

##### Decoder Module.

DistMult factorization Yang et al. (2014) is used to score a triplet
$(c_{i},r,c_{j})$,

$\displaystyle\begin{split}s(c_{i},r,c_{j})=\sigma(h_{c_{i}}^{T}R_{r}h_{c_{j}})\end{split}$
(10)

$\sigma$ is the logistic sigmoid function; $h_{c_{i}}$,
$h_{c_{j}}\in\mathbb{R}^{d}$ are the R-GCN encoded feature vectors for
concepts $c_{i}$, $c_{j}$ from Eq. 9. Each relation $r\in\mathcal{R}$ is also
associated with a diagonal matrix $R_{r}\in\mathbb{R}^{d\times d}$.

##### Training.

We train our graph autoencoder model using negative sampling following Yang et
al. (2014); Schlichtkrull et al. (2018). For triplets in
$\hat{\mathcal{E^{\prime}}}$ (positive samples), we create an equal number of
negative samples by randomly corrupting the positive triplets. The corruption
is performed by randomly modifying either one of the constituting relation or
the concept.

The target label is set to $y=1/0$ for the positive/negative triplets. In this
setup, the model is then trained with standard cross-entropy loss:

$\displaystyle\mathcal{L}_{\mathcal{G^{\prime}}}=-\frac{1}{2|\hat{\mathcal{E^{\prime}}}|}\sum_{(c_{i},r,c_{j},y)\in\mathcal{T}}(y\log
s(c_{i},r,c_{j})+$ (11) $\displaystyle(1-y)\log(1-s(c_{i},r,c_{j})))$

Once we train the graph autoencoder model, it will be ensured that knowledge
about unseen classes (crucial during evaluation) can possibly be explained via
the knowledge of seen classes and further via inter-class associations. In
other words, the encoded node representations $h_{i}$ will capture commonsense
information in the form of class associations and thus will be effective for
the downstream ZSL task when there is the possibility of encountering unseen
classes during evaluation.

###  4.3 Phase 2: Commonsense Embedding Extraction & ZSL

Figure 4: Fusion of CSE with semantic embedding.

The trained graph autoencoder model from Section 4.2, can now be used for the
commonsense embedding extraction. For a particular class name $C$, it is
performed as follows:

  1. 1.

First, we extract a subgraph from $\mathcal{G^{\prime}}$, where we take all
triplets which are within the vicinity of radius 2 of $C$. We call this graph
$\mathcal{G^{\prime}_{C}}$.

  2. 2.

We then make a forward pass of $\mathcal{G^{\prime}_{C}}$ through the encoder
of the pretrained graph autoencoder model. This results in feature vectors
$h_{j}$ for all unique nodes $j$ in $\mathcal{G^{\prime}_{C}}$.

  3. 3.

Finally we take the feature vector $h_{C}$ for node $C$, to obtain the
commonsense embedding $CSE_{C}$ for class $C$.

We hypothesize that since unseen classes will be connected to the seen classes
through the inter-class associations in $\mathcal{G^{\prime}}$, $CSE_{C}$ will
inherently capture relevant information likely to be helpful in ZSL.

###  4.4 Fusion: Integrating Commonsense Embedding into the ZSL framework

Once the commonsense embeddings for an image class-label are extracted, we
need a mechanism to fuse these embeddings with the semantic embeddings that
are used in the baselines. We carry out this using the following operations:

$\displaystyle\widetilde{s_{y}}$ $\displaystyle=FC^{Relu}_{e}(s_{y})$ (12)
$\displaystyle\widetilde{c_{y}}$ $\displaystyle=FC^{Relu}_{f}(CSE_{y})$
$\displaystyle s_{y}^{\prime}$
$\displaystyle=\widetilde{s_{y}}\oplus\widetilde{c_{y}}$

where $s_{y}$ and $CSE_{y}$ are the semantic and commonsense embeddings
respectively. The output dimension of $FC^{Relu}_{f}$ is set equal to the size
of the visual feature embedding size in the case of DeZSL. Finally,
$s_{y}^{\prime}$ is sent to the $\Theta$ in both the baselines models– DeZSL
and RN. This can be expressed using the equations:

$\displaystyle\Theta_{DeZSL}(s_{y}^{\prime})$
$\displaystyle=FC^{Relu}_{b}(FC^{Relu}_{a}(s_{y}^{\prime}))$ (13)
$\displaystyle\Theta_{RN}(s_{y}^{\prime})$
$\displaystyle=FC^{Relu}_{c}(s_{y}^{\prime})$

The outputs from $\Theta_{DeZSL}(s_{y}^{\prime})$ and
$\Theta_{RN}(s_{y}^{\prime})$ are fed to respective relation module of both
the baseline models (Eq. 7 and Eq. 5) – DeZSL and RN for training.

Although the adopted fusion scheme is relatively simpler than the ones
proposed in recent literature, we find it very effective. In this paper we
focus only on efficacy of commonsense knowledge in ZSL rather than
effectiveness of fusion methods.

Dataset | #instances |  $\mathbf{\mathcal{C}_{1}}$/$\mathbf{\mathcal{C}_{2}}$
---|---|---
AwA1 Lampert et al. (2009) | 30475 | 40/10
AwA2 Xian et al. (2017) | 37322 | 40/10
aP&Y Farhadi et al. (2009) | 15339 | 20/12
Table 1: Description of the ZSL benchmark datasets used. Note:
$\mathcal{C}_{1}$/$\mathcal{C}_{2}$ \- No. of seen/unseen classes

| Class pairs
---
<Seen, Unseen>
| Cosine
---
similarity
| Commonsense Graph
---
Associations
<Aeroplane, Motorbike> | 0.03 | aeroplane $\xrightarrow{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{Synonym}}$ airplane$\xrightarrow{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{RelatedTo}}$aerodynamics$\xrightarrow{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{HasContext}}$motorracing$\xrightarrow{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{RelatedTo}}$motorbike
<Building, Statue> | 0.32 | building$\xrightarrow{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{RelatedTo}}$build$\xrightarrow{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{RelatedTo}}$statue
<Tvmonitor, Chair> | 0.28 | tv_monitor$\xrightarrow{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{Synonym}}$television monitor $\xrightarrow{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{IsA}}$monitor$\xrightarrow{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{AtLocation}}$desk $\xrightarrow{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{SimilarTo}}$chair
<Jetski, Motorbike> | 0.45 | jetski$\xrightarrow{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{RelatedTo}}$motorbike

Table 2: Examples of path between seen and unseen classes in ConceptNet.

Model Variants | AwA1 | AwA2 | aP&Y
---|---|---|---
DeZSL | RN | DeZSL | RN | DeZSL | RN
HA | 68.25 | 56.47 | 65.34 | 44.12 | 25.53 | 28.90
HA+CSE | 68.41 | 60.54 | 68.71 | 56.55 | 34.47 | 29.19
HA+WNet | 67.10 | 47.14 | 60.83 | 41.3 | 20.2 | 24.0
DWE | 45.16 | 38.03 | 43.29 | 25.49 | 27.42 | 38.23
DWE+CSE | 46.58 | 40.36 | 44.53 | 32.15 | 28.65 | 39.83
DWE+WNet | 42.12 | 37.27 | 32.7 | 35.26 | 14.08 | 34.55
HA+DWE | 66.75 | 43.38 | 66.01 | 40.88 | 20.91 | 38.45
HA+DWE+CSE | 67.01 | 47.6 | 68.52 | 46.49 | 22.35 | 37.65
HA+DWE+WNet | 65.15 | 47.38 | 63.3 | 46.27 | 17.22 | 27.53

Table 3: Results of model variants & WordNet(WNet). Bold values denote best
performance in a block.

##  5 Experiments

Datasets | Wang et. al | Wang et. al + HA | Wang et. al + DWE | HA+CSE | DWE+CSE
---|---|---|---|---|---
DeZSL | RN | DeZSL | RN
AwA1 | 34.55 | 46.96 | 41.40 | 68.41 | 65.54 | 47.58 | 41.36
AwA2 | 34.59 | 47.25 | 43.34 | 68.71 | 56.55 | 44.53 | 42.15
aP&Y | 14.76 | 34.50 | 33.19 | 34.47 | 29.19 | 28.65 | 40.23

Table 4: Comparison with Wang et al. (2018)

###  5.1 Experimental Setup

We validate our commonsense embedding by performing experiments on three
benchmark datasets on the task of ZSL. We choose two mid-sized datasets namely
Animals with Attributes1 (AwA1) Lampert et al. (2009) and Animals with
Attributes2 (AwA2) Xian et al. (2017). They share the same animal classes
(such as horse, zebra, sheep, blue-whale etc.) but different images. aPascal &
aYahoo (aP&Y) Farhadi et al. (2009) is small dataset containing image-classes
form varied domains (such as dogs, aeroplane, jet-ski, potted-plant etc.).
Details regarding these datasets are provided in Table 1. We adopt GBU
settings Xian et al. (2017) commonly used in ZSL literature. We use ConceptNet
5.5 Speer et al. (2017) in all our experiments.

###  5.2 Features Used

We use the following features in our ZSL baselines.

Visual Embedding: As used in Xian et al. (2017); Zhang et al. (2017), our
visual embeddings are 2048-dim top-layer pooling units of the 101-layered
ResNet. It is pre-trained on ImageNet 1K Deng et al. (2009) and not fine-
tuned.

Semantic Embeddings: As discussed in the earlier sections, we use two
different kinds of semantic embeddings:

  * •

Distributed Word Embeddings (DWE): We use word2vec vectors pre-trained on part
of Google News dataset Mikolov et al. (2013a) to cast each class name into a
300-dimensional vector representation.

  * •

Human Defined Attributes: Human defined attributes (such as _furry,stripes_)
are 85 dimensional attribute vectors for Awa1 and AwA2. For aP&Y, we have 64
dimensional attribute vectors.

###  5.3 Implementation Details

In both the baselines, number of nodes in FC layers are chosen from — 512,
1024, 2048 and 4096 and are empirically determined using 10% data from the
training set validaiton set. Both baselines are trained using ADAM optimizer
Kingma and Ba (2014) with a learning rate of $10^{-5}$.

###  5.4 Model Variants and Baselines

In order to study the effect of commonsense knowledge, we look into the
following variants:

  * •

HA: Only using human defined attributes.

  * •

DWE: Only constituting of word2vec DWEs.

  * •

HA+DWE: Fusion of HA + DWE (as in Fig. 4).

  * •

HA+CSE: We fuse human defined attributes with commonsense embedding.

  * •

DWE+CSE: Word2vec is fused with commonsense embedding in order to compare
against performance of DWE.

  * •

HA+DWE+CSE: We fuse CSE with the fusion of HA and DWE for comparison with
HA+DWE.

  * •

Wang et al. (2018): We compare our results with Wang et al. (2018) as their
approach is closest to ours. The purpose of this experiment is to check
whether the existing knowledge graph-based approaches can produce results
similar to ours. Wang et al. (2018) use a knowledge graph, NELL, and find the
shortest path between each class pair in it. This process generates a
subgraph. They apply a GCN on this subgraph to compute the node embeddings. As
some of these nodes actually represent the classes in the dataset, they then
feed these class embeddings into ZSL. We re-implemented their method by
replacing NELL with ConceptNet. Note that the only similarity between Wang et
al. (2018) and our work is that we both use graph convolution, but the
methodologies and motivation are completely different. Wang et al. (2018) use
GCN on a knowledge graph to transform initial pre-trained word embeddings and
find an association among the transformed embeddings to visual embedding.
However, we extract sub-graphs for all seen and unseen classes from the
knowledge graph and pass them through a graph convolutional autoencoder model
to extract a commonsense embedding. As such, our results are superior to Wang
et al. (2018)’s as indicated in Table 4.

###  5.5 Evaluation Scheme

We follow conventional ZSL setting – training and testing classes are
completely disjoint. This setting is standard for majority of ZSL works Wang
et al. (2019) including the original papers we formulate our baselines upon
Zhan et al. (2017); Sung et al. (2018). All experiments report top-1 accuracy.

###  5.6 Results and Analysis

We feed commonsense embedding (CSE) into two strong baseline models as
discussed in §3.1. To do so, we fuse commonsense CSE with two different
semantic embeddings 1) human defined attributes (HA); 2) word2vec (DWE).
Experimental results are shown in Table 3.

We observe a common trend of increase in ZSL recognition accuracies when CSE
is fused with HA, DWE as well as HA+DWE. This demonstrates the role that
commonsense knowledge can play in discovering meaningful associations between
seen and unseen classes, which in turn facilitates knowledge transfer. We
notice that the overall performance of the models on aP&Y is lesser compared
to the other two datasets. We believe this happens because the seen and unseen
class labels in the $aP\&Y$ dataset are from varied domains e.g., _bird,
statue, train_. Wherein, in the other two datasets the class labels are from
only _animal_ domain.

##### Comparison with WordNet:

Can we replicate our performance with a knowledge graph that does not contain
commonsense information? To answer this, we replaced ConceptNet with
WordNet(WNet) and applied our method. Wordnet Fellbaum (2010) is a lexical
knowledge graph that contains semantic connections such as _synonym, hypernym,
hyponym, antonym_ among the words. WordNet does not contain commonsense
related information that we see in ConceptNet. Performances consistently drop
by a large margin when WordNet is replaced with ConceptNet (Table 3).

##### Effect of Commonsense Knowledge:

We take a qualitative look at the effectiveness of commonsense knowledge in
Table 2. Here, Column 1 indicates a pair of <seen, unseen> classes. Column 2
shows cosine similarity of their word2vec class embeddings. Column 3 reports
association of these classes in commonsense graph. Although the pairs are not
contextually similar as reflected in cosine similarity between their word2vec
embeddings, they still have a path of associations between them. Even in cases
of visually similar classes such as < Jetski, Motorbike > we see low cosine
similarity of 0.45. Whereas in our commonsense graph, they have a direct
relation (’relatedTo’) –
jetski$\xrightarrow{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{RelatedTo}}$motorbike.
This ensures that commonsense embedding of ’Motorbike’ can carry a great deal
of information about the embedding of ’Jetski’ because neighborhood message
passing is the underlying formulation of the trained GCN model. Our
corresponding ZSL performance gain also corroborates with this.

##### Comparison with Wang et al. (2018):

The results of this experiment are shown in Table 4. It can be seen that our
proposed approach outperforms Wang et al. (2018) by a significant margin. It
shows that our knowledge graph encoding method and its usage in the ZSL
baselines are superior.

##  6 Conclusion

In this work, we built upon ConceptNet to extract commonsense embeddings to
address the task of Zero Shot Learning (ZSL). To generate commonsense
embeddings, we applied Graph Convolutional Network on the ConceptNet
commonsense graph. Experimental results showed that our method achieved
significant performance improvement over two strong baselines on three well
known datasets.

## References

  * Akata et al. (2013) Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. 2013.  Label-embedding for attribute-based classification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 819–826.
  * Andrich et al. (2009) Christian Andrich, Leo Novosel, and Bojan Hrnkas. 2009.  Common sense knowledge.
  * Deng et al. (2014) Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. 2014.  Large-scale object classification using label relation graphs.  In _European conference on computer vision_ , pages 48–64. Springer.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee.
  * Farhadi et al. (2009) Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. 2009.  Describing objects by their attributes.  In _Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on_ , pages 1778–1785. IEEE.
  * Fellbaum (2010) Christiane Fellbaum. 2010.  Wordnet.  In _Theory and applications of ontology: computer applications_ , pages 231–243. Springer.
  * Frome et al. (2013) Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov. 2013.  Devise: A deep visual-semantic embedding model.  In _Advances in neural information processing systems_ , pages 2121–2129.
  * Gilmer et al. (2017) Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017.  Neural message passing for quantum chemistry.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 1263–1272. JMLR. org.
  * Jayaraman and Grauman (2014) Dinesh Jayaraman and Kristen Grauman. 2014.  Zero-shot recognition with unreliable attributes.  In _Advances in neural information processing systems_ , pages 3464–3472.
  * Jiang et al. (2017) Huajie Jiang, Ruiping Wan g, Shiguang Shan, Yi Yang, and Xilin Chen. 2017.  Learning discriminative latent attributes for zero-shot classification.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 4223–4232.
  * Kampffmeyer et al. (2019) Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P Xing. 2019.  Rethinking knowledge graph propagation for zero-shot learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 11487–11496.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kodirov et al. (2015) Elyor Kodirov, Tao Xiang, Zhenyong Fu, and Shaogang Gong. 2015.  Unsupervised domain adaptation for zero-shot learning.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2452–2460.
  * Kodirov et al. (2017) Elyor Kodirov, Tao Xiang, and Shaogang Gong. 2017.  Semantic autoencoder for zero-shot learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3174–3183.
  * Kumar Verma et al. (2018) Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, and Piyush Rai. 2018.  Generalized zero-shot learning via synthesized examples.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4281–4289.
  * Lampert et al. (2009) CH. Lampert, H. Nickisch, and S. Harmeling. 2009.  Learning to detect unseen object classes by between-class attribute transfer.  In _CVPR_.
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b.  Distributed representations of words and phrases and their compositionality.  In _Advances in neural information processing systems_ , pages 3111–3119.
  * Norouzi et al. (2013) Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, and Jeffrey Dean. 2013.  Zero-shot learning by convex combination of semantic embeddings.  _arXiv preprint arXiv:1312.5650_.
  * Qiao et al. (2017) Ruizhi Qiao, Lingqiao Liu, Chunhua Shen, and Anton van den Hengel. 2017.  Visually aligned word embeddings for improving zero-shot learning.  _arXiv preprint arXiv:1707.05427_.
  * Roy et al. (2018) Abhinaba Roy, Jacopo Cavazza, and Vittorio Murino. 2018.  Visually-driven semantic augmentation for zero-shot learning.  In _BMVC_ , page 85.
  * Salakhutdinov et al. (2011) Ruslan Salakhutdinov, Antonio Torralba, and Josh Tenenbaum. 2011.  Learning to share visual appearance for multiclass object detection.  In _CVPR 2011_ , pages 1481–1488. IEEE.
  * Schlichtkrull et al. (2018) Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018.  Modeling relational data with graph convolutional networks.  In _The Semantic Web - 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings_ , volume 10843 of _Lecture Notes in Computer Science_ , pages 593–607. Springer.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_.
  * Speer et al. (2017) Robert Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Thirty-First AAAI Conference on Artificial Intelligence_.
  * Sung et al. (2018) Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018.  Learning to compare: Relation network for few-shot learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1199–1208.
  * Wang et al. (2019) Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. 2019.  A survey of zero-shot learning: Settings, methods, and applications.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 10(2):1–37.
  * Wang et al. (2018) Xiaolong Wang, Yufei Ye, and Abhinav Gupta. 2018.  Zero-shot recognition via semantic embeddings and knowledge graphs.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6857–6866.
  * Xian et al. (2017) Yongqin Xian, Bernt Schiele, and Zeynep Akata. 2017.  Zero-shot learning-the good, the bad and the ugly.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4582–4591.
  * Yang et al. (2014) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014.  Embedding entities and relations for learning and inference in knowledge bases.  _arXiv preprint arXiv:1412.6575_.
  * Zhan et al. (2017) Xueying Zhan, Yaowei Wang, Yanghui Rao, Haoran Xie, Qing Li, Fu Lee Wang, and Tak-Lam Wong. 2017.  A network framework for noisy label aggregation in social media.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 484–490, Vancouver, Canada. Association for Computational Linguistics.
  * Zhang et al. (2017) Li Zhang, Tao Xiang, and Shaogang Gong. 2017.  Learning a deep embedding model for zero-shot learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2021–2030.