# Multi-task Prompt Words Learning for Social Media Content Generation

Haochen Xue∗ , Chong Zhang∗, Chenzhi Liu∗, Fangyu Wu, Xiaobo Jin† * Equal
Contribution. $\dagger$ Corresponding Authors Xiaobo Jin:
<EMAIL_ADDRESS>This work was partially supported by the “Qing Lan
Project” in Jiangsu universities, Research Development Fund with No.
RDF-22-01-020 and Suzhou Science and Technology Development Planning Programme
(Grant No. ZXL2023176)  School of Advanced Technology, Xi’an Jiaotong-
Liverpool University, Suzhou, China
Email: {Haochen.Xue20, Chong.Zhang19<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The rapid development of the Internet has profoundly changed human life.
Humans are increasingly expressing themselves and interacting with others on
social media platforms. However, although artificial intelligence technology
has been widely used in many aspects of life, its application in social media
content creation is still blank. To solve this problem, we propose a new
prompt word generation framework based on multi-modal information fusion,
which combines multiple tasks including topic classification, sentiment
analysis, scene recognition and keyword extraction to generate more
comprehensive prompt words. Subsequently, we use a template containing a set
of prompt words to guide ChatGPT to generate high-quality tweets. Furthermore,
in the absence of effective and objective evaluation criteria in the field of
content generation, we use the ChatGPT tool to evaluate the results generated
by the algorithm, making large-scale evaluation of content generation
algorithms possible. Evaluation results on extensive content generation
demonstrate that our cue word generation framework generates higher quality
content compared to manual methods and other cueing techniques, while topic
classification, sentiment analysis, and scene recognition significantly
enhance content clarity and its consistency with the image.

###### Index Terms:

Artificial Intelligence Generated Content
Prompt Learning, Multimodal Data, Social Media

##  I Introduction

Social media plays a vital role in today’s world, not only as a key platform
for human connection, information sharing and cultural interaction but also
provides individuals with the opportunity to showcase themselves and benefit
on both a personal and professional level [1]. It also provides companies with
effective tools to enhance their marketing strategies and increase product
exposure. Therefore, creating high-quality tweets can significantly enhance an
individual’s social media presence and personal brand, and increase engagement
and connection with followers, promoting the sharing and exchange of ideas and
opinions. For enterprises, high-quality tweets can effectively attract more
attention, strengthen brand image and trust, and effectively disseminate
information and promotions to target audiences, thereby promoting business
development and market expansion [2].

However, despite the rapid advancement of artificial intelligence technology
in content creation, particularly with the use of GPT for text production [3],
its application in social media remains underdeveloped. Besides the issue, for
users, especially those new to social media, creating visually appealing
photos and engaging text content can be challenging [4]. Tasks like photo
editing and making captions can be difficult even for skilled users. In
general, editing pictures and content is a difficult and time-consuming job
for ordinary users. Bridging these gaps, the application of the technology not
only holds the potential to unlock new avenues for innovation and diversity in
social media content but also brings convenience to users.

The goal of our work is to achieve controllable automatic generation of social
media content through existing multi-modal artificial intelligence technology.
Specifically, we first generate a preliminary content description of the image
based on multi-modal technology. Next, based on the generated content and
original images, we perform our multi-task prompt words learning (MPWL): topic
classification, sentiment analysis, scene recognition and content keyword
extraction. These four types of prompt words are then fed into GPT through
specific templates to optimize GPT’s performance on social media generation
tasks. With the help of MPWL and ChatGPT, content can be generated that is
concise, smooth, consistent with pictures, and in line with human aesthetics.
Finally, the GroundingDINO [5] algorithm is applied to detect people in the
image so that the cropped image is centered on the person. Our work will
provide tool convenience, content diversity, and greater time efficiency for
content creation in social media in the future.

Our main contributions are as follows:

  * •

We have applied multimodal processing and large language model to the field of
social media content generation. As far as we show, we are the first work to
use these techniques for the automatic generation of social media content.

  * •

We propose a generation framework of prompt words based on multi-modal
information fusion, which combines tasks such as topic classification,
sentiment analysis, scene recognition and keyword extraction to generate more
comprehensive prompt words for social media to guide and control ChatGPT to
generate high-quality of tweets.

  * •

In the current lack of evaluation tools and standards in the field of content
generation, we use the ChatGPT tool to make it possible to evaluate
experimental results on a large scale, which is more standardized and fair
than manual evaluation.

  * •

In our experiments, the tweets generated by our just-in-time learning method
were of higher quality than manually crafted tweets, while also outperforming
tweets generated by other just-in-time learning methods.

##  II Related Work

###  II-A Artificial Intelligence-Generated Content

The research field of Artificial Intelligence-Generated Content (AIGC) has
witnessed a surge in interest, particularly for its remarkable capacity to
autonomously create diverse content forms, encompassing text, images, and
videos [6]. Within the domain of text generation, researchers have delved into
sophisticated methods utilizing deep learning techniques to craft coherent and
varied texts, as evidenced by studies such as [7] and [8]. A pivotal
development in the AIGC landscape is the emergence of Large Language Models
(LLMs), exemplified by GPT-3 [9], built on the transformative Transformer
architecture. These models, trained on massive datasets, embody extensive
general world knowledge and exhibit exceptional prowess in natural language
processing tasks. Distinguished by their large-scale training datasets and a
significant number of parameters, LLMs excel in capturing intricate linguistic
patterns and nuances. Their effectiveness spans a spectrum of tasks, including
natural language inference, question answering, and code generation [10, 11,
12].

Recently, researchers have leveraged the few-shot capabilities of LLMs across
diverse applications, such as translation, text summarization, and common
sense reasoning [13, 14]. Brooks et al. [13] harnessed GPT-3 for generating
instructions and editing captions datasets, contributing to the training of
models for image editing. Schick et al. [14] utilized GPT-3’s few-shot
capability to curate a dataset of external API calls, subsequently fine-tuning
another LLM. In a similar vein, Peng et al. [15] utilized GPT-4 outputs to
construct an instruction-following dataset, facilitating the subsequent fine-
tuning of other LLMs in a supervised learning fashion.

Despite these advancements, the field of generative AI faces challenges,
especially in the intricate domain of social media content generation, where
tasks extend beyond text to include the creation of graphics. Additionally,
controlling generative AI remains a formidable task, given that text prompts
serve as the primary interface. While re-prompting in a chat-like environment
offers a method for refining outputs [16], the precise identification of
issues and corresponding solutions remains a challenge. To address these
challenges, this work proposes a novel Multi-task Prompt Word Learning
approach. This approach aims to enhance control over the generation process of
GPT and elevate the quality of generated social media content.

###  II-B Prompt Learning

To adapt pre-trained Large Language Models (LLMs) for downstream tasks, prompt
learning strategies [17, 18] traditionally employ task-specific textual tokens
to elicit task-specific textual knowledge [18, 19]. For instance, in CLIP
[19], a manually crafted template like ”a photo of a [CLASS]” is employed to
construct textual embeddings for zero-shot prediction. However, these manually
designed prompts exhibit limitations in offering comprehensive descriptions of
downstream tasks as they lack consideration for specific knowledge pertinent
to the task at hand.

To overcome this limitation, Context Optimization (CoOp) [20] replaces
manually crafted prompts with learnable soft prompts inferred from labeled
few-shot samples. However, CoOp’s drawback lies in generating unique and fixed
learnable prompts for each task’s images, neglecting the nuanced
characteristics of individual images. In response, Conditional Context
Optimization (CoCoOp) [21] is introduced, generating image-conditional context
for each image and amalgamating it with textual-conditional context for prompt
tuning. Specifically, a lightweight neural network is employed to generate a
vector representing learnable text prompts. ProDA [22], in its pursuit of
high-quality task-related tokens, incorporates prompt prior distribution
learning. Additionally, ProGrad [23] selectively updates prompts aligned with
the gradient of the ”general knowledge” generated by the original prompts.

Graph-ToolFormer [24] significantly enhances LLMs’ capabilities in graph
reasoning tasks. To enhance efficiency and alleviate computational load in
LLMs, LLMLingua [25] integrates budget controllers, iterative token-level
compression, and directive-based adjustments. Prompt Distillation [26]
transforms discrete prompt words (traditional text prompts) into continuous
prompt vectors, reducing computational load during input processing, as
continuous vectors are more amenable to the model compared to traditional
text. DenseCLIP [27] employs a context-aware prompt strategy for generating
dense prediction tasks, while CLIP-Adapter [28] applies an adapter to fine-
tune visual or text embeddings.

##  III Methodology

Figure 1: The pipeline of our algorithm is as follows: 1) Image features and
text features are fused after BERT and VIT; 2) Based on the fused image
features and text features, we use multi-task learning to generate prompt
words; 3) Twitter text is generated based on ChatGPT and a template filled
with prompt words; 4) The synthesized square image and Twitter text are
combined into the final tweet containing images and text.

The framework of our algorithm is as follows: 1) Image features and text
features are fused after BERT and VIT; 2) With the fused image features and
text features, we use multi-task learning including topic classification,
sentiment analysis, scene recognition and keyword extraction to generate
prompt words; 3) Twitter text is generated based on ChatGPT and a template
filled with prompt words; 4) The synthesized square image and Twitter text are
combined into the final tweet containing images and text. Below we introduce
each component of the algorithm in detail.

###  III-A Crude Content Generation

It is first processed using a ViT ImageProcessor to convert these images into
a vectorized semantic representation. These vectors are then fed into
HuggingGPT [29], an image captioning model that is trained to encode and
decode visual information using ViT and GPT-2’s [30] causal language model,
respectively. Through the above process, a preliminary text description is
generated for each image patch. It is important to note that we improve the
accuracy of the model by fine-tuning the COCO dataset[31]. Through the
inherent randomness of the model and the adopted Top-k decoding strategy, we
improve the diversity of the generated text, producing several possible
descriptions for each image.

In order to get a more accurate text description of the image, we first
generate $4$ random text descriptions for the image, then use CLIP
(Contrastive Language-Image Pre-training) [32] to score the matching of each
text and image. Finally, the text description with the highest score is
selected as the description of the image. CLIP here is a text-to-image pre-
trained model designed to learn alignment between multi-modal entities such as
images and text. Given raw inputs of text and images, CLIP will output a
similarity representation between them, where $-1$ means completely dissimilar
and $1$ means completely similar.

###  III-B Multi-task Prompt Words Learning

In this section, we will generate image-aligned text, specifically using large
language cue learning methods to generate textual descriptions that are
consistent with the visualization’s context. Prompt learning is a creative and
detail-oriented task that involves strategic choices of words, phrases,
symbols, and formatting, all designed to guide the model in producing high-
quality and relevant text.

To this end, we generate prompt words through the following multiple methods:
1) extract keywords $K=\\{k_{1},k_{2},\ldots,k_{m}\\}$ from a series of
candidate sentences; 2) classify multi-modal data to determine the topic words
$T$ of the text; 3) perform sentiment analysis on text data and evaluate their
emotional polarity $Y$ ($Y$ may be positive, negative and neutral); 4)
identify the scene $Z$ in which the story takes place through text and images.
Subsequently, we will use the prompt template of Figure 2 to synthesize the
final prompt text description with the above information $T$, $Y$, $K$ and
$Z$.

Figure 2: Prompt word template for multi-prompt word learning

#### Keyword Extraction

After obtaining the text description of the image, we first perform part-of-
speech filtering on all words in the text: analyze the part-of-speech of each
word with the help of CoreNLP [33] and retain semantic words such as
adjectives, names, and verbs as candidates $\mathcal{W}$. The final keywords
will be selected from these words.

The next step is to measure the importance of keywords: mask the sentences in
the paragraph and calculate the distance between the two sentences before and
after the masking. Given a word sequence corresponding to a sentence
$S=(w_{1},w_{2},\cdots,w_{k},\cdots,w_{n})$, we encode it into a
768-dimensional vector through BERT[34] and then calculate the cos similarity
between words degree sentence and its masked version to measure the importance
of word $w_{k}$

$h(w_{k})=\frac{\textrm{BERT}(S)^{T}\textrm{BERT}(S/w_{k})}{\|\textrm{BERT}(S)\|_{2}\|\textrm{BERT}(S/w_{k})\|_{2}},$
(1)

where $S/w_{k}$ represents the sentence after removing (masking) the word
$w_{k}$. Here, $h(w_{k})$ basically measures the similarity distance between
two sentences. The smaller the distance $h(w_{k})$, the more important the
word $w_{k}$ is. Finally, we select the most important 3 words from the
candidate words of the sentence as keywords.

#### Fusion of Multimodal Features

In this section, we attempt to fuse the feature representations of multiple
models to generate more accurate prompt representations that will support
subsequent multiple learning tasks.

To achieve this fusion effectively, we begin by utilizing specific pre-trained
models for distinct modalities. We use pre-trained ViT-384 [35] as the
backbone to extract image features and pre-trained BERT as the backbone to
extract text features.

Subsequently, we normalize all image features and word-level features to
obtain matrices $I_{l\times
m}=\begin{bmatrix}i_{1}&i_{2}&\cdots&i_{m}\end{bmatrix}$ and $W_{l\times
n}=\begin{bmatrix}w_{1}&w_{2}&\cdots&w_{n}\end{bmatrix}$, where vectors
$i_{j}$ and $w_{j}$ are unit vectors in $l$-dimensional space respectively.

In order to better integrate the semantic representation of text and images,
we use the idea of Transformer to regenerate the representation of text and
images

$\displaystyle\mathcal{C}$ $\displaystyle=$
$\displaystyle\textrm{softmax}\left(\frac{I^{T}W}{\sqrt{l}}\right)$ (2)
$\displaystyle\widehat{I}$ $\displaystyle=$ $\displaystyle W\mathcal{C}^{T},$
(3) $\displaystyle\widehat{W}$ $\displaystyle=$ $\displaystyle I\mathcal{C}.$
(4)

Here, $\widehat{I}$ and $\widehat{W}$ will be used as inputs for multiple
subsequent tasks including topic classification, sentiment analysis and scene
recognition.

#### Topic Categorization

In order to ensure the diversity of GPT model generation, we will build a
topic model to classify all samples into five categories: sports,
entertainment, technology, politics, and business. We used a web crawler to
crawl 1500 samples containing images and text from Twitter to build our model,
where the training, validation, and test sets were randomly divided in
80:10:10.

In order to process $\widehat{W}$, we design a decoder to complete the task of
topic classification, as shown in Figure 3. The Dense/ReLU layer initially
processes the fused features, providing basic nonlinearity. Next, the Add/Norm
layer with its residual connections and normalization ensures the stability
and depth of feature learning. Next comes the Feed Forward layer, which
further enhances the model’s ability to capture complex patterns. Adaptive
average pooling adjusts feature sizes for efficient processing. Finally, a
combination of dense layers and softmax functions converts these processed
features into classification probability distributions, which are crucial for
accurate topic classification.

Figure 3: Network architecture for topic analysis task

#### Sentiment Analysis

Below we predict the sentiment polarity of the semantics of image and text
representations and use it in prompt templates.

For text sequence data, we use LSTM to predict the sentiment polarity of the
text $\widehat{W}$ integrated into the image. LSTM contains LSTM cells that
process the input data of each time step in a recurrent fashion. LSTM cells
include forget gates, input gates, output gates, cell states and hidden
states. The forget gate, input gate and output gate in Eqn. (5), (6) and (7)
can operate the input data $z_{t}$ and the hidden state $h_{t-1}$ to obtain
the corresponding gating coefficients, where $\sigma$ is the sigmoid
activation function, and $\bm{W}$ and $\bm{b}$ are weights and biases. Then,
as shown in Eqn. (8), (9) and (10), the forget gate can choose to forget the
unimportant information in the previous cell state, and the input gate can
retain the important information of the input data, so as to obtain the new
cell state $c_{t}$. Finally, the output gate determines the important
information that the current cell state can output, and use the new hidden
state $h_{t}$ as the output of LSTM.

$\displaystyle f_{t}$ $\displaystyle=$
$\displaystyle\sigma(\bm{W^{f}}[h_{t-1},z_{t}]+\bm{b^{f}})$ (5) $\displaystyle
i_{t}$ $\displaystyle=$
$\displaystyle\sigma(\bm{W^{i}}[h_{t-1},z_{t}]+\bm{b^{i}})$ (6) $\displaystyle
o_{t}$ $\displaystyle=$
$\displaystyle\sigma(\bm{W^{o}}[h_{t-1},z_{t}]+\bm{b^{o}})$ (7)
$\displaystyle\tilde{c}_{t}$ $\displaystyle=$
$\displaystyle\tanh(\bm{W^{c}}[h_{t-1},z_{t}]+\bm{b^{c}})$ (8) $\displaystyle
c_{t}$ $\displaystyle=$ $\displaystyle f_{t}*c_{t-1}+i_{t}*\tilde{c}_{t}$ (9)
$\displaystyle h_{t}$ $\displaystyle=$ $\displaystyle o_{t}*\tanh(c_{t})$ (10)

#### Scene Recognition

Figure 4: Network architecture of scene recognition task

In the field of scene recognition, the accurate pairing of images and text and
the effective analysis of scene-related nouns are crucial. The transformer
module we designed, as shown in Fig. 4, contains encoders and decoders to meet
these needs. The decoder accepts three types of input: flat image features
denoted $I$, text features labeled $W$, and modality-aware prompts
$\widehat{W}$. These inputs work together within the model to enhance the
recognition and interpretation of the scene depicted in the image.

In the multi-head attention mechanism of our decoder, the text features $W$
act as the query $Q$. They encapsulate semantic details relevant to the scene,
helping the model pinpoint image fragments that resonate with the accompanying
text. These textual features are often derived from image-related
descriptions, thereby directing the model’s attention to the parts of the
image that correspond to the textual narrative.

The modality-aware prompt $\widehat{W}$ acts as a key $K$, providing
additional contextual insights. They help guide the model’s focus to important
image areas, especially those that are closely related to textual
descriptions. Essentially, these cues act as navigation aids, directing the
model’s attention to the most important elements of the image.

At the same time, the image feature $I$ serves as the value $V$, which
reflects the visual essence of the image. Within the multi-head attention
framework, a combination of text queries and modality-aware cues are used to
embed scene-related information into image features $I$. This synergy
ultimately forms the scene-aware prompt, symbolized by $R$.

In the encoding stage, scene-aware prompts $R$ and text information $W$ are
further refined through the attention mechanism. Here, $R$ assumes the roles
of $K$ and $V$, while $W$ plays the role of $Q$. This setup enables the
encoder to enhance the interaction between visual and textual elements,
thereby improving the accuracy of scene understanding. The output of the
encoder is an enhanced feature fusion that interweaves the visual narrative of
the image with its textual description, providing a more nuanced and precise
context for scene recognition. The scene recognition model’s classification
head then leverages this rich representation to extract indicator words that
accurately describe the scene.

###  III-C Synthesis of Grid Images

Generally, Twitter is a platform that allows everyone to show themselves from
multiple aspects and angles, so we usually need to process each picture into a
character-centered image. Specifically, we detect people from images using the
detection algorithm GroundingDINO, and then extend a set of images to the same
size through an image cropping algorithm. These steps can cover some
background information in the image. On the other hand, it makes the cropped
photos more beautiful.

##  IV Experiment

To demonstrate the superior performance of our model, we conducted comparative
experiments to compare our method with other prompting methods and manual
Twitter content generation. Ablation studies were designed to verify the
importance of sub-tasks within Multi-Task Prompt Learning for generating high-
quality tweets. Furthermore, a generalizability experiment was implemented to
illustrate the universality of the networks employed in our sub-tasks.

###  IV-A Dataset

In the comparative experiment, both the proposed Multi-task Prompt Word
Learning method and each of the comparative methods [24, 25, 26] generated a
dataset of 200 tweets. Each tweet exhibits a multimodal composition,
seamlessly blending textual and visual elements. The textual component is
meticulously confined to a maximum length of 40 characters. The tweet content
is deliberately curated to align thematically with five distinct categories:
Sports, Entertainment, Technology, Politics, and Business.

In the subsequent generalizability experiment, we utilized widely recognized
public datasets. For sentiment analysis network, we employed MVSA [36],
Twitter15 [37], Twitter17 [38]. While Twitter15 and Twitter17 datasets
comprise text and image modalities extracted from tweets, MVSA is a dataset
featuring text and image modalities derived from social media posts. In the
scene recognition segment, we applied Scene-15 [39], MIT Indoor-67 [40], and
Places-205 [41] to assess diverse scene recognition networks. Scene-15
concentrates on general outdoor and indoor scenes, MIT Indoor-67 specializes
in varied indoor scenes, and Places-205 encompasses over 200 different scenes
and environments, offering a comprehensive evaluation for scene recognition
tasks. It’s essential to emphasize that due to the absence of pertinent
datasets and models exhibiting comparable performance, the generalizability
experiments excluded topic classification.

TABLE I: Evaluation of Generated Social Media Content

| Manual Twitter | Graph-ToolFormer [24] | LLM-Lingua [25] | Prompt Distillation [26] | Multi-task Prompt Words Learning (ours)
---|---|---|---|---|---
The generated tweets |  |  |  |  |
Relevance & Clarity | 8.20/10 | 7.55/10 | 7.15/10 | 6.90/10 | 8.75/10
Creativity & Originality | 7.35/10 | 5.85/10 | 5.45/10 | 4.50/10 | 7.30/10
Coherence & Structure | 7.10/10 | 5.85/10 | 5.45/10 | 5.30/10 | 7.60/10
Emotional Impact | 6.75/10 | 4.55/10 | 5.40/10 | 4.40/10 | 7.25/10
Engagement | 7.15/10 | 5.70/10 | 5.55/10 | 5.25/10 | 7.60/10
Overall | 7.50/10 | 5.85 | 5.45 | 5.35 | 8.05/10

TABLE II: Ablation Study of Multi-task Prompt Word Learning Method

Method | Relevance & Clarity | Creativity & Originality | Coherence & Structure | Emotional Impact | Engagement | Overall
---|---|---|---|---|---|---
MPWL w/o Scene Recognition | 8.35 | 7.20 | 7.55 | 7.25 | 7.15 | 7.70
MPWL w/o Topic Classification | 7.90 | 7.15 | 7.45 | 7.20 | 6.80 | 7.35
MPWL w/o Sentiment Analyse | 7.75 | 7.15 | 7.40 | 6.15 | 6.60 | 7.15
w/o MPWL | 6.60 | 4.35 | 5.20 | 5.05 | 5.15 | 5.20
Ours (with MPWL) | 8.75 | 7.30 | 7.60 | 7.25 | 7.60 | 8.05

###  IV-B Implementation Details

We utilize the unified GPT-4 version [16] by OpenAI without further branching.
For Multi-task Prompt Word Learning, the topic classification model undergoes
500 epochs with a batch size of 4 and a learning rate of 0.005.
Simultaneously, the sentiment analysis network is refined over 500 epochs on
the Twitter-15 dataset with a batch size of 16 and a consistent learning rate
of 0.005. The dedicated scene recognition network is trained on the Scene-15
dataset for 300 epochs, utilizing a batch size of 16 and a slightly increased
learning rate of 0.008. All models leverage a single NVIDIA 3090 GPU
throughout training.

In the comparative experiment, we have designed a template to facilitate GPT
in scoring tweets. As illustrated in Figure 5, the evaluation of generated
tweet content encompasses four main aspects: Relevance and Clarity, Creativity
and Originality, Coherence and Structure, and Emotional Impact and Engagement.
Each aspect is rated on a scale from 1 to 10. This approach aims to reduce
biases and variations typically found in human scoring methods. In the
ablation study, this template is similarly employed, but with the addition of
blanks to account for absent subtasks.

For the generalizability experiment, we isolate sub-networks in Multi-Task
Prompt Learning and compare their performance with the corresponding task’s
network on universal datasets. We use Accuracy and F1-score to measure the
performance of different methods.

Figure 5: The Designed Template for Scoring Generated Tweets

###  IV-C Comparative Experiments

As shown in Table I, the quality of tweets generated under the guidance of
Multi-task Prompt Word Learning (MPWL) is relatively higher compared to
manually crafted tweets. MPWL outperforms in almost all evaluated aspects,
except for Creativity and Originality, where it slightly falls manually
crafted tweets. The lesser performance of other prompting learning methods
[24, 25, 26] can be linked to their unsuitability for social media content
creation. These methods [24, 25, 26] lack the specialized features necessary
to extract themes, emotions, and locations from images, and they do not have
structured templates crucial for crafting standard tweets. As a result, their
tweets often lack the emotional depth, creativity, and overall quality that
are typical of manually crafted tweets. In contrast, MPWL shows more skill in
these areas.

###  IV-D Ablation study

To further validate the effectiveness of proposed Multi-task Prompt Word
Learning (MPWL), we conducted ablation experiments with different settings.
Specifically, we performed experiment by removing the MPWL, denoted as “w/o
MPWL” in Table II. Additionally, experiments were conducted by removing each
component from the MPWL, labeled as “MPWL w/o Scene Recognition”, “MPWL w/o
Topic Classification”, or “MPWL w/o Sentiment Analyse”. The results reveal
that: (1) MPWL with all components achieves the best performance on generated
content, while removing any module results in sub-optimal results; (2) “w/o
MPWL” performs worse than MPWL, highlighting the effectiveness of scene
recognition, topic classification and sentiment analysis in enhancing the
overall quality of the content.

###  IV-E Generalizability Experiments

To further validate the generalizability of Multi-Task Media Prompting method,
we test its sub-network in different tasks, including sentiment analysis, and
scene recognition. This diverse range of tasks provided a comprehensive
evaluation of its adaptability. As shown in Table III IV, the results
confirmed that Multi-Task Media Prompting not only performs on par with the
most commonly used existing methods [42, 43, 44, 45, 46, 47] but also
showcases its superiority in managing diverse data types and meeting different
analytical requirements. This extension into various tasks emphasized the
flexibility and effectiveness of Multi-Task Media Prompting in a wide range of
applications.

TABLE III: Performance comparison (Accuracy(%) and F1(%)) of different Sentiment analysis models. | MVSA | Twitter15 | Twitter17
---|---|---|---
Model |  ACC $\uparrow$ |  F1 $\uparrow$ |  ACC $\uparrow$ |  F1 $\uparrow$ |  ACC $\uparrow$ |  F1 $\uparrow$
CNN-Multi [42] | 66.30 | 64.19 | 64.38 | 63.48 | 61.87 | 60.26
mBERT(BOTH) [43] | 68.46 | 66.83 | 75.31 | 70.18 | 69.61 | 67.12
TomBert [44] | 69.86 | 65.84 | 76.18 | 71.27 | 70.50 | 68.04
Ours | 69.12 | 66.56 | 76.31 | 71.56 | 70.21 | 68.43
TABLE IV: Performance comparison (Accuracy(%) and F1(%)) of different Scene recognition models. | Scene-15 | MIT Indoor-67 | Places-205
---|---|---|---
Model |  ACC $\uparrow$ |  F1 $\uparrow$ |  ACC $\uparrow$ |  F1 $\uparrow$ |  ACC $\uparrow$ |  F1 $\uparrow$
DAG-CNN [45] | 92.90 | 89.41 | 77.50 | 73.41 | 56.20 | 52.28
Hybrid CNNs [46] | 91.21 | 88.86 | 82.24 | 79.56 | 64.53 | 61.89
Multi-scale CNNs [47] | 95.18 | 92.87 | 86.04 | 82.69 | 70.17 | 67.56
Ours | 95.34 | 93.17 | 86.07 | 82.86 | 70.23 | 67.71

##  V Conclusion

In our work, we apply artificial intelligence technology to content generation
for social media and propose a prompt word generation framework based on
multi-modal information fusion, which uses topic classification, sentiment
analysis, and scene recognition. Generate more comprehensive prompt words to
generate high-quality tweets through ChatGPT. Furthermore, in the absence of
effective and objective evaluation methods in the field of content generation,
we propose to use the ChatGPT tool to achieve large-scale evaluation of
content generation methods. The final results show that our approach produces
higher-quality tweets compared to other on-the-fly learning methods and
manually crafted posts. In future work, we will focus on further processing
and beautification of images.

## References

  * [1] A. M. Kaplan and M. Haenlein, “Users of the world, unite! the challenges and opportunities of social media,” _Business horizons_ , vol. 53, no. 1, pp. 59–68, 2010.
  * [2] H. Alves, C. Fernandes, and M. Raposo, “Social media marketing: a literature review and implications,” _Psychology & Marketing_, vol. 33, no. 12, pp. 1029–1038, 2016.
  * [3] G. Davis, M. Grierson _et al._ , “Investigating attitudes of professional writers to gpt text generation ai based creative support tools,” _._ , 2022.
  * [4] C. T. Carr and R. A. Hayes, “Social media: Defining, developing, and divining,” _Atlantic journal of communication_ , vol. 23, no. 1, pp. 46–65, 2015.
  * [5] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu _et al._ , “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” _arXiv preprint arXiv:2303.05499_ , 2023.
  * [6] L. Yunjiu, W. Wei, and Y. Zheng, “Artificial intelligence-generated and human expert-designed vocabulary tests: A comparative study,” _SAGE Open_ , vol. 12, no. 1, p. 21582440221082130, 2022.
  * [7] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” _arXiv preprint arXiv:1710.10196_ , 2017.
  * [8] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz, “Multimodal unsupervised image-to-image translation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 172–189.
  * [9] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [10] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann _et al._ , “Palm: Scaling language modeling with pathways,” _Journal of Machine Learning Research_ , vol. 24, no. 240, pp. 1–113, 2023.
  * [11] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma _et al._ , “Scaling instruction-finetuned language models,” _arXiv preprint arXiv:2210.11416_ , 2022.
  * [12] G. Poesia, O. Polozov, V. Le, A. Tiwari, G. Soares, C. Meek, and S. Gulwani, “Synchromesh: Reliable code generation from pre-trained language models,” _arXiv preprint arXiv:2201.11227_ , 2022.
  * [13] T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 18 392–18 402.
  * [14] T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” _arXiv preprint arXiv:2302.04761_ , 2023.
  * [15] B. Peng, C. Li, P. He, M. Galley, and J. Gao, “Instruction tuning with gpt-4,” _arXiv preprint arXiv:2304.03277_ , 2023.
  * [16] OpenAI, “Gpt-4 technical report,” 2023.
  * [17] Z. Gan, L. Li, C. Li, L. Wang, Z. Liu, J. Gao _et al._ , “Vision-language pre-training: Basics, recent advances, and future trends,” _Foundations and Trends® in Computer Graphics and Vision_ , vol. 14, no. 3–4, pp. 163–352, 2022.
  * [18] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” _ACM Computing Surveys_ , vol. 55, no. 9, pp. 1–35, 2023.
  * [19] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8748–8763.
  * [20] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models,” _International Journal of Computer Vision_ , vol. 130, no. 9, pp. 2337–2348, 2022.
  * [21] ——, “Conditional prompt learning for vision-language models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16 816–16 825.
  * [22] Y. Lu, J. Liu, Y. Zhang, Y. Liu, and X. Tian, “Prompt distribution learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5206–5215.
  * [23] B. Zhu, Y. Niu, Y. Han, Y. Wu, and H. Zhang, “Prompt-aligned gradient for prompt tuning,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2023, pp. 15 659–15 669.
  * [24] J. Zhang, “Graph-toolformer: To empower llms with graph reasoning ability via prompt augmented by chatgpt,” _arXiv preprint arXiv:2304.11116_ , 2023.
  * [25] H. Jiang, Q. Wu, C.-Y. Lin, Y. Yang, and L. Qiu, “Llmlingua: Compressing prompts for accelerated inference of large language models,” _arXiv preprint arXiv:2310.05736_ , 2023.
  * [26] L. Li, Y. Zhang, and L. Chen, “Prompt distillation for efficient llm-based recommendation,” in _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_ , 2023, pp. 1348–1357.
  * [27] Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang, J. Zhou, and J. Lu, “Denseclip: Language-guided dense prediction with context-aware prompting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 082–18 091.
  * [28] P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao, “Clip-adapter: Better vision-language models with feature adapters,” _International Journal of Computer Vision_ , pp. 1–15, 2023.
  * [29] Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang, “Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface,” _arXiv preprint arXiv:2303.17580_ , 2023.
  * [30] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” _ACM computing surveys (CSUR)_ , vol. 54, no. 10s, pp. 1–41, 2022.
  * [31] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 2014, pp. 740–755.
  * [32] M. Li, R. Xu, S. Wang, L. Zhou, X. Lin, C. Zhu, M. Zeng, H. Ji, and S.-F. Chang, “Clip-event: Connecting text and images with event structures,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 16 420–16 429.
  * [33] C. D. Manning, M. Surdeanu, J. Bauer, J. R. Finkel, S. Bethard, and D. McClosky, “The stanford corenlp natural language processing toolkit,” in _Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations_ , 2014, pp. 55–60.
  * [34] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [35] K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu _et al._ , “A survey on vision transformer,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 45, no. 1, pp. 87–110, 2022.
  * [36] T. Niu, S. Zhu, L. Pang, and A. El Saddik, “Sentiment analysis on multi-view social data,” in _MultiMedia Modeling: 22nd International Conference, MMM 2016, Miami, FL, USA, January 4-6, 2016, Proceedings, Part II 22_. Springer, 2016, pp. 15–27.
  * [37] X. Liu, A. Nourbakhsh, Q. Li, R. Fang, and S. Shah, “Real-time rumor debunking on twitter,” in _Proceedings of the 24th ACM international on conference on information and knowledge management_ , 2015, pp. 1867–1870.
  * [38] B. Wang and W. Lu, “Learning latent opinions for aspect-level sentiment classification,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [39] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories,” in _2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06)_ , vol. 2. IEEE, 2006, pp. 2169–2178.
  * [40] A. Quattoni and A. Torralba, “Recognizing indoor scenes,” in _2009 IEEE conference on computer vision and pattern recognition_. IEEE, 2009, pp. 413–420.
  * [41] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, “Learning deep features for scene recognition using places database,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [42] G. Cai and B. Xia, “Convolutional neural networks for multimedia sentiment analysis,” in _Natural Language Processing and Chinese Computing: 4th CCF Conference, NLPCC 2015, Nanchang, China, October 9-13, 2015, Proceedings 4_. Springer, 2015, pp. 159–167.
  * [43] J. Yu and J. Jiang, “Adapting bert for target-oriented multimodal sentiment classification,” in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence_. IJCAI, 2019.
  * [44] ——, “Adapting bert for target-oriented multimodal sentiment classification,” in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence_. IJCAI, 2019.
  * [45] S. Yang and D. Ramanan, “Multi-scale recognition with dag-cnns,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1215–1223.
  * [46] G.-S. Xie, X.-Y. Zhang, S. Yan, and C.-L. Liu, “Hybrid cnn and dictionary-based models for scene recognition and domain adaptation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 27, no. 6, pp. 1263–1274, 2015.
  * [47] L. Herranz, S. Jiang, and X. Li, “Scene recognition with cnns: objects, scales and dataset bias,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 571–579.