# Are Large Language Models (LLMs) Good Social Predictors?

Kaiqi Yang1, Hang Li1, Hongzhi Wen1, Tai-Quan Peng2, Jiliang Tang1, Hui Liu1
1Department of Computer Science and Engineering, Michigan State University
2Department of Communication, Michigan State University
<EMAIL_ADDRESS>
(Nov 2023)

###### Abstract

The prediction has served as a crucial scientific method in modern social
studies. With the recent advancement of Large Language Models (LLMs), efforts
have been made to leverage LLMs to predict the human features in social life,
such as presidential voting. These works suggest that LLMs are capable of
generating human-like responses. However, we find that the promising
performance achieved by previous studies is because of the existence of input
shortcut features to the response. In fact, by removing these shortcuts, the
performance is reduced dramatically. To further revisit the ability of LLMs,
we introduce a novel social prediction task, Soc-PRF Prediction, which
utilizes general features as input and simulates real-world social study
settings. With the comprehensive investigations on various LLMs, we reveal
that LLMs cannot work as expected on social prediction when given general
input features without shortcuts. We further investigate possible reasons for
this phenomenon that suggest potential ways to enhance LLMs for social
prediction.

Are Large Language Models (LLMs) Good Social Predictors?

Kaiqi Yang1, Hang Li1, Hongzhi Wen1, Tai-Quan Peng2, Jiliang Tang1, Hui Liu1
1Department of Computer Science and Engineering, Michigan State University
2Department of Communication, Michigan State University<EMAIL_ADDRESS>

##  1 Introduction

Prediction is one of the crucial elements of the scientific methods in social
studies Hofman et al. (2017), with a body of literature Liben-Nowell and
Kleinberg (2003); Bakshy et al. (2011); Cheng et al. (2014) devoted to
estimating unobserved or missing data based on observed features.
Historically, social prediction is made by statistical models such as linear
regression Uyanık and Güler (2013). With the development of machine learning,
supervised methods have been adopted, including random forest, Support Vector
Machines (SVM), and neural networks Chen et al. (2021b). However, the classic
machine learning methods notably rely on extensive labeled training datasets,
which is labor-intensive, especially in social studies. Additionally, the
predictive power of machine learning methods is limited Mackenzie (2015);
Athey (2018) and can hardly model the complicated phenomenon in social life.

Meanwhile, Large Language Models (LLMs) have advanced various text-related
tasks, such as question-answering Zhuang et al. (2023); Tan et al. (2023),
code-generation Nijkamp et al. (2022); Chen et al. (2021a), and math word
problems solving Zhou et al. (2022); Wei et al. (2022)). The extensive world
knowledge Zhao et al. (2023) and inference abilities Creswell et al. (2022) of
LLMs have the potential to mitigate the limitations of classic machine
learning methods in predicting features of social datasets. Therefore, there
are recent works leveraging LLMs in predicting and simulating human responses,
such as voting decisions Argyle et al. (2022); von der Heyde et al. (2023) and
political attitude Rosenbusch et al. (2023). They take advantage of LLMs to
augment existing datasets with previously inaccessible features and promising
performance is reported. However, our preliminary investigation revisiting the
case of voting prediction Argyle et al. (2022) with LLMs indicates that their
performance is bolstered by the presence of shortcuts to the desired response
features. Specifically, these shortcuts arise when the input contains features
directly associated with the feature to be predicted, leading to the
exceptional performance of both machine learning models and LLM-based methods.
Unfortunately, this efficiency comes with a downside: it overlooks the
essential task of uncovering authentic relationships between various features
and the label. When these shortcuts are eliminated, we observe a significant
decline in the LLMs’ effectiveness in addressing social study issues, as
detailed in Section 2. This observation leads us to question the true
capability of LLMs in social predictions, challenging the prevailing
perception of their prowess Argyle et al. (2023).

To investigate this problem, we introduce a set of studies that utilize
general social features as input and simulate real-world settings of feature
prediction. In particular, to comprehensively understand the power of LLMs in
social predicting, we introduce a new task, Soc-PRF Prediction (stands for
Social Profile Prediction). It is designed to predict the social features of
individuals while accounting for selected features as explanatory and response
variables. Besides, informed by social studies Bailey (1998), we categorize
social features into two groups that capture individuals’ features from
different perspectives. This enables us to design three distinct settings,
which rigorously categorize features into groups and assess LLMs’ predictive
capacities. In this work, we evaluate various LLMs, including closed-sourced
models GPT 3.5 OpenAI (2022), GPT 4 Achiam et al. (2023), and Gemini Pro Anil
et al. (2023), as well as lighter open-sourced models like Llama-7B,
Llama-7B-chat Touvron et al. (2023) and Mistral-7B Jiang et al. (2023). Our
studies suggest that LLMs cannot work on social prediction with general input
features without shortcuts. We further explore the potential reasons and
future directions to enhance LLMs for social prediction.

##  2 Revisit Voting Prediction with LLMs

Large Language Models (LLMs) have demonstrated impressive performance across
several societal domains, notably in predicting voting decisions in the United
States Argyle et al. (2022); Veselovsky et al. (2023). In this section, we
revisit the voting prediction study with LLMs in Argyle et al. (2022).

The voting prediction study in Argyle et al. (2022) adopts the American
National Election Studies ANES  to construct the dataset. ANES is a survey
conducted in every presidential election year, with features about American
public life, especially political views and decisions. To elicit LLMs’
prediction of individual voting decisions, the study selects 10 input
features, i.e., racial/ethnic self-identification, gender, age, ideological
self-identification, party identification, political interest, church
attendance, if discussing politics with family/friends, patriotism feelings,
state of residence. With all the features above, prompts of individual
profiles are constructed with a first-person template and a question to elicit
prediction. Then the prompts are fed into LLMs for completion, and the words
filled in by LLMs serve as the predicted voting decisions. An example of the
prompts is:

> Racially, I am white. I am male. Ideologically, I describe myself as
> conservative. Politically, I am a strong Republican … In 2016, I voted for:

Intuitively two of the input features are likely equivalent to voting
decisions, i.e. ideological self-placement and party identification. It is
evident from political studies Miller (1991); Dalton (2016) that given the
partisan nature of American politics, voting decisions are closely related to
these two features. To validate the intuition, we calculate their Cramer’s V
scores with the voting decisions. Cramer’s V is a measurement of association
between features, ranging from 0 to 1; the score 0 indicates no association
and 1 indicates a perfect association. We find that these two features are
highly correlated with the vote decisions. For example, in ANES 2016 wave
data, these two features have Cramer’s V scores of $0.86$ and $0.76$,
respectively. This indicates their strong associations with voting decisions
and consequently, they can become the shortcuts to make predictions.

Next, we conduct further experiments to study the impact of such shortcuts on
prediction performance. We choose both GPT-based approaches and classic
supervised machine learning models. For GPT-based approaches, we deploy GPT
3.5 as the basis and follow the prompts and zero-shot setting in Argyle et al.
(2022) to make predictions based on individual profiles. For classic
supervised machine learning models, we choose the Random Forest Classifier.
Since the supervised classifier needs labeled data to train, we split the
dataset into 80%/10%/10% as training, validation, and test sets. There are two
settings for each method: (1) Full, taking the full set of input features; (2)
w/o shortcut, taking input features without the shortcut features. To evaluate
the performance, we deploy accuracy as the metric, given the balanced
distribution of the voting decision (51.9% vs. 48.1%). Besides, we adopt
Cohen’s Kappa $\kappa$ as a metric to evaluate the agreement between
prediction and true data. Cohen’s Kappa $\kappa$ has values ranging from 0 to
1, where 1 indicates stronger agreement and 0 indicates almost no agreement.

Figure 1: The performance of voting prediction. In this table, GPT stands for
the LLM-based approach and we choose GPT 3.5 following Argyle et al. (2022).
The Full indicates settings with all input features, and w/o shortcut stands
for settings without the two shortcut features.

As shown in Figure 1, the GPT-based approach with all input features achieved
the accuracy of 90.82% and the Cohen’s Kappa $\kappa$ of 0.83, which aligns
with Argyle et al. (2022). However, when removing the two shortcut features,
the performance of both methods drops dramatically. For example, the
performance of GPT 3.5 drops from the accuracy of 90.82% and $\kappa$ of 0.83
to an accuracy of 61.60% and $\kappa$ of 0.43; similarly, the performance of
Random Forest drops from 90.29%, 0.78 to 69.22%, 0.23. Given the balanced
distribution of the voting decision feature, the performance without shortcut
features is considerably unsatisfactory.

Our preliminary study suggests that the promising performance of social
prediction via LLMs reported by prior works Argyle et al. (2022) could be from
the existence of shortcut features. This finding motivates us to question if
LLMs are really powerful in social prediction. To explore this question, we
design a set of studies that avoid shortcut features as inputs and resemble
realistic scenarios in the following section.

##  3 Social Profile Prediction

In this section, we introduce a social prediction task that evaluates the
predictive power of LLMs without shortcuts. First, we construct a social
prediction dataset based on survey data where we validate that there are no
shortcut features. Then we introduce three settings of evaluations to simulate
real-world scenarios. Finally, we show the performance of LLMs’ prediction in
the proposed settings and discuss the results.

###  3.1 Task and Dataset

As illustrated in Section 2, the inclusion of shortcut features can affect the
evaluation of the power of LLMs in social prediction. Therefore, we design
Soc-PRF Prediction that takes individual social features as input to predict
other missing features in profiles. Next, we introduce the dataset for the
task.

The dataset derives from Gallup World Poll Gallup (2009), one of the most
prestigious social surveys that guarantees reliability and offers various
types of features. Initialized in 2006, the Gallup World Poll has been
conducted in over 150 countries and follows strict random sampling. Questions
in the Gallup World Poll are designed by political scientists, measuring key
indicators of social life, such as law and order, financial life, civic
engagement, etc. Besides, it collects individual demographic data to construct
the survey dataset Tortora et al. (2010).

In this paper, we construct the dataset based on Gallup World Poll Gallup
(2009) and its corresponding questions. We pick the data from the USA and the
data we use in this work is primarily collected between 2016 and 2020. To keep
the basic information complete, we remove all the samples with missing data in
demographic features. After the data cleaning, the dataset includes 4,941
profiles of American individuals (samples). Additionally, we pick a set of
features from the survey to construct the profiles, which encompasses 16
social features reflecting a variety of socio-demographic characteristics,
attitudes, and behaviors.

###  3.2 Task Settings

In social studies, social datasets have predominantly been derived from two
methodologies: traditional surveys and online data collection Couper (2017);
Diaz et al. (2016); Callegaro et al. (2014). As one characteristic of social
features, mutability measures the features’ propensity to change or be
influenced by social context. Following social studies Bailey (1992);
Brensinger and Eyal (2021); Sen and Wasow (2016); Halley (2017), social
features can be roughly divided into two mutability groups: high-mutability
and low-mutability. In most social datasets, features with high mutability and
low mutability can hardly be collected simultaneously. For example, although
survey data collected through in-person interviews is of high quality, it
predominantly captures features of low mutability; the dynamic tracking of
highly mutable features across all topics and times is nearly impractical due
to associated costs. On the other hand, online data collection methods, such
as crawling posts from social networking platforms, have the advantage of
collecting high-mutability features. Facilitated by natural language
processing (NLP) tools Alghamdi and Alfalqi (2015); Vayansky and Kumar (2020);
Hussein (2018); Yue et al. (2019), real-time human attitudes and opinions are
easy to collect. Yet, features of low mutability (e.g. demographic features)
often remain inaccessible due to privacy constraints.

Among 16 social features in our proposed dataset, we assign them to low-
mutability and high-mutability groups, respectively. The low-mutability
features are socio-demographic features, including age, gender, marriage,
education, employment, income, and urbanicity. Here the age, gender, marriage
status, employment status, urbanicity of residence refer to the individual
status when taking the interview, while education refers to the highest
completed level of education, and income is the annual household income of
last year. The high-mutability features are all about attitudes or behaviors
of social life, whose topics include Internet access, social life, economic
confidence, civic engagement, and approval of leadership. For each topic,
there are one to three questions around it. In save of space, we denote the
questions as IA, SL1, SL2, EC1, EC2, CE1, CE2, CE3, AL, respectively. The
details of questions are shown in Appendix A.1.

According to features’ mutability, we design three settings to assess the
capability of LLMs in predicting missing features for the individual profiles,
which simulate real-world scenarios for social data: from low-mutability
features to predicting high-mutability features (for survey data), and from
high-mutability features to predicting high-mutability or low-mutability
features (for online data). Following the prior works especially Argyle et al.
(2022), we set all three settings as zero-shot, without taking any labeled
data as input.

low2high. In this setting, the input features possess low mutability, and the
output features exhibit high mutability. This setting is designed for
traditional survey datasets, where low-mutability features (e.g. demographic
features) are comprehensively collected, but high-mutability features (such as
attitudes) are sparse.

To construct prompts of individual profiles, we adopt the template in Argyle
et al. (2022), designing the prompt as a self-description of individuals and
inducing the LLMs to predict missing features by completing the self-
description. One example of the prompt is:

> I am a male in the USA. I am 42 years old. My current marital status is
> married. My highest completed level of education is middle level. My current
> employment status is employed. My Annual Household Income is $12600. I am
> from a suburb of a large city.
>
> When I’m asked "Do you have access to the Internet in any way, whether on a
> mobile phone, a computer, or some other device?", my answer is

In the provided prompt, the underlined text indicates the values of individual
features, and italicized content presents the question to elicit responses.
For the subsequent settings, we utilize prompts with the same template.

high2low: This setting denotes the prediction from high-mutability to low-
mutability features. To construct the input profiles of individuals, we take
values from all 9 high-mutability features, with a question about one low-
mutability feature. Serving as the inverse setting of low2high, this setting
is designed for profile construction using online data: aided with NLP tools,
the in-time attitudes of individuals can be captured from online posts with
ease, yet their demographic features are inaccessible.

Figure 2: Correlation between features in the dataset. The metric is Cramer’s
V, and values close to 1 indicate strong correlations.

high2high. In this setting, high-mutability features are utilized as input to
predict other high-mutability features. Different from high2low setting, in
order to avoid shortcuts, the features sharing the same topic with the
response feature are excluded from the input profile prompts; rather, we put
all high-mutability features of each topic as inputs. This setting simulates a
specific real-world scenario, where individuals’ attitudes toward one topic
are collected, but their opinions on other topics of interest remain
unexpressed.

Evaluation Metrics. Most features in the dataset have imbalanced
distributions. For example, the feature IA has 91.82% samples with "yes"
labels, while only 8.18% samples with "no". In this situation, accuracy is not
a suitable metric for imbalanced predictions Gu et al. (2009). As a result, we
employ AUC as the metric to evaluate the classification performance.

###  3.3 Feature Analysis

To ensure that there are no shortcut features used as input for predictions,
we first check the correlations (i.e., the Cramer’s V) of all feature pairs.
As shown in Figure 2, most of the Cramer’s V scores are less than 0.5. The
maximum (between CE2 and CE3) is 0.58, which is merely a moderate level of
correlation. This relatively high correlation is because they share a similar
topic (i.e., civic engagement). Therefore, in the following evaluations, we
will not consider features that share the same topic with the response as
inputs.

To evaluate the predictive power of the selected features, we follow the
traditional supervised setting. Take the setting of low2high as the example,
we first chose Random Forest Classifier as the basis model, and split the
dataset by 80%/10%/10% as training, validation, and test sets. The results are
shown in Figure 3, the AUC scores are much higher than those of the random
guessing method. For example, the AUC score of IA is 95.07, while its
corresponding score of random guessing is 48.34. These observations suggest
that though the selected features are not shortcuts, they are still powerful
in predicting the target responses.

Figure 3: Performance of Random Forest and Random Guessing. The metric is AUC.

###  3.4 LLMs as the Predictor

In this section, we leverage LLMs for the Soc-PRF Prediction task with the
aforementioned three settings. The results for the three settings are
illustrated in Table 1, Table 2, and Figure 4, respectively. In the tables,
"Random" indicates the random guessing method. Note that for the settings
high2high, we only partially show the results because the observations are
similar. We note that the performance of LLMs is closely similar to the random
guessing and is far from satisfactory. The poor results appear consistently in
all the settings and with all the LLMs. These observations indicate that LLMs
are struggling to distinguish individual features with the given information
in the proposed settings.

Table 1: Performance of LLMs of setting low2high.

Model | IA | SL1 | SL2 | EC1 | EC2 | CE1 | CE2 | CE3 | AL
---|---|---|---|---|---|---|---|---|---
Random | 48.34 | 52.09 | 48.47 | 52.12 | 50.07 | 49.89 | 49.16 | 49.32 | 48.60
Llama-7B | 50.00 | 50.00 | 50.00 | 48.75 | 55.41 | 50.00 | 50.00 | 50.00 | 50.00
Llama-7B-chat | 50.00 | 50.00 | 50.00 | 50.95 | 51.80 | 50.00 | 50.00 | 50.00 | 50.00
Mistral-7B | 50.00 | 50.00 | 50.00 | 53.12 | 56.89 | 50.00 | 50.00 | 50.00 | 50.00
Gemini Pro | 50.00 | 50.00 | 50.00 | 50.76 | 60.93 | 50.00 | 50.00 | 50.00 | 50.00
GPT-3.5 | 50.00 | 50.00 | 50.00 | 52.63 | 58.20 | 50.00 | 50.00 | 50.00 | 50.00
GPT-4 | 50.00 | 50.00 | 50.00 | 53.82 | 56.57 | 50.00 | 50.00 | 50.00 | 50.00

Table 2: Performance of LLMs (GPT 3.5 and GPT 4) of setting high2low.

Model | age | gender | marriage | education | employment | income | urbanicity
---|---|---|---|---|---|---|---
Random | 49.50 | 49.62 | 49.45 | 49.99 | 50.54 | 48.14 | 50.22
Llama-7B | 33.50 | 49.81 | 50.00 | 55.15 | 50.00 | 50.05 | 49.85
Llama-7B-chat | 40.00 | 50.00 | 50.00 | 35.21 | 50.33 | 51.18 | 50.09
Mistral-7B | 33.55 | 49.81 | 50.00 | 55.15 | 50.00 | 50.05 | 49.85
Gemini Pro | 38.80 | 51.14 | 50.00 | 66.70 | 50.00 | 50.10 | 49.75
GPT-3.5 | 41.35 | 50.00 | 51.29 | 57.76 | 49.59 | 50.95 | 50.94
GPT-4 | 40.75 | 50.00 | 50.88 | 65.65 | 52.01 | 53.80 | 52.09

Figure 4: Performance of GPT 3.5 of setting high2high. The metric is AUC
score. The sign "-" indicates no valid data, either because the input features
(Y-axis) and output features (X-axis) share the same topic, or they are not
conducted simultaneously in the survey.

###  3.5 Discussions

####  3.5.1 Population v.s. Individual

As shown in the previous subsection, even advanced LLMs like GPT-4 encounter
challenges in accurately predicting social features, often yielding outcomes
similar to random guessing. To explore the underlying reason for such
phenomena, we use the distribution comparisons between predicted features and
true ones under the setting low2high as our case studies; the results are
shown in Figure 5. From the analysis, we have the following observations.
First, LLM’s prediction of less mutable features, such as IA and SL, is prone
to share similar distribution patterns with the true ones. This fact indicates
that LLMs do contain some global knowledge about these social features at the
population level, but they face challenges in building precise connections to
different individual samples during the prediction. Thus, when making
individual-level predictions, LLMs may simply generate random samples from the
feature distribution. Second, the population-level patterns of highly mutable
features, such as different CEs, are seemingly not captured by existing LLMs,
and LLMs always prefer to generate negative responses on these features. This
fact indicates that building accurate predictors with LLMs for those highly
mutable features is more challenging as it not only requires LLMs to establish
the connection to individual samples but also external population-level
knowledge about the features.

Figure 5: Distributions of Social Features. Note that the last two features
(civic engagement behaviors) are more mutable than the first two.

####  3.5.2 Incorporating Labeled Data

Although the performance of zero-shot LLMs’ prediction is much worse than our
expectation, the strong performance of the random forest classifier in Figure
3 indicates that our designed prediction task is reasonable if sufficient
labeled data is considered. Based on this finding, we explore the
effectiveness of incorporating supervision signals to LLMs based on the
low2high setting as one example. Following the prior studies Brown et al.
(2020); Song et al. (2023), we leverage the in-context learning ability of
LLMs Dong et al. (2022); Zhang et al. (2023) to incorporate a few labeled
samples as demonstrations. To be specific, for each individual profile, we
sample a few other individual profiles from the dataset as a reference, whose
year and marriage features are of the same group. Then, besides constructing
vanilla prompts as introduced in Section 3.2, we add the full information
(including the input and output features) of these reference samples in the
prompts. Finally, the LLMs are asked to make predictions as the original
settings. One example of such prompts is:

> Here are self-descriptions of two people: "I am a male in the USA … my
> answer is yes"; "I am a female in the USA … my answer is no"; …
>
> I am a male in the USA. I am 42 years old …;
>
> When I’m asked "Do you have access to the Internet in any way, whether on a
> mobile phone, a computer, or some other device?", my answer is

With the augmented prompts, we introduce samples with similar input features
and show the true labels of these samples. Like the supervised methods, these
demonstrations allow LLMs to make predictions with the help of supervision
signals. To provide comprehensive information, the positive and negative
labels are balanced within the reference samples. The results of experiments
with 2 and 4 demonstrations are shown in Table 3. With all selected features,
the prompts with demonstrations help LLMs to achieve better prediction
performance. However, when incorporating 4 demonstrations into the prompt, we
observe marginal or no improvement compared with the setting with 2
demonstrations. This observation suggests that how to effectively incorporate
more demonstrations for LLMs for social prediction still faces great
challenges.

####  3.5.3 Enriching Input Features

As aforementioned, in reality, no matter survey data or online data, we face
constraints to collect sufficient features to describe individuals. In fact,
we find that such constraints may also limit the power of LLMs to discriminate
different individuals in social prediction. For example, in the low2high
setting, we can identify a sub-group from the population characterized by all
low-mutability features: male, married, age between 30 and 60, higher level of
education, fully employed, annual household income within the middle 30%,
living in suburbs of large cities. Within this subgroup, all samples share the
same input; however, their responses are varied significantly. For example, to
the question CE1, 60.53% of samples answered "no" and 29.47% answered "yes".
This variability indicates the lack of discriminative features poses
significant challenges in precisely predicting individual responses. To unlock
the potential of LLMs in predicting social features, enriching input features
is crucial.

Table 3: The performance of LLMs (GPT 3.5) with a few demonstrations.

| Zero-Shot | 2 Demos | 4 Demos
---|---|---|---
IA | 50.00 | 71.61 | 82.67
SL2 | 50.00 | 50.60 | 50.04
EC1 | 52.63 | 50.52 | 53.47
CE1 | 50.00 | 60.17 | 55.34
CE2 | 50.00 | 53.22 | 52.79
AL | 50.00 | 52.03 | 50.80

##  4 Related Work

In this section, we give an overview of works related to classic machine
learning methods and LLMs in social studies.

###  4.1 Machine Learning Methods in Social Prediction

Quantitative social studies have deployed machine learning methods to model
and predict social features in replace of the classical statistical models
like OLS and logistic regression Chen et al. (2021b); Hindman (2015). In
studies of criminology, K-means and neural networks are deployed to predict
criminal behaviors Reier Forradellas et al. (2020). To predict communication
phenomena, the random forest models are utilized to predict the view count of
online posts Hsu et al. (2017). Besides, Dong et al. (2018) uses the SVM
classifier to predict public emotions to social news. In addition, deep
learning methods have also been widely used in social prediction. For example,
CNN models are used to process image data in social studies Messer and Fausser
(2019); Dong et al. (2018); RNN and its variants have been utilized to predict
sequential data like stock changes Wu et al. (2018) and user interests Liu et
al. (2018). However, these methods either suffer from the need for massive
training data or the limited predictive power of models, highlighting the need
for more powerful prediction models.

###  4.2 Using LLMs to Predict Social Features

With the advent of LLMs, predicting social features with LLMs has been studied
by numerous works Ziems et al. (2023); Veselovsky et al. (2023). Among social
studies, LLMs have been deployed to predict the potential responses or
outcomes with ease, especially in scenarios where traditional methods are
constrained by cost or ethical concerns. In economics, Phelps and Russell
(2023) studied game theory by examining cooperative and competitive behaviors
with LLMs. Within political science, Wu et al. (2023) deployed LLMs to predict
the ideological views of politicians. For communication studies, LLMs are used
to simulate and predict the potential outcomes of toxic discourse Törnberg et
al. (2023), the political affiliation of Twitter posts Törnberg (2023), etc.

Additionally, there are growing interests in leveraging LLMs with social
survey and interview, aiming to replicate human-like responses to certain
questions or attributes of individuals. For example, Argyle et al., 2022
proposed "silicon samples" that deploy LLMs to simulate the people in a survey
or interview and predict their partisan views and voting decisions. Dillion et
al., 2023 examined the LLMs response to psychological tests, comparing the
decisions and judgements from LLMs and humans. Aher et al., 2023 proposed sets
of experiments to check LLMs response to interview and games. Besides, fine-
tuning LLMs is a promising method for better prediction of social attitudes
across years of surveys Kim and Lee (2023). At the same time, discussions
Jansen et al. (2023) are hold about the potential and risks of deploying LLMs
in social survey studies.

##  5 Conclusion

In this study, we introduce a survey-based social prediction task to assess
the LLMs’ predictive ability using general features. Through the replication
of experiments and ablation studies of voting prediction tasks, we reveal a
significant performance gap between input prompts with and without shortcut
features. To further study the LLMs’ predictive ability, we propose a real-
world survey dataset with rigorously selected features. Based on it, we
demonstrate the inability of LLMs to predict social features only with general
features. Furthermore, our empirical studies further showcase the potential
reasons that constrain the LLMs’ predictive power. In our future research, we
aim to explore the efficient methods of providing supervision signals and
reference information to improve LLMs prediction performance. Moreover, with
the abundant social survey and online data, we plan to use fine-tuning methods
to fit the LLMs knowledge with social prediction tasks.

##  6 Limitations

In this study, we examine the predictive power of LLMs. We replicate a voting
prediction study and find the shortcuts producing plausible results. However,
we do not investigate how to select informative features that enhance
prediction and what is the upper bound of this task. Then we conduct
comprehensive experiments of social prediction, and contend that incorporating
labeled data and enrich input features could benefit social prediction.
However, we do not provide experiments to validate these suggestions. Lastly,
the LLMs are not further tuned and the prompts are adopted from prior works.
Tailoring LLMs and proper prompts to meet the needs of social prediction tasks
could be a potential direction to explore. Such adjustments could potentially
unlock ability and applicability of LLMs in social prediction, further
improving the performance of LLMs.

## References

  * Achiam et al. (2023) Josh Achiam et al. 2023.  Gpt-4 technical report.
  * Aher et al. (2023) Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai. 2023.  Using large language models to simulate multiple humans and replicate human subject studies.  In _International Conference on Machine Learning_ , pages 337–371. PMLR.
  * Alghamdi and Alfalqi (2015) Rubayyi Alghamdi and Khalid Alfalqi. 2015.  A survey of topic modeling in text mining.  _Int. J. Adv. Comput. Sci. Appl.(IJACSA)_ , 6(1).
  * (4) ANES.  User’s guide and codebook for the anes 2012 time series study.
  * Anil et al. (2023) Rohan Anil et al. 2023.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_.
  * Argyle et al. (2022) Lisa P. Argyle, E. Busby, Nancy Fulda, Joshua Ronald Gubler, Christopher Michael Rytting, and David Wingate. 2022.  Out of One, Many: Using Language Models to Simulate Human Samples.  _Political Analysis_ , 31:337 – 351.
  * Argyle et al. (2023) Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023.  Out of one, many: Using language models to simulate human samples.  _Political Analysis_ , 31(3):337–351.
  * Athey (2018) Susan Athey. 2018.  The impact of machine learning on economics.  In _The economics of artificial intelligence: An agenda_ , pages 507–547. University of Chicago Press.
  * Bailey (1992) Kenneth D Bailey. 1992.  Globals, mutables, and immutables: a new look at the micro-macro link.  _Quality and Quantity_ , 26(3):259–276.
  * Bailey (1998) Kenneth D Bailey. 1998.  A theory of mutable and immutable characteristics: Their impact on allocation and structural positions.  _Quality and Quantity_ , 32(4):383–398.
  * Bakshy et al. (2011) Eytan Bakshy, Jake M Hofman, Winter A Mason, and Duncan J Watts. 2011.  Everyone’s an influencer: quantifying influence on twitter.  In _Proceedings of the fourth ACM international conference on Web search and data mining_ , pages 65–74.
  * Brensinger and Eyal (2021) Jordan Brensinger and Gil Eyal. 2021.  The sociology of personal identification.  _Sociological Theory_ , 39(4):265–292.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Callegaro et al. (2014) Mario Callegaro, Reginald P Baker, Jelke Bethlehem, Anja S Göritz, Jon A Krosnick, and Paul J Lavrakas. 2014.  _Online panel research: A data quality perspective_.  John Wiley & Sons.
  * Chen et al. (2021a) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021a.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_.
  * Chen et al. (2021b) Yunsong Chen, Xiaogang Wu, Anning Hu, Guangye He, and Guodong Ju. 2021b.  Social prediction: a new research paradigm based on machine learning.  _The Journal of Chinese Sociology_ , 8:1–21.
  * Cheng et al. (2014) Justin Cheng, Lada Adamic, P Alex Dow, Jon Michael Kleinberg, and Jure Leskovec. 2014.  Can cascades be predicted?  In _Proceedings of the 23rd international conference on World wide web_ , pages 925–936.
  * Couper (2017) Mick P Couper. 2017.  New developments in survey data collection.  _Annual review of sociology_ , 43:121–145.
  * Creswell et al. (2022) Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022.  Selection-inference: Exploiting large language models for interpretable logical reasoning.  _arXiv preprint arXiv:2205.09712_.
  * Dalton (2016) Russell J Dalton. 2016.  Party identification and its implications.  _Oxford research encyclopedia of politics_.
  * Diaz et al. (2016) Fernando Diaz, Michael Gamon, Jake M Hofman, Emre Kıcıman, and David Rothschild. 2016.  Online and social media data as an imperfect continuous panel survey.  _PloS one_ , 11(1):e0145406.
  * Dillion et al. (2023) Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023.  Can ai language models replace human participants?  _Trends in Cognitive Sciences_.
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_.
  * Dong et al. (2018) Rida Dong, Oinke Peng, Xintong Li, and Xinyu Guan. 2018.  Cnn-svm with embedded recurrent structure for social emotion prediction.  In _2018 Chinese Automation Congress (CAC)_ , pages 3024–3029. IEEE.
  * Gallup (2009) G Gallup. 2009.  World poll methodology.  Technical report, Technical Report, Washington, DC.
  * Gu et al. (2009) Qiong Gu, Li Zhu, and Zhihua Cai. 2009.  Evaluation measures of the classification performance of imbalanced data sets.  In _Computational Intelligence and Intelligent Systems: 4th International Symposium, ISICA 2009, Huangshi, China, October 23-25, 2009. Proceedings 4_ , pages 461–471. Springer.
  * Halley (2017) Janet E Halley. 2017.  Sexual orientation and the politics of biology: A critique of the argument from immutability.  In _Sexual orientation and rights_ , pages 3–68. Routledge.
  * Hindman (2015) Matthew Hindman. 2015.  Building better models: Prediction, replication, and machine learning in the social sciences.  _The Annals of the American Academy of Political and Social Science_ , 659(1):48–62.
  * Hofman et al. (2017) Jake M Hofman, Amit Sharma, and Duncan J Watts. 2017.  Prediction and explanation in social systems.  _Science_ , 355(6324):486–488.
  * Hsu et al. (2017) Chih-Chung Hsu, Ying-Chin Lee, Ping-En Lu, Shian-Shin Lu, Hsiao-Ting Lai, Chihg-Chu Huang, Chun Wang, Yang-Jiun Lin, and Weng-Tai Su. 2017.  Social media prediction based on residual learning and random forest.  In _Proceedings of the 25th ACM international conference on Multimedia_ , pages 1865–1870.
  * Hussein (2018) Doaa Mohey El-Din Mohamed Hussein. 2018.  A survey on sentiment analysis challenges.  _Journal of King Saud University-Engineering Sciences_ , 30(4):330–338.
  * Jansen et al. (2023) Bernard J Jansen, Soon-gyo Jung, and Joni Salminen. 2023.  Employing large language models in survey research.  _Natural Language Processing Journal_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Kim and Lee (2023) Junsol Kim and Byungkyu Lee. 2023.  Ai-augmented surveys: Leveraging large language models for opinion prediction in nationally representative surveys.  _arXiv preprint arXiv:2305.09620_.
  * Liben-Nowell and Kleinberg (2003) David Liben-Nowell and Jon Kleinberg. 2003.  The link prediction problem for social networks.  In _Proceedings of the twelfth international conference on Information and knowledge management_ , pages 556–559.
  * Liu et al. (2018) Chi Harold Liu, Jie Xu, Jian Tang, and Jon Crowcroft. 2018.  Social-aware sequential modeling of user interests: A deep learning approach.  _IEEE Transactions on Knowledge and Data Engineering_ , 31(11):2200–2212.
  * Mackenzie (2015) Adrian Mackenzie. 2015.  The production of prediction: What does machine learning want?  _European Journal of Cultural Studies_ , 18(4-5):429–445.
  * Messer and Fausser (2019) Uwe Messer and Stefan Fausser. 2019.  Predicting social perception from faces: A deep learning approach.  _arXiv preprint arXiv:1907.00217_.
  * Miller (1991) Warren E Miller. 1991.  Party identification, realignment, and party voting: Back to the basics.  _American Political Science Review_ , 85(2):557–568.
  * Nijkamp et al. (2022) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022.  Codegen: An open large language model for code with multi-turn program synthesis.  _arXiv preprint arXiv:2203.13474_.
  * OpenAI (2022) OpenAI. 2022.  Openai chatgpt.
  * Phelps and Russell (2023) Steve Phelps and Yvan I Russell. 2023.  Investigating emergent goal-like behaviour in large language models using experimental economics.  _arXiv preprint arXiv:2305.07970_.
  * Reier Forradellas et al. (2020) Ricardo Francisco Reier Forradellas, Sergio Luis Náñez Alonso, Javier Jorge-Vazquez, and Marcela Laura Rodriguez. 2020.  Applied machine learning in social sciences: neural networks and crime prediction.  _Social Sciences_ , 10(1):4.
  * Rosenbusch et al. (2023) Hannes Rosenbusch, Claire E Stevenson, and Han LJ van der Maas. 2023.  How accurate are gpt-3’s hypotheses about social science phenomena?  _Digital Society_ , 2(2):26.
  * Sen and Wasow (2016) Maya Sen and Omar Wasow. 2016.  Race as a bundle of sticks: Designs that estimate effects of seemingly immutable characteristics.  _Annual Review of Political Science_ , 19:499–522.
  * Song et al. (2023) Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. 2023.  Llm-planner: Few-shot grounded planning for embodied agents with large language models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2998–3009.
  * Tan et al. (2023) Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023.  Can chatgpt replace traditional kbqa models? an in-depth analysis of the question answering performance of the gpt llm family.  In _International Semantic Web Conference_ , pages 348–367. Springer.
  * Törnberg (2023) Petter Törnberg. 2023.  Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning.  _arXiv preprint arXiv:2304.06588_.
  * Törnberg et al. (2023) Petter Törnberg, Diliara Valeeva, Justus Uitermark, and Christopher Bail. 2023.  Simulating social media using large language models to evaluate alternative news feed algorithms.  _arXiv preprint arXiv:2310.05984_.
  * Tortora et al. (2010) Robert D Tortora, Rajesh Srinivasan, and Neli Esipova. 2010.  The gallup world poll.  _Survey methods in multinational, multiregional, and multicultural contexts_ , pages 535–543.
  * Touvron et al. (2023) Hugo Touvron et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Uyanık and Güler (2013) Gülden Kaya Uyanık and Neşe Güler. 2013.  A study on multiple linear regression analysis.  _Procedia-Social and Behavioral Sciences_ , 106:234–240.
  * Vayansky and Kumar (2020) Ike Vayansky and Sathish AP Kumar. 2020.  A review of topic modeling methods.  _Information Systems_ , 94:101582.
  * Veselovsky et al. (2023) Veniamin Veselovsky, Manoel Horta Ribeiro, Akhil Arora, Martin Josifoski, Ashton Anderson, and Robert West. 2023.  Generating faithful synthetic data with large language models: A case study in computational social science.  _arXiv preprint arXiv:2305.15041_.
  * von der Heyde et al. (2023) Leah von der Heyde, Anna-Carolina Haensch, and Alexander Wenz. 2023.  Assessing bias in llm-generated synthetic datasets: The case of german voter behavior.  Technical report, Center for Open Science.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Wu et al. (2018) Huizhe Wu, Wei Zhang, Weiwei Shen, and Jun Wang. 2018.  Hybrid deep sequential modeling for social text-driven stock prediction.  In _Proceedings of the 27th ACM international conference on information and knowledge management_ , pages 1627–1630.
  * Wu et al. (2023) Patrick Y Wu, Joshua A Tucker, Jonathan Nagler, and Solomon Messing. 2023.  Large language models can be used to estimate the ideologies of politicians in a zero-shot learning setting.  _arXiv preprint arXiv:2303.12057_.
  * Yue et al. (2019) Lin Yue, Weitong Chen, Xue Li, Wanli Zuo, and Minghao Yin. 2019.  A survey of sentiment analysis in social media.  _Knowledge and Information Systems_ , 60:617–663.
  * Zhang et al. (2023) Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. 2023.  Trained transformers learn linear models in-context.  _arXiv preprint arXiv:2306.09927_.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.
  * Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. 2022.  Least-to-most prompting enables complex reasoning in large language models.  _arXiv preprint arXiv:2205.10625_.
  * Zhuang et al. (2023) Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023.  Toolqa: A dataset for llm question answering with external tools.  _arXiv preprint arXiv:2306.13304_.
  * Ziems et al. (2023) Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023.  Can large language models transform computational social science?  _arXiv preprint arXiv:2305.03514_.

##  Appendix A Appendix

###  A.1 Questions for Features

We categorize the selected 16 features into two groups, i.e. high-mutability
and low-mutability features. The details of high-mutability features are shown
in Table 4 and those of low-mutability features are shown in Table 5. The
column "Question Abbrev." indicates the abbreviation of the features, which
are broadly used in this work. The column "Question Identifiers" indicates the
identifier labels of the corresponding questions in the original Gallup
survey.

Topic |  | Question
---
Abbrev.
| Question
---
Identifiers
Question |  Options
Communication Use | IA | WP16056 |  Do you have access to the internet in any way, whether on a mobile phone, a computer, or some other device? |  yes, no
Social Life | SL1 | WP27 |  If you were in trouble, do you have relatives or friends you can count on to help you whenever you need them, or not? |  yes, no
| SL2 | WP10248 |  In the city or area where you live, are you satisfied or dissatisfied with the opportunities to meet people and make friends? |  satisfied, dissatisfied
Economic Confidence | EC1 | WP148 |  Right now, do you think that economic conditions in this country, as a whole, are getting better or getting worse? |  better, worse
| EC2 | M30 |  How would you rate your economic conditions in this country today – as excellent, good, fair, or poor? |  excellent, good, fair, poor
Civic Engagement | CE1 | WP108 |  Have you donated money to a charity in the past month? |  yes, no
| CE2 | WP109 |  Have you volunteered your time to an organization in the past month? |  yes, no
| CE3 | WP110 |  Have you helped a stranger or someone you did not know who needed help? |  yes, no
Approval of Leadership | AL | WP150 |  Do you approve or disapprove of the job performance of the leadership of this country? |  approve, disapprove

Table 4: Questions and Options of High-mutability Features of Gallup Dataset.

Immutable Attribute |  | Question
---
Abbrev.
| Question
---
Identifiers.
Options
Age | age | age |  -
Gender | gender | WP1219 |  1\. Man, 2. Woman
Marital Status | marriage | WP1223 |  1\. Single/Never been married, 2. Married, 3. Separated, 4. Divorced, 5. Widowed, 6. Domestic Partner;
Highest Completed Level of Education | education | WP3117 |  1\. Completed elementary education or less (up to 8 years of basic education); 2. Secondary - 3 years Tertiary/Secondary education and some education beyond secondary education (9-15 years of education); 3. Completed four years of education beyond high school and/or received a 4-year college degree;
Employment Status | employment | EMP_2010 |  1\. Employed full time for an employer, 2. Out of workforce, 3. Employed part time do not want full time, 4. Employed full time for self, 5. Employed part time want full time, 6. Unemployed;
Annual Household Income | income | INCOME_1 |  -
Living of Urbanicity | urbanicity | WP14 |  1\. A suburb of a large city, 2. A small town or village, 3. A large city, 4. A rural area or on a farm;

Table 5: Questions and Options of Low-mutability Features of Gallup Dataset.

####  A.1.1 Feature Convert Methods

In the main experiments, there are features of integer or several classes,
such as income, employment, etc. We convert them into groups (with the number
of groups no larger than four). For income, we calculate the 35% and 65%
percentiles of the annual household income. Based on them, we categorize
income into three classes: lower level, middle level, and higher level. For
features with more than 4 classes, we combine similar classes to make the
number of classes as 2 or 3.


*natbibCitation *BibTex

# Data Augmentation for Deep Graph Learning:
A Survey

Kaize Ding, Zhe Xu, Hanghang Tong, and Huan Liu

###### Abstract

Graph neural networks, a powerful deep learning tool to model graph-structured
data, have demonstrated remarkable performance on numerous graph learning
tasks. To address the data noise and data scarcity issues in deep graph
learning, the research on graph data augmentation has intensified lately.
However, conventional data augmentation methods can hardly handle graph-
structured data which is defined in non-Euclidean space with multi-modality.
In this survey, we formally formulate the problem of graph data augmentation
and further review the representative techniques and their applications in
different deep graph learning problems. Specifically, we first propose a
taxonomy for graph data augmentation techniques and then provide a structured
review by categorizing the related work based on the augmented information
modalities. Moreover, we summarize the applications of graph data augmentation
in two representative problems in data-centric deep graph learning: (1)
optimal graph learning which focuses on enhancing the data usability of the
input graph via graph data augmentation; and (2) low-resource graph learning
which targets on enlarging the labeled training data scale through graph data
augmentation. For each problem, we also provide a hierarchical problem
taxonomy and review the existing literature related to graph data
augmentation. Finally, we point out promising research directions and the
challenges in future research.

###### Index Terms:

Graph Neural Networks, Deep Graph Learning, Graph Data Augmentation.

## Acknowledgments

The authors would like to thank…

|  Michael Shell Biography text here.
---|---
John Doe Biography text here.
---
Jane Doe Biography text here.
---