#  Community-Cross-Instruct: Unsupervised Instruction Generation for Aligning
Large Language Models to Online Communities

Zihao He, Rebecca Dorn, Siyi Guo, Minh Duc Chu, Kristina Lerman
USC Information Sciences Institute
{zihaoh, rdorn, siyiguo<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Social scientists use surveys to probe the opinions and beliefs of
populations, but these methods are slow, costly, and prone to biases. Recent
advances in large language models (LLMs) enable creating computational
representations or “digital twins” of populations that generate human-like
responses mimicking the population’s language, styles, and attitudes. We
introduce Community-Cross-Instruct, an unsupervised framework for aligning
LLMs to online communities to elicit their beliefs. Given a corpus of a
community’s online discussions, Community-Cross-Instruct automatically
generates instruction-output pairs by an advanced LLM to (1) finetune an
foundational LLM to faithfully represent that community, and (2) evaluate the
alignment of the finetuned model to the community. We demonstrate the method’s
utility in accurately representing political and fitness communities on
Reddit. Unlike prior methods requiring human-authored instructions, Community-
Cross-Instruct generates instructions in a fully unsupervised manner,
enhancing scalability and generalization across domains. This work enables
cost-effective and automated surveying of diverse online communities.

Community-Cross-Instruct: Unsupervised Instruction Generation for Aligning
Large Language Models to Online Communities

Zihao He, Rebecca Dorn, Siyi Guo, Minh Duc Chu, Kristina Lerman USC
Information Sciences Institute {zihaoh, rdorn, siyiguo<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Social scientists use surveys and focus groups to learn the opinions, needs
and concerns of diverse populations. However, designing surveys and recruiting
participants is a slow and costly process, limiting the utility of these
instruments for probing public opinion. Surveys are prone to biases, such as
the social desirability bias Gordon (1987), where respondents may alter their
responses to sensitive questions to appear more socially acceptable Bergen and
Labonté (2020), non-response bias Hill et al. (1997), where participants fail
to answer questions, and self-selection bias due to the choices participants
make to participate in the survey Heckman (1990). In addition, social stigmas
may taint responses Goel and Salganik (2010), especially for hard-to-reach and
marginalized groups.

Figure 1: Illustration of Community-Cross-Instruct to align an LLM to a
community. (1) Open-ended instructions and multi-choice survey questions are
generated by an advanced LLM from the community data. (2) A foundational LLM
is aligned to community through instruction-tuning on the open-ended
instructions. (3) The alignment of the finetuned LLM to the community is
measured on the survey questions.

Recent breakthroughs in generative AI and especially large language models
(LLMs) enable new capabilities for creating computational representations of
human populations — their digital twins El Saddik (2018) — by ingesting vast
textual data they create, for example, in online discussion forums. These LLM-
based models generate human-like responses that mimic the language,
communication styles, and attitudes of populations they are aligned to,
allowing us to probe their worldviews, biases, and sentiments in a cost-
effective and automated manner. Previous works have leveraged such LLM-based
representations to mine opinions and learn political attitudes of online
communities Jiang et al. (2020); He et al. (2024b). However, they finetune
GPT-2 Radford et al. (2019) directly on textual data from communities, where
the finetuned model is not able to answer questions that may be in different
format, e.g., multiple choice, as it is not an instruction-tuned model.

To address these challenges, we introduce Community-Cross-Instruct: a fully
unsupervised framework for aligning LLMs to online communities through
instruction-tuning. The finetuned LLMs serve as digital twins of communities,
which can be automatically surveyed to elicit their views (Figure 1).

Specifically, given a corpus of comments and submissions in different forums
on Reddit, Community-Cross-Instruct uses an advanced LLM (Claude-3) to
automatically curate (1) CommInst: a set of open-ended instructions (Fig.
2(a)) for community-specific instruction tuning, and (2) CommSurvey, a set of
multi-choice survey questions (Fig. 2(b)). Each instruction in CommInst and
each survey question in CommSurvey are paired with answers from different
communities. We finetune foundational LLMs (GPT-3.5 or Llama-3) on CommInst,
in order to align them to different communities, and evaluate the finetuned
LLMs on CommSurvey to measure alignment. See Figure 3 for a high-level
overview of the framework. Existing methods for self-supervised instruction
generation rely on a seed set of instructions curated by domain experts Wang
et al. (2022); Chen et al. (2024b), which is not generalizable. We overcome
this bottleneck by incorporating the readily available community data into the
instruction generation pipeline. Our Community-Cross-Instruct framework
distills knowledge from communities using an advanced LLM to generate
instructions for aligning foundational LLMs.

Our key contributions can be summarized as

  * •

We introduce Community-Cross-Instruct, a novel unsupervised framework for
aligning foundational LLMs to online communities, by finetuning them on the
automatically curated set of open-ended community-specific instruction-answer
pairs (CommInst). The models’ alignment to communities is measured using
another set of automatically generated multiple choice questions and answers
(CommSurvey).

  * •

Using data from Reddit forums, we show that our method improves the fidelity
of community representation (alignment) in two domains— _politics_ and
_fitness_ —yielding significant alignment improvement over standard persona
adaptation methods.

Our work highlights the potential of generative AI to help researchers gain
insights from online communities. By leveraging LLMs to create digital twins
of these communities, researchers can more accurately and efficiently
understand the nuances of public opinion, attitudes, and behaviors. Our
framework not only enhances the fidelity of community representation but also
paves the way for more effective approaches to studying social phenomena in
the digital age.

#### Clarification on contributions.

We aim to align LLMs to online communities through instruction-tuning, so that
the aligned LLMs can be flexibly surveyed to probe the communities’ views on
different topics. To prepare data for instruction tuning, we propose
Community-Cross-Instruct, and the automatic creation of the open-ended
instruction-answer pairs (CommInst) is the core contribution. However, in
order to demonstrate that the finetuned LLMs are indeed aligned to communities
and can be treated as their digital twins, we propose to automatically
generate survey questions and answers (CommSurvey) to efficiently evaluate LLM
alignment. This is a secondary contribution of this paper, and we do not claim
that CommSurvey is the best way to measure alignment. Therefore, our main goal
is to propose a method to prepare LLMs as a proxy for surveying different
online communities, rather than automatically creating such surveys.

#### On the use of the term _alignment_.

Inspired by Santurkar et al. (2023), throughout this paper, we use _alignment_
to refer only to the alignment of views and opinions of LLMs and humans.

##  2 Problem Definition

A topical domain (e.g., _politics_ or _fitness_) includes $n$ communities
{$C_{1}$, $C_{2}$, …., $C_{n}$} each with different views and beliefs. Members
of each community $C_{i}$ collectively author text corpus $D_{i}$ (e.g.,
discussions on Reddit forums) expressing views and exhibiting behaviors. Our
goal is to align an LLM $f$ to each community $C_{i}$ using its texts $D_{i}$,
such that the aligned LLM $f^{\prime}_{i}$ learns the complex mindset of the
community and responds to inputs in the community’s voice. By administering
surveys to the aligned LLMs
$\\{f^{\prime}_{1},f^{\prime}_{2},...,f^{\prime}_{n}\\}$, we obtain responses
from different communities, thereby capturing their ideological differences.

To align an LLM $f$ to a community $C$, we finetune it on a set of
demonstrations (instruction-answer pairs) Wang et al. (2023); Ouyang et al.
(2022); Chen et al. (2024b) $I=\\{(X_{j},Y_{j})\\}_{j=1}^{m}$, where the
instructions are open-ended questions probing the community’s views on
different topics, and the corresponding answers are aligned with each
community’s ideology. Figure 2(a) shows an example demonstration in the
politics domain. We propose Community-Cross-Instruct (Fig. 3), a framework to
automatically generate community-specific demonstrations $I$ with an advanced
LLM $\hat{f}$ (Claude-3) based on the community’s text corpus $D$ and use
these demonstrations to instruction-tune a foundational LLM $f$ (GPT-3.5 or
Llama-3) through a process we call “Cross-Instruct”.

Figure 2: Example of (a) an instruction from CommInst-Pol and (b) a survey
question from CommSurvey-Pol. The instruction/survey question is paired with
answers from different communities. Figure 3: Overview of Community-Cross-
Instruct, with an illustrative example of the _politics_ domain. (1) Data
collection for each community within the desired domain. (2) BERTopic clusters
the data and identifies prominent topics. (3) For each topic, the advanced LLM
is prompted with (i) on-topic examples from each community and (ii)
instruction-answer task definition and explanation. (4) Creation of (a) open-
ended instruction-answer pairs and (b) multi-choice survey questions and
answers, for fine-tuning and evaluating a sub-population representation
foundational LLM respectively.

##  3 CommInst: A Collection of Community-Specific Instructions

To align LLMs to different communities within each domain, we curate a dataset
CommInst for community-specific instruction tuning. CommInst-Pol and CommInst-
Fit are the two subsets for the _politics_ and _fitness_ domains with 6,687
and 4,395 unique instructions respectively. Each instruction is an opinion-
eliciting question about a topic relevant to the domain (Figure 2(a)), which
appears in multiple demonstrations. Each demonstration’s instruction is paired
with an open-ended answer from a different community.

###  3.1 Online Community Corpora Collection

Reddit is a vibrant social media platform hosting discussion forums
(subreddits) on a wide range of topics Hofmann et al. (2022); Chen et al.
(2024a). In this paper we focus on two domains: _politics_ and _fitness_ ,
which contain distinctive subreddits, or communities, with complex social
dynamics. Political subreddits are valuable expressions of diverse public
opinions and viewpoints. In the age of rising polarization, LLMs can help
researchers track the complex evolving ideological landscape and effectively
elicit public opinions. Meanwhile, the fitness subreddits feature a wealth of
sensitive, health-related conversations, and the latest diet and fitness fads,
often discussed using inscrutable insider jargon. We can track the direction
of these discussions, identifying emerging trends, risks, and potential health
misinformation. This insight allows public health officials to promptly
address harmful health advice and intervene where necessary. Additionally, it
provides an early warning system to detect the spread of dangerous health
practices, ensuring that corrective measures can be implemented swiftly to
protect community health.

We identify a set of representative subreddits for each domain based on
personal knowledge and by querying ChatGPT. We manually aggregate and filter
results, obtaining the following political online communities: _r/Anarchism_ ,
_r/Anarcho_Capitalism_ , _r/Conservative_ , _r/Libertarian_ , _r/Republican_ ,
and _r/democrats_ ; For fitness, we investigate: _r/Fitness_ , _r/loseit_ ,
_r/keto_ , _r/xxfitness_ , and _r/GettingShredded_. More details of these
communities are presented in Appendix A.1.

We collect comments and submissions from January 2019 to December 2022 and
remove those that were deleted by the author or the moderator. The statistics
of the collected comments and submissions are shown in Appendix A.1. To
balance the number of comments and submissions across different communities,
for each community, we keep a maximum of 140K comments and 30K submissions.
Therefore, there are at most 170K texts, which makes up the text corpus $D$,
including comments and submissions, for each community.

###  3.2 Topic Modeling

For a domain, denote the combined corpus from all $n$ communities by
$\mathbb{D}=D_{1}\cup D_{2}\cup....\cup D_{n}$. We use BERTopic Grootendorst
(2022) on $\mathbb{D}$ to identify topics $g$. More details about BERTopic can
be found in Appendix B.1. After topic modeling, each text $d$ is assigned a
topic $g(d)$. The number of topics detected in _politics_ and _fitness_ are
425 and 628 respectively. The top 10 most frequent topics in politics and
fitness are shown in Appendix B.1 respectively. _Politics_ topics include
abortion, climate, guns, and racism. _Fitness_ topics include body measures,
keto diet, and weight lifting.

For each community $C_{i}$, we split the text corpus $D_{i}$ into smaller
chunks $D_{i}=\\{\hat{D}^{i}_{1},\hat{D}^{i}_{2},....\\}$, where each chunk
$\hat{D}$ consists of texts belonging to a single topic $g(\hat{D})$.

###  3.3 Open-ended Instruction-Answer Pair Generation

For a specific domain, starting with an empty demonstration pool, we randomly
sample without replacement a text chunk $D_{i}$ for each topic $t_{k}\in T$
from each community. We include the sampled text chunks on topic $t_{k}$ from
the $n$ communities into a prompt and then use the prompt to query the
advanced LLM to generate open-ended instructions and answers on the topic. The
prompting template is shown in Appendix B.2. We query _claude-3-opus-20240229_
using the Anthropic API. In the prompt, we first specify the topic by the
topic keywords, and then the comments in the sampled chunks from different
communities. Next we ask the model to generate instructions that can be
answered based on the texts, and that the instructions should elicit different
responses from the $n$ communities. Each generated instruction is paired with
different answers from different communities (Figure 2(a)). It is then added
to the pool as $n$ demonstrations. Note that some topics may be discussed more
in certain communities than others, so some instructions may be paired with
answers from less than $n$ communities. We iteratively repeat this process
until there are no text chunks left for topic $t_{k}$ from any community,
until we shift to the next topic $t_{k+1}$. After the iterative generation
process, the instruction-answer pool will contain demonstrations covering
various topics, which will use to finetune foundational LLMs. Please refer to
Appendix A.2 for more details about CommInst.

For fair evaluating the finetuned LLMs, we split CommInst-Pol in two ways:

  * •

Random split. CommInst-Pol is randomly split into training and eval sets:
CommInst-Pol-train and CommInst-Pol-eval, in a ratio of 85:15.

  * •

Topical split. We randomly select 10 topics in _politics_ and collect
demonstrations related to the topics into CommInst-Pol-eval-topic. The
remaining demonstrations are collected as CommInst-Pol-train-topic. In this
way, CommInst-Pol-train-topic and CommInst-Pol-eval-topic has no topic
overlap.

We repeat the same process for CommInst-Fit.

##  4 CommSurvey: A Collection of Community-Specific Survey Questions

To measure the alignment of the finetuned foundational LLM to the community,
we administer a survey of multi-choice questions to the model and evaluate the
agreement between the model’s responses and the community members’ responses.
However, manually designing surveys and collecting human responses from online
communities is a non-trivial process, which is costly and time-consuming.
Instead, we curate another dataset called CommSurvey for evaluating the
alignment of the finetuned LLM’s views to the community using survey
questions. CommSurvey consists of automatically generated multi-choice survey
questions and answers, where each question is paired with answers from
different communities (Figure 2(b)). CommSurvey-Pol and CommSurvey-Fit are the
two subsets for _politics_ and _fitness_ respectively, consisting of 4,457 and
2,930 multi-choice questions respectively, covering a wide range of topics.

politics | r/Anarchism |  | r/Anarcho_
---
Capitalism
r/Conservative | r/Libertarian | r/Republican | r/democrats
0.75 | 0.60 | 0.75 | 0.45 | 0.75 | 0.65
fitness | r/Fitness | r/loseit | r/keto | r/xxfitness | r/GettingShredded |
0.75 | 0.70 | 0.60 | 0.50 | 0.65 |
Table 1: Agreement between human annotators’ survey responses and the advanced
LLM (Claude-3) generated survey answers in different domains and subreddits,
measured by accuracy.

###  4.1 Multi-choice Survey Generation.

We initialize an empty multi-choice survey pool. Then, following the open-
ended instruction-answer pairs generation in $\S$3.3, we iteratively generate
multi-choice questions and answers (Figure 2(b)) using the same advanced LLM
(Claude-3) and add them to the pool. We also explored other advanced LLMs,
such as GPT-4 or Gemini, but none of them was able to generate high-quality
questions or answers as Claude-3. The prompting template is shown in Figure 8.
We assume that the answers generated by the advanced LLM are the “semi-ground
truths”, and that they faithfully represent the views of the corresponding
communities.

Note that there is a correspondence between instructions in CommInst and
survey questions CommSurvey, as they are generated using nearly identical
prompts. Following the correspondence, we split CommSurvey-Pol and CommSurvey-
Fit following the same two ways for CommInst. Due to the correspondence, there
is no information leakage between CommInst-Pol-train and CommSurvey-Pol-eval,
or between CommInst-Pol-train-topic and CommSurvey-Pol-eval-topic.

###  4.2 Validating Generated Survey Answers

We verify the faithfulness of the “semi-ground truths” for the communities via
human annotation. For each domain, we randomly sample 20 questions from the
survey (see Fig. 9 and Fig. 10 in the Appendix) and ask two expert annotators
to answer them from the perspective of the subreddit. We also tried directly
posting the surveys to the subreddits and having community members answer
them. However, our posts were immediately removed by moderators and our
accounts were banned from certain subreddits. This further indicates the
difficulty of directly surveying online communities.

For _politics_ , each annotator fills out the survey for three subreddits; and
for _fitness_ , two annotators answer the questions for three subreddits and
two subreddits respectively. For each subreddit, we compare the LLM-generated
“semi-ground truths” to human responses, and present the accuracy in Table 1.
In both domains, “semi-ground truths” achieve strong agreement with human
annotations for most subreddits, which gives confidence that the advanced LLM-
generated survey answers can be used as “semi-ground truth” to evaluate
finetuned foundational LLMs. However, for _r/Libertarian_ in _politics_ , the
agreement is less satisfactory. After inspecting the online posts from
_r/Libertarian_ , we find that libertarian principles often emphasize
individual freedom and skepticism of government intervention, which can lead
to highly context-dependent and situation-specific responses. This ambiguity
can result in different interpretations of the same question for the human
annotator and the advanced LLM. Considering the low agreement, we will not
report the results for _r/Libertarian_ in $\S\ref{sec:exp}$.

##  5 Experiments

We demonstrate the effectiveness of the pipeline on the Reddit _politics_ and
_fitness_ domains. We align LLMs to Reddit communities by finetuning them on
relevant demonstrations from CommInst. After finetuning, we administer surveys
from CommSurvey and evaluate the finetuned LLMs by comparing their responses
to the “semi-ground truths”, in order to measure alignment.

###  5.1 Experimental Setup

We experiment with two foundational LLMs – GPT-3.5 and Llama-3.

#### Finetuning.

We finetune the foundational LLMs on CommInst-Pol-train and CommInst-Fit-
train. We use the OpenAI finetuning API to finetune _gpt-3.5-turbo-0125_ for 1
epoch, as we find that the loss converges within 1 epoch. The batch size is
automatically determined by the API, and the finetuning takes ~40mins. For
Llama-3, the implementation is based on LlamaFactory Zheng et al. (2024), and
we finetune _Llama-3-8B-Instruct_ with batch size 16 for 2 epochs on 4 H100
GPUs. The finetuning takes ~10mins.

#### Measuring Alignment.

We administer survey questions in CommSurvey-Pol-veal and CommSurvey-Fit-veal
to the finetuned foundational LLMs. In the prompt, we provide the question and
the four options verbatim, and then add the instruction “Select only one
answer by stating either A, B, C, or D. Do not provide any additional
explanation or rationale for your choice.”, for the sake of easier matching of
the model’s response to the option. For GPT-3.5, the temperature is set to 0.9
and the model provides 5 generations for each question. For Llama-3,
temperature is set to 0.9, $top\\_p$ is set to 0.9, and the model provides 10
generations for each question. The majority vote from the different
generations for each question is taken. We compare the model responses to the
“semi-ground truths” and report the accuracy as a measure of alignment.

###  5.2 Baselines

#### Vanilla LLMs.

We administer survey questions to the vanilla unfinetuned GPT-3.5 and Llama-3
and evaluate their answers.

#### Vanilla LLMs+Steering.

When prompting the vanilla language models to answer survey questions, we
steer them to mimic the community, by specifying in the prompt that “Select
only one answer that best aligns with the opinions of members from subreddit
r/[subreddit]”. Steering the vanilla LLMs can nudge them to respond for the
community. However, it is important to note that steering only applies to
predefined communities that are developed via a predefined tag, such as
subreddits, where we can easily specify community names in the prompt to steer
the LLMs. Although in this paper we focus on communities associated with
online forums, Community-Cross-Instruct generalizes to other online
communities, including organically formed communities in the retweet network
Chu et al. (2024) or the news co-sharing network Mosleh et al. (2021); He et
al. (2024b), as long as their texts are readily available. It is non-trivial
to summarize the organically-formed communities concisely in language, but we
can still align LLMs to them via Community-Cross-Instruct.

###  5.3 Main Results

We compare survey answers generated by the finetuned LLMs to the “semi-ground
truth” and report the accuracy in Figure 4 as a measure of alignment with the
corresponding community. In most cases, steering LLMs towards specific
communities creates better alignment by relying on the models’ pre-existing
knowledge about the communities. However, such pre-knowledge may be
incomplete, inaccurate, or outdated. Therefore, by finetuning LLMs on
instructions distilled from the text corpora of communities, Community-Cross-
Instruct outperforms steering and achieves the best results, demonstrating its
effectiveness in aligning LLMs to different communities.

(a) politics

(b) fitness.

Figure 4: Alignment of finetuned LLMs to the corresponding communities
measured by accuracy, by comparing the LLMs’ survey responses to the “semi-
ground truths”.

The base models show worse alignment with _r/Conservative_ and _r/Republican_
than other Reddit communities. This observation conforms to findings by
previous works Santurkar et al. (2023); He et al. (2024a); Chen et al. (2024b)
that LLMs tend to display a liberal bias and poor alignment with
conservatives. After explicitly steering them towards specific communities,
their alignment with the two subreddits significantly increases, but it is
still lower than that with other communities. Finetuning the LLMs with
Community-Cross-Instruct further improves the alignment, although it is still
lower than for other communities. This observation highlights the risks of
deploying LLMs in political scenarios due to the entrenched political biases.

For the _r/Anarchism_ subreddit, GPT-3.5-CrossInst achieves similar alignment
as the steering baseline. We suspect that this is because views about
anarchism are captured by the pre-existing knowledge of GPT-3.5. Therefore,
infusing such knowledge again by Community-Cross-Instruct barely leads to any
gain.

For the _r/Fitness_ and _r/xxfitness_ communities, steering both LLMs worsens
the alignment compared to the vanilla model. _r/Fitness_ is a general fitness
subreddit and _r/xxfitness_ is a fitness community for women, both of which
may have a broader and more diverse range of opinions compared to other
subreddits. Steering the models to align with a singular “community opinion”
might oversimplify the nuanced perspectives within these fitness communities,
leading to decreased alignment accuracy. However, _Community-Cross-Instruct_
effectively distills knowledge from the community corpora during finetuning.
On _r/GettingShredded_ , Community-Cross-Instruct is outperformed by the
steering baseline for GPT-3.5, but not Llama-3. _r/GettingShredded_ has the
least amount of comments and posts, producing much fewer demonstrations for
finetuning (Appendix 3). The insufficient data might be less effective for
finetuning a larger LLM like GPT-3.5111We assume that GPT-3.5 has more than 8B
parameters., leading to inferior performance.

###  5.4 Out-of-Topic Generalizability

Our final goal is to create LLMs aligned to different communities, which can
be used to answer any survey question from the perspective of the community.
In the real world, the surveys may contain questions covering topics that do
not appear in the finetuning data CommInst. In this study, we finetune Llama-3
on CommInst-Pol-train-topic, and evaluate it on CommSurvey-Pol-eval-topic, to
make sure that there is no overlap in topic coverage. The results in Fig. 5
show that Community-Cross-Inst consistently outperforms both baselines,
demonstrating its superiority in aligning with views of different communities,
and generalize the learned views to unseen topics.

Figure 5: Cross-topic alignment of finetuned LLMs to the corresponding
communities. The LLMs are finetuned on demonstrations and evaluated on survey
questions on different topics.

##  6 Related Works

Self-Improved Instruction Tuning. A survey of instruction-tuning approaches
Zhang et al. (2023) outlines several methods for models to autonomously self-
improve their instruction set. These include generating instructions for pre-
existing texts Li et al. (2023), eliciting interaction between different model
iterations Chen et al. (2024c), and bootstrapping from an existing instruction
set to generate new ones Wang et al. (2022). Additionally, a human-in-the-loop
framework builds upon self-generalization by iterating between human and
machine-generated instructions Guo et al. (2024). Despite these advances,
these strategies require manual input. In this work, we build on self-
instruct, a method to enhance instruction tuning by eliciting synthetic model
instruction generations. We alter the original framework by removing the
required set of manually written seed instructions Wang et al. (2022). This
work significantly contributes to the field by offering a scalable and fully
autonomous solution to instruction tuning, paving the way for more adaptive
and intelligent models.

Community Alignment. Aligning with the dialects and behaviors of online
communities is crucial for accurately modeling subpopulations Dorn et al.
(2023, 2024). Subpopulation representative models (SRMs) Simmons and Hare
(2023) can be used to emulate some characteristics of a particular
subpopulation, particularly as LLMs can provide fine-grained, demographically-
correlated outputs Argyle et al. (2023a). Using self-instruct with questions
from the American Trans Panel as the seed instruction set, LLMs can be
ideologically manipulated to reflect particular political opinions Chen et al.
(2024b). To learn about partisan communities, Jiang et al. (2022) repeatedly
prompt GPT 2 with tweets authored by prominent community members to generate
synthetic responses to election survey questions. Argyle et al. (2023b) find
that exposing GPT 3 to thousands of socio-demographic backstories leads the
model to obtain a complex understandings of sociological concepts. ‘Media diet
models’ train BERT using MLM on news articles, evaluating the alignment of
model predictions with a survey of community members who follow the media Chu
et al. (2023). Feldman et al. (2022) expose GPT 3 to Yelp reviews to expand
insights past sentiment information. Prompting language models with (survey
question, individual ID, survey year) triplets from the Americans in the
General Social Surveys has helped social scientists impute missing survey data
and predict opinions Kim and Lee (2023). These techniques rely on manually
created datasets at least as a seed dataset to align the models with the
desired communities, limiting SRBMs to high-resource environments.

Surveys. Social scientists use surveys to systematically collect data from
populations to characterize their beliefs, attitudes, opinions, and behaviors.
While designing the survey, researchers consider the order, wording, and
format of questions (open-ended vs multiple-choice) to increase clarity and
ensure the validity and reliability of the survey data. Survey design also
aims to minimize biases, such as the non-response bias Hill et al. (1997),
fatigue due to overly long surveys Choi and Pak (2005), social desirability
bias Gordon (1987) when asking survey participants questions about sensitive
topics (due to the desire to conform to perceived social expectations), or
those involving social stigmas Goel and Salganik (2010). These issues, as well
as the variety of existing biases, call for new approaches to survey design.

##  7 Conclusion

In this paper, we introduced and explored Community-Cross-Instruct, a fully
automated framework to represent online communities and elicit their views. By
automating the process of surveying diverse online communities, Community-
Cross-Instruct offers a cost-effective and efficient alternative to
traditional survey methods used by social scientists. Through finetuning LLMs
based on community-specific instructions and answers, Community-Cross-Instruct
enables the generation of accurate responses that align with the beliefs and
perspectives of different online communities. This innovative framework has
demonstrated its effectiveness in representing various communities, including
political and fitness forums on Reddit. Community-Cross-Instruct opens up new
possibilities for researchers to gain insights into the diverse views and
opinions present in online communities, paving the way for deeper
understanding and analysis of digital societies. Moving forward, we will apply
the framework to model organic online communities, such as those in the
retweet network.

## Limitations

#### Identical LLM for Generating both Datasets.

We use Claude-3 for generating both CommInst for finetuning LLMs and
CommSurvey for evaluating LLM alignment, as we couldn’t find a different
advanced LLM that was able to generate high-quality data as Claude-3. This
reliance on a single LLM introduces a limitation, as it may lead to a bias
where the evaluation dataset (CommSurvey) is inherently aligned with the model
used for instruction generation (CommInst), potentially overestimating the
alignment accuracy of the finetuned models. To address this limitation, we
plan to incorporate a diverse set of advanced LLMs as they become available in
future iterations of our framework. Additionally, we will include human-in-
the-loop validation to ensure the generated datasets maintain high quality and
representativeness, mitigating any biases introduced by the single LLM
dependency. Furthermore, we will explore cross-validation techniques and
third-party evaluations to benchmark the performance of our finetuned models,
ensuring robustness and generalizability of our results beyond the influence
of a single LLM.

#### Group Approximation.

To approximate group-level behavior, community members represented in the
minority are inherently excluded. Further, measuring group alignment using a
single-answer multiple choice questionnaire does not account for bimodal
distribution. We hope to build on this work by experimenting with survey
designs that account for more diversity in communities.

#### Hallucination Potential.

Adapting models to communities poses a risk of language models hallucinating
or providing misinformation in their community representation. We hope to
build upon this work with in-depth experiments on model hallucination and
misrepresentation in the subgroup representation task.

#### Prompt Perturbations.

LLM responses are sensitive to slight changes in prompts Salinas and
Morstatter (2024). In this work, we work primarily with one prompt for
instruction generation. We would be interested to see how model’s generated
instructions shift with different prompting schemas.

## Ethics Statement

Users might not be informed of how their data (reflected in their posts) are
used to produce digital twins that mirror their voices and the purpose of the
survey constructed based on their data. Furthermore, automatically surveying
communities can reveal unconsented insights of certain individuals or groups
of people online. To address these ethical concerns, we will implement several
measures. Firstly, we will ensure that all data used for creating digital
twins is anonymized, stripping any personally identifiable information to
protect user privacy. Secondly, we will seek to aggregate data in a manner
that focuses on community-level insights rather than individual-level
analysis, thereby reducing the risk of unconsented personal exposure. Thirdly,
we will obtain explicit consent from users where possible, clearly
communicating the purpose and scope of our research. Lastly, we will adhere to
ethical guidelines and institutional review board (IRB) requirements to ensure
that our methods respect the privacy and rights of the individuals and
communities involved.

## References

  * Argyle et al. (2023a) Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023a.  Out of one, many: Using language models to simulate human samples.  _Political Analysis_ , 31(3):337–351.
  * Argyle et al. (2023b) Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023b.  Out of one, many: Using language models to simulate human samples.  _Political Analysis_ , 31(3):337–351.
  * Bergen and Labonté (2020) Nicole Bergen and Ronald Labonté. 2020.  “everything is perfect, and we have no problems”: detecting and limiting social desirability bias in qualitative research.  _Qualitative health research_ , 30(5):783–792.
  * Chen et al. (2024a) Kai Chen, Zihao He, Keith Burghardt, Jingxin Zhang, and Kristina Lerman. 2024a.  Isamasred: A public dataset tracking reddit discussions on israel-hamas conflict.  _arXiv preprint arXiv:2401.08202_.
  * Chen et al. (2024b) Kai Chen, Zihao He, Jun Yan, Taiwei Shi, and Kristina Lerman. 2024b.  How susceptible are large language models to ideological manipulation?  _arXiv preprint arXiv:2402.11725_.
  * Chen et al. (2024c) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024c.  Self-play fine-tuning converts weak language models to strong language models.  _arXiv preprint arXiv:2401.01335_.
  * Choi and Pak (2005) Bernard CK Choi and Anita WP Pak. 2005.  Peer reviewed: a catalog of biases in questionnaires.  _Preventing chronic disease_ , 2(1).
  * Chu et al. (2023) Eric Chu, Jacob Andreas, Stephen Ansolabehere, and Deb Roy. 2023.  Language models trained on media diets can predict public opinion.  _arXiv preprint arXiv:2303.16779_.
  * Chu et al. (2024) Minh Duc Chu, Aryan Karnati, Zihao He, and Kristina Lerman. 2024.  Characterizing online eating disorder communities with large language models.  _arXiv preprint arXiv:2401.09647_.
  * Dorn et al. (2024) Rebecca Dorn, Lee Kezar, Fred Morstatter, and Kristina Lerman. 2024.  Harmful speech detection by language models exhibits gender-queer dialect bias.  _arXiv preprint arXiv:2406.00020_.
  * Dorn et al. (2023) Rebecca Dorn, Negar Mokhberian, Julie Jiang, Jeremy Abramson, Fred Morstatter, and Kristina Lerman. 2023.  Non-binary gender expression in online interactions.  _arXiv preprint arXiv:2303.04837_.
  * El Saddik (2018) Abdulmotaleb El Saddik. 2018.  Digital twins: The convergence of multimedia technologies.  _IEEE multimedia_ , 25(2):87–92.
  * Feldman et al. (2022) Philip Feldman, Aaron Dant, James R Foulds, and Shemei Pan. 2022.  Polling latent opinions: A method for computational sociolinguistics using transformer language models.  _arXiv preprint arXiv:2204.07483_.
  * Goel and Salganik (2010) Sharad Goel and Matthew J Salganik. 2010.  Assessing respondent-driven sampling.  _Proceedings of the National Academy of Sciences_ , 107(15):6743–6747.
  * Gordon (1987) Randall A Gordon. 1987.  Social desirability bias: A demonstration and technique for its reduction.  _Teaching of Psychology_ , 14(1):40–42.
  * Grootendorst (2022) Maarten Grootendorst. 2022.  Bertopic: Neural topic modeling with a class-based tf-idf procedure.  _arXiv preprint arXiv:2203.05794_.
  * Guo et al. (2024) Hongyi Guo, Yuanshun Yao, Wei Shen, Jiaheng Wei, Xiaoying Zhang, Zhaoran Wang, and Yang Liu. 2024.  Human-instruction-free llm self-alignment with limited samples.  _arXiv preprint arXiv:2401.06785_.
  * He et al. (2024a) Zihao He, Siyi Guo, Ashwin Rao, and Kristina Lerman. 2024a.  Whose emotions and moral sentiments do language models reflect?  _arXiv preprint arXiv:2402.11114_.
  * He et al. (2024b) Zihao He, Ashwin Rao, Siyi Guo, Negar Mokhberian, and Kristina Lerman. 2024b.  Reading between the tweets: Deciphering ideological stances of interconnected mixed-ideology communities.  _arXiv preprint arXiv:2402.01091_.
  * Heckman (1990) James J Heckman. 1990.  Selection bias and self-selection.  In _Econometrics_ , pages 201–224. Springer.
  * Hill et al. (1997) Anthony Hill, Julian Roberts, Paul Ewings, and David Gunnell. 1997.  Non-response bias in a lifestyle survey.  _Journal of Public Health_ , 19(2):203–207.
  * Hofmann et al. (2022) Valentin Hofmann, Hinrich Schütze, and Janet B Pierrehumbert. 2022.  The reddit politosphere: A large-scale text and network resource of online political discourse.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 16, pages 1259–1267.
  * Jiang et al. (2022) Hang Jiang, Doug Beeferman, Brandon Roy, and Deb Roy. 2022.  Communitylm: Probing partisan worldviews from language models.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 6818–6826.
  * Jiang et al. (2020) Julie Jiang, Emily Chen, Shen Yan, Kristina Lerman, and Emilio Ferrara. 2020.  Political polarization drives online conversations about covid-19 in the united states.  _Human Behavior and Emerging Technologies_ , 2(3):200–211.
  * Kim and Lee (2023) Junsol Kim and Byungkyu Lee. 2023.  Ai-augmented surveys: Leveraging large language models for opinion prediction in nationally representative surveys.  _arXiv preprint arXiv:2305.09620_.
  * Li et al. (2023) Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. 2023.  Self-alignment with instruction backtranslation.  _arXiv preprint arXiv:2308.06259_.
  * Mosleh et al. (2021) Mohsen Mosleh, Gordon Pennycook, Antonio A Arechar, and David G Rand. 2021.  Cognitive reflection correlates with behavior on twitter.  _Nature communications_ , 12(1):921.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback, 2022.  _URL https://arxiv. org/abs/2203.02155_ , 13.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Salinas and Morstatter (2024) Abel Salinas and Fred Morstatter. 2024.  The butterfly effect of altering prompts: How small changes and jailbreaks affect large language model performance.  _arXiv preprint arXiv:2401.03729_.
  * Santurkar et al. (2023) Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023.  Whose opinions do language models reflect?  In _International Conference on Machine Learning_ , pages 29971–30004. PMLR.
  * Simmons and Hare (2023) Gabriel Simmons and Christopher Hare. 2023.  Large language models as subpopulation representative models: A review.  _arXiv preprint arXiv:2310.17888_.
  * Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020.  Mpnet: Masked and permuted pre-training for language understanding.  _Advances in neural information processing systems_ , 33:16857–16867.
  * Wang et al. (2022) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022.  Self-instruct: Aligning language models with self-generated instructions.  _arXiv preprint arXiv:2212.10560_.
  * Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  Self-instruct: Aligning language models with self-generated instructions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13484–13508.
  * Zhang et al. (2023) Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023.  Instruction tuning for large language models: A survey.  _arXiv preprint arXiv:2308.10792_.
  * Zheng et al. (2024) Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024.  Llamafactory: Unified efficient fine-tuning of 100+ language models.  _arXiv preprint arXiv:2403.13372_.

##  Appendix A Data Details

###  A.1 Subreddits

We briefly introduce each subreddit in Table 2. Statistics of data from
different subreddits are shown in Table 3.

Subreddit | Description
---|---
r/Anarchism |  | r/Anarchism is a forum with 268K members for discussing anarchism, assuming
---
participants are anarchists, allies, or learners.
r/Anarcho_Capitalism |  | r/Anarcho_Capitalism is a subreddit with 197K members dedicated to discussing
---
free-market capitalist anarchism and related topics, advocating for a society
where
voluntary interactions enhance liberty and opportunity for all.
r/Conservative |  | r/Conservative, with 1.1M members, offers the largest space on Reddit for fiscal
---
and social conservatives to explore and discuss political and cultural issues
from
a distinctly conservative perspective.
r/Libertarian |  | r/Libertarian is a forum with 506K members focused on libertarianism and the
---
principles of a free society, explicitly rejecting anti-libertarian ideologies
like communism.
r/Republican |  | r/Republican is a partisan subreddit with 191K members specifically designed
---
as a space for Republicans to discuss issues among themselves.
r/democrats |  | r/democrats is a subreddit with 438K members dedicated to supporting the
---
Democratic Party, offering daily updates, policy analysis, and opportunities
to
help elect Democrats at all levels.
r/Fitness |  | r/Fitness is the largest community dedicated to achieving physical fitness
---
goals with 12M members
r/loseit |  | r/loseit is a supportive community with 4M members for individuals of all
---
sizes to discuss healthy and sustainable weight loss methods
r/keto |  | r/keto is a community of 3.6M members for sharing experiences and advice
---
about the low-carb Ketogenic diet, which supports a range of health
conditions from diabetes to epilepsy
r/xxfitness |  | r/xxfitness is a community of 2.2M members for female and gender
---
non-binary/gender non-conforming Redditors to discuss fitness
r/GettingShredded |  | r/GettingShredded is a community of 162K members for anyone aiming to lose
---
body fat, improve their physique, and boost confidence through healthy fat
loss
and strength training discussions.
Table 2: Reddit forums used in this study. Descriptions of (top) _politics_
and (bottom) _fitness_ subreddits.

###  A.2 Statistics of CommInst

Table 3 describes the number of demonstrations in each subreddit. The variance
is due to their different topic coverage. Figure 6 presents the distribution
of instruction lengths and answer lengths in CommInst-Pol and CommInst-Fit.

Politics
---
| r/Anarch. | r/Anarcho. | r/Conserv. | r/Democr. | r/Libert. | r/Republic.
Comments | 182,192 | 589,354 | 2,391,745 | 142,274 | 1,950,089 | 140,572
Submissions | 31,567 | 41,264 | 356,842 | 90,001 | 40,967 | 48,808
Demonstrations | 2,430 | 3,027 | 2,943 | 3,159 | 2,928 | 2,574
Fitness
| r/Fitness | r/loseit | r/keto | r/xxfitness | r/GettingShredded
Comments | 907,839 | 967,645 | 603,920 | 420,082 | 78,053 |
Submissions | 2,430 | 3,027 | 2,943 | 3,159 | 2,928 |
Demonstrations | 3,438 | 2,793 | 1,716 | 2,883 | 1,215 |
Table 3: Dataset size measured by number of comments and submissions for each
community in political and fitness communities, and the number of learned
open-ended instruction-answer pairs (demonstrations).

(a) Instructions in CommInst-Pol

(b) Instructions in CommInst-Fit

(c) Answers in CommInst-Pol

(d) Instructions in CommInst-Fit

Figure 6: Distribution of instruction and answer lengths in CommInst-Pol and
CommInst-Fit.

##  Appendix B Implementation Details

###  B.1 Topic Modeling with BERTopic

Following the pipeline of BERTopic Grootendorst (2022), we first obtain the
text embeddings using _all-mpnet-base-v2_ Song et al. (2020). The embedding
dimensions are reduced with UMAP, with n_neighbors 15 and n_components 5. Then
the embeddings are clustered using HDBSCAN, with min_cluster_size 50. After
fitting the model, we remove texts that are not assigned any topics. Table 4
and Table 5 present the top-10 most frequent topics in _politics_ and
_fitness_.

idx | Topic Keywords | Frequency
---|---|---
1 | covid, vaccinated, masks, vaccines, healthcare, deaths, flu, pandemic, coronavirus, cdc | 48,787
2 | abortion, fetus, abortions, roe, body, choice, unborn, rights, parenthood, cells | 11,256
3 | climate, change, solar, emissions, pipeline, fuels, coal, power, supply, grid | 9,035
4 | guns, firearms, shootings, firearm, amendment, laws, armed, rifles, militia, nra | 8,833
5 | police, officer, officers, gun, force, law, crime, video, training, situation | 7,934
6 | kamala, woman, pelosi, vp, hillary, president, herself, biden, party, with | 7,879
7 | racist, racism, whites, racists, racial, minorities, democrats, privilege, whiteness, discrimination | 7,785
8 | libertarian, libertarians, libertarianism, party, government, left, ideology, vote, political, conservative | 6,666
9 | anarchism, anarchy, anarchist, anarchists, anarcho, hierarchies, communism, ideology, government, system | 6,311
10 | biden, bernie, obama, candidate, voters, debate, election, democrats, hillary, sanders | 5,768
Table 4: Top-10 most frequent topics in _politics_. Each topic is represented by the topic-10 keywords. idx | Topic Keywords | Frequency
---|---|---
1 | lbs, bmi, down, cw, gained, progress, healthy, tips, weeks, started | 18,296
2 | keto, ketosis, carbs, diet, 20g, protein, macros, started, calories, ketogenic | 13,078
3 | leggings, clothes, wear, pants, shorts, bras, shirts, clothing, dress, lululemon | 7,063
4 | motivation, journey, progress, yourself, change, motivated, success, start, proud, back | 5,142
5 | water, scale, weighing, retention, trend, normal, plateau, fluctuates, gain, weekly | 4,880
6 | sleep, morning, hours, work, gym, motivation, schedule, waking, exercise, routine | 4,486
7 | she, people, overweight, weight, obese, comment, healthy, skinny, talk, shaming | 4,192
8 | bf, percentage, fat, bodyfat, body, lean, bulk, estimates, lbs, pic | 3,930
9 | reps, sets, program, max, amrap, lifts, ranges, bench, 5x10, hypertrophy | 3,788
10 | shoulder, elbow, injury, arm, knees, physio, exercises, bicep, curls, heal | 3,775
Table 5: Top-10 most frequent topics in _fitness_. Each topic is represented
by the topic-10 keywords.

###  B.2 Prompting Template for CommInst

The prompting template for generating the open-ended instruction-answer pairs
in CommInst is shown in Figure 7.

Figure 7: Template prompt to generate CommInst.

###  B.3 Prompting Template for Generating CommSurvey

The prompting template for generating the multiple choice instruction-answer
pairs in CommSurvey is shown in Figure 8.

Figure 8: Template prompt to generate CommSurvey.

##  Appendix C Survey Sample

Samples of the surveys provided to human annotators for generating “semi-
ground truth data” are provided in Figures 9 and 10 (political and fitness
domains, respectively).

Figure 9: Sample of the multiple choice survey generated for human evaluation
in the politics domain. Figure 10: Sample of the multiple choice survey
generated for human evaluation in the fitness domain.