# FeedbackMap: a tool for making sense of open-ended survey responses

Doug Beeferman  Massachusetts Institute of TechnologyCambridge, MAUSA
<EMAIL_ADDRESS>and  Nabeel Gillani  Northeastern UniversityBoston, MAUSA
<EMAIL_ADDRESS>
(2023)

###### Abstract.

Analyzing open-ended survey responses is a crucial yet challenging task for
social scientists, non-profit organizations, and educational institutions, as
they often face the trade-off between obtaining rich data and the burden of
reading and coding textual responses. This demo introduces FeedbackMap, a web-
based tool that uses natural language processing techniques to facilitate the
analysis of open-ended survey responses. FeedbackMap lets researchers generate
summaries at multiple levels, identify interesting response examples, and
visualize the response space through embeddings. We discuss the importance of
examining survey results from multiple perspectives and the potential biases
introduced by summarization methods, emphasizing the need for critical
evaluation of the representation and omission of respondent voices.

datasets, natural language processing, text analysis, surveys

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: The 26th ACM Conference On Computer-Supported
Cooperative Work And Social Computing; Oct 14–18, 2023; Minneapolis, MN

##  1\. Introduction

Open-ended survey questions can give richer information to researchers than
closed-ended questions, with lower risks of certain kinds of bias (Reja et
al., 2003). But in deciding whether to add such a question, survey makers face
a trade-off between obtaining rich data and the burden of reading and coding
textual responses. We seek to reduce that burden and thereby make it more
compelling for surveyors to add questions with free-form textual answers. We
introduce FeedbackMap 111 A public version is available at
https://feedbackmap.org and its source code is available at
https://github.com/Plural-Connections/feedbackmap for organizations wishing to
host it themselves., a tool that summarizes a collection of open-ended
responses in multiple ways.

Our goal is a kind of multi-document summarization, which is well-studied in
the context of news stories and business communication. Survey responses
differ from these kinds of document collections in that they are from
individuals expressing different perspectives in response to a common
question. A summary of any text loses some of the nuance of the input, but
with survey responses this means that individual perspectives may be erased. A
survey summary may systematically prefer the majority response, or exclude
certain types of responses due to hidden biases in the summarization model.
Without careful evaluation, the analyst may jump to conclusions that influence
actions taken on behalf of the communities they serve. As such, there are
three priorities that guide the development of FeedbackMap: (1) Give the
researcher multiple perspectives on the data; (2) show interactions between
open-ended responses and known categorical variables; and (3) connect findings
back to individual responses to the survey.

##  2\. Related Work

###  2.1. Natural language processing for qualitative analysis

Qualitative researchers use tools like NVIVO (NVivo, 2023) to support manual
open-ended analyses, but they are often expensive to access and do little to
reduce the time burden of qualitative analyses. In recent years, researchers
have turned to automated methods to gather new insights from open-ended
corpora in time-efficient ways (Nelson, 2020; Roberts et al., 2014). However,
many of the most popular and frequently used methods are also dated in how
they operate in settings of data sparsity (like Latent Dirichlet Allocation
(Blei et al., 2003)), rendering them less effective in surfacing nuanced and
actionable findings from text corpora. Even when such tools make use of recent
and more powerful methods (like BERTopic (Grootendorst, 2022)), they often
require writing code to model and analyze bespoke datasets, creating barriers
for those who are not trained as engineers and analysts. Startups are jumping
in to fill this void, but may charge fees that make them inaccessible to
community organizations with limited budgets. There appears, then, to be a gap
between effectiveness, user-friendliness, and cost in the domain of open-ended
survey response analysis. Researchers are beginning to design and prototype
platforms to bridge this gap (Goldman et al., 2022; Mellon et al., 2022), yet
more work is needed to make these tools more general and accessible to wider
audiences while simultaneously remaining flexible enough to incorporate
emerging capabilities from the NLP community.

###  2.2. Biases in open-ended survey response analysis

Bias is inherent in any type of voluntary survey, and in particular, open-
ended survey questions. A recent study by Pew Research found that women,
younger adults, Hispanic and Black adults, and individuals with less formal
education were less likely to answer open-ended survey questions (Dunn and
Gómez, 2023), reflecting findings from prior work (Andrews, 2005). Of course,
even when people do respond, feelings of “question threat” may contribute to
biased or misleading responses (Bradburn et al., 1978).

Bias stems both from who responds as well as how responses are analyzed.
Practices like concept mapping (Jackson and Trochim, 2002) and computing
inter-coder reliability across multiple coders for the same content (O’Connor
and Joffe, 2020) seek to account for researcher biases in the analysis
process. FeedbackMap’s multiple data presentations seek to mitigate biases
that may be reinforced through a single analysis frame, yet the extent to
which this succeeds, and how it may help address issues of response biases
further upstream in the feedback-giving process—remain important open
questions for our work.

##  3\. System Overview

FeedbackMap is a Web application implemented using the Streamlit framework
(Streamlit, 2023). Organizations that host the tool can customize the language
models (LMs) to use for writing summaries and computing embeddings. LMs may be
API-based, such as the GPT3 model from the OpenAI API (OpenAI, 2023)–the
choice for summaries in our publicly deployed instance of the tool–or a local
transformer-based model from PyTorch-transformers (Wolf et al., 2020). Other
customizations include the types of summarization available to the end user
and their corresponding LM prompts. The tabs shown to the end user as they use
the Web application are described below.

###  3.1. Welcome tab

The Welcome tab invites the researcher to upload a comma-separated value (CSV)
file containing the results of their survey. The tool is designed to work best
with the format that is produced by Google Forms, but it will try to
accommodate any CSV or JSONL-formatted file. While the input file may be
arbitrarily large, the records will be randomly sampled for analysis if they
exceed a threshold (5000 rows, in the case of our publicly deployed instance.)

###  3.2. Summary tab

The Summary tab gives an overview of the selected data file and asks the
researcher to pick one of the open-ended questions in the data to analyze.
Here, by “open-ended” we mean that the survey asked for a free-form textual
response. While we can’t be certain of the question type from a schema-less
CSV file, in practice it’s usually easy to tell: FeedbackMap infers which
columns of data are open-ended and which are categorical by analyzing the
distribution of answer values for each question. The open-ended questions are
shown alongside the rate of nonempty responses for the question, while the
categorical questions are shown alongside information about the observed value
distribution. For categorical questions identified as multi-select, each value
is counted separately. Menus next to the categorical questions let the user
constrain the analysis (for example, to consider only the responses from one
U.S. state) prior to clicking on an open-ended question.

Figure 1 shows an excerpt of a synthetic survey data file, and the
corresponding Summary tab that FeedbackMap displays for the file.

Figure 1. An excerpt of the input file (left) and summary tab (right) for our
synthetic survey response data. The dataset consists of 1,020 GPT-4-generated
responses to questions about age, state of residence, favorite meal, and the
current weather, among others. The script to generate this data and its output
are available in the demo repository.

###  3.3. Analysis tab

The analysis tab contains the key offerings of the tool, letting the user
explore the open-ended responses to a question of interest in various ways,
each way contained within a collapsible subsection. Figure 2 shows the
subsections of the analysis tab for our hypothetical input file. We discuss
these in more detail below:

(a) Top-level abstractive summmary

(b) Topic scatterplot

(c) Cluster summaries (clustering by response text)

(d) Cluster summaries (clustering by U.S. state)

Figure 2. Some elements of the Analysis tab when the user selects the question
”What is your favorite meal?”

####  3.3.1. Top-level abstractive summary

This section shows a top-level summary of the responses to the selected open-
ended question. It is generated using an LM and prompt that may be controlled
in the code configuration. The prompt to the LM is a random sample of
responses (sample size chosen to maximize the use of the LM’s input context
window), followed by an instructional statement such as the default, ”Briefly,
what do these responses have in common?”

####  3.3.2. Topic scatterplot

This section shows a two-dimensional interactive scatterplot that organizes
the individual responses by topic, such that responses about similar topics
are close to each other. Users may read the individual responses by hovering
over one of the points. This section uses a familiar pipeline of techniques
(used by, e.g., BERTopic (Grootendorst, 2022)) to go from the raw text to the
scatterplot: namely, it computes the embedding for each response according to
a sentence embedding model, and then projects the set to 2 dimensions. By
default, the points are clustered and color-coded according to the result of a
clustering algorithm applied to these same embeddings. We use UMAP (McInnes et
al., 2018) for projection and HDBSCAN (McInnes et al., 2017) for clustering.
The user may choose to override this ”auto-clustering” and instead group by
one of the categorical variables in the survey data. Labels for the clusters
are determined by terms in the responses that have high pointwise mutual
information with respect to the other clusters.

####  3.3.3. Interesting examples

Here FeedbackMap uses the LM to ask for noteworthy responses. As in the top-
level summary, this happens by prompting the LM with a random selection of
responses, followed by a fixed instructional prompt. The default instructional
portion is “What are 3 interesting responses and why?”, which elicits
rationales for the LM’s choices. A “Pick again” prompt allows the user to
rerun the generation, based on a new random sample.222There is no guarantee
that the LM generates examples that are actually in the sample we include in
the prompts; so far we have not seen a case of a hallucinated example, but
this is certainly possible and should be part of any evaluation of the tool.

####  3.3.4. Cluster summaries

This section shows an LM-generated summary for each cluster, color-coded to
match the scatterplot.

####  3.3.5. Top words and phrases

We identify words and collocations (multi-word terms) that are frequent in the
data and show how they interact with the selected category in tabular form.
Columns of the table correspond to categorical values and can be sorted to
surface terms that are highly associated with the category.

##  4\. Discussion and evaluation plan

Since deploying FeedbackMap we have shared it with three users, including a
school district administrator, a non-profit founder, and a political
scientist, all of whom had suitable data sets for it. Reactions were positive,
and varied with respect to which features were seen as most useful, with the
top-level and category-specific summaries being highlights for all three
users. We hope to run a more rigorous field study of the tool with these users
and others, pending IRB approval. We also plan to measure the impact of survey
summarization on users with respect to the kinds of knowledge they impart and
conceal about the underlying data set.

## References

  * (1)
  * Andrews (2005) Mark Andrews. 2005\.  Who Is Being Heard? Response Bias in Open-ended Responses in a Large Government Employee Survey. In _AAPOR - ASA Section on Survey Research Methods_.
  * Blei et al. (2003) D. M. Blei, A. Y. Ng, and M. I. Jordan. 2003.  Latent Dirichlet Allocation.  _Journal of Machine Learning Research_ (2003), 993–1022.  Issue 3.
  * Bradburn et al. (1978) Norman M. Bradburn, Seymour Sudman, Ed Blair, and Carol Stocking. 1978.  Question Threat and Response Bias.  _Public Opinion Quarterly_ 42, 2 (1978), 221–234.
  * Dunn and Gómez (2023) Amina Dunn and Vainney Gómez. 2023.  Nonresponse rates on open-ended survey questions vary by demographic group, other factors.  Pew Research Center.  https://www.pewresearch.org/decoded/2023/03/07/nonresponse-rates-on-open-ended-survey-questions-vary-by-demographic-group-other-factors/
  * Goldman et al. (2022) Ariel Goldman, Cindy Espinosa, Shivani Patel, Francesca Cavuoti, Jade Chen, Alexandra Cheng, Sabrina Meng, Adil Patil, Lydia B. Chilton, and Sarah Morrison-Smith. 2022.  QuAD: Deep-Learning Assisted Qualitative Data Analysis with Affinity Diagrams. In _CHI EA ’22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–7.
  * Grootendorst (2022) Maarten Grootendorst. 2022\.  BERTopic: Neural topic modeling with a class-based TF-IDF procedure.  _arXiv preprint arXiv:2203.05794_ (2022).
  * Jackson and Trochim (2002) Kristin M. Jackson and William M. K. Trochim. 2002.  Concept Mapping as an Alternative Approach for the Analysis of Open-Ended Survey Responses.  _Organizational Research Methods_ 5, 4 (2002).
  * McInnes et al. (2017) Leland McInnes, John Healy, and Steve Astels. 2017\.  hdbscan: Hierarchical density based clustering.  _J. Open Source Softw._ 2, 11 (2017), 205.
  * McInnes et al. (2018) Leland McInnes, John Healy, and James Melville. 2018\.  Umap: Uniform manifold approximation and projection for dimension reduction.  _arXiv preprint arXiv:1802.03426_ (2018).
  * Mellon et al. (2022) Jonathan Mellon, Jack Bailey, Ralph Scott, James Breckwoldt, and Marta Miori. 2022.  Does GPT-3 know what the Most Important Issue is? Using Large Language Models to Code Open-Text Social Survey Responses At Scale.  _Using Large Language Models to Code Open-Text Social Survey Responses At Scale (December 22, 2022)_ (2022).
  * Nelson (2020) Laura K. Nelson. 2020\.  Computational Grounded Theory: A Methodological Framework.  _Sociological Methods & Research_ 49, 1 (2020), 3–42.
  * NVivo (2023) NVivo. 2023.  NVivo.  https://lumivero.com/products/nvivo Accessed: 2023-05-11.
  * O’Connor and Joffe (2020) Cliodhna O’Connor and Helene Joffe. 2020.  Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines.  _International Journal of Qualitative Methods_ 19 (2020), 1–13.
  * OpenAI (2023) OpenAI. 2023.  OpenAI API.  https://beta.openai.com Accessed: 2023-05-11.
  * Reja et al. (2003) Urša Reja, Katja Lozar Manfreda, Valentina Hlebec, and Vasja Vehovar. 2003.  Open-ended vs. close-ended questions in web questionnaires.  _Developments in applied statistics_ 19, 1 (2003), 159–177.
  * Roberts et al. (2014) Margaret E. Roberts, Brandon M. Stewart, Dustin Tingley, Christopher Lucas, Jetson Leder-Luis, Shana Kushner Gadarian, Bethany Albertson, and David G. Rand. 2014\.  Structural Topic Models for Open-Ended Survey Responses.  _American Journal of Political Science_ 58, 4 (2014), 1064–1082.
  * Streamlit (2023) Streamlit. 2023\.  Streamlit – The fastest way to build and share data apps.  https://streamlit.io Accessed: 2023-05-11.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020\.  Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_. 38–45.