*

A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Department of
Computer Science
National University of Singapore

Incorporating Different Verbal Cues to Improve Text-Based Computer-Delivered
Health Messaging

_Author:_

Samuel Rhys COX

_Supervisor:_
Associate Professor
Wei Tsang OOI
Co-supervisor:
Assistant Professor
Brian Y. LIM
Examiners:
Associate Professor Zhao Shengdong
Associate Professor Bimlesh Wadhwa

November, 2023

###### Abstract

The ubiquity of smartphones has led to an increase in on demand healthcare
being supplied. For example, people can share their illness-related
experiences with others similar to themselves, and healthcare experts can
offer advice for better treatment and care for remediable, terminal and mental
illnesses. As well as this human-to-human communication, there has been an
increased use of human-to-computer digital health messaging, such as chatbots.
These can prove advantageous as they offer synchronous and anonymous feedback
without the need for a human conversational partner. However, there are many
subtleties involved in human conversation that a computer agent may not
properly exhibit. For example, there are various conversational styles,
etiquettes, politeness strategies or empathic responses that need to be chosen
appropriately for the conversation. Encouragingly, computers are social actors
(CASA) posits that people apply the same social norms to computers as they
would do to people. On from this, previous studies have focused on applying
conversational strategies to computer agents to make them embody more
favourable human characteristics. However, if a computer agent fails in this
regard it can lead to negative reactions from users. Therefore, in this
dissertation we describe a series of studies we carried out to lead to more
effective human-to-computer digital health messaging.

In our first study, we use the crowd to write diverse motivational messages to
encourage people to exercise. Yet, it is not a simple case of asking the crowd
to write messages, as people may write similar ideas leading to duplicated and
redundant messages. To combat this, previous work has shown people example
ideas to direct them to less commonly thought of ideas. However, these studies
often relied on manually labelling and selecting example ideas. Therefore, we
developed Directed Diversity, a system which uses word embeddings to
automatically and scalably select collectively diverse examples to show to
message writers. In a series of user studies we asked people to write messages
either using examples selected by Directed Diversity, random examples, or no
examples. We evaluated the diversity of messages using a selection of
diversity metrics from different domains, human evaluations, and expert
evaluations, and found that our method led to more diverse messages compared
to the baseline conditions. Additionally, subjective evaluations of the
messages found that directed diversity generated more informative and equally
motivating messages compared to the baseline conditions.

Our second study investigates the effect of a health chatbot’s conversational
style on the quality of response given by a user. The tone and language used
by a chatbot can influence user perceptions and the quality of information
sharing. This is especially important in the health domain where detailed
information is needed for diagnosis. In two user studies on Amazon Mechanical
Turk, we investigated the effect of a chatbot’s conversational style (formal
vs casual) on user perceptions, likelihood to disclose, and quality of user
utterances. In general, results indicate the benefits of using a formal
conversational style within the health domain. In the first user study, 187
users were asked for their likelihood to disclose either their income level,
credit score, or medical history to a chatbot to assist finding life
insurance. When the chatbot asked for medical history, users perceived a
formal conversation style as more competent and appropriate. In the second
user study, 156 users discussed their dental flossing behaviour with a chatbot
which used either a formal or casual conversational style. We found that users
who do not floss every day wrote more specific and actionable user utterances
when the chatbot used a formal conversational style.

In our final study, we investigate the format used by a chatbot when
referencing a user’s utterance from a previous chatting session. Our
motivation for this study is driven in part by the Personalisation-Privacy
Paradox: a tension between providing personalised services to users to improve
their experience, and providing less personalised services so as not to seem
invasive to users. To explore this trade-off, we investigated how a chatbot
references previous conversations with a user and its effects on a user’s
privacy concerns and perceptions. In a three-week longitudinal between-
subjects study, 169 participants talked to a chatbot that either, (1-None):
did not explicitly reference previous user utterances, (2-Verbatim):
referenced previous utterances verbatim, or (3-Paraphrase): used paraphrases
to reference previous utterances. Participants found Verbatim and Paraphrase
chatbots more intelligent and engaging. However, the Verbatim chatbot also
raised privacy concerns with participants. Upon study completion, we conducted
semi-structured interviews with 15 participants to gain insights such as to
why people prefer certain conditions or had privacy concerns. These findings
can help designers choose an appropriate form of referencing previous user
utterances and inform in the design of longitudinal dialogue scripting.

In summary, we have researched how to create more effective digital health
interventions starting from generating health messages, to choosing an
appropriate formality of messaging, and finally to formatting messages which
reference a user’s previous utterances.

###### Acknowledgements

I would like to offer my thanks to the following for their continued guidance
and support:

  * •

Associate Professor Wei Tsang Ooi

  * •

Assistant Professor Brian Y. Lim

  * •

My parents, Grace, Ashraf, Bob, Therisa and Anna.

###### Contents

  1. 1 Introduction
    1. 1.1 Motivation
    2. 1.2 Research Questions
      1. 1.2.1 Generating Diverse and Efficacious Health Messages
      2. 1.2.2 Choosing an Appropriate Conversational Style for a Health Chatbot
      3. 1.2.3 Choosing an effective format for a health chatbot to reference a user’s previous utterances
    3. 1.3 Thesis Outline
  2. 2 Background Research and Related Work
    1. 2.1 How should messaging be generated?
      1. 2.1.1 Machine-generated messages
      2. 2.1.2 Human-generated messages
    2. 2.2 Crowd-generated messages: Can the crowd perform as well as experts?
    3. 2.3 Downsides of the crowd, and how to overcome them
    4. 2.4 How should messages be formatted and stylised?
    5. 2.5 How should messaging be selected?
      1. 2.5.1 Selecting messages algorithmically
    6. 2.6 When should messages be sent to users?
    7. 2.7 Summary of Related Work
  3. 3 Generating Diverse and Efficacious Health Messages
    1. 3.1 Introduction
    2. 3.2 Background and Related Work
      1. 3.2.1 Cognitive Psychology of Creative Ideation
      2. 3.2.2 Creativity Support Tools for Crowd Ideation
      3. 3.2.3 Supporting Collective Intelligence with Artificial Intelligence
    3. 3.3 Technical Approach
      1. 3.3.1 Phrase Extraction
      2. 3.3.2 Phrase Embedding
      3. 3.3.3 Phrase Selection
      4. 3.3.4 Directing with Prompts of Grouped Phrases
    4. 3.4 Diversity Prompting Evaluation Framework
      1. 3.4.1 Research Questions and Experiments
      2. 3.4.2 Diversity and Creativity Measures of Prompting and Ideation
      3. 3.4.3 Factor analyses to draw constructs from experiment variables
    5. 3.5 Evaluation: applying framework to study Directed Diversity
      1. 3.5.1 Characterisation Simulation Study
      2. 3.5.2 Ideation User Study
      3. 3.5.3 Validation User Studies
      4. 3.5.4 Summary of Answers to Research Questions
    6. 3.6 Discussion
      1. 3.6.1 Need for Sensitive and Mechanistic Measures of Creativity
      2. 3.6.2 Generalization of Directed Diversity to other Domains
      3. 3.6.3 Generalization of Evaluation Framework
    7. 3.7 Conclusion
  4. 4 The Effects of a Chatbot’s Language Formality on a Users’ Information Sharing
    1. 4.1 Introduction
    2. 4.2 Related Work
      1. 4.2.1 Designing Chatbots to Act Like People
      2. 4.2.2 Improving Self-Disclosure to Chatbots
      3. 4.2.3 Changing Chatbot Conversational Style
    3. 4.3 Study 1: The Effect of Conversational Style on Disclosure
      1. 4.3.1 Experiment Conditions
      2. 4.3.2 Design Considerations
      3. 4.3.3 Participants
      4. 4.3.4 Procedure
      5. 4.3.5 Measures
      6. 4.3.6 Post Interaction Survey
    4. 4.4 Study 1: Results and Findings
      1. 4.4.1 Qualitative findings from user feedback
    5. 4.5 Study 2: The Effect of Conversational Style on Response Quality
      1. 4.5.1 Experiment Conditions
      2. 4.5.2 Participants
      3. 4.5.3 Procedure
      4. 4.5.4 Subjective Measures
      5. 4.5.5 Analysing User Utterances
      6. 4.5.6 Post Interaction Survey
    6. 4.6 Study 2: Results and Findings
      1. 4.6.1 Response Informativeness
    7. 4.7 Summary of Key Findings:
    8. 4.8 Discussion
    9. 4.9 Limitations
    10. 4.10 Conclusion
  5. 5 The Effect of How a Health Chatbot Formats and References a User’s Previous Utterances
    1. 5.1 Introduction
    2. 5.2 Related Work
      1. 5.2.1 Increasing Social Presence and User Perceptions of Chatbots
      2. 5.2.2 Privacy Concerns Among Chatbot Users
    3. 5.3 User Study
      1. 5.3.1 Chatbot Script
      2. 5.3.2 Implementation Details
      3. 5.3.3 Participants
      4. 5.3.4 Procedure
      5. 5.3.5 Measures
    4. 5.4 User Study Results
      1. 5.4.1 General Chatbot Perceptions
      2. 5.4.2 Privacy Perceptions
      3. 5.4.3 Self-declared flossing frequency and beliefs
    5. 5.5 Semi-Structured Interviews
      1. 5.5.1 Participants
      2. 5.5.2 Procedure
      3. 5.5.3 Findings
    6. 5.6 Summary of Key Findings
    7. 5.7 Discussion
    8. 5.8 Limitations and Future Work
    9. 5.9 Conclusion
  6. 6 Conclusion and Future Work
    1. 6.1 Generating and maintaining diverse content - LLMs or the crowd?
      1. 6.1.1 Directed Diversity: comparing LLM and crowd-written messages
      2. 6.1.2 Prompt engineering with other previous crowdsourcing instructions
    2. 6.2 The importance of message style
  7. A Chapter 3 Appendices
    1. A.1 Definitions of Prompt Selection Variables
    2. A.2 Intra-Prompt Diversity Metrics based on Embedding Distances
    3. A.3 Definitions of Prompt Adoption Metrics
    4. A.4 Pairwise Embedding Distances of Phrases and Messages
    5. A.5 Results of Characterisation Simulation Study
    6. A.6 Factor Loadings from Factor Analysis in User Studies
    7. A.7 Survey Screenshots in User Studies
      1. A.7.1 Ideation User Study
    8. A.8 Validation User Studies
    9. A.9 Examples of Prompts Shown to and Messages Written by Ideators
    10. A.10 Linear Mixed Models and statistical analysis results of Prompt Creativity, Prompt-Ideation Mediation, and Ideation Diversity
    11. A.11 Investigating Confound of Prompt Understandability on Ideation Diversity
    12. A.12 Examples of Message-Group Ranking
  8. B Chapter 5 Appendices
    1. B.1 Plots of weekly measures
    2. B.2 Health related outcomes
    3. B.3 Full Chatbot Script
      1. B.3.1 Week 1 Script
      2. B.3.2 Week 2 Script
      3. B.3.3 Week 3 Script
    4. B.4 Paraphrases Used
      1. B.4.1 Paraphrases given in Week 2:
      2. B.4.2 Paraphrases given in Week 3:
    5. B.5 Semi-Structured Interview Guide
  9. C Chapter 6 Appendices
    1. C.1 Generating exercise plans (using instructions from Agapie et al. [6])
      1. C.1.1 Prompt used to generate an exercise plan using GPT-4. Prompt uses instructions taken from Agapie et al. [6].
      2. C.1.2 GPT-4 exercise plan output:

###### List of Figures

  1. 3.1 Pipeline of the overall technical approach to extract, embed, and select phrases to generate diverse prompts. a) Phrase extraction by collecting phrases from online articles and discussion forums (shown as pages), filtering phrases to select a clean subset (shown as the black dash for each phrase); b) Phrase embedding using the Universal Sentence Encoder [46] to compute the embedding vector of each phrase (shown as scatter plot); c) Phrase Selection by constructing the minimal spanning tree to select optimally spaced phrases (see Figure 3.3 for more details).
  2. 3.2 Demonstration of pairwise embedding angular distances between an example text items (first data row) and neighboring text items. Text items with semantically similar words have smaller distances. For interpretability, we highlighted words to indicate darker color with higher cosine similarity to the first phrase.
  3. 3.3 Procedure to direct ideation towards diverse phrases (top) and away from prior or redundant ideas (bottom). To attract ideation with diverse prompts: a) start with embeddings of corpus-extracted phrases; b) construct minimum spanning tree (MST); c) traverse tree to select distant prompts from clusters (most distant points as green dots, in clustered phrases as green ellipses); d) selected prompts are the most diverse. To repel ideation from prior ideas, e) compute embeddings of prior ideas (red hollow dots); f) compute prompt-ideation pairwise distances of all prompts from each prior ideation, exclude phrases (dotted black circles) with pairwise distance less than a user- defined threshold (red bubble), and construct the MST with remaining phrases; g) traverse MST to select a user- defined number of prompts; h) selected prompts are diverse, yet avoids prior ideas.
  4. 3.4 Diversity prompting evaluation framework to evaluate prompting to support diverse ideation along the ideation chain. We pose research questions (RQ3.1-3.3) between each step to validate the ideation diversification process. For each step, we manipulate or measure various experiment constructs to track how well ideators are prompted to generate creative ideas. Except for prompt selection, each construct refers to a statistical factor determined from factor analyses of multiple dependent variables. Constructs are grouped in colored blocks indicating different data collection methods as follows: (Blue - Computed embedding-based metric, Green - ratings from ideators, Red - ratings from validators, Purple - thematic coding of ideations).
  5. 3.5 Metrics of distances between two points in a multi-dimensional vector space. Each metric can be calculated for an individual text item. These metrics can apply to the embedding of phrases or ideations.
  6. 3.6 Metrics of diversity of phrases or ideation embeddings in a vector space. These capture more characteristics of diversity than average distances in Table 2. Each metric can only be calculated collectively for multiple items.
  7. 3.7 Comparison of diversity metrics for canonical examples of distributed points in a 2D space. Points farther apart mean higher diversity. Here, we calculate Euclidean instead of angular distance, but intuitions are similar.
  8. 3.8 Constructs from factor analyses of variables along ideation chain. Factor loadings in Appendix Tables A.7 to A.14.
  9. 3.9 2D UMAP projection showing how diversely selected prompts and resulting ideation messages are distributed based on Directed or Random prompt selection technique and prompt size (number in brackets). Each point represents the embedding of a text item. Light grey points represent all phrases in the extracted corpus, dark grey points represent selected phrases from the simulation study (Section 3.5.1) and blue dots represent the ideated messages written by crowdworkers in the ideator user study (Section 3.5.2). Gradient lines connect ideation messages to their stimulus prompts.
  10. 3.10 Example prompts shown to participants in different conditions: None (left, $g=0$, Directed(1) (center, $g=1$), and Directed(3) (right, $g=3$). Phrase texts would be different for Random(1) and Random(3) selection techniques.
  11. 3.11 Results of ideators’ perceived prompt creativity (Top) and ideation effort (Bottom) for different Prompt Selection technique and Prompt Size. All factors values on a 5-point Likert scale (–2 = “Strongly Disagree” to 2 = “Strongly Agree”). Dotted lines indicate extremely significant $p<.0001$ comparisons, otherwise very significant with p-value stated; solid lines indicate no significance at $p>.01$. Error bars indicate 90% confidence interval.
  12. 3.12 Results of computed individual and collective diversity from ideations for different prompt configurations. See Figure 3.11 caption for how to interpret charts.
  13. 3.13 Metrics of creativity of ideation based on categories and themes derived from our thematic analysis of generated ideas. Metrics are shown for categories, but are the same for themes.
  14. 3.14 Results of diversity in categories and themes derived from thematic analysis of ideations.
  15. 3.15 Results of perceived individual and collective creativity from the three validation user studies.
  16. 4.1 The chatbot conversation for both casual and formal conversational styles when asking for the user’s credit score.
  17. (a) Casual chat condition: with F-score of 42 (similar to F-score of normal spoken English conversation [135])
  18. (b) Formal chat condition: example with F-score of 50.5 (similar to F-score of prepared English speech [135])
  19. 4.2 Key figures for Study 1. Graphs 4.2(b) and 4.2(c) show significant differences between casual and formal conversational styles when medical history was requested.
  20. (a) Participants were most likely to disclose income level and least likely to disclose medical history
  21. (b) When asked for their medical history, participants perceived a chatbot as more competent if it used a formal conversational style
  22. (c) When asked for their medical history, participants perceived a chatbot as more appropriate if it used a formal conversational style
  23. 4.3 The Study 2 chatbot for both casual and formal conversational style conditions. The interface is as seen by participants with chatbot utterances in the grey speech bubbles and actual participant responses in green speech bubbles. The F-score for the casual condition was 40.5, and the F-score for the formal condition was 47.5.
  24. 4.4 Users who perceived flossing as more important provided longer utterances.
  25. 4.5 Count of specificity and actionability labelling for user utterances. Participants provide more specific and actionable utterances when conversing with a formal chatbot compared to a casual chatbot.
  26. (a) Formal participants provided fewer “bad” and more “ok” specificity responses than Casual participants.
  27. (b) Formal participants provided fewer “bad” actionability responses than Casual participants.
  28. 5.1 An extract from week 3 of the user study showing the 3 levels of chatbot referencing format. Grey bubbles are chatbot utterances, and teal bubbles are user utterances. Differences in referencing format between conditions are circled in red.
  29. 5.2 Experiment procedure. Participants were divided into 3 conditions, and talked to a chatbot once a week for 3 weeks about their dental flossing habits and beliefs.
  30. 5.3 User utterances and their potential None, Verbatim and Paraphrase chatbot responses across all 3 weeks. Grey bubbles are chatbot utterances, and teal bubbles are user utterances. Red arrows show where a user utterance would be referenced (by Verbatim and Paraphrase) in the following week.
  31. 5.4 Outcome measures by question asked in the final week of the study. Significance $p<0.05$ indicated by $\ast$, and $p<0.10$ indicated by +.
  32. 6.1 Prompt instructions given to GPT-4. Instructions for crowd-workers (similar to Phrase-GPT) found in [74].
  33. 6.2 Prompting GPT-4 using scenarios from De Vries et al. [79]. Scenarios describe the stage of change (see Transtheoretical Model [253]) of an individual
  34. (a) Using a scenario describing a precontemplation stage of change.
  35. (b) Using a scenario describing a maintenance stage of change.
  36. A.1 Independent variables used in the simulation and user studies to manipulate how prompts are shown to ideators.
  37. A.2 Metrics of prompt diversity for all phrases in a single prompt.
  38. A.3 Metrics indicating how much of prompt text and concepts are adopted into the ideations.
  39. A.4 Distribution of pairwise distances between the extracted phrases (N=3,666). The pairwise distances ranged from Min=0.057 to Max=0.586, Median=0.430, inter-quartile range 0.394 to 0.460, SD= 0.047.
  40. A.5 Distribution of pairwise distances between the messages (N=250) ideated in the pilot study with no prompting (None). The pairwise distances ranged from Min=0.169 to Max=0.549, Median=0.405, inter-quartile range 0.376 to 0.432, SD=0.043.
  41. A.6 Influence of prompt selection technique, prompt size, and prompt count on various distance and diversity metrics. Higher values for all metrics indicate higher diversity. Span and Sparseness results are not shown, but are similar to Mean Distance. Note that we computed the mean of MST edge distances instead of sum, which is independent of number of prompts. Error bars are extremely small, and not shown for simplicity.
  42. A.7 The rotated factor loading of factor analysis on metrics of prompt distance and consistency. Factors explained 73.6% of the total variance. Bartlett’s Test for Sphericity to indicate common factors was significant ($\chi^{2}$= 5810, $p<.0001$).
  43. A.8 The rotated factor loading of factor analysis on metrics of perceived helpfulness of prompts. Factors explained 68.9% of the total variance. Bartlett’s Test for Sphericity to indicate common factors was significant ($\chi^{2}$= 2575, $p<.0001$).
  44. A.9 The rotated factor loading of factor analysis on metrics of prompt adoption. Factors explained 65.1% of the total variance. Bartlett’s Test for Sphericity to indicate common factors was significant ($\chi^{2}$= 1315, $p<.0001$).
  45. A.10 The rotated factor loading of factor analysis on diversity metrics of generated messages. Factors explained 75.2% of the total variance. Bartlett’s Test for Sphericity to indicate common factors was significant ($\chi^{2}$= 2676, $p<.0001$).
  46. A.11 The rotated factor loading of factor analysis on metrics of perceived quality of the generated messages. Factors explained 80.9% of the total variance. Bartlett’s Test for Sphericity to indicate common factors was significant ($\chi^{2}$= 5810, $p<.0001$).
  47. A.12 The rotated factor loading of factor analysis on metrics of group ranking of the generated messages. Factors explained 93.9% of the total variance. Bartlett’s Test for Sphericity to indicate common factors was significant ($\chi^{2}$= 366, $p<.0001$). For usability, “unrepetitive” was measured with the word “repetitive” in the survey.
  48. A.13 The rotated factor loading of factor analysis on metrics of message distinctness. Factors explained 74.8% of the total variance. Bartlett’s Test for Sphericity to indicate common factors was significant ($\chi^{2}$= 1022, $p<.0001$).
  49. A.14 The rotated factor loading of factor analysis on metrics of ideation effort. Factors explained 59.0% of the total variance. Bartlett’s Test for Sphericity to indicate common factors was significant ($\chi^{2}$= 1008, $p<.0001$).
  50. A.15 The instructions in the Ideation User Study for the None condition.
  51. A.16 For the None condition, users are asked to write a message that is between one and three sentences long.
  52. A.17 The instructions in the Ideation User Study for the Random(1) and Directed(1) conditions.
  53. A.18 Random(1) and Directed(1) prompts consisted of one phrase per prompt. Different phrases were used for each trial.
  54. A.19 Random(1) and Directed(1) prompts consisted of one phrase per prompt. Different phrases were used for each trial.
  55. A.20 Random(3) and Directed(3) prompts consisted of three phrases per prompt. Different phrases were used for each trial.
  56. A.21 Ideators are asked to evaluate the message they wrote by providing Likert scale ratings for many different factors along with a short reflection about the message writing process. The screenshot above shows the evaluation screen for Directed(3).
  57. A.22 The instructions for individual message rating tasks.
  58. A.23 Validators rated a randomly selected message on a 7-point Likert scale and gave a justification.
  59. A.24 Validators were asked to rate the difference between two messages in a message-pair.
  60. A.25 The instructions for group message ranking tasks.
  61. A.26 Validators were asked to rank groups of messages for motivation, informativeness and repetitiveness. Note that while we used the word “repetitive” for usability in the survey, we analysed this dependent variable as “unrepetitive” to be consistent with other diversity metrics.
  62. A.27 Messages generated in our study and the phrase prompt(s) that were shown to ideators.
  63. A.28 Statistical analysis of responses due to effects (one per row), as linear mixed eﬀects models, all with Participant as random eﬀect, Prompt Selection and Prompt Size as fixed effects, their interaction effect. a) model for manipulation check analysis of how prompt configurations affect perceived prompt creativity (RQ3.1.2); b) model for mediation analysis of how prompt configurations affect ideation effort (RQ3.2.2). n.s. means not significant at $p>.01$. $p>F$ is the significance level of the fixed effect ANOVA. $R^{2}$ is the model’s coefficient of determination to indicate goodness of fit.
  64. A.29 Statistical analysis and results of mediation effects (RQ3.2.3) of how prompt configurations (a) and perceived prompt creativity (b) affect ideation diversity. See Table A.28 caption to interpret tables. Positive and negative numbers in second column represent estimated model coefficients indicating how much each fixed effect influences the response.
  65. A.30 Statistical analysis of how prompt selection influences ideation diversity defined by different metrics (RQ3.3): a) individual diversity, b) collective diversity, and c) thematic diversity. See Table A.28 caption for how to interpret tables.
  66. A.31 Statistical analysis of how prompt selection influences ideation creativity as validated by different methods (RQ3.3.1): a) individual rating, b) collective ranking, and c) collective pairwise rating. See Table A.28 caption for how to interpret tables.
  67. A.32 Statistical analysis of a) how ideators’ understanding of phrases influences ideation diversity and b) how similar Directed ideations are to their prompts compared to other None and Random Messages.
  68. A.33 Results of computed individual diversity from ideations for different prompt configurations for (left) prompts that users understood ($>0$) or did not and (right) ideations that were fast or slow.
  69. A.34 Results of prompt-message distance (how dissimilar a prompt is from a message) comparing different messages with respect to Directed(3) prompts.
  70. A.35 Examples of the factors with low $(<$ Median) and high ($\geq$ Median) scores for “Ideations Unrepetitive” and “Ideations Informative”.
  71. B.1 Desire to continue measures for weeks 1 to 3 by condition
  72. B.2 Positive engagement measures for weeks 1 to 3 by condition
  73. B.3 Negative engagement measures for weeks 1 to 3 by condition
  74. B.4 Intelligence measures for weeks 1 to 3 by condition
  75. B.5 Competence measures for weeks 1 to 3 by condition
  76. B.6 Flossing Frequency for weeks 1 to 3.
  77. B.7 Flossing Frequency for weeks 1 to 3 by condition.
  78. B.8 Weekly differences from week 1 flossing frequency
  79. B.9 Weekly differences from week 1 flossing frequency by experiment condition
  80. B.10 Change in intention to floss from week 1 to week 3 by experiment condition
  81. B.11 Change in self-efficacy to floss from week 1 to week 3 by experiment condition
  82. B.12 Change in perceived susceptibility from week 1 to week 3 by experiment condition
  83. C.1 Exercise plan generated by GPT-4 in response to prompt from Section C.1.1.

##  Chapter 1 Introduction

This thesis investigates the use of verbal cues in computer driven text-based
messaging to assist users in developing healthy habits, and describes several
of our studies that answer related research questions. In this introduction
chapter, Section 1.1 describes the motivation behind our work, and the context
behind each of our studies. Section 1.2 lays out our research questions, and
explains the methods used to answer them. Finally, Section 1.3 provides an
outline of the chapters in this report.

###  1.1 Motivation

As part of a healthy lifestyle and to avoid chronic diseases, it is
recommended that we adhere to various health guidelines [263, 105, 134, 282,
229]. These could include getting sufficient exercise [263, 105], eating a
balanced and healthy diet [134, 282], flossing our teeth [272, 121, 68] or
getting enough sleep [229]. To help us adhere to these recommendations, we can
turn to social support: the provision of assistance or comfort, such as
sharing advice or emotional support [284, 32]. Social support can help us
perform more healthy behaviours [32, 65], in part by improving our motivation
[267] and self-efficacy [93].

Traditionally, this social support would come from human connections such as
health experts, friends, or family [124], and could be provided in person or
via messages [39]. However, this support pool is finite and may not be readily
available to everyone due to lack of social connections, financial
constraints, or time constraints [103]. Fortunately, a solution to this gap in
human support can come from the use of health messaging technologies [3, 298,
18], which can provide immediacy in their responses to users [245].

Health messaging technologies are already publicly available [325, 312, 61],
and could message users via nudging interventions (e.g., push-notifications to
remind someone to exercise [254, 120, 130]), or via longer two-way
conversations (e.g., chatbots to discuss someone’s cancer treatment [47, 22]).
Additionally, the technology that makes this possible is becoming accessible
to more people, and (as of 2020) nearly 50% of the world population owns a
smartphone [227]. This gives the option for low-cost and highly accessible
support for people with their health needs [245, 170], as the technology is
scalable provided that infrastructure (reliable Internet or phone service)
exists. Furthermore, health messaging technologies have seen some acceptance
among both end-users [219, 125, 130] (such as chatbots for anonymous symptom
sharing [219]) and medical professionals [236] (such as for information
sharing and appointment scheduling). However, both physicians and consumers
have raised lack of empathy and inability to understand human emotions as
potential barriers to engagement with health messaging technology [219, 236].

These concerns lead us to a potential impasse. Namely, the question of “How
should technology communicate with us?”. We want technology which people
perceive as understanding their emotions and that responds empathetically.
However, we do not want technology that enters the uncanny valley [137, 330,
290] where communication is seen as almost human-like, but something is not
quite right. Equally we do not want technology that responds in a way seen as
inappropriate to a given situation [72, 56]. For example, users may find it
less appropriate if a chatbot uses a casual conversational style when
discussing their sensitive health information [72]. In this case
conversational style is an example of a social cue, a signal that saliently
conveys meaning to people and leads them to apply social rules, expectations
and scripts to interactions [101]. Social cues can be split into those that
are verbal cues (such as conversational style [72] and conversational content
[71, 74, 259]); auditory cues [38] (such as voice qualities and
vocalisations); visual cues [177] (such as agent appearance, proxemics, and
kinesics); and invisible cues [101] (such as chronemics and haptics).

Effects of social cues and their resulting social scripts led to the
development of the interaction paradigm Computers are Social Actors (CASA)
[220, 111]. CASA states that people mindlessly apply the same social
heuristics to computers as they do to humans, by following associated social
cues. For example, people may carry forward their own existing gender biases
when interacting with an agent [205], or may expect politeness conventions to
be upheld under certain contexts [33, 58]. The CASA paradigm has thereby
encouraged people to develop computers that either: communicate using
favourable human traits (such as a tone-aware chatbot [144]), or exploit human
biases (such as a chatbot that mirrors a user’s race [186]) to improve the
effectiveness of interactions.

This desire to create more effective (such as motivational or appropriate)
computer-delivered communication is a primary motivation behind our studies.
Specifically, we want to create computer-delivered health messaging that is
perceived more positively by people, so as to better help them meet their
health needs. We focus on text-based computer-delivered verbal cues used in
messaging both at the conception stage (with the generation and curation of
efficacious messages), and the end-user stage with empirical studies to
investigate verbal cues that make messaging more effective. We focus on verbal
cues in particular as the content and style of conversational messages is a
primary driving force behind user perceptions and behaviour in response to
computer-delivered health messaging. On from this, in our studies we
investigate both the conversation content (what is said), as well as the
conversational style (how it is said).

###  1.2 Research Questions

This dissertation aims to improve the effectiveness of verbal cues in text-
based computer-delivered health messaging. We believe that messaging can be
improved: (1) in the way it is generated; (2) by adopting an appropriate
conversational style for health communication; and (3) by altering the
presence and format of a chatbot’s references to previous user utterances. The
high-level research question of this dissertation is therefore:

> How can we make text-based computer-delivered verbals cues in health
> messaging more effective?

Below we will outline our three sets of studies in more detail, alongside
their high-level research questions.

####  1.2.1 Generating Diverse and Efficacious Health Messages

Our first study investigates how text-based health messages can be generated
that are both efficacious and diverse. By generating diverse text-based health
messages, we collect diverse verbal cues: i.e., conversational content and
conversational styles of messages. One method to generate large amounts of
health content is to use collections of people hereby referred to as “the
crowd” [79]. Previous work has studied how to use the crowd to generate health
content such as motivational messages [79], exercise plans [6], and solutions
to back pain [141]. Using the crowd is low cost (compared to using experts),
and can generate personable and relatable content by drawing on people’s
personal experiences [66]. The ability to produce large quantities of diverse
and efficacious health content is particularly useful for interventional
messaging or chatbots, where large numbers of messages are needed both to
avoid user boredom in repeated interventions [168, 42, 293], and in order to
train algorithms for health chatbot systems [337].

However, the crowd may produce similar or identical ideas, that leads to
redundancy and wasted human effort [16, 28]. Previous work within
crowdsourcing and creativity have shown example ideas to the crowd to help
them generate diverse ideas [279, 145], but these systems often rely on manual
human effort to run. Therefore, we want to use the crowd to generate diverse
and efficacious health messages, while also using a scalable and more
automatic system. This leads us to our first research question:

> RQ1: How can we use the crowd to automatically and scalably generate diverse
> verbal cues in health messaging?

To help answer this we developed Directed Diversity [74], a method that
harnesses embedding distances to scalably and automatically allocate diverse
example ideas to crowd-workers. By giving diverse examples to workers, we
generate a collectively diverse corpus of messages. In our user studies we
showed users short phrases related to physical exercise (e.g., “hard workout
may feel”) to inspire workers to write motivational messages to encourage
exercise. To automate our process, we used existing online health content to
generate the examples shown to workers, and developed an algorithm to select
semantically diverse examples to show to workers. To measure the efficacy of
our crowd-generated messages, we asked people to hypothetically answer how
they themselves would find the message if it was a notification on their
phone. We also used human evaluations and objective measures to evaluate the
diversity of messages. It was found that our method helped the crowd generate
diverse content when compared to both randomly selected examples and no
examples, while still generating efficacious motivational messages. This study
is described in detail in Chapter 3.

####  1.2.2 Choosing an Appropriate Conversational Style for a Health Chatbot

In our second study [72], we investigated the effect of a health chatbot’s
conversational style on a user’s quality of information disclosure. In order
to help people improve their health, chatbots need to be given enough
information. However, people may be unwilling to disclose their information if
a chatbot is perceived negatively [236]. Previous work has investigated the
effect of conversational style on quality of multiple-choice survey responses,
both with a standard survey and with a survey chatbot [161, 45, 322]. However,
we are interested in the effect of conversational style on the quality of
disclosure to a health chatbot, in terms of likelihood to disclose and quality
of user utterances. This led us to our second research question:

> RQ2: How does the conversational style (i.e., language formality) of a
> health chatbot affect the quality of a user’s self-disclosure?

On from this, we investigated the effect of a health chatbot’s conversational
style on the quality of responses from users [72]. We conducted two user
studies to investigate the impact of formal and casual conversational styles
on human perceptions (such as likelihood to disclose, and perceptions of the
chatbot) and the quality of responses given to the chatbot. In our studies
users perceived a formal conversational style to be more appropriate and
competent, and a formal conversational style was found to produce higher
quality user utterances. This study is described in detail in Chapter 4.

####  1.2.3 Choosing an effective format for a health chatbot to reference a
user’s previous utterances

Health interviewers have been found to be effective both by adopting higher
levels of social presence [310, 332] (such as paraphrasing a user’s
utterances) or by adopting lower levels of social presence [268, 59, 221]
(such as by providing high-level references to user utterances). These two
conflicting schools of thought lead us to investigate the effect of varying
levels of a health chatbot’s social presence by changing the format used when
referencing a user’s previous utterances. This leads us to our third research
question:

> RQ3: How does the format used when a health chatbot references a user’s
> utterances from previous chatting sessions impact a user’s perception of a
> chatbot and why?

As a result of this, in our third study [71], we conducted a between-subjects
longitudinal study where we compared three different formats that a chatbot
could use when referencing user utterances from previous chatting sessions. We
found that a format that referenced users’ past utterances explicitly made the
chatbot seem more intelligent and engaging, but also raised privacy concerns.
We then conducted semi-structured interviews to further investigate why people
viewed the referencing formats differently. This study is described in Chapter
5.

###  1.3 Thesis Outline

In this thesis, I will outline background literature, as well as the three
sets of studies that we have completed. The structure of this report is
outlined below:

##### Chapter 2: Background Research and Related Work

This chapter investigates the topics related to the project and outlines any
relevant research. In our related work chapter, we contextualise our three
studies by providing a background overview of health communication technology.
As we are interested in text-based health messaging, we first start by giving
an overview of health message generation and curation. We then describe
previous work that has attempted to improve health messaging technology, and
more specifically are interested in text-based communication such as nudges or
conversations to encourage healthy behaviours.

##### Chapter 3: Generating Diverse and Efficacious Health Messages

Our first studies in Chapter 3 draw on the crowd to generate diverse and
efficacious health messages, that could be used as interventions or to train
chatbots. For this, we developed and evaluated a method to scalably and
automatically generate more diverse and efficacious health messages by using
embedding distances to send crowd-workers collectively diverse phrases to
inspire their ideation [74]. This study is published as:

cox2021diverse.

##### Chapter 4: The Effects of a Chatbot’s Language Formality on a Users’
Information Sharing

Our studies in Chapter 4 draw a connection between health chatbots and their
users. More specifically, we investigated the effect of a health chatbot’s
conversational style, and found that a formal conversational style may lead to
higher quality user responses [72]. This study is published as:

cox2022does.

##### Chapter 5: The Effect of how a Health Chatbot Formats and References a
User’s Previous Utterances

Finally, our study in Chapter 5 investigated the effect of varying how a
chatbot refers to user utterances from a previous chatting session, and we
found that, while people think more positively of chatbots that explicitly
reference their previous utterances, they also find them more privacy
violating. This study is published as:

cox2023referencing.

##### Chapter 6: Conclusion and Future Work

In Chapter 6, we offer concluding remarks, we reflect on our findings and
implications, and offer perspective about future work.

##  Chapter 2 Background Research and Related Work

In this chapter, we will outline the scope of literature in relation to the
research that we have undertaken. We will provide an overview of literature
from the message generation process, to the content of messages themselves,
and to the appropriate timing for messaging to take place.

###  2.1 How should messaging be generated?

Before any health messaging can take place, we need either a bank of existing
health messages or an appropriate method to create health messages. Messages
could be in the form of one time interventions to encourage a healthy
behaviour (such as SMS messages or app notifications [66, 254, 184]); or
messages that emulate a conversation with another human (such as chatbot
utterances [239, 186]). These messages can be generated by humans [66, 141,
172, 6, 5] or by machines through natural language generation (NLG) [189, 191,
257, 179, 35].

####  2.1.1 Machine-generated messages

Machine-generated messages have been used to generate empathic responses such
as for customer service [144] and general chitchat [191, 189]. While NLG does
not rely on people to write messages directly, it requires a large body of
training data [257], human feedback (such as subjective human ratings for
reinforcement learning from human feedback used in commercial LLMs [347, 62,
352]), and the reliability of messages (such as appropriateness,
grammaticality, tone and accuracy) may be difficult to ensure [285, 179].

However, with the rapid rise and advancement of large language models (LLMs)
[347], chatbots have become increasingly capable in responding to user queries
and can be successfully applied to a variety of creative tasks and problem
solving [35, 302, 73], such as providing medical advice [343, 60]. Yet (at the
time of writing), while these LLMs (such as GPT-4 [35], LaMDA [307], and more
[347]) are able to adapt to a variety of user tasks, “hallucinated” responses
may be generated whereby the LLM generates often plausible sounding but
factually incorrect statements [35]. Cases of AI hallucination within
healthcare are already documented [35, 179, 302, 60, 150], such as LLMs
misunderstanding medical vocabulary or providing advice that does not follow
medical guidelines [60, 150]. For example, Lee et al. described the use of
GPT-4 to act as a chatbot for medicine, whereby the chatbot claimed to have
and a master’s degree and family members suffering from diabetes [179] (see
also Figure 1.8 p. 12 [35] for two examples of GPT-4 hallucination in the
healthcare domain).

These potential inaccuracies put the user at risk if incorrect health advice
is given, and lower the acceptability of AI-driven health chatbots [219] or
health interventions [273]. Nadarzynski et al. conducted in person interviews
and an online survey to assess the acceptability of AI-driven health chatbots
[219]. Interviews found that participants may be hesitant to adopt healthcare
chatbots as there was a perceived risk due to miscommunication between the
chatbot and its users, with some participants describing their concerns that
users may not be able to describe their health issues to a chatbot in a way
which the chatbot can understand. Additionally, some interviewees perceived a
risk of harm to users if inadequate or inaccurate information was provided by
a chatbot, and expressed concern that AI-driven chatbots may not understand
the emotions of users. Yet, as limitations (such as “hallucinations”) are
inevitably overcome the adoption of LLMs to healthcare is a promising and
exciting development, and (with ongoing positive mainstream news) acceptance
of healthcare chatbots may become more commonplace.

####  2.1.2 Human-generated messages

Human-generated messages can either be generated by experts (such as health
experts or a research team), or by a collection of non-experts which will we
refer to as “the crowd” [80]. Expert-generated health messages have been used
in studies, such as for interventions [184, 254] or retrieval-based chatbot
scripts [239]. This content has the reliability of an expert, and may be
grounded in health theory. However, it is expensive to generate using experts
and not easily scalable.

###  2.2 Crowd-generated messages: Can the crowd perform as well as experts?

In contrast, the crowd can be used to generate large quantities of content
more scalably (although caution needs to be made to ensure the quality of
content generated [162]). The crowd could consist of a collection of peers who
suffer from similar health issues [66, 141, 172, 6], or a collection of
individuals who are paid to complete a task online (such as over the platforms
Amazon Mechanical Turk or Prolific) [5, 79]. Past studies have used the crowd
to generate generate utterances for chatbots [342, 154, 332], and motivational
messages to encourage healthy behaviours [79, 66, 168, 172, 78].

To compare these two methods of human generation, past studies have compared
the efficacy and diversity of crowd and expert-generated health content [66,
172, 80, 5, 6]. Coley et al. asked both experts and smokers (the crowd) to
write smoking cessation messages [66], and found that crowd-generated messages
were more diverse than those written by experts. They hypothesised that this
is due to the crowd incorporating more diverse past experiences into their
writing. Kristan et al. asked both experts and young adults (the crowd) to
write messages to reduce alcohol consumption [172]. Young adults then rated
messages as more motivating if they were peer-generated by the crowd, leading
Kristan et al. to posit that messages written by peers more “authentic” and
less “manipulative” than those from experts. De Vries et al. used both experts
and the crowd to generate messages to motivate physical activity, and found
that (for certain users) motivational messages written by the crowd are as
motivating as those written by experts [80].

###  2.3 Downsides of the crowd, and how to overcome them

While these studies may suggest that the crowd should be used in lieu of
experts, importance needs to be placed on designing crowdsourcing systems to
avoid low quality input through in-built quality control [162]. Past studies
have used fault-tolerant sub-tasks [20, 163, 225] and applied agreement
filters and peer-review [20, 163, 320, 90] to counteract this.

Giving feedback to the crowd can be used to improve the quality of generated
content [90]. Mamykina et al. suggest showing expert and peer generated
feedback, or showing answers from other users to improve crowdsourcing [200].
Agapie et al. found that exercise plans crowdsourced using scaffolding were in
some cases as (or more) effective as those produced by experts [5]. Using
multiple design iterations within crowdsourcing has been investigated in a
number of studies [91, 193]. “Serial” iteration can be used to repeatedly
refine a piece of content. However this can lead to reaching a local rather
than global optimal solution [40]. Alternatively, refining multiple
alternatives in “parallel” iterations has been shown to increase content
diversity [91], and generated more authentic and diverse feedback from
individuals [308]. As well as this, previous work has investigated the
effectiveness of providing live feedback to crowdworkers [90, 180]. Lee et al.
generated design guidelines to assist in the running of crowdsourcing studies
featuring live-feedback to collaborate [180].

Additionally, the crowd can produce similar or identical ideas, which leads to
redundancy and wasted human effort [16, 279, 28, 165, 261]. This causes
additional issues with regards to health messaging. Since behaviour change is
typically a long-term process, messaging needs to be repeated over a long
period of time [67]. However, seeing the same or similar messages repeatedly
can lead to boredom, annoyance, content blindness or even avoidance, resulting
in poor adherence and dropouts [42, 293]. Hence, researchers have called for
the need to diversify messaging [168, 242] and have offered practical methods
to create diversity, such as Kocielnik and Hsieh’s [168] use of strategies
based on cognitive space. Kocielnik and Hsieh [168] used writing prompts to
produce diverse messages to encourage physical exercise based on target-
diverse (cognitively close to target action) and self-diverse (cognitively
close to message recipient) messages. They found cognitively diverse messages,
maintained the same quality and meaning, but did not encounter the same
negative effects of repeated exposure. Additionally within creativity
literature methods have been proposed which give crowd-workers examples to
help generate diverse content [279, 145, 115]. For example, Siangliulue et al.
asked participants to manually categorise ideas as they generated them, which
allowed their system to show participants example ideas that were different to
their previously generated ideas [279]. BlueSky used the crowd to construct
taxonomies to constrain sub-topics for ideation [145].

However, these methods to diversify content generated by the crowd are not
fully-automated and rely on manual effort to label idea categories, or are
restricted to a discrete number of idea categories. Therefore we developed
Directed Diversity [74], a method which scalably and automatically allocates
diverse prompts to crowd-workers. Our method extracted a corpus of example
ideas, and used embedding distances to select diverse examples for prompting
ideation. Our study is described in detail in Chapter 3.

###  2.4 How should messages be formatted and stylised?

Now that we’ve described how messaging can be generated, we will discuss the
content of this messaging. From Section 2.2 we have heard how the written
style of messaging may vary between experts and the crowd [66, 80], with
crowd-generated content perhaps using less technical and more everyday
language [66, 172, 80]. Additionally, we have discussed how previous work
aimed to generate health messaging that adopted a semantically diverse range
of content [168]. However, it may not always be clear what style or message
content would be appropriate for a given situation. The range of message
content could be vast such as messaging to encourage a particular behaviour
(e.g., reminders [289], action plans [6], implementation intentions [214],
recommendations [254], or acts of deception [70]), reflective messaging [169],
and motivational messages [79, 214], while conversational styles can also vary
greatly by changing the formality [135, 72], readability, verbosity [8, 86],
and sentiment [324] of language.

A useful paradigm to call upon is the computers as social actors (CASA)
paradigm [220]. CASA posits that people are likely to apply similar social
norms to computer interactions, as they do for human interactions. This has
led to a desire to create computer messaging that exhibits favourable human-
like qualities such as empathy [190], listening [332], differing
conversational styles and tone [194], and adopting favourable politeness
strategies [33, 151] or behaviour change techniques [210, 314].

Interventional messaging (via smartphone notifications and chatbots) has been
used to increase physical activity [120, 168, 254]. Gouveia et al. developed
Habito [120] to send people persuasive (e.g., “try walking longer while on the
phone”) and informative (e.g., “you’ve burned 200 calories by walking today”)
messages. They found that participants receiving persuasive messages were more
likely to start walking and to walk further. Additionally, there have been HCI
studies to encourage reflection on past physical activity to produce insights
or encourage change [301, 169, 183, 178]. Lee et al. found that personalised
prompts can be delivered to individuals to motivate them to exercise more
[178]. Similarly to this, Kocielnik et al. developed Reflection Companion a
conversational agent to assist people in reflecting on their physical activity
[169]. They found that delivering reflection prompts through a conversational
agent increased motivation, empowerment, and adoption of new behaviours during
a 2-week field study.

Recently, chatbots have been used in the healthcare domain. Chatbots offer
scalability, while having the potential for the flexibility and personability
advantages of interviews with health experts [245]. Commercially available
chatbots include Vik, which disseminates information for breast cancer support
and can give medication reminders [47, 22]. Bibault et al. found that messages
from Vik were as informative and helpful as those from physicians [22].
Tschanz et al. propose a chat-like smartphone app (eMMA) to manage patient
medication through a chatbot [311] focused on patient interactions and
medication management. Chatbots have also been used for mental health support
[186, 239, 1, 249]. For example, Park et al. found a chatbot using
motivational interviewing techniques can be helpful to encourage reflection
and stress reduction [239]. On from this, physicians agree that chatbots can
help in most automatic simple tasks in healthcare scenarios such as
disseminating information or arranging appointments [236]. Additionally, the
increased anonymity of chatbots can lead to greater disclosure and openness
among users [245].

Previous work has also investigated the conversational style of chatbots [72,
45, 332, 161]. A chatbot using a casual conversational style was found to
improve the quality of course evaluations when compared to a standard survey
[322]. Kim et al. surveyed adolescents about their Internet usage (via
multiple-choice questions), and compared a chatbot using casual language
against a chatbot using language taken from a standardised survey[161]. People
found a casual conversational style more engaging, and the responses were
found to be equally efficacious between both conditions. Similarly, Celino et
al. found people preferred a casual survey and it showed same level of
reliability and increased level of response quality (reduced satisficing)
compared to traditional multiple-choice surveys [45]. Kocielnik et al. [167]
compared a chatbot against a standard survey for health screening and found
that users perceived a chatbot as more engaging. Xiao et al. found that an AI-
driven chatbot produced better quality open text responses and high levels of
participat engagement compared to a standard survey [332].

These studies investigated the use of a chatbot as a data collection tool.
This was either by comparing multiple-choice responses given to a chatbot
using a casual conversational style against a standard survey [161, 45, 322],
or by comparing the quality of open-text responses given to a chatbot compared
to those elicited by a standard survey [332]. Yet the baseline conditions
using standarised surveys are often written in concise language that may not
mirror more conversational formal language. Therefore, in Chapter 4 we compare
the use of casual and formal conversational styles of a health chatbot, and
assess its effect on the both the quality of a user’s open-text responses, and
on a user’s perceptions [72]. Furthermore, to investigate how to improve user
perceptions further, in Chapter 5 we describe an experiment where we vary the
presence and format of references to previous user utterances and measure the
impact on a user’s privacy concerns and perceptions (such as intelligence and
engagement) of the chatbot [71].

###  2.5 How should messaging be selected?

Next, we will discuss how messages can be selected. Messages can be selected
manually by human users (such as a research team or health experts) or
algorithmically by a computer. However, while messages can be chosen manually
based on their expected effectiveness, this is not scalable, is labour
intensive, and could be skewed by human biases. Due to these issues of
scalability, we will therefore focus on literature describing algorithmic
message selection techniques.

####  2.5.1 Selecting messages algorithmically

Messages can be chosen algorithmically either: (1) for ongoing turn-taking
conversations between a human and a chatbot or (2) for one time interventions
to nudge a user. Messages selected for turn-taking conversations are typically
based on a dataset of messages, by selecting the message that is most likely
to follow on from the previous messages in a conversation. This could then
lead to a retrieval based method (where an existing message is used verbatim)
or generative methods as described in Section 2.1.

Messages selected for one-time interventions can be chosen based on past
rewarding behaviour. For example, by looking at previous outcomes, we could
choose messages which have led to the biggest increase in a desired behaviour,
or messages which have been most likely to result in a desired behaviour.
Within healthcare, studies have demonstrated the usefulness of using
reinforcement learning techniques to select messaging. Rabbi et al. developed
MyBehavior [254] to use multi-armed bandits to exploit messages intended to
maximise calorie loss and physical activity. Similarly, Yom-Tov et al.
developed a system that used a bandit-style algorithm to send messages to
patients who are following a personalised plan for physical exercise [339].

###  2.6 When should messages be sent to users?

Message selection as described in Section 2.5.1 can also be based on a user’s
context. For example, messages could be selected using data such as a user’s
location or mood, a user’s demographics, the time of day, or meta-data
collected by a device such as applications used on a smartphone [185, 335].

Micro-randomised trials can be used to select the optimal intervention for a
given user based on contextual data and prior intervention success. DIAMANTE
[335] delivered motivational messages to increase physical activity, and used
reinforcement learning to adaptively display more effectively performing
messages more often. They experimented with different message factors (such as
behaviour change technique deployed and message timing) to find and select the
most effective messages for different contexts.

Liao et al. [185] selected notifications based on a user’s context (such as
location and time of day). Petovic et al. [243] developed an interruptibility
model to infer the most opportune time to send a notification based on a
user’s past behaviour. Kunzler et al. [173] investigated how different factors
influence the receptivity of health interventions by considering intrinsic
factors (i.e., device type, age, gender, and personality) and contextual
factors (i.e., GPS location, physical activity, date/time, Battery status,
Lock/Unlock events, Screen on/of events, Wi-Fi connection state and Proximity
sensor).

###  2.7 Summary of Related Work

This preceding related work section outlined the use of computer-delivered
health messaging. In order to contextualise our upcoming chapters, we gave an
overview of health messaging, beginning first with the source of the messages
(be it human- or machine-generated), before discussing the potential content
type and style of messages, and finally discussing the medium of message
delivery. As discussed above, this medium of delivery can either be computer-
delivered interventional messages (such as those investigated further in
Chapter 3 where we investigated message generation) or two-way conversations
between a human and an agent (such as the studies in Chapters 4 and 5 where we
investigated the conversational style of health chatbots). In the following
chapters, we describe empirical studies that we conducted in order to further
knowledge in these areas.

##  Chapter 3 Generating Diverse and Efficacious Health Messages

In Chapters 1 and 2, we gave an overview of the thesis and highlighted
relevant literature. In doing so, we discussed the importance of generating a
diverse range of non-repetitive health messages. Diverse messages reduce
negative effects such as perceived boredom and annoyance and increase positive
effects such as perceived informativeness and helpfulness [168, 112]. In this
chapter, we describe a method to generate diverse verbal cues or more
specifically, conversational content (in this case motivational messages to
encourage physical activity). Our method, Directed Diversity, uses language
embedding distances to allocate a collectively diverse range of prompts to
users to aid in the generation of diverse motivational messages. We ran a
series of user studies to generate and validate messages generated using our
method. Our method produced more diverse messages, whilst also maintaining or
improving more abstract human concepts such as motivation, helpfulness and
informativeness. Abstract concepts such as these are useful in creating a more
effective human-like chatbot. This chapter is based on one publication:

cox2021diverse.

###  3.1 Introduction

Crowdsourcing has been used to harness the power of human creativity at scale
to perform creative work such as text editing [20, 64, 276], iterating designs
[91], information synthesis [198], and motivational messaging [168, 66, 79].
In such tasks, empowering crowd workers to ideate effectively and creatively
is key to achieving high-quality results. Different prompting techniques have
been proposed to stimulate creativity and improve the diversity of ideas [5,
91, 168, 79], but they suffer from ideation redundancy, where multiple users
express identical or similar ideas [28, 165, 261, 279]. Current efforts to
avoid redundancy include iterative or adaptive task workflows [341],
constructing a taxonomy of the idea space [145], and visualizing a concept map
of peer ideas [279], but these require much manual effort and are not
scalable.

Instead, we propose an automatic prompt selection mechanism — Directed
Diversity — to scale crowd ideation diversity. Directed Diversity composes
prompts with one or more phrases to stimulate ideation. It helps to direct
workers towards new ideas and away from existing ideas with the workflow: 1)
extract phrases from text corpuses in a target domain, 2) embed phrases into a
vector embedding, and 3) automatically select phrases for maximum diversity.
These phrases are then shown as prompts to ideators to stimulate ideation. The
phrase embedding uses the Universal Sentence Encoder (USE) [46] to position
phrases within an embedding vector space. Using the embedding vectors, we
calculated distances between phrases to optimally select phrases that are
farthest apart from one another; this maximizes the diversity of the selected
phrases. Hence, Directed Diversity guides ideators towards under-utilized
phrases or away from existing or undesirable phrases.

The embedding space provides a basis to calculate quantitative, distance-based
metrics to estimate diversity in selected phrases and prompts, and
subsequently ideated messages. These metrics can complement empirical
measurements from user studies to evaluate prompts and ideations. We curate
multiple measures and evaluation techniques and propose a Diversity Prompting
Evaluation Framework to evaluate perceived and subjective creativity and
objective, computed creativity, and diversity of crowd ideations. We
demonstrate the framework with experiments on Directed Diversity to 1)
evaluate its efficacy to select diverse prompts in a simulation study, 2)
measure the perceived diversity of selected prompts and effort to generate
ideas in an ideation study, and 3) evaluate the creativity and diversity of
generated ideas in validation studies using quantitative and qualitative
analyses. The experiments were conducted with the use case of writing
motivational messages to encourage physical activity [5, 6, 168, 79], though
we discuss how Directed Diversity can apply to other crowd ideation tasks. In
summary, our contributions are:

  1. 1.

We present Directed Diversity, a corpus-driven, automatic approach that
leverages embedding distances based on a language model to select diverse
phrases by maximizing a diversity metric. Using these constrained prompts,
crowdworkers are directed to generate more diverse ideas. This results in
improved collective creativity and reduced redundancy.

  2. 2.

A Diversity Prompting Evaluation Framework to evaluate the efficacy of
diversity prompting along an ideation chain. This draws constructs from
creativity and diversity literature, metrics computed from a language model
embedding, and is validated with statistical and qualitative analyses.

  3. 3.

We applied the evaluation framework in a series of four experiments to
evaluate Directed Diversity for prompt selection, and found that it can
improve ideation diversity without compromising ideation quality, but at a
cost of higher user effort.

###  3.2 Background and Related Work

We discuss related research on supporting crowd ideation with the cognitive
basis for creative ideation, how creativity support tools help crowd ideation,
and how artificial intelligence can help collective intelligence.

####  3.2.1 Cognitive Psychology of Creative Ideation

Different cognitive models of creativity have been proposed to explain how
ideation works. Memory-based explanation models describe how people retrieve
information relevant to a cue (prompt) from long-term memory and process it
generate ideas [4, 89, 197, 224, 223]. Since retrieval is dependent on
prompts, they need to be sufficiently diverse to stimulate diverse ideation
[223], otherwise people may fixate on a few narrow ideas [149]. Ideation-based
models [81] explain how individuals can generate many ideas through complex
thinking processes, including analogical reasoning [122, 10, 208], problem
constraining [286], and vertical or lateral thinking [117]. We focus on
prompting to promote memory-based retrieval than these other reasoning
processes. Besides cue-based retrieval and thinking strategies, other factors
influence ideation creativity, such as personal traits, motivation to perform
the task, and domain-relevant skills that can affect individual creativity
[299]. We provide technological support to improve the creative mental
process, rather than to select creative personalities, recruit domain experts,
or improve task motivation. Next, we discuss how different cognitive factors
have been leveraged at scale to support creative ideation with the crowd.

####  3.2.2 Creativity Support Tools for Crowd Ideation

Creativity Support Tools have been widely studied in HCI to enable
crowdworkers to ideate more effectively and at scale [109, 110]. Showing
workers ideas from their peers has been very popular [114, 278, 279], but can
have limited benefit to creativity if peer ideas are too distant from the
ideators’ own ideas [51]. Other approaches include employing contextual
framing to prompt ideators to imagine playing a role for the task [232] or
using avatars for virtual interactions while brainstorming [201]. While these
methods focus on augmenting individual creativity, they do not coordinate the
crowd, so multiple new ideations may be redundant. More recent approaches
apply provide more explicit guidance to workers. IdeaHound [279] visualizes an
idea map to encourage workers to focus on gaps between peer ideas, but does
not inform what ideas or topics will fill the gaps. BlueSky [145] and de Vries
et al. [79] use crowd or expert annotators to construct taxonomies to
constrain the sub-topics for ideation, but these taxonomies require
significant manual effort to construct and are difficult to scale. Chan et al.
[49] employed latent Dirichlet allocation (LDA) to automatically identify
topics, but this still requires much manual curation which does not scale to
many topics. With Directed Diversity, we automatically extract a phrase corpus
and embed the phrases as vectors, and select diverse phrases for focused
prompting. We employ pre-trained language model to provide crowd ideation
support, thus we next discuss how artificial intelligence can support
collective intelligence.

####  3.2.3 Supporting Collective Intelligence with Artificial Intelligence

Collective Intelligence is defined as groups of individuals (the collective)
working together exhibiting characteristics such as learning, judgement and
problem solving (intelligence) [199]. Crowdsourcing is a form of collective
intelligence exhibited when crowdworkers work towards a task mediated by the
crowdsourcing platform. However, managing crowdwork to ensure data quality and
maximize efficiency is difficult because of the nature and volume of the
tasks, and varying abilities and skills of workers [326]. HCI research has
contributed much towards this with interfaces to improve crowdworker
efficiency, designing incentives for workers, and workflows to validate work
quality [25, 132, 209, 296, 326]. Furthermore, recent developments in
artificial intelligence (AI) provides opportunities to complement human
intelligence to improve the quality and efficiency of crowd work [326, 162],
optimize task allocation [76, 100], adhere to budget constraints [158], and
dynamically control quality [30]. With Directed Diversity, we used AI to
optimize ideation diversity by shepherding the crowd towards more desired and
diverse ideation with diverse prompt selection.

###  3.3 Technical Approach

We aim to improve the collective diversity of crowdsourced ideas by presenting
crowdworker ideators, with carefully selected prompts that direct them towards
newer ideas and away from existing ones. The prompts presented to the ideators
consist of one or more phrases that represent ideas that are distinct and
different from prior ideas. Prompts can have one or more phrases. As a running
example throughout the technical discussion and experiments, we apply our
approach to the application of motivational messages for healthy physical
activity, where it is important to collect diverse motivational messages [168,
79]. Figure 3.1 shows the 3-step overall approach to extract, embed, and
select phrases. We next describe each of these steps in detail.

Figure 3.1: Pipeline of the overall technical approach to extract, embed, and
select phrases to generate diverse prompts. a) Phrase extraction by collecting
phrases from online articles and discussion forums (shown as pages), filtering
phrases to select a clean subset (shown as the black dash for each phrase); b)
Phrase embedding using the Universal Sentence Encoder [46] to compute the
embedding vector of each phrase (shown as scatter plot); c) Phrase Selection
by constructing the minimal spanning tree to select optimally spaced phrases
(see Figure 3.3 for more details).

####  3.3.1 Phrase Extraction

##### Phrase Selection

We want to show people phrases relevant to exercise and motivation to inspire
them to write new motivational messages. To automatically generate a large and
diverse set of relevant phrases, we followed a data-driven approach by
extracting phrases from a large corpus of documents about exercising, weight
loss and healthy living. We created our corpus using two main sources:
informative and credibly sourced health news articles, and diverse online
health communities (OHCs).

First, we collected articles from online news websites focusing on health-
related topics. Such articles are written in well-formed, informative, as well
as neutral and impersonal language. Using relevant keywords as search terms,
we identified the most popular health news websites. We manually inspected
their content and selected 3 websites based on their large number of articles
and health-related categories, and their policies of specifying sources of
health information: www.health.harvard.edu, www.medicinenet.com,
www.webmd.com. Note that we limited ourselves to categories such as exercise,
nutrition and healthy habits, and omitted categories such as pregnancy or
drugs and supplements.

In contrast to neutral and impersonal news articles, we collected user-
generated posts submitted to OHCs on the popular content aggregation and
discussion website Reddit (www.reddit.com). In general, user posts feature a
more emotional tone, stronger sentiments and personal anecdotes. The content
on Reddit is organized into “subreddits” that allow for organizing content and
discussion around a general topic. For our corpus, we identified subreddits
related to physical activity or motivation based on their popularity in terms
of the number of submitted posts. We focused on subreddits where the majority
of posts are user-written content in contrast to images, videos, or links to
external content. Given these criteria we collected the posts from 20 relevant
subreddits (90daysgoal, advancedfitness, advancedrunning, bodyweightfitness,
c25k, crossfit, fitness, gainit, getmotivated, ketogains, kettlebells,
leangains, loseit, motivation, powerlifting, running, selfimprovement,
swimming, weightroom, xxfitness). Previous work has shown that OHCs with
similar health focuses can differ in their behaviour change goals [52], and by
using multiple subreddits we aim to draw on more collective wisdom and
diversity.

Overall, our final corpus consists of 3,235 articles and 32,721 user posts.

##### Phrase Cleaning

To extract all relevant phrases from this corpus, we first split each article
and post into sentences and perform Part-of-Speech (POS) tagging using an off-
the-shelf text processing tool. We use the grammatical sequence of POS tags to
identify and extract phrases as sequence of words forming a syntactic
constituent in a sentence — in contrast to simple n-grams that might contain
sequences of words of overlapping syntactic constituents. From each sentence,
e.g., “Regular exercising helps to improve people’s health at any age.”, we
extract verb phrases (e.g., “helps to improve”), noun phrases (e.g., “regular
exercising”, “people’s health”, “age”) and prepositional phrases (e.g., “at
any age”). To further expand our set of phrases, we also combine adjoining
verb and noun phrases to generate noun-verb phrases (e.g. “regular exercising
helps to improve”) and verb-noun phrases (e.g., “helps to improve people’s
health”). Despite limiting ourselves to health-related articles and user-
generated content, not all sentences and let alone all phrases in themselves
are related to this topic. For the final set of phrases to use as potential
prompts for workers, we apply the following filter criteria to all phrases.

  1. 1.

We remove all phrases that contain uncommon words not found a dictionary. This
is particularly useful for user posts that often contain typos, slang or other
stylistic devices (e.g., emoticons or expressive lengthenings).

  2. 2.

We remove all phrases with less than 3 or more than 5 words, since short
phrases may not sufficiently inspire and longer phrases may lead to fixation.

  3. 3.

We remove all phrases with no word related to exercising, nutrition or health.
We use WordNet lexical database [213] to check for each word in a phrase, if
the word or any of its hypernyms is within a set of manually selected
concepts. For example, the word treadmill has the hypernym exercise device,
which makes it a relevant word for our phrase filter.

  4. 4.

We remove all sub-phrases and overlapped phrases (e.g., exclude “federal
exercise recommendations” and keep “federal exercise guidelines rose”).

The application of all filter steps, resulted in 3,666 phrases that may serve
as prompts for our user study.

####  3.3.2 Phrase Embedding

The corpus of extracted phrases provides a large set of potential phrases for
prompting, but we seek to select phrases that are least similar to one
another. For each phrase, we obtain a multi-dimensional vector representation,
called an embedding, so that the phrase is a data point in an idea space.
Similar work by Siangliulue et al. [278] obtained embeddings of $N=52$ ideas
by training a Crowd Kernel model [300] from 2,818 triplet annotations is not
scalable to our corpus of $N=$ 3,666 phrases, since that would need
$N(N-1)(N-2)/3=16.4$ million triplets. Instead, similar to Chan et al.’s [51]
use of GloVE [244], we use pre-trained language models based on deep learning
to encode each word or sentence as a vector representation. Specifically, we
use the more recent Universal Sentence Encoder (USE) [46] to obtain embeddings
for phrases in our corpus, compute their pairwise distances, and selected a
maximally diverse subset of phrases. Our approach is generalizable to other
language embedding techniques [340].

Figure 3.2: Demonstration of pairwise embedding angular distances between an
example text items (first data row) and neighboring text items. Text items
with semantically similar words have smaller distances. For interpretability,
we highlighted words to indicate darker color with higher cosine similarity to
the first phrase.

To obtain the phrase embedding presentation, we use a pre-trained USE
model111Pre-trained Universal Sentence Encoder model
(https://tfhub.dev/google/universal-sentence-encoder/4) , which was trained
using both unsupervised learning on Wikipedia, web news, web question-answer
pages, and discussion forums, and supervised learning on Stanford Natural
Language Inference (SNLI) corpus. to obtain embedding vectors for each phrase.
With USE, all embeddings are 512-dimensional vectors are located on the unit
hypersphere, i.e., all vectors are unit length, and only their angles are
different. Hence, the dissimilarity between two phrase embeddings $x_{i}$ and
$x_{j}$ is calculated as the angular distance $arccos(x_{i},x_{j})$, which is
between 0 and $\pi$. For our phrase corpus, the pairwise distance between
phrases ranged from Min = 0.06 to Max = 0.58, Median=0.4, inter-quartile range
0.39 to 0.46, SD=0.043; see Appendix Figure A.4. We use the same USE model to
compute embeddings and distances for ideated messages. For a dataset of 500
motivational messages ideated in a pilot study with no prompting, the pairwise
distance between ideations ranged from Min=0.169 to Max=0.549, Median=0.405,
inter-quartile range 0.376 to 0.432, SD=0.043; see Appendix Figure A.5. Table
3.2 shows example phrases and messages and their corresponding pairwise
dissimilarity distances. With the embedding vectors and pairwise distances for
all phrases, the next step selects diverse phrases with which to prompt
ideators.

####  3.3.3 Phrase Selection

Given the embeddings of the curated phrases, we want to select the subset of
phrases with maximum diversity. Mathematically, this is the dispersion problem
or diversity maximization problem of “arranging a set of points as far away
from one another as possible”. Among several diversity formulations [54], we
choose the Remote-MST diversity formulation [129] (also called Remote-tree
[54] or functional diversity [247]) that defines diversity as the sum of edge
weights of a minimum spanning tree (MST) over a set of vertices. It is robust
against non-uniformly distributed data points (e.g., with multiple clusters,
see Table 3.7). We construct the minimum spanning tree by performing
agglomerative hierarchical clustering on the data points with single linkage
[281]. Next, we describe how we select phrases as prompts to direct ideators
towards diverse phrases, or away from prior ideas. Figure 3.3 illustrates the
technical approach.

Figure 3.3: Procedure to direct ideation towards diverse phrases (top) and
away from prior or redundant ideas (bottom). To attract ideation with diverse
prompts: a) start with embeddings of corpus-extracted phrases; b) construct
minimum spanning tree (MST); c) traverse tree to select distant prompts from
clusters (most distant points as green dots, in clustered phrases as green
ellipses); d) selected prompts are the most diverse. To repel ideation from
prior ideas, e) compute embeddings of prior ideas (red hollow dots); f)
compute prompt-ideation pairwise distances of all prompts from each prior
ideation, exclude phrases (dotted black circles) with pairwise distance less
than a user- defined threshold (red bubble), and construct the MST with
remaining phrases; g) traverse MST to select a user- defined number of
prompts; h) selected prompts are diverse, yet avoids prior ideas.

##### Directing towards Diverse Phrases

For phrase selection, we aim to select a fixed number of points n from the
corpus with maximum diversity. This is equivalent to finding a maximal edge-
weighted clique in a fully connected weighted graph, which is known to be NP-
hard [142]. Hence, we propose a scalable greedy approach that uses the
dendrogram representation of the MST resulting from the hierarchical
clustering. Starting from the root, we set the number of clusters to the
desired number of phrases $n$. For each cluster $C_{r}$, we select the phrase
that is most distant from other points, with largest minimum pairwise distance
from all points from outside the cluster, i.e.,

$x_{r}=argmax_{i\in C_{r}}(min_{j\notin C_{r}}d(x_{i},x_{j})$

where $x_{r}$ is the diverse phrase selected in cluster $C_{r}$, $x_{i}$ is a
point in cluster $C_{r}$ and $x_{j}$ is a point in the corpus not in $C_{r}$,
and d is the pairwise distance between $x_{i}$ and $x_{j}$. This method has
$O(n^{2})$ time complexity and runs in less than one second on a desktop PC
for 3.6k phrases; it is generalizable and can be substituted for other
approximate algorithms to select most diverse points [54, 146]. Figure 3.3
(top row) illustrates the phrase selection method to direct towards areas
without ideations:

> a) Start with all phrases in a corpus represented as USE embedding points.

> b) Construct a dendrogram (MST) from all points, using single-linkage
> hierarchical clustering.

> c) Set number of clusters equal to desired number of diverse phrases. For
> each cluster, find the most distant phrase.

> d) Selected phrases are the approximately most diverse from the corpus, for
> the desired number of phrases.

##### Directing Away from Prior Ideas

Other than directing ideators towards new ideas with diverse prompts, it is
important to help them to avoid prior ideas written by peers. We further
propose a method to remove corpus phrases that are close to prior ideas so
that ideators do not get prompted to write ideas similar to prior ones. The
method, illustrated in Figure 3.3 (bottom row), is similar as before, but with
some changes:

> e) Add the embedding points of prior ideas to the corpus.

> f) Calculate phrase-ideation distance $d(x_{i}^{P},x_{j}^{I})$ for each
> phrase $x_{i}^{P}$ and ideation $x_{i}^{I}$ and exclude phrases too close to
> the ideas, i.e., $d<\delta$ , where $\delta$ is an application-dependent
> threshold, $\delta=0.29$ in our case.

> g) Same as step (c), but different clusters, since fewer points are
> clustered.

> h) Same as step (d), but different prompts would be selected, even if the
> number of phrases is the same.

####  3.3.4 Directing with Prompts of Grouped Phrases

Instead of prompting with only one phrase, prompting with multiple related
terms can help ideators to better understand the concept being prompted and
generate higher quality ideas [50, 222, 283]. We extend the phrase selection
method to group multiple phrases in a single prompt using the following greedy
algorithm. After step (a), we 1) sorted phrases by descending order of minimum
pairwise distance for each phrase to produce a list of seed candidates, 2) for
each seed phrase, perform a nearest neighbors search to retrieve a specified
prompt size (number of phrases $g$ in a prompt) and remove the selected
neighbors from the seed list, 3) repeat seed neighbor selection until $n$ seed
phrases have been processed. We grouped the phrases into a prompt and
calculate its embedding point $x_{i}^{Pr}$ as the angular average of all
phrases $x_{k}^{P}$ in the prompt, i.e.,
$x_{i}^{Pr}=\Sigma_{(k=1)}^{g}x_{k}^{P}/Z$, where
$Z=||\Sigma_{(k=1)}^{g}x_{k}^{P}||_{2}$ is the magnitude of the vector sum and
$x_{i}^{Pr}$ is also a unit vector. We then perform steps (b) to (d) with the
prompts $x_{i}^{Pr}$ instead of individual phrases. Note that the corpus of
prompts will be smaller than the corpus of phrases. This approach has disjoint
prompts that do not share phrases, but there can be alternative approaches to
group phrases 222An alternative approach is, after step (c), to simply group
nearest neighbours. However, this will cause the prompt embeddings to be
shifted after the diversity is maximized, so it may reduce the diversity of
the selected prompts..

###  3.4 Diversity Prompting Evaluation Framework

To evaluate the effectiveness of the Directed Diversity prompt selection
technique to improve the collective creativity of generated ideas, we define
an ideation chain as a four step process (Figure 3.4 top): 1) setting the
prompt selection technique will influence 2) the creativity of selected
prompts (prompt creativity), 3) the ideation process of the ideators (prompt-
ideation mediation), and 4) the creativity of their ideation (ideation
creativity). We propose a Diversity Prompting Evaluation Framework, shown in
Figure 3.4, to measure and track how creative and diverse information
propagates along this ideation chain to evaluate how and whether a creativity
prompting technique improves various measures of creativity and diversity in
outcome ideas. Note that our proposed framework is descriptive to curate many
useful metrics, but not prescriptive to recommend best metrics.

####  3.4.1 Research Questions and Experiments

Prompt stimuli act along the ideation chain to increase ideation diversity,
but it is unclear how well they work and at which point along the chain they
may fail. We raise three research questions between each step in the ideation
chain, which we answer in four experiments (Section 3.5.2) with various
measures and factors.

RQ3.1 How do the prompt techniques influence the perceived diversity of
prompts? (RQ3.1a) How do they affect diversity in prompts? (RQ3.1b) How well
can users perceive differences in creativity and diversity in these prompts?
These questions relate to the prompt selection technique effectiveness and
serve as a manipulation check. We answer them in a Characterisation Simulation
Study (Section 3.5.1) with objective diversity measures, and an Ideation User
Study (Section 3.5.2) with subjective measures perceived prompt diversity
measures.

RQ3.2 How does diversity in prompts affect the ideation process for ideators?
(RQ3.2a) Do differences in diversity affect ideation effort? (RQ3.2b) How well
do ideators adopt and apply the content of the prompts? (RQ3.2c) How does
prompt creativity affect diversity in ideations? We answer these questions as
a mediation analysis in the Ideation User Study (Section 3.5.2) with objective
measures of task time and similarity between ideations and stimulus prompts,
thematically coded creativity metrics, and perceived ease of ideation.

RQ3.3 How do prompt selection techniques affect diversity in ideations? Having
validated the manipulation checks, we evaluate the effectiveness of prompt
selection techniques in questions in the Ideation User Study (Section 3.5.2)
with subjective measures self-assessed creativity and thematically coded
creativity metrics, and two Validation User Studies (Section 3.5.3) with
subjective measures of perceived creativity.

Figure 3.4: Diversity prompting evaluation framework to evaluate prompting to
support diverse ideation along the ideation chain. We pose research questions
(RQ3.1-3.3) between each step to validate the ideation diversification
process. For each step, we manipulate or measure various experiment constructs
to track how well ideators are prompted to generate creative ideas. Except for
prompt selection, each construct refers to a statistical factor determined
from factor analyses of multiple dependent variables. Constructs are grouped
in colored blocks indicating different data collection methods as follows:
(Blue - Computed embedding-based metric, Green - ratings from ideators, Red -
ratings from validators, Purple - thematic coding of ideations).

##### Independent variables of Prompt Specifications

We manipulated prompt selection technique, prompt count, and prompt size as
independent variables; these are detailed in Appendix Table A.1. We chose
Random prompt selection as a key baseline where selection is non-trivial and
data-driven based on our corpus, but not intelligently selected for diversity.

####  3.4.2 Diversity and Creativity Measures of Prompting and Ideation

We measured diversity and creativity for selected prompts and generated ideas
with embedding-based and human rated metrics. We color code variable names
based on data collection method as in Figure 3.4.

##### Embedding-based Diversity Metrics for Prompts and Ideations

Although crowd creativity research has focused on the mean pairwise distance
as a metric for idea diversity, our literature review has revealed many
definitions and metrics. Here, we describe computational metrics calculated
from the embedding-based distances. Inspired by Stirling’s general framework
diversity framework [292], we collect definitions from crowd ideation [48, 91,
145, 278, 279], ecology [84, 246, 318], recommender systems [107, 157, 351,
317], and theoretical computer science [54, 129]. These cover many aspects of
diversity to characterize the mean distance and minimum Chamfer distance
between points, MST-based dispersion, sparseness of points around the median,
span from the centroid, and entropy to indicate the evenness of points in the
embedding vector space. Table 3.5 and Table 3.6 describe distance metrics for
individual and collective text items, respectively. These metrics describe
nuances of diversity, which we illustrate with example distributions in Table
3.7. Other measures of diversity and divergence [54] can be included in the
framework, which we defer to future work. Next, we describe human-subjects
ratings to validate these embedding-based metrics with measures that do not
depend on the embeddings to avoid circular dependency.

Figure 3.5: Metrics of distances between two points in a multi-dimensional
vector space. Each metric can be calculated for an individual text item. These
metrics can apply to the embedding of phrases or ideations. Figure 3.6:
Metrics of diversity of phrases or ideation embeddings in a vector space.
These capture more characteristics of diversity than average distances in
Table 2. Each metric can only be calculated collectively for multiple items.
Figure 3.7: Comparison of diversity metrics for canonical examples of
distributed points in a 2D space. Points farther apart mean higher diversity.
Here, we calculate Euclidean instead of angular distance, but intuitions are
similar.

##### Creativity Measures for Ideations

Along with the computed diversity metrics, we evaluate with qualitative
characteristics of creativity. From creativity literature, we draw from
Torrance’s [309] description of several measures for creativity, including
quality, flexibility and originality. Quality measures whether an ideation is
“usable, practical, or appropriate” [218]. We asked ideators to self-assess on
a 5-point Likert scale their message’s effectiveness (towards motivation) and
creativity. We ask validator crowdworkers to rate each individual ideation on
a 7-point Likert scale whether it is effective (motivating [79]),
helpful333Note that a message could be helpful but written in a way that
participants do not find motivating. [168, 293], and informative [168] towards
encouraging physical activity; rank collections of ideations on effectiveness,
informativeness and unrepetitiveness.; and rate the pairwise difference
between ideation pairs from each collection. Note that Directed Diversity was
not designed to improve quality, since these metrics were not explicitly
modeled. Flexibility [287] measures how many unique ideas were generated, and
originality [345] measures how infrequently each idea occurs. These require
expert annotation to identify distinct categories. We conducted a thematic
analysis on the messages using open coding [116] to derive categories and
affinity diagramming [41] to consolidate categories to themes (see Table
LABEL:tab:directed-thematic). We calculate the flexibility and originality
measures based on the coded categories (fine-grained) and themes (coarser)
described in Appendix Table LABEL:tab:Directed-Tab7.

##### Creativity Measures for Prompts

As a manipulation check, it is important to verify that prompts that are
computed as more diverse, are perceived by ideators as more creative. Since
perceived creativity encompasses more qualitative effects, computed diversity
may not be correlated with creativity. Thus, we measure the creativity and
usefulness of prompts by asking about prompt understandability, relevance to
domain topic (physical activity), relevance to task 444Note that a prompt
could be relevant to the domain, but not motivating. (motivation), helpfulness
to inspire ideation, and unexpectedness [218] along 7-point Likert scales.

##### Mediating Variables for Prompt-Ideation Process

Even if more diverse prompts can facilitate more creative ideation, it is
important to understand whether this requires more effort and time, how the
consistency of phrases within prompts affect ideation, and how well ideators
adopt words and concepts from the phrases into their ideations. We measure
effort as ease of ideation with a 7-point Likert scale survey question. For
individual creativity, fluency [108] is defined as the number of ideas an
individual writes within a fixed time. Chan et al. had also measured fluency
for an 8-minute crowd ideation task [51]. In contrast, we asked ideators to
only write one idea per prompt without time constraint, so we measure the
inverse relation of ideation task time to generate one ideation [14].
Specifically, since task time is skewed, we use –Log(ideation time) to
represent fluency. For prompts with more than one phrase, the similarity
between phrases can affect their perceived consistency. Therefore, we measure
the intra-prompt mean phrase and prompt average phrase Chamfer distances
(Appendix Table A.2) to indicate the similarity between intra-prompt phrases.
We measure the adoption of the prompt ideas by calculating the proportion of
words from phrases in the ideations as prompt recall and prompt precision, and
computing the prompt-ideation distance between the embeddings of the prompt
and ideation (Appendix Table A.3).

####  3.4.3 Factor analyses to draw constructs from experiment variables

With the numerous variables from our experiments, we observed some may be
correlated since they measure similar notions or participants may confound
questions to have similar meanings. We employed an iterative design-analytical
process to organize and consolidate variables into factors with the following
steps.

  * •

Identify metrics of creativity and diversity from a literature review from
various research domains, such as ecology, creativity, crowdsourcing,
theoretical computer science, recommender systems (Section 3.4.2). Ideate
additional measures and questions to capture user behavior and opinions when
generating and validating ideas. We refine and reduce measures based on survey
pilots and usability testing.

  * •

Collect measurements of each metric with different methods: 1) Compute
embedding-based metrics from prompts shown and messages written. This was
computed individually for each text item (e.g., mean pairwise distance) and
collectively for all text items in each prompt technique (e.g., Remote-MST
diversity). 2) Measure perception ratings and behavioral measures regarding
reading prompts and ideating messages and rating messages. We asked text
rationale to help with interpretations. 3) Measure subjective thematic
measures to qualitatively assess the collective creativity with thematic
analysis and idea counting.

  * •

Perform factor analysis on quantitative data to organize correlated variables
into fewer factors. Variables are first grouped by data collection method
555E.g., individual text item metrics, collective text items metrics, ratings
of text item from ideators, ratings of text item from validators, ratings of
collection of text items from validators. and analyzed together. To determine
the number of factors, we examined scree plots and verified grouped variables
as consistent with constructs from literature. The final number of factors are
statistically significant by the Bartlett Test of Sphericity (all $p<.0001$).
See Appendix Tables A.7 to A.14 for the results of the factor analysis,
including factor loadings and statistical significance. Table 3.8 summarizes
the learned factors from 42 variables that we developed.

  * •

Perform statistical hypothesis testing using these learned factors to answer
our research questions.

Figure 3.8: Constructs from factor analyses of variables along ideation chain.
Factor loadings in Appendix Tables A.7 to A.14.

###  3.5 Evaluation: applying framework to study Directed Diversity

We have described a general descriptive framework for evaluating diversity
prompting. We applied it to evaluate our proposed Directed Diversity prompt
selection technique against baseline approaches (no prompting, random prompt
selection) in a series of experiments (characterisation, ideation, individual
validation, collective validation), for the use case of crowd ideating
motivational messages for physical activity. Here, we describe the procedures
for each experiment and their results.

####  3.5.1 Characterisation Simulation Study

The first study uses computational methods to rapidly and scalably evaluate
prompt selection techniques. This helps us to fine tune prompt parameters to
maximize their potential impact in later human experiments.

##### Experiment Treatments and Method

We varied three independent variables (prompt selection, prompt count, prompt
size) to measure the impact on 7 dependent variables of distance and diversity
metrics. We varied Prompt Selection technique (None, Random, or Directed) to
investigate how much Directed Diversity improves prompt diversity with respect
to baseline techniques. For None prompt selection, we simulated ideation with
500 ideas collected from a pilot study where crowd ideators wrote messages
without prompts. We simulated Random selection by randomly selecting phrases
from the phrase corpus (Section 3.3.1) and Directed selection with our
technical approach (Sections 3.3.1 to 3.3.4). If we assume that prompt
embeddings are an unbiased estimator for ideation embeddings, then this gives
an approximation of ideation diversity due to prompting. We conducted
experiments for directing towards diverse prompts and for directing away from
the 500 pilot prior ideations. We varied the number of prompts (Prompt Count,
$n=50$,150,…,950) to simulate how diversity increases with the number of
ideation tasks performed. This investigates how diversity increases as the
budget for crowd tasks increases. To investigate how well Directed selection
avoids prior ideations, we varied the number of repeller prior ideations
(Repeller Prior Ideations Count, $n_{R}=50$,100,150,200). We varied the number
of phrases in prompts (Prompt Size, $g=1$ to 5) to simulate ideating on one or
more phrases in each prompt. We computed the prompt embedding as the average
of all phrases in the prompt. For Random selection, we randomly chose phrases
to group together for each prompt. This random neighbour selection will lead
to variation in prompt consistency, but does not bias the prompt embedding on
average. For Directed selection, phrases in each prompt were chosen as
described in Section 3.3.3.

##### Results on Manipulation Efficacy Analysis (RQ3.1a)

We visualized (Figure 3.9) the phrase embeddings to help to interpret how the
selected prompts are distributed, whether they are well spread out, clustered,
etc. We used Uniform Manifold Approximation and Projection (UMAP) [206] to
reduce the 512 dimensions of USE to a 2D projection. Hyperparameters were
selected such that the 2D points in UMAP had pairwise distances correlated
with that of the 512-dimension USE embeddings.

Figure 3.9: 2D UMAP projection showing how diversely selected prompts and
resulting ideation messages are distributed based on Directed or Random prompt
selection technique and prompt size (number in brackets). Each point
represents the embedding of a text item. Light grey points represent all
phrases in the extracted corpus, dark grey points represent selected phrases
from the simulation study (Section 3.5.1) and blue dots represent the ideated
messages written by crowdworkers in the ideator user study (Section 3.5.2).
Gradient lines connect ideation messages to their stimulus prompts.

We can see that Directed prompt selection led to prompts that were more spread
out, and less redundant from prior ideation. This is more pronounced for
higher prompt size ($g=3$). Random(3) had lower diversity than None with
tighter clustering of prompts (grey points in middle-bottom graph) than of
messages (blue points in left graph). This was because Random(3) prompts
averaged their embeddings from multiple phrases, such that this variance of
means of points is smaller than the variance of points666This is analogous to
standard error is to standard deviation.. We further conducted a
characterisation study with 50 simulations for each prompt configuration to
confirm that Directed Diversity improves diversity and reduces redundancy from
prior ideations for various embedding-based metrics (see Appendix A.5 \-
Figure A.6).

####  3.5.2 Ideation User Study

The Ideation User Study serves as a manipulation check that higher prompt
diversity can be perceived by ideators, and as an initial evaluation of
ideation diversity based on computed and thematically coded metrics.

##### Experiment Treatment and Procedure

We conducted a between-subjects experiment with two independent variables
prompt selection technique (None, Random, Directed) and prompt size ($g=1$ and
$3$), and kept constant prompt count $n=250$. The None condition (no prompt)
allows us to measure if the quality of ideations become worse due to the undue
influence of phrases in prompts. The Random condition provides a strong
baseline since it also leverages the extracted phrases in the first step of
Directed Diversity. A prompt size of $g>1$ can provide more contexts to help
ideators understand the ideas in the phrases, but may also lead to more
confusion if the phrases are not consistent (too dissimilar). Figure 3.10
shows example prompts that ideator participants see in different conditions.
The experiment apparatus and survey questions were implemented in Qualtrics
(see Appendix Figures A.15 to A.21 for instructions and question interface).

Figure 3.10: Example prompts shown to participants in different conditions:
None (left, $g=0$, Directed(1) (center, $g=1$), and Directed(3) (right,
$g=3$). Phrase texts would be different for Random(1) and Random(3) selection
techniques.

##### Experiment Task and Procedure

Participants were tasked to write motivational messages and answer questions
with the following procedure:

  1. 1.

Read the introduction to describe the experiment objective and consent to the
study.

  2. 2.

Complete a 4-item word associativity test [53] to screen for English language
skills.

  3. 3.

Write 5 messages to motivate for physical activity for a fitness mobile app.
For each message, one at a time,

    1. a.

On the first page, depending on condition, see no prompt or a prompt with one
or three phrases selected randomly or by Directed Diversity (see Figure 5),
then write a motivational message in one to three sentences. This page is
timed to measure ideation task time.

    2. b.

Rate on a 5-point Likert scale the experience of ideating the current message:
ease of ideation, self-assessed success in writing motivationally, and success
in writing creatively; perception of the prompt on: understandability,
relevance to domain topic (physical activity), relevance to task (motivation),
helpfulness for inspiration, and unexpectedness (described in Section 3.4.2).

    3. c.

Reflect and describe in free text on their rationale, thought process, phrase
word usage, and ideation effort. We analyze these quotes to verify our
understanding of the collected quantitative data.

  4. 4.

Answer demographics questions, and end the survey by receiving a completion
code.

##### Experiment Data Collection and Statistical Analyses

We recruited participants from Amazon Mechanical Turk with high qualification
($\geq$5000 completed HITs with $>$97% approval rate). Of 282 workers who
attempted the survey, 250 passed the test to complete the survey (88.7% pass
rate). They were 45.2% female, between 21 and 70 years old (M$=38.6$); 76.4%
of participants have used fitness apps. Participants were compensated after
screening and were randomly assigned to one prompt selection technique.
Participants in the None condition were compensated with USD$1.80, while
others with USD$2.50 due to more time needed to answer the additional survey
questions about prompts. Participants completed the survey in median time 15.4
minutes and were compensated $>$USD$8/hour. We collected 5 messages per
participant, 50 participants per condition, 250 ideations per condition, and
1,250 total ideations.

For all response variables, we fit linear mixed effects models described in
Appendix Tables A.28 to A.31. To allow a 2-factor analysis, we divided
responses in the None(0) condition (no prompt, 0 phrases) randomly and evenly
to None(1) and None(3). Results are shown in Figure 3.11. We performed post-
hoc contrast tests for specific differences identified. Due to the large
number of comparisons in our analysis, we consider differences with $p<.001$
as significant and $p<.005$ as marginally significant. Most significant
results reported are $p<.0001$. This is stricter than a Bonferroni correction
for 50 comparisons (significance level = .05/50). We next describe the
statistically significant results for prompt mediation check (RQ3.1b),
mediation analysis (RQ3.2a, 3.2b), and ideation evaluation (RQ3.3). We include
participant quotes from their rationale text response where available and
relevant.

##### Results of Manipulation Check on Creativity and Mediation on Ideation
Effort (RQ3.1b, 3.2a)

We discuss findings on how ideators perceived creativity factors in prompts
and how prompt configurations affected their ideation effort. Figure 3.11
(Top) shows that compared to Random, Directed Diversity selected prompts that
were more unexpected (good for diversity); but were slightly more difficult to
understand (by half unit on 5-point Likert scale), very slightly less relevant
(1/4 unit), and of slightly lower quality (1/2 unit). However, the relevance
of the selected diverse prompts was not explicitly controlled. P173 in
Directed(1) felt that the phrase “first set of challenges is” was
“straightforward and gave me the idea of what to write. It was very easy”;
whereas P157 in felt that the phrase “review findings should be” “didn’t
really have anything I could think to tie towards a motivational message. I
tried to think of it as looking back to see progress in terms of reviewing
your journey.” Random prompts with more phrases were harder to understand,
perhaps, because they were randomly grouped and are less semantically similar.
P128 in Random(3) found that “these [phrases] were hard to combine since they
deal with different aspects of exercise. Also the weight lifting seems to be
not the best thing for addressing obesity, so that was hard to work in.”

Figure 3.11: Results of ideators’ perceived prompt creativity (Top) and
ideation effort (Bottom) for different Prompt Selection technique and Prompt
Size. All factors values on a 5-point Likert scale (–2 = “Strongly Disagree”
to 2 = “Strongly Agree”). Dotted lines indicate extremely significant
$p<.0001$ comparisons, otherwise very significant with p-value stated; solid
lines indicate no significance at $p>.01$. Error bars indicate 90% confidence
interval.

We found that ideation effort was mediated by prompt factors. Figure 3.11
(Bottom) shows that Directed prompts were least easy to use for ideation, and
less adopted than Random selected prompts. This is consistent with Directed
prompts being less understandable than Random. Ideating with 1-phrase prompts
increased ideation time from 44.1s by 21.6s (48.9%) compared to None, and
viewing 3 phrase increased time further by 11.9s. In summary, Directed
Diversity may improve diversity by selecting unexpected prompts, but at some
cost of ideator effort and confusion. This cost compromises prompt adoption
and suggests that directing diversity may not work. Yet, as we will show
later, Directed Diversity does improve ideation creativity. We analyzed the
confound of understandability further in Appendix Section A.11. Next, we
investigate if prompts characteristics mediate more ideation creativity.

##### Results of Mediation Analysis of Diversity Propagation from Prompt to
Ideation (RQ3.2c)

We found that prompt configuration and perceived prompt creativity mediated
the individual diversity of ideated messages (RQ3.2b). Appendix Table A.29a
(in) shows that Ideation Mean (or Min) Pairwise Distance increased with Prompt
Mean (or Min) Pairwise Distance by +0.176 (or +0.146), and marginally with
Intra-Prompt Phrase Mean Distance by +0.021 (or +0.020). This means that
farther Prompts stimulated farther Ideations, and higher variety of Phrases
within each prompt drove slightly farther Ideations too. Hence, prompt
diversity (mean pairwise distance) influenced ideation diversity, and prompt
redundancy (minimum pairwise distance) influenced ideation redundancy.
Appendix Table A.29b shows that as Prompt Relevance decreased by one Likert
unit (on 5-point scale), ideation mean pairwise distance decreased by 0.0034
(7.9% of ideation pairwise distance SD of 0.043) and ideation minimum pairwise
distance decreases by 0.0056 (13% of SD). This suggests that prompting with
irrelevant phrases slightly reduced diversity, since users had to have to
conceive their own inspiration; e.g., P165 in Directed(1) “couldn’t make sense
of the given messages, so I tried my best to make something somewhat
motivational and correct from them.”. Prompt understandability and quality did
not influence ideation individual diversity (p=n.s.). In summary, selecting
and presenting computationally diverse and less redundant prompts increased
the likelihood of crowdworkers ideating messages that are more computationally
diverse and less redundant.

##### Results on Evaluating Individual, Collective Objective, Thematic
Ideation Diversity (RQ3.3)

Having shown the mediating effects of diverse prompts, we now evaluate how
prompt selection techniques affect self-assessed creativity ratings, objective
diversity metrics of ideations, and thematically coded diversity metrics of
ideations. To carefully distinguish between the commonly used mean pairwise
distance with the less used minimum pairwise distance, we performed our
analyses on them separately. We calculated one measurement of each collective
diversity metric in Table 3.6 for all messages in each prompt selection
condition, and computed uncertainty estimations from synthesized 50 bootstrap
samples 777For each dataset, randomly sample with replacement from the
original dataset until the same dataset size is reached. to generate 50
readings of each diversity metric. We performed factor analyses on the metrics
as described in Section 3.4.3, and performed statistical analyses on these
factors as described in Appendix Table A.30. Analyses on both individual
diversity and collective diversity measures had congruent results (Figure
3.12), though results for collective diversity had more significant
differences ($p<.001$). For collective diversity, our factor analysis found
that Ideation Dispersion was most correlated with mean pairwise distance, and
Ideation Evenness with entropy and mean of Chamfer distance. Directed(3)
improved Ideation Dispersion from None, while Random reduced Dispersion (even
more for 3 vs. 1 phrases). Directed prompts improved Ideation Evenness more
than Random with respect to None. There was no significant difference for
self-assessed Ideation Quality (p=n.s., Table A.31a in Appendix).

Figure 3.12: Results of computed individual and collective diversity from
ideations for different prompt configurations. See Figure 3.11 caption for how
to interpret charts.

##### Thematic Ideation Diversity

The previous ideation diversity metrics were all computational. We next assess
diversity with human judgement based on thematic analysis. By performing a
thematic analysis, we can gain an understanding of the diversity of messages
on a more qualitative level. Specifically, we categorised messages based on
verbal cues such as behaviour change techniques from health behaviour science
[210, 271, 233, 211], or the informational content of messages (such as
specific exercises, locations, times, or authorities being referenced). To
conserve manpower when evaluating ideations, we limited thematic coding and
crowdworker validation to ideations from three conditions of prompts with 1
phrase, i.e., None, Random(1), and Directed(1).

To conduct our thematic analysis of ideated messages, we used open coding of
grounded theory [116] to derive categories. These categories were added,
reduced, merged, and refined by iteratively assessing the messages. We then
consolidated the categories into themes using affinity diagramming [21]. The
thematic analysis was primarily performed by one co-author researcher with
regular discussion with co-authors. We calculated inter-rater reliability on a
random 10% subset of messages was coded independently by another co-author to
obtain a Krippendorff’s alpha with MASI distance of $\alpha=0.82$, which
indicated good agreement. Note that while thematic analyses and affinity
diagramming are popular methods to interpret qualitative data, we use them
here for quantifying qualitative themes and categories of messages.

Our thematic analysis resulted in 239 categories which we consolidated to 53
themes. For example categories of “Squats” or “Dance” are in the “Exercise
Suggestion” theme; and the category “Strong immune system” is in the “Health
Benefits” theme. See Table LABEL:tab:directed-thematic for full list of
categories and themes.

Table 3.1: Themes and categories identified with the thematic analysis of messages Theme | Categories
---|---
Ambiguous Benefits |  Ambiguous benefits
Anecdote |  Anecdote
Appeal to “Obvious” Knowledge |  Appeal to “obvious” knowledge
Appeal by Cohorts |  Appeal to children $\mid$ Appeal to older adults $\mid$ Appeal to weight
Appeal to Fear |  Appeal to fear
Appeal to Guilt |  Appeal to guilt
Appeal to Shame |  Appeal to shame
Appeal to Social Approval |  Appeal to social approval
Barrier to Ability |  Barrier to ability
Barrier to Boredom |  Encourage exercise variety $\mid$ Prompt to try something new $\mid$ Tips to make exercise less boring/more fun
Barrier to Comfort |  Barrier to comfort
Barrier to Cost |  Cheap exercises $\mid$ Lower healthcare/insurance costs
Barrier to Effort |  It will get easier $\mid$ Recommending less effortful routines or exercises $\mid$ Take a short break then carry on
Barrier to Energy |  Barrier to energy
Barrier to Enjoyment |  Prompt to research fun exercise $\mid$ Recommending enjoyable activity
Barrier to Motivation |  Barrier to motivation
Barrier to Resources |  At home exercises $\mid$ No equipment needed $\mid$ No gym available
Barrier to Self-Efficacy |  Don’t feel bad if confused $\mid$ Don’t need certificate or qualifications $\mid$ Improving self-confidence $\mid$ Recommending exercises within ability
Barrier to Time |  Barrier to time
Call to Action |  Call to action
Call to Authority |  Citing health experts $\mid$ Unspecified authority
Collective Societal Benefits |  Collective societal benefits
Equipment |  Bench press $\mid$ Exercise machine $\mid$ Exercise machines (unspecified $\mid$ Rubber exercise tubing/band $\mid$ Swimming gear $\mid$ Treadmill $\mid$ Vertical or horizontal press $\mid$ Work - standing desk
Exercise Suggestion |  Aerobic exercises $\mid$ Aerobics $\mid$ Anaerobic exercise $\mid$ Biking $\mid$ Body weight exercises $\mid$ Cardiovascular exercise $\mid$ Climbing $\mid$ Competitive cycling $\mid$ Dance $\mid$ Diving $\mid$ Double clean $\mid$ Exercise through chores $\mid$ Handstand $\mid$ High intensity exercise $\mid$ Hot yoga $\mid$ Internal rotation workouts $\mid$ Iron Yoga $\mid$ Jump on bed $\mid$ Jumping jacks $\mid$ Lift weights $\mid$ Lifting luggage $\mid$ Meditation $\mid$ Pull-ups $\mid$ Pushing kids on swings $\mid$ Push-ups $\mid$ Resistance exercises $\mid$ Ring Pull-ups $\mid$ Running $\mid$ Seated leg-raises $\mid$ Sit-ups $\mid$ Snatches $\mid$ Sports $\mid$ Squats $\mid$ Strength training $\mid$ Strenuous/moderate/vigorous exercise $\mid$ Stretching $\mid$ Swimming $\mid$ Tennis $\mid$ Using stairs $\mid$ Vertical and horizontal presses $\mid$ Volleyball $\mid$ Walk your dog $\mid$ Walking $\mid$ Water exercises $\mid$ Work/desk exercise $\mid$ Yoga
Fear of Injury |  Don’t overexert yourself $\mid$ Recommending exercises to avoid injury $\mid$ Research good techniques to avoid injury $\mid$ Take breaks $\mid$ Tips to avoid injury for outdoor activities
Food and Drink |  Avoid steroids, pills and drugs $\mid$ Avoid unhealthy food $\mid$ Exercise supplements $\mid$ Exercise to avoid medication and drugs $\mid$ Food recommendation $\mid$ Staying hydrated $\mid$ Stress eating advice
Future Life |  Improve quality of life $\mid$ Live longer
Goals |  Journaling to track your goal $\mid$ Set Actionable goals $\mid$ Set daily exercise goal $\mid$ Set goals based on health recommendations $\mid$ Set unspecified goal $\mid$ Set weight goal $\mid$ Tips to reach goals $\mid$ Visualising meeting goals
Health Advice |  Advice for diabetics $\mid$ See a doctor if you are worried
Health Benefits |  Better mobility $\mid$ Bone health $\mid$ Cardio Health $\mid$ Fluid regulation $\mid$ Help with foot problems $\mid$ Lowers blood pressure $\mid$ More stamina $\mid$ Pain and strain relief $\mid$ Slows aging process $\mid$ Strong immune system $\mid$ Unspecified health benefit
Health Risks |  Arthritis $\mid$ Breathing difficulty $\mid$ Cancer $\mid$ Depression $\mid$ Diabetes $\mid$ Heart disease $\mid$ Hernias $\mid$ Obesity $\mid$ Unspecified health risk
Improving Appearance |  “Look better” $\mid$ Beach ready $\mid$ Chest and back $\mid$ Improved posture $\mid$ Look more appealing to potential partners $\mid$ Nice butt $\mid$ Six-pack abs
Inspirational Phrase |  Inspirational phrase
Lack of Knowledge |  Research exercise routines $\mid$ Research nutrition $\mid$ Study exercise form
Lack of Social Support |  Exercise with an expert $\mid$ Exercise with friends $\mid$ Family want you to be healthy $\mid$ Find places to support you $\mid$ Impress your doctor $\mid$ Interacting with others $\mid$ Join a health club $\mid$ Join an exercise class $\mid$ Meeting new people $\mid$ Playing/exercising with children $\mid$ Social competition
Mental Health Benefits |  Feeling better $\mid$ Happier $\mid$ Improved sleep $\mid$ Improves cognitive abilities $\mid$ Lower depression $\mid$ Lowers anxiety $\mid$ Relaxing $\mid$ Release endorphins $\mid$ Self-esteem $\mid$ Stress reduction $\mid$ Unspecified mental health benefits
Muscle Building |  Biceps $\mid$ Core strength $\mid$ Leg Muscles $\mid$ Physically stronger $\mid$ Shoulder muscles $\mid$ Triceps $\mid$ Unspecified Muscle building $\mid$ Upper body strength
Overcoming Beliefs |  Changing mindset
Overcoming Family Obligations |  Overcoming family obligations
Overcoming Self-Consciousness |  Overcoming self-consciousness
Push to Do More |  Prompt to increase $\mid$ Prompt to steadily increase
Push to Start |  Prompt to start exercising $\mid$ Prompt to stop sedentary lifestyle
Rewards |  Prizes from exercise competitions $\mid$ Reward with food $\mid$ Reward with new clothes $\mid$ Reward with new exercise equipment $\mid$ Unspecified reward
Self-Empowerment |  Self-empowerment
Self-Forgiving |  Self-forgiving
Self-Reflection |  Self-reflection
Social Comparison |  Avoid social comparisons $\mid$ Downwards Social comparison $\mid$ Upwards Social Comparison
Specific Locations |  Around the neighbourhood $\mid$ At desk exercises $\mid$ At home $\mid$ At school $\mid$ Beach $\mid$ Church/community centre $\mid$ Front yard $\mid$ Gym $\mid$ Outdoors $\mid$ Park $\mid$ Travelling/airport $\mid$ Walk to train station
Specificity |  Appropriate Exercises (e.g. “Try what’s best for you”) $\mid$ Developing habits $\mid$ Even small amounts of exercise $\mid$ Exercise daily $\mid$ Exercise regularly $\mid$ Follow exercise plan/routine $\mid$ Specific amount or distance to exercise $\mid$ Specific days a week to exercise $\mid$ Specific minutes to exercise
Time to See Results |  Dedicate time $\mid$ Fast results $\mid$ Promise of results $\mid$ Tips to progress faster
Time to Exercise |  Anytime $\mid$ End of the day $\mid$ Morning exercise $\mid$ Spring $\mid$ Summer/hot weather
Use of Technology |  Exergame $\mid$ Experts review your exercises from an app $\mid$ Follow videos $\mid$ Listen to music/podcast $\mid$ Reflect on progress $\mid$ Use apps for exercise tips $\mid$ Use apps for workout schedules $\mid$ Use apps to track progress $\mid$ Watch TV while exercising
Weight Loss |  Aid digestion $\mid$ Boost metabolism $\mid$ Burning calories $\mid$ Burning cellulite $\mid$ Maintaining weight $\mid$ Slimming down

Next, to quantify categories and themes (and compare between conditions) we
used metrics from creativity literature, namely the flexibility [287] and
originality [345] of ideations. Flexibility [287] measures how many unique
ideas (conceptual categories) was generated, and originality [345] measures
how infrequently each conceptual category occurs. Finally, we calculate the
flexibility and originality measures based on the coded categories (fine-
grained) and themes (coarser) described in Figure 3.13.

Figure 3.13: Metrics of creativity of ideation based on categories and themes
derived from our thematic analysis of generated ideas. Metrics are shown for
categories, but are the same for themes.

Figure 3.14 shows results from our statistical analysis. We found that
ideations generated with Directed prompts had higher Flexibility and
Originality in categories and themes than with Random or None, demonstrating
that Directed prompts generated more diverse verbal cues (conversational
content). Ideations from Random prompts mostly had higher Flexibility and
Originality compared to None, but the theme Originality was significantly
lower. This could be because Random prompts primed ideators to fixate on fewer
broad ideas (themes), instead of the higher number of fine-grained idea
categories.

Figure 3.14: Results of diversity in categories and themes derived from
thematic analysis of ideations.

In summary, despite lower ideation ease and understandability with Directed
prompts (Section 3.5.2), we found objective and thematic evidence that
Directed Diversity improved ideation diversity compared to Random and None.
Additionally, we found that directed (through our thematic analysis) led to
the generation of more diverse (conversational content) verbal cues. Next, we
describe how crowdworkers rated these ideations.

####  3.5.3 Validation User Studies

The third and fourth studies employed third-party crowdworkers to assess the
creativity of ideated messages from the Ideation User Study, to answer (RQ3.3)
How do prompt selection techniques affect diversity in ideations? This
provides a less biased validation than asking ideators to self-assess. We
conducted three experiments with different questioning format to strengthen
the experiment design. Appendix Figures A.22 to A.24 details the
questionnaires.

##### Individual Validation: Experiment Treatment and Procedure

For the individual validation study, we conducted a within-subjects experiment
with prompt selection technique (None, Random, Directed) as independent
variable, and controlled prompt size ($g=1$). Each participant assessed 25
ideation messages chosen randomly from the three conditions. Participants went
through the same procedure as in the ideation user study, but with a different
task in step 3:

  * •

Assess 25 messages regarding how well they motivate for physical activity. For
each message,

    1. a.

Read a randomly chosen message.

    2. b.