# When Are Combinations of Humans and AI Useful?

Michelle Vaccaro  MIT Center for Collective Intelligence, Sloan School of
Management, Massachusetts Institute of Technology, Cambridge, USA  Institute
for Data, Systems, and Society, Schwarzman College of Computing, Massachusetts
Institute of Technology, Cambridge, USA  Abdullah Almaatouq  MIT Center for
Collective Intelligence, Sloan School of Management, Massachusetts Institute
of Technology, Cambridge, USA  Thomas Malone  MIT Center for Collective
Intelligence, Sloan School of Management, Massachusetts Institute of
Technology, Cambridge, USA

Inspired by the increasing use of AI to augment humans, researchers have
studied human-AI systems involving different tasks, systems, and populations.
Despite such a large body of work, we lack a broad conceptual understanding of
when combinations of humans and AI are better than either alone. Here, we
addressed this question by conducting a meta-analysis of over 100 recent
experimental studies reporting over 300 effect sizes. First, we found that, on
average, human-AI combinations performed significantly worse than the best of
humans or AI alone. Second, we found performance losses in tasks that involved
making decisions and significantly greater gains in tasks that involved
creating content. Finally, when humans outperformed AI alone, we found
performance gains in the combination, but when the AI outperformed humans
alone we found losses. These findings highlight the heterogeneity of the
effects of human-AI collaboration and point to promising avenues for improving
human-AI systems.

##  1 Introduction

People increasingly work with artificial intelligence (AI) tools in fields
including medicine, finance, and law, as well as in daily activities such as
traveling, shopping, and communicating. These human-AI systems have tremendous
potential given the complementary nature of humans and AI – the general
intelligence of humans allows us to reason about diverse problems, and the
computational power of AI systems allows them to accomplish specific tasks
that people find difficult.

In fact, a large body of work suggests that integrating human creativity,
intuition, and contextual understanding with AI’s speed, scalability, and
analytical power can lead to innovative solutions and improved decision-making
in areas such as healthcare [1], customer service [2], and scientific research
[3]. On the other hand, a growing number of studies reveal that human-AI
systems do not necessarily achieve better results when compared to the best of
humans or AI alone. Challenges such as communication barriers, trust issues,
ethical concerns, and the need for effective coordination between humans and
AI systems can hinder the collaborative process [4, 5, 6, 7, 8, 9].

These seemingly contradictory results raise important questions: When do
humans and AI complement each other? And by how much? To address these issues,
we conducted a systematic literature review and meta-analysis in which we
quantified synergy in human-AI systems and identified factors that explain its
presence (or absence) in different settings.

Extending the concepts of strong and weak synergy from human teams [10, 11],
we focused on two types of synergy: (1) strong synergy, where the human-AI
group performs better than both the human alone and the AI alone; and (2) weak
synergy, where the human-AI group performs better than either the human or the
AI alone but not both.

When evaluating human-AI systems, many studies focus on a specific kind of
weak synergy where the baseline is the performance of the humans alone [12,
13, 14, 15]. This measure can serve important purposes in contexts for which
full automation cannot happen for legal, ethical, safety, or other reasons.
But when talking about the potential of human-AI systems, most people
implicitly assume that the combined system should be better than either alone;
otherwise, they would just use the best of the two [16]. Thus they are looking
for strong synergy. In light of these considerations, a growing body of work
emphasizes evaluating and searching for strong synergy in human-AI systems [4,
17, 8, 18, 19, 20].

To evaluate this synergy in human-AI systems, we analyzed 370 unique effect
sizes from 106 different experiments published between January 2020 and June
2023 that included the performance of the human-only, AI-only, and human-AI
systems. On average, we found evidence of weak synergy, meaning that the
average human-AI systems performed better than the human alone. But we did not
find strong synergy on average, meaning that the average human-AI systems
performed worse than either the human alone or the AI alone. So, in practice,
if we consider only the performance dimensions the researchers studied, it
would have been better to use either a human alone or an AI system alone
rather than the human-AI systems studied.

While this overall result may appear discouraging, we also identified specific
factors that did or did not contribute to strong synergy in human-AI systems.
On the one hand, for example, much of the recent research in human-AI
collaboration has focused on using AI systems to help humans make decisions by
providing not only suggested decisions but also confidence levels or
explanations. But we found that neither of these factors significantly
affected the performance of human-AI systems.

On the other hand, little work has investigated the effects of task types and
the relative performance of humans alone and AI alone. But we found that both
factors significantly affected human-AI performance. Our work thus sheds
needed light on promising directions for designing future human-AI systems to
unlock the potential for greater synergy.

##  2 Results

Our initial literature search yielded 5126 papers, and, per the review process
described in the Methods (see section 4.1), we identified 74 that met our
inclusion criteria (see Figure LABEL:fig:prisma). These papers reported the
results of 106 unique experiments, and many of the experiments had multiple
conditions, so we collected a total of 370 unique effect sizes measuring the
impact of human-AI collaboration on task performance. Figure
LABEL:fig:barplots highlights the descriptive statistics for the experiments
in our analysis. We synthesize these data in a three-level meta-analytic model
(see the Methods section 4.2). We also make the materials required to
reproduce our results publicly accessible through an Open Science Framework
(OSF) repository.

###  2.1 Overall Levels of Human-AI Synergy

In our primary analyses, we focus on strong human-AI synergy, so we compare
the performance of the human-AI systems to a baseline of the human alone or
the AI alone, whichever performed best. We found that the human-AI systems
performed significantly worse overall than this baseline. The overall pooled
effect was negative ($g=-0.23$, $p=0.0047$, 95% confidence interval (CI)
$-0.39$ to $-0.07$) and considered small according to conventional
interpretations [76].

On the other hand, when we compared the performance of the human-AI systems to
a different baseline – the humans alone – we found substantial evidence of
weak human-AI synergy. The human-AI systems performed significantly better
than humans alone, and this pooled effect size was positive ($g=0.64$,
$p<0.0001$, 95% CI $0.53$ to $0.74$) and medium to large in size [76]. Figure
1 displays a forest plot of these effect sizes. In other words, the human-AI
systems we analyzed were, on average, better than humans alone but not better
than both humans alone and AI alone. For effect sizes that correspond to other
types of human-AI synergy, see Figure LABEL:fig:ForestPlot_Synergy_Other.

(a)

(b)

Figure 1:  Forest plots of all effect sizes ($k=370$) included in the meta-
analysis. The positions of the points on the $x$-axes represent the values of
the effect sizes, and the bars indicate the 95% confidence interval for the
effect sizes. The colors of the points and lines correspond to the values of
the effect sizes, with negative effect sizes colored red (no human-AI synergy)
and positive effect sizes colored green (human-AI synergy). The black dotted
line corresponds to an effect size of Hedges’ $g=0$, which means that the
human-AI group performed the same as the baseline. The circle at the bottom of
the graph represents the meta-analytic average effect size and confidence
interval.

###  2.2 Heterogeneity of Human-AI Synergy

We also found evidence for substantial heterogeneity of effect sizes in our
estimations of both strong synergy ($I^{2}=97.7\%$) and weak synergy
($I^{2}=93.8\%$) (see table Table LABEL:tab:het_table for more details). Our
moderator analysis identified characteristics of participants, tasks, and
experiments that led to different levels of human-AI synergy, and it helps
explain sources of this heterogeneity. Figure 2 provides a visualization of
the results of meta-regressions with our moderators. Definitions of the
subgroups for different moderator variables are included in Table
LABEL:tab:mod_desc, and more details of the regressions are included in Table
LABEL:tab:mod_table.

Figure 2:  Results from the three-level meta-regression models for the
moderator variables. Here, $K$ is the number of included effect sizes for the
moderator subgroup level, and the estimated effect size with the corresponding
95% confidence interval. The symbols in front of the moderator indicate if
there is a statistically significant difference between the subgroups for
strong synergy (*) and weak synergy (∧).

First, we found that the type of task significantly moderated strong human-AI
synergy ($p=0.0061$). Among decision tasks – those in which participants
decided between a finite set of options – the pooled effect size for strong
synergy was significantly negative ($g=-0.27$, 95%CI $-0.44$ to $-0.10$),
which indicates performance losses from combining humans and AI. In contrast,
among creation tasks – those in which participants created some sort of open
response content – the pooled effect size for strong synergy was positive
($g=0.19$, 95%CI $-0.09$ to $0.48$), pointing to strong synergy between humans
and AI. Even though the average performance gains for creation tasks were not
significantly different from 0 (presumably because of the relatively small
sample size of $n=34$), the difference between losses for decision tasks and
gains for creation tasks was statistically significant. Relatedly, we found
that the type of data involved in the task significantly moderated both strong
and weak synergy in human-AI systems ($p<0.0001$).

Second, we found that the performance of the human and AI relative to each
other impacted both types of human-AI synergy ($p<0.0001$). As shown in Figure
2, when the human alone outperformed the AI alone, the combined human-AI
system outperformed both alone with an average pooled effect size for strong
synergy of $g=0.46$ (95%CI $0.28$ to $0.66$, a medium-sized effect [76]). But
when the AI alone outperformed the human alone, performance losses occurred in
the combined system relative to the AI alone, with a negative effect size for
strong synergy of $g=-0.54$ (95%CI $-0.71$ to $-0.37$, a medium -sized effect
[76]).

The performance of the human and AI relative to each other also affected the
degree of weak synergy in the human-AI systems. When the AI outperformed the
human alone, greater performance gains tended to occur in the human-AI systems
relative to the human alone ($p<0.0001$), and the pooled effect size for this
type of weak synergy was positive and medium to large in magnitude ($g=0.74$,
95%CI $0.63$ to $0.85$) (see Figures LABEL:fig:acc_synergy and
LABEL:fig:acc_synergy_h for a more detailed visualization of this result for
decision tasks).

Additionally, we found that the type of AI involved in the experiment
moderated strong synergy in human-AI systems ($p=0.0264$), and the
experimental design moderated weak synergy in human-AI systems ($p=0.0290$).

The remaining moderators we investigated were not statistically significant
for strong or weak human-AI synergy (explanation, confidence, participant
type, division of labor).

##  3 Discussion

Systems that combine human intelligence and AI tools can address multiple
issues of societal importance, from how we diagnose disease to how we design
complex systems [22, 23, 24]. But some studies show how augmenting humans with
AI can lead to better outcomes than humans or AI working alone [25, 24, 26],
while others show the opposite [4, 7, 9]. These seemingly disparate results
raise two important questions: How effective is human-AI collaboration, in
general? And under what circumstances does this collaboration lead to
performance gains versus losses? Our study analyzes over three years of recent
research to provide insights into both of these questions.

Performance Losses from Human-AI Collaboration: Regarding the first question,
we found that, on average among recent experiments, human-AI systems did not
exhibit strong synergy: the human-AI groups performed worse than either the
human alone or the AI alone. This result complements the qualitative
literature reviews on human-AI collaboration [77, 28, 29], which highlight
some of the surprising challenges that arise when integrating human and
artificial intelligence. For example, people often rely too much on AI systems
("overreliance"), using its suggestions as strong guidelines without seeking
and processing more information [30, 31, 6]. Other times, however, humans rely
too little on AI ("underreliance"), ignoring its suggestions because of
adverse attitudes toward automation [7, 31, 32].

Interestingly, we found that, among this same set of experiments, weak synergy
(better than humans) did exist in the human-AI systems: the human-AI groups
performed better than the humans working alone. Thus, even though, on average,
the human-AI combinations did not achieve strong synergy, they did, on
average, augment human performance.

Moreover, the discrepancy between strong and weak synergy in human-AI groups
mirrors that found in the literature on synergy in human-only groups. In both
cases, substantial empirical evidence points to the existence of weak synergy,
but there is relatively spare evidence of strong synergy in task performance
[10, 11]. This result can occur, of course, because by definition the baseline
for strong synergy is more stringent than that for weak synergy. It may also
occur, however, because obtaining strong human-AI synergy requires different
forms of human-AI interaction, or because the recent empirical studies were
not appropriately designed to elicit strong synergy.

Moderating Effect of Task Type: With the large data set we collected, we also
performed additional analyses of factors that influence the effectiveness of
human-AI collaboration. We found that the type of task significantly moderated
strong synergy in human-AI systems with decision tasks associated with
performance losses and creation tasks associated with performance gains.

We hypothesize that this advantage for creation tasks occurs because even when
creation tasks require the use of creativity, knowledge, or insight for which
humans perform better, they often also involve substantial amounts of somewhat
routine generation of additional content that AI can perform as well or better
than humans. For instance, generating a good artistic image usually requires
some creative inspiration about what the image should look like, but it also
often requires a fair amount of more routine fleshing out of the details of
the image. Similarly, generating many kinds of text documents often requires
knowledge or insight that humans have and computers do not, but it also often
requires filling in boilerplate or routine parts of the text as well.

With most of the decision tasks studied in our sample, however, both the human
and the AI system make a complete decision, with the humans usually making the
final choice. Our results suggest that with these decision tasks, better
results might have been obtained if the experimenters had designed processes
in which the AI systems did only the parts of the task for which they were
clearly better than humans. Only 3 of the 100+ experiments in our analysis
explore such processes with a pre-determined delegation of separate sub-tasks
to humans and AI. With the 4 effect sizes from these 3 experiments, we found
that, on average, strong human-AI synergy ($g=0.22,N.S.$) occurred (see
Section LABEL:div_labor for a more detailed discussion of these experiments).

Moderating Effect of Relative Human/AI Performance: Interestingly, when the AI
alone outperformed the human alone, substantial performance losses occurred in
the human-AI systems ($g=-0.54$). When the human outperformed the AI alone,
however, performance gains occurred in the human-AI systems ($g=0.46$). This
finding shows that the human-AI performance cannot be explained with a simple
average of the human alone and AI alone. In such a case, strong human-AI
synergy could never exist [33].

Most (>95%) of the human-AI systems in our data set involved humans making the
final decisions after receiving input from the AI algorithms. In these cases,
one potential explanation of our result is that, when the humans are better
than the algorithms overall, they are also better at deciding in which cases
to trust their own opinions and in which to rely more on the algorithm’s
opinions.

For example, [34] use an experimental design in which subjects in the human-AI
condition saw a problem instance, an AI prediction for that instance, and in
some cases, additional descriptions of the accuracy of the AI on this type of
instance. The same experimental design, with the same task interface,
participant pool, and accuracy of the AI system, was used for three separate
tasks: fake hotel review detection, satellite image classification, and bird
image classification. For fake hotel review detection, the researchers found
that the AI alone achieves an accuracy of 73%, the human alone achieves an
accuracy of 55%, and the human-AI system achieves an accuracy of 69%. In this
case, we hypothesize that, since the people are less accurate, in general,
than the AI algorithms, they are also not good at deciding when to trust the
algorithms and when to trust their own judgement, so their participation
results in lower overall performance than for the AI algorithm alone.

On the other hand, [34] find that, for bird image classification, the AI alone
achieves an accuracy of 73%, the human alone achieves an accuracy of 81%, and
the human-AI system achieves an accuracy of 90%. Here, the humans alone are
more accurate than the AI algorithms alone, so we hypothesize that the humans
are good at deciding when to trust their own judgements versus those of the
algorithms, and, thus, the overall performance improves over either humans or
AI alone.

Surprisingly Insignificant Moderators: We also investigated other moderators
such as the presence of an explanation, the inclusion of the confidence of the
AI output, and the type of participant evaluated. These factors have received
much attention in recent years [4, 24, 26, 35]. Given our result that, on
average across our 300+ effect sizes, they do not impact the effectiveness of
human-AI collaboration, we think researchers may wish to de-emphasize this
line of inquiry and instead shift focus to the significant and less researched
moderators we identified: the baseline performance of the human and AI alone,
the type of task they perform, and the division of labor between them.

Limitations: Importantly, we want to highlight some general limitations of our
meta-analytic approach to aid with the interpretation of our results. First,
our quantitative results apply to the subset of studies we collected through
our systematic literature review. To evaluate human-AI synergy, we required
that papers report the performance of (1) the human alone, (2) the AI alone,
and (3) the human-AI system. We can, however, imagine tasks that a human and /
or AI cannot perform alone but can when working with the other. Our analysis
does not include such studies.

Second, we calculate effect sizes that correspond to different quantitative
measures such as task accuracy, error, and quality. By computing Hedges’ $g$,
a unitless standardized effect size, we can describe important relations among
these experiments in ways that make them comparable across different study
designs with different outcome variables [36]. The studies in our data set,
though, come from different samples of people – some look at doctors [37, 38,
39], others at crowdworkers [4, 34, 6], and still others at students [40, 41,
13] –- and this variation can limit the comparability of the effect size to a
degree [36]. Additionally, the measurement error can also vary across
experiments. For example, some studies estimate overall accuracy based on the
evaluation of as many as 500 distinct images [25] whereas others estimate it
based on the evaluation of as few as 15 distinct ones [42]. As is typical for
meta-analyses [43], in our three-level model, we weight effect sizes as a
function of their variance across participants, so we do not account for this
other source of variation in measurement.

Third, although we did not find evidence of publication biases, it remains
possible that they exist, which would impact our literature base and, by
extension, our meta-analytic results. However, we expect that if there were a
publication bias operating here, it would be a bias to publish studies that
showed significant gains from combining humans and AI. And since our overall
results showed the opposite, it seems unlikely that they are a result of
publication bias.

Fourth, our results only apply to the tasks, processes, and subject pools that
researchers have chosen to study, and these configurations may not be
representative of the ways human-AI systems are configured in practical uses
of AI outside of the laboratory. In other words, even if there is not a
publication bias in the studies we analyzed, there might be a research topic
selection bias at work.

Fifth, the quality of our analysis depends on the quality of the studies we
synthesized. We tried to control for this issue by only including studies
published in peer-reviewed publications, but the rigor of the studies may
still vary in degree. For example, studies used different attention check
mechanisms and performance incentive structures, which can both affect the
quality of responses and thus introduce another source of noise into our data.

Finally, we find a high level of heterogeneity among the effect sizes in our
analysis. The moderators we investigate account for some of this
heterogeneity, but much remains unexplained. We hypothesize that interaction
effects exist between the variables we coded (e.g. explanation and type of
AI), but we do not have enough studies to detect such effects. There are also
certainly potential moderators that we did not analyze. For example,
researchers mostly used their own experimental platforms and stimuli, which
naturally introduce sources of variation between their studies. As the human-
AI collaboration literature develops, we hope future work can identify more
factors that influence human-AI synergy and assess the interactions among
them.

A Roadmap for Future Work: Finding Human-AI Synergy: Even though our main
result suggests that – on average – combining humans and AI leads to
performance losses, we do not think this means that combining humans and AI is
a bad idea. On the contrary, we think it just means that future work needs to
focus more specifically on finding effective processes that integrate humans
and AI. Our other results suggest promising ways to proceed.

Develop Generative AI for Creation Tasks. In our broad sample of recent
experiments, the vast majority (about 85%) of the effect sizes were for
decision-making tasks in which subjects chose among a predefined set of
options. But in these cases we found that the average effect size for strong
synergy was significantly negative. On the other hand, only about 10% of the
effect sizes researchers studied were for creation tasks—those that involved
open-ended responses. And in these cases we found that the average effect size
for strong synergy was positive and significantly greater than that for
decision tasks. This result suggests that studying human-AI synergy for
creation tasks—many of which can be done with generative AI—could be an
especially fruitful area for research.

Much of the recent work on generative AI with human-subjects, however, tends
to focus on attitudes towards the tool [44, 45], interviews or think-alouds
with participants [46, 47, 48], or user experience instead of task performance
[49, 50, 51]. Furthermore, the relatively little work that does evaluate
human-AI collaboration according to quantitative performance metrics tends to
only report the performance of the human alone and human-AI combination (not
the AI alone) [52]. This limitation makes evaluating human-AI synergy
difficult as the AI alone may be able to perform the task at a higher quality
and speed than the participants involved in the experiment, typically
crowdworkers. We thus need studies that further explore human-AI collaboration
across diverse tasks while reporting the performance of the human alone, AI
alone, and human-AI system.

Develop Innovative Processes. Additionally, as discussed in [33], human-AI
synergy requires that humans be better at some parts of a task, AI be better
at other parts of the task, and the system as a whole be good at appropriately
allocating subtasks to whichever partner is best for that subtask. Sometimes
that is done by letting the more capable partner decide how to do allocation
of subtasks, and sometimes it is done by assigning different subtasks a priori
to the most capable partner (see Section LABEL:div_labor of the SI for
specific examples from experiments in our dataset). In general, to effectively
use AI in practice, it may be just as important to design innovative processes
for how to combine humans and AI as it is to design innovative technologies
[53].

Develop More Robust Evaluation Criteria for Human-AI Systems. Many of the
experiments in our analysis evaluate performance according to a single measure
of overall accuracy, but this measure corresponds to different things
depending on the situation, and it omits other important criteria for human-AI
systems. For example, as one approaches the upper bound of performance, such
as 100% accuracy, the improvements necessary to increase performance usually
become more difficult for both humans and AI systems. In these cases, we may
wish to consider a metric that applies a non-linear scaling to the overall
classification accuracy and thus takes such considerations into account [54].

More importantly, there are many practical situations where good performance
depends on multiple criteria. For instance, in many high-stakes settings such
as radiology diagnoses and bail predictions, relatively rare errors may have
extremely high financial or other costs. In these cases, even if AI can, on
average, perform a task more accurately and less expensively than humans, it
may still be desirable to include humans in the process if the humans are able
to reduce the number of rare but very undesirable errors. One potential
approach for situations like these is to create composite performance measures
that incorporate the expected costs of various kinds of errors. The weak
synergy measure described is also appropriate for these high-stakes settings.

In general, we encourage researchers to develop, employ, and report more
robust metrics that take into account factors such as task completion time,
financial cost, and the practical implications of different types of errors.
These developments will help us better understand the significance of
improvements in task performance as well as the effects of human-AI
collaborations.

Develop Commensurability Criteria. As researchers continue to study human-AI
collaboration, we also urge the field to develop a set of commensurability
criteria, which can facilitate more systematic comparisons across studies and
help us track progress in finding areas of human-AI synergy. These criteria
could provide standardized guidelines for key study design elements such as:

  1. 1.

Task Designs: Establish a set of benchmark tasks that involve human-AI
systems.

  2. 2.

Quality Constraints: Specify acceptable quality thresholds or requirements
that the human, AI, and human-AI system must meet.

  3. 3.

Incentive Schemes: Outline incentive structures (e.g. payment schemes,
bonuses) used to motivate human participants.

  4. 4.

Process Types: Develop a taxonomy of interaction protocols, user interface
designs, and task workflows for effective human-AI collaboration.

  5. 5.

Evaluation Metrics: Report the performance of the human, AI, and human-AI
system according to well-defined performance metrics.

To further promote commensurability and research synthesis, we encourage the
field to establish a standardized and open reporting repository, specifically
for human-AI collaboration experiments. This centralized database would host
the studies’ raw data, code, system outputs, interaction logs, and detailed
documentation, adhering to the proposed reporting guidelines. As such, it
would facilitate the replication, extension, and synthesis of research in the
field. For example, by applying advanced machine learning techniques on such a
dataset, we could develop predictive models to guide the design of human-AI
systems optimized for specific constraints and contexts. Additionally, it
would provide a means to track progress in finding greater areas of human-AI
synergy.

In conclusion, our results demonstrate that human-AI systems often perform
worse than humans alone or AI alone. But our analysis also suggests promising
directions for the future development of more effective human-AI systems. We
hope that this work will help guide progress in developing such systems and
using them to solve some of our most important problems in business, science,
and society.

##  4 Methods

We conducted this meta-analysis in accord with the guidelines from Kitchenham
2004 on systematic reviews, and we follow the standards set forth by the
Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)
[55].

###  4.1 Literature Review

####  4.1.1 Eligibility Criteria

Per our pre-registration (see here), we applied the following criteria to
select studies that fit our research questions. First, the paper needed to
present an original experiment that evaluates some instance in which a human
and an AI system work together to perform a task. Second, it needed to report
the performance of (1) the human alone, (2) the AI alone, and (3) the human-AI
system according to some quantitative measure(s). As such, we excluded studies
that reported the performance of the human alone but not the AI alone, and
likewise we excluded studies that reported the performance of the AI alone but
not the human alone. Following this stipulation, we also excluded purely meta-
analyses and literature reviews, theoretical work, qualitative analyses,
commentaries, opinions, and simulations. Third, we required the paper to
include the experimental design, the number of participants in each condition,
and the standard deviation of the outcome in each condition, or enough
information to calculate it from other quantities. Finally, we required the
paper to be written in English.

####  4.1.2 Search Strategy

Given the interdisciplinary nature of human-AI interaction studies, we
performed this search in multiple databases covering conferences and journals
in the computer sciences, information sciences, and social sciences, as well
as other fields. Through consultation with a library specialist in these
fields, we decided to target the Association for Computing Machinery Digital
Library (ACM DL), Association for Information Systems eLibrary (AISeL), and
the Web of Science Core Collection (WoS) for our review. To focus on current
forms of artificial intelligence, we limited the search to studies published
between January 1, 2020 and June 30, 2023.

To develop the search string, we began by distilling the facets of studies
that evaluated the performance of a human-AI system. We required the
following: (1) a human component, (2) a computer component, (3) a
collaboration component, and (4) an experiment component. Given the
multidisciplinary nature of our research question, papers published in
different venues tended to refer to these components under various names [8,
13, 56, 57, 12]. For broad coverage, we compiled a list of synonyms and
abbreviations for each component and then combined these terms with Boolean
operations, resulting in the following search string:

  1. 1.

Human: human OR expert OR participant OR humans OR experts OR participants

  2. 2.

AI: AI OR "artificial intelligence" OR ML OR "machine learning" OR "deep
learning"

  3. 3.

Collaboration: collaborate OR assist OR aid OR interact OR help

  4. 4.

Experiment: "experiment" OR "experiments" OR "user study" OR "user studies" OR
"crowdsourced study" OR "crowdsourced studies" OR "laboratory study" OR
"laboratory studies"

Search term: (1) AND (2) AND (3) AND (4) in the abstract. See Table
LABEL:tab:search_strings for the exact syntax for each database.

To further ensure comprehensive coverage, we also conducted a forward and
backwards search on all studies we found that meet our inclusion criteria.

####  4.1.3 Data Collection and Coding

We conducted the search in each of these databases in July 2023. To calculate
our primary outcome of interest – the effect of combining human and artificial
intelligence on task performance – we recorded the averages and standard
deviations of the task performance of the human alone, the AI alone, and the
human and AI working with each other, as well as the number of subjects in
each of these conditions (see LABEL:data_calc for details).

Many authors reported all of these values directly in the text of the paper. A
notable number, however, reported them indirectly by providing 95% confidence
intervals or standard errors instead of the raw standard deviations. For
these, we calculated the standard deviations using the appropriate formulas
[74]. Additionally, multiple papers did not provide the exact numbers needed
for such formulas, but the authors made the raw data of their study publicly
accessible. In these cases, we download the datasets and computed the averages
and standard deviations using Python or R. If relevant data were only
presented in the plots of a paper, we contacted the corresponding author to
ask for the numeric values. If the authors did not respond, we used a Web Plot
Digitizer [75] to convert plotted values into numerical values. For papers
that conducted an experiment that met our inclusion criteria but did not
report all values need to calculate the effect size, we also contacted the
corresponding author directly to ask for the necessary information. If the
author did not respond, we could not compute the effect size for the study and
did not include it in our analysis.

We considered and coded for multiple potential moderators of human-AI
performance, namely: (1) publication date (2) pre-registration status (3)
experimental design (4) data type (5) task type (6) task output (7) AI type
(8) AI explanation (9) AI confidence (10) participant type and (11)
performance metric. See Table LABEL:tab:mod_desc for a description of each of
these moderator variables. The additional information we recorded served
descriptive and exploratory purposes.

Many papers conducted multiple experiments, contained multiple treatments, or
evaluated performance according to multiple measures. In such cases, we
assigned a unique experiment identification number, treatment identification
number, and measure identification number to the effect sizes from the paper.
Note that we defined experiments based on samples of different sets of
participants.

###  4.2 Data Analysis

####  4.2.1 Calculation of Effect Size

We computed Hedges’ $g$ to measure the effect of combining human and
artificial intelligence on task performance [60]. For strong synergy, Hedges’
$g$ represents the standardized mean difference between the performance of the
human-AI system performance and the baseline, which can be the human alone or
AI alone, whichever one performs better, on average. For weak synergy (better
than humans alone), Hedges’ $g$ represents the standardized mean difference
between the performance of the human-AI system and the baseline of the human
alone.

We chose Hedges’ $g$ as our measure of effect size because it is unitless, so
allows us to compare human-AI performance across different metrics, and it
corrects for upward bias in the raw standardized mean difference (Cohen’s $d$)
[60]. See SectionLABEL:calc_effect in the SI for more details about this
calculation.

####  4.2.2 Meta-Analytic Model

The experiments from the papers we analyzed varied considerably. For example,
they evaluated different tasks, they recruited participants from different
backgrounds, and they employed different experimental designs. Since we
expected substantial between-experiment heterogeneity in the true effect
sizes, for our analysis we used a random-effects model that accounts for
variance in effect sizes that comes from both sampling error and "true"
variability across experiments [61].

Additionally, some of the experiments we considered generated multiple
dependent effect sizes: they could involve multiple treatment groups, and they
could assess performance according to more than one measure, for example. The
more commonly used meta-analytic models assume independence of effect sizes,
which makes them unsuitable for our analysis [62]. We thus adopted a three-
level meta-analytic model in which effect sizes are nested within experiments,
so we explicitly took the dependencies in our data into account in the model
[62, 63]. Furthermore, we used robust variance estimate (RVE) methods to
compute consistent estimates of the variance of our effect sizes estimates
and, relatedly, standard errors, confidence intervals, and statistical tests,
which account for the dependency of sampling errors from overlapping samples
that occurred in experiments that compared multiple treatment groups to a
single control group [64]. When evaluating the significance of our result, we
applied the Knapp and Hartung adjustment and computed a test statistic,
standard error, $p$-value, and confidence interval based on the
$t$-distribution with degrees of freedom, where denotes the number of effect
size clusters (i.e. number of experiments) and denotes the number of
coefficients in the model. To perform our moderator analyses, we conducted
separate meta-regressions for each of our moderator variables.

To interpret the degree of heterogeneity in our effect sizes, we calculated
the popular $I^{2}$ statistic following [65], which quantifies the percentage
of variation in effect sizes that is not from sampling error. Furthermore, to
distinguish the sources of this heterogeneity, we also calculated multilevel
versions of the statistic, following [63].

###  4.3 Bias Tests

In the context of our meta-analysis, publication bias may occur if researchers
publish experiments that show evidence of significant human-AI synergy more
frequently than those that do not. Such actions would affect the data we
collected and distort our findings. To evaluate this risk, we adopted multiple
diagnostic procedures outlined by Viechtbauer and Cheung [66]. First, we
created funnel plots that graph the observed effect sizes on the $x$-axis and
the corresponding standard errors on the $y$-axis [67]. In the absence of
publication bias, we expect the points to fall roughly symmetrically around
the y-axis. We enhanced these plots with colors indicating the significance
level of each effect size to help distinguish publication bias from other
causes of asymmetry [68]. A lack of effect sizes in regions of statistical
non-significance points to a greater risk of publication bias. We visually
inspected and performed Egger’s regression test [69] as well as the rank
correlation test [70] to evaluate the results in the funnel plot. These tests
examine the correlation between the observed effect sizes and their associated
sampling variances. A high correlation indicates asymmetry in the funnel plot,
which may stem from publication bias.

Figure LABEL:fig:funnel_plots displays the funnel plot of the included effect
sizes, and we do not observe significant asymmetry or regions of missing data
in the plot for our primary outcome, strong synergy. Egger’s regression did
not indicate evidence of publication bias in the sample
($\beta=-0.67,p=0.4378$), nor did the rank correlation test
($\tau=0.05,p=0.1211$). Taken as a whole, these tests suggest that our results
for strong synergy are robust to potential publication bias.

Importantly, however, we do find potential evidence of publication bias in
favor of studies that report results in which the human-AI system outperforms
the human alone (weak synergy). In this case, Egger’s regression does point to
publication bias in the sample ($\beta=1.95,p=0.0016$), as does the rank
correlation test ($\tau=0.19,p<0.0001$).

The discrepancy between the symmetry in the funnel plot for strong synergy
versus asymmetry in the funnel plot for weak synergy may reflect how many
researchers and journals implicitly assume an interest in weak synergy,
comparing the human-AI system to the human alone.

###  4.4 Sensitivity Analysis

For our primary analysis, we developed a three-level meta-analytic model that
accounted for variance in the observed effect sizes (first level), variance
between effect sizes from the same experiment (second level), and variance
between experiments (third level). We then calculated cluster-robust standard
errors, confidence intervals, and statistical tests for our effect size
estimates in which we defined clusters at the level of the experiment. This
model accounts for the dependency in effect sizes that result from evaluating
more than one treatment against a common control group and assessing
performance according to more than one measure. It does, however, consider the
experiments in our analysis as independent from each other, even if they come
from the same paper. We find this assumption plausible because the experiments
recruited different sets of participants and entailed different tasks or
interventions.

As a robustness check, though, we performed a sensitivity re-analysis in which
we clustered at the paper level instead of the experiment level. This
multilevel model accounts for variance in the observed effect sizes (first
level), variance between effect sizes from the same paper (second level), and
variance between papers (third level). We still calculated cluster-robust
standard errors, confidence intervals, and statistical tests for our effect
size estimates in which we defined clusters at the level of the experiment
because the participant samples only overlapped on the level of the
experiment. Using this approach, we found a comparable overall effect size for
strong synergy of $g=-0.25$ ($p=0.0008$) and for weak synergy of $g=0.60$
($p<0.0001$).

We also evaluated the robustness of our results to outlying and influential
data points. To detect such data, we computed the residuals and Cook’s
distance for each effect size. We considered residual values greater or less
than three standard deviations from the mean as outliers, and following [71],
we considered values greater than $4/N$ as high influence, where $N$ is the
number of data points in our analysis. Using this approach, we identified 11
outliers. We performed a sensitivity re-analysis on a dataset excluding these
effect sizes, which resulted in similar effect size for strong synergy
($g=-0.24$, $p=0.0010$, 95% CI $[-0.38,-0.10]$).

Additionally, we conducted leave-one-out analyses, in which we performed a
series of sensitivity re-analysis on the different subsets of effect sizes
obtained by leaving out one effect size in our original dataset. We also
conducted leave-one-out analyses at the experiment and publication levels.
These tests show how each effect size, experiment, and publication affect our
overall estimate of the effect of human-AI collaboration on task performance.
Summary effect sizes for strong synergy ranged from $-0.28$ to $-0.19$ with
$p<0.05$ in all cases, indicating the robustness of our results to any single
effect size, experiment, or paper; likewise, summary effect sizes for weak
synergy (better than humans alone) ranged from $0.61$ to $0.66$ with $p<0.05$
in all cases, indicating the robustness of our results to any single effect
size, experiment, or paper.

Lastly, we conducted a sensitivity re-analysis on a dataset that omits the
effect sizes we estimated, either using WebPlotDigitizer or the information
provided by the authors in their paper, and again we found almost identical
results ($g=-0.18$, $p=0.0254$, 95% CI $[-0.34,-0.02]$).

We performed all quantitative analysis with the R statistical programming
language, and we primarily relied on the package metafor [72].

We compiled the data used in this analysis based on the studies identified in
our systematic literature review. We make the data we collected available via
the Open Science Framework repository
(https://osf.io/wrq7c/?view_only=b9e1e86079c048b4bfb03bee6966e560).

We share the code used to conduct our analysis via the Open Science Framework
repository (https://osf.io/wrq7c/?view_only=b9e1e86079c048b4bfb03bee6966e560).

##  5 Acknowledgements

MV was funded by the Accenture Technology Convergence Fellowship. Additional
funding was provided by Toyota Research Institute, by the MIT Quest for
Intelligence, and by the National Research Foundation, Prime Minister’s
Office, Singapore under its Campus for Research Excellence and Technological
Enterprise (CREATE) program. The funding organizations had no role in study
design, data collection and analysis, decision to publish or preparation of
the manuscript. We also thank Jessika Kim and Garrett Campagna for their help
collecting the data for this project.

##  6 Author Contributions

MV conceived the study idea with feedback from AA and TM. MV collected the
data and performed the statistical analysis, with feedback from AA and TM. MV,
AA, and TM wrote the manuscript.

##  7 Competing Interests

The authors declare no competing interests.

## References

  * [1] Adam Bohr and Kaveh Memarzadeh  “The rise of artificial intelligence in healthcare applications”  In _Artificial Intelligence in Healthcare_ Elsevier, 2020, pp. 25–60
  * [2] Luminița Nicolescu and Monica Teodora Tudorache  “Human-computer interaction in customer service: The experience with AI chatbots—A systematic literature review”  In _Electronics_ 11.10 MDPI, 2022, pp. 1579
  * [3] Brian Koepnick et al.  “De novo protein design by citizen scientists”  In _Nature_ 570.7761 Nature Publishing Group UK London, 2019, pp. 390–394
  * [4] Gagan Bansal et al.  “Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance”  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , CHI ’21  , Yokohama, Japan,: Association for Computing Machinery, 2021  DOI: 10.1145/3411764.3445717
  * [5] Zana Buçinca, Phoebe Lin, Krzysztof Z. Gajos and Elena L. Glassman  “Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems”  In _Proceedings of the 25th International Conference on Intelligent User Interfaces_ , IUI ’20  Cagliari, Italy: Association for Computing Machinery, 2020, pp. 454–464  DOI: 10.1145/3377325.3377498
  * [6] Vivian Lai, Han Liu and Chenhao Tan  ““Why is ’Chicago’ Deceptive?” Towards Building Model-Driven Tutorials for Humans”  In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , CHI ’20  Honolulu, HI, USA: Association for Computing Machinery, 2020, pp. 1–13  DOI: 10.1145/3313831.3376873
  * [7] Yunfeng Zhang, Q. Liao and Rachel K.. Bellamy  “Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making”  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , FAT* ’20  Barcelona, Spain: Association for Computing Machinery, 2020, pp. 295–305  DOI: 10.1145/3351095.3372852
  * [8] Gagan Bansal et al.  “Updates in human-AI teams: Understanding and addressing the performance/compatibility tradeoff”  In _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence_ , AAAI’19/IAAI’19/EAAI’19  Honolulu, Hawaii, USA: AAAI Press, 2019  DOI: 10.1609/aaai.v33i01.33012429
  * [9] Michelle Vaccaro and Jim Waldo  “The effects of mixing machine learning and human judgment”  In _Communications of the ACM_ 62.11 New York, NY, USA: Association for Computing Machinery, 2019, pp. 104–110  DOI: 10.1145/3359338
  * [10] James R Larson Jr  “In search of synergy in small group performance”  Psychology Press, 2013
  * [11] Abdullah Almaatouq, Mohammed Alsobay, Ming Yin and Duncan J Watts  “Task complexity moderates group synergy”  In _Proceedings of the National Academy of Sciences_ 118.36 National Acad Sciences, 2021, pp. e2101062118
  * [12] Zi-Hao Bo et al.  “Toward human intervention-free clinical diagnosis of intracranial aneurysm via deep neural network”  In _Patterns_ 2.2 Elsevier, 2021
  * [13] Melanie M Boskemper, Megan L Bartlett and Jason S McCarley  “Measuring the efficiency of automation-aided performance in a simulated baggage screening task”  In _Human Factors_ 64.6 Sage Publications Sage CA: Los Angeles, CA, 2022, pp. 945–961
  * [14] Elizabeth Bondi et al.  “Role of human-AI interaction in selective prediction”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 36, 2022, pp. 5286–5294
  * [15] Max Schemmer et al.  “A meta-analysis of the utility of explainable artificial intelligence in human-AI decision-making”  In _Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society_ , 2022, pp. 617–626
  * [16] H James Wilson and Paul R Daugherty  “Collaborative intelligence: Humans and AI are joining forces”  In _Harvard Business Review_ 96.4, 2018, pp. 114–123
  * [17] Gagan Bansal et al.  “Is the most accurate AI the best teammate? Optimizing AI for teamwork”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 35, 2021, pp. 11405–11414
  * [18] Bryan Wilder, Eric Horvitz and Ece Kamar  “Learning to complement humans”  In _arXiv preprint arXiv:2005.00582_ , 2020
  * [19] Charvi Rastogi, Liu Leqi, Kenneth Holstein and Hoda Heidari  “A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementarity”  In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ 11, 2023, pp. 127–139
  * [20] Hussein Mozannar et al.  “Effective Human-AI Teams via Learned Natural Language Rules and Onboarding”  In _Advances in Neural Information Processing Systems_ 36, 2024
  * [21] Jacob Cohen  “Statistical Power Analysis for the Behavioral Sciences”  Academic press, 2013
  * [22] Won Shin, Jeongyun Han and Wonjong Rhee  “AI-assistance for predictive maintenance of renewable energy systems”  In _Energy_ 221 Elsevier, 2021, pp. 119775
  * [23] Gali Noti and Yiling Chen  “Learning When to Advise Human Decision Makers”  In _International Joint Conference on Artificial Intelligence_ , 2022  URL: https://api.semanticscholar.org/CorpusID:252545048
  * [24] Valerie Chen, Q. Liao, Jennifer Wortman Vaughan and Gagan Bansal  “Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations”  In _Proc. ACM Hum.-Comput. Interact._ 7.CSCW2 New York, NY, USA: Association for Computing Machinery, 2023  DOI: 10.1145/3610219
  * [25] Carlo Reverberi et al.  “Experimental evidence of effective human–AI collaboration in medical decision-making”  In _Scientific reports_ 12.1 Nature Publishing Group UK London, 2022, pp. 14952
  * [26] Han Liu, Vivian Lai and Chenhao Tan  “Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making”  In _Proc. ACM Hum.-Comput. Interact._ 5.CSCW2 New York, NY, USA: Association for Computing Machinery, 2021  DOI: 10.1145/3479552
  * [27] Vivian Lai et al.  “Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies”  In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’23  <conf-loc>, <city>Chicago</city>, <state>IL</state>, <country>USA</country>, </conf-loc>: Association for Computing Machinery, 2023, pp. 1369–1385  DOI: 10.1145/3593013.3594087
  * [28] Fabian Sperrle et al.  “A survey of human-centered evaluations in human-centered machine learning”  In _Computer Graphics Forum_ 40, 2021, pp. 543–568  Wiley Online Library
  * [29] Mansoureh Maadi, Hadi Akbarzadeh Khorshidi and Uwe Aickelin  “A review on human–AI interaction in machine learning and insights for medical applications”  In _International journal of environmental research and public health_ 18.4 MDPI, 2021, pp. 2121
  * [30] Linda J Skitka, Kathleen L Mosier and Mark Burdick  “Does automation bias decision-making?”  In _International Journal of Human-Computer Studies_ 51.5 Elsevier, 1999, pp. 991–1006
  * [31] Zana Buçinca, Maja Barbara Malaya and Krzysztof Z. Gajos  “To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making”  In _Proc. ACM Hum.-Comput. Interact._ 5.CSCW1 New York, NY, USA: Association for Computing Machinery, 2021  DOI: 10.1145/3449287
  * [32] Helena Vasconcelos et al.  “Explanations Can Reduce Overreliance on AI Systems During Decision-Making”  In _Proc. ACM Hum.-Comput. Interact._ 7.CSCW1 New York, NY, USA: Association for Computing Machinery, 2023  DOI: 10.1145/3579605
  * [33] Kate Donahue, Alexandra Chouldechova and Krishnaram Kenthapadi  “Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness”  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’22  Seoul, Republic of Korea: Association for Computing Machinery, 2022, pp. 1639–1656  DOI: 10.1145/3531146.3533221
  * [34] Ángel Alexander Cabrera, Adam Perer and Jason I. Hong  “Improving Human-AI Collaboration With Descriptions of AI Behavior”  In _Proc. ACM Hum.-Comput. Interact._ 7.CSCW1 New York, NY, USA: Association for Computing Machinery, 2023  DOI: 10.1145/3579612
  * [35] Charvi Rastogi et al.  “Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making”  In _Proc. ACM Hum.-Comput. Interact._ 6.CSCW1 New York, NY, USA: Association for Computing Machinery, 2022  DOI: 10.1145/3512930
  * [36] Larry V Hedges  “What are effect sizes and why do we need them?”  In _Child development perspectives_ 2.3 Wiley Online Library, 2008, pp. 167–171
  * [37] Maia Jacobs et al.  “How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selection”  In _Translational psychiatry_ 11.1 Nature Publishing Group UK London, 2021, pp. 108
  * [38] Philipp Tschandl et al.  “Human–computer collaboration for skin cancer recognition”  In _Nature Medicine_ 26.8 Nature Publishing Group US New York, 2020, pp. 1229–1234
  * [39] Ekaterina Jussupow, Kai Spohrer, Armin Heinzl and Joshua Gawlitza  “Augmenting medical diagnosis decisions? An investigation into physicians’ decision-making process with artificial intelligence”  In _Information Systems Research_ 32.3 INFORMS, 2021, pp. 713–735
  * [40] Ziyao He, Yunpeng Song, Shurui Zhou and Zhongmin Cai  “Interaction of Thoughts: Towards Mediating Task Assignment in Human-AI Cooperation with a Capability-Aware Shared Mental Model”  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , CHI ’23  <conf-loc>, <city>Hamburg</city>, <country>Germany</country>, </conf-loc>: Association for Computing Machinery, 2023  DOI: 10.1145/3544548.3580983
  * [41] Garston Liang, Jennifer F Sloane, Christopher Donkin and Ben R Newell  “Adapting to the algorithm: How accuracy comparisons promote the use of a decision aid”  In _Cognitive research: principles and implications_ 7.1 Springer, 2022, pp. 14
  * [42] Andrea Papenmeier, Dagmar Kern, Gwenn Englebienne and Christin Seifert  “It’s Complicated: The Relationship between User Trust, Model Accuracy and Explanations in AI”  In _ACM Trans. Comput.-Hum. Interact._ 29.4 New York, NY, USA: Association for Computing Machinery, 2022  DOI: 10.1145/3495013
  * [43] Michael Borenstein, Larry V Hedges, Julian PT Higgins and Hannah R Rothstein  “Introduction to meta-analysis”  John Wiley & Sons, 2021
  * [44] Lauren Wilcox, Robin Brewer and Fernando Diaz  “AI Consent Futures: A Case Study on Voice Data Collection with Clinicians”  In _Proceedings of the ACM on Human-Computer Interaction_ 7.CSCW2 ACM New York, NY, USA, 2023, pp. 1–30
  * [45] Elise Karinshak, Sunny Xun Liu, Joon Sung Park and Jeffrey T Hancock  “Working with AI to persuade: Examining a large language model’s ability to generate pro-vaccination messages”  In _Proceedings of the ACM on Human-Computer Interaction_ 7.CSCW1 ACM New York, NY, USA, 2023, pp. 1–29
  * [46] Veera Vimpari, Annakaisa Kultima, Perttu Hämäläinen and Christian Guckelsberger  ““An Adapt-or-Die Type of Situation”: Perception, Adoption, and Use of Text-to-Image-Generation AI by Game Industry Professionals”  In _Proceedings of the ACM on Human-Computer Interaction_ 7.CHI PLAY ACM New York, NY, USA, 2023, pp. 131–164
  * [47] Xingyu ’Bruce’ Liu et al.  “Experiencing Visual Captions: Augmented Communication with Real-time Visuals using Large Language Models”  In _Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ , UIST ’23 Adjunct  , San Francisco, CA, USA,: Association for Computing Machinery, 2023  DOI: 10.1145/3586182.3615978
  * [48] Eunkyung Jo, Daniel A Epstein, Hyunhoon Jung and Young-Ho Kim  “Understanding the benefits and challenges of deploying conversational AI leveraging large language models for public health intervention”  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , 2023, pp. 1–16
  * [49] Savvas Petridis et al.  “Anglekindling: Supporting journalistic angle ideation with large language models”  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , 2023, pp. 1–16
  * [50] Maurice Jakesch et al.  “Co-writing with opinionated language models affects users’ views”  In _Proceedings of the 2023 CHI conference on human factors in computing systems_ , 2023, pp. 1–15
  * [51] Piotr Mirowski, Kory W Mathewson, Jaylen Pittman and Richard Evans  “Co-writing screenplays and theatre scripts with language models: Evaluation by industry professionals”  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , 2023, pp. 1–34
  * [52] Shakked Noy and Whitney Zhang  “Experimental evidence on the productivity effects of generative artificial intelligence”  In _Science_ 381.6654 American Association for the Advancement of Science, 2023, pp. 187–192
  * [53] Paul Baier, David DeLallo and John J. Sviokla  “Your Organization Isn’t Designed to Work with GenAI” Section: Technology and analytics  In _Harvard Business Review_ , 2024  URL: https://hbr.org/2024/02/your-organization-isnt-designed-to-work-with-genai
  * [54] Andres Campero et al.  “A test for evaluating performance in human-computer systems” (see Supplementary Information, Section S1.2)  In _arXiv preprint arXiv:2206.12390_ , 2022
  * [55] David Moher et al.  “Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement”  In _Annals of internal medicine_ 151.4 American College of Physicians, 2009, pp. 264–269
  * [56] Matthew Groh, Ziv Epstein, Chaz Firestone and Rosalind Picard  “Deepfake detection by human crowds, machines, and machine-informed crowds”  In _Proceedings of the National Academy of Sciences_ 119.1 National Acad Sciences, 2022, pp. e2110013119
  * [57] Heliodoro Tejeda, Aakriti Kumar, Padhraic Smyth and Mark Steyvers  “AI-assisted decision-making: A cognitive modeling approach to infer latent reliance strategies”  In _Computational Brain & Behavior_ 5.4 Springer, 2022, pp. 491–508
  * [58] Julian PT Higgins, Tianjing Li and Jonathan J Deeks  “Choosing effect measures and computing estimates of effect”  In _Cochrane handbook for systematic reviews of interventions_ Wiley Online Library, 2019, pp. 143–176
  * [59] Ankit Rohatgi  “Webplotdigitizer: Version 4.5”  In _URL https://automeris. io/WebPlotDigitizer_ 411, 2020
  * [60] Larry V Hedges  “Distribution theory for Glass’s estimator of effect size and related estimators”  In _journal of Educational Statistics_ 6.2 Sage Publications Sage CA: Thousand Oaks, CA, 1981, pp. 107–128
  * [61] Larry V Hedges and Ingram Olkin  “Statistical methods for meta-analysis”  Academic press, 2014
  * [62] Wim Van den Noortgate, José Antonio López-López, Fulgencio Marín-Martínez and Julio Sánchez-Meca  “Meta-analysis of multiple outcomes: A multilevel approach”  In _Behavior research methods_ 47 Springer, 2015, pp. 1274–1294
  * [63] Mike W-L Cheung  “Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach.”  In _Psychological methods_ 19.2 American Psychological Association, 2014, pp. 211
  * [64] Larry V Hedges, Elizabeth Tipton and Matthew C Johnson  “Robust variance estimation in meta-regression with dependent effect size estimates”  In _Research synthesis methods_ 1.1 Wiley Online Library, 2010, pp. 39–65
  * [65] Julian PT Higgins and Simon G Thompson  “Quantifying heterogeneity in a meta-analysis”  In _Statistics in medicine_ 21.11 Wiley Online Library, 2002, pp. 1539–1558
  * [66] Wolfgang Viechtbauer and Mike W-L Cheung  “Outlier and influence diagnostics for meta-analysis”  In _Research synthesis methods_ 1.2 Wiley Online Library, 2010, pp. 112–125
  * [67] Jonathan AC Sterne and Matthias Egger  “Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis”  In _Journal of clinical epidemiology_ 54.10 Elsevier, 2001, pp. 1046–1055
  * [68] Jaime L Peters et al.  “Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry”  In _Journal of clinical epidemiology_ 61.10 Elsevier, 2008, pp. 991–996
  * [69] Matthias Egger, George Davey Smith, Martin Schneider and Christoph Minder  “Bias in meta-analysis detected by a simple, graphical test”  In _Bmj_ 315.7109 British Medical Journal Publishing Group, 1997, pp. 629–634
  * [70] Colin B Begg and Madhuchhanda Mazumdar  “Operating characteristics of a rank correlation test for publication bias”  In _Biometrics_ JSTOR, 1994, pp. 1088–1101
  * [71] Naomi Altman and Martin Krzywinski  “Analyzing outliers: Influential or nuisance?”  In _Nature Methods_ 13.4 Nature Publishing Group, 2016, pp. 281–283
  * [72] Wolfgang Viechtbauer  “Conducting meta-analyses in R with the metafor package”  In _Journal of Statistical Software_ 36.3, 2010, pp. 1–48  DOI: 10.18637/jss.v036.i03

## References

  * [73] Matthew J Page et al.  “The PRISMA 2020 statement: an updated guideline for reporting systematic reviews”  In _Bmj_ 372 British Medical Journal Publishing Group, 2021
  * [74] Julian PT Higgins, Tianjing Li and Jonathan J Deeks  “Choosing effect measures and computing estimates of effect”  In _Cochrane handbook for systematic reviews of interventions_ Wiley Online Library, 2019, pp. 143–176
  * [75] Ankit Rohatgi  “Webplotdigitizer: Version 4.5”  In _URL https://automeris. io/WebPlotDigitizer_ 411, 2020
  * [76] Jacob Cohen  “Statistical Power Analysis for the Behavioral Sciences”  Academic press, 2013
  * [77] Vivian Lai et al.  “Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies”  In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’23  <conf-loc>, <city>Chicago</city>, <state>IL</state>, <country>USA</country>, </conf-loc>: Association for Computing Machinery, 2023, pp. 1369–1385  DOI: 10.1145/3593013.3594087
  * [78] Min Hun Lee et al.  “A Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment”  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , CHI ’21  <conf-loc>, <city>Yokohama</city>, <country>Japan</country>, </conf-loc>: Association for Computing Machinery, 2021  DOI: 10.1145/3411764.3445472
  * [79] Patrick Hemmer et al.  “Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction”  In _Proceedings of the 28th International Conference on Intelligent User Interfaces_ , IUI ’23  Sydney, NSW, Australia: Association for Computing Machinery, 2023, pp. 453–463  DOI: 10.1145/3581641.3584052
  * [80] Vivian Lai et al.  “An Exploration of Post-Editing Effectiveness in Text Summarization”  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ Seattle, United States: Association for Computational Linguistics, 2022, pp. 475–493  DOI: 10.18653/v1/2022.naacl-main.35

appendix