# A Systematic Review on Reproducibility in Child-Robot Interaction

Micol Spitale<EMAIL_ADDRESS>0000-0002-3418-1933 University of CambridgeUK
,  Rebecca Stower<EMAIL_ADDRESS>KTH Royal Institute of TechnologySweden ,
Elmira Yadollahi<EMAIL_ADDRESS>KTH Royal Institute of TechnologySweden ,
Maria Teresa Parreira<EMAIL_ADDRESS>Cornell UniversityUSA ,  Nida Itrat
Abbasi<EMAIL_ADDRESS>University of CambridgeUK ,  Iolanda Leite
<EMAIL_ADDRESS>KTH Royal Institute of TechnologySweden and  Hatice Gunes
<EMAIL_ADDRESS>University of CambridgeUK

(2023)

###### Abstract.

Research reproducibility – i.e., rerunning analyses on original data to
replicate the results – is paramount for guaranteeing scientific validity.
However, reproducibility is often very challenging, especially in research
fields where multi-disciplinary teams are involved, such as child-robot
interaction (CRI). This paper presents a systematic review of the last three
years (2020-2022) of research in CRI under the lens of reproducibility, by
analysing the field for transparency in reporting. Across a total of 325
studies, we found deficiencies in reporting demographics (e.g. age of
participants), study design and implementation (e.g. length of interactions),
and open data (e.g. maintaining an active code repository). From this
analysis, we distil a set of guidelines and provide a checklist to
systematically report CRI studies to help and guide research to improve
reproducibility in CRI and beyond.

survey, child-robot interaction, reproducibility, replication, transparency

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Human-centered computing HCI
theory, concepts and models††ccs: Human-centered computing Empirical studies
in HCI

##  1\. Introduction

The replication crisis – also referred to as a “crisis of confidence” – refers
to a series of events circa 2015 that called into question the scientific
validity of many psychological studies (Jost et al., 2020), due to the
difficulties in replicating apparently well-established psychological
findings, as well as the differences in results between original and
replicated studies. Although initially emerging in psychology, many of the
concerns raised, such as lack of open access to data, materials, and/or
experimental design apply also to other (social) sciences. Among these are
both Human Robot Interaction (HRI) and its related sub-field of Child Robot
Interaction (CRI), where social and psychological relationships between humans
and robots are often the focus of the research. Given its novelty and rapidly
evolving progress, CRI in particular suffers from fragmented and heterogeneous
literature, varying research goals, and a lack of standardised methods and
metrics.

Recent efforts have brought forth conversations related to replication
specifically within CRI (Stower et al., 2023, 2021), with authors appealing
for more works that address the main challenges in HRI with children whilst
still ensuring high-quality reporting and data sharing. However, clear open
science guidelines on reproducibility in HRI and related sub-fields are still
missing. Consequently, the goal of this review is to systematically evaluate
CRI works in the context of replicability. We aim to both provide an overview
of the field as well as develop guidelines for CRI researchers.

In the following sections, we discuss both HRI and CRI in the context of the
replication crisis. This is followed by the systematic review methodology
(Section 2) and presentation and discussion of the results (Section 3). From
these, we developed a set of guidelines specifically tailored for CRI
researchers. Finally, we discuss open challenges and future directions for the
field of CRI as a whole (Section 5).

###  1.1. Replication Crisis

The replication crisis was initially sparked in 2015 when a team of
psychologists, led by Brian Nosek, published a paper in Science that provided
evidence of a systemic issue with the reproducibility of psychological studies
(Nosek et al., 2015). Specifically, they found that, across 100 studies, only
40% were able to have their findings reproduced. The remaining experiments
either did not replicate or produced mixed results. In addition, the
replications that did succeed had less robust results than the original
studies. In comparison to studies with weaker experimental findings, those
that tended to replicate had more results that were highly significant
originally. Since then, many efforts have been made to improve reproducibility
in psychology, including preregistering studies, a priori power analyses (with
the goal of encouraging larger sample sizes), and increased sharing of
research materials (Nosek et al., 2022). However, those efforts have not yet
been extended to other related fields such as human-robot interaction, where
reproducibility is still very challenging.

####  1.1.1. Replication in HRI

Within the HRI literature, many researchers have already brought forth
reflections and recommendations to address the issues of replicability
(reconducting the same experiment) and reproducibility (rerunning analyses on
original data to replicate the results) (Lücking et al., 2018; Hoffman and
Zhao, 2020; Gunes et al., 2022; Leichtmann et al., 2022; Cordero et al.,
2022). These studies highlight that ensuring reproducibility is essential for
building a strong knowledge base, identifying best practices, and fostering
the development of effective and safe robotic systems. However, translating
these recommendations into practice remains a strong barrier to much HRI
research.

One of the main factors that hinders reproducibility in HRI is the lack of
transparent, rigorous reporting (Baxter et al., 2016; Veling and McGinn, 2021;
Leichtmann et al., 2022). Recently, Gunes et al. (2022) published a review on
reproducibility practices in HRI. Their review identifies technical
challenges, research biases, and social systematic biases which can emerge in
HRI research and pose threats to reproducibility. Technical challenges
include, for example, the cost of commercially available robot platforms, and
barriers to code/software sharing. Research biases discussed are null-
hypothesis significance testing, sample size bias, positive results bias, and
physical robot bias. Social systemic biases are classism bias, racial, ethnic,
and cultural sampling (i.e., WEIRD populations), and robot morphology biases.
They outline some of the practices that are being taken to address these
issues, but stress that there is still much room for improvement within the
field.

Leichtmann et al. (2022) also investigated the main contributors and potential
solutions for the “replicability crisis” (Collaboration, 2015) in HRI. They
mention the effect of publication bias, which is rooted in the scientific
system as the “search for novelty”. The emphasis towards publishing only
significant and/or novel research findings provides little incentive for
researchers to engage in replicability/reproducibility studies, while also
leading to questionable research practices (e.g., p-hacking) (Romero, 2019).
Other causes include low-powered studies (due to small sample sizes and/or
overly complex research designs) and the lack of a research framework that can
be utilised across the field. Such lack of theory is reflected in disagreement
even about what constitutes a successful replication (Scheel et al., 2021).

Nonetheless, recently a number of researchers have been providing positive
examples of the importance of replicating HRI studies. Ullman et al. (2021)
presented results of three studies targeted at investigating if being “in the
loop”, i.e., involved with a robot’s actions, increases trust. They found that
the initial, statistically significant effect failed to replicate once
repeated with a larger sample size. Similarly, Irfan et al. (2018) also
discussed the applicability of the replication crisis to HRI. They present
findings where they failed to replicate a social facilitation effect with
robots. They speculated about the reasons for their failures, such as the
small effect sizes of social facilitation, the setting in which experimental
data is collected, or a bias towards publishing only positive results. Other
studies have made some effort in lowering barriers to reproducibility in HRI
by proposing replication studies (Strait et al., 2020). For example, Strait et
al. (2020) conducted a replication study wherein they performed the same
experiment (investigating the joint-simon effect with a NAO robot) at three
different locations. Whilst these works highlight some of the specific
challenges that HRI faces, namely external validity – more studies are needed
to understand just to what extent other variables, such as the experimental
context, are impacting the results.

####  1.1.2. Replication in CRI

This absence of research theory and well-established methods is particularly
pronounced in the field of child-robot interaction (Belpaeme et al., 2013).
CRI has become a rapidly growing research field in recent years, driven by the
potential of robots to support children’s learning (Belpaeme et al., 2013),
development (Yadollahi et al., 2022), and well-being (Abbasi et al., 2022b;
Abbasi et al., 2022a). Children are a diverse and dynamic population, with
wide variations in cognitive and emotional development, as well as individual
differences. This makes designing experiments that are both ecologically valid
and controlled a difficult task (Stower et al., 2023). The relative novelty
and rapid evolution of CRI as a research field contribute to its
reproducibility challenges. Compared to HRI, CRI is still in its early stages,
resulting in fragmented and heterogeneous literature with varying research
goals and practices (Belpaeme et al., 2013). Consequently, whilst CRI inherits
many of the challenges from HRI regarding reproducibility, it is not limited
to them.

First, recruiting the desired number and population for CRI studies can prove
to be especially challenging (Stower et al., 2023), and the inherent
variability and diversity of children as a participant group presents
challenges in designing controlled experiments that yield consistent results.
In fact, children encompass a wide range of ages, developmental stages,
cognitive abilities, and cultural backgrounds, making it difficult to
generalise findings across different populations.

Second, children’s engagement, preferences, and responses to robots can vary
significantly, resulting in contextual dependencies (Chen et al., 2020) that
require careful consideration when attempting to replicate experiments.
Children’s behaviours and responses can be influenced by factors such as the
appearance (Spitale et al., 2020), behaviour (Zhang et al., 2023), and
capabilities of the robot (Beran et al., 2011), as well as the specific task
or activity being performed. Replicating these interactions across different
contexts and settings often necessitates customisation and adaptation of the
robotic platform and experimental protocols, further complicating
reproducibility efforts.

Third, ethical considerations play a critical role in CRI (Langer et al.,
2023), posing additional challenges for reproducibility (Tolksdorf et al.,
2021b). Interactions between robots and children have the potential to impact
children’s attitudes, beliefs, and behaviours. As a result, researchers must
navigate complex ethical issues surrounding privacy, autonomy, and the well-
being of child participants. Adhering to ethical guidelines, obtaining
informed consent from both children and their parents/guardians, and ensuring
participant safety requires careful attention, potentially influencing the
reproducibility of studies in CRI (Baxter et al., 2016). Such ethical
considerations also affect the design stage of the studies, where researchers
face trade-offs between research interests and ethical reflections (e.g., data
collection vs. data privacy) but also constrains the sharing of experimental
data (Baxter et al., 2016).

Lastly, CRI (as HRI), often involves multidisciplinary teams of researchers
from different domains, such as psychology, education, engineering, and
design, which may have different assumptions, theories, and practices. This
multitude of disciplines can introduce diverse perspectives and assumptions
that need to be harmonised for replication efforts (Stower et al., 2021).
Thus, ensuring reproducibility in CRI requires not only technical rigor but
also interdisciplinary collaboration, transparency, and openness to
alternative perspectives (Stower et al., 2023).

###  1.2. Our Contribution

In sum, CRI research faces empirical challenges related to the usability,
deployment, and sustainability of long-term interactions, as well as ethical
considerations of conducting user studies with children (Tolksdorf et al.,
2021b; Couto et al., 2022). This lack of standardised methods, metrics, and
best practices can hinder the reproducibility of studies. As a result, CRI as
a field is currently lacking specific guidelines and recommendations that may
assist researchers in overcoming the above-mentioned challenges surrounding
CRI and thus foster its reproducibility. To address this gap, we conducted a
systematic review targeted at assessing current reproducibility standards
within CRI, with the goal of proposing a set of guidelines for CRI
researchers.

We conducted a systematic review based on PRISMA guidelines (Moher et al.,
2009) on the last three years of CRI research (2020-2022) to capture the most
recent efforts of the CRI community with respect to reproducibility. We
provide an overview of the field, identifying recent trends in robotic
applications, robots used, the child populations involved, study design
practices etc. This is followed by an analysis of the field in terms of
transparency in reporting. We considered factors such as which study details
were reported and to what extent they were described (e.g., whether the paper
reported only the mean age of the sample but not the age range). Finally, from
our findings, we distilled a set of guidelines for enhancing reproducibility
of CRI studies, while also discussing the main open challenges and future
directions of reproducibility in CRI.

The main contributions of this work are the following:

  * •

We present the first systematic review that focuses on reproducibility and
replication in child-robot interaction.

  * •

We propose a set of guidelines to improve reproducibility in CRI (which can be
further extended to HRI or HCI) by discussing the main challenges that
emerged;

  * •

We provide a checklist to systematically report child-robot interaction
studies to help and guide researchers in improving reproducibility of their
works;

  * •

We draw a research agenda by highlighting the main open challenges in the
field of CRI in terms of reproducibility.

##  2\. Methods

Figure 1. PRISMA schema steps.

This section describes the methodology adopted to conduct our literature
review, including the screening procedure (Section 2.1) and the description of
data extraction and analysis (Section 2.2). We pre-registered this
work111https://osf.io/h8bka/?view_only=04088596f64846229b236cb962d9ea8c via
the OSF platform following all the guidelines of Open Science (Collaboration,
2015).

###  2.1. Screening Procedure

We followed the guidelines described in Nightingale (2009) to conduct this
systematic review. We used the PRISMA framework (Moher et al., 2009) as it
represents the state-of-the-art workflow for systematic reviews and it
includes the following steps (as depicted in Figure 1):

  1. (1)

identification: searching specific databases (how) within a specific time
range (when);

  2. (2)

screening: filtering out manuscripts whose title and abstract do not meet the
eligibility criteria;

  3. (3)

selection: reading the full text of the manuscripts and excluding the ones
that do not meet eligibility criteria;

  4. (4)

inclusion: selecting the final list of the included manuscripts whose data are
extracted.

####  2.1.1. Search Query

For generating the list of manuscripts to include in this survey, we re-used a
list of already screened manuscripts that were extracted via Scopus from
January 2020 to December 2021 in another work. Then, we identified the newest
manuscripts from January 2022 to July 2022 using the same approach (i.e., the
same database and search query). We searched for these manuscripts via the
Scopus database (as in Catania et al. (2022)). The list of terms for the
search included both robot technology and the children population. The
manuscripts were exported on 12/07/2022 and the search query used was the
following:

    TITLE-ABS-KEY  ("school"  OR  "education"  OR  "rehabilitation")
    AND  ("child*"  OR  "teen*") AND  "robot"
    AND  ("study"  OR  "experiment*"  OR  "field study"
    OR  "into the wild") AND  PUBYEAR  =  2022.

After the extraction, we filtered out the duplicates and stored references in
an Excel file, publicly available in the GitHub
repository222https://github.com/becbot/CRI_Replication.

####  2.1.2. Eligibility Criteria

We defined a set of inclusion and exclusion criteria as follows. The papers
were included if:

  * •

they included a social interaction between a child and a robot;

  * •

they reported either physical or online studies, as long as they met all other
criteria;

  * •

they described some evaluation by the child (i.e., not only teachers/ parents/
healthcare providers, etc).

Papers were excluded if:

  * •

they were review/theoretical papers;

  * •

they included only adults (older than 18 years old);

  * •

they were purely technical (e.g., medical robotics).

####  2.1.3. Selection Process

All the manuscripts extracted from the database were first screened by reading
their titles and abstracts. Then we conducted a more in-depth analysis of the
full texts to ensure that the manuscripts met the eligibility criteria. Before
starting the screening process, a random sample of 20 papers was taken from
the 2022 Scopus search and screened by 5 reviewers to ensure consensus on
inclusion/exclusion criteria. All other papers were randomly divided by the
reviewers and screened by one person. Titles and abstracts were screened based
on the eligibility criteria mentioned above. If a reviewer was unsure about a
paper, they flagged it for further discussion between all reviewers. The same
reviewers were then assigned to a different subset of the manuscripts for
full-text screening and data extraction.

We collected a total of 2862 papers using the search query (See Figure 1).
Then, we screened the manuscripts based on their titles and abstracts and
selected a total of 607. After that, we added 524 manuscripts from a previous
screening and reviewed the full texts. This resulted in a final number of 296
manuscripts (reported in the Supplementary Materials). Among the 296
manuscripts, we identified a total of 325 studies (i.e., some of the papers
reported more than one study). When one of the reviewers was uncertain about
the inclusion of a paper, it was assigned to one of the other reviewers to
make a final decision. The final list of the extracted papers can be found in
a publicly available spreadsheet stored in the GitHub
repository333https://github.com/becbot/CRI_Replication.

###  2.2. Data Extraction and Analysis

We identified a set of variables that can inform the transparency/
reproducibility of the included papers and extracted the corresponding data.
Note that some variables were extracted on a paper level (e.g., author
discipline, application context), whereas others were extracted on a study
level (e.g., number of participants, robot model). See Table 1 for a summary
of the extracted/coded variables. We only extracted data from the content of
the main paper and did not cover the supplementary material (if present), so
as to follow the same extraction procedure across all papers (including those
without any extra materials). However, we did note if supplementary material,
appendices, or links to external repositories were included in the paper.

Variable | Category | Type | Example | Level |
---|---|---|---|---|---
General | Authors’ discipline | categorical | Computer science, Psychology, etc | Paper |
Country | categorical | USA, Finland, Japan, etc. | Study |
Open access | categorical | Yes, No | Paper |
Ethics approval | categorical | Yes, No | Paper |
Consent | categorical | Yes, No | Paper |
| Preregistration | categorical | Yes, No | Paper |
Open Science | Open materials | categorical | Yes, No, Partially available | Paper |
Open measures | categorical | Yes, No, Partially available | Paper |
Open code | categorical | Yes, No, Partially available, N/A | Paper |
Code repository | categorical | Yes, No, Partially available, N/A | Paper |
Open analysis | categorical | Yes, No, Partially available, N/A | Paper |
Open data | categorical | Yes, No, Partially available, N/A | Paper |
Study repository | categorical | Yes, No, Partially available, N/A | Paper |
Design | Application scenario | categorical | Education, Therapy, Play, … | Study |
Number of sessions | numerical | 1, 2, 10, … | Study |
Length of single interaction | numerical | 5 minutes, 10 minutes, 23 minutes … | Study |
with robot |  |  |  |
Population | categorical | Neurotypical, ASD, Mixed, … | Study |
Age range | numerical | 4-5 years-old | Study |
Mean age | numerical | 10.26, 6.3, 7.7,… | Study |
Gender reporting | categorical | Yes, No | Study |
Age and gender reporting | categorical | Yes, No | Study |
after exclusion |  |  |  |
Number of participants | numerical | 10, 25, … | Study |
Results reporting | qualitative | Figure, in-text, Not reported | Study |
| Analyses reported | qualitative | Quantitative, Qualitative, Not reported | Study |
Robot/Technology | Robot model | categorical | NAO, Cozmo, Misty, … | Study |
Mode of interaction | categorical | Real-life, Video, Life-mediated | Study |
Robot operation | categorical | Autonomous, Wizard of Oz, | Study |
|  |  | Semi-autonomous |  |

Table 1. List of variables extracted for each paper, the corresponding
category, type, and example of values. N/A refers to ”Not Applicable”.

We first extracted general information about the paper (citation, the
disciplines of the authors, and the country(s) in which the study was
conducted). Author discipline was coded based on the departmental affiliations
reported in the paper. However, we acknowledge that this method is imperfect,
as researchers’ departments do not necessarily have to correspond with their
research backgrounds. Additionally, whilst some papers report departments,
divisions, or even specific labs to which the authors belong, others report
only the university affiliation/s, making it sometimes difficult to identify a
specific discipline.

We identified whether the paper reported obtaining ethical approval (0,1) and
informed consent (0,1). We extracted these two variables separately, as it is
possible in some cases that ethical approval is not required. Whether the
paper was open-access (0,1) and had pre-registered hypotheses (0,1) was also
recorded.

Next, we extracted data related to the experimental design. First, two of the
authors coded the application context (i.e., if there was a specific
application such as healthcare or education that the study fit within). The
different categories of application scenarios were decided post-hoc based on
discussion between the two coders. Where studies could potentially fit
multiple applications (e.g., (Liu et al., 2022) could be both physical
healthcare and education), the most relevant application was chosen, such that
each paper was only coded to one application. The population/s being tested,
and the number of participants were also extracted. If no specific population
was reported (e.g., children with ASD), we coded the population as
“neurotypical”. We also extracted demographic information of the participants,
such as the mean age, minimum and maximum age, and gender breakdown of
participants. We further noted whether age and gender information was reported
prior to or after any exclusion of participants.

As studies in child-robot interaction can vary both in the number of
interactions the child has with the robot and the length of these
interactions, both of these were extracted from the papers. Number of
interactions was coded as “single”, “multiple”, or “not reported”. However, we
found it challenging to code the exact length of the interaction, especially
when multiple interactions with the robot were considered. It was not always
clear whether the lengths reported referred to the entire study session (i.e.,
including questionnaires, introduction, etc.) or only the interaction with the
robot. Additionally, some authors reported the range of the interaction’s
duration (e.g., 15-20 minutes), whilst others reported the average time,
making it even more difficult to consistently code the manuscripts. Hence, we
only coded the length of the interaction as reported (1) or not reported (0).

The type of interaction is also relevant in CRI, as some studies have shown
that video and real-life interactions are not always comparable (Bainbridge et
al., 2011). Consequently, we recorded whether the interaction occurred in
“real life”, was “live-mediated” (e.g., via video call), or used “videos” or
“pictures” of robots as stimuli. Studies in which it was not possible to tell
the mode of interaction were coded as “not reported”.

Similarly, as there is no single definition of what constitutes a “social
robot”, we considered also the robot model being used in the study, as well as
how the robot was controlled during the interaction. The robot model was
extracted directly from the papers. Papers that used their own custom robot
were coded as “custom”, otherwise if no robot model was mentioned this was
classed as “not reported”. For robot operation, we identified five different
modes of operation: “autonomous” (no experimenter needed to operate the robot
during the interaction), “semi-autonomous” (some experimenter input is needed
at various stages), “pre-scripted” (robot operated independently, but
following a defined set of interaction steps), “Wizard of Oz”, where a human
operator completely controlled the robot, and “teleoperated”, which is where
the participant themselves controls some elements of the robot’s behaviour. We
classified the mode of operation based on the authors’ own
definitions/descriptions of the interaction, and if none was provided we coded
this as “not reported”. This means there are potential inconsistencies in
coding between the papers, as different authors may define robot autonomy in
different ways.

Finally, we coded a set of variables relating to the open-science practices of
the papers. First, we checked whether the paper had open materials, i.e., if
it would be possible to replicate the task or interaction, given access to all
of the materials in the information provided444Note that this does not
consider how the materials are actually obtained (e.g., cost, commercial
availability), we only took into account if the materials are reported in
sufficient detail that someone with access to them could replicate the study..
Second, we coded whether the study had open measures, if applicable (that is,
if it provided the materials used such as questionnaires, interview protocols,
coding schemes, etc.). We distinguished between materials (stimuli or
equipment needed to set-up and run the experiment, such as specific tasks or
games) and measures (tools needed to capture and record data). Next, we coded
whether the study had open code (if they provided the code used to run the
robot, for instance) and/or a code repository. We separated these two
variables as, in theory, a study could include all the code necessary to
replicate the study within the paper, or could use entirely pre-existing code
bases without needing their own repository. If the study used picture or video
stimuli and thus could be replicated without needing any code, this was coded
as not applicable. We then coded whether the data were open access, as well as
if there was any data repository associated with the paper. Papers that
explicitly stated that data could not be made available were coded as “not
applicable”. We classified study repositories as “active” (e.g., an external
link to GitHub or OSF), “inactive” (broken or dead links), “supplementary”,
“in-text” (all data was reported in text), or “available upon request” if an
explicit data availability statement was made. Table 1 collects the list of
variables extracted for each paper and the corresponding data type (i.e.,
numerical, categorical, or qualitative), as well as if data were extracted on
the paper or study level.

##  3\. Results and Discussion

Following data extraction, we performed descriptive analyses on the included
manuscripts (frequency, mean and SD) grouped by variable classes (i.e.,
general, open science, design, and robot/technology).

###  3.1. General Overview

We first analysed the geographical distribution of the included papers, the
disciplines of the authors, as well as the application context of the papers.

####  3.1.1. Geographical Distribution

Figure 2. Map of the distribution of the studies across countries worldwide.

Within the surveyed manuscripts, we extracted the countries where each study
was conducted, as depicted in Figure 2. Out of the 256 included manuscripts,
6.08% did not report where their study/s were conducted. The country with the
highest number of reported studies was the US, accounting for 11.49% of all
manuscripts. 3 papers reported data collection across multiple countries. From
a reproducibility standpoint, reporting where the study was conducted
complemented reporting relevant ethical procedures. Reporting which ethical
guidelines were followed also benefits researchers in decomposing what type of
ethical standards were in place and how these could transfer to a different
country’s ethical guidelines.

The majority of studies were conducted within the Western world, confirming
other HRI works which have highlighted the disproportionate number of studies
conducted with WEIRD populations (Gunes et al., 2022). Replication efforts
across different countries, cultures, and demographics are needed to help
understand to what extent findings relevant to CRI are generalisable across
children’s development, versus tied to specific contexts.

####  3.1.2. Author Disciplines

We checked whether the authors’ disciplines were reported in the included
manuscripts (Figure 3). 50.34% of papers fully reported the authors’
disciplines, 22.64% partially reported them (e.g., the disciplines reported
covered just part of the authors), and 27.03% of papers did not report the
disciplines at all. The diversity of the disciplines (Figures 4 and 5) attests
to the interdisciplinary nature of the field. Different disciplines value
different styles of reporting, thus enhancing the inconsistency we observe in
reporting of CRI research. Added to this, a lack of reporting homogeneity
among different fields and publishing venues imposed some limitations in
extracting the authors’ disciplines, confirming the need for better reporting
guidelines targeted at interdisciplinary fields such as HRI, HCI, and CRI.

Figure 3. Reporting of country and discipline information across the included
papers. Figure 4. Frequency distribution of disciplines across the first
authors in the included papers. Disciplines which occurred less than 1% of the
time across the dataset were coded as “Other”.  Figure 5. Frequency
distribution of disciplines across all authors in the included papers.
Disciplines which occurred less than 1% of the time across the dataset were
coded as “Other”.

####  3.1.3. Applications

From the surveyed papers 10 different application scenarios were identified
(Figure 6). 38.18% of the surveyed manuscripts reported an educational
application, 25.34% reported a therapeutic application, 15.88% reported a
social application (e.g., (Martin et al., 2020) investigated children’s
helping behaviour towards a humanoid robot), 9.46% reported a developmental
psychology application (e.g., (Wang et al., 2020) examined infants’
perceptions of cooperation involving robotic agents), 4.05% reported a mental
health application, 2.36% reported a health application, 1.69% reported a
creativity application, 1.35% reported a home application, 1.35% reported a
technical application, and 0.34% reported a design application.

These results highlight a tendency towards high-stakes scenarios, namely
health-related applications (as well as education-based studies). This is yet
another reason why establishing guidelines for reproducibility in CRI is
pertinent, as it requires additional ethical and practical considerations that
need to be planned a priori, but also reported on adequately (Tolksdorf et
al., 2021b). Nevertheless, extracting the applications of the surveyed paper
is not always straightforward, while some posit their research in a dedicated
application, others aim for more application-agnostic reporting.

Figure 6. Frequency distribution of application contexts of the included
papers. Each paper was coded to one application.

###  3.2. Demographics

Next, we considered the sample demographics across the included papers,
including the targeted population/s, age and gender distributions, and sample
size.

####  3.2.1. Population

The majority of the surveyed papers, 64.86%, included neurotypical children.
19.26% reported that they included children with autism spectrum disorder
(ASD) and 7.09% reported that they included a mixture of populations (e.g.,
(Clabaugh et al., 2020) involved both neurotypical children and children with
autism in a one-month in-home human-robot interaction study). 2.36% of the
surveyed papers reported that they included a population with multiple
diagnoses. 1.01% of studies reported that their population included children
with cerebral palsy, 1.01% included children with cognitive impairments, 0.68%
included deaf children, and 0.68% reported including hospitalised children.
The remainder of the studies included one each of the following populations:
children with behavioural disorders, cystic fibrosis, diabetes, down syndrome,
dysgraphia, language disorders, and physical disability (Figure 7).

A high percentage (35.14%) of the surveyed works focused on non-neurotypical
populations or other conditions. Recruiting and working with these populations
is challenging (Stower et al., 2023; Yadollahi et al., 2021), leading
researchers to have smaller participant pools, exclude many participants, and
have shorter-duration studies. All of these factors play into reproducibility
(Leichtmann et al., 2022). With vulnerable populations and high-stakes
application scenarios, the need for literature that appropriately reports on
the recruitment process and challenges, as well as provides a clear
description of the study design and procedure, becomes obvious.

Figure 7. Frequency distribution of child populations targeted in the included
papers. Populations which occurred less than 1% of the time were coded as
”Other”

####  3.2.2. Sample characteristics and reporting

We checked whether the authors reported the data about the age and gender of
the participants before and after exclusion in their studies. 45.85% of the
included studies did not report any screening procedure to include or exclude
the participants in their studies. Of the remaining studies, 21.54% reported a
screening procedure but did not report if the age and gender data included in
the paper referred to the participants after the screening procedure or not.
24.31% reported the age and gender information only after the screening,
whereas 8.31% reported age and gender only before the screening process. Zero
of the surveyed papers reported age and gender both before and after
screening. More details about reporting across the surveyed papers can be seen
in Figure 8.

Figure 8. Percentage of studies reporting age, gender, and sample sizes.

Our results suggest that authors often reported participants’ age and gender
without referring to recruitment strategies and exclusion criteria used. To
promote reproducibility and transparency practices, authors should report
details of the sample after exclusion or mention if participants were
excluded, and why. This is particularly critical in specifying if the reported
age and gender correspond to the participant group before or after the
exclusion.

The median sample size across all studies was $N=25$ $(min=1,max=570)$. 6
studies (1.85%) did not report the sample size. Although appropriate sample
size depends on the specific type of experiment (e.g., case study, randomised
control trial) and design (number of variables, expected effect size), the
overall low median sample size suggests a trend towards likely underpowered
studies within CRI. This is problematic from a reproducibility perspective, as
underpowered studies both reduce the likelihood of a truly significant effect
being detected (Type II error / false negative) but also increase the chance
of a Type I error (false positive) occurring when significant effects are
present (Button et al., 2013). Figures 9 and 10 detail the sample size
distribution.

Figure 9. Distribution of sample sizes across the included studies. Figure 10.
Distribution of sample sizes across studies with $n<100$.

####  3.2.3. Age

Figure 11. Frequency distribution of ages of children in the included papers.

The majority of the studies included in the manuscripts (75.69%) reported
either the age range (minimum and maximum age) or mean age, while 24.31% of
studies did not report either. The median age across all studies which
reported age information was 7.65 y.o., with a mean age of 7.69 y.o., a
maximum age of 17.4 y.o., a minimum age of 0.77 y.o., and a variance of 9.47
years. Figure 11 depicts the frequency distribution of age across all the
studies included in this survey.

####  3.2.4. Gender

Regarding gender, 71.69% of the included studies reported the gender of the
children involved in their studies, while 28.31% did not report their
participants’ gender. Out of the reported ones, 47% of participants were
female and 53% were male. Only one study reported including non-binary
participants. The average ratio of female to male participants, across all
studies, was 0.42 (Figure 12).

Figure 12. Distribution of the male-to-female participant ratio across the
included papers. Higher proportions refer to a higher number of males.

###  3.3. Design of Interactions

We analysed the number and length of interactions with the robot, the number
and type of robots used in the studies, the mode of interaction with the
robot, and how the robot was controlled.

####  3.3.1. Study Design

Regarding the number of interaction sessions, 36.31% of the studies included
in the surveyed papers reported that they conducted studies with multiple
sessions (i.e., more than one interaction), and 56.92% reported that they
conducted single-session studies. 6.77% of the studies in the surveyed
manuscripts did not report the number of sessions with the robot. We then
checked whether the studies reported the length of each session with the
robot. 42.76% of the studies included in the manuscripts did not report the
length of the interaction, compared to 57.23% that did (Figure 13).

Many CRI papers have called for more long-term studies to better assess the
generalisability of CRI findings. Reporting the details of the interaction,
including the number of sessions and the length, therefore allows other
researchers to assess the validity and applicability of the study.
Furthermore, in the particular cases of educational or healthcare scenarios
with children, such information can enable researchers to assess the patterns
of engagement or other outcomes that emerge from the interaction.

In addition, many studies included an “introduction” phase with the robot,
where the robot introduced itself and/or performed a small activity such as a
dance. Reporting such interactions is also important, as it helps understand
how children were primed to interact with the robot. This is also relevant
from an ethical perspective, as such introductions can help to appropriately
calibrate children’s expectations about the robot’s capabilities, as well as
help alleviate any concerns children might have about interacting with the
robot.

Figure 13. Percentage of studies reporting the number and length of
interactions with the robot/s.

####  3.3.2. Robot Models

We extracted the number of robots used in the surveyed papers. 88.62% of
studies reported using one robot, 6.46% reported using two robots, 2.77% used
three robots, 1.23% used four robots, 0.62% reported five robots, while just 1
paper (0.31%) reported using thirteen robots.

The most commonly used robot model was the NAO robot, which was used in 34.50%
of studies, followed by Pepper (6.00%), Cozmo (3.75%), Zeno (2.25%), Robovie
(1.75%) and Furhat (1.50%) (see Figure 14). Jibo, Kaspar, LEGO Mindstorms,
Pleo, and QTRobot also appeared 5 times each (1.25%) across the included
studies. The remaining studies (17.50%) used either other commercial robots
(e.g., MiRo-E) or their own custom robot (see Figure 14 for details). While
only a small percentage (2%) of the works did not report the robot model that
was used, we note that this too is extremely valuable information, as prior
work has shown that studies may not replicate if a different robot model is
used (Irfan et al., 2018).

Figure 14. Frequency distribution of the most commonly used robot models in
the included papers. Robot models which appeared 5 times or less across the
dataset were coded as “Other”.

####  3.3.3. Mode of interaction

Only 3 (0.92%) of the surveyed manuscripts did not report what type of
interaction was included in their studies. Of the remaining papers, 89.85%
reported that the interaction was in real-life, 5.23% reported that the
interaction was through video (i.e., participants were asked to watch a video
of a robot as in (Nijssen et al., 2021) where the authors assessed the effect
of anthropomorphic appearance and affective state attributions by watching
videos), 1.85% reported that the interaction was live-mediated (i.e., virtual
or augmented reality robot’s applications, as in (Shahab et al., 2022), where
a social robot in virtual reality was used with children with high functioning
autism for music education), 0.92% reported using a picture of a robot (i.e.,
children were asked to look at a picture of a robot as in (Oranc and Küntay,
2020), where the authors explored children’s perception of social robots as a
source of information across different domains), 0.62% reported that the
interaction was via a virtual robot (not a physical robot), 1 (0.31%) study
reported that the interaction was carried out in augmented reality, and
another 1 (0.31%) reported that the interaction was hybrid (e.g., (Yang et
al., 2020) combines traditional physical education teaching with a robot
teaching).

Figure 15. Frequency distribution of the mode of interaction with the robot
across the included studies.

####  3.3.4. Robot operation

Across all studies, Wizard of Oz was the most commonly used method for
operating the robot (24.32%), followed by autonomous (20.06%), pre-scripted
(13.98%), teleoperated (7.60%), and semi-autonomous (6.08%). 4 studies (1.22%)
reported using more than one method to operate the robot/s, especially if more
than one type of robot was involved in the study (e.g., a combination of
Wizard of Oz and autonomous, as in (de Souza Jeronimo et al., 2022)). 20.32%
of papers did not report how the robot was controlled, and for another 6.38%
the mode of operation was not applicable (for example, if the study did not
involve a live interaction with the robot). Figure 16 displays these findings,
and Figure 17 provides an overview of the reporting on study design across the
surveyed papers.

Equally as important, a significant number of the surveyed papers did not
report the mode of robot operation (20.92%). Research in highly
interdisciplinary fields can overlook technical contributions, which is why
the CRI field might overlook the need to fully describe the systems used.
However, this might change if better guidelines and reporting standards are
set – prior work on reproducibility (Baxter et al., 2016; Riek, 2012) has also
pointed out robot operation as an important reporting point towards more
reproducible HRI.

Figure 16. Frequency distribution of the mode of operation of the robot/s
across the included studies. Figure 17. Percentage of studies which reported
the robot model, mode of operation, and type of interaction.

###  3.4. Ethical and Open Science Guidelines

Finally, we analysed the adherence of the included papers to current ethical
and open science guidelines, including ethical approval and/or informed
consent, pre-registration, open access, and availability of code, data,
measures, and materials (Figures 18 and 19).

####  3.4.1. Ethical approval and informed consent

A fundamental aspect of guaranteeing open science is the reporting of the
appropriate data protection safeguard measures adopted in compliance with data
protection and ethics (e.g., GDPR for EU countries). 58.11% of papers reported
that their studies were approved by a relevant ethics committee (e.g.,
Institutional Review Board, IRB). 41.22% of the manuscripts did not report any
ethics information, while 2 papers (0.68%) stated that this was not applicable
(e.g., (Serholt et al., 2020) was conducted in Sweden where ethics approval is
not required or mandatory for user studies). Analogously, we also checked the
papers that reported whether they asked participants to fill out and sign
informed consent. 68.92% of the manuscripts reported asking participants
(parents/guardians) to sign an informed consent form, while 31.08% did not
include any consent information. Of the 205 manuscripts that reported asking
for informed consent, 49 did not include a corresponding ethical approval.
Reporting ethical approval and consent is important, not only for readers and
reviewers evaluating these studies but also to promote good practices across
the field.

####  3.4.2. Preregistration and Open Access

Only 3.38% of the 296 manuscripts included in this survey reported that they
pre-registered their works (e.g., using the Open Science Framework), while the
majority (96.62%) did not. Over 296 papers, 128 (43.24%) were open access,
while 168 (56.76%) were not. While most institutions guarantee access to a
large number of literature resources, this might still constitute a barrier
for researchers to conduct informed research studies.

The low number of pre-registered works could be partially explained by a lack
of incentive, as many conferences and journals in HRI do not enforce such
practices as a prerequisite for publication. Additionally, preregistration is
sometimes met with resistance within the HRI community (Gunes et al., 2022),
where some researchers may feel that they will be unfairly penalised for
conducting exploratory analyses with pre-registered study designs. Given this
and the pertinence of piloting studies, maybe preregistration could be
reviewed as a practice in terms of where it chronologically lays on a study
development process – for instance, to be considered as a stage that takes
place after pilot studies, once the study design is finalized.

Figure 18. Percentage of studies which reported ethical approval, informed
consent, open access, and pre-registration.

####  3.4.3. Open science

We then extracted the main open science variables following the European
guidelines555https://research-and-
innovation.ec.europa.eu/strategy/strategy-2020-2024/our-digital-future/open-
science_en: open materials, open measures, open code, and open data (Fig. 19).
Our results showed that 36.49% of the included manuscripts reported the
materials used in their studies, 23.65% partially reported the materials used
(i.e., the paper reported the exercises delivered with instructions without
providing the images of the exercises necessary to accomplish it, e.g., (Yang
et al., 2020)), 39.86% did not report any detail of the materials used.

Figure 19. Percentage of studies which had open materials, open measures, open
code, and open data.

Among the surveyed manuscripts, 58.11%, reported the measures used for running
their analysis and computing their results, 13.51% partially reported them
(e.g., (Tian et al., 2020)), whereas 28.38% did not report the measures they
used. Also, our results showed that only 4.39% of the surveyed manuscripts
provided the code, 1.35% provided the code partially (e.g., (Zhexenova et al.,
2020)), while the majority of the surveyed papers, 88.51%, did not provide the
code used in their studies. For 17 papers (5.74%), the code was not applicable
because their studies did not include any implementation (e.g., (Kewalramani
and Veresov, 2022)).

We also checked whether a code repository was included in the surveyed
manuscripts. 4.41% included a link to a code repository, 1 (0.34%) partially
included this information (i.e., the code link was included but not longer
active, e.g., (Konijn et al., 2022)). Most papers, 69.15%, did not include any
repository.

Most of the surveyed papers (83.11%) did not provide (access to) the data
collected or used in the study. 8.45% provided full access to the data (either
in text, in supplementary materials, or an external data file/repository),
whereas 4.39% only provided partial data (e.g.,(Tolksdorf et al., 2021a)).
2.70% among the surveyed manuscripts reported that the data could be made
available upon request, while for 4 papers (1.35%), this information was not
applicable (for example, the study used data that could not be made available
due to ethical or privacy constraints).

We also checked how many manuscripts included a study repository or other
supplemental materials. We found that most of the studies (224, 75.38%) did
not report any study repository in their manuscript. For 21 manuscripts
(7.09%), the study repository reporting was not applicable (i.e., they haven’t
implemented any code in their studies). 6.78% of the manuscripts reported an
active study repository, 5.07% added a statement that the data can be made
available upon request, 3.72% included supplementary materials, 1.69% reported
an inactive link to the study repository.

A very significant portion of the works included in our analysis did not have
open materials, i.e. did not provide the different components used for the
study (survey questions, experiment code, study data, analysis code, etc. –
for instance, only 4.39% of papers provided code). These are important
elements of a study that are prominent for not only thorough evaluation of the
study methods but also for reproducibility and replicability. We note, as
Baxter et al. (2016) mention, that CRI poses particular challenges when it
comes to data sharing, namely identifiable data. Children’s identity
protection and right to privacy should be a top priority for all researchers,
and thus any reproducibility good practices must never override these
considerations. Nonetheless, strategies for bettering the sharing of study
materials should be discussed, developed, and implemented whenever possible.
The lack of guidelines in reporting such data in turn posited challenges and
limitations in extracting the variables described in Data Extraction and
Analysis section.

##  4\. Guidelines to improve reproducibilty

Child-robot interaction tackles similar obstacles to replicate published
studies as its parent field, human-robot interaction (Baxter et al., 2016).
However, CRI is a more recent field, which thus suffers from the problem of a
lack of definitions and standardised guidelines. To add to this, the
interdisciplinarity of the field results in a multitude of not only
application contexts but also author disciplines. Different disciplines value
different styles of reporting, thus enhancing the inconsistency we observe in
reporting in CRI works. Hence, we conducted this systematic review and the
findings of this work enable us to distill a set of guidelines to improve
reproducibility as follows.

Our results suggest that most of the surveyed papers reported investigating
applications related to health (e.g., (Neerincx et al., 2021; Beyer-Wunsch and
Reichstein, 2020)) and education (e.g., (Lopez-Caudana et al., 2020; de Souza
Jeronimo et al., 2022)) for the target group of CRI, i.e., children, as
depicted in Figure 6. In those application scenarios, literature (Mott et al.,
2022) highlights the importance of ethical considerations for such a
population due to concerns about trust and deception, data privacy and
security. Langer et al. (2023) also pinpoints the necessity of ensuring
transparency in consent data usage, limitations, and destruction (Livingstone
et al., 2018). However, the surveyed papers still lack a comprehensive
reporting of ethical approval, informed consent and preregistration as
displayed in Figure 18. Therefore, we recommend reporting the ethical
approval, and when possible, preregistration of studies. Detailed reporting of
how consent was acquired (for example, by providing the consent form in a
study repository) is also pivotal.

We found that most of the surveyed papers (e.g., (Ismail et al., 2021; Calvo-
Barajas et al., 2021)) reported details about the system and the study design,
but most of them had small sample sizes (with a median of 25 participants), as
shown in Figure 9. Thus, replication studies are called for. In order to
replicate a study, however, researchers require an in-depth description of the
system used (robot model and mode of operation, the deployment scenario –
e.g., what did the room look like, who was in the room). In line with
application settings and vulnerable population groups, past works (DiPaola et
al., 2023; Hoffman and Zhao, 2020) suggested that it is not enough to briefly
describe the system and study design but their reporting must be thoughtful.
Fraune et al. (2022) also presented lessons learned for designing HRI studies
by highlighting that researchers should properly document the task and the
entire experiment in their papers so others can replicate the study with the
same conditions and a similar environment. Therefore, we recommend that
robotic systems and study design should be described in detail. Ideally, code
for robot operation and any other aspects of the system should be freely
available for any researcher attempting to replicate the study.

We found that not all the surveyed papers reported information about the
demographics of children included in their studies (see Figure 8). Literature
(Cordero et al., 2022; Fraune et al., 2022) highlights the importance of
thorough reporting on demographics (age, gender, and inclusion/exclusion
criteria). For example, Fraune et al. (2022) reported that children in
different developmental stages may display different behaviour when
interacting with robots. Another consideration is the fact that the range of
age is extremely relevant in children: when designing an CRI study, a 2-year
age range could be too broad, due to differences in cognitive developmental
stages. Information about demographics allows researchers to understand the
results within the child developmental context but also provides the
groundwork for comparing replication studies (and why, sometimes, results
might not replicate (Ullman et al., 2021; Leichtmann et al., 2022)). We then
recommend that recruitment methods and exclusion criteria should also be
provided.

Our results show that the surveyed papers lacked in providing materials,
measures and especially open code and data, as depicted in Figure 19. As
mentioned, past works (Koole and Lakens, 2012; Stower et al., 2021) stressed
that providing study materials is paramount for achieving reporting
transparency, promoting reproducibility, and incentivising replicability.
Thus, we recommend researchers maintain study repositories and provide as many
resources as possible (deployment and analysis code, surveys, forms, etc.).

Following our findings in Section 3 and similarly to prior work (Gunes et al.,
2022; Baxter et al., 2016), in Table 2 we provide a checklist that authors can
use to self-assess the quality of their reporting. We note that manuscript
length restrictions may exist, for which other strategies can be used
(supplementary material, online repository, etc.). Publishing venues can
exclude these sections from the page limit; in any case, this information
should be easily traceable for readers.

Table 2. Checklist to systematically report child-robot interaction studies in
particular, that can be also extended to user studies with robots in general.

Study Stage | List of items | We recommend reporting..
---|---|---
Study |  $\square$ Preregistration | .. if you have preregistered the study (before or after piloting)
Conceptualisation |  $\square$ Ethical approval | .. whether the study was approved by an ethics committee
|  $\square$ Sample Size | .. the target number of participants aiming to be recruited,
|  | and how this number was obtained (e.g, power-analysis,
|  | pilot studies, previous work)
System Design |  $\square$ Robot Model | .. which robot platform was used in the study (e.g., Nao)
|  $\square$ Mode of interaction | .. the modality of the interaction (e.g., real-life, video)
|  | between the human and the robot
|  $\square$ Robot operation | .. the autonomy level of the robot, both in terms of
|  | perception and generation (e.g., autonomous, wizard-of-Oz)
|  $\square$ Open code | .. the code used for implementing the interaction with
|  | the robot, specifying the study repository if applicable.
Study Design |  $\square$ Pilot tests | .. if applicable, how pilot studies were carried out
and Deployment |  | and results/conclusions from these studies
|  $\square$ Consent | .. how informed consent was provided (ideally, share the
|  | consent form used for the study)
|  $\square$ Recruitment criteria | .. which criteria and methods were used for recruitment
|  $\square$ Population | .. the population involved in the study
|  | (neurotypical, neurodivergent, etc.)
|  $\square$ Number of participants | .. the number of participants involved in the study
|  $\square$ Age | .. the age range and mean age of the participants
|  | (if applicable, provide this for each condition of study)
|  $\square$ Gender | .. the genders of participants involved in the study
|  | (if applicable, provide this for each condition of study)
|  $\square$ Screening information | .. whether the participants were screened and the
|  | information about age and gender before and after
|  | the screening exclusion
|  $\square$ Experiment setting | .. what was the setting when running the study
|  | (what the room looked like, who was present, etc.)
|  $\square$ Number of sessions | .. the number of sessions during which the participants
|  | interacted with the robot
|  $\square$ Length of interaction(s) | .. the length (i.e., duration) of each interaction with the robot
|  $\square$ Debriefing | .. how participants were debriefed about
|  | study goals and research questions
|  $\square$ Open materials and measures | .. the measures used for collecting the data
|  | (surveys, task materials, interview questions, etc.)
Analysis |  $\square$ Open analysis | .. the analysis conducted for analysing the data collected
|  | (if applicable, provide script used)
|  $\square$ Open data | .. the data collected if fully anonymised and
|  | in compliance with privacy rights, or the reason why
|  | these data cannot be openly available

##  5\. General Discussion

This section describes from a high-level the implications of our review in
terms of reporting, transparency, and reproducibility, and the open challenges
and future directions for reproducibility in CRI.

###  5.1. Reporting, Transparency, and Reproducibility

Challenges specific to reproducibility in CRI are found in our analysis.
Namely, studies in CRI often involve vulnerable population demographics (non-
neurotypical groups), and there is a tendency towards high-stakes scenarios.
This requires additional considerations when designing a study. The study’s
motivation and design need to be carefully planned and controlled, and
participants and their guardians need to be adequately informed about all the
foreseeable risks and benefits of their participation. Reporting ethical
approval and consent is important, not only for readers and reviewers
evaluating these studies but also to promote good practices across the field
(Tolksdorf et al., 2021b). Further, with vulnerable populations and high-
stakes application scenarios, the need for literature that appropriately
reports on the recruitment process and challenges, as well as provides a clear
description of the study design and procedure, becomes obvious.

The interdisciplinarity of researchers and research questions in CRI is also
an emergent finding from this work. This is reflected in inconsistent
reporting, which constituted a challenge even for the data extraction method
used in this analysis. However, the lack of standardised methods and metrics,
and interdisciplinary collaborations can be seen as not a limitation but
rather as an opportunity to build theory and good practices in the field, that
can be communicated and applied effectively (Chrysostomou et al., 2017; Stower
et al., 2023).

While reporting is one of the primary elements for promoting reproducibility,
we recognise that other constraints may arise in realistic settings. The lack
of standardised questionnaires and metrics can be potentially addressed by
Institutional Review Boards (IRBs) or ethics review committees of the
institutions of the authors, although this might be impractical in
interdisciplinary collaborations (as is often the case with CRI), as the
guidelines for surveys can conflict. Another factor hindering extensive
reporting is also manuscript length restrictions, which are included in the
paper submission guidelines for most conferences and other publishing venues.
However, we identify strategies that can help address these restrictions,
namely including these data as supplementary material, or creating (and
actively maintaining) study repositories. Below, we reflect on the
implementation of better practices, from the perspective of different
stakeholders.

###  5.2. Open Challenges and Future Directions

Enforcing appropriate reporting in papers to enable reproducibility in the
field of CRI requires a collective effort involving various stakeholders.
While each entity plays a role, it is essential to recognise the shared
responsibility among publishing venues, institutions, and authors (Koole and
Lakens, 2012; Munafò et al., 2017).

Publishing venues hold a significant responsibility in ensuring the integrity
and reproducibility of research. They can establish and enforce guidelines
that promote transparent and rigorous reporting practices. Journals and
conference proceedings can require authors to provide detailed methodological
information, access to datasets, and source code when applicable.
Additionally, they can encourage replication studies and require authors to
address limitations and potential sources of variation explicitly. We provide
the example of the ACM Interaction Design and Children conference (IDC) (IDC,
[n.d.]), which requires authors to submit papers with a “Selection and
Participation of Children” section in which “the authors of the paper should
describe how children were selected (…), what consent processes were followed
(…), how they were treated, how data sharing was communicated, and especially
any additional ethical considerations”. Analogously the Affective Computing
and Intelligence Interaction conference (ACII) has introduced a mandatory
section on “Ethical Statement” since 2022 because “a large diversity of
research done in the Affective Computing community today also means that there
is a large diversity of ethical issues. Not every project will encounter every
issue, but almost all projects will encounter some issues”.

Institutions also play a vital role in fostering reproducibility. They can
promote research integrity and provide support and resources for researchers
to conduct reproducible studies. Institutions can offer training and workshops
on transparent research practices, data management, and statistical analysis.
They can also encourage collaborative efforts among researchers to replicate
and validate findings. Furthermore, institutions can establish policies that
incentivise and reward researchers for practicing open science and
reproducibility, such as recognising open access publications or replication
studies in evaluation and promotion processes.

Ethical approval procedures to conduct studies with human participants or
children are in place in the majority of countries and institutions. While
each country, university, or department has its own procedures and
requirements, adhering to these standards and reporting them can not only
provide other researchers with the knowledge that the studies were ethically
approved but also removes barriers to reproducing the study, given that it
adheres with the ethical procedures in other countries. In the case of
vulnerable populations, particularly children with any type of difficulties
(e.g. autism or dysgraphia) being able to reproduce the studies goes beyond
notions of reproducibility to the realms of helping them through utilising
recent scientific advances.

Finally, authors themselves hold direct responsibility for reporting their
research in a manner that allows for reproducibility. We should strive to
provide clear and comprehensive descriptions of their methods, materials, and
data collection procedures. Authors can adhere to reporting guidelines
specific to CRI or follow broader guidelines like the Transparency and
Openness Promotion (TOP) guidelines 666https://www.cos.io/initiatives/top-
guidelines. Additionally, where possible, authors should openly share their
datasets, code, and any other relevant research materials to enable
independent verification and replication of their work.

##  6\. Conclusion

Ensuring appropriate reporting and reproducibility in science requires a
collective effort. The current review investigates how the field of child-
robot interaction fares on different reproducibility metrics with an
additional overview of the field by analysing 225 CRI papers published from
2020-2022. Our findings revealed that considering the targeted population
(i.e., children) and high-stakes scenarios (e.g. healthcare), the field has
significant progress yet to make. We identified different areas where
reporting was not comprehensive enough to enable reproducible work. Based on
these, we developed a set of guidelines that could help researchers in the
field to reflect on various stages of their study design, development, and
reporting to ensure contribution to the reproducibility culture. Publishing
venues, institutions, and authors all play crucial roles in establishing and
promoting standards and practices that support transparent and replicable
research. By working together, these stakeholders can contribute to advancing
the field of CRI and fostering a scientific community committed to rigorous
and reliable research practices.

## Acknowledgments

This work was partially funded by the S-FACTOR project from NordForsk, Digital
Futures Research Center, and Vinnova Competence Center for Trustworthy Edge
Computing Systems and Applications at KTH. M.S. and H.G. are supported by the
EPSRC/UKRI under grant ref. EP/R030782/1 (ARoEQ). We thank Dr. Mart Couto and
Dr. Shruti Chandra for sharing the list of their screened paper on child-robot
interaction from January 2020 to December. We would also like to thank the
organisers of the RPL Summer School, for facilitating the initial conception
of this work.

## Author Contributions

M.S. and R.S. conceived of the initial study. M.S., R.S, E.Y, M.P, and N.A.,
performed paper screening and data extraction. R.S. analysed the data. M.S.
and R.S. wrote the methods and results. M.S. wrote the guidelines and
checklist for future CRI researchers. E.Y. and M.P. wrote the introduction and
general discussion sections. H.G. and I.L. oversaw the project and provided
guidance. All authors proofread the manuscript.

## References

  * (1)
  * IDC ([n.d.]) [n.d.].  ACM Interaction Design and Children 2023.  https://idc.acm.org/2023/full-short-papers/.  Accessed: 2023-05-17.
  * Abbasi et al. (2022a) Nida Itrat Abbasi, Micol Spitale, Joanna Anderson, Tamsin Ford, Peter B Jones, and Hatice Gunes. 2022a.  Can robots help in the evaluation of mental wellbeing in children? An empirical study. In _2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 1459–1466.
  * Abbasi et al. (2022b) Nida Itrat Abbasi, Micol Spitale, Peter B Jones, and Hatice Gunes. 2022b.  Measuring mental wellbeing of children via human-robot interaction: Challenges and opportunities.  _Interaction Studies_ 23, 2 (2022), 157–203.
  * Bainbridge et al. (2011) Wilma A Bainbridge, Justin W Hart, Elizabeth S Kim, and Brian Scassellati. 2011.  The benefits of interactions with physically present robots over video-displayed agents.  _International Journal of Social Robotics_ 3 (2011), 41–52.
  * Baxter et al. (2016) Paul Baxter, James Kennedy, Emmanuel Senft, Severin Lemaignan, and Tony Belpaeme. 2016\.  From characterising three years of HRI to methodology and reporting recommendations. In _2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, 391–398.
  * Belpaeme et al. (2013) Tony Belpaeme, Paul Baxter, Joachim De Greeff, James Kennedy, Robin Read, Rosemarijn Looije, Mark Neerincx, Ilaria Baroni, and Mattia Coti Zelati. 2013.  Child-robot interaction: Perspectives and challenges. In _Social Robotics: 5th International Conference, ICSR 2013, Bristol, UK, October 27-29, 2013, Proceedings 5_. Springer, 452–459.
  * Beran et al. (2011) Tanya N Beran, Alejandro Ramirez-Serrano, Roman Kuzyk, Meghann Fior, and Sarah Nugent. 2011\.  Understanding how children understand robots: Perceived animism in child–robot interaction.  _International Journal of Human-Computer Studies_ 69, 7-8 (2011), 539–550.
  * Beyer-Wunsch and Reichstein (2020) Pia Beyer-Wunsch and Christopher Reichstein. 2020.  Effects of a humanoid robot on the well-being for hospitalized children in the pediatric clinic-An experimental study.  _Procedia Computer Science_ 176 (2020), 2077–2087.
  * Button et al. (2013) Katherine S Button, John PA Ioannidis, Claire Mokrysz, Brian A Nosek, Jonathan Flint, Emma SJ Robinson, and Marcus R Munafò. 2013.  Power failure: why small sample size undermines the reliability of neuroscience.  _Nature reviews neuroscience_ 14, 5 (2013), 365–376.
  * Calvo-Barajas et al. (2021) Natalia Calvo-Barajas, Giulia Perugia, and Ginevra Castellano. 2021.  The effects of motivational strategies and goal attainment on children’s trust in a virtual social robot: A pilot study. In _Interaction Design and Children_. 537–541.
  * Catania et al. (2022) Fabio Catania, Micol Spitale, and Franca Garzotto. 2022\.  Conversational agents in therapeutic interventions for neurodevelopmental disorders: a survey.  _ACM Computing Surveys (CSUR)_ (2022).
  * Chen et al. (2020) Huili Chen, Hae Won Park, Xiajie Zhang, and Cynthia Breazeal. 2020\.  Impact of interaction context on the student affect-learning relationship in child-robot interaction. In _Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction_. 389–397.
  * Chrysostomou et al. (2017) Dimitrios Chrysostomou, Paolo Barattini, Johan Kildal, Yue Wang, Jacopo Fo, Kerstin Dautenhahn, Francois Ferland, Adriana Tapus, and Gurvinder S. Virk. 2017\.  ReHRI’17 - Towards Reproducible HRI Experiments: Scientific Endeavors, Benchmarking and Standardization. In _Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction_ (Vienna, Austria) _(HRI ’17)_. Association for Computing Machinery, New York, NY, USA, 421–422.  https://doi.org/10.1145/3029798.3029800
  * Clabaugh et al. (2020) Caitlyn Clabaugh, Shomik Jain, Balasubramanian Thiagarajan, Zhonghao Shi, Leena Mathur, Kartik Mahajan, Gisele Ragusa, and Maja Matarić. 2020.  Month-long, in-home socially assistive robot for children with diverse needs. In _Proceedings of the 2018 International Symposium on Experimental Robotics_. Springer, 608–618.
  * Collaboration (2015) Open Science Collaboration. 2015\.  Estimating the reproducibility of psychological science.  _Science_ 349, 6251 (2015), aac4716.  https://doi.org/10.1126/science.aac4716 arXiv:https://www.science.org/doi/pdf/10.1126/science.aac4716
  * Cordero et al. (2022) Julia R Cordero, Thomas R Groechel, and Maja J Matarić. 2022\.  A Review and Recommendations on Reporting Recruitment and Compensation Information in HRI Research Papers. In _2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. 1627–1633.  https://doi.org/10.1109/RO-MAN53752.2022.9900744
  * Couto et al. (2022) Marta Couto, Shruti Chandra, Elmira Yadollahi, and Vicky Charisi. 2022.  Child-robot interaction: Design, evaluation, and novel solutions.  , 151–156 pages.
  * de Souza Jeronimo et al. (2022) Bruno de Souza Jeronimo, Anna Priscilla de Albuquerque Wheler, José Paulo G de Oliveira, Rodrigo Melo, Carmelo JA Bastos-Filho, and Judith Kelner. 2022.  Comparing Social Robot Embodiment for Child Musical Education.  _Journal of Intelligent & Robotic Systems_ 105, 2 (2022), 28\.
  * DiPaola et al. (2023) Daniella DiPaola, Vicky Charisi, Cynthia Breazeal, and Selma Sabanovic. 2023.  Children’s Fundamental Rights in Human-Robot Interaction Research: A Systematic Review. In _Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_ (Stockholm, Sweden) _(HRI ’23)_. Association for Computing Machinery, New York, NY, USA, 561–566.  https://doi.org/10.1145/3568294.3580148
  * Fraune et al. (2022) Marlena R. Fraune, Iolanda Leite, Nihan Karatas, Aida Amirova, Amélie Legeleux, Anara Sandygulova, Anouk Neerincx, Gaurav Dilip Tikas, Hatice Gunes, Mayumi Mohan, Nida Itrat Abbasi, Sudhir Shenoy, Brian Scassellati, Ewart J. de Visser, and Takanori Komatsu. 2022.  Lessons Learned About Designing and Conducting Studies From HRI Experts.  _Frontiers in Robotics and AI_ 8 (2022).  https://doi.org/10.3389/frobt.2021.772141
  * Gunes et al. (2022) Hatice Gunes, Frank Broz, Chris S Crawford, Astrid Rosenthal-von der Pütten, Megan Strait, and Laurel Riek. 2022.  Reproducibility in Human-Robot Interaction: Furthering the Science of HRI.  _Current Robotics Reports_ (2022), 1–12.
  * Hoffman and Zhao (2020) Guy Hoffman and Xuan Zhao. 2020.  A Primer for Conducting Experiments in Human–Robot Interaction.  _J. Hum.-Robot Interact._ 10, 1, Article 6 (oct 2020), 31 pages.  https://doi.org/10.1145/3412374
  * Irfan et al. (2018) Bahar Irfan, James Kennedy, Séverin Lemaignan, Fotios Papadopoulos, Emmanuel Senft, and Tony Belpaeme. 2018.  Social psychology and human-robot interaction: An uneasy marriage. In _Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction_. 13–20.
  * Ismail et al. (2021) Luthffi Idzhar Ismail, Fazah Akhtar Hanapiah, Tony Belpaeme, Joni Dambre, and Francis Wyffels. 2021\.  Analysis of attention in child–robot interaction among children diagnosed with cognitive impairment.  _International Journal of Social Robotics_ 13 (2021), 141–152.
  * Jost et al. (2020) Céline Jost, Brigitte Le Pévédic, Tony Belpaeme, Cindy Bethel, Dimitrios Chrysostomou, Nigel Crook, Marine Grandgeorge, and Nicole Mirnig (Eds.). 2020.  _Human-Robot Interaction_. Vol. 12.  Springer International Publishing.  https://doi.org/10.1007/978-3-030-42307-0
  * Kewalramani and Veresov (2022) Sarika Kewalramani and Nikolay Veresov. 2022.  Multimodal creative inquiry: Theorising a new approach for children’s Science meaning-making in Early Childhood Education.  _Research in Science Education_ (2022), 1–21.
  * Konijn et al. (2022) Elly A Konijn, Brechtje Jansen, Victoria Mondaca Bustos, Veerle LNF Hobbelink, and Daniel Preciado Vanegas. 2022.  Social robots for (second) language learning in (migrant) primary school children.  _International Journal of Social Robotics_ (2022), 1–17.
  * Koole and Lakens (2012) Sander L. Koole and Daniël Lakens. 2012.  Rewarding Replications: A Sure and Simple Way to Improve Psychological Science.  _Perspectives on Psychological Science_ 7, 6 (2012), 608–614.  https://doi.org/10.1177/1745691612462586 arXiv:https://doi.org/10.1177/1745691612462586  PMID: 26168120.
  * Langer et al. (2023) Allison Langer, Peter J Marshall, and Shelly Levy-Tzedek. 2023\.  Ethical Considerations in Child-Robot Interactions.  _Neuroscience & Biobehavioral Reviews_ (2023), 105230.
  * Leichtmann et al. (2022) Benedikt Leichtmann, Verena Nitsch, and Martina Mara. 2022\.  Crisis ahead? Why human-robot interaction user studies may have replicability problems and directions for improvement.  _Frontiers in Robotics and AI_ 9 (2022).
  * Liu et al. (2022) Kai Liu, Cheng Guo, et al. 2022\.  The Auxiliary Function of Intelligent Mobile Robot in the Standard Training of Children’s Sports Movement.  _Wireless Communications and Mobile Computing_ 2022 (2022).
  * Livingstone et al. (2018) Sonia Livingstone, Giovanna Mascheroni, and Elisabeth Staksrud. 2018.  European research on children’s internet use: Assessing the past and anticipating the future.  _New media & society_ 20, 3 (2018), 1103–1122.
  * Lopez-Caudana et al. (2020) Edgar Lopez-Caudana, Maria Soledad Ramirez-Montoya, Sandra Martínez-Pérez, and Guillermo Rodríguez-Abitia. 2020.  Using robotics to enhance active learning in mathematics: A multi-scenario study.  _Mathematics_ 8, 12 (2020), 2163.
  * Lücking et al. (2018) Phillip Lücking, Florian Lier, Jasmin Bernotat, Sven Wachsmuth, Selma Ŝabanović, and Friederike Eyssel. 2018.  Geographically distributed deployment of reproducible hri experiments in an interdisciplinary research context. In _Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction_. 181–182.
  * Martin et al. (2020) Dorothea U Martin, Madeline I MacIntyre, Conrad Perry, Georgia Clift, Sonja Pedell, and Jordy Kaufman. 2020.  Young children’s indiscriminate helping behavior toward a humanoid robot.  _Frontiers in psychology_ 11 (2020), 239.
  * Moher et al. (2009) David Moher, Alessandro Liberati, Jennifer Tetzlaff, Douglas G Altman, and the PRISMA Group*. 2009\.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.  _Annals of internal medicine_ 151, 4 (2009), 264–269.
  * Mott et al. (2022) Terran Mott, Alexandra Bejarano, and Tom Williams. 2022\.  Robot co-design can help us engage child stakeholders in ethical reflection. In _2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, 14–23.
  * Munafò et al. (2017) Marcus R Munafò, Brian A Nosek, Dorothy VM Bishop, Katherine S Button, Christopher D Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J Ware, and John Ioannidis. 2017\.  A manifesto for reproducible science.  _Nature human behaviour_ 1, 1 (2017), 1–9.
  * Neerincx et al. (2021) Anouk Neerincx, Thirza Hiwat, and Maartje De Graaf. 2021\.  Social robot for health check and entertainment in waiting room: Child’s engagement and parent’s involvement. In _Adjunct Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization_. 120–125.
  * Nightingale (2009) Alison Nightingale. 2009\.  A guide to systematic literature reviews.  _Surgery (Oxford)_ 27, 9 (2009), 381–384.
  * Nijssen et al. (2021) Sari RR Nijssen, Barbara CN Müller, Tibor Bosse, and Markus Paulus. 2021.  You, robot? The role of anthropomorphic emotion attributions in children’s sharing with a robot.  _International Journal of Child-Computer Interaction_ 30 (2021), 100319\.
  * Nosek et al. (2015) Brian A Nosek, George Alter, George C Banks, Denny Borsboom, Sara D Bowman, Steven J Breckler, Stuart Buck, Christopher D Chambers, Gilbert Chin, Garret Christensen, et al. 2015\.  Promoting an open research culture.  _Science_ 348, 6242 (2015), 1422–1425.
  * Nosek et al. (2022) Brian A Nosek, Tom E Hardwicke, Hannah Moshontz, Aurélien Allard, Katherine S Corker, Anna Dreber, Fiona Fidler, Joe Hilgard, Melissa Kline Struhl, Michèle B Nuijten, et al. 2022\.  Replicability, robustness, and reproducibility in psychological science.  _Annual review of psychology_ 73 (2022), 719–748.
  * Oranc and Küntay (2020) Cansu Oranc and Aylin C Küntay. 2020.  Children’s perception of social robots as a source of information across different domains of knowledge.  _Cognitive Development_ 54 (2020), 100875.
  * Riek (2012) Laurel D. Riek. 2012\.  Wizard of Oz Studies in HRI: A Systematic Review and New Reporting Guidelines.  _J. Hum.-Robot Interact._ 1, 1 (jul 2012), 119–136.  https://doi.org/10.5898/JHRI.1.1.Riek
  * Romero (2019) F Romero. 2019.  Philosophy of Science and the Replicability Crisis. In _Philos. Compass_.  https://doi.org/10.1111/phc3.12633
  * Scheel et al. (2021) Anne M. Scheel, Leonid Tiokhin, Peder M. Isager, and Daniël Lakens. 2021.  Why Hypothesis Testers Should Spend Less Time Testing Hypotheses.  _Perspectives on Psychological Science_ 16, 4 (2021), 744–755.  https://doi.org/10.1177/1745691620966795 arXiv:https://doi.org/10.1177/1745691620966795  PMID: 33326363.
  * Serholt et al. (2020) Sofia Serholt, Lena Pareto, Sara Ekström, and Sara Ljungblad. 2020.  Trouble and Repair in Child–Robot Interaction: a study of complex interactions with a robot tutee in a primary school classroom.  _Frontiers in Robotics and AI_ 7 (2020), 46.
  * Shahab et al. (2022) Mojtaba Shahab, Alireza Taheri, Mohammad Mokhtari, Azadeh Shariati, Rozita Heidari, Ali Meghdari, and Minoo Alemi. 2022.  Utilizing social virtual reality robot (V2R) for music education to children with high-functioning autism.  _Education and Information Technologies_ (2022), 1–25.
  * Spitale et al. (2020) Micol Spitale, Silvia Silleresi, Giulia Cosentino, Francesca Panzeri, and Franca Garzotto. 2020\.  ” Whom would you like to talk with?” exploring conversational agents for children’s linguistic assessment. In _Proceedings of the interaction design and children conference_. 262–272.
  * Stower et al. (2021) Rebecca Stower, Mike Ligthart, Caroline Van Straten, Natalia Calvo-Barajas, Ella Velner, and Thomas Beelen. 2021.  Interdisciplinary research methods for child-robot relationship formation. In _Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction_. 700–702.
  * Stower et al. (2023) Rebecca Stower, Mike E.U. Ligthart, Micol Spitale, Natalia Calvo-Barajas, and Simone M. de Droog. 2023\.  CRITTER: Child-Robot Interaction and Interdisciplinary Research. In _Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_ (Stockholm, Sweden) _(HRI ’23)_. Association for Computing Machinery, New York, NY, USA, 926–928.  https://doi.org/10.1145/3568294.3579955
  * Strait et al. (2020) Megan Strait, Florian Lier, Jasmin Bernotat, Sven Wachsmuth, Friederike Eyssel, Robert Goldstone, and Selma Šabanović. 2020\.  A three-site reproduction of the joint simon effect with the nao robot. In _Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction_. 103–111.
  * Tian et al. (2020) Xiaoyi Tian, Nichola Lubold, Leah Friedman, and Erin Walker. 2020.  Understanding Rapport over Multiple Sessions with a Social, Teachable Robot. In _Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part II 21_. Springer, 318–323.
  * Tolksdorf et al. (2021a) Nils F Tolksdorf, Camilla E Crawshaw, and Katharina J Rohlfing. 2021a.  Comparing the effects of a different social partner (social robot vs. human) on children’s social referencing in interaction. In _Frontiers in Education_ , Vol. 5. Frontiers Media SA, 569615.
  * Tolksdorf et al. (2021b) Nils F Tolksdorf, Scarlet Siebert, Isabel Zorn, Ilona Horwath, and Katharina J Rohlfing. 2021b.  Ethical considerations of applying robots in kindergarten settings: Towards an approach from a macroperspective.  _International Journal of Social Robotics_ 13 (2021), 129–140.
  * Ullman et al. (2021) Daniel Ullman, Salomi Aladia, and Bertram F Malle. 2021\.  Challenges and opportunities for replication science in hri: A case study in human-robot trust. In _Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction_. 110–118.
  * Veling and McGinn (2021) Louise Veling and Conor McGinn. 2021.  Qualitative research in HRI: A review and taxonomy.  _International Journal of Social Robotics_ 13, 7 (2021), 1689–1709.
  * Wang et al. (2020) Ying Wang, Yun-Hee Park, Shoji Itakura, Annette Margaret Elizabeth Henderson, Takayuki Kanda, Naoki Furuhata, and Hiroshi Ishiguro. 2020\.  Infants’ perceptions of cooperation between a human and robot.  _Infant and Child Development_ 29, 2 (2020), e2161.
  * Yadollahi et al. (2021) Elmira Yadollahi, Shruti Chandra, Marta Couto, Angelica Lim, and Anara Sandygulova. 2021.  Children, Robots, and Virtual Agents: Present and Future Challenges. In _Interaction Design and Children_. 682–686.
  * Yadollahi et al. (2022) Elmira Yadollahi, Marta Couto, Pierre Dillenbourg, and Ana Paiva. 2022.  Motivating Children to Practice Perspective-Taking Through Playing Games with Cozmo. In _2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 1482–1489.
  * Yang et al. (2020) Dapeng Yang, Eung-Soo Oh, and Yingchun Wang. 2020.  Hybrid physical education teaching and curriculum design based on a voice interactive artificial intelligence educational robot.  _Sustainability_ 12, 19 (2020), 8000.
  * Zhang et al. (2023) Xiajie Zhang, Cynthia Breazeal, and Hae Won Park. 2023\.  A Social Robot Reading Partner for Explorative Guidance. In _Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_. 341–349.
  * Zhexenova et al. (2020) Zhanel Zhexenova, Aida Amirova, Manshuk Abdikarimova, Kuanysh Kudaibergenov, Nurakhmet Baimakhan, Bolat Tleubayev, Thibault Asselborn, Wafa Johal, Pierre Dillenbourg, Anna CohenMiller, et al. 2020\.  A comparison of social robot to tablet and teacher in a new script learning context.  _Frontiers in Robotics and AI_ 7 (2020), 99.