###### Abstract

In this paper, we illustrate our work on improving the accessibility of
Cyber–Physical Systems (CPS), presenting a study on human–robot interaction
where the end-users are either deaf or hearing-impaired people. Current trends
in robotic designs include devices with robotic arms and hands capable of
performing manipulation and grasping tasks. This paper focuses on how these
devices can be used for a different purpose, which is that of enabling robotic
communication via sign language. For the study, several tests and
questionnaires are run to check and measure how end-users feel about
interpreting sign language represented by a humanoid robotic assistant as
opposed to subtitles on a screen. Stemming from this dichotomy, dactylology,
basic vocabulary representation and end-user satisfaction are the main topics
covered by a delivered form, in which additional commentaries are valued and
taken into consideration for further decision taking regarding robot-human
interaction. The experiments were performed using TEO, a household companion
humanoid robot developed at the University Carlos III de Madrid (UC3M), via
representations in Spanish Sign Language (LSE), and a total of 16 deaf and
hearing-impaired participants.

###### keywords:

accessibility; anthropomorphic robotic hands; assistive robotics;
Cyber–Physical Systems; dactylology; household companion; humanoid;
human–robot interaction; robotics; sign language; statistics; survey;
vocabulary

8 1 57 Received: 27 November 2018; Accepted: 2 January 2019; Published: date
Sign Language Representation by TEO Humanoid Robot: End-User Interest,
Comprehension and Satisfaction Jennifer J. Gago *,†,‡, Juan G. Victores †,‡
and Carlos Balaguer † Jennifer J. Gago, Juan G. Victores and Carlos Balaguer
Correspondence<EMAIL_ADDRESS>or<EMAIL_ADDRESS>Tel.:
+34-91-624-6241 Current address: Av. de la Universidad, 30, 28911 Leganés,
Madrid. These authors contributed equally to this work.

##  1 Introduction

User accessibility and Universal Design (UD, also known as Design For All),
are currently getting a growing consideration worldwide to reduce the physical
and attitudinal barriers among people of all ages and abilities UD . Regarding
deaf and hearing-impaired people accessibility, Spanish and Catalan Sign
Languages were recognised to be official languages in Spain by the national
Parliament (BOE 27/2007) in 2007 b0 .

Several measures regarding the learning of this language from an early age and
empowering deaf people to request interpreters in public and private services
and areas have been taken. Following this approach, the use of resources that
enhance and enable oral communication, such as lipreading, hearing aids,
subtitling and other technological advances, has been declared a fundamental
right. These measures aim to overcome any kind of discrimination of people
with hearing disabilities in their access to information and communication,
keeping in mind their heterogeneity and the specific needs of each group.

Regarding UD, there is a need to focus on the development of products that are
easily accessible to as many people as possible, without the need to adapt or
redesign them in a special way. In order to meet these objectives in the field
of Cyber–Physical Systems (CPS), human–robot interaction must be not only
accessible, but also usable. This guarantees easy access attributes and the
possibility of understanding and learning how to communicate with the robot in
a natural and intuitive way, without the need to investigate or get additional
assistance.

Finding a way to make sure deaf or hearing-impaired individuals feel
comfortable about interacting with technology is a step forward towards
achieving the accessibility goal. The most widely used resource is to display
subtitles on a screen, since sign language interpretation is not always an
available option and it represents numerous challenges regarding its correct
use and implementation. For that reason, there are many open questions whether
or not it is likely that sign language users feel comfortable interacting with
a robot in their everyday language.

###  1.1 Challenges of Representing Sign Language

Representing sign language is a complex task which needs from advanced
software and hardware to be done properly. It is not only a matter of
precision, speed and movement fluidity, it is important to consider that
signing is commonly complemented with facial expressions, shoulder raising,
mouth morphemes, head tilt/nod/shake among other non-verbal communication
signals that affect the meaning of the message, those are part of a set of
behaviours called “non-manual markers” b4 .

The complexity of sign language is the main reason why it is still a quite
incipient developing area in human–robot interaction, in comparison to other
topics. There are relatively few projects related to robot reproduction of
sign language. The assistant android developed in 2014 by Toshiba Corporation
in collaboration with other Japanese technological institutes can mimic some
simple movements, such as greetings and signing in Japanese bbc . In addition,
humanoids Robovie R3 (five-fingered robot) and Nao robot (three-fingered
robot) were tested by the Istanbul Technical University for tutoring sign
language in adults and children with typical hearing b1 ; b2 . This work
proved the relevance of the hand anthropomorphism in sign language vocabulary
comprehension. There are other studies regarding the design and development of
robotic hands which have covered this topic independently from a humanoid
robot, as it is the case of Project Aslan, from the University of Antwerp,
which consists in a text dactylology translator arm aslan .

Participatory Design (PD) has been considered, since involving users,
designers and technology in a process of development and obtaining a distinct
and diverse set of perspectives is highly valuable when developing a universal
user oriented product CP . It is important to take into consideration that the
representation of sign language in CPS may be controversial without the
feedback and participation of deaf and hearing-impaired people in the signing
learning and implementation process. It is important to meet the expectations
and needs of the target audience of this work before investing time and
resources in specialising robots in certain areas. That is the main principle
underlying this project.

###  1.2 TEO as a Household Companion

TEO, also known as RH-2, is a full-size humanoid robot developed by
researchers at the Robotics Lab research group, from UC3M. It features 28
Degrees of Freedom (DOF), two actuated hands and several sensors to provide it
with information about its environment.

Regarding manipulation, TEO features two 6 DOF arms, each with a five-finger
dexterous hand, which can be seen in Figure 1. Thanks to their anthropomorphic
characteristics, humanoid robots can perform human tasks such as greetings,
waiter functions, folding and unfolding clothes, ironing and painting b3 ; b5
. Task performance is achieved by perception-manipulation loops through a
variety of machine learning techniques.

Figure 1: Humanoid robot TEO performing informal greeting with Dextra TPMG90-2
robotic hands.

As shown in the previous subsection, developing a robotic interpreter is an
ambitious long-term project, since grammar, dialects, idioms and facial
expression analyses would be needed. Currently, human–robot interaction with
TEO relies on short command sentences delivered in both directions, so it is
an affordable start point to test the user acceptance. To illustrate the
interaction mentioned before, performing a greeting would consist in TEO using
its voice to ask for the user name and, right after receiving that
information, a short welcome sentence would be sent through the speakers.
Therefore, the point of this work is to ensure this kind of communication can
be established via sign language.

###  1.3 TEO Robotic Hands and Sign Language

The development and adaption of new anthropomorphic robotic hands for TEO
started in September 2017. Dextra TPMG90-2 is the version name of the current
undergraduated hand prototypes operative and available in the robot b9 . They
each have 15 DOF (14 for flexion/extension and 1 for abduction/adduction) and
6 actuators. The motion transmission system is based on a tendon-driven
mechanism.

Underactuation could have been an issue regarding adaptability and precision,
since each single actuator is in charge of flexing and extending all the
phalanges of a single finger, with the exception of the thumb which is
governed by two actuators. Contrary to this assumption, due to the phalange
inner design depicted in Figure 2, the finger shows a natural gradual joint
rotation that starts from the proximal phalange and allows the hand to develop
movements similar to the one of the human hand.

Figure 2: The phalange inner design allows the finger to show a natural
gradual joint rotation starting from the proximal phalange.

Dactylology or fingerspelling requires a certain degree of position accuracy.
Figure 3 shows how Dextra TPMG90-2 is able to represent the complete Spanish
Sign Language (LSE) dactylology. This dactylology and its outcome demonstrate
how reasonable is to expect a positive performance in robotic hand signing.
Since the hand is able to reproduce the complete alphabet, the following step
is to test it with deaf and hearing-impaired users not related to the project
to obtain and evaluate feedback.

Figure 3: Spanish dactylology developed by Dextra TPMG90-2 robotic hand for
joint position configuration.

##  2 A Preliminary Study: Subtitles or Sign Language

A general solution to procure deaf and hearing-impaired accessible
communication in media and technology is to display subtitles. This settlement
presents some advantages, such as ease of understanding, speed, or simplicity
of implementation; and some disadvantages, such as the need of users’
literacy, or the requirement of a sufficiently big readable screen.

To measure the target audience preferences regarding human–robot interaction,
a preliminary study is performed in this section to obtain the rate of users
that prefer sign language over subtitles in this assistive robotics context,
before and after watching a TEO humanoid robot demonstration. These
preferences are asked and shown as it is important to measure the user
interest regarding the use of sign language within the context of humanoid
robotics, before engaging in deeper studies.

###  2.1 Preliminary Study Experimental Setup

A group of 16 anonymous deaf and hearing-impaired users were recruited in
collaboration with CILSEM (Spanish Sign Language Interpreters of Madrid
Association) and Signapuntes Lengua de Signos (an LSE forum) and asked to
choose between using sign language or subtitles with a humanoid robot, before
and after watching a demonstration in which the robot asks “how are you?” in
LSE. The sampling group consists of 16 Spanish men and women between 22 and 56
years old. The only characteristic taken into consideration for this study is
the users’ age, as the generational factor is considered to be the determining
factor to measure users’ predisposition to interact with or use technology.

A statistical test is carried out to check the consistency in responses across
the two options: sign language or subtitles. The same question is delivered on
more than one occasion for each of the individuals included in the
investigation, so the focus is on comparing whether the measurements made at
two different times are the same or if, on the contrary, there is a
significant change. McNemar’s test b6a  fits perfectly for this purpose, since
the data has one nominal variable with two categories and one independent
variable with two connected groups, the sample is random, and sign language
and subtitles are mutually exclusive b13 .

The importance of delivering this multiple choice test prior to the
comprehension test needs to be highlighted. If most users refuse the idea of
using LSE to interact with a robot in both cases, the utility of the project
should be reconsidered. Otherwise, if any or both of the cases receive a
positive feedback, there would be sound arguments to continue with the
research.

###  2.2 Preliminary Study Results

The experimental outcome is shown in Figure 4. The user’s predisposition to
communicate with robots was over 80% positive, and more than 65% of reticent
users changed their minds after their first experience with TEO. The
experimental outcome predicts a positive response to human–robot interaction.
However, a statistical analysis is needed to ensure this, which is performed
in this section.

Figure 4: User preference rate regarding human–robot interaction before and
after interacting with TEO.

Table 1 cluster the data, before and after demonstration, to analyse it via
McNemar’s test. If there were no association between the results before and
after demonstration, it is reasonable to expect the number of pairs where
users before demonstration preferred sign language but users after
demonstration did not (top right), to equal the number of pairs where the
users after demonstration preferred sign language but the users before
demonstration did not (bottom left). In this study, there were two discordant
pairs (results before demonstration and results after demonstration had
different exposure to the demonstration factor). There were a 100% of pairs
where users after demonstration preferred communicating via sign language but
users before demonstration did not (bottom left), and no pairs where users
before demonstration preferred communicating via sign language but users after
demonstration did not (top right).

Table 1: McNemar’s test table that shows user preferences regarding human–robot interaction before and after the demonstration. |  | After demonstration
---|---|---
|  | Sign language | Subtitles | Total
Before demonstration | Sign language | 13 | 0 | 13
Subtitles | 2 | 1 | 3
Total | 15 | 1 | 16

Under the null hypothesis, with a sufficiently large number of discordants
(elements of the antidiagonal), the chi-square ($\chi^{2}$) test indicates
that the distribution of the samples is chi-squared with 1 degree of freedom.

$\chi^{2}={(b-c)^{2}\over b+c}$ (1)

When the elements of the antidiagonal sum less than 25, it is not well-
approximated by the chi-squared distribution b6 . An alternative to the chi-
squared distribution is the exact binomial test:

${\displaystyle{\text{exact-P-value}}=2\sum_{i=b}^{n}{n\choose
i}0.5^{i}(1-0.5)^{n-i}}$ (2)

Edwards proposed a continuity corrected version of the McNemar test to
approximate the binomial exact-P-value, which is the most widely used variant
nowadays b7 :

$\chi^{2}={(|b-c|-1)^{2}\over b+c}.$ (3)

From Equation (3), chi-squared equals 0.500 with 1 degrees of freedom. The P
value is calculated with McNemar’s test with the continuity correction and
shows the probability of observing a large discrepancy between the number of
the two kinds of discordant pairs. The two-tailed P value equals 0.4795. By
conventional criteria, this difference is considered to be not statistically
significant. Therefore, the percentage difference after and before watching
TEO’s demonstration can be attributed to chance and there is no consistent
evidence of the effectiveness of TEO’s performance in increasing the liking or
interest rate. The odds ratio and its confidence interval cannot be calculated
because one of the discordant values is zero.

##  3 Experimental Setup: Materials and Methods

The first decision-making regarding the comprehension test setup is to
consider how this test would be distributed. In order to preserve coherence in
this experimental test, it is decided to keep using an anonymous online form
distributed by LSE institutions and simulation-based multimedia files. There
are several reasons for using simulation. On the one hand, this study aims to
present the experiments in simulation as a first step within long-term work,
where further studies will be performed with the physical humanoid robot.
Therefore, the simulation outcome allows us to anticipate the effects of the
embodiment and the robot appearance on user satisfaction and comprehension. On
the other hand, it is convenient to use a neutral background and a simplified
representation of the humanoid robot that allows the respondents to focus on
the gestures, since a non-neutral background could affect the quality of the
gesture identification.

TEO’s signing simulation is developed by using OpenRAVE and QtCoin viewer,
which provides a suitable environment for testing and developing. For that
purpose, XML files were created to store all robot and scene descriptions. An
example of this simulation can be found in Figure 5.

Figure 5: Frame of TEO’s simulation signing letter E for the dactylology test.
Vocabulary obtained from CNSE Foundation for the Suppression of Communication
Barriers images and signs LSE database.

Usability testing is used to observe how easy to use sign language with TEO is
by testing it with end-users. Participants are asked to complete these tests
to detect problematic or confusing situations. Regarding the required number
of participants to get acceptable results, Virzi b9a , and more recently Lewis
b10  and Turner b11 , have published influential articles on the topic of
sample size in usability testing. According to these authors, five is a proper
number for usability testing, so counting with 16 samples would be enough to
develop a precise and reliable study and reach a successful conclusion b12 .

The subjects of the test are randomly selected deaf and hearing-impaired
subjects, contacted by CILSEM and Signapuntes Lengua de Signos. There is no
detailed information given before the beginning of the test, and they are
kindly asked to complete a form to obtain feedback about a signing humanoid
robot. As commented before, the test is completely anonymous. The only
personal information collected from the participants is their age, in order to
detect any tendency regarding preferences or understanding.

The developed test consists in two main parts: dactylology and vocabulary
recognition. These two tests are selected to cover the study of the hand
signing accuracy and the ability to communicate by using the upper part of the
robot body. After the comprehension test with TEO, the user is ready to
measure their satisfaction, so they will be asked to answer some questions
about their experience.

Every test section is compulsory, which means that the responses cannot be
submitted until the whole test is completed. There are just three additional
optional questions about user preferences which can be completed at the end of
each section.

###  3.1 Dactylology

Fingerspelling needs to be precise to be understood properly. There are some
letters in LSE which share a quite similar hand configuration, so transitions,
speed, and arm orientation must be treated carefully to obtain good results.
It must be taken into account that TEO does not include anything similar to a
human mouth, so it is not possible to aid the understanding of the signs with
lip-speaking.

The confusion matrix of a class problem is a square matrix in which the
columns are named according to the expected result, and the rows are named
according to the experimental results. This kind of matrix is the tool
selected for showing explicitly when one letter is confused with another
letter. It is a powerful tool since it allows to work separately with
different types of errors.

The selected tool needs the provided test to check each one of the 30 letters
of the Spanish alphabet. In order to avoid predictability and check if
transitions between letters may cause any kind of confusion, the letters are
shown in groups of three, so the user is asked to fill 10 blank gaps with 3
letters each. The letters are represented in a loop, so the first frame of
each loop is marked with a blue dot to help the user to identify the beginning
of the letter signing.

###  3.2 Basic House Vocabulary

The representation of sign language vocabulary involves the action of the
upper body, which includes hands, arms and head. This makes it specially
important to coordinate all the simultaneous movements to make them seem
human-like and, therefore, be more understandable by the end-user.

The tested vocabulary is selected according to the household companion context
and considering some similar words to make it possible to apply the confusion
matrix in this case, as well as in the dactylology test. There were nine
related words and one unconnected word. “Iron” is the only unconnected word,
selected due to its significance, since ironing is one of the most complex and
relevant tasks that TEO can develop. “Machine” and “clothes”; “door”, “kichen”
and “closet”; “bedroom” and “table”; and “living room” and “telephone” are the
related words that are expected to lead to confusion. Figure 6 shows an
example of the kind of similarity tested, where the arms’ movement is quite
similar in both cases, and the position of the fingers is fundamental to
understand the difference in meaning.

In this case, as house vocabulary comprises a much wider group of words than
the Spanish alphabet and to avoid obtaining unexpected results that could
affect the confusion matrix and the following study, the users have to select
the word from a ten choices drop-down list. Each word is shown independently,
so, in accordance with the dactylology test, each user submits a ten-time
outcome.

Figure 6: Initial and final frames of TEO’s simulation signing (A) “machine”
and (B) “clothes” for the vocabulary test. Vocabulary obtained from CNSE
Foundation for the Suppression of Communication Barriers images and signs LSE
database.

###  3.3 User Satisfaction

An important part of this experimental test is to measure satisfaction, since
it is fundamental that the end-users are not only able to communicate with the
humanoid robot by using sign language, but that they can also do it in the
most comfortable way. Six topics have been considered in order to measure user
satisfaction, inspired in the users’ overall valuation test developed for the
ASIBOT assistive robot b12 , which are the following:

  * •

Aesthetics: Although TEO is still in an experimental phase and the way it
currently looks is temporary. The outcome shows the way this topic affects the
interaction experience.

  * •

Anthropomorphism: The degree of anthropomorphism or human resemblance of the
humanoid robot may influence the emergence of the uncanny valley b8 , so it
must be taken into consideration.

  * •

Future prospects: Since the technology shown in this test is under
development, it is important to know if the user is willing or not to use it
in the near future.

  * •

Comfort: Uncomfortable experiences should not be present in assistive
robotics, since these robots are made to work in close interaction with
people; therefore, comfort must be handled properly.

  * •

Comprehension ease: The user may find some difficulties to comprehend the way
TEO reproduces LSE which sometimes cannot be completely detected by error-
proofing tests.

  * •

Usefulness: Although preferences regarding robot communication are asked at
the beginning of the form, end-users might consider human–robot interaction
pointless after the tests.

The Likert scale is a measurement tool that, unlike binary questions that can
be answered affirmatively or negatively, allows to measure attitudes and know
the degree of conformity of the respondent with any proposed statement b13a .
It is especially appropriate in this context in which we want our end-users to
provide their opinion quantitatively. In this sense, the response categories
will serve to capture the intensity of the respondent’s feelings toward each
affirmation.

The most important requirement in this scale is that the distance between each
possible answer choice is the same. It allows quantitative studies across
different covered topics that have more than two outcome values b14 . There is
no clear consensus among researchers about the number of response levels. The
most commonly used scale consists in five levels; but four, seven, or ten
levels are also frequently used b15 . Adding levels results in obtaining more
diverse valuations, as it avoids central tendency bias (CTB). CTB theory
explains that in an item of only five levels, participants tend to avoid the
two extreme options, obtaining very little variation.

The CTB effect could be softened by balancing positive and negative levels in
the scale (symmetric scale) and letting the user respond to the test
anonymously to avoid the pressure of being judged for selecting extreme
options. A symmetric scale could also help to avoid acquiescence bias, which
is a tendency of the respondent to agree or show positive feedback b17 . Since
a five-level scale allows neutral response and two different levels of
agreement and disagreement, which simplifies decision taking, it is selected
for the user satisfaction test. Table 2 shows the displayed options in the
final survey.

Table 2: Five-level Likert scale used in the user satisfaction questionnaire. Scale Value | Opinion
---|---
$-$2 | Strongly disagree
$-$1 | Disagree
0 | Neither agree nor disagree
1 | Agree
2 | Strongly agree

It is difficult to treat neutral responses, such as the “neither agree nor
disagree” presented in the table, but it is recommended to offer the
possibility of taking this option if the respondent is unsure about their
opinion or cannot decide between a positive or a negative answer. About
considering a middle option as “unsure” or “neutral”, a study developed by R.
Amstrong found the differences to be imperceptible b16 .

###  3.4 Optional Questions

Some optional questions are provided in the delivered form at the end of each
previous described test sections, to obtain further information regarding the
respondent preferences. The three questions presented in the questionnaire
are:

  * •

Regarding human–robot interaction, would you prefer any alternative method to
using sign language or reading subtitles?

  * •

Why do you prefer the way of interacting with TEO that you selected?

  * •

What would you improve about TEO signing performance?

The answers to these questions could provide additional details that would
help us to understand some issues that need to be fixed in future
developments.

##  4 Experimental Results

Experimental results were collected two weeks after delivering the online form
to the institutions involved in its distribution. This limit on the period of
time for receiving the form was established to assure distribution only within
the reach of the target end-user group, as the link to the form was open and
based on trust of anonymous user data. A total of 16 users participated up to
that date.

###  4.1 Dactylology

Dactylology answers, provided the fact that robot movements were programmed by
LSE non-experts, were surprisingly accurate and insightful. Table 3 depicts
the confusion matrix that compares expected and obtained results. The elements
in the main diagonal show the amount of correct answers for each specific
letter. The ‘Other” row contains the sum of answers that are not elements of
the expected answers. It is noticeable at first sight that, except for the
letters F and RR which will be commented within this section, all letters
obtained a correct answer rate above 50%. One third of the alphabet was
completely understood (10 letters), with no failed attempts (discarding
outlier answers). Finally, the mean shows that approximately 82% of the
answers were correct (369 correct answers over 450), which can be considered a
successful outcome.

Table 3: Confusion matrix: dactylology. The elements of diagonal, which
represent correct answers, are marked in bold. Elements with a shaded
background mean 100% correct answers, discarding those of the “Other” row. In
the Spanish alphabet, the CH, LL, RR and Ñ represent individual letters.

---
|  | Result Expected
|  | A | B | C | CH | D | E | F | G | H | I | J | K | L | LL | M | N | Ñ | O | P | Q | R | RR | S | T | U | V | W | X | Y | Z
| A | 14 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| B | 0 | 13 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| C | 0 | 0 | 15 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| CH | 0 | 0 | 0 | 15 | 0 | 0 | 0 | 0 | 4 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| D | 0 | 0 | 0 | 0 | 15 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| E | 0 | 0 | 0 | 0 | 0 | 13 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| F | 0 | 0 | 0 | 0 | 0 | 0 | 6 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0
| G | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 15 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| H | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 9 | 0 | 0 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| I | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 13 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 1
| J | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 12 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 6
| K | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| L | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 15 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| LL | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 12 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| M | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 15 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0
| N | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 15 | 6 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0
| Ñ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 9 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| O | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 14 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| P | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 11 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 4 | 0 | 0 | 0
| Q | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 15 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| R | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 15 | 6 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0
| RR | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 7 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0
| S | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 14 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| T | 0 | 0 | 0 | 0 | 0 | 0 | 9 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 13 | 0 | 0 | 0 | 1 | 0 | 0
| U | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 11 | 2 | 0 | 0 | 0 | 0
| V | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 11 | 0 | 0 | 0 | 0
| W | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 10 | 0 | 0 | 0
| X | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 13 | 0 | 0
| Y | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 13 | 0
| Z | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 8
Result Obtained | Other | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1

####  4.1.1 Individual Letter Error Analysis

Taking a deeper look at the matrix helps to clarify the source of errors in
individual letter recognition. The most controversial letters, of which the
initial frames are shown in Figure 7, are F, H, K, Ñ, RR and Z, with a correct
answer rate less or equal to 75%. An independent study for each letter is
convenient to identify causes of confusion.

Figure 7: Initial frames of challenging letters in dactylology. Dactylology
obtained from CNSE Foundation for the Suppression of Communication Barriers
images and signs LSE database.

  * •

Letter F is the most erratic letter of the experiment. It was mistaken for
letter T in 60% of the attempts. These letters have relatively similar finger
position configuration, as can be seen in Figure 3. It is remarkable that only
13% of users mistaken letter T for letter F. Reviewing the F simulation file
its has been noticed that thumb position may have resulted confusing, and it
has been modified for future experiments.

  * •

Letter H is mistaken for letter CH in almost 27% of the attempts. Both letters
have the same finger position configuration, but they differ in arm movement.
Letter CH has not been mistaken for letter H in any attempt so the arm
movement for CH was pronounced after this analysis.

  * •

Letter K is mistaken for letters H and P in 20% of the attempts each. In this
case, there is over a 7% of coincidence regarding letter H and no coincidence
at all regarding letter P. Letter K is a specially complex case, since the
position of the middle finger is not so evident as it is in other letters and
there must have been some implementation errors that should be rectified with
the help of an LSE expert.

  * •

Letter Ñ is mistaken for letter N 40% of the attempts. The only difference
between these letters is that letter N is static and letter Ñ requires
movement. The solution provided to decrease this error is to make the movement
more noticeable to avoid being confused with a letter transition.

  * •

Letter RR is mistaken for letter R in 40% of the attempts. The casuistic is
exactly the same as in the Ñ-N case. Therefore, the same solution is provided.

  * •

Letter Z is mistaken for letter J in 40% of the attempts. Both letters need
motion and they share finger position configuration with letter I. Letter J
performs a circular movement while letter Z performs a zig-zag movement. The
second one was developing this movement in an almost horizontal plane, so it
was not easily understandable. The solution was to change the angle of
movement execution.

Some other letters show small inaccuracies of which sources are not as
immediately perceptible as these previous ones, so further analysis is
required to find new root causes.

####  4.1.2 Age Influence in Dactylology

Figure 8 shows the relation between the number of correct answers and the age
of the users. The negative slope of the linear trendline shows a light
tendency towards misunderstanding the dactylology developed by TEO in relation
to age increase. To measure letter transition understanding, the answer is
considered correct only if the user is able to understand the complete set of
three letters, which means that the movements between letters have not
influenced the correct perception of the dactylology.

The regression channel, which is the area between dotted lines in Figure 8, is
based on the linear regression that represents a simple trendline that is
projected using the least squares method. Consequently, this line turns out to
be an average line of the correct answer rate that is changing. It can be
considered as an “equilibrium” result line, while any deviation from it up or
down indicates the higher activity of correct or wrong answers, respectively
b18 . The distance between the channel bands and the regression line is equal
to the standard deviation value of the correct answer rate with respect to the
regression line. The upper and lower channel lines therefore contain between
themselves approximately 68% of all user answer data.

Figure 8: Graph of number of correct answers per participant in dactylology
test versus participant’s age (years) with linear trendline and linear
regression channel which contains approximately 68% of all answers.

For this dactylology test $d$, the trendline equation and the coefficient of
determination $R^{2}_{d}$ obtained by the least squares method are shown in
Equations (4) and (5), respectively. Since the regression line is relatively
far from some of the points, the $R^{2}_{d}$ of the regression is quite low.

$y_{d}=-0.071x_{d}+8.2056$ (4) ${\displaystyle
R^{2}_{d}={\sigma_{X_{d}Y_{d}}^{2}\over\sigma_{X_{d}}^{2}\sigma_{Y_{d}}^{2}}}=0.1076$
(5)

The standard deviation $s_{d}$ for the regression channel included in Figure 8
is shown in Equation (6). Using this value and Equation (4), upper and lower
lines on the regression channel are drawn.

${\displaystyle
s_{d}={\sqrt{\frac{\sum_{i=1}^{N}(y_{d_{i}}-{\overline{y_{d}}})^{2}}{N-1}}}=2.4281}$
(6)

For these sets of three words, this standard deviation shows that a range of
dactylology understanding approximately between 42% and 90% can be expected
from users in their early twenties, in comparison to the 18–67% approximated
range for middle aged people.

It should be taken into consideration for this analysis that there is an
outlier due to one user which, through a manual review of the answers, can be
determined to have answered the comprehension tests arbitrarily. Consequently,
if this outlying data is omitted, the data presented in Equations (4)–(6)
presents the variations presented in Equations (7)–(9), respectively.

$y_{d^{\prime}}=-0.0996x_{d^{\prime}}+9.608$ (7) ${\displaystyle
R^{2}_{d^{\prime}}=0.3453}$ (8) ${\displaystyle s_{d^{\prime}}=1.9346}$ (9)

As expected, this change presents a steeper negative slope and the coefficient
of determination $R^{2}_{d}$ has increased more than three times while the
standard deviation $s_{d}$ has decreased. As this model is more adjusted to
the variable when the outlier is omitted, it can be concluded that the
tendency to misunderstand the dactylology in relation to age is more
pronounced than previously stated.

###  4.2 Basic House Vocabulary

Vocabulary test results were significantly more positive than the dactylology
ones. The correct answer rate can be checked in the confusion matrix shown in
Table 4.

The average of correct answers per user is 13.3, which means a 83% of success
rate (133 correct answers over 160 answers). The lowest understood word
achieved a 62.5% of correct answers (10 correct answers over 16 answers), so
almost two thirds of the users were able to understand even the most
challenging words. This result in vocabulary understanding was expected, since
word signing, in this particular case, does not require a high level of detail
and it is more figurative than fingerspelling.

Table 4: Confusion matrix: basic house vocabulary. The elements of diagonal,
which represent correct answers, are marked in bold. Elements with a shaded
background mean 100% correct answers (15 correct, and 16 in the exceptional
case where there was a correct answer within the outlying data).

---
|  | Result Expected
|  | Table | Door | Bedroom | Closet | Telephone | Machine | Kitchen | Clothes | Iron |  | Living room
---
| Table | 14 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0
| Door | 0 | 10 | 2 | 2 | 0 | 0 | 1 | 0 | 0 | 0
| Bedroom | 0 | 2 | 14 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| Closet | 0 | 4 | 0 | 11 | 0 | 0 | 0 | 0 | 0 | 1
| Telephone | 0 | 0 | 0 | 0 | 16 | 0 | 0 | 0 | 0 | 1
| Machine | 0 | 0 | 0 | 0 | 0 | 12 | 0 | 0 | 1 | 2
| Kitchen | 1 | 0 | 0 | 0 | 0 | 0 | 14 | 0 | 0 | 0
| Clothes | 0 | 0 | 0 | 0 | 0 | 3 | 0 |  15 | 0 | 0
| Iron | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |  15 | 0
Result Obtained |  | Living room
---
1 | 0 | 0 | 3 | 0 | 1 | 0 | 0 | 0 | 12

####  4.2.1 Vocabulary Error Analysis

In order to detect some irregularities and check if the groups of similar
words produced confusion among users, a detailed error analysis is developed.

  * •

“Machine” and “clothes” are two words in LSE that are similar, since the main
difference between them is the position of the hand, but the arms develop
relatively the same movement. “Machine” was mistaken for “clothes” in almost
19% of the answers. The word “clothes” was however never mistaken for
“machine”. Since the difference between both words is a matter of open/close
fist variation it is possible that users are not so used to the word “machine”
or even that the first word that appeared in the drop-down list was “clothes”.

  * •

“Door”, “kichen” and “closet” are words that require a similar arm and hand
movement, with some variations in the order the hands are positioned. “Door”
was mistaken for “closet” 25% of the attempts and only a 13% in the inverse
order. This difference may be attributed to simulation, since the hand
position order was correctly developed. “Kitchen” was mistaken for “door” over
a 6% of the attempts but not a single time in the opposite way, so it is not
considered significant. There were no connections at all between “closet” and
“kitchen” in either direction.

  * •

“Bedroom” and “table” could have been confused since they share similar
movements, but they were not confused at any time.

  * •

“Living room” and “telephone” are two words that require signing in the head
area. “Telephone” was identified 100% of the attempts, which is an interesting
rate, considering there is one user who submitted most of their answers wrong.
The reason for this accuracy may be explained through the fact that the
Spanish sign for “telephone” could be understood internationally without any
LSE knowledge. “Living room” was not confused at any time with “telephone”,
but it was confused approximately 13% of the attempts with “machine”, which is
an outcome that cannot be explained from the consulted LSE signing database
point of view.

An unexpected result was the confusion between “closet” and “living room”,
with about 19% error rate in the mentioned order, and just a 6% in the inverse
order. There is no relation at all between the way both words were simulated,
so it may indicate an implementation error or may be biased by frequency of
everyday use.

The only independent word, which is “iron”, did not represent any challenge
for users, since it presented a 94% of success rate. Some other words also
shown small inaccuracies, which sources are relatively difficult to determine.

####  4.2.2 Age Influence in Vocabulary

Figure 9 shows the relation between the rate of correct answers and the age of
the users. The negative trendline slope shows a even lighter tendency to
misunderstand the vocabulary in relation to age than the one presented in
Figure 8.

Figure 9: Graph of number of correct answers per participant in vocabulary
test versus participant’s age (years) with linear trendline and linear
regression channel which contains approximately 68% of all answers.

For this vocabulary test $v$, the resulting equation used to draw the
trendline and the coefficient of determination $R^{2}_{v}$ are shown in
Equations (10) and (11). As expected, the slope of the negative trendline is
less than one third of the dactylology trendline slope, which means that the
tendency to misunderstand sign language using words in relation to age is
almost insignificant. Since the regression line is relatively far from a high
percentage of the points, the $R^{2}_{v}$ of the regression is quite low.

$y_{v}=-0.022x_{v}+9.0529$ (10) ${\displaystyle
R^{2}_{v}={\sigma_{X_{v}Y_{v}}^{2}\over\sigma_{X_{v}}^{2}\sigma_{Y_{v}}^{2}}}=0.0121$
(11)

The standard deviation for the regression channel included in Figure 9 is
shown in Equation (12). Using this value and Equation (10), upper and lower
lines on the regression channel can be drawn.

${\displaystyle
s_{v}={\sqrt{\frac{\sum_{i=1}^{N}(y_{v_{i}}-{\overline{y_{v}}})^{2}}{N-1}}}=2.2425}$
(12)

This standard deviation shows that a range of vocabulary understanding between
63% and 100% can be expected from users in their twenties, quite similar to
the 56–100% range for middle aged people.

If the outlier is also not being considered in this case, the data presented
in Equations (10)–(12) presents some variations, which are presented in
Equations (13)–(15), respectively.

$y_{v^{\prime}}=-0.0511x_{v^{\prime}}+10.482$ (13) ${\displaystyle
R^{2}_{v^{\prime}}=0.1447}$ (14) ${\displaystyle s_{v^{\prime}}=1.533747356}$
(15)

As occurred with dactylology, this change presents over a double steeper
negative slope, and the coefficient of determination $R^{2}_{v}$ has increased
almost twelve times, while the standard deviation $s$ has decreased
significantly. As this model is more adjusted to the variable when the outlier
is omitted, it can be concluded that the tendency to misunderstand the
dactylology in relation to age is slightly more pronounced than previously
stated.

Considering the obtained data, it can be concluded that this high vocabulary
understanding correct answer rate and this small pronounced slope in
comparison with the dactylology outcomes may be due, not only to the signing
simplicity, but also to the fact that letters were displayed in sets of three,
while words were tested independently and not in a sentence in order to
simplify analysis, so it is understandable that the error rate decreases.

###  4.3 User Satisfaction

Table 5 shows the satisfaction questionnaire individual results, sorted by age
and measured in a [$-$2, 2] Likert scale. Average overall user satisfaction
over this experimental work results in a promising 0.78 (69.5%), roughly
between a neutral and positive position.

This data is grouped and analysed with the purpose of drawing relevant
conclusions. Figure 10 gives a breakdown of this outcome, where no negative
mean values can be observed, but some relevant different satisfaction levels
are found.

Figure 10: Overall satisfaction classified by topic. A five level Likert scale
has been used, where the values mean: ($-$2) Strongly disagree, ($-$1)
Disagree, (0) Neither agree nor disagree, (1) Agree, (2) Strongly agree. Table
5: Satisfaction topics outcome classified by age in ascending order. A five
level Likert scale has been used, where the values mean: ($-$2) Strongly
disagree, ($-$1) Disagree, (0) Neither agree nor disagree, (1) Agree, (2)
Strongly agree.

---
|  | Satisfaction topics |
|  | Usefulness |  | Comprehension ease
---
Comfort |  | Future prospects
---
Anthropomorphism | Aesthetics |  Satisfaction mean
| 22 |  -1 |  -2 |  -0 |  -1 |  -1 |  -1 |  -1.00
| 23 |  -2 |  -2 |  -2 |  -2 |  -2 |  -2 |  -2.00
| 23 |  -2 |  -2 |  -2 |  -2 |  -2 |  -2 |  -2.00
| 23 |  -1 | -1 |  -1 |  -2 | -1 | -1 |  -0.17
| 24 |  -2 |  -1 |  -1 |  -2 |  -1 |  -0 |  -1.00
| 26 |  -1 |  -1 |  -1 |  -1 |  -1 |  -1 |  -1.00
| 27 |  -1 |  -0 |  -2 |  -2 |  -1 |  -2 |  -1.33
| 29 |  -2 |  -2 |  -2 |  -2 |  -2 |  -2 |  -2.00
| 29 |  -2 |  -2 |  -0 |  -1 |  -1 | -1 |  -0.83
| 34 |  -2 |  -2 |  -2 |  -2 |  -1 |  -2 |  -1.83
| 40 |  -1 | -1 | -1 |  -1 | -1 | -1 | -0.33
| 40 |  -0 | -1 | -1 | -1 | -1 |  -0 | -0.67
| 47 |  -1 |  -1 |  -1 |  -1 |  -1 |  -1 |  -1.00
| 48 |  -2 | -1 |  -2 |  -2 |  -0 | -1 |  -0.67
| 48 |  -1 |  -1 | -1 |  -0 | -1 | -1 | -0.17
Age | 56 | -1 |  -0 | -1 | -1 | -2 | -2 | -1.17

Top valued topics were future prospects and usefulness, with a 1.2 average
satisfaction or, which is the same, an 80% positive feedback. This result
demonstrates the user willingness to use this technology, and the high level
of expectation the use of LSE with a humanoid robot this first contact has
generated.

Comfort and comprehension ease, with 0.8 (70%) and 0.7 (67%) of average
satisfaction, respectively, occupy the following positions in the ranking. A
reasonable explanation to find these topics lower rated than the previous ones
is that there are various letters and vocabulary which have presented some
understanding difficulties and have led to confusion. In any case, as proved,
these minor inconveniences have not influenced the user expectation. Finally,
the least favourable marks are aesthetics and anthropomorphism, with a 62% and
58%. These topics are closely associated to the robot appearance. Since TEO is
still being developed at both software and hardware level, it is
comprehensible that there are divergent opinions about the way it looks. In
either case, this nearly neutral anthropomorphism user perception should not
be interpreted as a negative outcome, since resemblance to a human being is
not only unnecessary, but also a characteristic to be avoided in assistive
robotics.

####  4.3.1 Age Influence in Satisfaction

Age influence in overall user satisfaction is related to its influence in
dactylology and vocabulary understanding. Figure 11, where the satisfaction-
age relation is shown, presents a negative trendline that goes through the
neutral line, so it is the first graphic in which the trend drops almost a 50%
from the youngest to the oldest age.

Figure 11: Graph of user satisfaction relation versus age (years) with linear
trendline and linear regression channel which contains approximately 68% of
all answers. A five level Likert scale has been used, where the values mean:
($-$2) Strongly disagree, ($-$1) Disagree, (0) Neither agree nor disagree, (1)
Agree, (2) Strongly agree.

For this satisfaction questionnaire $s$, the resulting equation used to draw
the trendline and the $R^{2}_{s}$ value are shown in Equation (16) and
Equation (17).

$y_{s}=-0.0568x_{s}+2.6947$ (16) ${\displaystyle
R^{2}_{s}={\sigma_{X_{s}Y_{s}}^{2}\over\sigma_{X_{s}}^{2}\sigma_{Y_{s}}^{2}}}=0.42$
(17)

The standard deviation $s_{s}$ for the regression channel included in Figure
11 is shown in Equation (18). Using this value and Equation (16), the upper
and lower lines on the regression channel can be drawn.

${\displaystyle
s_{s}={\sqrt{\frac{\sum_{i=1}^{N}(y_{s_{i}}-{\overline{y_{s}}})^{2}}{N-1}}}=0.9827}$
(18)

This standard deviation shows that between 62% and 100% user satisfaction can
be expected from users in their twenties, in comparison to a 27% to 62% range
for middle aged people. The input from the user that previously provided
outlying data is not affecting these results dramatically, and is therefore
taken as a valid part of the experiment.

Taking a deeper look into Table 5, further conclusions can be drawn. For users
between 20 and 40 years old, the main disadvantages of this work are related
to anthropomorphism and aesthetics; while users between 40 and 60 years old,
also find quite inconvenient the comprehension ease and comfort. This
satisfaction distribution helps to identify what fields need to be improved to
reach all users. Although it is reasonable to expect that people around this
second group may not be so used to technology as the millennial generation,
and that their answers could also be influenced by the online test format, it
is convenient to reach a universal level of ease of understanding for the wide
range of people that could need to communicate with the robot.

###  4.4 User Additional Comments

Approximately 63% of users left additional optional comments with their
personal opinions. This feedback is highly valuable, since it offers the
opportunity to detect further aspects that need improvement, while it sets the
basis for communication error understanding.

####  4.4.1 Alternatives for Human–Robot Interaction

End-users are asked to provide alternative possibilities to the two options
considered in this paper: subtitles and sign language. This question was
answered by 31% of the participants, of which only one user actually provided
an alternative solution: using a robotic mouth which is able to enunciate
accurately while using sign language. The rest of opinions encouraged the use
of sign language as the best available option.

####  4.4.2 Justification of Preference

User preference justification must be divided between users which finally
selected sign language and the one that selected subtitles as the ideal option
after TEO signing demonstration.

The reasons for selecting sign language can be summed up in the following
points:

  * •

Sign language is clearer.

  * •

Sign language is more understandable.

  * •

Many deaf people find it difficult to read or interpret a text.

  * •

Interpreting signing is effortless for the user, since they use sign language
in a daily basis.

Two points were shared by one user to justify the subtitles selection:

  * •

Lack of facial expression and lip-speaking.

  * •

Depending on the context, a sign can have several different meanings.

Overall, there are various reasons regarding comfort which lead people to
prefer interacting with the robot via sign language. Nevertheless, it must be
taken into consideration that there are some drawbacks, such as the lack of
facial expression, which may hinder the communication.

####  4.4.3 Proposals for Improvement

The submitted suggestions were of major importance to detect which areas
required improvement. There were several individual comments which stressed
that the representation of LSE was clear and understanding the robot is a
matter of practice. The remaining comments are listed bellow.

  * •

Some words are not sign language or they are not used anymore, such as “living
room”. This note highlights the importance of working with people specialised
in LSE to implement this language.

  * •

Human-like appearance is demanded. Human misconception of what to expect from
a robot may be biased due to science fiction culture and may lead some users
to feel disillusioned by the humanoid appearance or “behaviour”.

  * •

Hand motion seems too rigid. The robot is made of hard materials and actuated
by electrical actuators, so it is complicated to reproduce a smooth motion as
human muscles can perform.

  * •

Bigger size of the images in the form would be required. This is an important
point since it could justify the increasing failure tendency in understanding
the robot related to age, considering that the decline of vision generally
associated with age.

##  5 Discussion

Regarding the developed research, detected several challenges that may be
addressed in order to enhance the analysis of the results can be described. As
mentioned, only the user’s age was considered as the main characteristic to
evaluate the tests and questionnaire outcome. However, it would be highly
useful to ask the participants for their education level and to let them rate
their frequency of use and familiarity with technology. These elements could
help factor out some outlying responses and classify the data more accurately,
especially in future studies where a larger sampling group will be managed.

A ten choice drop-down list has been used in the vocabulary test to measure
the performance of the robot. This was done to provide ease and avoid fatigue
of the respondents, while simultaneously avoid obtaining a high proportion of
outlying responses that could negatively affect the confusion matrix. Possible
redesign alternatives in relation to the format of the test within the
Participatory Design process essentially fall into one of the following two
categories: (1) to have the robot perform more actions, forming complex
sentences aiming at completing a full dialogue, or (2) having the respondents
provide more custom or personalised answers, moving from a set of closed
responses to an open interview format. While these options are not mutually
exclusive are definitively appealing, they are prone to lead into the same
kind of pitfall, which is: how to quantitatively analyse and evaluate the
respondent’s answers to obtain statistically relevant results. However, there
is an incentive for focusing on how to circumvent these challenges, as the
long-term goal of this study is to establish a complete and effective
human–robot communication.

Even though there are some potential limitations that need to be handled, such
as the need for sign language expertise and the development of a more complex
sign language reproduction by TEO, the excellent results obtained with
simulation show the importance of focusing on making further advances towards
full communication via sign language. One of the considered paths to face
these issues is to develop machine learning algorithms to learn from LSE
datasets that contain collections of signs performed by professional
interpreters. The developed system would additionally enable learning new
signs –or in different languages– from data obtained by low-cost sensors.

##  6 Conclusions

Given the worldwide need for user accessibility and UD in assistive robotics,
this work provided a pioneer study of end-user interest, comprehension and
satisfaction regarding the reproduction of sign language by a humanoid robot.

The willingness of the end-users of the study towards using sign language with
a humanoid robot was almost 94% positive, which is reaffirmed in the user
satisfaction questionnaires after the comprehension tests, where usefulness
and future prospects are valued with the highest marks. Both dactylology and
vocabulary tests resulted in 82% and 83% correct answer rate respectively,
with a relatively pronounced tendency to acceptance in relation to a younger
age. Most errors encountered on dactylology and vocabulary should be mendable
by modifying finger joint configuration or pronouncing the movement, so
further iterations of experiments could be performed to prove if the confusing
signs are fixed. Most users distinguished the robot appearance as its most
remarkable inconvenience, which is a reasonable outcome since the robot used
for testing is an experimental platform and its appearance is constantly
changing.

The most challenging issue regarding this project has been attempting to
reproduce sign language with the lack of facial expressions and other non-
manual markers. This circumstance may cause understanding problems to some
users and would be a potential barrier regarding the development of more
complex communication. Concerning basic instructions communication, the tests
have shown a proficient human–robot interaction.

The data collected over these experiments has provided quantitative
measurements on end-user satisfaction, as well as useful insight regarding
user needs. The experimental results shed light towards new improvements and
developments to make assistive robotics and CPS more usable for deaf and
hearing-impaired users.

Conceptualization, J.J.G. and J.G.V.; Data curation, J.J.G.; Formal analysis,
J.G.V.; Funding acquisition, C.B.; Investigation, J.J.G. and J.G.V.;
Methodology, J.J.G.; Project administration, J.G.V. and C.B.; Resources, C.B.;
Software, J.J.G. and J.G.V.; Supervision, J.G.V. and C.B.; Validation, J.J.G.;
Visualization, J.G.V.; Writing—original draft, J.J.G.; Writing—review and
editing, J.G.V. and C.B.

The research leading to these results has received funding from the
RoboCity2030-III-CM project (Robótica aplicada a la mejora de la calidad de
vida de los ciudadanos. fase III; S2013/MIT-2748), funded by Programas de
Actividades I+D en la Comunidad de Madrid and cofunded by Structural Funds of
the EU.

###### Acknowledgements.

The authors thank CILSEM (Spanish Sign Language Interpreters of Madrid
Association) and Signapuntes Lengua de Signos for their kind collaboration
with this project. The authors declare no conflict of interest. The following
abbreviations are used in this manuscript:
BOE | Official Gazette of the Spanish Government
---|---
CPS | Cyber–Physical Systems
DOF | Degrees of Freedom
LSE | Spanish Sign Language
PD | Participatory Design
TEO | Task Environment Operator
UC3M | University Carlos III de Madrid
UD | Universal Design
References

## References

  * (1) Story, M.F.; Mueller, J.L.; Mace, R.L. _The Universal Design File: Designing for People of All Ages and Abilities_ , Revised ed.; Center for Universal Design: Washington, DC, USA, 1998.
  * (2) Spanish Law 27/2007, October 23rd, Which Recognizes the Spanish Sign Languages and Regulates the Means of Support for Oral Communication of Deaf People, Hearing Impaired and Deafblind. [Online]. Boletín Oficial del Estado (BOE), October 2007, No. 255, pp. 43251—43259. Available online: https://www.boe.es/buscar/doc.php?id=BOE-A-2007-18476 (accesed on 1 September 2017).
  * (3) Metaxas, D.; Liu, B.; Yang, F.; Yang, P.; Michael, N.; Neidle, C. _Recognition of Nonmanual Markers in American Sign Language (ASL) Using Non-Parametric Adaptive 2D-3D Face Tracking_ ; LREC: Boston, MA, USA, 2012.
  * (4) Kelion, L. Toshiba’s robot is designed to be more human-like. _BBC News_ , 9 March 2016. Available online: https://www.bbc.com/news/technology-35763917 (accessed on 18 September 2018).
  * (5) Kose, H.; Yorganci, R. Tale of a robot: Humanoid Robot Assisted Sign Language Tutoring. Presented at the 2011 11th IEEE-RAS International Conference on Humanoid Robots, Bled, Slovenia, 26–28 October 2011.
  * (6) Uluer, P.; Akalın, N.; Köse, H. A New Robotic Platform for Sign Language. Tutoring Humanoid Robots as Assistive Game Companions for Teaching Sign Language. _Int. J. Soc. Robot._ 2015, _7_ , 571–585, doi:10.1007/s12369-015-0307-x.
  * (7) Goossens, M. Optimisation of a Humanoid Sign Language Robot. Bachelor’s Thesis, Universiteit Antwerpen, Antwerpen, Belgium, 2016.
  * (8) Asaro, P.M. Transforming society by transforming technology: the science and politics of participatory design. _Account. Manag. Inf. Technol._ 2000, _10_ , 257–290, doi:10.1016/S0959-8022(00)00004-7.
  * (9) Estevez, D.; Fernandez-Fernandez, R.; Victores, J.G.; Balaguer, C. Improving and evaluating robotic garment unfolding: A garment-agnostic approach. Presented at the 2017 IEEE International Conference on Autonomous Robot Systems and Competitions, Coimbra, Portugal, 26–28 April 2017.
  * (10) Estevez, D.; Victores, J.G.; Fernandez-Fernandez, R.; Balaguer, C. Robotic ironing with 3D perception and force/torque feedback in household environments. In Proceedings of the IEEE/RSJ IROS, Vancouver, BC, Canada, 24–28 September 2017; pp. 6484–6489, doi:10.1109/IROS.2017.8206556.
  * (11) Gago, J.J.; Victores, J.G. Desarrollo e integración de mano robótica antropomórfica en el robot humanoide TEO. Bachelor’s Thesis, Dept. Sist. Autom., Universidad Carlos III de Madrid (UC3M), Madrid, Spain, 2018.
  * (12) McNemar, Q. Note on the sampling error of the difference between correlated proportions or percentages. _Psychometrika_ 1947, _12_ , 153–157, doi:10.1007/BF02295996.
  * (13) McCrum-Gardner, E. Which is the correct statistical test to use? _Br. J. Oral Maxillofac. Surg._ 2008, _46_ , 38–41, doi:10.1016/j.bjoms.2007.09.002.
  * (14) Fagerland, M.W.; Lydersen, S.; Laake, P. The McNemar test for binary matched-pairs data: mid-p and asymptotic are better than exact conditional. _BMC Med. Res. Methodol._ 2013, _13_ , doi:10.1186/1471-2288-13-91.
  * (15) Edwards, A. Note on the “correction for continuity” in testing the significance of the difference between correlated proportions. _Psychometrika_ 1948, _13_ , 185–187, doi:10.1007/bf02289261.
  * (16) Virzi, R.A. Refining the test phase of usability evaluation: How many subjects is enough? _Hum. Factors_ 1992, _34_ , 457–468.
  * (17) Lewis, J.R. Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. _Int. J. Hum.-Comput. Interact._ 2009, _13_ , 445–479.
  * (18) Turner, C.W.; Lewis, J.R.; Nielsen, J. Determining usability test sample size. In _International Encyclopedia of Ergonomics and Human Factors_ , 2nd ed.; CRC Press: Boca Raton, FL, USA, 2006; Volume 3, pp. 3084–3088.
  * (19) Huete, A.J.; Victores, J.G.; Martinez, S.; Gimenez, A.; Balaguer, C. Personal Autonomy Rehabilitation in Home Environments by a Portable Assistive Robot. _IEEE TSMC Part C (Appl. Rev.)_ 2012, _42_ , 561–570, doi:10.1109/TSMCC.2011.2159201.
  * (20) Mori, M.; MacDorman, K.F.; Kageki, N. The Uncanny Valley [From the Field]. _IEEE Robot. Autom. Mag._ 2012, _19_ , 98–100, doi:10.1109/MRA.2012.2192811.
  * (21) Likert, R. A Technique for the Measurement of Attitudes. _Arch. Psychol._ 1932, _22_ , 1–55.
  * (22) Burns, A.; Burns, R. _Basic Marketing Research_ , 2nd ed.; Prentice Hall: Upper Saddle River, NJ, USA, 2007; p. 250.
  * (23) Dawes, J. Do Data Characteristics Change According to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales. _Int. J. Mark. Res._ 2008, _50_ , 61–77.
  * (24) Watson, D. Correcting for Acquiescent Response Bias in the Absence of a Balanced Scale: An Application to Class Consciousness. _Sociol. Methods Res._ 1992, _21_ , 52–88, doi:10.1177/0049124192021001003.
  * (25) Armstrong, R. The midpoint on a Five-Point Likert-Type Scale. _Percept. Motor Skills_ 1987, _64_ , 359–362, doi:10.2466/pms.1987.64.2.359.
  * (26) Seal, H.L. The historical development of the Gauss linear model. _Biometrika_ 1967, _54_ , 1–24, doi:10.1093/biomet/54.1-2.1.


education, income, and occupation. The contribution of occupations is
calculated by summing over the ten different occupational group dummies. The
left panel depicts results for Norway, while the right panel shows results for
Sweden (data for Denmark is not available for this exercise).

Table B1: Summary Statistics Children 1951 - 1979.

| Number of Children | Son Share Zero | Daughter Share Zero
---|---|---|---
| DK | NO | SE | DK | NO | SE | DK | NO | SE
| (2) | (3) | (4) | (5) | (6) | (7) | (8) | (9) | (10)
1951 |  | 53768 | 101153 |  | 0.0218 | 0.0093 |  | 0.0792 | 0.0109
1952 |  | 57474 | 101048 |  | 0.0247 | 0.0099 |  | 0.0780 | 0.0096
1953 |  | 59492 | 100978 |  | 0.0272 | 0.0083 |  | 0.0785 | 0.0085
1954 |  | 59334 | 96577 |  | 0.0276 | 0.0097 |  | 0.0801 | 0.0084
1955 |  | 60253 | 98415 |  | 0.0279 | 0.0113 |  | 0.0767 | 0.0091
1956 |  | 61008 | 98904 |  | 0.0324 | 0.0147 |  | 0.0810 | 0.0148
1957 |  | 60136 | 97445 |  | 0.0306 | 0.0166 |  | 0.0760 | 0.0151
1958 |  | 60152 | 96016 |  | 0.0302 | 0.0167 |  | 0.0697 | 0.0154
1959 |  | 60297 | 95185 |  | 0.0299 | 0.0179 |  | 0.0659 | 0.0162
1960 |  | 59326 | 92591 |  | 0.0295 | 0.0194 |  | 0.0639 | 0.0168
1961 |  | 59948 | 94567 |  | 0.0282 | 0.0228 |  | 0.0595 | 0.0169
1962 | 69948 | 59843 | 96813 | 0.0412 | 0.0299 | 0.0217 | 0.0592 | 0.0583 | 0.0178
1963 | 73901 | 61073 | 101760 | 0.0417 | 0.0298 | 0.0214 | 0.0555 | 0.0567 | 0.0170
1964 | 74955 | 63298 | 110549 | 0.0416 | 0.0253 | 0.0207 | 0.0587 | 0.0471 | 0.0177
1965 | 76700 | 63988 | 110364 | 0.0431 | 0.0256 | 0.0204 | 0.0586 | 0.0414 | 0.0169
1966 | 79340 | 64874 | 110074 | 0.0454 | 0.0236 | 0.0195 | 0.0594 | 0.0403 | 0.0162
1967 | 73204 | 64721 | 108302 | 0.0444 | 0.0226 | 0.0198 | 0.0593 | 0.0405 | 0.0159
1968 | 67242 | 65963 | 100853 | 0.0429 | 0.0242 | 0.0187 | 0.0588 | 0.0369 | 0.0159
1969 | 64121 | 66462 | 95375 | 0.0391 | 0.0235 | 0.0215 | 0.0536 | 0.0332 | 0.0160
1970 | 63892 | 63636 | 96388 | 0.0363 | 0.0226 | 0.0211 | 0.0518 | 0.0318 | 0.0173
1971 | 67457 | 64517 | 99960 | 0.0371 | 0.0226 | 0.0229 | 0.0518 | 0.0265 | 0.0175
1972 | 67514 | 63452 | 98766 | 0.0433 | 0.0221 | 0.0220 | 0.0565 | 0.0240 | 0.0177
1973 | 64060 | 60694 | 97018 | 0.0532 | 0.0230 | 0.0247 | 0.0651 | 0.0247 | 0.0183
1974 | 63527 | 59327 | 97644 | 0.0562 | 0.0222 | 0.0257 | 0.0683 | 0.0250 | 0.0201
1975 | 64604 | 56014 | 91500 | 0.0572 | 0.0232 | 0.0256 | 0.0740 | 0.0259 | 0.0202
1976 | 58824 | 53354 | 86260 | 0.0572 | 0.0263 | 0.0261 | 0.0726 | 0.0273 | 0.0194
1977 | 55902 | 51076 | 84042 | 0.0560 | 0.0102 | 0.0283 | 0.0719 | 0.0104 | 0.0195
1978 | 55893 | 51893 | 81184 | 0.0577 | 0.0107 | 0.0269 | 0.0706 | 0.0117 | 0.0205
1979 | 53578 | 51791 | 83740 | 0.0579 | 0.0111 | 0.0285 | 0.0736 | 0.0113 | 0.0188

  * •

Notes: The table provides summary statistics for cohort size and the share of
individuals with zero incomes in the child generation. Columns (2)-(4) present
the cohort size, columns (5)-(7) indicate the share of sons with zero incomes,
and columns (8)-(10) indicate the share of daughters with zero incomes. All
statistics are displayed separately by county and birth year of the child.

Table B2: Summary Statistics Parents 1951 - 1979.

| Number of Children | Father ID Missing | Father Share Zero | Mother ID Missing | Mother Share Zero
---|---|---|---|---|---
| DK | NO | SE | DK | NO | SE | DK | NO | SE | DK | NO | SE | DK | NO | SE
| (2) | (3) | (4) | (5) | (6) | (7) | (8) | (9) | (10) | (11) | (12) | (13) | (14) | (15) | (16)
1951 |  | 53768 | 101153 |  | 3086 | 2332 |  | 0.0292 | 0.0976 |  | 1109 | 487 |  | 0.5128 | 0.2856
1952 |  | 57474 | 101048 |  | 2642 | 1826 |  | 0.0294 | 0.0868 |  | 851 | 385 |  | 0.4782 | 0.2363
1953 |  | 59492 | 100978 |  | 1652 | 1633 |  | 0.0321 | 0.0847 |  | 488 | 373 |  | 0.4432 | 0.2103
1954 |  | 59334 | 96577 |  | 1449 | 1480 |  | 0.0339 | 0.0837 |  | 380 | 254 |  | 0.3622 | 0.1913
1955 |  | 60253 | 98415 |  | 1218 | 1320 |  | 0.0373 | 0.0795 |  | 370 | 202 |  | 0.3202 | 0.1700
1956 |  | 61008 | 98904 |  | 1154 | 1259 |  | 0.0364 | 0.0782 |  | 300 | 174 |  | 0.2771 | 0.1466
1957 |  | 60136 | 97445 |  | 1074 | 1210 |  | 0.0353 | 0.0740 |  | 256 | 180 |  | 0.2422 | 0.1171
1958 |  | 60152 | 96016 |  | 925 | 1197 |  | 0.0340 | 0.0731 |  | 236 | 134 |  | 0.2107 | 0.1027
1959 |  | 60297 | 95185 |  | 833 | 1080 |  | 0.0325 | 0.0700 |  | 185 | 108 |  | 0.1900 | 0.0828
1960 |  | 59326 | 92591 |  | 749 | 1106 |  | 0.0312 | 0.0716 |  | 207 | 91 |  | 0.1729 | 0.0743
1961 |  | 59948 | 94567 |  | 754 | 859 |  | 0.0291 | 0.0681 |  | 180 | 2 |  | 0.1583 | 0.0644
1962 | 69948 | 59843 | 96813 | 1315 | 826 | 900 | 0.0460 | 0.0306 | 0.0689 | 212 | 175 | 8 | 0.1618 | 0.1474 | 0.0577
1963 | 73901 | 61073 | 101760 | 1255 | 780 | 926 | 0.0478 | 0.0315 | 0.0668 | 177 | 155 | 6 | 0.1538 | 0.1381 | 0.0531
1964 | 74955 | 63298 | 110549 | 1209 | 778 | 972 | 0.0469 | 0.0328 | 0.0664 | 129 | 126 | 4 | 0.1421 | 0.1324 | 0.0472
1965 | 76700 | 63988 | 110364 | 1082 | 857 | 947 | 0.0460 | 0.0340 | 0.0651 | 117 | 99 | 6 | 0.1314 | 0.1261 | 0.0440
1966 | 79340 | 64874 | 110074 | 937 | 852 | 890 | 0.0437 | 0.0325 | 0.0619 | 86 | 82 | 4 | 0.1170 | 0.1133 | 0.0385
1967 | 73204 | 64721 | 108302 | 779 | 839 | 861 | 0.0424 | 0.0324 | 0.0594 | 59 | 88 | 7 | 0.1117 | 0.1015 | 0.0346
1968 | 67242 | 65963 | 100853 | 341 | 828 | 697 | 0.0419 | 0.0319 | 0.0565 | 12 | 96 | 8 | 0.1046 | 0.0965 | 0.0302
1969 | 64121 | 66462 | 95375 | 305 | 813 | 631 | 0.0439 | 0.0317 | 0.0556 |  $<$10 | 72 | 6 | 0.1047 | 0.0901 | 0.0278
1970 | 63892 | 63636 | 96388 | 306 | 768 | 647 | 0.0461 | 0.0339 | 0.0545 | 11 | 91 | 7 | 0.1033 | 0.0907 | 0.0266
1971 | 67457 | 64517 | 99960 | 288 | 720 | 566 | 0.0460 | 0.0347 | 0.0514 |  $<$10 | 90 | 2 | 0.0990 | 0.0921 | 0.0237
1972 | 67514 | 63452 | 98766 | 305 | 760 | 567 | 0.0490 | 0.0364 | 0.0511 |  $<$10 | 75 | 4 | 0.0993 | 0.0916 | 0.0232
1973 | 64060 | 60694 | 97018 | 314 | 793 | 568 | 0.0516 | 0.0380 | 0.0513 |  $<$10 | 66 | 5 | 0.0962 | 0.0908 | 0.0232
1974 | 63527 | 59327 | 97644 | 294 | 736 | 483 | 0.0532 | 0.0411 | 0.0506 |  $<$10 | 88 | 4 | 0.0914 | 0.0920 | 0.0292
1975 | 64604 | 56014 | 91500 | 326 | 760 | 464 | 0.0564 | 0.0387 | 0.0503 |  $<$10 | 82 | 2 | 0.0928 | 0.0899 | 0.0289
1976 | 58824 | 53354 | 86260 | 310 | 778 | 436 | 0.0566 | 0.0426 | 0.0526 |  $<$10 | 64 | 0 | 0.0971 | 0.0895 | 0.0285
1977 | 55902 | 51076 | 84042 | 288 | 740 | 400 | 0.0568 | 0.0411 | 0.0525 |  $<$10 | 78 | 1 | 0.0948 | 0.0859 | 0.0286
1978 | 55893 | 51893 | 81184 | 280 | 767 | 379 | 0.0563 | 0.0405 | 0.0561 |  $<$10 | 72 | 0 | 0.0883 | 0.0808 | 0.0306
1979 | 53578 | 51791 | 83740 | 290 | 802 | 399 | 0.0560 | 0.0408 | 0.0559 |  $<$10 | 96 | 3 | 0.0861 | 0.0786 | 0.0317

  * •

Notes: The table provides summary statistics for cohort size and the share of
individuals with zero incomes in the parent generation. Columns (2)-(4)
present the cohort size, columns (5)-(7) and (11)-(13) shows the number of
observations with missing father ID or mother ID, columns (8)-(10) and
(14)-(16) indicates the share of fathers or mothers with zero income. All
statistics are displayed separately by country and birth year of the child.

Table B3: IRA Coefficients, Trends and Differences Across Countries and Time.

| 1951 | 1962 | 1979 | Trend 1962-1979 |  $\Delta$ p-value
---|---|---|---|---|---
IRA Spec. | NO | SE | DK | NO | SE | DK | NO | SE | DK | NO | SE | DK-NO | DK-SE | NO-SE
All | 0.156 | 0.167 | 0.190 | 0.170 | 0.180 | 0.265 | 0.234 | 0.225 | 0.530 | 0.379 | 0.277 | 0.065 | 0.004 | 0.176
|  |  |  |  |  |  |  |  | (0.035) | (0.018) | (0.033) |  |  |
Son-Parent | 0.242 | 0.245 | 0.225 | 0.222 | 0.233 | 0.280 | 0.241 | 0.235 | 0.360 | 0.085 | 0.014 | 0.000 | 0.000 | 0.736
|  |  |  |  |  |  |  |  | (0.036) | (0.024) | (0.061) |  |  |
Daughter-Parent | 0.146 | 0.158 | 0.197 | 0.173 | 0.169 | 0.276 | 0.262 | 0.240 | 0.592 | 0.552 | 0.428 | 0.363 | 0.067 | 0.020
|  |  |  |  |  |  |  |  | (0.035) | (0.037) | (0.038) |  |  |
Son-Father | 0.253 | 0.248 | 0.220 | 0.236 | 0.242 | 0.241 | 0.213 | 0.211 | 0.139 | -0.160 | -0.224 | 0.000 | 0.000 | 0.378
|  |  |  |  |  |  |  |  | (0.035) | (0.022) | (0.060) |  |  |
Son-Mother | 0.068 | 0.080 | 0.098 | 0.089 | 0.101 | 0.198 | 0.150 | 0.155 | 0.619 | 0.324 | 0.307 | 0.000 | 0.007 | 0.723
|  |  |  |  |  |  |  |  | (0.026) | (0.026) | (0.041) |  |  |
Daughter-Father | 0.137 | 0.139 | 0.175 | 0.144 | 0.152 | 0.216 | 0.195 | 0.192 | 0.342 | 0.325 | 0.239 | 0.778 | 0.570 | 0.421
|  |  |  |  |  |  |  |  | (0.031) | (0.032) | (0.034) |  |  |
Daughter-Mother | 0.073 | 0.077 | 0.120 | 0.119 | 0.112 | 0.227 | 0.221 | 0.194 | 0.731 | 0.607 | 0.541 | 0.439 | 0.181 | 0.107
|  |  |  |  |  |  |  |  | (0.036) | (0.037) | (0.034) |  |  |

  * •

Notes: Columns (1)-(7) report the IRA coefficients of separated regressions in
the years 1951, 1962 and 1979 separately for Denmark, Norway and Sweden.
Columns (8)-(10) report the coefficient of the fitted regression lines of
country specific regressions of the IRA coefficient on a linear trend for the
years 1962 to 1979. The trend coefficients and corresponding standard errors
have been multiplied by 100 in order to avoid too many digits after the
separator. Columns (11)-(13) report rounded p-values for the null hypothesis
that the slopes for the respective countries (see column header) are equal.
Robust standard errors are reported in parentheses.

Table B4: IRA Coefficients and Trends - Details (United States).

|  Parents | Father | Mother
---|---|---|---
| Child | Son | Daughter | Son | Daughter
A: Weights |  |  |  |  |
Pooled IRA |  0.317*** |  0.336*** |  0.195*** |  0.097*** |  0.137***
|  (0.017) |  (.022) |  (0.031) |  (0.025) |  (0.029)
Trend $\times$ 100 |  0.603*** |  -0.240 |  0.980*** |  0.136 |  1.047***
|  (0.149) |  (0.205) |  (0.277) |  (0.253) |  (0.292)
N |  5,392 |  2,272 |  1,637 |  2,477 |  2,205
B: No weights |  |  |  |  |
Pooled IRA |  0.335*** |  0.360*** |  0.237*** |  0.107*** |  .152***
|  (0.013) |  (0.020) |  (0.025) |  (0.021) |  (0.022)
Trend $\times$ 100 |  0.449*** |  -0.263* |  0.728*** |  0.268 |  0.917***
|  (0.118) |  (0.178) |  (0.229) |  (0.202) |  (0.213)
N |  5,392 |  2,272 |  1,637 |  2,477 |  2,205
C: SRC sample |  |  |  |  |
Pooled IRA |  0.294*** |  0.327*** |  0.192*** |  0.098*** |  0.126***
|  (0.018) |  (0.023) |  (0.0353) |  (0.026) |  (0.032)
Trend $\times$ 100 |  0.433** |  -0.393 |  1.156*** |  0.180 |  0.727
|  (0.162) |  (0.218) |  (0.305) |  (0.266) |  (0.327)
N |  2,927 |  1,583 |  904 |  1,497 |  1,001

  * •

Notes: The table presents estimates of the IRA and linear trends in the IRA
separately for different child-parent combinations. Due to the small sample
sizes, trends have been estimated directly on the underlying micro data by
regressing cohort-specific child ranks on cohort-specific parent ranks
interacted with a linear time trend. The trend coefficients and standard
errors have been multiplied by 100 in order to avoid too many digits after the
separator. Panel A contains estimates for the full PSID sample using provided
sample weights, Panel B uses the full sample without weights and Panel C
includes estimates on the nationally representative SRC sample. Standard
errors are in parentheses. P-values indicated by * $<$ 0.1, ** $<$ 0.05, ***
$<$ 0.01.

Table B5: IRA Coefficients and Trends - Age 30 (United States).

|  Parents | Father | Mother
---|---|---|---
| Child | Son | Daughter | Son | Daughter
A: Weights |  |  |  |  |
Pooled IRA |  0.327*** |  0.318*** |  0.222*** |  0.120*** |  0.151***
|  (0.015) |  (0.022) |  (0.026) |  (0.025) |  (0.027)
Trend $\times$ 100 |  0.643*** |  0.133 |  0.661*** |  0.610** |  0.571**
|  (0.129) |  (0.193) |  (0.220) |  (0.239) |  (0.262)
N |  6,652 |  2,664 |  2,109 |  2,685 |  2,611
B: No weights |  |  |  |  |
Pooled IRA |  0.345*** |  0.341*** |  0.263*** |  0.148*** |  0.168***
|  (0.012) |  (0.018) |  (0.021) |  (0.019) |  (0.020)
Trend $\times$ 100 |  0.457*** |  0.102 |  0.510*** |  0.429** |  0.567***
|  (0.101) |  (0.59) |  (0.183) |  (0.176) |  (0.181)
N |  6,652 |  2,663 |  2,109 |  2,686 |  2,611
C: SRC sample |  |  |  |  |
Pooled IRA |  0.303*** |  0.310*** |  0.225*** |  0.097*** |  0.133***
|  (0.016) |  (0.023) |  (0.028) |  (0.027) |  (0.030)
Trend $\times$ 100 |  0.528*** |  0.020 |  0.586** |  0.661** |  0.352
|  (0.146) |  (0.210) |  (0.245) |  (0.261) |  (0.307)
N |  3,451 |  1,757 |  1,161 |  1,460 |  1,142

  * •

Note: The table presents estimates of the IRA and linear trends in the IRA
separately for different child-parent combinations Children’s income is
measure at age 30. Due to the small sample sizes, trends have been estimated
directly on the underlying micro data by regressing cohort-specific child
ranks on cohort-specific parent ranks interacted with a linear time trend. The
trend coefficients and standard errors have been multiplied by 100 in order to
avoid too many digits after the separator. Panel A contains estimates for the
full PSID sample using provided sample weights, Panel B uses the full sample
without weights and Panel C includes estimates on the nationally
representative SRC sample. Standard errors are in parentheses. P-values
indicated by * $<$ 0.1, ** $<$ 0.05, *** $<$ 0.01.

Table B6: PSID Sample Size with Parent-Child Links by Birth Cohort (United
States).

|  | Parents | Father | Mother
---|---|---|---|---
Child birth year |  | Child | Son | Daughter | Son | Daughter
1947 |  | 76 | 27 | 26 | 35 | 39
1948 |  | 107 | 38 | 39 | 48 | 56
1949 |  | 143 | 52 | 44 | 73 | 65
1950 |  | 171 | 57 | 72 | 68 | 99
1951 |  | 218 | 76 | 81 | 101 | 108
1952 |  | 193 | 70 | 78 | 88 | 102
1953 |  | 239 | 87 | 98 | 116 | 117
1954 |  | 235 | 78 | 96 | 101 | 128
1955 |  | 267 | 103 | 98 | 122 | 136
1956 |  | 263 | 86 | 106 | 107 | 147
1957 |  | 247 | 95 | 85 | 126 | 115
1958 |  | 220 | 75 | 95 | 95 | 119
1959 |  | 159 | 79 | 37 | 107 | 46
1960 |  | 177 | 96 | 34 | 121 | 54
1961 |  | 105 | 54 | 27 | 62 | 38
1962 |  | 117 | 50 | 37 | 63 | 53
1963 |  | 125 | 49 | 41 | 64 | 56
1964 |  | 100 | 47 | 30 | 56 | 43
1965 |  | 91 | 42 | 20 | 49 | 40
1966 |  | 88 | 39 | 17 | 52 | 34
1967 |  | 95 | 49 | 21 | 60 | 33
1968 |  | 66 | 32 | 17 | 39 | 26
1969 |  | 99 | 49 | 32 | 55 | 42
1970 |  | 87 | 40 | 20 | 50 | 35
1971 |  | 92 | 40 | 21 | 60 | 32
1972 |  | 111 | 49 | 22 | 63 | 43
1973 |  | 107 | 50 | 20 | 65 | 36
1974 |  | 117 | 51 | 25 | 64 | 48
1975 |  | 128 | 55 | 36 | 70 | 48
1976 |  | 132 | 58 | 37 | 63 | 53
1977 |  | 130 | 66 | 23 | 41 | 21
1978 |  | 138 | 66 | 34 | 27 | 30
1979 |  | 179 | 93 | 43 | 29 | 41
1980 |  | 142 | 65 | 36 | 33 | 26
1981 |  | 148 | 77 | 30 | 40 | 24
1982 |  | 120 | 58 | 27 | 26 | 30
1983 |  | 160 | 74 | 32 | 38 | 42
Total |  | 5,392 | 2,272 | 1,637 | 2,477 | 2,205

  * •

Notes: The table presents the number of cohort-specific parent-child links
that were used to produce the main results from the PSID survey data.

Table B7: Comparison of Trends 1962 - 1979.

|  | Denmark | Norway | Sweden
---|---|---|---|---
Panel A: Son - Father |  |  |
| IRA | 0.1385 | -0.1598 | -0.2243
|  | (0.0349) | (0.0222) | (0.0605)
| LW | 0.1504 | -0.2062 | -0.1898
|  | (0.0239) | (0.0350) | (0.0613)
| Difference | -0.0118 | 0.0464 | -0.0346
|  | (0.0423) | (0.0414) | (0.0861)
Panel B: Son - Mother |  |  |
| IRA | 0.6186 | 0.3244 | 0.3069
|  | (0.0256) | (0.0262) | (0.0408)
| LW | 0.2994 | -0.1200 | 0.0175
|  | (0.0353) | (0.0273) | (0.0495)
| Difference | 0.3192 | 0.4444 | 0.2894
|  | (0.0436) | (0.0379) | (0.0642)
Panel C: Daughter - Father |  |  |
| IRA | 0.3416 | 0.3247 | 0.2388
|  | (0.0309) | (0.0318) | (0.0339)
| LW | 0.0658 | -0.0385 | -0.0897
|  | (0.0324) | (0.0314) | (0.0201)
| Difference | 0.2758 | 0.3632 | 0.3285
|  | (0.0447) | (0.0447) | (0.0394)

  * •

Notes: IRA indicates linear trends estimated through all coefficients of the
intergenerational rank association. LW specifies linear trends estimated
through all coefficients obtained from applying the Lubotsky-Wittenberg
method. The trend coefficients and corresponding standard errors have been
multiplied by 100 in order to avoid too many digits after the separator.
Difference indicates differences between LW and IRA trends and tests the null-
hypothesis of equality in trends between the IRA and LW coefficients.
Heteroskedasticity robust standard errors are in parentheses.

##  Appendix C Calibrating Parameters in Model

Each set of country-cohort model parameters for trend decomposition are —
loosely described — calibrated in the following steps:

  1. 1.

If the cohort is the first cohort of observation for a given country (say,
1951 in Sweden and Norway and 1962 in Denmark), draw a random set of values
for the parameters $\psi_{t},\kappa_{t},\alpha_{t},\phi_{t}^{M}$ and
$\phi_{t}^{D}$. If the cohort is not the first cohort of observation,
initialize the algorithm with the optimal set of parameters from the last
cohort associated with the same country. These become the ’search parameters’
until they are replaced by parameters that are likely superior in terms of
fitting the true data as outlined in the next steps.

  2. 2.

Simulate 100,000 parent-child pairs282828The same 100,000 ’families’ are drawn
in order to ensure that ’stochastic’ differences in simulations neither affect
the convergence process nor the evolution of calibrated parameters over the
cohorts and repeat the following procedure until there is a sufficiently close
match between empirical rank associations as outlined in equations (6a)-(6e)
and the simulated counterparts:292929Or stop the algorithm early if it stops
converging, i.e. it seems that a much better match cannot be achieved.

    1. (a)

Compute skills and incomes for all individuals (father, mother, son and
daughter) using the set of ’search parameters’ along with randomly drawn
values for $x^{k}_{it}$ and $\varepsilon^{k}_{it}$.

    2. (b)

Compute rank associations as listed in equations as outlined in equations
equations (6a)-(6e) from the simulated data and match them to the associated
rank associations as estimated in the real data.

    3. (c)

If the convergence criterion is not met, adjust the parameters using a
customized variation of gradient descent. These now become the ’search
parameters’.

In Figure C1, we illustrate how the implied empirical association between the
two types of income in the (calibrated) simulated data compares to the
empirical association between the same two incomes as observed in the data.
These figures validate the quality of the calibration exercise.

Figure C1: Validation of Calibration Exercise.

Notes: Each panel displays the empirical association between two incomes as
observed in the data as well as the implied empirical association between the
same two types of income in the simulated data as calibrated in the
decomposition model. “Birth Year” refers to birth year of the child in each
parent-child pair.