# Characterizing Vaccination Movements on YouTube in the United States and
Brazil

Marcelo Sartori Locatelli<EMAIL_ADDRESS>0000-0002-0893-1446 ,
Josemar Caetano  0000-0002-0522-2572<EMAIL_ADDRESS>Universidade
Federal de Minas GeraisBelo HorizonteMinas GeraisBrazil ,  Wagner Meira Jr
0000-0002-2614-2723 Universidade Federal de Minas GeraisBelo HorizonteMinas
GeraisBrazil<EMAIL_ADDRESS>and  Virgilio Almeida  0000-0001-6452-0361
Universidade Federal de Minas GeraisBelo HorizonteMinas GeraisBrazil
<EMAIL_ADDRESS>
(2022)

###### Abstract.

In the context of COVID-19 pandemic, social networks such as Facebook,
Twitter, YouTube and Instagram stand out as important sources of information.
Among those, YouTube, as the largest and most engaging online media
consumption platform, has a large influence in the spread of information and
misinformation, which makes it important to study how the platform deals with
the problems that arise from disinformation, as well as how its users interact
with different types of content. Considering that United States (USA) and
Brazil (BR) are two countries with the highest COVID-19 death tolls, we asked
the following question: What are the nuances of vaccination campaigns in the
two countries? With that in mind, we engage in a comparative analysis of pro
and anti-vaccine movements on YouTube. We also investigate the role of YouTube
in countering online vaccine misinformation in USA and BR. For this means, we
monitored the removal of vaccine related content on the platform and also
applied various techniques to analyze the differences in discourse and
engagement in pro and anti-vaccine ”comment sections”. We found that American
anti-vaccine content tend to lead to considerably more toxic and negative
discussion than their pro-vaccine counterparts while also leading to 18%
higher user-user engagement, while Brazilian anti-vaccine content was
significantly less engaging. We also found that pro-vaccine and anti-vaccine
discourses are considerably different as the former is associated with
conspiracy theories (e.g. ccp), misinformation and alternative medicine (e.g.
hydroxychloroquine), while the latter is associated with protective measures.
Finally, it was observed that YouTube content removals are still insufficient,
with only approximately 16% of the anti-vaccine content being removed by the
end of the studied period, with the United States registering the highest
percentage of removed anti-vaccine content(34%) and Brazil registering the
lowest(9.8%).

social media, vaccination movements, textual analysis

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
33rd ACM Conference on Hypertext and Social Media; June 28-July 1, 2022;
Barcelona, Spain††booktitle: Proceedings of the 33rd ACM Conference on
Hypertext and Social Media (HT ’22), June 28-July 1, 2022, Barcelona,
Spain††price: 15.00††doi: 10.1145/3511095.3531283††isbn:
978-1-4503-9233-4/22/06††ccs: Human-centered computing Empirical studies in
collaborative and social computing

##  1\. Introduction

Political polarization and disinformation are among the main challenges to
expand global vaccination campaigns against the COVID-19 pandemic. Online
platforms such as Facebook, Twitter, YouTube and Instagram are social networks
whose global structure shapes the flow of information and affect individual
and collective behavior in issues such as vaccine, elections, extremism
(Franks et al., 2008) and racism. Social networks are also the main places
where disinformation (Wang et al., 2019), denialist narratives, confusion and
political disputes find a fertile ground on which to grow and spread
throughout society (Mitra et al., 2016). For these reasons, it is important to
understand the dynamics of online battles between anti- and pro-vaccination
groups in order to better comprehend the actions of both groups, so as to
enable more effective courses of action to be taken to mitigate the
consequences of such battles. Online influence operations are used by
different political and ideological groups to disseminate information, lies,
conspiracy theories and misleading facts. United States and Brazil have been
on the global epicenter of the COVID-19 infodemic (Nielsen et al., 2021). Both
countries have the highest death tolls since the start of the
pandemic111https://covid19.who.int/. According to the World Health
Organization, the coronavirus infodemic undermines the global response and
jeopardizes measures to control the pandemic (Zarocostas, 2020). Different
studies found that anti-vaccine groups manage to manipulate social media with
effective communication strategies that capture audiences to disseminate their
messages. Commenting on online platforms have been considered one of the most
popular forms of public online participation (Kim et al., 2021a). Comments are
the way in which online collective debates take form around specific topics
promoted by videos. They represent the relation between different groups of
society with official vaccination campaigns and reflect the degree of
polarization around vaccination.

Social platforms play an important role in identifying and removing online
anti-vaccine misinformation to limit the consequences it can do to a country’s
vaccination movement. We monitored counter influence operations to
characterize the language of comments and videos that were removed from
YouTube.

In this paper, we present two case studies that investigate the behavior of
anti and pro-vaccine groups in different political contexts, represented by
the use of social media in the United States and Brazil. In order to
understand the nuances of vaccination campaigns in the two countries, we have
the four following research questions, that are explored in our analysis of
the data sets we have collected from YouTube.

  * •

What are the characteristics that differentiate pro-vaccine and anti-vaccine
online comments?

  * •

What kind of language drives engagement in online comments?

  * •

What are similarities and differences between pro and anti-vaccination
campaigns in United States and Brazil?

  * •

What are characteristics of the pro and anti-vaccine videos removed from
YouTube?

In this work we examine characteristics of pro and anti-vaccine movements. Our
analysis explores the polarization dynamics of comment sections of political
and ideological YouTube channels and videos. On YouTube, we observe that both
language and engagement level are characteristics of different narratives of
the ideological groups involved in battles around vaccines. It has been
observed by other references that frequent commenters are more political, more
polarized, and more toxic than the average user of social media platforms (Kim
et al., 2021b). Through the use of NLP tools, we study the nature and language
of comments of both pro- and anti-vaccination groups.

Our data collection efforts resulted in a dataset with more than 8.5M comments
in more than 3.0M YouTube accounts in the two countries. We analyze this large
dataset extensively:

  * •

We found that negative emotions are much more popular and prevalent in anti-
vaccine comment sections and content.

  * •

We noticed the removal of anti-vaccine content on YouTube seems to be
insufficient to stop anti-vaccine videos.

  * •

We showed the contrast between the two groups: anti-vaccine discourse is
associated with conspiracy theories, misinformation and alternative medicine,
while pro-vaccine discourse is related to protective measures.

  * •

We found that user-user engagement is much higher on comments in removed
videos, which could suggest that content deemed inappropriate leads to more
discussion.

##  2\. Related Work

Numerous studies have been done in the past regarding disinformation and anti-
vaccine campaigns, producing countless results throughout the years. Many of
those results in regards to anti-vaccination were reviewed by Hussain et al.
(Hussain et al., 2018), who gave an overview of the movement, including the
influence of technology. They mention previous work that found that 32% of the
videos on YouTube about immunization were anti-vaccine and had higher rates
and more views than pro-vaccine videos (Keelan et al., 2007). Another study
identified that 43% of websites found in search engines when searching for
”imunization” and ”vaccination” were anti-vaccine (Davies et al., 2002).

The cross-platform nature of such campaigns was illustrated by the research
conducted by (Wilson and Starbird, 2020), which investigated an online cross-
platform disinformation campaign targeting a rescue group operating in Syria
called White Helmets. They found that anti-White Helmets accounts did more
”cross-posting” from YouTube to Twitter, creating a bridge between the two
platforms, directing Twitter users to a large set of videos critiquing the
previously mentioned group.

Another relevant study in the context of disinformation-campaigns was
conducted by (Broniatowski et al., 2018), which explores the influence of
bots, Russian trolls and content polluters on vaccine discourses on Twitter.
They find that these kinds of users tend to tweet about vaccination at higher
rates than the average user. Additionally, they conclude that accounts posing
as legitimate users create false equivalency, eroding public consensus on
vaccination.

In this same vein, (Starbird et al., 2019) examines the critical social
impacts of disinformation, its extension beyond the narrow window of automated
or paid actors such as bots and hired trolls—the window that tends to get the
most media and research attention, and the need for more robust, cross-
disciplinary treatments of information operations.

The competition between pro and anti-vaccination views was illustrated by
(Johnson et al., 2020), which explores the interconnected clusters across
geographical locations and languages. They found that, while smaller, anti-
vaccination clusters are highly entangled with undecided clusters, while pro-
vaccine clusters are more peripheral. Their theoretical framework predicts the
dominance of anti-vaccination views in a decade. They propose that the
insights provided by the framework are used to interrupt the shift to negative
views.

Focusing instead on the the role of social media companies in misinformation
regarding Covid-19, Burki (Burki, 2020) presents distinct views by experts.
While some argue that the companies’ pledges to act against anti-vaccine
movements were insufficient, others see issues of freedom of speech and fear
that the removal of posts would turn the authors into martyrs.

By exploring user behavior in YouTube live streaming chat during Trump’s 2020
presidential campaingn in the United States Liebeskind et al. (Liebeskind et
al., 2021) find, through the use of Natural Language Processing algorithms,
that most messages express sentiment, with positive sentiment outweighing the
negative. They also found that heavy users tend to use more emojis and express
more sentiment while using less abusive language. Finally, they concluded that
sarcasm invokes engagement.

The toxicity in YouTube comment sections was studied by A. Obadimu et al.
(Obadimu et al., 2019), who examined comments posted on anti- and pro-NATO
channels. They discovered that those on anti-NATO channels have a higher
tendency to be toxic. By applying Latent Dirichlet Allocation(LDA) topic
modelling, they also found that topics extracted from anti-NATO channels were
more geared towards negative components and geographical locations when
compared to their pro-NATO counterparts.

##  3\. Data Collection

In order to understand ideological battle on social media around vaccines, we
need to look at videos, channels and cultural contexts. We are interested in
two contexts of using YouTube, represented by two different countries: the
United States and Brazil. In addition to the two different countries, we focus
on two classes of groups, pro-vaccination and anti-vaccination in each
country. We collected data from different YouTube channels, videos and
comments to allow us to compare distinct types of content and analyze their
characteristics. For this means, YouTube Data API v3 was used to collect data
on 142 YouTube channels that contain videos related to the COVID-19 pandemic
and the vaccination process. These channels were discovered by different
means, such as keyword searches, suggestions by the YouTube algorithm after
the keyword searches, journalistic reports222These include: The Disinformation
Dozen, CCDH ¡https://www.counterhate.com/disinformationdozen¿, “Canais na
internet ganharam dinheiro com fake news sobre Covid, informa Google à CPI”
(Title in English: Internet channels made money with fake news about Covid,
Google informs Parliamentary Inquiry Committee)., inspection of YouTube links
shared on Facebook’s CrowdTangle, as well as through a searchable snapshot of
the Parler social network333Parler Snapshot ¡https://parler.adatascienti.st/¿,
which includes posts, comments and users that existed before it was shut down
on January 10th. Channels of important and popular political figures in the
online platforms (e.g., Brazilian president Jair Bolsonaro and his family),
educational (e.g., “University of Sao Paulo channel”) or government
institutions (e.g., “Ministry of Health”), several news sources (e.g., CNN,
Forbes, The New York Times), hospitals (e.g., ”Hospital Israelita Albert
Einstein”), well-known medical doctors (e.g., “Drauzio Varella”) or users who
have a large number of subscribers or followers (e.g., “Atila Iamarino”) were
also tracked. The category (anti- or pro-vaccine) that each channel belongs to
was determined manually upon inspection of the channel videos.

Data on videos posted by these channels was collected and filtered using a set
of relevant keywords444shorturl.at/bpzCD. This set of keywords is related to
the COVID-19 pandemic and the vaccination process. The resulting dataset
includes the video title, the channel title, the video ID, the channel ID, the
time of publication, the description of the video, tags, and topic categories.
The comments and transcript of each video that contained at least one of the
keywords used were also collected. In total, 16820 videos were found to be
relevant to the proposed study: 6648 anti-vaccine videos and 10172 pro-vaccine
videos. The data collection period started on 11th July 2021 and ended on 3rd
October 2021.

A total of 8.540.234 comments were collected, of which 172.578 had at least
one hashtag and 107.429 had a link to at least one of the following websites :
YouTube, Facebook, Instagram, Twitter, Reddit, and Gab. Table 1 shows basic
statistics grouped by Channel Category and Channel Country for the dataset
with YouTube comments. Table 2 shows the distribution of links to relevant
social networks within the database.

Channel Category | Channel Country | Accounts Number | Total Comments | Comments per Account | % incl URL
---|---|---|---|---|---
Anti-Vaccine | BR | 539.678 | 2.308.712 | 4,28 | 21,54%
Pro-Vaccine | BR | 350.774 | 649.644 | 1,85 | 22,33%
Anti-Vaccine | USA | 280.979 | 584.844 | 2,08 | 13,50%
Pro-Vaccine | USA | 1.233.565 | 3.017.758 | 2,45 | 12,99%
Anti-Vaccine | OTHER | 248.463 | 709.947 | 2,86 | 14,57%
Pro-Vaccine | OTHER | 437.707 | 1.261.304 | 2,88 | 18,14%

Table 1. Basic Statistics by Channel Category and Country for the YouTube Dataset. % incl URL represents the percentage of the total number of comments that contain URLs. While building this table, we discarded 8.025 comments that have neither country nor category. URL | Brazil | United States | Others
---|---|---|---
youtu.be | 9.485 | 31.520 | 23.601
youtube.com | 4.432 | 18.976 | 15.602
facebook.com | 759 | 1.619 | 1.382
twitter.com | 161 | 979 | 1.023
instagram.com | 191 | 255 | 181
reddit.com | 3 | 135 | 100
gab.com | 7 | 255 | 181
Total | 15.038 | 53.739 | 42.070
Table 2. Number of YouTube comments with links to websites by country.

##  4\. Methods and Tools

We use a four-dimension approach to analyze the content of comments, and
videos. Through our analyses, we evaluate (a) engagement, (b) tone of the
language, (c) toxicity and (d) lexical analysis. We also developed a tool to
monitor counter misinformation operations, i.e videos and channels removal.

###  4.1. Engagement

User-user engagement, such as the number of replies to comments, are useful
measures for understanding how users interact with different kinds of content.
This may be useful in the context of influence operations in social media
since, by finding how the engagement varies among videos/posts, it is possible
to evaluate whether or not the comment section seems to be influenced by the
nature of the content.

Each social media platform allows their users to interact in different ways,
for example, while Twitter and Gab allow retweeting/reblogging, liking posts,
replying to posts and many other forms of interaction, YouTube comments only
track replies and likes. As such, the engagement metric used for each platform
has to be calculated differently to account for such diversity.

YouTube user-user engagement has previously been explored in literature
(Ksiazek et al., 2016), where it was measured as the total number of replies
in a given comment divided by the number of comments in the video where it was
posted:

(1) $\beta=\dfrac{\operatorname{Total\ number\ of\
replies}}{\operatorname{Number\ of\ comments\ in\ video}},$

where $\beta$ denotes the reply-based engagement.

When looking at user-user engagement metrics, it is important to consider the
popularity of the analyzed videos, as previous studies (Ksiazek et al., 2016)
show a negative relationship between video popularity and this metric.

Besides, for US videos, specifically, VADER (Hutto and Gilbert, 2014)
sentiment analysis,a tool which is tailored for English text and maps lexical
features to emotion intensities known as sentiment scores, was used to
determine comments’ sentiment. By using the sentiment metrics for each
comment, we wanted to explore if the sentiment of a comment interfered with
its engagement and whether the category(anti or pro-vaccine) of the video in
which the comment was posted lead to comments of a given sentiment being more
or less popular. As previous studies have shown that in other social networks
such as Twitter (Featherstone et al., 2020), negative sentiments lead to
higher engagement regardless of the nature of the content, it was interesting
to explore if such tendencies were also true for YouTube.

###  4.2. Tone of comments

We want to understand whether variations regarding the tone of the comments
vary as a function of the video nature (i.e., pro- or anti-vaccine), as well
as whether the tone is different when people mention other platforms in an
effort to increase the reach of the comment.

####  4.2.1. Valence

We look at how words are associated with each group using the measure of
political valence proposed in (Conover et al., 2011). We adapt political
valence to the context of this study as a measure that encodes the relative
prominence of a term among Pro-Vaccine and Anti-Vaccine users. Valence _V_ of
a term _t_ is defined as:

(2)
$V(t)=2*\frac{\frac{N(t,A)}{N(A)}}{\frac{N(t,P)}{N(P)}+\frac{N(t,A)}{N(A)}}-1$

where _N(t,P)_ and _N(t,A)_ are the counts of occurrences of a term _t_ , and
_N(P)_ and _N(A)_ is the number of total terms in the Pro-Vaccine and Anti-
Vaccine texts (i.e., comments), respectively. This equation bounds the valence
between -1, for terms only used in Pro-Vaccine texts, and +1, for terms
appearing only in Anti-Vaccine texts.

The valence metric was then used to analyze the usage of words in comments
from Anti-Vaccine and Pro-Vaccine channels during 2020 and 2021, seeking to
capture the changes in the tone of comments from one year to another.

####  4.2.2. Toxicity

The toxicity of comments was evaluated using the Perspective API
555https://www.perspectiveapi.com/ tool. We worked with the definition of
toxicity used by the tool to identify whether a text (i.e., comment) could be
perceived as toxic. Toxicity refers to “a rude, disrespectful, or unreasonable
comment that is likely to make you leave a discussion”. Perspective is a tool
developed by Google that leverages deep learning to estimate the toxicity
score of a text. Therefore, we can use such score to measure how much toxicity
on average is published in comments on American and Brazilian channels.

###  4.3. Lexical Categories Analysis

Given the considerable number of video removals, it became interesting to
ascertain why the removal of these videos occurred and, thus, it was important
to explore their statistics when compared to other videos of the same category
that are still online. For this, a sample with the same number of videos was
drawn from the dataset and their statistics were collected for comparison. The
ECDF distributions in Figure 1 show the distribution of video statistics for
the sample and removed videos.

To further compare the removed videos and the sample, it was interesting to
study the difference in emotions present in their transcripts and comment
sections. For this,the Empath (Fast et al., 2016) tool was used, as it is
useful for analyzing text across lexical categories and also generating new
lexical categories to use for an analysis. For this analysis, the following
categories were used among the available Empath categories:

• positive: ’joy’, ’love’, ’politeness’.

• negative: ’aggression’, ’anger’, ’disgust’, ’hate’, ’kill’, ’nervousness’,
’pain’, ’rage’, ’sadness’, ’suffering’, ’terrorism’, ’violence’.

Ottoni et al. (Ottoni et al., 2018) proposed an analysis of right-wing
YouTubers using Empath which could be used for our analysis, thus, following
their proposed methodology, the following steps as well as the textual
processing steps of lemmatization were performed. First, the Empath for
transcripts and comments were obtained, then, the mean Empath for comments
from each video was obtained. After that, the mean Empath for each channel was
calculated, followed by the cosine similarity between comment and transcript
Empath for each video.The similarity between a video’s comments and its
transcript is defined as

(3) $S_{v}=\cos(E_{v,transcript},\overline{E}_{v,comments}),$

where $S_{v}$ denotes the similarity between the comments and transcript of
video v, $E_{v,transcript}$ denotes the Empath metrics for the video
transcript and $\overline{E}_{v,comments}$ denotes the mean Empath metrics for
the video comments.

By following the proposed steps using these relevant Empath categories, it is
possible to obtain information on how much each comment and video is related
to a given category. This information is useful to determine which categories
prevail on a given kind of content and can be used to correlate video and
comments content for each Empath category.

As Empath currently does not natively support Portuguese, this analysis was
only done for American videos.

(a) Comment count ECDF

(b) View count ECDF

(c) Like count ECDF

(d) Dislike count ECDF

Figure 1. Video Statistics ECDFs

.

###  4.4. Counter misinformation operations

It is well-known that while YouTube is an important source of vaccine-related
information to many people, it is also plagued by misinformation (Donzelli et
al., 2018). The platform is aware of this issue and follows four principles in
an attempt to mitigate the impact of fake news in its environment: (i)
rewarding trusted content creators, (ii) removing content that violates
YouTube policies, (iii) reducing recommendations of borderline content, and
(iv) raising up authoritative sources for news and information
sources666https://www.youtube.com/howyoutubeworks/our-commitments/fighting-
misinformation/. However, the measures taken in accordance to the principles
above have been shown to be insufficient (Hussein et al., 2020).

In this study, we analyze the content removal in YouTube by the means of
tracking the status of the videos on the database mentioned above. This was
achieved using the YouTube Data API v3 on a daily basis to detect video take
downs, be it by the user or by the platform. The period during which video
removals were tracked ranges from May 30,2021 to October 3,2021.

##  5\. Results

###  5.1. Engagement

Tables 3 and 4 show the mean YouTube video views in the United States and in
Brazil respectively. As mentioned, it is important to know the video
popularity before looking at engagement metrics, since it is expected that
lower popularity leads to higher engagement. The data from these tables will
be discussed in the following sections.

| Absolute | Relative
---|---|---
All videos | 143139.5 | 1.0
Anti-vaccine | 57576.5 | 0.40
Pro-vaccine | 188843.2 | 1.31
All Removed videos | 20095.2 | 0.14
Removed Anti-vaccine | 16256.9 | 0.11
Removed Pro-vaccine | 84449.3 | 0.58
Table 3. Mean YouTube views per category on American videos | Absolute | Relative
---|---|---
All videos | 71190.1 | 1.0
Anti-vaccine | 82902.3 | 1.16
Pro-vaccine | 60948.1 | 0.85
All Removed videos | 41124.0 | 0.57
Removed Anti-vaccine | 41124.0 | 0.57
Removed Pro-vaccine* | - | -
*There were no removed pro-vaccine videos in Brazil
Table 4. Mean YouTube video views per category on Brazilian videos

####  5.1.1. United States

Table 5 shows the relative mean engagement with YouTube comments for each
category and sentiment. The numbers were normalized by dividing all values by
the mean engagement of all the comments in American videos. It is important to
acknowledge that the comments from removed videos were obtained before the
video was taken down.

| General | Anti-vaccine | Pro-vaccine | Removed anti-vaccine | Removed pro-vaccine
---|---|---|---|---|---
All comments | 1.0 | 1.13 | 0.95 | 8.11 | 3.82
Comments w/links | 0.59 | 0.82 | 0.55 | 4.35 | 3.90
Comments w/hashtags | 0.72 | 0.95 | 0.68 | 16.96 | 2.32
Negative sentiment |
All comments | 0.95 | 1.24 | 0.86 | 9.64 | 4.66
Comments w/links | 0.92 | 1.38 | 0.84 | 3.44 | 3.37
Comments w/hashtags | 1.08 | 1.24 | 1.04 | 11.79 | 2.79
Positive sentiment |
All comments | 1.15 | 1.14 | 1.16 | 7.51 | 3.77
Comments w/links | 0.91 | 0.91 | 0.90 | 9.38 | 6.99
Comments w/hashtags | 0.85 | 1.42 | 0.73 | 32.37 | 4.50
*The mean engagement of all comments is 0.0006061521773068392 |

Table 5. Mean relative user-user engagement of comments per category of
American YouTube videos

It can be seen that there is significantly higher engagement for removed
videos irrespective of the sentiment of the comment. It is also interesting
that while, in general, negative pro-vaccine comments show lower engagements,
the opposite happens with positive sentiments, which may suggest that optimism
is a more popular, or at least more discussed, view in this kind of content.
In anti-vaccine content, however, negative sentiments are slightly more
engaged with, which could suggest that pessimism causes more discussion in
this category. Additionally, the fact that both positive and negative
sentiments have greater mean engagement than the mean for all anti-vaccine
comments is explained by neutral comments having lower mean engagement.

Another interesting aspect to consider is the much higher user-user engagement
in the removed videos, which could be an indicative that content deemed
inappropriate by YouTube (or by the user who decided to remove their own
video), sparks more discussion.

Additionally, comments with links to other social network websites or other
YouTube videos have shown, in most categories, comparable mean engagement to
the mean for that category. But, when considering the whole database, they
have shown considerably lower mean engagement. This could suggest that users
in YouTube comment sections, in general, are less interested in links to other
websites/videos than in purely conversational comments.

When contrasting the relative popularity of content belonging to a given
category and its user-user engagement, it can be seen that, while the
expectation would be for less popular(on average) videos to have higher
engagement, it can be seen that this does not always happen, for example,
despite being more popular on average, comments in removed pro-vaccine videos
have much lower mean engagement when compared to comments in anti-vaccine
videos.

####  5.1.2. Brazil

Table 6 shows the relative mean engagement with YouTube comments for each
category for Brazil. The numbers were obtained by dividing all values by the
mean engagement of all the comments in Brazilian videos.

| General | Anti-vaccine | Pro-vaccine | Removed anti-vaccine | Removed pro-vaccine**
---|---|---|---|---|---
All comments | 1.0 | 0.48 | 1.32 | 0.69 | -
Comments w/links | 0.83 | 0.18 | 3.45 | 0.48 | -
Comments w/hashtags | 0.25 | 0.14 | 1.11 | 0.27 | -
| *The mean engagement of all comments is 0.0008826060131351426
---
*There were no removed pro-vaccine videos in Brazil

Table 6. Mean relative user-user engagement of comments per category of
Brazilian YouTube videos

It can be seen that, for Brazil, anti-vaccine content had much lower mean
user-user engagement when compared with pro-vaccine, which is the opposite of
what happened in the case of the United States. However, similarly to the
United States, comments with links and hashtags in general had relatively
lower engagement, which could suggest that YouTube users are more interested
in purely conversational comments.

Unlike in the United States, removed Brazilian videos did not show extreme
user-user engagement, however, they did show greater engagement than the
videos of the same category that were not removed.

###  5.2. Tone of comments

####  5.2.1. Valence

Figure 2 and Figure 3 shows valence score of words used, from April 2020 to
June 2021, in US and Brazilian comments 777The words in the figure are
translated from Portuguese to English., respectively. We observe that, in both
countries, anti-vaccine discourse is more associated with conspiracy
theories(e.g., communist, ccp, chinese), misinformation and alternative
medicine (e.g., inject_disinfectant, quercetin, ivermectin,
hydroxychloroquine). Whereas pro-vaccine discourse is associated with election
and protective measures (e.g. president, administration, mask, healthy).

Figure 2. Valence of words used in 2020 and 2021 in comments from US. Figure
3. Valence of words used in 2020 and 2021 in comments from Brazil.

####  5.2.2. Toxicity

Figure 4(a) and Figure 4(b) show the toxicity probability of US comments and
Brazilian comments, respectively. We observe that, in both countries, comments
on Anti-Vaccine channels have more toxicity probability than those on Pro-
Vaccine channels. For instance, in the United States, 75% of comments on Anti-
Vaccine channels have toxicity probability between 0.125 and 0.55, whereas on
the Pro-Vaccine channels, 75% of comments have toxicity probability between
0.09 and 0.27. In Brazil, we observe a similar characteristic. However, the
majority of comments (75%) published in Pro-Vaccine are even less toxic than
the comments published in Pro-Vaccine in the United States.

(a) Toxicity probability of YouTube comments from US.

(b) Toxicity probability of YouTube comments from Brazil.

Figure 4. Toxicity probability of YouTube comments.

###  5.3. Lexical Categories Analysis

To compare the different video categories, for both removed as well as the
samples, the previously mentioned mean Empath metric for each video was used
to generate Figures 5 and 6.

Figure 5. Empath for removed videos comments and transcripts. The y axis
represents the mean Empath metric. Figure 6. Empath for sample videos comments
and transcripts.The y axis represents the mean Empath metric.

From them, it can be seen that in all categories, comments tend to have
greater metrics in positive categories, while transcripts do not seem to show
higher scores in negative categories in general. Additionally, transcripts
from videos that are still online seem to have greater positive scores than
their removed counterparts.

Besides that, it can be observed that surprisingly, comments in the sampled
non-removed pro-vaccine videos are more negative than those from removed
videos, which could be a consequence of the low sample size. However, comments
on anti-vaccine videos have shown greater negative scores in all categories
when compared to the sample, with the exception of the “rage” category.
Unsurprisingly, in most categories, especially for the removed videos, anti-
vaccine content has greater negative scores.

Figure 7 presents a matrix in which each element is the correlation between
average similarities and an Empath category. From it, it can be seen that pro
vaccine videos with less fraction of negative emotions on their transcripts
have high similarity between caption and comments. Anti-vaccine transcripts,
however, showed positive correlations with negative Empath categories,
suggesting that a higher fraction of negative emotions correlates to a high
similarity, in other words, whenever content creator and commenters’ views are
aligned, anti-vaccine videos show more negative emotions, while pro-vaccine
videos show less negative emotions. Both of these statements are especially
true for removed content, as the correlations in absolute value, are higher
for them.

Figure 7. Correlation between similarities and Empath categories

Looking at the comments it can be seen that removed pro vaccine comment
sections have low correlation with positive emotions. Furthermore, removed
anti-vaccine comments also show low correlations for some of the negative
Empath categories.

###  5.4. Counter misinformation operations

During the data collection period, a total of 1141 videos were removed, of
which 648 are from the United States, 306 are from Brazil, and 187 are from
other countries. It is important to highlight that while most removed videos
are anti-vaccine, 47 pro-vaccine videos were removed in the United States.
Moreover, 5 anti-vaccine channels from the United States and 2 from other
countries were taken down. No channels from Brazil were removed during the the
discussed period.

As it can be seen, while it is true that a large number (approximately 16%) of
anti-vaccine videos were removed, most of the anti-vaccine videos were still
available at the end of the tracking period, which illustrates why the
measures taken by YouTube are said to be insufficient. It is important to note
that most, if not all, pro-vaccine content removals happened due to the video
creators themselves and not due to actions taken by YouTube. An even lower
percentage of removals was found when looking at channels, where only 7% of
anti-vaccine channels were removed.

Another interesting aspect to consider is that the United States registered
the largest absolute number and percentage of removed anti-vaccine videos, as
601 out of 1755 videos were removed(34%). Brazil registered the lowest ratio
of removals, with only 306 out of 3121 videos being removed(9.8%). This could
suggest that YouTube measures for filtering videos are more effective in some
countries.

##  6\. Conclusion

In the context of the COVID-19 pandemic, online social media platforms were
important sources of information for many people. However, amidst the endless
stream of content posted on those platforms, misinformation was widespread. As
anti-vaccine content flooded the social networks, the efforts to prevent their
spread grew, leading to different reactions from the various platforms.
YouTube, the platform of interest for this study due to its large reach, dealt
with this kind of content by the means of demonetizing (Burki, 2020), raising
awareness via information panels
888https://support.google.com/youtube/answer/9004474?hl=en, reducing
recommendation for questionable videos, and content removal.

Despite the aforementioned attempts to filter content and reduce
disinformation, most comments and videos were still accessible at the end of
the data collection period, with only about 16% of anti-vaccine videos being
taken down. For this reason, it became clear that it is important to study the
difference between anti and pro-vaccine content to ascertain what
differentiates those two categories and what are common characteristics among
people who consume these kinds of videos, seeking to understand the dynamics
of ”comment sections”.

By analyzing comments in both anti and pro-vaccine videos, we found that, on
average, English speaking users engage more with anti-vaccine content. This is
specially noticeable when looking at the removed videos, which are supposedly
more controversial, as the user-user engagement on those is significantly
higher than the average of the database. Additionally, we note that positive
sentiments seem to be more popular in pro-vaccine ”comment sections”, while
the same can be said for negative sentiments in anti-vaccine ”comment
sections”. Moreover, during the study, it was possible to observe higher
toxicity probability on anti-vaccine comments, corroborating to the
observation that negative statements are more popular in this environment.
This is further explored when looking at the similarities between videos and
their comments, as it is observed that anti-vaccine videos are more similar to
their ”comment section” when they show negative emotions, with the opposite
being true for pro-vaccine content.

In conclusion, in this study, the removal of videos by the YouTube platform
was monitored and different methods were used to analyze anti and pro-vaccine
”comment sections”, finding key differences between them. Future work could
explore similar concepts for different social network platforms such as
Facebook, Instagram, and Twitter, seeking to ascertain whether the tendencies
observed match those found on YouTube. Finally, the removal of content on
YouTube could be explored over a larger time frame with a larger database, in
an attempt to better understand the effectiveness of the actions taken by the
platform.

###### Acknowledgements.

The authors would like to thank the participants of the project “Measuring The
Impact Of Covid Misinformation Influence Operations On Public Health
Outcomes”, with support from the Carnegie Endowment for International Peace’s
Partnership for Countering Influence Operations, for valuable discussions and
feedbacks during the year of 2021. This work was funded in part by Brazilian
agencies CNPq, CAPES and FAPEMIG, and by projects INCTCyber, MASWeb and CIIA-
Saude.

## References

  * (1)
  * Broniatowski et al. (2018) David A Broniatowski, Amelia M Jamison, SiHua Qi, Lulwah AlKulaib, Tao Chen, Adrian Benton, Sandra C Quinn, and Mark Dredze. 2018.  Weaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate.  _American journal of public health_ 108, 10 (2018), 1378–1384.
  * Burki (2020) Talha Burki. 2020\.  The online anti-vaccine movement in the age of COVID-19.  _The Lancet Digital Health_ 2, 10 (2020), e504–e505.
  * Conover et al. (2011) Michael Conover, Jacob Ratkiewicz, Matthew Francisco, Bruno Gonçalves, Alessandro Flammini, and Filippo Menczer. 2011.  Political Polarization on Twitter. In _Proc. 5th International AAAI Conference on Weblogs and Social Media (ICWSM)_.  http://www.aaai.org/ocs/index.php/ICWSM/ICWSM11/paper/view/2847
  * Davies et al. (2002) Paul Davies, Simon Chapman, and Julie Leask. 2002\.  Antivaccination activists on the world wide web.  _Archives of disease in childhood_ 87, 1 (2002), 22–25.
  * Donzelli et al. (2018) Gabriele Donzelli, Giacomo Palomba, Ileana Federigi, Francesco Aquino, Lorenzo Cioni, Marco Verani, Annalaura Carducci, and Pierluigi Lopalco. 2018.  Misinformation on vaccination: A quantitative analysis of YouTube videos.  _Human vaccines & immunotherapeutics_ 14, 7 (2018), 1654–1659.
  * Fast et al. (2016) Ethan Fast, Binbin Chen, and Michael S Bernstein. 2016\.  Empath: Understanding topic signals in large-scale text. In _Proceedings of the 2016 CHI conference on human factors in computing systems_. 4647–4657.
  * Featherstone et al. (2020) Jieyu D Featherstone, George A Barnett, Jeanette B Ruiz, Yurong Zhuang, and Benjamin J Millam. 2020\.  Exploring childhood anti-vaccine and pro-vaccine communities on twitter–a perspective from influential users.  _Online Social Networks and Media_ 20 (2020), 100105.
  * Franks et al. (2008) Daniel W Franks, Jason Noble, Peter Kaufmann, and Sigrid Stagl. 2008.  Extremism propagation in social networks with hubs.  _Adaptive Behavior_ 16, 4 (2008), 264–274.
  * Hussain et al. (2018) Azhar Hussain, Syed Ali, Madiha Ahmed, and Sheharyar Hussain. 2018\.  The anti-vaccination movement: a regression in modern medicine.  _Cureus_ 10, 7 (2018).
  * Hussein et al. (2020) Eslam Hussein, Prerna Juneja, and Tanushree Mitra. 2020\.  Measuring misinformation in video search platforms: An audit study on YouTube.  _Proceedings of the ACM on Human-Computer Interaction_ 4, CSCW1 (2020), 1–27.
  * Hutto and Gilbert (2014) Clayton Hutto and Eric Gilbert. 2014.  Vader: A parsimonious rule-based model for sentiment analysis of social media text. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 8.
  * Johnson et al. (2020) Neil F Johnson, Nicolas Velásquez, Nicholas Johnson Restrepo, Rhys Leahy, Nicholas Gabriel, Sara El Oud, Minzhang Zheng, Pedro Manrique, Stefan Wuchty, and Yonatan Lupu. 2020.  The online competition between pro-and anti-vaccination views.  _Nature_ 582, 7811 (2020), 230–233.
  * Keelan et al. (2007) Jennifer Keelan, Vera Pavri-Garcia, George Tomlinson, and Kumanan Wilson. 2007.  YouTube as a source of information on immunization: a content analysis.  _jama_ 298, 21 (2007), 2482–2484.
  * Kim et al. (2021a) Jin Woo Kim, Andrew Guess, Brendan Nyhan, and Jason Reifler. 2021a.  The distorting prism of social media: How self-selection and exposure to incivility fuel online comment toxicity.  _Journal of Communication_ 71, 6 (2021), 922–946.
  * Kim et al. (2021b) Jin Woo Kim, Andrew Guess, Brendan Nyhan, and Jason Reifler. 2021b.  The Distorting Prism of Social Media: How Self-Selection and Exposure to Incivility Fuel Online Comment Toxicity.  _Journal of Communication_ (09 2021).  https://doi.org/10.1093/joc/jqab034 arXiv:https://academic.oup.com/joc/advance-article-pdf/doi/10.1093/joc/jqab034/40301657/jqab034.pdf  jqab034.
  * Ksiazek et al. (2016) Thomas B Ksiazek, Limor Peer, and Kevin Lessard. 2016\.  User engagement with online news: Conceptualizing interactivity and exploring the relationship between online news videos and user comments.  _New media & society_ 18, 3 (2016), 502–520.
  * Liebeskind et al. (2021) Chaya Liebeskind, Shmuel Liebeskind, and Shoam Yechezkely. 2021\.  An Analysis of Interaction and Engagement in YouTube Live Streaming Chat. In _2021 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/IOP/SCI)_. IEEE, 272–279.
  * Mitra et al. (2016) Tanushree Mitra, Scott Counts, and James W Pennebaker. 2016\.  Understanding anti-vaccination attitudes in social media. In _Tenth International AAAI Conference on Web and Social Media_.
  * Nielsen et al. (2021) Rasmus Kleis Nielsen, Anne Schulz, and Richard Fletcher. 2021\.  An ongoing infodemic: how people in eight countries access news and information about coronavirus a year into the pandemic.  _Reuters Institute for the Study of Journalism_ (2021).
  * Obadimu et al. (2019) Adewale Obadimu, Esther Mead, Muhammad Nihal Hussain, and Nitin Agarwal. 2019.  Identifying toxicity within YouTube video comment. In _International conference on social computing, Behavioral-cultural modeling and prediction and behavior representation in modeling and simulation_. Springer, 214–223.
  * Ottoni et al. (2018) Raphael Ottoni, Evandro Cunha, Gabriel Magno, Pedro Bernardina, Wagner Meira Jr, and Virgílio Almeida. 2018.  Analyzing right-wing youtube channels: Hate, violence and discrimination. In _Proceedings of the 10th ACM conference on web science_. 323–332.
  * Starbird et al. (2019) Kate Starbird, Ahmer Arif, and Tom Wilson. 2019.  Disinformation as collaborative work: Surfacing the participatory nature of strategic information operations.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (2019), 1–26.
  * Wang et al. (2019) Yuxi Wang, Martin McKee, Aleksandra Torbica, and David Stuckler. 2019.  Systematic literature review on the spread of health-related misinformation on social media.  _Social science & medicine_ 240 (2019), 112552.
  * Wilson and Starbird (2020) Tom Wilson and Kate Starbird. 2020.  Cross-platform disinformation campaigns: lessons learned and next steps.  _Harvard Kennedy School Misinformation Review_ 1, 1 (2020).
  * Zarocostas (2020) John Zarocostas. 2020\.  How to fight an infodemic.  _The lancet_ 395, 10225 (2020), 676.


# An Analysis of Personalized Speech Recognition System Development for the
Deaf and Hard-of-Hearing

Lester Phillip Violeta and Tomoki Toda Nagoya University, Japan
E-mail<EMAIL_ADDRESS>
###### Abstract

Deaf or hard-of-hearing (DHH) speakers typically have atypical speech caused
by deafness. With the growing support of speech-based devices and software
applications, more work needs to be done to make these devices inclusive to
everyone. To do so, we analyze the use of openly-available automatic speech
recognition (ASR) tools with a DHH Japanese speaker dataset. As these out-of-
the-box ASR models typically do not perform well on DHH speech, we provide a
thorough analysis of creating personalized ASR systems. We collected a large
DHH speaker dataset of four speakers totaling around 28.05 hours and
thoroughly analyzed the performance of different training frameworks by
varying the training data sizes. Our findings show that 1000 utterances (or
1-2 hours) from a target speaker can already significantly improve the model
performance with minimal amount of work needed, thus we recommend researchers
to collect at least 1000 utterances to make an efficient personalized ASR
system. In cases where 1000 utterances is difficult to collect, we also
discover significant improvements in using previously proposed data
augmentation techniques such as intermediate fine-tuning when only 200
utterances are available.

##  1 Introduction

Deaf or hard of hearing (DHH) people typically have their development of
speech stunted due to hearing impairedness. This results in difficult-to-
understand speech [1], or commonly referred to as atypical speech, which makes
daily conversations a tedious task for them. As communication is one of the
essential aspects of being human, there is a need to develop more systems to
support DHH speakers and provide them with more communication alternatives.

Recent advancements in artificial intelligence (AI) has given way for
computers to assist humans in their everyday life. One particular field with
substantial improvements in the past decade is automatic speech recognition
(ASR) [2, 3, 4], the conversion of speech into its corresponding text labels.
Owing to neural network-based approaches and the availability of large-scale
datasets, several systems have been proposed that could transcribe text at an
astounding accuracy rate. Smart devices, in particular voice assistants, have
used this technology as a means of communication between users’ speech and a
control system to execute the said command, enabling several conveniences such
as controlling appliances around the house. Thus, with the growing popularity
of ASR, several devices and software applications now support using speech to
navigate through the device or the application. For DHH users, using ASR to
transcribe their spoken speech into text becomes available as an alternative
communication option. With the huge benefits and conveniences that ASR-based
smart devices bring, improving and developing personalized ASR systems for DHH
speakers can allow them a communication alternative and be able to use these
devices as an assistant in everyday life.

Although ASR systems have been successful with typical speech, it has
struggled with transcribing DHH speech due to the difference and high
variability in the DHH speech patterns compared to typical speech. The deaf
speaker usually cannot control the high-frequency components of speech [5] due
to their hearing conditions, making how other humans and ASR models perceive
their speech entirely different from typical speakers. Consequently, this
means that although ASR models are usually trained on large-scale typical
speech (usually a minimum of 1000 hours), the success of these ASR models is
also limited to just typical speech. Replicating the same success on DHH
speech would require training the ASR model on a large-scale DHH speaker
dataset; however, recording at a large scale, due to strict ethics regulations
and the effort it takes to setup recordings, makes this a difficult task to
resolve. Thus, speakers with atypical speech have found these devices rather
difficult to use [6, 7]. However, as mentioned, DHH users would be the ones
who could benefit the most from ASR-based devices.

In this research, we provide an analysis of how to create personalized ASR
systems for DHH speakers. Our goal is to provide a thorough analysis such that
researchers would be able to understand the training size/performance tradeoff
when collecting speech data, and how to make use of methodologies used in the
literature to properly develop personalized ASR systems. Our previous work has
shown the effectiveness of large-scale pretraining and using imperfect
synthetic data to improve recognition accuracy in electrolaryngeal speech
recognition [8]; however, there is still some doubt on whether the previously
proposed method works on different types of atypical speech or on larger
dataset sizes. Thus, this extension of the analysis provides information on
the effects of varying the training data sizes and how it affects the
performance of each ASR method. Moreover, compared to other atypical datasets
used in the literature, we collect a large dataset of four DHH speakers with
28.05 hours in total, allowing us a thorough examination of the dataset. Our
main findings are as follows:

  * •

We perform a thorough analysis with publicly available ASR resources on four
different DHH Japanese speakers to make a personalized ASR system. We train
personalized models and analyze different scenarios with different training
data sizes and data augmentation techniques to give researchers an idea of
what requirements are needed to train high-performing personalized ASR
systems.

  * •

Through our findings, we confirm that our previously proposed speech
synthesis-based data augmentations indeed improve ASR performance in low-
resourced scenarios (200 utterances) even on DHH speech datasets. However,
using at least 1000 utterances of real recordings to fine-tune pretrained
models already shows huge performance boosts in ASR performance, and thus
recommend researchers collect around this number of utterances as it is the
best cost/performance tradeoff point.

  * •

Finally, we emphasize the effectiveness of creating personalized ASR systems,
as when conducting data augmentation, using the synthesized speech of the
target speaker was always better than using real recordings from other
speakers.

##  2 Related literature

Since the number of devices and software applications that support the use of
speech as an input grows more and more each day, using ASR as a communication
alternative is becoming more available as an option. However, research such as
[6, 7] has investigated that for DHH and atypical speech, currently
productionized or open-sourced ASR models have up to 60% increase in word
error rates (WER) compared to typical speech. Thus, although ASR-based
technology has been gaining popularity and more support on devices, current
models are still far from being inclusive to all people, and more
investigations need to be done to resolve this issue.

Although not focused on DHH speech, several research works have investigated
improving ASR performance for atypical speakers. The most common method to
resolve this is by pretraining the model with large-scale typical speech data,
and subsequently fine-tuning on the target low-resourced atypical speech data
[9, 10, 11, 12]. Through large-scale pretraining, the model can generalize to
the target better by using these as the initialization model parameters,
rather than training the model from randomized initialization model
parameters. The larger the pretraining dataset used, the more generalized the
model could be and thus more robust to different types of speech during fine-
tuning.

Research such as [11, 13] has shown that personalized ASR models will always
outperform ASR models trained on other speakers’ data. Although this work has
shown that simply fine-tuning a pretrained model with target speakers can
massively improve ASR performance, our previous research in [8] shows that
naively fine-tuning the pretrained model on the target speaker is not very
effective in a low-resourced scenario, such as when there are only 200
utterances available for each speaker. To further improve performance, we
proposed using an intermediate fine-tuning approach using imperfect synthetic
speech by using text-to-speech (TTS) [8]. The main point made here was that
even low-quality synthetic speech of the target speaker (in this case, the
electrolaryngeal speaker) could be utilized to improve ASR performance,
contrary to the common intuition that data augmentations should always be of
high quality. This has opened doors in improving low-resourced ASR, by
requiring looser requirements to train ASR models of a target atypical
speaker. However, more work has yet to be done to verify the effectiveness of
this method on other types of atypical speakers (such as DHH) or if it would
still work well in scenarios with a larger training data size as the
experiments were only conducted using a small-scale electrolaryngeal speaker
dataset.

Figure 1:  An overview of the ASR training process. We follow the setup
described in [8] and expand on the investigations of this work on deaf and
hard-of-hearing speakers.

##  3 Methodology

###  3.1 Development of ASR model

We expand on our previous work shown in [8], to verify the effectiveness of
the intermediate fine-tuning task on other atypical speech datasets and with
different training data sizes. Following this experiment, we primarily used
setups based on pretraining on a large-scale typical speech dataset and
subsequently fine-tuning on each target speaker. We then perform a thorough
analysis of their behavior to understand how to develop efficient ASR models.
To generate synthetic speech used for data augmentation, we use the same TTS
setup used in that experiment. The TTS setup involves using a pretrained model
and fine-tuning it on each of the speakers to create a larger training data.
Then, text from a separate speech dataset is used to generate the larger
speech dataset of each speaker. An overview of the ASR model training
procedure is shown in Figure 1.

###  3.2 DHH dataset overview

The dataset we used was collected from four different DHH Japanese speakers in
the comfort of their homes. To record the dataset, we asked them to utter the
phrases from the Japanese speech corpus of Saruwatari Lab of the University of
Tokyo (JSUT) dataset [14], a dataset spoken by a native female Japanese
speaker. The dataset contains 7696 utterances and is commonly used for speech
synthesis tasks. However, speakers were allowed to only record a subset of the
dataset or skip some utterances if they felt uncomfortable saying some of the
phrases. All datasets were recorded at a 44.1kHz sampling rate. A detailed
look at the dataset can be found in Table 1. Compared to commonly used
atypical speech datasets such as UASpeech [15] and TORGO [16], our dataset is
larger in total and contains a large number of utterances per speaker, making
personalized ASR more possible. Moreover, compared to UASpeech [15] which only
contains word-level utterances, our dataset contains sentence-level
utterances, making it a more realistic scenario for certain personalized ASR
scenarios. With a larger dataset, we aim to analyze the behavior of the
network when varying the training data sizes.

Table 1: An overview of the DHH speaker dataset collected. Speaker ID | Gender | No. of utterances | Total length (Hours)
---|---|---|---
S1 | Male | 5000 | 11.41
S2 | Male | 4982 | 10.69
S3 | Female | 902 | 1.64
S4 | Female | 2493 | 4.31
Total | - | 13377 | 28.05

##  4 Experiments

###  4.1 Experimental conditions

To conduct the experiments, we used ESPnet [17, 18], an open-sourced speech
processing toolkit. For the ASR model architecture, we used a Conformer [19]
encoder and a Transformer [20] decoder due to their proven strength in ASR
tasks [21]. As for the TTS models, we used a Transformer [20] encoder and
decoder. Since the TTS models only convert the text to a mel-spectrogram, we
used a pretrained model based on HiFiGAN [22] as the vocoder to convert mel-
spectrograms to waveforms. The ASR and TTS pretrained models are made openly-
available by ESPNet, while we also used a pretrained HiFiGAN
model111https://github.com/kan-bayashi/ParallelWaveGAN on the JSUT [14]
dataset. To pretrain the ASR model, we used the large-scale LaboroTV [23]
dataset, which contains around 2000 hours of Japanese speech collected from
various TV recordings. On the other hand, to pretrain the TTS model, we used
the entire JSUT [14] dataset.

To vary the training data sizes, we subset the datasets into different groups
depending on the number of utterances available. More details on the number of
utterances used are shown in Fig. 1 as they differ for each speaker due to the
total dataset size. We evaluated the trained model using a test set with 250
utterances for speakers S1, S2, and S4, and 50 utterances for speaker S3.
Although the TTS models were trained and synthesized the utterances in 24 kHz,
the synthesized audio would be downsampled to 16 kHz when used to train the
ASR model. Thus, all audio used in the ASR models were in a 16 kHz sampling
rate.

###  4.2 Data augmentation setups

We describe the different setups used in our experiment. We follow the
intermediate fine-tuning (IF) setup detailed in [8] and extend upon their
findings.

  * •

Direct: Directly using the out-of-the-box pretrained model and evaluating the
performance without performing any fine-tuning.

  * •

Direct - speaker-independent: Fine-tuning the dataset with the other DHH
speakers’ audio data. This means that we do not use the target DHH speaker’s
audio data to fine-tune the model.

  * •

Naive fine-tuning: Simply fine-tuning the pretrained model on the target
speaker without any extra procedures.

  * •

IF - TTS: Based on the intermediate fine-tuning setup in [8]. We generate a
larger dataset of the target DHH speaker using TTS, and use these during
intermediate fine-tuning.

  * •

IF - TTS with text-swapping: Similar to the TTS setup, but randomly swapping
the text labels within the dataset. Note that the text labels are only swapped
during the intermediate fine-tuning stage. We further investigate this setup
due to its reported success in [8] but non-intuitive setup.

  * •

IF - speaker-independent: Instead of using synthetic speech, we use the real
speech recordings from other DHH speakers during intermediate fine-tuning.
This allows us to assess if the improvements in intermediate fine-tuning come
from the speaker-dependent or speaker-independent characteristics of the data.

Figure 2:  Character error rate (CER) of the different developed ASR systems
for all speakers (S1, S2, S3, and S4). The x-axis indicates the amount of data
used in the training and development sets.

###  4.3 Evaluation metric

To evaluate the ASR models, we use the character error rate (CER), as seen in
Equation 1, a commonly used metric for Japanese ASR. Given an N number of
characters and the number of wrongly predicted characters, which are
categorized into three types, insertions (I), deletions (D), and substitutions
(S), the CER can be calculated as the sum of the insertions, deletions, and
substitutions divided by the number of characters, multiplied by 100. Thus,
the lower the CER value, the better the performance of the ASR model.

$\operatorname{CER}=\frac{\operatorname{I}+\operatorname{D}+\operatorname{S}}{\operatorname{N}}\operatorname{x}100$
(1)

##  5 Results and Analysis

###  5.1 ASR model performance using out-of-the-box pretrained models

We present the summary of our results in Figure 2. As displayed, in all cases
where fine-tuning is not used, the ASR model always performed above 80% CER as
shown by the results of using the Direct setup. Thus, despite the ASR model
being pretrained on a large-scale dataset, it still has difficulties adapting
to the DHH speech, showing the difficulties faced by deaf speakers using out-
of-the-box speech recognition devices. Aside from this, fine-tuning on speech
data of other DHH speakers improves performance, but still not effective
enough as shown in the Direct - speaker independent setup, proving the need
for personalized ASR models. On the other hand, when fine-tuning the model on
the available utterances (as shown by the Naive setup), the model’s
performance almost always improves, where the only exception is when using 200
utterances, similar to the findings in [8]. However, using a larger dataset of
the target speaker should be sufficient enough for the model to generalize. In
all speakers, we find improvements of between 5% to 30% CER by increasing the
number of utterances to 1000 from 200. Thus, we recommend researchers collect
at least 1000 utterances as the ASR models can be already be effectively fine-
tuned using this amount of data.

Figure 3:  Difference of the character error rate (CER) of the Naive and IF-
TTS setups across different training data sizes.

###  5.2 Extending investigations on intermediate fine-tuning

We investigate whether our previously proposed data augmentation techniques
for improving electrolaryngeal speech recognition also perform well on DHH
speech. Using data augmentation techniques such as TTS to create a larger
dataset of the target speaker and using intermediate fine-tuning, we see that
this fine-tuning method can significantly improve performance in this low-
resourced scenario, verifying that our method proposed in [8] can also be
transferred to DHH speech data. However, we observe that using intermediate
fine-tuning became less effective as the training data size increased. Amongst
the different setups, we find that the IF-TTS setup was the most stable and
showed the most improvements in the low-resourced scenarios. Although the IF-
TTS text-swapping setup was reported to be successful in [8], we do not see
the same trend in our experiments as we observe its instability. Specifically,
in cases where the training data is too low or too high, we observe a collapse
and the ASR model fails. One hypothesis on why this is the case is because of
the difference in speech features between electrolaryngeal speech and DHH
speech. For electrolaryngeal speech, the speaker identity is disrupted, but
the intelligibility is still there, as several phonemes can still be
pronounced almost the same as a typical speaker. On the other hand, DHH speech
contains the speaker identity but has unintelligible speech. Thus, since the
intelligibility information between the pretraining data (LaboroTV) and the
target data (electrolaryngeal speech) could be transferred and shared, the
intermediate fine-tuning worked better on the electrolaryngeal dataset. On the
other hand, in the case of our experiments, since the intelligibility
information between the pretraining data (LaboroTV) and the target data (DHH
speech) vastly differed, the information was not effectively shared. ASR
models typically focus on the intelligibility information and disregard
speaker identity information, which also explains why the text-swapping was
not as effective in the case of DHH speakers. Finally, we observe that using
data augmentations of the target speaker, even when the quality is low, was
still always better than using real recordings of other DHH speakers as shown
by the IF - speaker-independent setup results.

As a final observation, although we verified the validity of the intermediate
fine-tuning in the low-resource scenario, we still find that it cannot beat
the performance when using real recordings of the target speaker as the
training data. A detailed look on the comparison of the Naive and IF-TTS
setups are shown in Fig 3. As seen in the figure, using imperfect synthetic
speech only became successful in the 200 utterance setup as there was a larger
CER difference between using the two setups; however, as more real data was
collected and used during training, the effects of using imperfect synthetic
speech became close to none. Thus, although we acknowledge the difficulty of
obtaining training data from the target speaker, we urge researchers to
collect at least 1000 utterances to easily develop efficient personalized ASR
systems, and such that they would not need to do any additional procedures
such as intermediate fine-tuning and speech synthesis to improve the ASR
models. On the other hand, in cases where collecting data is difficult,
researchers can still opt to use the intermediate fine-tuning procedure, as it
has been verified to work even on other types of atypical speech.

##  6 Conclusion

We presented a thorough analysis of developing personalized ASR systems for
DHH Japanese speakers. We provided a thorough extension of our previous work
to verify its effectiveness by analyzing its behavior when varying the number
of utterances in the training data and with a different atypical speech
dataset. We collected a large-scale DHH dataset, totaling around 28.05 hours
of four speakers. Our findings show the following observations: 1) the most
efficient setup is to collect 1000 utterances from the target speaker, as we
see the most performance boost compared to using just 200 utterances, 2) in
cases where 1000 utterances are difficult to collect, data augmentation
techniques such as using intermediate fine-tuning with synthetic speech is
still an available option to improve performance, and finally, 3) we
emphasized the importance of using personalized models, as using real
recordings from the target speaker as training data greatly outperforms using
synthetic speech or recordings from other speakers. Future work would be to
improve the speech intelligibility of atypical speakers using voice conversion
techniques such that they can easily use out-of-the-box ASR models.

## Acknowledgment

This work was partly supported by AMED under Grant Number JP21dk0310114,
Japan, and a project, JPNP20006, commissioned by NEDO.

## References

  * [1] M. J. Osberger and N. S. McGarr, “Speech production characteristics of the hearing impaired,” ser. Speech and Language, N. J. LASS, Ed., vol. 8. Elsevier, 1982, pp. 221–283.
  * [2] F. Jelinek, “Continuous speech recognition by statistical methods,” _Proceedings of the IEEE_ , vol. 64, no. 4, pp. 532–556, 1976.
  * [3] H. A. Bourlard and N. Morgan, _Connectionist speech recognition_. Kluwer Academic Publishers, 1994.
  * [4] A. Graves, A.-R. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in _Proc. ICASSP_ , 2013, pp. 6645–6649.
  * [5] C. Lorenzi, G. Gilbert, H. Carn, S. Garnier, and B. Moore, “Speech perception problems of the hearing impaired reflect inability to use temporal fine structure,” _Proceedings of the National Academy of Sciences of the United States of America_ , vol. 103, pp. 18 866–9, 01 2007.
  * [6] A. T. Glasser, K. R. Kushalnagar, and R. S. Kushalnagar, “Feasibility of using automatic speech recognition with voices of deaf and hard-of-hearing individuals,” in _Proc. ASSETS_. New York, NY, USA: Association for Computing Machinery, 2017, p. 373–374.
  * [7] M. Moore, H. Venkateswara, and S. Panchanathan, “Whistle-blowing ASRs: Evaluating the Need for More Inclusive Speech Recognition Systems,” in _Proc. Interspeech_ , 2018, pp. 466–470.
  * [8] L. P. Violeta, D. Ma, W.-C. Huang, and T. Toda, “Intermediate Fine-Tuning Using Imperfect Synthetic Speech for Improving Electrolaryngeal Speech Recognition,” in _Proc. ICASSP_. IEEE, 2023.
  * [9] R. L. MacDonald, P.-P. Jiang, J. Cattiau, R. Heywood, R. Cave, K. Seaver, M. A. Ladewig, J. Tobin, M. P. Brenner, P. C. Nelson, J. R. Green, and K. Tomanek, “Disordered Speech Data Collection: Lessons Learned at 1 Million Utterances from Project Euphonia,” in _Proc. Interspeech_ , 2021, pp. 4833–4837.
  * [10] J. Shor, D. Emanuel, O. Lang, O. Tuval, M. Brenner, J. Cattiau, F. Vieira, M. McNally, T. Charbonneau, M. Nollstadt, A. Hassidim, and Y. Matias, “Personalizing ASR for Dysarthric and Accented Speech with Limited Data,” in _Proc. Interspeech_ , 2019, pp. 784–788.
  * [11] J. R. Green, R. L. MacDonald, P.-P. Jiang, J. Cattiau, R. Heywood, R. Cave, K. Seaver, M. A. Ladewig, J. Tobin, M. P. Brenner, P. C. Nelson, and K. Tomanek, “Automatic Speech Recognition of Disordered Speech: Personalized Models Outperforming Human Listeners on Short Phrases,” in _Proc. Interspeech_ , 2021, pp. 4778–4782.
  * [12] L. P. Violeta, W. C. Huang, and T. Toda, “Investigating Self-supervised Pretraining Frameworks for Pathological Speech Recognition,” in _Proc. Interspeech_ , 2022, pp. 41–45.
  * [13] J. Tobin and K. Tomanek, “Personalized automatic speech recognition trained on small disordered speech datasets,” in _Proc. ICASSP_ , 2022, pp. 6637–6641.
  * [14] R. Sonobe, S. Takamichi, and H. Saruwatari, “JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesis,” 2017. [Online]. Available: https://arxiv.org/abs/1711.00354
  * [15] H. Kim, M. H. Johnson, J. Gunderson, A. Perlman, T. Huang, K. Watkin, S. Frame, H. V. Sharma, and X. Zhou, “Dysarthric speech database for universal access research,” in _Proc. Interspeech_ , 2008, pp. 1741–1744.
  * [16] F. Rudzicz, A. Namasivayam, and T. Wolff, “The torgo database of acoustic and articulatory speech from speakers with dysarthria,” _Language Resources and Evaluation_ , vol. 46, pp. 1–19, 01 2010.
  * [17] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. Enrique Yalta Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai, “ESPnet: End-to-End Speech Processing Toolkit,” in _Proc. Interspeech_ , 2018, pp. 2207–2211.
  * [18] S. Watanabe, F. Boyer, X. Chang, P. Guo, T. Hayashi, Y. Higuchi, T. Hori, W.-C. Huang, H. Inaguma, N. Kamo, S. Karita, C. Li, J. Shi, A. Subramanian, and W. Zhang, “The 2020 espnet update: New features, broadened applications, performance improvements, and future plans,” 06 2021, pp. 1–6.
  * [19] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented Transformer for Speech Recognition,” in _Proc. Interspeech_ , 2020, pp. 5036–5040.
  * [20] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [21] P. Guo, F. Boyer, X. Chang, T. Hayashi, Y. Higuchi, H. Inaguma, N. Kamo, C. Li, D. Garcia-Romero, J. Shi, J. Shi, S. Watanabe, K. Wei, W. Zhang, and Y. Zhang, “Recent developments on espnet toolkit boosted by conformer,” in _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 5874–5878.
  * [22] J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,” _Advances in NeurIPS_ , vol. 33, pp. 17 022–17 033, 2020.
  * [23] S. Ando and H. Fujihara, “Construction of a Large-Scale Japanese ASR Corpus on TV Recordings,” in _Proc. ICASSP_ , 2021, pp. 6948–6952.