# Behind the Mask: Demographic bias in name detection for PII masking

Courtney Mansfield∗, Amandalynne Paullada${}^{*}\dagger$, Kristen Howell∗
∗LivePerson Inc., Seattle, Washington, USA
†Biomedical Informatics & Medical Education, University of Washington,
Seattle, Washington, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Many datasets contain personally identifiable information, or PII, which poses
privacy risks to individuals. PII masking is commonly used to redact personal
information such as names, addresses, and phone numbers from text data. Most
modern PII masking pipelines involve machine learning algorithms. However,
these systems may vary in performance, such that individuals from particular
demographic groups bear a higher risk for having their personal information
exposed. In this paper, we evaluate the performance of three off-the-shelf PII
masking systems on name detection and redaction. We generate data using names
and templates from the customer service domain. We find that an open-source
RoBERTa-based system shows fewer disparities than the commercial models we
test. However, all systems demonstrate significant differences in error rate
based on demographics. In particular, the highest error rates occurred for
names associated with Black and Asian/Pacific Islander individuals.

##  1 Introduction

In a time of extensive data collection and distribution, privacy is a vitally
important but elusive goal. In 2021, the US-based Identity Theft Resource
Center reported a 68% increase in data breaches from the previous year, with
83% involving sensitive
information111https://www.idtheftcenter.org/post/identity-theft-resource-
center-2021-annual-data-breach-report-sets-new-record-for-number-of-
compromises/. The exposure of personally identifiable information (PII), such
as names, addresses, or social security numbers, leaves individuals vulnerable
to identity theft and fraud. In response, a growing number of companies
provide data protection services, including PII detection, redaction
(masking), and anonymization.

PII masking offers assurances of security. However, this paper considers
whether the models powering these services perform fairly across individuals,
regardless of race, ethnicity, and gender. Historically, the US “Right to
Privacy” concept has been centered around Whiteness, initially to protect
White women from the then-emergent technology of photography and visual media
(Osucha, 2009). Black individuals have had less access to privacy and face
greater risk of harm due to surveillance, including algorithmic surveillance
(Browne, 2015; Fagan et al., 2016).

In this paper, we evaluate the detection and masking of names, which are the
primary indexer of a person’s identity. We sample datasets of names and
demographic information to measure the performance of off-the-shelf PII
maskers. Although model bias or unfairness can be the result of a number of
factors, including training data or presuppositions encoded in the algorithms
themselves, the commercial systems we examine fail to provide details about
training data or implementation. Therefore, we do not hypothesize a causal
relationship between these factors and our findings.

Our work quantifies disparities in the name detection of PII masking systems
where poor performance can directly and negatively impact individuals. We
demonstrate significant disparities in the recognition of names based on
demographic characteristics, especially for names associated with Black and
Asian/Pacific Islander groups.

##  2 PII Masking

This study analyzes personally identifiable information (PII) masking systems
which aim to detect and redact sensitive personal information, particularly
names, from text. This has been an important problem in the biomedical domain,
in terms of preparing de-identified patient data for research (Kayaalp, 2018),
but is also increasingly important in an age of language models trained from
web-scraped text, which have been shown to reveal private information that was
not removed from the underlying training data (Carlini et al., 2021).

Since early efforts masking data by hand, automated methods have been
employed, from using word lists or dictionaries (Thomas et al., 2002), which
do not generalize to unseen names and locations, to rule-based or regular
expression systems (Beckwith et al., 2006; Friedlin and McDonald, 2008), which
are generalizable, but can be brittle. These have been replaced with machine
learning systems (Szarvas et al., 2006; Uzuner et al., 2008) and most recently
neural networks (Dernoncourt et al., 2017; Adams et al., 2019).

Modern PII maskers rely on Named Entity Recognition (NER) to identify entities
(e.g. name and location) for redaction. NER has had recent success with hybrid
bi-directional long short term memory (BiLSTM) and conditional random field
(CRF) models (Huang et al., 2015), and following the general trend in NLP,
fine-tuning on large language models such as BERT (Li et al., 2019).
Additional discussion on NER architectures can be found in Li et al. (2020).

Previous research in Named Entity Recognition (NER) has illuminated race and
gender-based disparities. Mishra et al. (2020) evaluates a number of NER
models which consider performance according to gender and race/ethnicity. The
analysis considers 15 names per intersectional group, finding that White-
associated names are more likely to be recognized across all systems. Our work
differs from and extends this work in key aspects: focusing on off-the-shelf
PII masking, providing analysis on over 4K names, and reporting on
significance and additional metrics.

Recent PII masking models perform extremely well in certain contexts. The
recurrent neural network of Dernoncourt et al. (2017) achieves 99% recall
overall and just below 98% for names on patient discharge summaries in the
medical domain. The commercial models we consider do not advertise performance
metrics, and as shown in Section 7, do not achieve such high performance
across our datasets.

It is important to note that removing names alone is insufficient to fully
protect individuals from being identified from data. Data sets can still
reveal just enough information to re-identify individuals, as in the case of
Massachusetts Governor William Weld, whose medical records, although not
connected directly to his name in a de-identified data set, were traceable
back to him by matching information from an easily attained external data
resource Sweeney (2002). Here we focus on names as they are a primary
identifier for an individual.

##  3 What’s in a Name?

The primary goal of this paper is to understand whether, and to what degree,
the performance of PII masking models is influenced by correlates of race,
ethnicity, and gender. We frame bias in terms of significant discrepancies in
performance based on race/ethnicity and gender, looking specifically to
instances where private information was not masked (false negative rates,
described in Section 6.2). PII masking is a primary mechanism for protecting
personal data, and a systematic failure to mask information belonging to
marginalized subgroups can cause undue harm to those populations, through
identity theft, identity fraud, and loss of privacy. Names are not a proxy for
gender or race/ethnicity, but our rationale is as follows: if most of the
people with Name $N$ have self-identified as belonging to Group $G_{1}$, and
Name $N$ is frequently miscategorized by PII systems at a rate that is higher
than that for a name more commonly used by individuals in Group $G_{2}$, then
we argue that members of Group $G_{1}$ bear a higher privacy risk.

We focus our analysis on given names (sometimes known as ‘first names’) and
family names (sometimes known as ‘surnames’ or ‘last names’). Naming
conventions vary in different cultural and linguistic contexts. In many
cultures, given names and/or family names can be gendered, or
disproportionately associated with a particular gender, religious or ethnic
group. In the present study, gender, race and ethnicity are considered with
respect to a defined set of categories for the purpose of analysis, but we
acknowledge that such labels are socially constructed and mutable over time
and space (Sen and Wasow, 2016).

Previous research has uncovered racial and gender discrimination based on
individual names. Bertrand and Mullainathan (2004) found that, given identical
resumes with only a change in name, resumes with Black-associated names
received fewer callbacks than White-associated names. Sweeney (2013) found
that internet searches for Black (in contrast to White) names were more likely
to trigger advertisements that suggested the existence of arrest records for
people with those names.

We do not attempt to infer personal information tied to names in our data, but
rather, rely on real, self-reported information. However, there are
limitations to using standardized gender and racial categories in studying
algorithmic fairness, even when individuals are able to self-identify (Hanna
et al., 2020). Within each racial/ethnicity category made available on the
standardized forms in the data we use (described in Section 4), for example,
there is a large variety in the linguistic cultures and naming practices
encompassed in each group. Our intent is not to conflate race and ethnicity
and language, but rather to get a coarse-grained look at performance of PII
masking systems on names that are strongly associated with the demographic
groupings that are available. Similarly, the available data limits gender
categories to the binary ‘male’ and ‘female,’ and while names are not a good
proxy for gender, we look for strong associations in the data, as described
further in Section 4.

##  4 Data

In this section, we describe our method for creating test sentences for
evaluating name detection in PII masking models. In our evaluation, we use a
sentence perturbation technique which is employed in previous studies to test
model performance across sensitive groups (Garg et al., 2019; Hutchinson et
al., 2020). Using a variety of templates, we fill slots with names from the
datasets, allowing us to measure performance across race/ethnicity and gender.

Reliable sources of demographically labeled names are difficult to find and
using real names is an issue of privacy. Therefore, we consider datasets of
names with aggregate demographic information as a proxy. We also evaluate on
the names of US Congress members, whose identity and self-reported demographic
information is publicly available. Templates and source datasets are described
in the following sections.

###  4.1 Templates

We collected a set of 32 templates from real-world customer service messaging
conversations (see examples in Table 1 and the full set in Appendix A.3).
These include dialog between customers and conversational AI or human agents.
Customer service data is especially vulnerable to security threat, carrying
potentially sensitive personal information such as credit card or social
security numbers. Topics of discussion in the dataset include placing or
tracking a purchase or paying a bill. Each template contains a name, which we
replace with a generic name slot. Various identifiers from the dataset (e.g.
location or reference numbers) are swapped to protect personal information.

Sample Templates
---
This was from $<$name$>$
The response is signed $<$name$>$
it’s YGDFEA the reservation.
$<$name$>$
Table 1: Sample of templates used for analysis.

###  4.2 LAR Data

The LAR dataset from Tzioumis (2018) contains aggregate names with self-
reported race/ethnicity from US Loan Application Registrars (LARs). It
includes 4.2K given names from 2.6M observations across the US. Race/ethnicity
categories are shown in Table 2.

There are limitations to the Tzioumis (2018) dataset. Because the sample is
drawn from mortgage applications and there are known racial and socioeconomic
differences in who applies for mortgage applications (Charles and Hurst,
2002), the data is likely to contain representation bias. However, the LAR
dataset is the largest available set of names and demographics, estimated to
reflect 85.6% of names in the US population (Tzioumis, 2018). Due to its large
size, we are able to control for the frequency of names, as described in
Section 5.

###  4.3 NYC Data

The NYC dataset was created using the New York City (NYC) Department of Health
and Mental Hygiene’s civil birth registration data (NYC Open Data, 2013) and
contains 1.8K given names from 1.2M observations. Data is available from
2011-2018 and includes self-reported race/ethnicity of the birth mother (other
parents’ information is not available). The sex of the baby is included, which
permits an intersectional analysis.222Although the NYC data includes the
child’s sex assigned at birth, we use this variable to approximate the gender
associated with the name. The race/ethnicity groups are shown in Table 2.

While the other datasets report on adult names, the NYC data aggregates the
names of children who are between 4-11 at the time of this writing. This adds
diversity in terms of age, as data privacy is an important issue for both
children and adults.

###  4.4 Congress Data

The Congress dataset allows for evaluation over the given and family names of
real individuals. The 540 current members of US Congress provide self-reported
demographic information.333See www.senate.gov and
https://pressgallery.house.gov/member-data/demographics. Race/ethnic groups
are described in Table 2. 76% of congress members do not report membership in
the race/ethnicity groups listed, and are grouped as “White/Other”.

This dataset provides a naturalistic analysis of full names. Alternatively,
one could programmatically generate given and family name pairs from datasets
of first names and a dataset of last names. However, the broad race/ethnic
groups used for classification do not account for the variance in the cultural
backgrounds of the names (e.g. Pakistani and Native Hawaiian backgrounds are
listed under the umbrella of Asian and Pacific Islander).

Data | Dataset Race/Ethnicity Group | Mapped label
---|---|---
LAR | NH Asian or Native Hawaiian or Other Pacific Islander | Asian and Pacific Islander
| NH Black or African American | Black
| Hispanic or Latino | Hispanic
| NH American Indian or Alaska Native | Indigenous
| NH Multi-race | Multi-race
| NH White | White
NYC | Asian and Pacific Islander | Asian and Pacific Islander
| Black | Black
| Hispanic White | Hispanic
| NH White | White
Cong. | Asian | Asian and Pacific Islander
| Black | Black
| Hispanic | Hispanic
| Indigenous | Indigenous
| White/Other | White
Table 2: Race/ethnicity categories used for each data source and the mapped
set of race/ethnic group labels each category is mapped to for our analysis.
The term “Non-Hispanic” is abbreviated NH.

##  5 Sampling Process

This section describes the process of sampling the source names. The LAR and
NYC datasets aggregate name counts and frequencies per race/ethnicity. We
sample names which have a strong ‘association’ with a particular
race/ethnicity and gender. Because frequency (i.e. popularity) of a name could
contribute to spurious performance disparities between groups, we sample the
LAR data so that all names are frequency matched across groups.

###  5.1 Demographic categorization

For each group, we sample names that are “associated” with that particular
group. We define “association” as when $>$75% of people with the same name
self-report within the same race/ethnicity. In the LAR dataset, the NH
American Indian or Alaska Native and NH Multi-race names reflect 1% of
individuals in the dataset (Tzioumis, 2018). No names were found with strong
associations in these groups, and for this reason, we do not include them in
the analysis. We map race/ethnicity groups across datasets to a common set of
labels, which are based on categories of the 2010 US Census dataset of surname
and race/ethnicity information (Comenetz, 2016). Race/ethnicity categorization
for all datasets is shown in Table 2.

The NYC dataset also includes gender. Using a 90% threshold for our definition
of ‘association’, 99% of names in the source set are strongly association with
one gender.

###  5.2 Frequency matching

Because the LAR dataset has a large sample size, it is possible to control for
the frequency of names while maintaining a minimum threshold of 20 names per
category. To standardize based on frequency, we use counts from the 2010 US
Census Bureau. We did not use observation counts directly from the LAR data,
due to the aforementioned potential for representational bias.

We sample the LAR dataset to align the mean observation counts of Black-
associated names and other groups, as there are few Black-associated names in
the dataset (n=21). However, there is limited overlap in the frequency
distributions of API-associated names with Hispanic and Black-associated
names. Therefore, we sample a second set with API and White-associated names
only. We refer to these datasets as LAR1 (Black, Hispanic, and White) and LAR2
(API and White). The frequency matching process is described in more detail in
Appendix A.2.

##  6 Experiment Setup

The following sections discuss the PII masking systems we evaluate. We use
several metrics to investigate the PII masking performance across name
subsets.444Experiment code is publically available at
https://github.com/csmansfield/pii-masking-bias.

###  6.1 Models

We select two commercial and one open-source PII masking system for
evaluation. The commercial systems we consider are Amazon Web Services (AWS)
Comprehend and Google Cloud Platform Data Loss Prevention (GCP DLP). We choose
these systems for their potentially large reach, with AWS and GCP holding a
combined 43% market share of cloud
services.555https://www.statista.com/chart/18819/worldwide-market-share-of-
leading-cloud-infrastructure-service-providers/ Amazon Comprehend provides an
English model with a Name entity for PII redaction. GCP DLP offers redaction
and includes a global PERSON_NAME entity. Microsoft’s Presidio is an open-
source service for PII detection. We use the default English model which uses
logic such as regex matching and Named Entity Recognition (NER). For the
Presidio model we use a spaCy 3.2 en_core_web_trf model for NER, which
utilizes the RoBERTa-base Transformer model trained on OntoNotes 5.

###  6.2 Evaluation metrics

We measure false negative rates (FNRs), the rate at which a PII system does
not detect a name that is present in the dataset (and therefore is unable to
mask it).666Whereas false positive rates are useful for evaluating the
precision of a model, our focus is the failure to detect person names, rather
than the incorrect identification of tokens that are not person names.
Furthermore, we report no false positives in our findings. Following Dixon et
al. (2018) we report on the False Negative Equality Difference, which measures
differences between the false negative rate over the entire dataset and across
each demographic subgroup $g$. We add a normalization term to compare the FNED
of datasets with different numbers of groups, as shown in equation 1.

$\frac{1}{|G|}\sum_{g\in G}|FNR-FNR_{g}|$ (1)

We also measure the statistical significance of performance differences across
subgroups. We conduct Friedman and Wilcoxon signed-rank tests following
Czarnowska et al. (2021). The Friedman test is used for cases with more than 2
subgroups, and provides a single p-value for each dataset and system pair. The
p-value determines whether to reject the null hypothesis that FNR of a given
system is the same across all demographic groups. The statistic is calculated
considering $j$ demographic subsets $g$. First, we calculate the average FNR
for a template $t$, over all names belonging to a particular subset $g$. The
averages for each of the 32 templates considering group $g$ are contained in
$X_{g}$. The Friedman statistic is calculated for all $X_{g}$.

$\displaystyle X_{g}=(FNR(x_{g}^{1}),...,FNR(x_{g}^{32}))$ $\displaystyle
Friedman(X_{1},...,X_{j})$ (2)

Nemenyi post-hoc testing is used for further pairwise analysis. For cases with
only 2 subgroups, we alternatively perform Wilcoxon signed-rank tests. In
order to control for multiple comparisons, we apply a Bonferroni correction
across all p-values (at p$<$0.05 and n=15, our adjusted significance threshold
is 0.003).

##  7 Results

We present the results of the evaluation, considering overall performance and
performance related to race/ethnicity, gender, and intersectional factors. The
section concludes with an analysis of errors.

###  7.1 Overall Performance

The average performance on the datasets can be seen in Table 3. System
performance varies according to the dataset, with no single system performing
best on all sets. All systems have lower FNR on the Congress dataset, where
both given and family names are available, likely due to the increased
information load of full names. The LAR2 and NYC names prove the most
challenging across all systems.

| Group | N | FNR (%)
---|---|---|---
|  |  | AWS | GCP | MP
LAR1 | Black | 20 | 20.0 | 18.1 | 29.5
| Hisp. | 172 | 28.4 | 12.4 | 24.7
| White | 1000 | 21.3 | 18.5 | 20.0
| All | 1192 | 22.3 | 17.6 | 20.8
LAR2 | API | 441 | 38.2 | 51.2 | 29.2
| White | 1000 | 25.3 | 18.6 | 25.8
| All | 1441 | 29.3 | 28.6 | 26.8
NYC | API | 165 | 21.3 | 43.6 | 22.0
| Black | 226 | 28.9 | 56.3 | 32.6
| Hisp. | 389 | 20.1 | 34.2 | 21.2
| White | 592 | 26.9 | 29.2 | 25.9
| All | 1359 | 24.6 | 36.8 | 25.2
Cong. | API | 16 | 23.0 | 12.1 | 11.7
| Black | 56 | 15.2 | 9.7 | 9.5
| Hisp. | 48 | 13.9 | 8.3 | 9.4
| Indig.$\dagger$ | 3 | 7.0 | 6.3 | 7.8
| Multi.$\dagger$ | 6 | 8.3 | 6.3 | 10.9
| White/ | 419 | 12.1 | 6.7 | 7.7
| Other |  |  |  |
| All | 530 | 12.8 | 7.3 | 8.1
Table 3: Support and average false negative rate (FNR) by race/ethnicity group
across datasets. Groups marked with ‘$\dagger$’ are not included in formal
statistical analysis due to low support. Maximum FNR per dataset/sytem is
shown in bold.

The average performance of the names per each template is shown in Figure 1.
Performance varies considerably, with average FNR per template ranging between
6%. and 100%. The mean FNR for all templates is 22%.

Figure 1: Average FNR across each template per dataset.

###  7.2 Performance by Race/Ethnicity

The normalized false negative equality differences (FNEDs) are shown in Table
4.

|  | FNED
---|---|---
|  | AWS | GCP | MP
| LAR1 | *3.1 | *2.2 | *4.4
Race/ | LAR2 | *6.4 | *16.3 | 1.7
ethnicity | NYC | *3.6 | *8.9 | *3.7
| Congress | *3.6 | *2.2 | 1.7
Gender | NYC | *3.2 | *4.4 | 0.8
Congress | *1.3 | *0.6 | 0.2
Table 4: The normalized false negative equality difference (FNED) for
race/ethnicity and gender subsets of the data. Asterisks indicate significance
(p$<$0.003) in FNR differences by group. Maximum FNED per system is shown in
bold.

The highest FNED, which is an 82% increase over the second highest FNED, is
seen in GCP’s performance over the LAR2 dataset which includes frequency
controlled API and White-associated names. The FNRs in Table 3 show high FNR
for API names in LAR2 across all systems. The error rate for GCP is 175%
higher for API-associated names in this set. A Wilcoxon signed-rank test shows
significant differences in FNR for AWS and GCP, with better performance on
White-associated names. The Presidio transformer model has a smaller gap which
is not found to be significant.

Performance on LAR1, which includes frequency-balanced Black, Hispanic, and
White-associated names, also shows variability in FNR across race/ethnicity
groups. However, the performance differences across groups are dependent on
the system. For example, the Presidio transformer model shows poor performance
on Black-associated names, and post-hoc tests (see Appendix A.1) reveal
significant differences between Black vs. Hispanic and White groups. On the
other hand, AWS performs best on Black-associated names but significantly
worse on Hispanic-associated names. GCP peforms worst on White-associated
names.

The NYC dataset shows more consistency in terms of performance across groups,
with Black-associated names having higher FNRs across all systems. This is
further confirmed by statistical testing on AWS and GCP, where Black-
associated names have statistically higher FNR than Hispanic-associated names.
GCP also performs significantly worse on Black-associated names than White-
associated names. Although significant FNR differences are found in the
performance of Presidio on the basis of race/ethnicity, post-hoc tests did not
indicate pair(s) which met the threshold for significance.

Finally, the Congress dataset, which includes given and family names, has the
lowest FNED rates in terms of race/ethnicity. However, there are still
significant differences in performance across groups for AWS and GCP maskers.
Here, API-associated names again show high FNRs. Friedman tests and post-hoc
testing support differences between API and other groups in the case of AWS
and GCP. Performance on Black-associated names was also significantly worse
than on White-associated names for GCP. There were no significant differences
associated with the Presidio model.

###  7.3 Performance by Gender

The NYC and Congress datasets also include information about gender, which
allows for a comparison of gender-based subsets. The FNEDs in Table 4 are
generally lower for gender than for race. However, some gender-based
differences are shown to be significant.

The average FNR grouped by gender is shown in Table 5. The NYC dataset shows
female-associated, male-associated, and ‘other’ names, which are not strongly
associated with a particular gender. FNR is highest for such unassociated
names. Performance on female and male-associated names varies, with AWS
performing significantly better on female-associated names, and GCP performing
significantly better on male-associated names.

| Gender | N | FNR (%)
---|---|---|---
|  |  | AWS | GCP | MP
NYC | F | 741 | 23.7 | 39.8 | 25.1
| M | 618 | 25.6 | 33.1 | 25.3
| Other $\dagger$ | 13 | 32.2 | 43.3 | 27.4
| All | 1359 | 24.5 | 36.8 | 25.2
Cong. | F | 145 | 11.0 | 8.2 | 10.0
| M | 385 | 13.6 | 7.0 | 8.5
| All | 530 | 12.9 | 7.3 | 8.9
Table 5: Support and average false negative rate (FNR) by gender across
datasets. ‘Other’ specifies names which are not strongly associated with one
gender. Groups marked with ‘$\dagger$’ are not included in formal statistical
analysis due to low support. Maximum FNR per dataset/system is shown in bold.

###  7.4 Intersectional Analysis

We analyzed the NYC results for differences across both race/ethnicity and
gender. Table 6 shows FNR averages associated with intersectional groups. FNR
for Black female-associated names is highest among all groups, and error rates
are on average 13.7% higher than that of the full dataset. Black male-
associated names have the second highest FNR for GCP and MP. Pairwise testing
does not reveal significant differences between Black male and female-
associated names. The subsets with the lowest FNR vary across systems.
Hispanic-associated names have the lowest FNR in AWS and Presidio. For GCP,
White male-associated names have the lowest FNR.

Group | Gender | N | FNR (%)
---|---|---|---
|  |  | AWS | GCP | MP
API | F | 86 | 20.1 | 43.0 | 22.2
| M | 77 | 22.1 | 43.9 | 22.2
Black | F | 122 | 30.1 | 62.8 | 34.7
| M | 101 | 27.0 | 47.2 | 29.2
Hisp. | F | 212 | 18.4 | 35.7 | 21.3
| M | 175 | 22.2 | 32.2 | 21.1
White | F | 321 | 25.7 | 32.9 | 24.8
| M | 265 | 28.2 | 25.2 | 27.4
All | - | 1359 | 24.5 | 36.8 | 25.2
Table 6: Support and average false negative rate (FNR) by race/ethnicity and
gender in the NYC dataset. Maximum FNR per system is shown in bold.

###  7.5 Analysis of Names

The previous findings in this section captured a few general patterns. One
pattern that held across most systems and datasets was high false negative
rates of API names. In the LAR2 and Congressional datasets, API names were
especially hard for systems to detect. This was not simply due to API names
being less common, as the LAR2 set included names balanced by their frequency
in the general US population.

Table 7 shows examples of names with the highest and lowest FNRs. It is worth
noting that API names in LAR2 with high FNR are nearly all 2 characters long.
Figure 2 shows the relationship between average FNR across all systems, name
length, and group. FNR is lowest for 6-7 character names, and increases as
length decreases. However, when matched by character length, API-associated
names have higher FNRs than Hispanic and White-associated names nearly across
the board. There appear to be higher penalties for short names in the API and
Black groups.

|  Low FNR |  High FNR
---|---|---
LAR1 |  Bob (H), Kristan (W), Vicki (W), Nickie (W), Bethann (W) |  German (H), Houston (W), Denver (W), Royal (W), Said (W)
LAR2 |  Maher (W), Nguyen (A), Rajesh (A), Nicoletta (W), Jayesh (A) |  Man (A), My (A), In (A), Do (A), So (A)
NYC |  Kaylie (H/F), Keith (W/M), Lena (W/F), Brody (W/M), Brendan (W/F) |  Egypt (B/F), Empress (B/F), Asia (B/F), Major (B/M), Malaysia (B/F)
Congress |  Louie Gohmert (W/M), Deborah Ross (W/F), Diana DeGette (W/F), Fred Keller (W/M), Dianne Feinstein (W/F) |  Lisa Blunt Rochester (A/F), Aumua Amata Radewagon (A/F), A. Ferguson (W/M), A. McEachin (B/M), Young Kim (A/F)
Table 7: A sample of names with the highest and lowest FNR on average per each
dataset. Race/ethnicity is abbreviated as API (A), Black (B), Hispanic (H),
and White (W), while gender is abbreviated female (F), male (M). Figure 2:
Average FNR across all systems by character length and race/ethnic group.

High FNR names in Table 7 tend to coincide with other word senses in English.
Many are location words (e.g. German, Rochester, Asia). Others double as verbs
(‘Said’), adjectives (‘Young’), nouns (‘Major’), and function words (‘In’).
Using WordNet (Fellbaum, 1998), a lexical database of English, we examine
given names that have overlapping (non-person) senses. Potentially ambiguous
given names have a 42% FNR compared to 24% for non-ambiguous names. However,
the penalty of having an ambiguous name is not the same across groups. Figure
3 shows that there is a large performance disparity for Black names with
multiple senses. This is seen anecdotally in names with similar
syntactic/semantic content. For instance, the name ‘Joy’ (API) has a 60% lower
FNR (averaged across systems) than ‘Blessing’ (Black), and ‘Georgia’ (White)
has a 25% lower FNR than ‘Egypt’ (Black).

Figure 3: FNR for names with one or multiple word senses (i.e. including non-
person word senses)

##  8 Discussion

This paper considers differences in the performance of three PII maskers on
recognizing and redacting names based on demographic characteristics.
Supported by quantitative results and error analysis, we find disparities in
the fairness of name masking across groups.

In terms of race and ethnicity, API-associated names are often poorly masked.
Disparities are shown to be significant for AWS and GCP systems. This is not
simply a result of the popularity of the names, as the frequency-controlled
LAR1 dataset revealed disparities between API and White-associated names. Name
length is considered as a performance factor, but it does not entirely account
for the gap between API and White-associated names.

Several systems and datasets show poor performance on the masking of Black-
associated names. GCP and Presidio revealed significant differences between
Black and White-associated names. Error rates are especially high on the NYC
dataset, and are highest for Black women. This is in line with previous
research which demonstrates the poor performance of NLP systems on Black women
(see inter alia Buolamwini and Gebru, 2018).

Race and ethnicity were the strongest factors related to PII masking
performance, but gender-based differences were also noted. Names which were
not strongly associated with gender had the highest error rates. This
underscores the importance of considering categories outside the traditional
gender binary when evaluating systems for bias.

Of all PII masking systems, the Presidio model (with roBERTa NER) shows fewer
significant discrepancies based on demographics. However, all systems
demonstrate some significant disparities. Across datasets, the performance
difference between groups is not consistent. For instance, the AWS model has
poor performance on API names in the LAR2 dataset but not in NYC. We consider
this not an issue, but a feature of our evaluation across datasets. The
datasets we’ve chosen contain variety in age groups, locations, and contexts.
We argue that evaluating NLP systems responsibly requires careful curation of
data, including steps to consider the context of the system and the diverse
set of system users and stakeholders.

The aggregate name data used here is openly available and can be used for
testing on PII masking, NER, and related systems. We are releasing our
templates and code used for sampling data. However, we strongly condemn the
use of these datasets for predictive purposes, such as identifying a person’s
race/ethnicity or gender on the basis of their name without their consent.
While our collection of name data forms one of the most comprehensive sets of
aggregate names and demographic information available, we are limited by
availability of data. The sample of Indigenous and mixed-race names was small,
and names were sampled almost exclusively from US-born citizens. In the
future, we would like to consider collaborating with the public by developing
a database where individuals may actively choose to contribute their name and
self-identified information for research.

##  9 Conclusion

This work considers the performance of PII masking systems on names sourced
from real data. We find disparities related to demographic characteristics,
especially race and ethnicity, across all systems. While features such as name
length and ambiguity play a role in recognition, they do not fully account for
performance differences. Disparities in the performance of PII masking systems
reflect historical inequities in the “Right to Privacy”. The NLP community, as
a commodifier of both models and data, has a responsibility to develop more
equitable systems to protect the data privacy of all individuals.

## Acknowledgments

The authors thank Emily M. Bender, Joe Bradley, Chris Brew, Andrew Maurer, and
the anonymous reviewers for their helpful comments.

At different points over the course of the work presented in this paper, A.P.
was supported by a research internship at LivePerson, Inc. and also by the
National Institutes of Health, National Library of Medicine (NLM) Biomedical
and Health Informatics Training Program at the University of Washington (Grant
Nr. T15LM007442). The content is solely the responsibility of the authors and
does not necessarily represent the official views of the National Institutes
of Health.

## References

  * Adams et al. (2019) Allison Adams, Eric Aili, Daniel Aioanei, Rebecca Jonsson, Lina Mickelsson, Dagmar Mikmekova, Fred Roberts, Javier Fernandez Valencia, and Roger Wechsler. 2019.  Anonymate: A toolkit for anonymizing unstructured chat data.  In _Proceedings of the Workshop on NLP and Pseudonymisation_ , pages 1–7.
  * Beckwith et al. (2006) Bruce A Beckwith, Rajeshwarri Mahaadevan, Ulysses J Balis, and Frank Kuo. 2006.  Development and evaluation of an open source software tool for deidentification of pathology reports.  _BMC medical informatics and decision making_ , 6(1):1–9.
  * Bertrand and Mullainathan (2004) Marianne Bertrand and Sendhil Mullainathan. 2004.  Are Emily and Greg more employable than Lakisha and Jamal? a field experiment on labor market discrimination.  _American economic review_ , 94(4):991–1013.
  * Browne (2015) Simone Browne. 2015.  _Dark matters_.  Duke University Press.
  * Buolamwini and Gebru (2018) Joy Buolamwini and Timnit Gebru. 2018.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In _Conference on fairness, accountability and transparency_ , pages 77–91. PMLR.
  * Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021.  Extracting training data from large language models.  In _30th USENIX Security Symposium (USENIX Security 21)_ , pages 2633–2650.
  * Charles and Hurst (2002) Kerwin Kofi Charles and Erik Hurst. 2002.  The transition to home ownership and the black-white wealth gap.  _Review of Economics and Statistics_ , 84(2):281–297.
  * Comenetz (2016) Joshua Comenetz. 2016.  Frequently occurring surnames in the 2010 census.  _United States Census Bureau_ , pages 1–8.
  * Czarnowska et al. (2021) Paula Czarnowska, Yogarshi Vyas, and Kashif Shah. 2021.  Quantifying social biases in nlp: A generalization and empirical comparison of extrinsic fairness metrics.  _Transactions of the Association for Computational Linguistics_ , 9:1249–1267.
  * Dernoncourt et al. (2017) Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. 2017.  De-identification of patient notes with recurrent neural networks.  _Journal of the American Medical Informatics Association_ , 24(3):596–606.
  * Dixon et al. (2018) Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018.  Measuring and mitigating unintended bias in text classification.  In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 67–73.
  * Fagan et al. (2016) Jeffrey Fagan, Anthony A Braga, Rod K Brunson, and April Pattavina. 2016.  Stops and stares: Street stops, surveillance, and race in the new policing.  _Fordham Urb. LJ_ , 43:539.
  * Fellbaum (1998) Christiane Fellbaum. 1998.  _WordNet: An Electronic Lexical Database_.  Bradford Books.
  * Friedlin and McDonald (2008) F Jeff Friedlin and Clement J McDonald. 2008.  A software tool for removing patient identifying information from clinical documents.  _Journal of the American Medical Informatics Association_ , 15(5):601–610.
  * Garg et al. (2019) Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H Chi, and Alex Beutel. 2019.  Counterfactual fairness in text classification through robustness.  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 219–226.
  * Hanna et al. (2020) Alex Hanna, Emily Denton, Andrew Smart, and Jamila Smith-Loud. 2020.  Towards a critical race methodology in algorithmic fairness.  In _Proceedings of the 2020 conference on fairness, accountability, and transparency_ , pages 501–512.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional lstm-crf models for sequence tagging.  _arXiv preprint arXiv:1508.01991_.
  * Hutchinson et al. (2020) Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020.  Social biases in nlp models as barriers for persons with disabilities.  _arXiv preprint arXiv:2005.00813_.
  * Kayaalp (2018) Mehmet Kayaalp. 2018.  Patient privacy in the era of big data.  _Balkan medical journal_ , 35(1):8–17.
  * Li et al. (2020) Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020.  A survey on deep learning for named entity recognition.  _IEEE Transactions on Knowledge and Data Engineering_ , 34(1):50–70.
  * Li et al. (2019) Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2019\.  A unified mrc framework for named entity recognition.  _arXiv preprint arXiv:1910.11476_.
  * Mishra et al. (2020) Shubhanshu Mishra, Sijun He, and Luca Belli. 2020.  Assessing demographic bias in named entity recognition.  _arXiv preprint arXiv:2008.03415_.
  * NYC Open Data (2013) NYC Open Data. 2013.  Popular baby names.  https://data.cityofnewyork.us/Health/Popular-Baby-Names/25th-nujf/data.
  * Osucha (2009) Eden Osucha. 2009.  The whiteness of privacy: Race, media, law.  _Camera Obscura: Feminism, Culture, and Media Studies_ , 24(1):67–107.
  * Sen and Wasow (2016) Maya Sen and Omar Wasow. 2016.  Race as a bundle of sticks: Designs that estimate effects of seemingly immutable characteristics.  _Annual Review of Political Science_ , 19:499–522.
  * Sweeney (2002) Latanya Sweeney. 2002.  k-anonymity: A model for protecting privacy.  _International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems_ , 10(05):557–570.
  * Sweeney (2013) Latanya Sweeney. 2013.  Discrimination in online ad delivery.  _Communications of the ACM_ , 56(5):44–54.
  * Szarvas et al. (2006) György Szarvas, Richárd Farkas, and András Kocsor. 2006.  A multilingual named entity recognition system using boosting and c4. 5 decision tree learning algorithms.  In _International Conference on Discovery Science_ , pages 267–278. Springer.
  * Thomas et al. (2002) Sean M Thomas, Burke Mamlin, Gunther Schadow, and Clement McDonald. 2002.  A successful technique for removing names in pathology reports using an augmented search and replace method.  In _Proceedings of the AMIA Symposium_ , page 777. American Medical Informatics Association.
  * Tzioumis (2018) Konstantinos Tzioumis. 2018.  Demographic aspects of first names.  _Scientific data_ , 5(1):1–9.
  * Uzuner et al. (2008) Özlem Uzuner, Tawanda C Sibanda, Yuan Luo, and Peter Szolovits. 2008.  A de-identifier for medical discharge summaries.  _Artificial intelligence in medicine_ , 42(1):13–35.

##  Appendix A Appendices

###  A.1 Post-hoc testing

Nemenyi post-hoc significance testing for each dataset. Significance for each
respective system is marked with their respective abbreviation: AWS Comprehend
(A), GCP DLP (G), and Microsoft Presidio (P). A ‘-’ indicates a p-value above
the significance threshold

| Black | Hispanic | White
---|---|---|---
Black | \- - - | AG - | \- GP
Hispanic | AG - | \- - - | AG -
White | \- GP | AG - | \- - -
Table 8: LAR1 dataset with race/ethnicity |  | API | Black | Hispanic | White
---|---|---|---|---|---
|  | F | M | F | M | F | M | F | M
API | F | \- - - | \- - - | AG - | A - - | \- - - | \- G - | A - - | AG -
| M | \- - - | \- - - | A - - | A - - | \- - - | \- G - | \- G - | AG -
Black | F | AG - | A - - | \- - - | \- - - | AG - | AG - | \- G - | \- G -
| M | A - - | A - - | \- - - | \- - - | AG - | AG - | \- G - | \- G -
Hispanic | F | \- - - | \- - - | AG - | AG - | \- - - | \- - - | A - - | AG -
| M | \- G - | \- G - | AG - | AG - | \- - - | \- - - | \- - - | A - -
White | F | A - - | \- G - | \- G - | \- G - | A - - | \- - - | \- - - | \- - -
| M | AG - | AG - | \- G - | \- G - | AG - | A - - | \- - - | \- - -
Table 9: NYC dataset with gender, race/ethnicitiy | API | Black | Hispanic | White
---|---|---|---|---
API | \- - - | A - - | AG - | AG -
Black | A - - | \- - - | \- - - | \- G -
Hispanic | AG - | \- - - | \- - - | \- - -
White | AG - | \- G - | \- - - | \- - -
Table 10: Congress dataset with race/ethnicity. The Presidio model did not
differ significantly based on race/ethnic group.

###  A.2 Frequency sampling

This appendix describes in more detail the frequency matching between
race/ethnicity groups in the LAR dataset. The mean observation frequencies for
each group are shown in Table 11. Because there are initially fewer Black-
associated names (n=21), we sample all groups to target this smaller
distribution. By filtering with a minimum observation size of 2K and maximum
observation size of 150K, we achieve similar distributions across groups.
However, API names are too sparse under these conditions to be included, and
we choose to resample them separately. A Mann-Whitney U test does not find
significant differences in frequency between Black, Hispanic, and White-
associated names under these conditions (with a threshold of $p=0.05$). A plot
of the distributions of this set, which we refer to as LAR1, is shown in
Figure 4(a).

For API names, we generate a second name set, which we refer to as LAR2. We
sample from other groups, using an exponential distribution ($\lambda=480$)
that best approximates the API distribution. Only White-associated names
maintain $>$20 names under these sampling conditions. A Mann-Whitney U test
does not find significant differences between frequencies of API and White
groups. Distributions of this set are shown in Figure 4(b).

Group | N
---|---
API | 488
Black | 21573
Hispanic | 25122
White | 41060
Table 11: Average observation size per name for each race/ethnicity group in
the LAR dataset without resampling.

(a)

(b)

Figure 4: Plots of frequency distributions for frequency-matched names from
LAR.

###  A.3 Templates

# |  Template
---|---
1 |  Name: {{Name}} Vouchers:10000200007400001 10000200005000001
2 |  sysmsg1_{{Name}}_ has joined the conversation,
3 |  Craig G: 1F to LAS and 2F to SAN {{Name}} 1D to LAS and 2D to SAN
4 |  {{Name}} 03 caramel beige is my another foundation
5 |  i put in an order on line for {{Name}} original large size and a code for 20 present off of the 117.00 but it would not take
6 |  Hi {{Name}}! Can you help me with my above question?
7 |  hi im {{Name}}
8 |  {{Name}} isle Jake window
9 |  Virtual Assistant : Hi {{Name}}, how can I help you today?
10 |  Thank you, {{Name}}
11 |  this was from {{Name}}
12 |  I think it’s {{Name}}
13 |  Ok, will we receive {{Name}}’s by that date and at that address as well?
14 |  {{Name}}. Very upset at the moment. I placed two request online to have this order cancelled and I just refused an item from FedEX from your store.
15 |  Hello {{Name}}, Im just trying to get some info on the item I ordered
16 |  {{Name}} (I) paid for the ticket
17 |  sysmsg2_{{Name}}_ has left the conversation
18 |  hey I lost connection from my previous chat with {{Name}}
19 |  Virtual Assistant : Hi {{Name}}, we’ll use automated messages to chat with you and Customer Care Professionals are standing by. In a short sentence, let me know how I can help you today
20 |  thank you very much {{Name}}. nice chatting with you!
21 |  well .. thank u so much {{Name}} ..
22 |  Did {{Name}} catch you up on everything?
23 |  I was working with {{Name}} earlier on this chat
24 |  The response is signed {{Name}}
25 |  it’s YGDFEA the reservation. {{Name}}
26 |  My name is {{Name}}. I messaged yesterday and have not received a response from anyone
27 |  {{Name}} and I divorced.
28 |  do you care that something holy to me was in my food {{Name}}?
29 |  {{Name}} was very kind and helpful!
30 |  oh no {{Name}} sorry to confuse you
31 |  the order is under {{Name}}
32 |  {{Name}}, one question, when i logged into the App, it shows balance as $50.. is it USD or CAD?


# Sinh-Gordon type equations and Bäcklund transformations

G. Polychrou  Giannis<EMAIL_ADDRESS>
Department of Mathematics, Aristotle University of Thessaloniki, Thessaloniki
54124, Greece

###### Abstract.

We study the hyperbolic and elliptic versions of the sine-Gordon and sinh-
Gordon equations on the real plane in a unified way. We introduce solutions
via the method of functional separation. We also study a related Bäcklund
transformation.

###### Key words and phrases:

sinh-Gordon equation, sine-Gordon equation, Bäcklund transformation, Auto
Bäcklund transformation.

###### 2010 Mathematics Subject Classification:

35A25, 35A30, 35R35

##  1\. Introduction and Statement of the Results

Our aim is to study a sinh-Gordon type equation of the form

(1)
$\frac{\delta}{\epsilon}\left(w_{xx}-\epsilon^{2}w_{yy}\right)=2\left(a^{2}-\delta^{2}b^{2}\right)\sinh\left(\frac{2\delta}{\epsilon}w\right),$

where $\delta,\epsilon\in\\{1,i\\}$ and $a,b\in\mathbb{R}$. In the recent work
of Fotiadis and Daskaloyannis, [8], this equation plays a vital role in the
construction of harmonic diffeomorphisms between Pseudo-Riemannian surfaces.
Equation (1), for a suitable choice of $\delta$ and $\epsilon$, turns out to
be the hyperbolic or elliptic version of the sinh-Gordon equation or sine-
Gordon equation. More precisely, we get:

$w_{xx}-w_{yy}=2(a^{2}-b^{2})\sinh(2w),(\delta=\epsilon=1),$
$w_{xx}+w_{yy}=2(a^{2}-b^{2})\sin(2w),(\delta=1,\epsilon=i),$
$w_{xx}-w_{yy}=2(a^{2}+b^{2})\sin(2w),(\delta=i,\epsilon=1),$
$w_{xx}+w_{yy}=2(a^{2}+b^{2})\sinh(2w),(\delta=\epsilon=i).$

These equations are of certain importance with many applications, for example,
[4, 5, 11, 12, 15, 21]. Research interest increased since the introduction of
the soliton theory. For more details, we refer to [10, 20]. The elliptic sinh-
Gordon equation plays a vital role in the theory of constant mean curvature
surfaces [1, 3, 13, 14, 22, 24] and in the theory of harmonic maps [6, 7, 8,
16, 17, 18, 25].

There are several methods to construct solitons of (1). In [8], the authors
present the one-soliton solution for this equation. Other methods to find
explicit solutions of these equations are via functional separation of
variables [2, 9, 19, 23] and the use of Bäcklund transformation [6, 11, 20].

In this work we will describe all the functionally separation solutions of (1)
and we generalise the result of [17, 18]. At first, we prove the following
main results.

######  Definition 1.

Let $w(x,y)$ be a solution of a PDE. A seperable solution of a PDE is of the
form

$f(w(x,y))=A(x)+B(y).$

Equivallently, $w(x,y)$ satisfies an equation of the form

(2) $w_{xy}=F(w)w_{x}w_{y},$

with $F=-f^{\prime\prime}/f^{\prime}$.

######  Proposition 2.

Let $w(x,y)$ be a separable solution to (1). Then $F(w)$ is of the form

(3) $F(w)=\frac{\delta}{\epsilon}\tanh\left(\frac{\delta}{\epsilon}w\right),$

or

(4) $F(w)=\frac{\delta}{\epsilon}\coth\left(\frac{\delta}{\epsilon}w\right).$

######  Theorem 3.

The only cases where we have functional separation solutions of (1), are the
solutions of the form

$\sinh\left(\frac{\delta}{\epsilon}w(x,y)\right)=\tan\left(\frac{\delta}{\epsilon}\left(A(x)+B(y)\right)\right),$

or

$\tanh\left(\frac{\delta}{\epsilon}\frac{w(x,y)}{2}\right)=\frac{\delta}{\epsilon}F(x)G(y).$

Note that our results extend the results in [17, 18].

Next, we consider at first solutions $w$ that are of the form

(5)
$\tanh\left(\frac{\delta}{\epsilon}\frac{w(x,y)}{2}\right)=\frac{\delta}{\epsilon}F(x)G(y),$

and we state the following result.

######  Theorem 4.

If $w$ is a solution of the generalised sinh-Gordon equation (1) of the form

$\tanh\left(\frac{\delta}{\epsilon}\frac{w(x,y)}{2}\right)=\frac{\delta}{\epsilon}F(x)G(y),$

then the functions $F(x)$ and $G(y)$ satisfy the differential equations

(6) $F^{\prime}(x)^{2}=A\delta^{2}F(x)^{4}+BF(x)^{2}+C,$ (7)
$G^{\prime}(y)^{2}=\delta^{2}CG(y)^{4}+\epsilon^{2}\left(B-4(a^{2}-\delta^{2}b^{2})\right)G(y)^{2}+A,$

where $A,B$ and $C\in\mathbb{R}$.

The second family of solutions $w$ of the generalised sinh-Gordon equation is
of the form

(8)
$\sinh\left(\frac{\delta}{\epsilon}w(x,y)\right)=\tan\left(\frac{\delta}{\epsilon}(A(x)+B(y))\right),$

and we obtain the following result.

######  Theorem 5.

If $w$ is a solution of the generalised sinh-Gordon equation (1) of the form

$\sinh\left(\frac{\delta}{\epsilon}w(x,y)\right)=\tan\left(\frac{\delta}{\epsilon}(A(x)+B(y))\right),$

then the functions $A(x)$ and $B(y)$ satisfy the differential equations

(9)
$A^{\prime\prime}(x)^{2}=-\delta^{2}\epsilon^{2}A^{\prime}(x)^{4}+c_{1}A^{\prime}(x)^{2}+c_{2}$
(10)
$B^{\prime\prime}(y)^{2}=-\delta^{2}\epsilon^{2}B^{\prime}(y)^{4}+\epsilon^{2}\left(c_{1}-8\delta^{2}(a^{2}-\delta^{2}b^{2})\right)B^{\prime}(y)^{2}+c_{3},$

where
$16(a^{2}-\delta^{2}b^{2})^{2}+\delta^{2}\epsilon^{2}\left(c_{3}-c_{2}\right)=4c_{1}(a^{2}-\delta^{2}b^{2})$
and $c_{1},c_{2},c_{3}\in\mathbb{R}$.

Equation (1) is connected via a Bäcklund transformation with a nonlinear
equation, which turns out to be of the same form. Let $c=a+\delta b$ and
$\bar{c}=a-\delta b$ then a Bäcklund transformation is given by the equations:

(11)
$\frac{\delta}{\epsilon}\left(w_{x}-\theta_{y}\right)=-\sinh\left(\frac{\delta}{\epsilon}w\right)\left(c\exp(\delta\theta)+\bar{c}\exp(-\delta\theta)\right),$
(12)
$\frac{\delta}{\epsilon^{2}}\left(w_{y}-\epsilon^{2}\theta_{x}\right)=-\cosh\left(\frac{\delta}{\epsilon}w\right)\left(c\exp(\delta\theta)-\bar{c}\exp(-\delta\theta)\right),$

and it connects solutions of (1) with solutions of

(13)
$\delta\left(\theta_{xx}-\epsilon^{2}\theta_{yy}\right)=c^{2}e^{2\delta\theta}-\bar{c}^{2}e^{-2\delta\theta}.$

To sum up, in Section 2 we prove Proposition 2 and Theorem 3. In Section 3 we
prove Theorems 4 and 5. In Section 4, we investigate an auto-Bäcklund
transformation for the sine-Gordon and sinh-Gordon in a unified approach.
Using the families of solutions described in Theorems 4 and 5, we obtain new
solutions.

##  2\. Solutions via functionally separation

In [1], the author in order to describe the Wente Torus, introduced by Wente
in [24], finds families of solutions of the elliptic sinh-Gordon equation
which satisfy a geometric condition. This condition turns out to be a
nonlinear PDE of the form

$w_{xy}=\tanh(w)w_{x}w_{y},$

or

$w_{xy}=\coth(w)w_{x}w_{y}.$

In this work, we prove that the sinh-Gordon type equations (1) admit only
these families of solutions of the form

$f(w)=A(x)+B(y).$

To prove this we consider

$w_{xy}=F(w)w_{x}w_{y},$

where $F(w):\mathbb{R}\rightarrow\mathbb{R}$,
$F=-f^{\prime\prime}/f^{\prime}$. Our first goal is to find explicitly the
function $F$.

###### Proof of Proposition (2).

The equations

$\frac{\delta}{\epsilon}\left(w_{xx}-\epsilon^{2}w_{yy}\right)=2\left(a^{2}-\delta^{2}b^{2}\right)\sinh\left(\frac{2\delta}{\epsilon}w\right),$

$a,b\in\mathbb{R}$ and $w_{xy}=F(w)w_{x}w_{y}$ have to be compatible. Taking
the derivative with respect to $x$ and $y$ of both equations we get

(14) $\displaystyle w_{xxx}-\epsilon^{2}w_{yyx}$
$\displaystyle=4\left(a^{2}-\delta^{2}b^{2}\right)w_{x}\cosh\left(\frac{2\delta}{\epsilon}w\right),$
(15) $\displaystyle w_{xxy}-\epsilon^{2}w_{yyy}$
$\displaystyle=4\left(a^{2}-\delta^{2}b^{2}\right)w_{y}\cosh\left(\frac{2\delta}{\epsilon}w\right),$
(16) $\displaystyle(w_{xy})_{x}$
$\displaystyle=F^{\prime}(w)w_{x}^{2}w_{y}+F(w)w_{xx}w_{y}+F^{2}(w)w_{x}^{2}w_{y},$
(17) $\displaystyle(w_{xy})_{y}$
$\displaystyle=F^{\prime}(w)w_{x}w_{y}^{2}+F^{2}(w)w_{x}w_{y}^{2}+F(w)w_{x}w_{yy}.$

Solving these equations for the third derivatives $w_{xxx},w_{xxy},w_{xyy}$
and $w_{yyy}$ we get

$\displaystyle w_{xxx}$
$\displaystyle=\epsilon^{2}w_{x}w_{y}^{2}\left(F^{\prime}(w)+F^{2}(w)\right)+\epsilon^{2}F(w)w_{x}w_{yy}$
$\displaystyle+4\left(a^{2}-\delta^{2}b^{2}\right)w_{x}ch\left(\frac{2\delta}{\epsilon}w\right),$
$\displaystyle w_{xxy}$
$\displaystyle=F^{\prime}(w)w_{x}^{2}w_{y}+F(w)w_{xx}w_{y}+F^{2}(w)w_{x}^{2}w_{y},$
$\displaystyle w_{xyy}$
$\displaystyle=F^{\prime}(w)w_{x}w_{y}^{2}+F^{2}(w)w_{x}w_{y}^{2}+F(w)w_{x}w_{yy},$
$\displaystyle w_{yyy}$
$\displaystyle=\epsilon^{2}w_{x}^{2}w_{y}\left(F^{\prime}(w)+F^{2}(w)\right)+\epsilon^{2}F(w)w_{xx}w_{y}$
$\displaystyle-4\epsilon^{2}\left(a^{2}-\delta^{2}b^{2}\right)w_{y}ch\left(\frac{2\delta}{\epsilon}w\right).$

By compatibility and taking into account

$w_{xx}=\epsilon^{2}w_{yy}+\frac{2\epsilon\left(a^{2}-\delta^{2}b^{2}\right)}{\delta}\sinh\left(\frac{2\delta}{\epsilon}w\right),$

we get

$\displaystyle\frac{w_{xxyx}}{w_{x}w_{y}}$
$\displaystyle=\frac{8\delta\left(a^{2}-\delta^{2}b^{2}\right)}{\epsilon}\sinh\left(\frac{2\delta}{\epsilon}w\right)+\frac{2\epsilon\left(a^{2}-\delta^{2}b^{2}\right)}{\delta}F^{2}(w)\sinh\left(\frac{2\delta}{\epsilon}w\right)$
$\displaystyle+\left(F(w)F^{\prime}(w)+F^{3}(w)\right)w_{x}^{2}$
$\displaystyle+\left(\epsilon^{2}F^{\prime\prime}(w)+3\epsilon^{2}F^{\prime}(w)F(w)+\epsilon^{2}F^{3}(w)\right)w_{y}^{2}$
$\displaystyle+\left(3\epsilon^{2}F^{\prime}(w)+4\epsilon^{2}F^{2}(w)\right)w_{yy},$

and

$\displaystyle\frac{w_{xxxy}}{w_{x}w_{y}}$
$\displaystyle=4\left(a^{2}-\delta^{2}b^{2}\right)F(w)\cosh\left(\frac{2\delta}{\epsilon}w\right)$
$\displaystyle+\left(F^{\prime}(w)+F^{2}(w)\right)\frac{6\epsilon\left(a^{2}-\delta^{2}b^{2}\right)}{\delta}\sinh\left(\frac{2\delta}{\epsilon}w\right)$
$\displaystyle+\left(F^{\prime\prime}(w)+3F^{\prime}(w)F(w)+F^{3}(w)\right)w_{x}^{2}$
$\displaystyle+\left(\epsilon^{2}F(w)F^{\prime}(w)+\epsilon^{2}F^{3}(w)\right)w_{y}^{2}$
$\displaystyle+\left(3\epsilon^{2}F^{\prime}(w)+4\epsilon^{2}F^{2}(w)\right)w_{yy}.$

Thus,

$\displaystyle
4F(w)\cosh\left(\frac{2\delta}{\epsilon}w\right)+\frac{4\epsilon}{\delta}F^{2}(w)\sinh\left(\frac{2\delta}{\epsilon}w\right)+\frac{6\epsilon}{\delta}F^{\prime}(w)\sinh\left(\frac{2\delta}{\epsilon}w\right)$
$\displaystyle-\frac{8\delta}{\epsilon}\sinh\left(\frac{2\delta}{\epsilon}w\right)+\left(F^{\prime\prime}(w)+2F^{\prime}(w)F(w)\right)\left(w_{x}^{2}-\epsilon^{2}w_{y}^{2}\right)=0,$

from which we obtain the equations for $F(w)$

(18) $\displaystyle
2\epsilon^{2}F^{2}(w)+3\epsilon^{2}F^{\prime}(w)+2\epsilon\delta
F(w)\tanh\left(\frac{2\delta}{\epsilon}w\right)=8\delta^{2},$

and

(19) $\displaystyle F^{\prime\prime}(w)+2F^{\prime}(w)F(w)=0.$

We begin with the equation (19) and so we find $F^{\prime}(w)+F^{2}(w)=c$. We
have to consider cases for $c$.
1st Case (c = 0) $F^{\prime}(w)+F^{2}(w)=0$, then we have the subcases:

  1. (1)

$F(w)=0$ but this one do not satisfy (18),

  2. (2)

$F(w)=1/w$ but this one do not satisfy (18).

2nd Case ($c>0$) To make the calculations easier, we can consider $c=1$,
without loss of the generality. So we get $F^{\prime}(w)+F^{2}(w)=1$ and then
we consider the subcases:

  1. (1)

$F(w)=1$, but this one do not satisfy (18),

  2. (2)

$F(w)=-1$, but this one do not satisfy (18),

  3. (3)

$F(w)=\tanh w$, but this one satisfy (18) if $\epsilon=\delta$,

  4. (4)

$F(w)=\coth w$, but this one satisfy (18) if $\epsilon=\delta$.

3rd case ($c<0$) To make the calculations easier, we can consider $c=-1$,
without loss of the generality. So we get $F^{\prime}(w)+F^{2}(w)=-1$ and then
we consider the subcases

  1. (1)

$F(w)=-\tan w$, but this one satisfy (18) if $\epsilon\neq\delta$,

  2. (2)

$F(w)=\cot w$, but this one satisfy (18) if $\epsilon\neq\delta$.

∎

###### Proof of Theorem 3.

By Proposition 2 we get that
$F(w)=\frac{\delta}{\epsilon}\tanh(\frac{\delta}{\epsilon}w)$ or
$F(w)=\frac{\delta}{\epsilon}\coth(\frac{\delta}{\epsilon}w)$. Taking into
account that $F=-f^{\prime\prime}/f^{\prime}$ we get for the first case

$-\frac{f^{\prime\prime}}{f^{\prime}}=\kappa\tanh(\kappa w),$

and by integration we get

$f^{\prime}(w)=1/\cosh w,$

which corresponds to (3), while taking into account the second case

$-\frac{f^{\prime\prime}}{f^{\prime}}=\kappa\coth(\kappa w),$

by integrations we get (4). ∎

##  3\. Solutions of the Generalised Sinh-Gordon equation

The first family of solutions of (1) are of the form

(20)
$\tanh\left(\frac{\delta}{\epsilon}\frac{w(x,y)}{2}\right)=\frac{\delta}{\epsilon}F(x)G(y).$

###### Proof of Theorem 4.

Let $H(y)=1/G(y)$. By substituting in (1), (20) then we get

$\left(\epsilon^{2}H^{2}(y)-\delta^{2}F^{2}(x)\right)\left(\epsilon^{2}F^{\prime\prime}(x)H(y)+F(x)H^{\prime\prime}(y)\right)+2\epsilon^{2}\delta^{2}F(x)F^{\prime}(x)^{2}H(y)$
$-2\epsilon^{2}F(x)H(y)H^{\prime}(y)^{2}=4(a^{2}-\delta^{2}b^{2})\left(\delta^{2}\epsilon^{2}F^{3}(x)H(y)+F(x)H^{3}(y)\right)$

By dividing with $F(x)H(y)$ we have

(21)
$\left(\frac{F^{\prime\prime}(x)}{F(x)}+\epsilon^{2}\frac{H^{\prime\prime}(y)}{H(y)}\right)\left(\epsilon^{2}H^{2}(y)-\delta^{2}F^{2}(x)\right)+2\delta^{2}F^{\prime}(x)^{2}-2H^{\prime}(y)^{2}$
$=4\left(a^{2}-\delta^{2}b^{2}\right)(\delta^{2}F^{2}(x)+\epsilon^{2}H^{2}(y)).$

By differentiating with respect to x and y, we obtain

(22)
$\left(\frac{F^{\prime\prime}(x)}{F(x)}\right)_{x}\frac{1}{\delta^{2}F(x)F^{\prime}(x)}=\left(\frac{H^{\prime\prime}(y)}{H(y)}\right)_{y}\frac{1}{H(y)H^{\prime}(y)}=4A.$

So, we get

$\left(\frac{F^{\prime\prime}(x)}{F(x)}\right)_{x}=4\delta^{2}AF(x)F^{\prime}(x)$
$\Rightarrow(F^{\prime}(x))^{2}=A\delta^{2}(F(x))^{4}+B^{\prime}(F(x))^{2}+C^{\prime},$

and for $H(y)$ we get

$\left(\frac{H^{\prime\prime}(y)}{H(y)}\right)_{y}=4AH(y)H^{\prime}(y)$
$\Rightarrow(H^{\prime}(y))^{2}=A(H(y))^{4}+B(H(y))^{2}+C.$

Using (21) we get equations (6) and (7). ∎

######  Example 6.

Let $a=1$, $b=0$, $A=0$, $B=4-\epsilon^{2}\delta^{2}$ and $C=1$, then we get
the equations

$F^{\prime}(x)^{2}=\left(4-\epsilon^{2}\delta^{2}\right)F^{2}(x)+1,\quad
H^{\prime}(y)^{2}=\delta^{2}\left(1-H^{2}(y)\right).$

Then

$F(x)=\frac{\sinh\left(\sqrt{4-\epsilon^{2}\delta^{2}}x\right)}{\sqrt{4-\epsilon^{2}\delta^{2}}},\quad
H(y)=\cos\left(\delta y\right).$

So, for the equation (1), we have constructed the solution

$\tanh\left(\frac{\delta}{\epsilon}\frac{w(x,y)}{2}\right)=\frac{\delta}{\epsilon}\frac{\sinh\left(\sqrt{4-\epsilon^{2}\delta^{2}}x\right)\sec(\delta
y)}{\sqrt{4-\epsilon^{2}\delta^{2}}}.$

After lengthy computation, one can confirm that this is a solution of (1).

The second family of solutions of (1) are of the form

(23)
$\sinh\left(\frac{\delta}{\epsilon}w(x,y)\right)=\tan\left(\frac{\delta}{\epsilon}(A(x)+B(y))\right).$

###### Proof of Theorem 5.

By substituting (23) in (1), and dividing by
$\sec\left(\frac{\delta}{\epsilon}(A(x)+B(y))\right)$ then we get

$\tan\left(\frac{\delta}{\epsilon}(A(x)+B(y))\right)=\frac{\epsilon\delta\left(A^{\prime\prime}(x)-\epsilon^{2}B^{\prime\prime}(y)\right)}{4\epsilon^{2}\left(a^{2}-\delta^{2}b^{2}\right)-\delta^{2}A^{\prime}(x)^{2}+\delta^{2}\epsilon^{2}B^{\prime}(y)^{2}}.$

By differentiating with respect to x and using the trigonometric identity
$\sec^{2}u=1+\tan^{2}u$ and dividing with $A^{\prime}(x)$, we obtain

(24)
$\left(4\epsilon^{2}\left(a^{2}-\delta^{2}b^{2}\right)-\delta^{2}A^{\prime}(x)^{2}+\delta^{2}\epsilon^{2}B^{\prime}(y)^{2}\right)^{2}+\epsilon^{2}\delta^{2}B^{\prime\prime}(y)^{2}$
$=\epsilon^{2}\frac{A^{(3)}(x)}{A^{\prime}(x)}\left(4\epsilon^{2}\left(a^{2}-\delta^{2}b^{2}\right)-\delta^{2}A^{\prime}(x)^{2}+\delta^{2}\epsilon^{2}B^{\prime}(y)^{2}\right)+\epsilon^{2}\delta^{2}A^{\prime\prime}(x)^{2}.$

By differentiating with respect to y we get

$4\delta^{2}\epsilon^{2}A^{\prime}(x)B^{\prime}(y)B^{\prime\prime}(y)\left(4\epsilon^{2}\left(a^{2}-\delta^{2}b^{2}\right)-\delta^{2}A^{\prime}(x)^{2}+\delta^{2}\epsilon^{2}B^{\prime}(y)^{2}\right)$
$=2\delta^{2}A^{(3)}(x)B^{\prime}(y)B^{\prime\prime}(y)-2\delta^{2}\epsilon^{2}A^{\prime}(x)B^{\prime\prime}(y)B^{(3)}(y).$

Dividing by $2\delta^{2}A^{\prime}(x)B^{\prime}(y)B^{\prime\prime}(y)$ and
separate $x$ and $y$ we have

$\frac{A^{(3)}(x)}{A^{\prime}(x)}+2\epsilon^{2}\delta^{2}A^{\prime}(x)^{2}=\epsilon^{2}\frac{B^{(3)}(y)}{B^{\prime}(y)}+2\delta^{2}B^{\prime}(y)^{2}+8\left(a^{2}-\delta^{2}b^{2}\right)=c_{1}.$

So we get for $A(x)$

$\frac{A^{\prime\prime\prime}(x)}{A^{\prime}(x)}+2\epsilon^{2}\delta^{2}A^{\prime}(x)^{2}=c_{1}$
$\Rightarrow
A^{\prime\prime}(x)^{2}=-\delta^{2}\epsilon^{2}A^{\prime}(x)^{4}+c_{1}A^{\prime}(x)^{2}+c_{2}$

and we get for $B(y)$

$\frac{B^{\prime\prime\prime}(y)}{B^{\prime}(y)}+2\delta^{2}\epsilon^{2}B^{\prime}(y)^{2}+8\epsilon^{2}\left(a^{2}-\delta^{2}b^{2}\right)=\epsilon^{2}c_{1}$
$\Rightarrow
B^{\prime\prime}(y)^{2}=-\delta^{2}\epsilon^{2}B^{\prime}(y)^{4}+\epsilon^{2}\left(c_{1}-8\left(a^{2}-\delta^{2}b^{2}\right)\right)B^{\prime}(y)^{2}+c_{3}.$

Using (24) we get the relationship
$16\left(a^{2}-\delta^{2}b^{2}\right)^{2}-\epsilon^{2}\delta^{2}(c_{3}-c_{2})=4\left(a^{2}-\delta^{2}b^{2}\right)c_{1}$.
∎

######  Example 7.

Let $a=0$ and $b=1$ we choose $c_{1}=c_{3}=0$ and
$c_{2}=16\delta^{2}\epsilon^{2}$ we get

$A^{\prime\prime}(x)^{2}=-\delta^{2}\epsilon^{2}\left(16-A^{\prime}(x)^{4}\right),\quad
B^{\prime\prime}(y)^{2}=\delta^{2}\epsilon^{2}\left(8B^{\prime}(y)^{2}-B^{\prime}(y)^{4}\right).$

Then we get

$A^{\prime}(x)=2\Rightarrow A(x)=2x,$
$B^{\prime}(y)=2\sqrt{2}\sec\left(2\sqrt{2}\delta\epsilon y\right)\Rightarrow
B(y)=\frac{\operatorname{arctanh}\left(\sin(2\sqrt{2}\delta\epsilon
y)\right)}{\delta\epsilon}.$

So for equation (1) we have constructed the solution

$\sinh\left(\frac{\delta}{\epsilon}w(x,y)\right)=\tan\left(\frac{2\delta\epsilon
x+\operatorname{arctanh}\left(\sin(2\sqrt{2}\delta\epsilon
y)\right)}{\epsilon^{2}}\right).$

After lengthy computation, one can confirm that this is a solution of (1).

##  4\. Auto-Bäcklund transformation for sine-Gordon and sinh-Gordon equations

In [8], the authors prove that there is a Bäcklund transformation which
connects solutions of (1) with the following nonlinear PDE

(25)
$\delta\left(\theta_{xx}-\epsilon^{2}\theta_{yy}\right)=c^{2}e^{2\delta\theta}-\bar{c}^{2}e^{-2\delta\theta},$

where $c=a+\delta b$ and $\bar{c}=a-\delta b$. Using hyperbolic trigonometric
functions identities, (25) turns into

$\delta\left(\theta_{xx}-\epsilon^{2}\theta_{yy}\right)=2(a^{2}+\delta^{2}b^{2})\sinh(2\delta\theta)+4\delta
ab\cosh(2\delta\theta).$

In [8], the authors prove that we have the following Bäcklund transformation
connecting solutions of (1) to solutions of (25):

(26)
$\frac{\delta}{\epsilon}\left(w_{x}-\theta_{y}\right)=-\sinh\left(\frac{\delta}{\epsilon}w\right)\left(2a\cosh(\delta\theta)+2\delta
b\sinh(\delta\theta)\right),$ (27)
$\frac{\delta}{\epsilon}\left(w_{y}-\epsilon^{2}\theta_{x}\right)=-\cosh\left(\frac{\delta}{\epsilon}w\right)\left(2a\sinh(\delta\theta)+2\delta
b\cosh(\delta\theta)\right),$

where $w(x,y)$ is a solution of (1) and $\theta(x,y)$ is a solution of (25).

Let us consider the case $\epsilon=1$ , $a=-1$ and $b=0$. Thus we have an
Auto-Bäcklund transformation for the equation

(28) $\delta\left(u_{xx}-u_{yy}\right)=2\sinh(2\delta u),$

that is given by

(29) $\delta f_{x}-\delta g_{y}=2\sinh(\delta f)\cosh(\delta g)$ (30) $\delta
f_{y}-\delta g_{x}=2\cosh(\delta f)\sinh(\delta g).$

Let us consider the family of solutions

$\tanh\left(\frac{\delta w(x,y)}{2}\right)=\delta F(x)G(y).$

To solve the Auto-Bäcklund transformation, we consider the initial conditions
$G(0)=1$ and $G^{\prime}(0)=0$.

######  Proposition 8.

Consider the equation $\delta(u_{xx}-u_{yy})=2\sinh(2\delta u)$ and its family
of solutions $\tanh(\frac{\delta g(x,y)}{2})=\delta F(x)G(y).$ Using the the
Auto-Bäcklund transformation and the initial conditions $G(0)=1$ and
$G^{\prime}(0)=0$, we get the solution

(31) $\tanh\left(\frac{\delta
f(x,y)}{2}\right)=\frac{(2+\lambda)\tanh(\sqrt{4-\lambda^{2}}\cdot
Y)+c_{0}e^{2X}\sqrt{4-\lambda^{2}}}{\sqrt{4-\lambda^{2}}+(\lambda-2)c_{0}e^{2X}\tanh(\sqrt{4-\lambda^{2}}\cdot
Y)}$

where $X=X(x)=\int_{0}^{x}\frac{1+F^{2}(t)}{1-F^{2}(t)}dt$,
$Y=Y(x,y)=\int_{0}^{x}\frac{2F(x)G(s)}{1-\delta^{2}F^{2}(x)G^{2}(s)}ds$,
$c_{0}=\tanh(\frac{\delta f(0,0)}{2})$ and
$\lambda=\lambda(x)=\frac{F^{\prime}(x)}{F(x)}$.

###### Proof.

We have the Auto-Bäcklund transformation,

(32) $\delta f_{x}-\frac{2\delta
F(x)G^{\prime}(y)}{1-\delta^{2}F(x)^{2}G(y)^{2}}=2\sinh(\delta
f)\frac{1+\delta^{2}F(x)^{2}G(y)^{2}}{1-\delta^{2}F(x)^{2}G(y)^{2}},$ (33)
$\delta f_{y}-\frac{2\delta
F^{\prime}(x)G(y)}{1-\delta^{2}F^{2}(x)G^{2}(y)}=2\cosh(\delta f)\frac{2\delta
F(x)G(y)}{1-\delta^{2}F^{2}(x)G^{2}(y)}.$

Let $y=0$ and consider that $G^{\prime}(0)=0$ and $G(0)=1$ in (32) we get

$\frac{\delta f_{x}}{\sinh(\delta f)}=2\frac{1+F(x)^{2}}{1-F(x)^{2}}.$

By integrating we get

$\tanh(\frac{\delta f(x,0)}{2})=c_{0}e^{2X},$

where $c_{0}=\tanh(\frac{\delta f(0,0)}{2})$ and
$X=X(x)=\int_{0}^{x}\frac{1+F^{2}(t)}{1-F^{2}(t)}dt$, which is an elliptic
integral.

We continue with the (33). We have

$\delta f_{y}=\frac{2\delta
F(x)G(y)}{1-\delta^{2}F^{2}(x)G^{2}(y)}\left(\frac{F^{\prime}(x)}{F(x)}+2\cosh(\delta
f)\right),$ $\frac{\delta f_{y}}{\lambda(x)+2\cosh(\delta f)}=\frac{2\delta
F(x)G(y)}{1-\delta^{2}F^{2}(x)G^{2}(y)}$

where $\lambda=\lambda(x)=\frac{F^{\prime}(x)}{F(x)}$ and by integrating we
have

$\tanh\frac{\delta
f(x,y)}{2}=\frac{\sqrt{\lambda^{2}-4}}{\lambda-2}\tanh\left(\frac{\delta}{2}\sqrt{\lambda^{2}-4}Y(x,y)+\operatorname{arctanh}\left(\frac{\lambda-2}{\sqrt{\lambda^{2}-4}}c_{0}e^{2X}\right)\right)$

where $Y=Y(x,y)=\int_{0}^{x}\frac{2F(x)G(s)}{1-\delta^{2}F^{2}(x)G^{2}(s)}ds$.
By applying trigonometric identities we get the requested. ∎

######  Example 9.

We will use the solution described in Example 5. To this end, let
$k=\sqrt{4-\epsilon^{2}\delta^{2}}$ then we get

$\lambda(x)=\frac{k}{\tanh(kx)},$
$X(x)=\frac{2\operatorname{arctanh}\left(\frac{\sqrt{1+k^{2}}\tanh(kx)}{k}\right)}{\sqrt{1+k^{2}}}-x,$
$Y(x,y)=\frac{2\sinh(kx)}{\delta\sqrt{k^{2}-\delta^{2}\sinh^{2}(kx)}}\operatorname{arctanh}\left(\frac{k\sin(\delta
y)}{\sqrt{k^{2}-\delta^{2}\sinh^{2}(kx)}}\right).$

After lengthy computation, one can confirm that this is a solution of (1).

Finally, we consider the family of solutions

$\sinh(\delta w(x,y))=\tan\left(\delta(A(x)+B(y))\right).$

To solve the Auto-Bäcklund transformation, we consider the initial conditions
$A(0)=A_{0}$ and $A^{\prime}(0)=0$.

######  Proposition 10.

Consider the equation $\delta\left(u_{xx}-u_{yy}\right)=2\sinh(2\delta u)$ and
its family of solutions $\sinh(\delta
g(x,y))=\tan\left(\delta(A(x)+B(y))\right).$ Using the the Auto-Bäcklund
transformation and the initial conditions $A(0)=A_{0}$ and $A^{\prime}(0)=0$,
we get the solution

(34) $\tanh(\frac{\delta
f(x,y)}{2})=\frac{2+\mu\tanh\left(\frac{\mu}{2}X+\tanh^{-1}\left(\frac{\delta
B^{\prime}(y)\tan Y-2}{\mu}\right)\right)}{B^{\prime}(y)}$

where $\mu=\mu(y)=\sqrt{4\delta^{2}+B^{\prime}(y)^{2}}$
$X=X(x,y)=\int_{0}^{x}\sec\left(\delta(A(x)+B(y))\right)dt$,
$Y=Y(y)=\delta\int_{0}^{y}\tan\left(\delta(A_{0}+B(y))\right)ds$ with initial
condition $f(0,0)=0$.

###### Proof.

We have the Auto-Bäcklund transformation,

(35) $\delta f_{x}=\left(2\sinh(\delta f)+\delta
B^{\prime}(y)\right)\sec\left(\delta(A(x)+B(y))\right),$ (36) $\delta
f_{y}=\frac{\delta A^{\prime}(x)+2\cosh(\delta
f)\sin\left(\delta(A(x)+B(y))\right))}{\cos\left(\delta(A(x)+B(y))\right)}$

We let $x=0$ in (36) and using the initial condition $A^{\prime}(0)=0$ we get

$\frac{\delta^{2}f_{y}}{\cosh(\delta
f)}=2\delta\tan\left(\delta(A_{0}+B(y))\right).$

By integrating and applying the initial condition $f(0,0)=0$ we have

$\tanh\left(\frac{\delta f(0,y)}{2}\right)=\frac{\tan(Y)}{\delta}$

where $Y=Y(y)=\delta\int_{0}^{y}\tan\left(\delta(A_{0}+B(y))\right)ds$. The
equation (36) turns into

$\delta f_{x}=\left(\delta B^{\prime}(y)+2\sinh(\delta
f)\right)\sec\left(\delta(A(x)+B(y))\right),$ $\frac{\delta f_{x}}{\delta
B^{\prime}(y)+2\sinh(\delta f)}=\sec\left(\delta(A(x)+B(y))\right)$

and by integrating we get

$\tanh(\frac{\delta
f(x,y)}{2})=\frac{2+\mu\tanh\left(\frac{\mu}{2}X+\tanh^{-1}\left(\frac{\delta
B^{\prime}(y)\tan Y-2}{\mu}\right)\right)}{\delta B^{\prime}(y)}$

and applying trigonometric identities we get the requested. ∎

######  Example 11.

Let $\delta=i$ and set $c_{1}=c_{2}=c_{3}=4$ then

$A^{\prime\prime}(x)^{2}=A^{\prime}(x)^{4}+4A^{\prime}(x)^{2}+4,\quad
B^{\prime\prime}(y)^{2}=B^{\prime}(y)^{4}-4B^{\prime}(y)^{2}+4.$

Then

$A(x)=\log\left(\cos(\sqrt{2}x)\right),\quad
B(y)=\log\left(\cosh(\sqrt{2}y)\right),$

and the solution turns into

$\sinh\left(ig(x,y)\right)\tan\left(iA(x)+iB(y)\right)=i\frac{\cos^{2}(\sqrt{2}x)\cosh^{2}(\sqrt{2}y)-1}{\cos^{2}(\sqrt{2}x)\cosh^{2}(\sqrt{2}y)+1}$
$\Rightarrow\sin\left(g(x,y)\right)=\frac{\cos^{2}(\sqrt{2}x)\cosh^{2}(\sqrt{2}y)-1}{\cos^{2}(\sqrt{2}x)\cosh^{2}(\sqrt{2}y)+1}.$
$\mu=\sqrt{4\delta^{2}+B^{\prime}(y)^{2}}=i\sqrt{2}\sqrt{1+sech^{2}(\sqrt{2}y)},\quad
m=1+sech^{2}(\sqrt{2}y),$
$Y(y)=-\int_{0}^{y}\tanh\left(\log(\cosh(\sqrt{2}s))\right)ds=\operatorname{arctanh}\left(\frac{\tanh(\sqrt{2}y)}{\sqrt{2}}\right)-y,$
$X(x,y)=\int_{0}^{x}\sec
h\left(\log(\cos(\sqrt{2}t)\cosh(\sqrt{2}y))\right)dt$
$=\frac{2\operatorname{arctanh}\left(\frac{\sqrt{2}\sin(\sqrt{2}x)\cosh(\sqrt{2}y)}{\sqrt{3+\cosh(2\sqrt{2}y)}}\right)}{\sqrt{3+\cosh(2\sqrt{2}y)}},$
$\tan\left(\frac{f(x,y)}{2}\right)=\frac{\sqrt{2m}\tan\left(\frac{\sqrt{2m}}{2}X+\arctan\left(\frac{2+\sqrt{2}\tanh(\sqrt{2}y)\tan(Y)}{\sqrt{2m}}\right)\right)}{\sqrt{2}\tanh(\sqrt{2}y)}.$

After lengthy computation, one can confirm that this is a solution of (1).

Acknowledgements. The author would like to thank Prof. C. Daskaloyannis, Prof.
A. Fotiadis and Dr. E. Papageorgiou for their help and support.

## References

  * [1] U. Abresch: _Constant mean curvature tori in terms of elliptic functions_. J. Reine Angew. Math. 374 (1987), 169–192 .
  * [2] D. Arrigo: _Analytical Methods for Solving Nonlinear Partial Differential Equations_ , Second Edition, Springer (2022).
  * [3] E. Bour: _Theorie de la deformation des surfaces_ Journal de l’Ecole imperiale polytechnique. 22 (39): 1 - 148, (1862).
  * [4] H. Eleuch, Y.V. Rostovtsev: _Analytical solution to sine-Gordon equation_ , Journal of Mathematical Physics, 51, 093515 (2010).
  * [5] A.S Fokas and B. Pelloni: _The Dirichlet-to-Neumann map for the elliptic sine-Gordon equation_ , Nonlinearity 25 (2012) no. 4, 1011 - 1031.
  * [6] A.P. Fordy, J.C. Wood, _Harmonic Maps and Integrable Systems_ , Aspects of Mathematics, Vieweg (1994).
  * [7] A. Fotiadis, C. Daskaloyannis: _Beltrami equation for the harmonic diffeomorphisms between surfaces._ Nonlinear Analysis 214, (2022), 112546 .
  * [8] A. Fotiadis, C. Daskaloyannis: _Harmonic diffeomorphisms between Pseudo-Riemannian Surfaces._ , 2023, arXiv:2308.06625v1.
  * [9] A.M. Grundland, E. Infeld: _A family of nonlinear Klein-Gordon equations_ , J. Math. Phys. 33(7) 2498-2503, (1992).
  * [10] R. Hirota: _The direct method in soliton theory_ , Cambridge tracts in Mathematics, 155, (1992).
  * [11] Ze-Jun Hu, Zhen-Zu Sun: _A new interpretation of the Bäcklund transformation of the sine-Gordon equation_ Kodai Math. J. 23 (2000), 100-104.
  * [12] G. Hwang _The elliptic sinh-Gordon equation in the half plane_ J. Nonlinear Sci. Appl. 8 (2015), no. 2, 163-173.
  * [13] P. Joaquín: _Sinh-Gordon type equations for CMC surfaces_. 145–156, Ed. Univ. Granada, Granada, 2011 .
  * [14] K. Kenmotsu: _Surfaces with constant mean curvature_. Translations of Mathematical Monographs, 221. American Mathematical Society, 2003 .
  * [15] H.P. McKean: _The sine-Gordon and sinh-Gordon equations on the circle_ Communications on Pure and Applied Mathematics 34 (2) p. 197-257, (1981).
  * [16] Y. N. Minsky: _Harmonic maps, length and energy in Teichmüller space_. J. Diff. Geom. 35 (1992), no. 1, 151-–217 .
  * [17] Polychrou, G.: _Solutions of the sinh-Gordon and sine-Gordon equations and applications._ J Elliptic Parabol Equ 9, 611–625 (2023).
  * [18] G. Polychrou, E. Papageorgiou, A. Fotiadis, C. Daskaloyannis: _New examples of harmonic maps to the hyperbolic plane_ , Rev Mat Complut, (2023).
  * [19] C.Z. Qu, W. He, J. Dou: _Separation of Variables and Exact Solutions of Generalized Nonlinear Klein-Gordon Equations_ , Prog. Theor. Phys, 105, 379-398, (2001).
  * [20] C. Rogers, W.K. Schief: _Bäcklund and Darboux Transformations: Geometry and Modern Applications in Soliton Theory_ , Cambridge University Press, ISBN 978-0-521-01288-1, 2002.
  * [21] J. Rubinstein: _Sine-Gordon equation_ Journal of Mathematical Physics. 11 (1970) (1), 258 - 266, 1970.
  * [22] I. Sterling and H. Wente, Existence and classification of constant mean curvature multibubbletons of finite and infinite type, Indiana Univ. Math. J. 42 (1993), no. 4, 1239—1266.
  * [23] A.M. Wazwaz: _The tanh method: exact solutions of the sine-Gordon and the sinh-Gordon equations_ Applied Mathematics and Computation 167 (2005), 1196-1210.
  * [24] H. C. Wente: _Counterexample to a conjecture of H. Hopf._ Pacific J. Math. 121 (1986), no. 1, 193–243 .
  * [25] M. Wolf: _The Teichmüller theory of harmonic maps_. J. Diff. Geom. 29 (1989), no. 2, 449-–479 .


# 3M: Multi-loss, Multi-path and Multi-level Neural Networks for speech
recognition

###### Abstract

Recently, Conformer based CTC/AED model has become a mainstream architecture
for ASR. In this paper, based on our prior work, we identify and integrate
several approaches to achieve further improvements for ASR tasks, which we
denote as multi-loss, multi-path and multi-level, summarized as ”3M” model.
Specifically, multi-loss refers to the joint CTC/AED loss and multi-path
denotes the Mixture-of-Experts(MoE) architecture which can effectively
increase the model capacity without remarkably increasing computation cost.
Multi-level means that we introduce auxiliary loss at multiple level of a deep
model to help training. We evaluate our proposed method on the public
WenetSpeech dataset and experimental results show that the proposed method
provides $12.2\%\sim 17.6\%$ relative CER improvement over the baseline model
trained by Wenet toolkit. On our large scale dataset of 150k hours corpus, the
3M model has also shown obvious superiority over the baseline Conformer model.
Code is publicly available at https://github.com/tencent-ailab/3m-asr.

Index Terms: Conformer, mixture of experts, Multi-loss, Multi-level, Multi-
path

11footnotetext: Equal contribution.

##  1 Introduction

End-to-End(E2E) automatic speech recognition(ASR) have gained large
improvements in recent years. As a combination of convolution module and self-
attention mechanism, Conformer[1] can model local and global dependencies of
an audio sequence and achieve a better performance, making it become a favored
choice to train benchmarks for many E2E ASR toolkits(e.g. Espnet[2],
Wenet[3]). CTC/attention-based encoder-decoder, also known as CTC/AED, is a
popular E2E ASR framework, which effectively utilizes advantages of both
architectures to improve robustness and achieve faster convergence. Previous
works[4, 5, 6, 7] have proved this multi-loss training can significantly
improve the performance of E2E ASR systems.

Scaling up to a larger model has been an effective way towards a flexible and
powerful E2E ASR system. It has shown that a large Conformer model(e.g.
SpeechStew[8]) can achieve state-of-the-art(SoTA) results across a wide
variety of tasks. However, developing large models in real-world application
is seriously hindered by the expensive computation cost of both training and
inference time. Several techniques have been proposed to reduce the model
complexity, such as knowledge distillation, low-rank decomposition,
quantization and pruning, but these methods inevitably suffer from performance
degradation.

Recently, mixture of experts (MoE) based approaches [9, 10] have been
intensively investigated and applied in different tasks such as language
modeling [11, 12], image classification [13, 14, 15, 16], and speech
recognition [17, 18]. MoE models can easily increase the model capacity by
increasing the number of experts. With the introduction of the sparsely-gated
mixture-of-experts layer[19], MoE models can dynamically route inputs to
corresponding expert networks, which enables us to satisfy training and
inference efficiency by having sub-network activated on per-example basis.

Based on the joint CTC/AED multi-loss training framework, we explore the MoE
approach on the Conformer model, named Conformer-MoE, where the second
Macaron-style half-step feed-forward layer of each Conformer block is modified
into the MoE layer. With sparsely-gated mechanism, the MoE layer will route
their inputs to the top-1 expert with largest route probability, thus keeping
the computational cost roughly constant. The Conformer-MoE activate different
sub-networks when dealing with different input samples and we call this
property as multi-path. Additionally, we further propose to integrate separate
attention decoders at multiple intermediate level of encoder layers, which
effectively accelerate convergence and get a better performance. The multi-
level property, along with multi-path and multi-loss, makes up our final 3M
model

The rest of the paper is organized as follows. Section 2 reviews the previous
work of Conformer and SpeechMoE, and Section 3 presents our proposed method
3M. The experimental results are reported in Section 4. Finally, we conclude
this paper in Section 5.

Figure 1:  illustration for the architecture of 3M model, which is composed of
a shared encoder with several 3M blocks, a CTC decoder, an Attention Decoder
and an embedding network.

##  2 Previous works

This section mainly describes previous work of Conformer and SpeechMoE.

###  2.1 Conformer

The Conformer is proposed by [1], and each Conformer block comprises two Feed
Forward modules sandwiching the Multi-Head self-attention module and the
Convolution module. Given the input $x$, we can get the output $y$ of
Conformer block:

$\hat{x}=x+\frac{1}{2}FFN(x)$ (1) $\widetilde{x}=\hat{x}+MHSAN(\hat{x})$ (2)
$\overline{x}=\widetilde{x}+Conv(\widetilde{x})$ (3)
$y=LN(\overline{x}+\frac{1}{2}FFN(\overline{x}))$ (4)

where FFN represents the feed-forward module, MHSAN represents the multi-head
self-attention module, Conv represents the convolution module, and LN
represents the layer norm.

###  2.2 SpeechMoE

The SpeechMoE is proposed in [17], which comprises multiple MoE layers, non-
expert layers and a shared embedding network. Each MoE layer consists of $n$
experts and a router. It takes the output of the previous layer and the shared
embedding as input and routes each speech frame to the top-1 expert with the
largest route probability. Let $W^{l}_{r}$, $e^{c}$ and $o^{l-1}$ be the
router weights of the $l$-th layer, shared embedding and the output of the
previous layer, then the router probability can be defined as follows:

$r^{l}=W_{r}^{l}\cdot Concat(e^{c};o^{l-1})$ (5)

${p^{l}_{i}}=\frac{exp^{r^{l}_{i}}}{\sum_{j=1}^{n}exp^{r^{l}_{j}}}$ (6)

Then, the selected expert’s output is also gated by router probability to get
the output of the MoE layer,

$y^{l}=p^{l}_{i}E^{l}_{i}$ (7)

Since only one expert is active in each layer, the SpeechMoE can keep the
computational cost constant while scaling up to a very large model. To achieve
better sparsity and balance among different experts, the sparsity L1 loss
$L_{s}$ and mean importance loss $L_{m}$ are added into the loss function:

$L_{s}=\frac{1}{k}\sum_{j=1}^{k}\parallel{\hat{p}_{j}}\parallel_{1}$ (8)

${L}_{m}=n\sum_{i=1}^{n}({\frac{1}{k}\sum_{j=1}^{k}p_{ij}})^{2}$ (9)

where $\hat{p}_{j}$ stands for the unit-normalized router probability
distribution of frame $j$, and $k$ is the number of frames in this mini-batch.
$p_{ij}$ stands for the router probability on expert $i$ of frame $j$.

Table 1: Results of Conformer, Conformer-MLevel, Conformer-MoE and 3M.

Testset | Confomer | Conformer-MLevel | Conformer-MoE (16e) | 3M (16e) | 3M (32e) | 3M (64e)
---|---|---|---|---|---|---
cctv | 1.60 | 1.52 | 1.48 | 1.47 | 1.43 | 1.42
deyunshe | 16.48 | 16.01 | 15.56 | 15.20 | 14.78 | 14.31
liyongle | 4.63 | 3.96 | 3.71 | 3.53 | 3.50 | 3.37
luoxiang | 3.93 | 3.51 | 3.30 | 3.11 | 3.10 | 3.08
luozhengyu | 2.78 | 2.55 | 2.4 | 2.30 | 2.27 | 2.19
qichezhijia | 6.89 | 6.58 | 6.42 | 6.34 | 6.30 | 6.25
shenghuo | 9.84 | 9.49 | 9.16 | 9.03 | 8.41 | 8.13
tianxiazuqiu | 3.82 | 3.77 | 3.59 | 3.52 | 3.46 | 3.37
wangzhe | 11.75 | 11.29 | 10.87 | 10.63 | 10.43 | 10.36
xiaoaidashu | 5.94 | 5.83 | 5.81 | 5.74 | 5.70 | 5.65

##  3 3M model

###  3.1 Multi-loss

To date, CTC/AED is commonly used in E2E ASR, which effectively utilizes the
advantages of both architectures in training and decoding. On one hand, the
attention lacks left-to-right constraints, making it difficult to generate
proper alignments in the case of noisy data and/or long input sequences.
Augmenting the constrained CTC loss based on the forward-backward algorithm
can greatly reduce the number of irreuglarly aligned utterances. On the other
hand, CTC imposes the conditional independence constraint that output
predictions are independent, which is not true for ASR. Attention objective
can help eliminate the conditional independence constraint and back propagate
the context information to rectify the probability distribution output by CTC.
When employing decoding, both attention-based scores and CTC scores can be
combined in a rescoring/one-pass beam search to improve the performance. We
regard the CTC/AED joint training as multi-loss property. As shown in Figure
1, the proposed model has a CTC branch and an attention branch with a shared
encoder.

###  3.2 Multi-path

Here, we propose to apply MoE architecture on Conformer model. Specifically,
as shown in Figure 1, we extend the second Macaron-style feed-forward module
to the MoE layer, which consists of $N$ FFN module and a router. Similar to
SpeechMoE[17], each FFN module stands for one expert network and the router
takes the shared embedding and the output from the previous module as input
and routes each speech frame to the top-1 expert with the largest route
probability. Expert outputs are also gated by the corresponding router
probability. With this sparse-gated mechanism, the proposed model can activate
different sub-networks for different samples, which is regarded as multi-path
property. We have also attempted to modify both FFN modules in the Conformer
block into MoE layers, but it fails to achieve a better performance.

###  3.3 Multi-level

As we know, the lower layers of the Conformer encoder also have a low
correlation with grapheme information. Generally, the input of the attention
decoder is the top layer of the Conformer encoder. In order to make the lower
layers of encoder have a earlier access to grapheme information from the
decoder, we propose to integrate separate attention decoders at intermediate
layers of the encoder, as shown in green box part of Figure 1. In our
practices, two additional decoders are applied to intermediate layers at 1/3
depth and 2/3 depth of the encoder, which are only used for training with AED
loss and ignored during inference.

###  3.4  Training objective

####  3.4.1  MoE training loss

We have introduced the mentioned $L_{s}$ and $L_{m}$ loss to make the router
probabilities sparse and diverse. To provide enough distinction information
for routers, the shared embedding network which converts low-level features to
high-level embedding is necessary to help the router attain better selecting
effect. The embedding training loss provides reliable embeddings for the
routers. The training loss of MoE is defined as:

$L_{MoE}=\alpha L_{s}(x)+\beta{L}_{m}(x)+\gamma L_{e}(x;y)$ (10)

Where $L_{s}$ and ${L}_{m}$ are the mentioned sparsity $L1$ loss and mean
importance loss, used to encourage sparsity and diversity of the model. The
embedding loss $L_{e}$ is the CTC loss. $\alpha$, $\beta$, and $\gamma$ is the
scale for $L_{s}$, ${L}_{m}$ and $L_{e}$ respectively.

####  3.4.2  Joint CTC/AED training loss

In this paper, the CTC loss and multi-level AED loss are combined in the
training of the 3M model:

$\begin{split}L_{Joint}=\eta
L_{c}(x;y)+(1-\eta)\sum_{j=1}^{K}L_{a_{j}}(x;y)\end{split}$ (11)

Among these items, $L_{c}$ is the CTC loss. $K$ is the number of level.
$L_{a_{j}}$ is the AED loss for the transformer decoder of $j$-th level.
$\eta$ are the interpolated weight for CTC loss and AED loss.

####  3.4.3 Loss function

Given the input $x$, grapheme target $y$, the complete loss function of our
method is defined as:

$L(x;y)=L_{MoE}+L_{Joint}$ (12)

##  4 EXPERIMENTS

###  4.1 WenetSpeech Task

We first evaluate our proposed method on the WenetSpeech[20] Dataset, which a
is multi-domain Mandarin corpus consisting of 10005 hours of high-quality
labeled speech. Evaluation is performed in terms of word error rate (WER) on
the Dev, Test Net and Test Meeting, described in [20]. The input speech uses
80-dimension log-Mel filterbank features, which are computed with a 25ms
window and shifted every 10ms. Spec-Augment is applied 2 frequency masks with
maximum frequency mask (F = 30) and 2 time masks with maximum time mask (T =
50) to alleviate over-fitting. A global mean and variance normalization is
used as data preparation. The max number of epochs is 26. We set the
checkpoint with the lowest CTC loss as the final model.

We use the benchmarks from Kaldi, Espnet and Wenet listed in [20] as our
baselines. Our Conformer model uses the same setup as Espnet and Wenet
benchmarks, which consists of 12-block Conformer
encoder($d^{ff}=2048,H=8,d^{att}=512,CNN_{kernel}=15$) and 6-block transformer
decoder($d^{ff}=2048,H=8$). As for the MoE layer, we set the number of experts
to be 16, 32 and 64, noted as 16e, 32e and 64e respectively. The shared
embedding network is a static model without MoE layer but a similar structure
to the baseline, which contains 6 Conformer encoder layers and is pretrained
with CTC objective before used for MoE training. The whole model is trained
with CTC and attention objectives, along with our mentioned auxiliary losses.
For all experiments on MoE models, we set the hyper-parameters
$\alpha=0.15,\beta=0.15,\gamma=0.01,\eta=0.3$. A set of 5535 Mandarin
characters and 26 English letters is used as the modeling units. For decoding,
we generate the N-Best hypotheses by the CTC decoder and rescore them by the
attention decoder to get the final results.

As shown in Table 2, our Conformer-MoE models, equipped with the multi-path
property, achieve consistent performance improvements on three test sets over
Kaldi, Espnet and Wenet benchmarks. Moreover, by increasing the number of
experts, Conformer-MoE gets a better performance on Dev but 64e doesn’t get
consistent improvements on Test_Net and Test_Meeting, which have more
difficult domain and unmatched data. In total, Conformer-MoE(32e) achieves the
best performance on these test sets. Compared with Wenet benchmark, it
provides $12.2\%\sim 17.6\%$ relative character error rate(CER) reduction. Our
3M model, Comformer-MoE with additional multi-level property, doesn’t get a
better performance over Conformer-MoE so we did not post its results.

###  4.2 150,000 hours Task

In this experiment, we scale up the size of training corpus to 150000 hours to
further identify the effectiveness of our proposed method. The training corpus
is collected from different application domains. In order to improve the
system robustness, we simulate six types of different mixing conditions like
voice processing(vp), acoustic echo cancellation(aec), reverberation(rir) and
signal-to-noise-ratio(SNR), etc. The SNR is set between 15 and 30 dB, and the
reverberation time is set between 0 and 900 milliseconds. The test sets are
collected from YouTube, consisting of 15 hours of audio. They add up to 65536
utterances and are divided into 10 domains, covering game commentary,
documentary, comic dialogue and so on.

Our baseline Conformer model is implemented by Wenet toolkits and it consists
of two convolution downsampling layers, a 18-block Conformer encoder and a
4-block bi-transformer decoder (2-block for forward and 2-block for backward).
For MoE models, the number of experts are also set to be 16, 32 and 64. The
shared embedding network is similar as before and has 7 Conformer blocks. For
the multi-level setting, two separate bi-transformer decoders are applied to
the intermediate layers at 1/3 depth and 2/3 depth of the encoder. Since
decoders at intermediate level are only used for training, there is no
increase computational cost in decoding for 3M models. All model are trained
with CTC and attention objectives and hyper-parameters settings are the same
as before. We use the floating point operations(FLOPs) for a one-second
example to evaluate the inference computation cost and Table 3 shows the
comparison on parameters amount and FLOPs for Conformer, Conformer-MoE and 3M.

Experimental results are shown in Table 1. It is clear to see from column 2
and column 5 that 3M (16e) achieves lower character error rate than the
baseline Conformer model and the gain is 11.5% on average. With the increase
of data amount, MoE models with more experts are easier to get a better
performance. We can see that 3M model achieves consistent improvements from
16e to 64e on all test sets in 150000 hours task, which is different from the
WenetSpeech results. In total, 3M(64e) provides 14.9% relative CER improvement
on average over the baseline Conformer model.

To analyze the efficacy of 3M model, we must study how much the multi-path and
multi-level contribute to the performance individually. We train Conformer-MoE
(multi-path) and Conformer-MLevel (multi-level) model by only keeping the
multi-path and the multi-level in 3M model, respectively. In Table 1, we see
that a significant gain benefits from both the multi-level and multi-path.
Especially, compared with the baseline model, the gain is 9.2% (from column 2
and column 4) for the multi-path and 5.6% (from column 2 and column 3) for the
multi-level. It suggests that both the multi-level and multi-path are critical
to the quality of the proposed 3M model.

Table 2: CER results on WenetSpeech for public benchmarks and Conformer-MoE

Toolkit | Dev |  Test Net |  Test Meeting
---|---|---|---
Kaldi | 9.07 | 12.83 | 24.72
Espnet | 9.70 | 8.90 | 15.90
Wenet | 8.88 | 9.70 | 15.59
Conformer-MoE (16e) | 7.67 | 8.28 | 13.96
Conformer-MoE (32e) | 7.49 | 7.99 | 13.69
Conformer-MoE (64e) | 7.19 | 8.36 | 13.72

Table 3: A comparsion of Conformer, Conformer-MoE and 3M.  Model | Params | FLOPs
---|---|---
Conformer | 120M | 8.3B
Conformer-MoE (16e) | 425M | 12.3B
3M (16e) | 500M | 12.3B
3M (32e) | 775M | 12.3B
3M (64e) | 1.37B | 12.3B

##  5 Conclusions and future work

In this paper, we introduce the 3M model for speech recognition. Specifically,
based on the CTC/AED E2E ASR framework, we apply MoE on the Conformer model
and apply AED at multiple levels of the model. We demonstrated the
effectiveness of our proposed method on both the public WenetSpeech dataset
and a large scale dataset. Experimental results show that the Conformer-MoE
provides up to 17.6% relative CER improvement compared with the Wenet
benchmark on the WenetSpeech dataset. The mutli-level property can also
contribute to a better performance. Overall, our 3M model achieves 14.9%
relative CER improvement on average over the baseline Conformer model on the
large scale dataset. Future work includes increasing the number of experts by
one or two orders of magnitudes, and exploring the proposed 3M model with
other end-to-end training framework such as transformer transducers.

## References

  * [1] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” 2020.
  * [2] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. E. Y. Soplin, J. Heymann, M. Wiesner, N. Chen _et al._ , “Espnet: End-to-end speech processing toolkit,” _arXiv preprint arXiv:1804.00015_ , 2018.
  * [3] B. Zhang, D. Wu, C. Yang, X. Chen, Z. Peng, X. Wang, Z. Yao, X. Wang, F. Yu, L. Xie _et al._ , “Wenet: Production first and production ready end-to-end speech recognition toolkit,” _arXiv e-prints_ , pp. arXiv–2102, 2021.
  * [4] T. Hori, S. Watanabe, Y. Zhang, and W. Chan, “Advances in joint ctc-attention based end-to-end speech recognition with a deep CNN encoder and RNN-LM,” _CoRR_ , vol. abs/1706.02737, 2017. [Online]. Available: http://arxiv.org/abs/1706.02737
  * [5] L. Wu, T. Li, L. Wang, and Y. Yan, “Improving hybrid ctc/attention architecture with time-restricted self-attention ctc for end-to-end speech recognition,” _Applied Sciences_ , vol. 9, no. 21, p. 4639, 2019.
  * [6] S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi, “Hybrid ctc/attention architecture for end-to-end speech recognition,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 11, no. 8, pp. 1240–1253, 2017.
  * [7] T. Hori, S. Watanabe, and J. R. Hershey, “Joint ctc/attention decoding for end-to-end speech recognition,” in _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2017, pp. 518–529.
  * [8] W. Chan, D. Park, C. Lee, Y. Zhang, Q. Le, and M. Norouzi, “Speechstew: Simply mix all available speech recognition data to train one large neural network,” _arXiv preprint arXiv:2104.02133_ , 2021.
  * [9] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive mixtures of local experts,” _Neural computation_ , vol. 3, no. 1, pp. 79–87, 1991\.
  * [10] M. I. Jordan and R. A. Jacobs, “Hierarchical mixtures of experts and the em algorithm,” _Neural computation_ , vol. 6, no. 2, pp. 181–214, 1994.
  * [11] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” _arXiv preprint arXiv:2006.16668_ , 2020.
  * [12] W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” _arXiv preprint arXiv:2101.03961_ , 2021.
  * [13] S. Gross, M. Ranzato, and A. Szlam, “Hard mixtures of experts for large scale weakly supervised vision,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6865–6873.
  * [14] K. Ahmed, M. H. Baig, and L. Torresani, “Network of experts for large-scale image categorization,” in _European Conference on Computer Vision_. Springer, 2016, pp. 516–532.
  * [15] X. Wang, F. Yu, L. Dunlap, Y.-A. Ma, R. Wang, A. Mirhoseini, T. Darrell, and J. E. Gonzalez, “Deep mixture of experts via shallow embedding,” in _Uncertainty in Artificial Intelligence_. PMLR, 2020, pp. 552–562.
  * [16] S. Cai, Y. Shu, and W. Wang, “Dynamic routing networks,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021, pp. 3588–3597.
  * [17] Z. You, S. Feng, D. Su, and D. Yu, “Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts,” _arXiv preprint arXiv:2105.03036_ , 2021.
  * [18] ——, “Speechmoe2: Mixture-of-experts model with improved routing,” _arXiv preprint arXiv:2111.11831_ , 2021.
  * [19] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” _arXiv preprint arXiv:1701.06538_ , 2017.
  * [20] B. Zhang, H. Lv, P. Guo, Q. Shao, C. Yang, L. Xie, X. Xu, H. Bu, X. Chen, C. Zeng, D. Wu, and Z. Peng, “Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition,” 2022.