# 1-2-3 Reproducibility
for Quantum Software Experiments ††thanks: WM acknowledges supported from the
German Federal Ministry of Education and Research (grant 13N15646, “quantum
technologies”).

Wolfgang Mauerer  Technical University of
Applied Sciences Regensburg
Siemens AG, Corporate Research
<EMAIL_ADDRESS>Stefanie Scherzinger  Chair of Scalable Database
Systems
University of Passau
Passau, Germany
<EMAIL_ADDRESS>
###### Abstract

Various fields of science face a reproducibility crisis. For quantum software
engineering as an emerging field, it is therefore imminent to focus on proper
reproducibility engineering from the start. Yet the provision of reproduction
packages is almost universally lacking. Actionable advice on how to build such
packages is rare, particularly unfortunate in a field with many contributions
from researchers with backgrounds outside computer science. In this article,
we argue how to rectify this deficiency by proposing a 1-2-3 approach to
reproducibility engineering for quantum software experiments: Using a meta-
generation mechanism, we generate DOI-safe, long-term functioning and
dependency-free reproduction packages. They are designed to satisfy the
requirements of professional and learned societies solely on the basis of
project-specific research artefacts (source code, measurement and
configuration data), and require little temporal investment by researchers.
Our scheme ascertains long-term traceability even when the quantum processor
itself is no longer accessible. By drastically lowering the technical bar, we
foster the proliferation of reproduction packages in quantum software
experiments and ease the inclusion of non-CS researchers entering the field.

###### Index Terms:

Reproducibility engineering, quantum software engineering

##  I Introduction

The ACM reproducibility guidelines (v1.1) [1] consider an experiment
_reproducible_ when a different team with a different experimental setup is
able to confirm the published results.111The ACM has repeatedly changed the
definition of their reproducibility terminology in an incompatible way; we are
aware of the arising confusion. Despite the universally acknowledged
importance of reproducibility, the vast majority of researchers agree that we
are facing a reproducibility crisis [2, 3] in almost all domains of science,
including quantum computing research [4, 5, 6, 7].

In the emerging field of quantum software (QSW) engineering, reproducibility
engineering has also been recognised as a grand challenge [4, 5, 6, 7], with
first contributions on managing the reproducibility of software bugs [6, 4],
or on exploring the parameter search space for quantum optimisation tasks [8].
Yet so far, little actionable advice has been given on how to engineer QSW
experiments for reproducibility. As we will argue shortly, despite the general
credo that reproducibility is important, at this moment in time, published
articles that are accompanied by reproduction packages are rare.

Building a working _reproduction package_ goes beyond providing a DOI to some
repository hosting data, code, and setup instructions. Rather, a gold-standard
reproduction package [9] bundles all research artefacts required to conduct
the experiment (such as source code, libraries, or input data), and contains a
dispatcher script that allows for executing and evaluating the experiment via
a single command.

### Quantum-specific challenges

Building reproduction packages for QSW shares many challenges of classic
software engineering, most importantly managing complex software stacks that
non-trivially interact with hardware.

Reproducibility challenges specific to QSW experiments are plentiful [10]:
Quantum computing hardware is usually provided as a cloud service, with
vendor-controlled access and configurations. Reproducers may not have ready
access to the very same machine (especially after prolonged periods of time).
Even if they do, they are very likely to find the configuration changed, given
that most quantum hardware is still in its infancy and subject to change even
during the operational phase.

Apart from a unique model/type specification of the employed machine, it is
necessary to provide information on  (a) input generation methods,  (b) qbit
counts,  (c) connectivity topology,  (d) any methods used to transform inputs,
(e) approaches used to map/embed logical qbits onto physical qbits,  (f)
postprocessing methods and utilities employed, and  (g)
programming/initialisation and readout times.  Quantum software experiments
may involve manual tuning, which necessitates specifying any employed
heuristics/policies. Finally, details on how runtimes are measured are
important, especially in cloud settings that may involve interference by
access schedulers.

### Synopsis

We quantitatively review the state of reproducibility in QSW experiments, and
recognise a need for action. We then introduce a customisable _meta package
template_ that generates end-to-end, one-click reproductions with adaptation
examples for common quantum hardware. It can be found at the accompanying
website/DOI), together with a video tutorial. Only little input is required
from the scientists.

### Vision

The ease-of-use of our reproduction package template enables the proliferation
of working reproduction packages for QSW experiments. Moreover, lowering the
threshold for building reproduction packages can foster skills in QSW
engineering, an area difficult to master for newcomers [11].

TABLE I: Reproducibility packages in recent QSW publications. Venue | Year | # Papers | # Exp | # Src | # Repro
---|---|---|---|---|---
QSA@ICSA | 2021 | 4 | 2 | 1 | 0
Q-SET@QW | 2020 | 6 | 2 | 0 | 0
Q-SET | 2021 | 4 | 2 | 1 | 0
QCS@SAC | 2021 | 11 | 10 | 8 | 0
APEQS@FSE | 2020 | 4 | 1 | 0 | 0
QTOP@Netsys | 2019 | 18 | 12 | 2 | 1
Q-SE@ICSE | 2020 | 8 | 3 | 2 | 0
Q-SE@ICSE | 2021 | 8 | 3 | 1 | 1

##  II State of the Art

_State of reproducibility._ Table I provides an overview about the state of
reproducibility in quantum software. Based on eight workshops from 2019–2021,
where we currently observe most activities related to QSW, we reviewed 63
papers (column “# Papers”; detailed results are on the accompanying website).
We classified them according to whether they describe artefacts (column “#
Exp”; algorithmic source code or experiments to be run on quantum hardware or
simulators) that should be provided in a reproducible form. We also state the
number of papers that _at least_ provide source code on a non-permanent
software forge like GitHub (column “# Src”), and the fraction of papers that
provide a reproduction package (column “# Repro”; we count anything provided
under a DOI-safe permanent location, even if it does not meet the standards
suggested in Ref. [1]).

The conclusion is simple: Although most published research is concerned with
executing code and simulations, only slightly more than half of the
publications provide source code _at all_ , and only _two_ papers come with
artefacts on a DOI-safe location! Clearly, there is a mismatch between
desirable reproducibility qualities as mandated by professional associations
(and, therefore, the community itself), and scientific day-to-day reality.

Project-specificartefactsGenericartefactsBuildrecipeDockermeta-
containerContainer(source)CredentialsContainer(binary)$\atom$QPUdest.
integrates sourcedest. produced by sourcedata flow Figure 1: Components and
workflow for 1-2-3 reproducibility engineering: The scientist provides
project-specific artefacts (green). A docker meta-container then generates a
reproduction package consisting of a source container and a pre-built binary
docker container. Both containers can trigger the QSW experiment, record
measurement and configuration data, and analyse results. Credentials for
accessing commercial quantum hardware and cloud services are not encoded into
the reproduction package, but specified as parameter.

##  III A Template Proposal

To make QSW experiments reproducible with little extra effort for researchers,
we suggest a generic, adaptable template (with full source available on the
accompanying website) as reference for QSW experiments as illustrated in Fig.
1. It includes examples for dealing with DWave and IBMQ systems. Based on the
supplied artefacts, the generated package comprises  (a) the complete source
code for the calculations performed in the paper,  (b) results obtained on
quantum computers (to ensure traceability beyind quantum processor
availability) and via classical simulation,  (c) a concise documentation of
each reproduction step,  (d) a “one-click” dispatcher to run the pipeline,
(e) ideally, means of generating the underlying research paper, including
graphs and tables.

The replication package in both, source and pre-built binary form, is intended
to be made available at a DOI-safe, long-term stable location. The former is
easier to extend by peer researchers, but assumes external components on
volatile locations are still available. The latter ensures long-term
reproducibility, since it does not have external dependencies.

To avoid impediments for adoption that might arise from technologies like
docker—many researchers in QSW do not have computer science backgrounds, and
even if, it might not be systems-centric—, we suggest this generative
approach.

##  IV Conclusion

Reproducibility of experiments is key in scientific research. Yet the adoption
of methods to ensure this quality is currently sub-par in the quantum software
literature. Since the field is still in relative infancy, we propose
sustainable rectification of the situation from the start, by providing a
generic scheme that generates reproduction packages that are tailored to
quantum software experiments, and that can be used with minimal effort.

## References

  * [1] Association for Computing Machinery. (2020, 08) Artifact review and badging (version 1.1). [Online]. Available: https://www.acm.org/publications/policies/artifact-review-and-badging-current
  * [2] M. Baker, “1,500 scientists lift the lid on reproducibility,” _Nature News_ , vol. 533, no. 7604, p. 452, May 2016.
  * [3] B. R. Jasny, G. Chin, L. Chong, and S. Vignieri, “Again, and Again, and Again …,” _Science_ , vol. 334, no. 6060, pp. 1225–1225, 2011.
  * [4] J. Campos and A. Souto, “QBugs: A Collection of Reproducible Bugs in Quantum Algorithms and a Supporting Infrastructure to Enable Controlled Quantum Software Testing and Debugging Experiments,” in _Proc. Q-SE@ICSE 2021_ , 2021, pp. 28–32.
  * [5] J. Zhao, “Quantum Software Engineering: Landscapes and Horizons,” _CoRR_ , vol. abs/2007.07047, 2020.
  * [6] P. Zhao, J. Zhao, Z. Miao, and S. Lan, “Bugs4Q: A Benchmark of Real Bugs for Quantum Programs,” _CoRR_ , vol. abs/2108.09744, 2021.
  * [7] M. Fingerhuth, T. Babej, and P. Wittek, “Open source software in quantum computing,” _PLOS ONE_ , vol. 13, no. 12, p. e0208561, Dec. 2018.
  * [8] R. Shaydulin, K. Marwaha, J. Wurtz, and P. C. Lotshaw, “QAOAKit: A Toolkit for Reproducible Study, Application, and Verification of the QAOA,” _CoRR_ , vol. abs/2110.05555, 2021.
  * [9] B. J. Heil, M. M. Hoffman, F. Markowetz, S.-I. Lee, C. S. Greene, and S. C. Hicks, “Reproducibility standards for machine learning in the life sciences,” _Nature Methods_ , vol. 18, no. 10, pp. 1132–1135, Aug. 2021\.
  * [10] C. C. McGeoch, “Principles and guidelines for quantum performance analysis,” in _QTOP@NetSys_ , ser. Lecture Notes in Computer Science, vol. 11413\. Springer, 2017, pp. 36–48.
  * [11] R. Shaydulin, C. Thomas, and P. Rodeghero, “Making Quantum Computing Open: Lessons from Open Source Projects,” in _Proc. International Conference on Software Engineering Workshops_ , 2020, p. 451–455.


# Measuring Regulatory Barriers Using Annual Reports of Firms

Haosen Ge Ph.D. Candidate, Department of Politics, Princeton University,
<EMAIL_ADDRESS>
Existing studies show that regulation is a major barrier to global economic
integration. Nonetheless, identifying and measuring regulatory barriers
remains a challenging task for scholars. I propose a novel approach to
quantify regulatory barriers at the country-year level. Utilizing information
from annual reports of publicly listed companies in the U.S., I identify
regulatory barriers business practitioners encounter. The barrier information
is first extracted from the text documents by a cutting-edge neural language
model trained on a hand-coded training set. Then, I feed the extracted barrier
information into a dynamic item response theory model to estimate the
numerical barrier level of 40 countries between 2006 and 2015 while
controlling for various channels of confounding. I argue that the results
returned by this approach should be less likely to be contaminated by major
confounders such as international politics. Thus, they is well-suited for
future political science research.

##  1 Introduction

Regulations have drawn increasing attention from policymakers in recent years.
The World Trade Organization (WTO) identifies regulatory barriers as the most
salient obstacle to economic globalization in the past decade.111World
Investment Report 2008, 2018 Researchers show that regulations have become a
powerful tool wielded by multinational firms to protect their special
interests [buthe_new_2011, carpenter2013preventing, gulotty_narrowing_2018,
perlman2019, kennard2017firms]. Nonetheless, due to the complexity of
regulatory regimes, understanding the causes and effects of regulatory
barriers often turn out to be very challenging. The obstacle is especially
pronounced when scholars attempt to study regulatory barriers systematically.
Regulations vary considerably by country. It is generally infeasible to assess
how stringent regulations are by studying their texts, as understanding texts
of regulations requires in-depth knowledge of the operation of an industry in
a specific country. To make things worse, the distributional effects of
regulations are often so subtle that researchers cannot tell who the winners
and losers are by only reading the texts. For example, rules on automobile
parts in a country are applied to all auto manufacturers, domestic or foreign.
Yet, the impact of such regulations is far from uniform across auto
manufacturers. To a large extent, the difficulty of identifying and measuring
regulatory barriers has limited the research progress in that area. In this
paper, I propose a novel approach to measuring regulatory barriers at the
country-year level. Leveraging on the information included in the annual
reports of U.S. public companies, I build an original database of observed
regulatory barriers and use it to estimate the barrier level of each country
with a dynamic two-level item response theory model. This new approach helps
us reveal new empirical patterns that are previously unobservable to scholars.

My main source of information comes from the United States Securities and
Exchange Commission (SEC). SEC requires all U.S. public firms to disclose
information about their operation in their annual reports (i.e., 10-K forms).
The annual reports submitted to SEC are much more detailed than the annual
reports published to share information with the firms’ shareholders
222Detailed explanations on 10-Ks. Specifically, federal laws require every
public firm to list all major factors that may adversely impact their
performance in their 10-K forms. For that reason, companies that are subject
to regulatory barriers are required to report their encountered difficulties.
This valuable information provides us with the knowledge that we otherwise
would not know: whether a firm is adversely affected by specific regulations.
As explained by [kennard2017firms], regulations do not evenly affect all firms
in an industry; instead, it is usually the case that rules can tilt the
playground in favor of some firms but against others. It is almost impossible
for scholars to tell how the “playground” is tilted without acquiring any
information from the “players” (i.e., firms). Thus, the information on the
adverse impacts of regulations included in the firms’ 10-K forms can be
valuable for researchers.

To quantify the level of regulatory barriers for each country, I first convert
the information in these reports into a well-structured dataset using text
processing techniques. Then, I build a dynamic item response theory model to
estimate the level of regulatory barriers at the country-year level. I believe
that this new approach contributes to our existing knowledge in the following
aspects:

  1. 1.

Compared with traditional survey-based measurement, my approach can be easily
extended to include more countries for a longer time period. More importantly,
it can estimate the entry-deterrence effects of regulations under reasonable
assumptions. [gulotty_narrowing_2018] argues that one of the major effects of
regulations is entry-deterrence: large firms advocate more stringent industry
standards to forestall new firms from entering by raising the fixed cost of
operation. For example, increasing the quality standard of a product will
increase the cost of production, which can eliminate small producers and deter
other small firms from entering the market. However, existing survey-based
measurements only survey firms that are active in the market and hence cannot
estimate the entry-deterrence effects. In addition, due to the cost of running
surveys, the coverage of existing survey measurements is far from ideal. For
example, the NTM Business Surveys run by the International Trade Commission
only cover around 25 countries which do not include large countries such as
the United States, China, Japan, Germany, South Korea, India, and many
others.333They claim to have surveyed the EU, but I cannot find the data
anywhere on their website at the time of writing this chapter

  2. 2.

The proposed approach is more micro-founded than data collected by
international organizations, such as the Special Trade Concerns (STCs) Report
data compiled by WTO. Members/Observers of WTO can raise STCs against other
countries if they find laws and regulations in other countries discriminatory.
Scholars use whether a country is subject to STCs to measure whether the
country imposes any regulatory barriers. However, it is well documented that
the behaviors of countries in WTO are shaped by political concerns
[davis2012adjudicate]. The STCs records are very likely to be outcomes of both
domestic regulations and international politics. It is then problematic if
scholars of political science use it to study countries’ political behaviors.
However, the firm-level information in the annual reports is less susceptible
to international and domestic political factors. For that reason, my proposed
measurement should be better suited for studies in political science.

  3. 3.

Existing work has shown promising progress in estimating non-tariff barriers
to trade (e.g., [cooley2019barrier, martini2020backward]). The proposed
approach complements their contributions by adding information about barriers
to foreign direct investment and operations of foreign subsidiaries.

##  2 Information in 10-K Forms

The U.S. federal securities laws require public companies to disclose
information on an ongoing basis. All U.S. public firms must submit annual
reports, a.k.a. form 10-K, which provides a comprehensive overview of the
company’s business and financial condition. In the 10-K forms, a company must
offer a detailed description of its main products or services, major
subsidiaries, relevant regulations, major competition, and any possible risks
associated with its business. Therefore, the 10-K forms contain information on
a comprehensive set of regulatory barriers observed by international business
practitioners.

To better illustrate the type of information included in 10-K forms, I will
present some examples here. Many firms report that they have encountered
restrictive laws or regulations in certain countries:

  1. 1.

A wholesale drug company Nu Skin Enterprises reports in their 2006 10-K form:
“laws and regulations in Japan, Korea and China are particularly restrictive
and difficult.”

  2. 2.

A farm machinery producer Deere & Co reports that: “recent industry and
regulatory changes have negatively impacted John Deere’s competitive position
in the potential high growth Russian markets during the fiscal year.”

  3. 3.

An ophthalmic goods producer Cooper Companies INC claims that: “we have
difficulty gaining market share in countries such as Japan because of
regulatory restrictions and customer preferences.”

  4. 4.

A medical equipment producer Immucor Inc states that: “in addition to the
U.S., Europe, Canada and Japan, there are multiple countries worldwide that
also impose regulatory barrier to market entry.”

  5. 5.

An insurance company Gerova Financial Group Ltd claims that “The Chinese and
Vietnamese governments have imposed regulations in various industries,
including the leisure and hospitality and financial services industries, that
would limit foreign investors equity ownership or prohibit foreign investments
altogether in companies that operate in such industries.”

  6. 6.

A software company Versant Corp reports that they are faced with “burdens of
complying with a variety of foreign laws, including more protective employment
laws affecting our sizable workforce in Germany”.

  7. 7.

A technology company Kenexa Corp reports their concerns about the intellectual
property issues: “Further, the laws of some countries, and in particular
India, where we develop much of our intellectual property, do not protect
proprietary rights to the same extent as the laws of the United States.”

  8. 8.

A technology company National Instrument Corp reports the difficulties of
doing business in Hungary: “In response to significant and frequent changes in
the corporate tax law, the unstable political environment, a restrictive labor
code, the volatility of the Hungarian forint relative to the U.S. dollar and
increasing labor costs, we have doubts as to the long term viability of
Hungary as a location for our manufacturing and warehousing operations.”

It is evident from these examples that reporting regulatory barriers in 10-K
forms is a common practice among firms. However, one may question whether
reporting the existence of regulations is a good indicator of the firm being
adversely affected by them. In other words, the presence of regulations does
not necessarily imply whether they constitute barriers for firms. To address
this concern, I present three more examples to show that the information on
regulations included in the annual reports is in fact related to the
distributional effects of regulations.

  1. 1.

In 2005, China passes a regulation that mandates all truck manufacturers to
install electrical throttle to reduce emission. This piece of regulation
obviously affects all major truck producers serving the Chinese market.
Cummins, a U.S. based natural gas engines producer who has joint ventures in
China, reports in their 10-K form that “These (earning) increases were
partially offset by decreased earnings from DCEC (one of Cummins’ joint
venture in China) of $7 million due to reduced demand in China’s truck market
in response to regulatory changes.” It is quite evident that the operation of
Cummins is harmed by the newly implemented regulation. However, in the same
year, another U.S. based automobile parts manufacturers, Williams Control,
found the new regulation an opportunity rather than a hurdle. In their 10-K
form, they reported: “Increases in off-road volumes in China primarily results
from adoption of more stringent emissions standards, which mandate the
inclusion of electronic throttle controls on new vehicles, thus allowing us to
expand our customer base in this market.” We can immediately tell from this
two pieces of information that Cummins is the loser of the new regulation
while Williams Control is the winner.

  2. 2.

Still in 2005, a cosmetic and fragrance producer Inter Parfums reports that
the existing regulations in France have little effect on their operation: “our
fragrances that are manufactured in France are subject to certain regulatory
requirements of the European Union, but as of the date of this report, we have
not experienced any material difficulties in complying with such
requirements.”

The cases presented above demonstrate what these firms report is closely
related to how they are affected by the reported regulations. If a firm finds
a law positively impacts their business, they will have strong incentives to
report it truthfully since their 10-K forms are made available to all
shareholders. Meanwhile, if a firm encounters a harmful regulatory barrier, it
may or may not want to share it with the public, but federal laws make it
mandatory for them to disclose that piece of information. For these reasons, I
argue that firms’ annual reports are both informative and truthful, which
makes them ideal sources for studies of regulatory politics.

###  2.1 The Truthfulness of 10-K Forms

I will further justify the 10-Ks truthfulness in the subsection, as the
plausibility of my proposed barrier index crucially depends on the information
accuracy.

Since 10-K forms can significantly impact a firm’s stock market performance,
one may be suspicious about the authenticity of its information. The scrutiny
is especially warranted when we evaluate how firms report their encountered
regulatory barrier, due to the inherent ambiguity of regulations. In this
section, I would like to establish the reliability of 10-Ks by providing more
substantive information on how public companies write these reports and how
SEC enforces the disclosure requirements. Hopefully, readers can be more
assured about using 10-Ks as a data source.

Public firms have two main concerns when writing the annual reports: 1) they
are concerned with the shareholders instituting legal actions against them for
financial loss resulting from undisclosed issues, and 2) the punishment from
SEC if found guilty of hiding information. For example, In 2013, stockholders
sued a company called Dole Foods for failure to disclose positive information.
In the end, the judge found the company guilty of unfairly keeping the stock
price down.444See a report In this monitoring process, the shareholders serve
as a “fire alarm” that forces the public firms to disclose any positive and
negative information honestly [mccubbins1984congressional].

In addition to shareholders’ monitoring efforts, the SEC also actively takes
measures to ensure the company’s annual reports’ authenticity. After firms
submit their 10-K forms, the SEC staff will review the submissions to monitor
and enhance companies’ compliance with the requirements. If the review process
finds the disclosed information deficient in explanation or clarity, the SEC
staff will provide comments for a company to resolve the issues. Moreover, the
SEC has made disclosure of qualitative information (such as the risk factor
subsection) a focus of its corporate filing reviews [campbell2014information,
brown2018spillover]. From 2013 to 2017, the chairman, Mary Jo White emphasized
the effectiveness of information disclosure which urged firms to include more
relevant qualitative information in their disclosures.

In reality, the SEC also imposes significant punishment on firms that fail to
disclose crucial information. Facebook, for example, is expected to pay $100
million for making misleading disclosures regarding their user privacy
policy.555Facebook to Pay $100 Million for Misleading Investors About the
Risks It Faced From Misuse of User Data: link The SEC’s complaint alleges that
Facebook failed to disclose its risk of a data breach even after it had
discovered the misuse of its users’ information in 2015.

The above evidence suggests that the SEC is well aware of the issue of
dishonest reporting and has taken a series of actions to enforce the filing
requirement. According to a study by [brown2018spillover], firms improve their
filling quality after the SEC issues comments to them during the review
process.

For these reasons, public firms usually utilize two strategies to minimize the
legal risks: 1) they timely add issues that might adversely affect their
operation in the annual reports, and 2) once added, the issues are rarely
removed from a future version of the reports. I argue that these two features
make the annual reports ideal for measuring regulatory barriers. First, it is
very costly for public firms to misreport in the annual reports. Thus, as
researchers, we should be confident in the information quality. Second, the
reluctance of firms to remove negative issues from the report makes it easier
to interpret the reported barrier. Barrier estimates based on the reported
barrier information can be viewed as cumulative regulatory barriers instead of
the change in regulatory barrier levels. As a result, the 10-K forms can be
beneficial for international political economy research.

##  3 Text Processing

To quantify the information in the annual reports, I need to identify
instances where a firm reports being adversely affected by a specific piece of
regulation. The text in annual reports is not well-structured: different firms
have different reporting formats. In addition, the annual reports often
contain several hundred pages of text filled with technical terms. Thus,
converting the annual reports to a well-structured dataset is challenging.

In this paper, I adopted the following strategy to create a feasible data
processing pipeline, which combined a dictionary-based method with supervised
learning:

  1. 1.

First, I break each report into sentences by using a regular expression.

  2. 2.

Second, I select sentences using a dictionary of regulation-related words:
“regulation, regulator, regulatory, law, standard, quota, approval, policy,
intellectual property, requirement, permit, license”. In addition, I filter
out sentences that do not contain a country name.

  3. 3.

Next, I randomly sample 3,846 sentences for human coding, with the help of two
research assistants. If the sentence clearly indicates that the firm is
adversely affected by a regulation, the sentence is coded as “1” (i.e.,
positive). Otherwise, the sentence is coded as “0” (i.e., negative).

  4. 4.

Finally, I train a neural network model to predict the rest of the corpus. If
a report contains any sentence the model predicts to be positive, I code the
entire document as the firm reporting barriers in that country.

This strategy is a balance between accuracy and feasibility. Annotating annual
reports with human coders is almost impossible, as the reports are too long
and highly technical. However, the task is greatly simplified if we only code
specific sentences instead of the entire report. Admittedly, filtering the
document with a dictionary of regulation-related words introduces
(unmeasurable) errors. We leave improving the text processing pipelines to
future research.

Training a prediction model capable of dealing with complex sentence
structures with a limited sample size is the next challenging task. In this
paper, I choose a state-of-the-art neural language model, “BERT”, to perform
this task. “BERT” is a neural language model pre-trained by scientists at
Google [devlin2018bert]. It is designed to understand the context of a
sentence and predict the appropriate words suitable for the context. The model
is trained on a vast corpus that includes the entire Wikipedia (2,500 million
words) and BookCorpus (800 million words. As a result, “’BERT’ is a powerful
tool for most text classification tasks.

I train a classifier on my sample of 3,846 sentences using “BERT” as the
underlying workhorse model (i.e., fine-tuning). The training sample includes
3,486 sentences, while I leave the other 400 sentences as the test set. After
training, the model returns satisfactory prediction accuracy: among the 400
test sentences, the results are:

Table 1: Model Performance: Confusion Matrix | Actual True | Actual False
---|---|---
Predicted True | 44 | 48
Predicted False | 6 | 302
False Positive Rate | 0.12
False Negative Rate | 0.137
Total Error Rate | 0.135

Since the training set has very few positive examples (i.e., only a few
sentences contain information on regulatory barriers), monitoring both the
false positive and negative rates is crucial. In the sparse prediction task, a
model that blindly returns “0” can still achieve seemingly perfect accuracy
simply because the number of positive examples accounts for a tiny proportion
of the training set. Fortunately, my model performs well in distinguishing
positive and negative examples, illustrated by the low false negative rate. As
shown in Table 1, both the false positive rate and the false negative rate are
less than 15%, demonstrating that the model refrains from blindly assigning
“0”.

Using the fine-tuned BERT as the machine reader, I assign 0 or 1 to all
sentences in the corpus. If any sentence in an annual report is predicted to
be positive, I classify the example as the firm reporting regulatory barriers.
Then, I extract the country names by comparing the text with the list of all
country names. The final data contains a matrix of $I$ firms and $N$ countries
over $T$ years. The problem, however, is to aggregate information from the
firm-country-year triad level to the country-year dyad level, as we need to
compare regulatory barrier levels both across country and across time. A naive
approach is to take the mean reporting level at the country-year level, but
such an approach requires strong assumptions that are unsubstantiated by
theories. Therefore, I use a widely-accepted latent factor model to estimate
regulatory barriers, a similar solution to that used by
[hollyer2014measuring].

##  4 The Statistical Model

###  4.1 Setup

The statistical model aims to assign a scalar-valued index to each country in
each year while accounting for major firm and country heterogeneity. I propose
to use an item response theory model to estimate this quantity of interest, a
popular model often used in estimating ideological positions of legislators
[clinton2004statistical] but has become increasingly popular in studies of
international relations (e.g., [hollyer2014measuring, bailey2017estimating]).

First, I convert the firm reported barrier into a well-structured dataset
(Table 2). In a given year $t$, we have a $I\times J$ matrix $\\{U_{ijt}\\}$
with $i\in\\{1,2,3,\cdots,I\\}$ and $j\in\\{1,2,3,\cdots,J\\}$, where $I$
indexes the total number of firms in the sample and $J$ indexes the number of
countries:

Table 2: Data Structure | Country 1 | Country 2 | $\cdots$ | Country $J$
---|---|---|---|---
Firm 1 | $U_{11t}$ | $U_{12t}$ | $\cdots$ | $U_{1Jt}$
Firm 2 | $U_{21t}$ | $U_{22t}$ | $\cdots$ | $U_{2Jt}$
⋮ | ⋮ | ⋮ | ⋮ | ⋮
Firm $I$ | $U_{I1t}$ | $U_{I2t}$ | $\cdots$ | $U_{IJt}$

The term $U_{ijt}$ can take three possible values:

$\displaystyle U_{ijt}=\begin{cases}3&\text{firm $i$ does not enter country
$j$ in year $t$}\\\ 2&\text{firm $i$ enters country $j$ AND reports barrier in
year $t$}\\\ 1&\text{firm $i$ enters country $j$ AND does not reports barrier
in year $t$}\end{cases}$

I assume that each country $j$ has a regulatory barrier level $\theta_{jt}$ in
year $t$ that is observable to firms but is not to researchers. Each firm-
country dyad $(i,j)$ in year $t$ has two dyadic characteristics: an entry
cutoff $\alpha^{E}_{ij}$ and a reporting cutoff $\alpha^{R}_{ij}$, for which I
will provide a more detailed exposition later. The observed firm behavior
$U_{ijt}$ is a function of the country barrier level $\theta_{jt}$ and the
dyad-specific reporting and entry cutoffs. Note that the cutoffs are time-
invariant, a strong assumption to avoid the model identification problems.

The intuition behind such a setup is straightforward. Each firm $i$ observes
the regulatory barrier level of country $j$ in year $t$ (i.e., $\theta_{jt}$)
and chooses an action. The reporting and entry cutoffs capture the firms’
tolerance level of a country’s barrier. The concept of tolerance is an
abstraction from firms’ real-world calculations, which often include market
size, cultural similarity, and geographical distance. Because firms’ decisions
are affected by a multitude of factors, I wrap any factors that shape firms’
decisions but are not components of a country’s regulatory barriers into the
tolerance terms, to alleviate concerns of omitted variables bias caused by
firm and country heterogeneity.

Specifically, if a country’s barrier level is higher than a firm’s entry
tolerance cutoff (i.e., $\alpha^{R}_{ij}$), the firm will not operate in that
country; if, on the other hand, the country’s barrier level is below the
firms’ entry tolerance cutoff but higher than the reporting tolerance cutoff,
the firm will enter the country but report the barrier in their annual
reports; lastly, if the country’s barrier level is lower than both the entry
and reporting cutoff level, the firm will operate in that country and no
barrier reporting will be witnessed in the annual reports. To fix ideas, I
will provide a formal explanation of this logic later in this section.

For ease of exposition (and model estimation), I define a random latent firm-
country barrier level $U_{ijt}^{*}$:

$\displaystyle U^{*}_{ijt}=\theta_{j,t}+\epsilon_{ijt}$

That is, each firm $i$ perceives the barrier level of country $j$ in year $t$
as slightly different. The random disturbance $\epsilon_{ijt}$ captures any
idiosyncrasies at the firm-country-year level. Readers can interpret the
errors as factors that are unobservable to researchers but affect firms’
perception of countries’ regulatory barriers. Following the practice in the
scaling literature, I assume that the disturbance term follows a standard
normal distribution $\mathcal{N}(0,1)$.

Each firm $i$ compares the latent firm-country barrier level $U_{ijt}^{*}$
with its entry and reporting cutoffs and chooses an action according to the
following decision rule:

  * •

If country $j$’s latent barrier level is higher than firm-country $(i.j)$’s
dyadic entry cutoff (i.e., $U^{*}_{ijt}>\alpha_{ij}^{E}$), firm $i$ will not
enter country $j$ (i.e., $U_{ijt}=3$).

  * •

If country $j$’s barrier level is lower than $(i,j)$’s entry cutoff but the
barrier level is higher than the dyad’s reporting cutoff (i.e.,
$\alpha_{ij}^{E}>U^{*}_{ijt}>\alpha^{R}_{ij}$, firm $i$ will enter country $j$
but report encountering barrier (i.e., $U_{ijt}=2$).

  * •

If country $j$’s barrier level is lower than the dyad $(i,j)$’s reporting
cutoff (i.e., $U^{*}_{ijt}\leq\alpha_{ij}^{R}$), firm $i$ will enter country
$j$ and not report any barrier (i.e., $U_{ijt}=1$).

Note that the implicit assumption is that the entry cutoff is greater than the
reporting cutoff (i.e., $\alpha_{ij}^{E}>\alpha^{R}_{ij}$).

Since barrier level $U^{*}_{i,j,t}$ and $\theta_{jt}$ is unobservable to
researchers, the proposed approach leverages the connection between regulatory
barrier and firms’ entry and report behaviors to make inference about the
barrier level. It is worth noting that the model will interpret a firm not
entering a country as a signal of prohibitive regulatory barriers after taking
the firms’ tolerance level into account. This is admittedly the strongest
assumption of the model, as it fails to distinguish between a country being
unattractive and a country imposing a significant entry barrier. I try to
address this concern by only including the most globally active U.S. firms,
which I will explain in detail in the next subsection. Nonetheless, readers
should be cautious when interpreting the results.

This model can be translated into a statistical model by noting the
relationship between the theoretical model and our observed data.

$\displaystyle U_{ijt}=\begin{cases}1\quad\text{if}\quad
U^{*}_{ijt}\leq\alpha_{ij}^{R}\\\
2\quad\text{if}\quad\alpha_{ij}^{R}<U^{*}_{ijt}\leq\alpha_{ij}^{E}\\\
3\quad\text{if}\quad U^{*}_{ijt}>\alpha_{ij}^{E}\end{cases}$

Denote the set of $\\{\theta_{j,t}\\}$ as $\Theta$ and the set of
$\\{\alpha_{ij}^{E}\\}$ and $\\{\alpha_{ij}^{R}\\}$ as $\alpha^{E}$ and
$\alpha^{R}$. Let $U$ denote the observed data and $U^{*}$ the augmented data.
Then, we can write the full data likelihood with data augmentation as:

$\displaystyle\mathcal{L}(\Theta,\alpha^{E},\alpha^{R}|U,U^{*})=\prod_{t=1}^{T}\prod_{j=1}^{J}\prod_{i=1}^{I}[I(U_{ijt}=1,U^{*}_{ijt}\leq\alpha_{ij}^{R})+I(U_{ijt}=2,\alpha_{ij}^{R}<U^{*}_{ijt}\leq\alpha_{ij}^{E})$
$\displaystyle+I(U_{ijt}=3,U^{*}_{ijt}>\alpha_{ij}^{E})]\cdot\phi_{\theta_{jt}}(U_{ijt}^{*})$

where $\phi_{\theta_{jt}}(\cdot)$ denotes the probability density function of
$\mathcal{N}(\theta_{jt},1)$.

The high dimensionality of the model poses a considerable challenge for
efficient estimation. Therefore, I propose a Gibbs sampler with a Kalman
filter to achieve optimal estimation performance. Existing political science
and economics studies have demonstrated the superiority of such an estimation
strategy over a generic Gibbs sampler or a Metropolis-Hastings sampler
[martin2002dynamic, west2006bayesian], as it efficiently utilizes the time
series information.

###  4.2 Posterior

Recall that the model aims to calculate the mean barrier level for each
country in each year. We can obtain the desired quantity by calculating the
mean of each parameter’s posterior distributions.

First, priors distributions are necessary for calculating the posterior
distributions,

$\displaystyle\alpha_{ij}^{R}|\alpha_{ij}^{E}$
$\displaystyle\sim\mathcal{N}_{(-\infty,\alpha_{ij}^{E})}(0,5^{2})$
$\displaystyle\alpha_{ij}^{E}|\alpha_{ij}^{R}$
$\displaystyle\sim\mathcal{N}_{(\alpha_{ij}^{R},\infty)}(0,5^{2})$
$\displaystyle\theta_{j,0}$ $\displaystyle\sim\mathcal{N}(0,5^{2})$

Priors are designed to capture our existing knowledge of these parameters.
Since we lack information on the possible values of these parameters, I choose
diffuse priors with large variances. However, it is worth noting that the
likelihood function has flat regions that can not be distinguished using only
the data. Specifically, it is equally likely for the model to assign a large
positive number (e.g., $100$) or a small negative number (e.g., $-100$) to a
country’s barrier level, as we did not provide information to the model about
whether a positive number signifies a higher barrier level or a negative
number does. To overcome this issue, I set semi-informative priors for two
countries in the sample: Russia’s barrier level is positive while Singapore’s
is negative. It tells the model that a positive number entails a higher
barrier level than a negative one, as Russia has a higher level of barriers
than Singapore to U.S. firms, according to anecdotal evidence.

Next, using the priors, I sample the augmented data $U_{ijt}^{*}$ from its
conditional distribution:

$\displaystyle
p(U_{ijt}^{*}|U,\Theta,\alpha^{E},\alpha^{R})=\begin{cases}\mathcal{N}_{(-\infty,\alpha_{ij}^{R}]}(\theta_{jt},1)\quad&\text{if}\quad
U_{ijt}=1\\\
\mathcal{N}_{(\alpha_{ij}^{R},\alpha_{ij}^{E}]}(\theta_{jt},1)\quad&\text{if}\quad
U_{ijt}=2\\\
\mathcal{N}_{(\alpha_{ij}^{E},\infty)}(\theta_{jt},1)\quad&\text{if}\quad
U_{ijt}=3\\\ \end{cases}$

the notation $\mathcal{N}_{(a,b)}$ denotes a truncated normal distribution on
the support of $(a,b)$. This is the standard data augmentation step for models
with latent variables [albert1993bayesian].

After sampling the latent barrier $U_{ijt}^{*}$, I sample the reporting
cutoffs $\alpha_{ij}^{R}$ from its posterior distribution.

$\displaystyle f(\alpha_{ij}^{R}|U,U^{*},\Theta,\alpha^{E})$
$\displaystyle=L(\Theta,\alpha^{E},\alpha^{R}|U,U^{*})\cdot
p(\alpha_{ij}^{R}|\alpha_{ij}^{E})$
$\displaystyle\propto\prod_{t=1}^{T}[I(U_{ijt}=1,U^{*}_{ijt}\leq\alpha_{ij}^{R})+I(U_{ijt}=2,\alpha_{ij}^{R}<U^{*}_{ijt}\leq\alpha_{ij}^{E})]\cdot
p(\alpha_{ij}^{R}|\alpha_{ij}^{E})$

Note that the product of $T$ indicator functions truncate the prior
distribution $p(\alpha_{ij}^{R}|\alpha_{ij}^{E})$ and only allows it to have
positive densities over the interval $[\max(U_{ijt}^{*}|U_{ijt}=1),\\\
\min(U_{ijt}^{*}|U_{ijt}=2))$. Analogously, the posterior distribution of
$\alpha_{ij}^{E}$ is proportional to the prior distribution
$p(\alpha_{ij}^{E}|\alpha_{ij}^{R})$ truncated on the interval
$[\max(U_{ijt}^{*}|U_{ijt}=2),\min(U_{ijt}^{*}|U_{ijt}=3))$.

To sample the barrier level $\theta_{jt}$, we need to define its evolution
probability. That is, how a country’s barrier level fluctuates across years.
In this paper, I model the state transition as a random walk:

$\displaystyle\theta_{j,t}=\theta_{j,t-1}+\delta_{jt}$

where $\delta_{jt}$ follows a normal distribution $\mathcal{N}(0,\sigma^{2})$.
I fix the variance term $\sigma^{2}$ a priori as $1$ for identification.
However, it is worth noting that the variance term $\sigma$ smoothes the
barrier level across years because the degree to which an estimated
$\theta_{jt}$ shrinks back to the prior mean is inversely proportional to the
variance of the disturbance term $\delta_{jt}$. For example, if a country’s
barrier level is completely independent of one another across years, or in
other words, the barrier is not sticky at all, the disturbance variance would
be infinity. Thus, fixing it as $1$ assumes a sticky regulatory barrier.

Finally, we sample the quantity of interest $\theta_{jt}$ from the full
conditional posterior distribution $f(\Theta|\alpha^{R},\alpha^{E},U,U^{*})$.
A naive Gibbs Sampler approach requires sampling $\theta_{jt}$ conditional on
$\theta_{j,t-1}$, which fails to incorporate the sticky evolution process and
is hence inefficient. Literature has demonstrated that sampling highly
correlated parameters with Gibbs Sampler can also lead to difficult-to-
converge chains [martin2002dynamic]. To improve estimation, I adopt the Kalman
filter to sample the entire times series at a time rather than sampling from
the component by component conditional distributions (i.e., sample the entire
time series $\theta_{j,k}$, $k\in\\{1,2,\cdots,T\\}$ instead of each
$\theta_{jt}$ at a time). For ease of exposition, I omit the description of
the forward-filtering and backward-sampling procedure.

One major weakness of this approach is that it cannot provide a regulatory
barrier estimate for the U.S. because firms that file 10-K forms are U.S.
firms. The U.S. is the home country for these firms, while any other country
in which they operate is the host country. By definition, these firms must
enter the U.S. market. And it is reasonable to argue that the data generating
process is very different for the home and host countries. For these reasons,
I exclude the U.S. from my analyses.

##  5 Data and Results

###  5.1 Data

My primary data source is the 10-K forms published by the U.S. Security and
Exchange Commission (SEC). Since the model cannot distinguish between 1) a
firm does not seek market entry and 2) the regulatory barrier is too
restrictive for a firm to enter, the confounding effect of preference
undermines the interpretability of the results. Although the proposed model
accounts for time-invariant unobserved dyadic heterogeneity (e.g., an energy
company always prefers natural resource abundant countries), yearly
fluctuations in global economic/political conditions and firms’ financial
performance may still bias my results. I partially circumvent this thorny
issue by excluding 1) countries that host very few U.S. firms and 2) firms
that have minimal international commercial activities. In essence, I aim to
construct a sample in which firms’ preferences can be viewed as constant so
that any variation in firm behaviors must be due to the changing barrier
levels.

As a result, I adopt the following exclusion criteria:

  * •

I exclude countries that more than 95% of firms in my sample never entered.
These are mostly African countries.

  * •

I only include the top $1500$ firms that are most active in the international
market between 2006 and 2015. A firm is deemed more active if it consistently
operates in more countries.

The period of my focus is between 2006 and 2015. The 10-K database used in
this paper is compiled by [loughran2016textual], which covers 10-K reports
from 1993. At the time of download in 2019, I excluded years after 2015 to
avoid potential backfiling issues. Years before 2006 are also excluded to
reduce computational complexity, as information from that period is relatively
outdated.

After cleaning, I have a sample of 853 firms and 40 countries. Because I only
keep firms that are consistently in the sample between 2006 and 2015, the
number of firms drops from 1500 to 853.

###  5.2 Results

Figure 1: Temporal Change in Regulatory Barrier (Brazil, Canada, China, Japan)

Figure 1 presents the temporal change in the estimated barrier level for four
countries (Brazil, Canada, China, Japan); however, readers can find the entire
list of estimated barrier levels in the appendix. Since the four chosen
countries have close economic connections with the U.S., examining their
estimated barrier level is a preliminary test for the estimates’ plausibility.

Among the four countries, Canada has the lowest barrier level consistently.
Recall that the estimates draw information from the U.S. firms’ entry and
reporting decisions. In the case of Canada, it demonstrates that more U.S.
firms enter Canada, but fewer of them report barriers when compared with the
three other countries. It is worth noting that the number of U.S. firms that
report encountering barriers in Canada may still exceed the other three
countries. Yet, Canada’s barrier level is still estimated to be lower because
the number of firms operating in Canada can be much higher than in other
countries.

Compared to the other three countries, China also displays a relatively low
barrier level, which is counter-intuitive, as many U.S. firms report
encountering barriers in China in my datasets. Again, this result should be
driven by the large number of U.S. firms operating in China. However, the
estimated difference in barrier level between China and Canada is still
considerable, even though both countries host many U.S. firms. Thus, I believe
that the model aggregates the firms’ entry and reporting information in a
consistent and reasonable manner.

The estimates show a sharp jump in the barrier level across all countries from
2006 to 2007, suggesting a global shock in 2007. Among the four countries, the
jump is the largest for Canada, followed by Brazil, China, and Japan. It is
difficult to pinpoint the cause of the jump using only the information from
the dataset. However, I offer some suggestive evidence regarding the possible
causes. In 2006, there were 2,305 incidences of barrier reporting, while that
number increased to 2,636 in 2007, a 14.4% increase. A plausible reason could
be the 2007-2008 global financial crisis which should impact the regulatory
barrier levels globally. It appears that more U.S. firms had financial
difficulties in 2007 than in 2006. For example, here is a list of statements
on bankruptcy in the 2007 10-Ks, whose number increases by 20% from 2006 to
2007:

  * •

“Some of our current and former international customers, particularly
automobile manufacturers in Europe and Japan, were reluctant to do business
with us while we underwent chapter 11 bankruptcy.”

  * •

“The proposed transaction is subject to approval by the United States
Bankruptcy Court, receipt of required regulatory approvals, finalizing the
definitive purchase agreement for Akzo Nobel’s Crystex.”

Still, it is worth emphasizing that more systemic analyses are required to
understand the financial crisis’s effect on the observed barrier level jump.

Figure 2: Pairwise Comparison of Countries’ Average Barrier Level

Figure 2 presents a cross-country comparison of the estimated regulatory
barrier level. Each cell in the heat map is the barrier difference between a
pair of countries labeled by the axis ticks. The country’s barrier level is
the average of its levels across years. Specifically, a negative value in the
cell signifies that the country represented by the Y axis (i.e., party 1) has
a lower barrier level than that on the X axis (i.e., party 2). It can be
observed that the row of Canada is the bluest among all, which shows that
Canada has the lowest average barrier level among all countries.

Similarly, India, Germany, and France also display significantly lower barrier
levels than other countries on average. On the other hand, Philippines,
Norway, and New Zealand have relatively higher barriers than other countries
on average. On the other hand, China has a medium level of barriers compared
with the rest of the world.

Figure 1 and Figure 2 serve as preliminary validation tests of the estimated
barrier. However, more rigorous results are needed to establish the accuracy
and consistency of the proposed index. Thus, I present several statistical
analyses in the next section, which compare and contrast my index with other
popular measurements in the field.

##  6 Validation

###  6.1 Special Trade Concerns (STCs)

[gulotty_narrowing_2018] has shown in his book that the special trade concerns
(STCs) data, and more specifically, the technical barrier to trade (TBT) data
collected by the World Trade Organization (WTO) can inform researchers of the
regulatory barrier levels of major economies in the world.

As I briefly explained in the introduction section, my proposed index can be
superior to the STC-TBT data in two major aspects:

  * •

[gulotty_narrowing_2018] noted that: “the choice to raise a foreign regulation
as an STC is as much a political process as the choice to impose the
regulation in the ﬁrst place.” Thus, the STC-TBT data is likely to be heavily
influenced by international politics. This concern is challenging to eliminate
but can be fatal for researchers who study the correlation between regulatory
barriers and international relations. However, my estimated barrier is less
susceptible to such a concern as it is unlikely that international politics
may shape an individual firm’s decision to report in their annual reports.

  * •

It is known that STC-TBT data tend to target larger markets, as governments
need to balance the cost of filing an STC complaint and its benefit on the
domestic economy [fontagne2018let, gulotty_narrowing_2018]. Therefore, the
observed STC-TBT report distribution is heavily skewed along the market size
dimension: countries with larger market sizes are more likely to be included
in STC reports than their smaller counterparts, even if they have the same
barrier level. Admittedly, my index cannot eliminate the contaminating effect
of market sizes. Still, by accounting for time-invariant dyadic-specific
confounders, the proposed index should address the concern more satisfyingly.

(a) Comparison with STCs count

(b) Comparison with OECD FDI Regulatory Restrictiveness Index

I conduct two validation analyses to compare the STC-TBT data and my proposed
index. First, I normalize the STCs count that a country is subject to and its
estimated barrier by subtracting their mean and dividing by the standard
deviation. Then, I take the difference between these two data at the country
level and plot the difference against the market size of the countries,
measured by GDP. The result is presented in Figure 3(a). The horizontal axis
shows the log GDP of each country: countries with a large GDP are placed more
on the right. The vertical axis is the normalized difference between the STCs
count and the estimated barrier level: a negative value shows that my
estimated barrier level is lower than the normalized STCs count. We observe
that the difference is more pronounced in countries with large market sizes.
More specifically, the barrier level measured by my index is consistently
lower than that measured by the STCs count among countries with large market
sizes. On the contrary, the two measurements align quite well among small
countries. These findings are consistent with the observation that the STC-TBT
data often inflate the barrier level of large economies. The proposed index
suffers less from such a weakness.

Next, I regress my estimated barrier level on the count of STC-TBT reports
filed against each country between 2006 and 2015. Column (1), (2), and (3) in
Table 3 display the regression results. The bi-variate regression between the
proposed index and the STCs count returns a negative coefficient. However, the
coefficient becomes positive after accounting for country, year fixed effects,
and GDP/GDP per capita. Although the positive correlation fails to achieve
statistical significance, the results nonetheless corroborate my claims on the
contaminating effect of market sizes and the weakness of the STC-TBT data.

| Estimated Barrier
---|---
Model: | (1) | (2) | (3) | (4) | (5) | (6)
STCs Count | -0.0967∗∗∗ | 0.0375 | 0.0386 |  |  |
| (0.0230) | (0.0349) | (0.0351) |  |  |
FDI Restrictiveness Index |  |  |  | -0.3421 | -4.702∗∗∗ | -6.371∗∗∗
|  |  |  | (0.3014) | (0.9352) | (1.541)
Log GDP |  |  | 0.4774 |  |  | -0.5753∗∗
|  |  | (0.6072) |  |  | (0.2855)
Log GDP pc |  |  | -0.4406 |  |  | 0.4051
|  |  | (0.5846) |  |  | (0.2491)
Year |  | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | $\checkmark$
Country |  | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | $\checkmark$
_Fit statistics_ |  |  |  |  |  |
Observations | 400 | 400 | 310 | 231 | 231 | 203
R2 | 0.04266 | 0.72868 | 0.69764 | 0.0056 | 0.69117 | 0.68889
_Signif. Codes: ***: 0.01, **: 0.05, *: 0.1_
Table 3: Regression Analyses of STCs, FDI Restrictiveness Index and the
Estimated Barrier

###  6.2 OECD FDI Regulatory Restrictiveness Index

Next, I compare my barrier index with the popular OECD FDI Regulatory
Restrictiveness Index [koyama2006oecd, kalinova2010oecd]. The index covers
four types of regulatory measures: (1) foreign equity restrictions, (2)
screening and prior approval requirements, (3) rules for key personnel, and
(4) other restrictions on the operation of foreign enterprises. In this
analysis, I use the aggregate index at the country-year level.

First, I plot log GDP against the normalized difference between FDI regulatory
restrictiveness index and the estimated barrier (Figure 3(b)). There is a
slight negative relationship between the average difference and log GDP,
similar to what we observe in the STC-TBT case. China, India, and Canada are
still among the countries that enjoy a considerable negative difference. That
is, my proposed index assigns a significantly lower barrier level than the FDI
regulatory restrictiveness index. However, the observed negative correlation
is much less pronounced than it is in the STC-TBT case, which suggests that
the FDI regulatory restrictiveness index captures issues that differ from what
STC-TBT capture.

I proceed to regress my estimated barrier index on the FDI regulatory
restrictiveness index and report the results in column (4), (5), and (6) in
Table 3. However, it is surprising that the coefficients of the FDI
restrictiveness index are consistently negative, suggesting a strong negative
correlation between my proposed barrier estimates and the restrictiveness
index. The negative relationship persists even after accounting for market
sizes and country/year idiosyncrasies. It shows that countries with a higher
FDI regulatory restrictiveness index are often associated with a lower
regulatory barrier level, per my proposed estimates. However, readers need to
be cautious when interpreting this seemingly contradictory result, as the two
indices may simply reflect different aspects of regulatory barriers. The FDI
regulatory restrictiveness index focuses on the regulatory restrictions that
only affect foreign firms. At the same time, my proposed barrier aims to
measure regulation that affects both domestic and foreign firms but may
constitute hidden obstacles for foreign firms in practice. The results could
demonstrate that countries that use domestic regulations as hidden barriers
can adopt less restrictive FDI-targeting measures, as blatant restrictions on
foreign ownership may imply significant political costs both domestically and
internationally [kono2008democracy, esberg2021covert].

###  6.3 Trade, FDI, and Democracy

Finally, I examine the relationship between my proposed barrier estimates and
the U.S. trade volume, FDI flow, FDI stock, and regime types. Intuitively, a
higher barrier should be correlated with lower trade flow, lower FDI flow, and
lower FDI stock. I also revisit the classic debate between regime types and
regulatory barrier [milner2005move, kono2008democracy,
pandya2014democratization].

I visualize the results in Figure 4. Each panel represents the overall
correlation between my proposed barrier estimate and trade flow, FDI flow, FDI
stock, and the Polity 2 score. It can be clearly observed that a country with
a higher barrier level is associated with 1) lower trade volume with the U.S.,
2) receives less FDI from the U.S., and 3) has lower FDI stock from the U.S..
However, the correlation between the Polity 2 score and the estimated barrier
level is very weak. These results lend further credibility to my proposed
barrier estimates.

Figure 4: Correlation with Major Indicators (Polity, Trade, FDI Flow, FDI
Stock)

##  7 Conclusions

This paper offers a novel measurement of the elusive quantity: regulatory
barrier, contributing to the empirical literature on the political economy of
regulation and regulatory barrier. I leverage information in the annual
reports of U.S. public firms (i.e., 10-Ks forms) to best address two major
concerns when measuring regulatory barriers: (1) the confounding effect of
international politics and (2) the bias caused by market size. I show in a
series of validation analyses that the proposed barrier index shows patterns
consistent with our existing knowledge of regulatory barriers. Moreover, the
new barrier estimates display signs of better addressing the problems
mentioned in the previous text than the existing measurements.

The proposed barrier index serves as an additional measurement of regulatory
barriers, which may be superior to existing ones in some research contexts. I
intend not to claim that my estimate precisely captures the concept of
regulatory barrier and hence is the “best” one. However, my goal is to shed
light on new aspects of regulatory barrier that evades researchers’ attention
due to their hard-to-observe nature. There are obvious weaknesses in both the
information sources and my models. Nonetheless, my estimates offer new
insights into this important political and economic phenomenon.

The paper also contributes to international political economy and
international relations by proposing the firms’ annual reports as a new source
of information for further research. Since high-quality text data has become
increasingly important for empirical research in political science, the rich
information contained in the 10-Ks firms deserves more scholarly attention in
the future.

##  8 Appendix

###  8.1 The Full List

I list the estimated barrier levels for all 40 countries below.  Country 2006
2007 2008 2009 2010 2011 2012 2013 2014 2015 Singapore -0.18 0.68 0.70 0.71
0.68 0.67 0.62 0.61 0.63 0.64 Australia -1.12 0.24 0.18 0.17 0.16 0.17 0.16
0.15 0.13 0.13 Austria 0.12 0.81 0.78 0.77 0.78 0.76 0.73 0.76 0.76 0.79
Belgium -0.23 0.62 0.55 0.58 0.56 0.54 0.52 0.51 0.51 0.54 Bermuda 0.32 0.88
0.90 0.86 0.85 0.84 0.87 0.89 0.86 0.88 Brazil -0.41 0.43 0.43 0.42 0.39 0.35
0.29 0.25 0.25 0.24 Canada -5.56 -0.23 -0.23 -0.26 -0.25 -0.31 -0.32 -0.32
-0.32 -0.33 Cayman Islands 0.03 0.91 0.85 0.85 0.87 0.83 0.78 0.83 0.83 0.83
Chile 0.09 0.95 0.88 0.91 0.86 0.84 0.79 0.74 0.73 0.76 China -0.59 0.09 0.08
0.07 0.07 0.01 -0.00 -0.00 0.04 0.04 Denmark 0.09 0.84 0.78 0.80 0.76 0.76
0.75 0.77 0.78 0.74 France -0.51 0.17 0.15 0.16 0.16 0.13 0.15 0.16 0.16 0.17
Germany -0.47 0.14 0.13 0.13 0.13 0.10 0.08 0.08 0.06 0.08 India -0.62 -0.09
-0.11 -0.12 -0.14 -0.13 -0.16 -0.16 -0.17 -0.19 Indonesia 0.29 1.00 0.95 0.94
0.97 0.92 0.91 0.87 0.86 0.89 Ireland 0.03 0.62 0.55 0.52 0.51 0.51 0.43 0.43
0.48 0.50 Israel 0.34 0.99 1.01 1.01 1.02 0.97 0.98 0.99 0.95 0.92 Italy -0.11
0.38 0.37 0.37 0.38 0.39 0.31 0.29 0.30 0.33 Japan -0.30 0.16 0.11 0.12 0.17
0.15 0.05 0.08 0.12 0.13 Jersey -0.14 0.26 0.22 0.24 0.30 0.33 0.33 0.35 0.34
0.35 Luxembourg 0.39 0.87 0.86 0.82 0.79 0.72 0.68 0.64 0.63 0.60 Malaysia
0.32 0.75 0.71 0.70 0.69 0.67 0.65 0.62 0.59 0.61 Mexico -0.37 -0.05 -0.02
-0.01 -0.03 -0.06 -0.03 -0.04 -0.03 -0.06 Netherlands -0.02 0.26 0.26 0.28
0.26 0.24 0.20 0.21 0.21 0.20 New Zealand 0.40 0.73 0.67 0.71 0.68 0.65 0.62
0.62 0.61 0.62 Norway 0.51 0.92 0.85 0.88 0.85 0.85 0.80 0.81 0.76 0.74
Philippines 0.59 0.92 0.88 0.92 0.93 0.88 0.90 0.85 0.84 0.85 Poland 0.50 0.80
0.75 0.75 0.72 0.72 0.69 0.67 0.66 0.70 Puerto Rico 0.53 0.80 0.81 0.81 0.82
0.81 0.79 0.74 0.75 0.76 United Kingdom 0.60 0.82 0.68 0.65 0.64 0.61 0.57
0.54 0.50 0.47 Argentina 0.23 0.42 0.38 0.36 0.34 0.30 0.25 0.24 0.24 0.25
South Africa 0.65 0.81 0.77 0.77 0.75 0.71 0.68 0.67 0.63 0.65 South Korea
0.85 1.01 1.01 1.00 0.96 0.93 0.86 0.86 0.87 0.87 Spain 0.31 0.42 0.40 0.43
0.40 0.41 0.33 0.33 0.34 0.40 Sweden 0.58 0.70 0.62 0.64 0.66 0.63 0.62 0.62
0.62 0.59 Switzerland 0.43 0.55 0.52 0.51 0.50 0.46 0.45 0.46 0.43 0.44 Taiwan
0.61 0.66 0.67 0.68 0.67 0.64 0.60 0.59 0.60 0.62 Thailand 0.78 0.81 0.78 0.77
0.76 0.75 0.67 0.64 0.66 0.66 Turkey 1.02 1.03 0.98 0.93 0.89 0.86 0.79 0.81
0.79 0.78 Russia 0.03 0.05 0.03 0.01 -0.00 0.01 0.02 -0.02 0.02 -0.06

###  8.2 Regression Analyses of Trade, FDI, and Regime Type

| Estimated Barrier
---|---
Model: | (1) | (2) | (3) | (4) | (5) | (6) | (7) | (8)
Log Trade | -0.1268∗∗∗ |  |  |  | -0.0077 |  |  |
| (0.0093) |  |  |  | (0.0125) |  |  |
Log FDI Flow |  | -0.0628∗∗∗ |  |  |  | -0.0096 |  |
|  | (0.0206) |  |  |  | (0.0165) |  |
Log FDI Stock |  |  | -0.0464∗∗∗ |  |  |  | -0.0323 |
|  |  | (0.0163) |  |  |  | (0.0397) |
Polity2 |  |  |  | 0.0043 |  |  |  | -0.0260∗∗∗
|  |  |  | (0.0052) |  |  |  | (0.0067)
Log GDP |  |  |  |  | 0.5403 | 0.2625 | 0.3575 | 0.5571
|  |  |  |  | (0.5702) | (0.7483) | (0.6548) | (0.5560)
Log GDP pc |  |  |  |  | -0.5018 | -0.1287 | -0.0892 | -0.5041
|  |  |  |  | (0.6377) | (0.8399) | (0.7496) | (0.6230)
_Fixed-effects_ |  |  |  |  |  |  |  |
Country |  |  |  |  | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
Year |  |  |  |  | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
Observations | 760 | 217 | 268 | 620 | 620 | 189 | 238 | 620
R2 | 0.19626 | 0.04152 | 0.02957 | 0.00114 | 0.69519 | 0.65353 | 0.67206 | 0.69698
_Signif. Codes: ***: 0.01, **: 0.05, *: 0.1_

Table 4: Regression Analyses of Trade, FDI, and Regime Type

Table 4 presents the regression results of the same exercise. Column 1 to 4
reports the bi-variate regression results that are visualized in Figure 4. In
column 5 to 8, I include country/year fixed effects and economic variables as
covariates. Interestingly, the coefficient of Polity 2 flips and becomes
statistically significant, suggesting that democracies are associated with
lower estimated barrier. However, the coefficients of the three other economic
variables fail to achieve statistical significance after including the fixed
effects and covariates.


# ICASSP 2023 Deep Noise Suppression Challenge

###### Abstract

Deep Speech Enhancement Challenge is the 5th edition of deep noise suppression
(DNS) challenges organized at ICASSP 2023 Signal Processing Grand Challenges.
DNS challenges were organized during 2019-2023 to stimulate research in deep
speech enhancement (DSE) [1]. Previous DNS challenges were organized at
INTERSPEECH 2020, ICASSP 2021, INTERSPEECH 2021, and ICASSP 2022. From prior
editions, we learnt that improving signal quality (SIG) is challenging
particularly in presence of simultaneously active interfering talkers and
noise. This challenge aims to develop models for joint denosing,
dereverberation and suppression of interfering talkers. When primary talker
wears a headphone, certain acoustic properties of their speech such as direct-
to-reverberation (DRR), signal to noise ratio (SNR) etc. make it possible to
suppress neighboring talkers even without enrollment data for primary talker.
This motivated us to create two tracks for this challenge: (i) Track-1
Headset; (ii) Track-2 Speakerphone. Both tracks has fullband (48kHz) training
data and testset, and each testclips has a corresponding enrollment data
(10-30s duration) for primary talker. Each track invited submissions of
personalized and non-personalized models all of which are evaluated through
same subjective evaluation. Most models submitted to challenge were
personalized models, same team is winner in both tracks where the best models
has improvement of 0.145 and 0.141 in challenge’s Score as compared to noisy
blind testset.

Index Terms—  Deep Speech Enhancement, Personalized P.835, Perceptual Speech
Quality, Personalized Noise Suppression

##  1 INTRODUCTION

Deep Speech Enhancement (DSE) models perform significantly better than their
classical counterparts [2, 3, 4, 5, 1, 6]. We open-sourced training datasets
and test sets, Personalized ITU-T P.835 subjective evaluation framework [7] as
part of the previous DNS challenges. Our GitHub repository111
https://github.com/microsoft/DNS-Challenge open-sourced Personalized and Non-
personalized DNSMOS P.835 [8] and word accuracy (WAcc) APIs to empower
iterative model improvements for teams articipating in the challenge. This
reduces the barrier to entry in field and provides standard tools for
evaluation of DSE models. Like previous challenges, each track has two
testsets: (i) development (dev) test set which was released at the beginning
of the challenge; (ii) blind test set released a few days before the final
challenge deadline. While dev testset enables intermediate model evaluations,
blind set is used for final ranking of models based on challenge metric
(Score). Registration and submission of enhanced clips was done through CMT
site https://cmt3.research.microsoft.com/DNSChallenge2023, cloud storage and
Azure Blobs as per participant’s preference. Questions related to challenge
can be sent to<EMAIL_ADDRESS>
In this challenge, participants could use any datasets including external
corpora, challenge training datasets to do model training. We do not prohibit
the use of headset corpora for training speakerphone models and vice versa.
Participants were required to describe the datasets used for training their
models in sufficient detail in their extended journal papers, and provide a
brief coverage in 2-page ICASSP grand challenge paper. Challenge website
https://aka.ms/5th-dns-challenge has details of scope and requirements;
definitions of algorithmic latency, processing latency, causal model, Real-
time factor (RTF) and associated challenge rules; and name of winning teams
etc. We verified whether the top models are real-time or not and provide this
information on our website. Verification was done with a $NRT$ testset which
contains clips with overlapping segments. Previous challenge websites are
linked at https://aka.ms/dns-challenge.

We introduced the following changes in this challenge: (i) there are two
tracks: Headset, and Speakerphone both has desktop and mobile recordings in
testsets; (ii) All testclips in both tracks has 10-30s enrollment speech
(primary talker) with or without noise; (iii) Personalized P.835 framework is
improved and now includes voice recognition, robust spam filtering and more
accurate evaluation of enhanced testclips with noise and neighboring talkers;
(iv) Personalized P.835 framework uses cleaned enrollment speech which was
enhanced using a non-causal model. Cleaned enrollment speech helps human rater
perform subjective evaluation more consistently.; (v) Personalized and non-
personalized models for a track were sent to same subjective evaluation and
ranked together, i.e. personalized and non-personalized models are treated
alike and compared against each other. (vi) Track-1 and Track-2 has their
separate subjective evaluations to ensure we have headset testclips evaluated
together, and speakerphone testclips evaluated together.

Most of the models submitted to challenge were personalized models. Winning
teams in both tracks have similar ranking in each track. We invited challenge
participants to submit their Preference-2 models which were not sent for
subjective evaluation. These enhanced clips are expected to be used in semi-
supervised training of personalized DNSMOS P.835.

##  2 Challenge Tracks

Enhancing speech quality (i.e., eliminating speech distortion) and suppressing
noise, reverberation and neighboring talkers are two trade-offs DSE models
must handle. DSE models submitted to challenge were supposed to do joint
denoising and dereverberation in presence of neighboring (interfering)
talkers. The goal is to enhance audio signal and preserve the primary talker
while suppressing the neighboring talkers, noise, and reverberation. All
datasets used in this challenge were full band (48 kHz). It aimed to study
headset and speakerphone DSE in separate tracks thereby creating possibility
of new insights. Acoustic properties of headset scenarios can be leveraged in
developing models for suppressing neighboring talkers without enrollment
speech. Non-personalized model not requiring enrollment speech are suitable
for consumers/industry with strict privacy requirements.

This challenge has two tracks: Track-1 Headset; Track-2 Speakerphone. Dev and
blind testset for both tracks were different. The testsets were collected
using similar procedure except that the Track-1 testsets were collected using
Headset devices while Track-2 testsets were collected using Speakerphone
devices. Each test clip in both tracks has enrollment speech with 30s
duration. The enrollment speech can be noise-free or noisy and with or without
reverberation. This facilitates multi-condition enrollment of primary talkers
which serves as a measure of robustness for personalized models which use
enrollment speech as additional input for enhancing the testclips.
Participants could choose to work on models with speaker enrollment or without
it for one or both tracks. Each team was asked to submit 1-4 models depending
on what models they trained. Each participating team could submit a maximum of
one personalized and one non-personalized model for each track, e.g., a team
can submit one personalized and one non-personalized model for Track 1 but not
two personalized or two non-personalized model for Track 1. Similarly, another
team could submit 4 models, personalized and non-personalized model for Track
1 and personalized and non-personalized model for Track 2. This rule
facilitates almost equal representation of personalized and non-personalized
models, and also similar participation in both tracks.

While submitting the blind set enhanced clips, participants were asked to
submit their preference for models submitted in each Track, e.g. they had to
assign Preference-1 to their best model and Preference-2 to their second best
model in each track. From the subjective evaluation of dev testsets, we
noticed that subjective results were taking several days of time to get back
the results. Also, very large batches of enhanced testclips may also results
in some inconsistency in evaluation results. We used Preference-1 clips from
all teams in both tracks for final evaluation based on enhanced blind set.
Preference-2 enhanced clips could be leveraged in semi-supervised training of
Personalized DNSMOS (PDNSMOS) P.835 model or in another subjective evaluation
for generating labels for training of PDNSMOS P.835. All models for a track
were evaluated and ranked together i.e., both personalized and non-
personalized models for Track 1 went through one subjective evaluation.
Similarly, for Track 2, all models in one subjective evaluation. Participants
were encouraged to conduct experiments with both personalized and non-
personalized models to elucidate the benefits of personalization. Though, this
was not a requirement for this challenge.

##  3 Challenge Datasets

###  3.1 Training Data

We used a machine learning model for extracting the near-field headset-like
clean speech which is released as clean speech for headset track. We used
entire clean speech from 4th DNS Challenge as clean speech for speakerphone
track. Noise dataset and impulse responses are same as in 4th DNS Challenge
[1]. Participants can re-use near-field clean speech in Track 1 by convolving
it with impulse responses for generating the training data for Track 2. The
provided datasets include different languages with talker and device variety.
We also provide speaker ID information for all clean speech clips to
facilitate development of personalized models for both tracks. We also provide
code for extracting speaker embeddings based on state-of-the-art ECAPA-TDNN
embeddings trained on Voxceleb [9, 10, 11]. Along with clean speech, we added
clean speech with emotions such as crying, yelling, laughter, or singing to
training data. Entire training set consists of clean speech clips in English
and 10 non-English languages.

Clean speech in the Track-2 training set is a total of 760.53 hours: read
speech (562.72 hours), singing voice (8.80 hours), emotional speech (3.6
hours), Chinese Mandarin (185.41 hours). Track-1 clean speech is obtained by
doing near-end speech extraction on Track-2 dataset. Clean speech has four
subsets: (i) Read speech recorded in clean conditions; (ii) Singing clean
speech; (iii) Emotional clean speech; and (iv) non-English clean speech.

Noise data included in the training set is chosen from AudioSet [12] and is
identical to noise set in 4th DNS Challenge [1]. AudioSet is a collection of
about 2 million human-labeled 10s sound clips extracted from YouTube videos.
There are over a million clips in AudioSet with audio classes music and speech
and less than 200 clips for classes such as toothbrush, creak, etc.
Approximately 42% of the clips have a single class, but the rest may have 2 to
15 labels. Hence, we developed a sampling approach to balance the dataset in
such a way that each class has at least 500 clips. We also used a speech
activity detector to remove the clips with any kind of speech activity, to
strictly separate speech and noise data. Audioset clips were made available at
44.1kHz, we upsampled those to fullband (48 kHz). The resulting noise dataset
has 152 audio classes and 60,000 clips [1]. In total, there are 181 hours of
noise data in the training set.

###  3.2 Development Testset

Both test sets consist of fullband audio clips recorded in real-world
scenarios collected through crowd-sourcing where workers read provided text
prompts and record their voice using desktop/laptop/mobile devices in the
presence of noise and/or neighboring talkers. We include some new noise types
in the test set covering relevant real-world scenarios, device variety and
added paralinguistic test set as new category. Our dev test set consists of
real-world test clips recorded by crowd-sourced workers.

The development test set for the non-personalized track consists of 600 real
recordings. All clips contain noisy speech in the English language. Among
these, 193 test clips have emotional speech in the presence of noise. There
are six emotion types, namely happy, sad, angry, yelling, crying, and
laughter. Crowd-sourced workers were asked to read provided text prompts and
create emotional events in each test clip. The remaining clips contain the
voice of a talker reading text in the presence of the following noise types:
fan, air conditioner, typing, door shutting, clatter noise, car noise (i.e.,
standing near a car on a busy street or standing outside the car), kitchen
noise (noise from kitchen utensils, dish scrubbing etc.), dish washer, running
water, opening chips bags, munching or eating, creaking chair, heavy
breathing, copy machine, baby crying, dog barking, inside-car noise (e.g.,
sitting on a passenger seat in a car which is being driven by someone else),
mouse clicks, mouse scroll wheel, touch pad clicks, etc. Each test clip was
recorded at 48 kHz with a duration of 10 to 20 seconds. Workers were asked to
record in the near-field (close-talk) and far-field with distances of 1, 2,
and 3 meters. All test clips in the non-personalized development test set were
recorded using a laptop or desktop computer. Both development and blind test
set have 2.5 minutes of enrollment speech for primary talkers to be used in
personalized denoising. PDNS leverages speaker embedding (features) for
preserving only the primary talker in a noisy environment while suppressing
the neighboring talkers and noise. The development test set for the
personalized track consists of 1443 real recordings. All clips contain noisy
speech in the English language. Among these, 193 test clips have emotional
speech in the presence of noise and are identical to the emotional test clips
in the non-personalized track. There are 737 test clips where the primary
talker reads the provided text in the presence of the same noise types as
those in the non-personalized track. Each test clip was recorded at 48 kHz
with a duration of 10–20 seconds. Workers were asked to record in the near-
field (close-talk) and far-field with distances of 1, 2, and 3 meters. There
are 166 test clips with the primary talker speaking in the presence of a
neighboring talker and noise where both the noise and neighboring talker are
simultaneously active in the primary talker’s background. There are 347 test
clips where the primary talker is speaking in the presence of a neighboring
speaker with no background noise. Thus, we have simulated three scenarios for
PDNS: (i) primary talker in the presence of noise; (ii) primary talker in the
presence of neighboring talker; and (iii) primary talker in the presence of
simultaneously active neighboring talker and noise. All test clips in the
personalized development test set were recorded using a laptop or desktop
computer.

###  3.3 Blind Testset

We include some new noise types in the test set covering relevant real-world
scenarios, device variety and added paralinguistic test set as new category.
Blind testset consists of real testclips which were not previously used in any
challenge and not otherwise available publically. Our test set consists of
real-world test clips recorded by crowd-sourced workers. Some of the devices
used for collecting blind set includes [list of devices]. Blind set clips were
chosen to confirm blind set specifications which were also shared with crowd-
sourced workers. We performed rigorous quality assurance (QA) to ensure blind
set is representative of real-world scenarios in terms of speaker variety,
device variety, variety in acoustic scenarios, different direct-to-reverb-
ratio (DRR), different T60 which is achieved by changing the relative and
absolute position of primary and interfering talkers, noise source and
presence of reflecting surfaces etc.

Unlike previous challenges, Blind testset in this challenge contains
paralinguistic test clips. These contains standard forms of paralanguage [13]
including but not limited to The throat-clear, ”hmm” or ”mhm”, ”Huh?” or
”what?, Gasps, Sighs, Moans and groans, Deceptive speech, Sincere speech,
Speech with high-base, Speech with high-pitch, Speech with low-pitch,
Confident speech, Tired speech (when talker is tired), Persuasive speech,
Voice change mid-clip (i.e. mimicry in last 50% of the clip).

Blind testset will also include emotional speech including but not limited to
happy, sad, angry, yelling, crying, and laughter. Blind testset include real
test clips with high reverberation, high reverberation with noise, and noise
in presence of interfering talkers. Testset noises include but not limited to
Office scenarios (Typing, AC, Door shutting, Eating/munching, Copy machine,
Squeaking chair, Notification sounds etc.), Home scenarios (Baby crying, Dogs,
TV, Radiators, hair dryer, kitchen noise, running water etc.), Appliances
(Washer Dryer, Dishwasher, Coffee maker, kitchen noise, Vacuum cleaner etc.),
Fire alarm, Car, Inside parked car on busy road, In-car neighboring talkers,
Traffic Road, Car noise (from machinery, control systems, turn signal etc.),
Café, Coffee machine, Blender, Background babble, Airport Announcements etc.

Fig. 1: Results: Personalized P.835 subjective evaluation, Word Accuracy
(WAcc) and Challenge metric (Score) computed on blind testset for all teams in
(a) Track-1 Headset; (b) Track-2 Speakerphone.

##  4 Evaluation Setup

###  4.1 Baseline Models

Along with training datasets and testsets, we also provide a baseline model
(or enhanced clips) for both tracks.

Baseline models were personalized and non-personalized variants of models
presented in [14].

###  4.2 Subjective Evaluation

####  4.2.1 Cleaning Enrollment Clips

We cleaned (enhanced) enrollment clips with a non-causal model based on E3Net
architecture [15]. The subjective evaluation uses only 5s of enrollment
speech. We manually choose a 5s segment either from the enhanced enrollment
clip ensure all long pause (¿0.2s) are removed and resulting 5s audio is
normalized to ensure easier recognition of primary talker by human raters. The
non-causal DSE model is a fullband time domain domain based on end-to-end
enhancement network (E3Net) which was originally proposed for the task of
personalized speech enhancement using wideband signals [15].

Non-causal model is non-personalized and is trained to remove background noise
and reverberation for fullband speech. This model uses a learnable encoder and
decoder instead of STFT and iSTFT which can mitigate the problem of imperfect
phase reconstruction that exists in most time-frequency based speech
enhancement methods. This model is based on 20ms frames with a stride of 10ms.
The core block in non-causal model is a bidirectional LSTM which offers
improved performance for offline processing which is suitable for enhancing
enrollment clips for subjective evaluations.

This model was trained on training clean speech and noise from he 4th DNS
challenge [16]. Impulse responses used for training this model consists of
150,000 simulated impulse responses at sampling rate 48kHz using the Image
Source method [17]. The reverberant speech and noise signals were mixed with a
signal-to-noise ratio (SNR) drawn from a Gaussian distribution with N(-5,20)
dB. All training samples were 10s segments. Training data consists of
synthesized 1000 hours of fullband data. For increasing the robustness of the
model with respect to audios with different bandwidths, namely narrowband and
wideband audios, we also downsampled half of the 1000 hours training set to
16kHz and 8kHz respectively and upsampled them back to 48 kHz and combined
these with the training set. Thus, the final training dataset has 3000 hours
of data.

####  4.2.2 Personalized P.835 Framework

This challenge relies on ITU-T P.835 [7] subjective evaluation framework. A
modified version (Personalized) ITU-T P.835 was used for measuring the
performance of personalized DSE models. Personalized P.835 framework uses 5s
of clean enrollment speech for primary talkers to help human raters recognize
the primary talker’s voice while assigning subjective scores. Human raters
were instructed to focus on the quality of the voice of the primary talker
when two or more talkers were present in a test clip. In addition,
personalized P.835 subjective framework is improved to include voice
recognition, better spam filtering and more accurate evaluation of enhanced
testclips with noise and neighboring talkers.

In all test clips, we have only one primary talker in enrollment clip while
noisy test clip may have noise, reverberation, one or more neighboring talkers
in addition to a primary talker. The goal of DSE models is to preserve primary
talker’s speech while suppressing everything else.

Crowd-sourced workers doing subjective evaluation are instructed to rate
interfering talkers as undesirable signal so the model which suppresses
interfering talkers is rated higher. Personalized P.835 subjective framework
provides three scores, namely speech quality (SIG), background noise quality
(BAK), and overall audio quality (OVRL). We evaluated all challenge models
through our Personalized P.835 framework to get the subjective ratings used
for computing the challenge metric.

###  4.3 Word Accuracy

We computed word accuracy (WAcc) from a state-of-the-art speech recognition
system. We did WAcc computation ourselves during last week of the challenge to
ensure all models are evaluated in exactly same way. WAcc is an objective
metric for measuring the impact of speech enhancement on speech recognition
transcription service. WAcc is defined as
$\text{WAcc}=1-\text{WER}$,

where WER is the word error rate of speech recognition system. We transcribed
the entire blind set for both tracks to obtained the ground-truth. Dev testset
was not transcribed. We computed WAcc results internally. Unlike subjective
P.835 framework which uses only 7s of manually chosen segment from noisy or
enhanced clips, WAcc engine uses entire testclips. Blind set test clips are of
different duration from 10s to more than 6 minutes.

Blind testset was collected from crowd-sourced data collection using various
recording apps where the worker was provided with a text prompts to read from.
Since there are often reading errors, omissions of words etc., prompts do not
reflect the correction transcription. We used a five step approach to obtain
ground-truth transcription for blind testset. In first step, we just obtain
the prompts for each testclips in blind set. In second step, we obtained the
transcripts from the state-of-the-art speech recognition engine for each
testclips in blind set. In third step, expert human listeners listened each
testclips and generated the corresponding human transcripts. Expert listeners
were advised to listen audio clips several times until they were confident
about their transcription. In fourth step, we compute word error rate (WER)
for each testclip in blind set. Then, the testclips with WER ¿ 0.5 were chosen
for fifth round of listening. In fifth round of listening, human listeners
listen the clips with WER ¿ 0.5 and validated/corrected the human
transcriptions. We found that a very few clips were needed to be corrected at
fifth stage this elucidates the robustness of our transcription approach.

We found that several clips in blind set results had WER ¿ 0.5, it is often
due to insertion error where DSE models leaked the secondary talker’s voice
and those speech were transcribed by speech recognition engine. During five
step approach for generating ground-truth transcription, we found out that
there were several clips where neighboring talker was sounding very similar to
primary talker (e.g., same accent, same gender, similar loudness and sitting
close to each other), some of those clips took more than 10 times listening to
correctly transcribe, some of those were not possible to transcribe and hence
skipped. Thus, our WAcc was computed over entire duration of selected clips
with ground-truth transcripts. It is important to note that speech recognition
engine is not personalized and does not use any enrollment speech to focus on
primary talker. In future, it would be interesting to develop a personalized
speech recognition engine. WAcc ground-truth transcripts only contained words
spoken by the primary talker, thus treating interfering talker as undesirable
signal.

###  4.4 Challenge Metric

Like past challenges, models in both tracks were ranked in terms of a final
score obtained by weighted average of subjective P.835 scores and word
accuracy (WAcc). Four metrics, three personalized P.835 subjective scores
namely SIG, BAK, OVRL and WAcc from a speech recognition system were used to
evaluate challenge models. Metrics on the blind test set were combined into a
final score for ranking the models. Higher WAcc shows superior speech
enhancement performance with respect to speech recognition. Higher P.835
scores shows better subjective speech quality. The final score is computed as
$\text{Final score}=0.5[\text{WAcc}+0.25(\text{OVRL}-1)]$.

We evaluated the submitted models based on Personalized ITU-T P.835 subjective
evaluation scores, namely speech quality (SIG), background noise quality
(BAK), and overall audio quality (OVRL), and WAcc from a state-of-the-art
speech recognition system. WAcc is an objective metric for measuring the
impact of speech enhancement on speech recognition transcription service.

##  5 Results & Discussions

There were 11 submissions in Track-1 and 11 submissions in Track-2 out of
which 10 teams participated in both tracks. Almost all models in both tracks
were personalized models. We had two baseline models for Track-1 out of which
one was non-personalized model. We had personalized model as baseline for
Track-2. By dint of acoustic properties of primary speech in a headset
scenarios, a non-personalized model may be able to suppress neighboring
talkers and noise, hence we included a non-personalized model. Each team was
asked to provide enhanced clips from their models. Teams were allowed to
submit maximum of two models (at most one personalized and at most one non-
personalized) per track. Since subjective evaluations may take very long-time
if number of submitted models is large, we ask participants to provide their
Preference of models. We chose the first Preference model from each team in
their respective tracks for subjective evaluations. We also conducted a dev
testset subjective evaluation based on enhanced dev set and provided the
results to participants.

Fig. 1 show the subjective personalized P.835 scores, WAcc and challenge
metric (Score) for all teams sorted in decreasing order of performance. dMOS
for SIG, BAK, OVRL refers to the difference in SIG, BAK, OVRL between the
enhanced clip and corresponding noisy clip. Similarly, dWAcc is the difference
in WAcc between the enhanced clip and noisy clip. We conducted ANOVA test on
top models in each track to determine statistical significance (see
https://aka.ms/5th-dns-challenge) for ANOVA results.

##  6 Conclusions

This challenge has a more diverse blind test set collected through
crowdsourcing using multiple data vendors. We included paralinguistic
testclips and leakage testclips in blind set. We enhanced and chose the
enrollment speech segment for primary talkers, and noisy segment from
testclips to ensure robust subjective evaluations. We verified if winning
models are causal with helps of our $NRT$ testset. Results show degradation in
signal quality (SIG) for most of the models. It may be due to two reasons: (i)
testset is significantly challenging; (ii) Personalized model end up
suppressing the primary talker. Noticeable SIG degradation’s happen due to
suppression of primary talker’s speech and/or leakage of interfering talker
and noise. Detailed analysis of results from current and previous DNS
challenges will be covered in our extended OJSP journal paper. We encourage
all the challenge participants and readers to send us their feedback and
contribute to DNS Challenge repository.

## References

  * [1] Harishchandra Dubey, Vishak Gopal, Ross Cutler, Ashkan Aazami, Sergiy Matusevych, Sebastian Braun, Sefik Emre Eskimez, Manthan Thakker, Takuya Yoshioka, Hannes Gamper, et al.,  “ICASSP 2022 deep noise suppression challenge,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 9271–9275.
  * [2] Hyeong-Seok Choi, Hoon Heo, Jie Hwan Lee, and Kyogu Lee,  “Phase-aware single-stage speech denoising and dereverberation with U-net,”  arXiv preprint arXiv:2006.00687, 2020.
  * [3] Chandan KA Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al.,  “The interspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results,”  arXiv preprint arXiv:2005.13981, 2020.
  * [4] Chandan KA Reddy, Harishchandra Dubey, Kazuhito Koishida, Arun Nair, Vishak Gopal, Ross Cutler, Sebastian Braun, Hannes Gamper, Robert Aichner, and Sriram Srinivasan,  “INTERSPEECH 2021 Deep Noise Suppression Challenge,”  ISCA INTERSPEECH, 2021.
  * [5] Chandan K. A. Reddy, Harishchandra Dubey, Vishak Gopal, Ross Cutler, Sebastian Braun, Hannes Gamper, Robert Aichner, and Sriram Srinivasan,  “ICASSP 2021 Deep Noise Suppression Challenge,”  in IEEE ICASSP, 2021, pp. 6623–6627.
  * [6] Yangyang Xia, Sebastian Braun, Chandan KA Reddy, Harishchandra Dubey, Ross Cutler, and Ivan Tashev,  “Weighted speech distortion losses for neural-network-based real-time speech enhancement,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 871–875.
  * [7] Babak Naderi and Ross Cutler,  “Subjective evaluation of noise suppression algorithms in crowdsourcing,”  in ISCA INTERSPEECH, 2021.
  * [8] Chandan KA Reddy, Vishak Gopal, and Ross Cutler,  “Dnsmos p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 886–890.
  * [9] Nauman Dawalatabad, Mirco Ravanelli, François Grondin, Jenthe Thienpondt, Brecht Desplanques, and Hwidong Na,  “Ecapa-tdnn embeddings for speaker diarization,”  arXiv preprint arXiv:2104.01466, 2021.
  * [10] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck,  “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,”  arXiv preprint arXiv:2005.07143, 2020.
  * [11] “DNSMOS Git Repo,” https://huggingface.co/speechbrain/spkrec-ecapa-voxceleb,  [Online; accessed 2022-09-01].
  * [12] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter,  “Audio set: An ontology and human-labeled dataset for audio events,”  in 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017, pp. 776–780.
  * [13] “Paralanguage,” https://en.wikipedia.org/wiki/Paralanguage,  [Online; accessed 2022-09-01].
  * [14] Sefik Emre Eskimez, Takuya Yoshioka, Alex Ju, Min Tang, Tanel Parnamaa, and Huaming Wang,  “Real-time joint personalized speech enhancement and acoustic echo cancellation with e3net,”  arXiv preprint arXiv:2211.02773, 2022.
  * [15] Manthan Thakker, Sefik Emre Eskimez, Takuya Yoshioka, and Huaming Wang,  “Fast real-time personalized speech enhancement: End-to-end enhancement network (e3net) and knowledge distillation,”  arXiv preprint arXiv:2204.00771, 2022.
  * [16] Harishchandra Dubey, Vishak Gopal, Ross Cutler, Ashkan Aazami, Sergiy Matusevych, Sebastian Braun, Sefik Emre Eskimez, Manthan Thakker, Takuya Yoshioka, Hannes Gamper, and Robert Aichner,  “ICASSP 2022 Deep Noise Suppression Challenge,”  in ICASSP, 2022.
  * [17] Jont B Allen and David A Berkley,  “Image method for efficiently simulating small-room acoustics,”  The Journal of the Acoustical Society of America, vol. 65, no. 4, pp. 943–950, 1979.