<EMAIL_ADDRESS>[]University of Mannheim, L15, 1–6, 68161 Mannheim,
Germany []Data is available on OSF. []The authors declare no competing
interests. Smirnov

# Toxic comments reduce the activity of volunteer editors on Wikipedia

Ivan Smirnov  Camelia Oprea  RWTH Aachen University  Markus Strohmaier
University of Mannheim  GESIS - Leibniz Institute for the Social Sciences
Complexity Science Hub Vienna  0000-0002-9998-0058

###### Abstract

Wikipedia is one of the most successful collaborative projects in history. It
is the largest encyclopedia ever created, with millions of users worldwide
relying on it as the first source of information as well as for fact-checking
and in-depth research. As Wikipedia relies solely on the efforts of its
volunteer-editors, its success might be particularly affected by toxic speech.
In this paper, we analyze all 57 million comments made on user talk pages of
8.5 million editors across the six most active language editions of Wikipedia
to study the potential impact of toxicity on editors’ behaviour. We find that
toxic comments consistently reduce the activity of editors, leading to an
estimated loss of 0.5–2 active days per user in the short term. This amounts
to multiple human-years of lost productivity when considering the number of
active contributors to Wikipedia. The effects of toxic comments are even
greater in the long term, as they significantly increase the risk of editors
leaving the project altogether. Using an agent-based model, we demonstrate
that toxicity attacks on Wikipedia have the potential to impede the progress
of the entire project. Our results underscore the importance of mitigating
toxic speech on collaborative platforms such as Wikipedia to ensure their
continued success.

##  1 Introduction

Wikipedia is arguably one of the most successful collaborative projects in
history. It has become the largest and most-read reference work ever created,
and it is currently the fifth most popular website on the
Internet111https://www.semrush.com/website/top/ [[Assessed on 23.02.2023]].
Millions of users worldwide rely on Wikipedia as their first source of
information when encountering a new topic, for fact-checking and in-depth
research Singer et al. (2017). Even if caution might be required when
consulting less actively maintained pages Bruckman (2022), numerous studies
have shown that Wikipedia is a reliable source of information in areas ranging
from political science Brown (2011) to pharmacology Clauson et al. (2008) and
its accuracy is comparable to traditional encyclopedias Giles (2005) and
textbooks Kräenbring et al. (2014).

One of the most remarkable aspects of Wikipedia’s success is that its content
is exclusively created and curated by volunteer-editors, known as Wikipedians.
The English edition alone has 129,698 active
editors222https://en.wikipedia.org/wiki/List_of_Wikipedias [[Assessed on
23.02.2023]]. However, this volunteer-driven model also makes Wikipedia
susceptible to the inherent challenges associated with maintaining such a
large online community Kraut and Resnick (2012); Keegan and Fiesler (2017).
For example, it has been previously observed that Wikipedia is not free of
conflict, particularly in the form of so-called edit wars Yasseri et al.
(2012), which impose significant costs on the project Kittur et al. (2007) and
could negatively affect the quality of Wikipedia articles Arazy et al. (2011).

In this paper, we focus on the impact of toxic comments directed towards
editors on their activity. This aspect is less studied, but potentially not
less important, as affected by toxic comments, Wikipedians might reduce their
contributions or abandon the project altogether, threatening the success of
the platform Preece (2001).

Toxicity has been extensively studied on popular social media websites such as
Twitter Chatzakou et al. (2017); Guberman et al. (2016), Reddit Xia et al.
(2020); Almerekhi et al. (2020), and similar platforms Wich et al. (2022);
Silva et al. (2016). However, much of these research focuses on automated
toxicity detection and prevalence estimation rather than on evaluating its
impact Kiritchenko et al. (2021). As an online encyclopedia, Wikipedia is
often perceived as immune to toxicity and has a strict “No personal attacks”
policy Wikipedia (2022). Despite that, toxic speech and harassment have been
previously observed on the platform Arazy et al. (2013); Wikimedia Foundation
(2015b); Corple (2016); Wulczyn et al. (2017); Raish (2019). The effects of
such behaviors on editors’ contributions are, however, not well understood nor
well studied. The largest study to date relies on a voluntary opt-in survey of
the 3,845 Wikipedians conducted in 2015 Wikimedia Foundation (2015b). It
reports that 20% of users witnessing harassment have stopped contributing for
a while, 17% considered not contributing anymore and 5% stopped contributing
at all.

In this paper, we analyzed all 57 million comments made on user talk pages of
editors on the six most active language editions of Wikipedia (English,
German, French, Spanish, Italian, Russian) to understand the impact of toxic
speech on editors’ contributions (see Methods & Materials for our definition
of toxic comments). User talk pages are a place for editors to communicate
with each other either on more personal topics or to extend their discussion
from an article’s talk page. The majority of toxic comments are left on user
talk pages Qu et al. (2019). The comments we study were extracted from
revision histories of talk pages and, thus, include even those toxic comments
that were later archived or deleted by the page owner.

Figure 1:  After receiving a toxic comment many users temporarily reduce their
activity or leave the project completely.  The figure shows the activity of 50
randomly selected users who received exactly one toxic comment. Blue squares
indicate an active day, i.e. a day when at least one edit was done, starting
from the first contribution of a given user. Red triangles correspond to toxic
comments. Note that while some users are resilient and their activity is
seemingly unaffected by toxic comments, many users temporarily reduce their
activity or stop contributing altogether.

Figure 1. shows the activity of 50 randomly selected users who have received
exactly one toxic comment. While some users are seemingly unaffected by a
toxic comment, others temporarily reduce their activity or leave the project
completely. The aim of our paper is to quantify this effect on the entire
population of editors.

We estimate the number of lost active days due to a toxic comment by comparing
the number of active days before and after receiving a toxic comment. To
account for potential baseline change, we have matched users that received a
toxic comment with similarly active users who received a non-toxic comment. We
have separately studied if toxic comments increase the probability of users
leaving the project altogether. Finally, we have used an agent-based model to
model the potential impact of an increased number of toxic comments on
Wikipedia.

##  2 Results

Figure 2:  After receiving a toxic comment, editors become less active.  On
average, users are more active near the time when they receive a toxic comment
(peak at zero for the red line in panel a). Average activity across all users
who have received a toxic comment is lower in all $100$ days after the event
compared to the corresponding days before (dashed and solid red lines in panel
b). This cannot be explained by a baseline drop in activity after a non-toxic
comment (dashed and solid blue lines in panel b). Similar results hold not
only for the English edition but also for the other five editions (c-g).

###  2.1 Loss of editor activity

To estimate the effect of a toxic comment, we compute the proportion of users
who were active on day X before or after receiving a toxic comment (Figure 2).
We find that, on average, editors are more active near the time when they
receive a toxic comment, with a peak at $24$ hours prior to the comment. At
this time point, more than $40\%$ of editors were active, as shown by the red
line in Figure 2a. This is a rather unsurprising observation since toxic
comments are often made as a reaction to an edit made by a user and, thus,
users are expected to be active around the time of a toxic comment. Note that
if the timestamps around which the curve is centered are shuffled (black line
in Figure 2a) then this pattern disappears completely as expected.

We also find that average activity across all users who have received a toxic
comment is lower during all 100 days after the event compared to the
corresponding days before (dashed and solid red lines in Figure 2b), e.g.
smaller number of users is active five days after receiving a toxic comment
than five days before receiving it. To rule out the possibility that this is
due to a general drop in activity over time or a drop in activity after any
comment, we select a control group of users who have received a non-toxic
comment, and whose average activity in the $100$ days before the comment is
the same as the average activity of users who received a toxic comment (see
Methods & Materials for details).

We observe a similar characteristic peak around the non-toxic comment, likely
due to both toxic and non-toxic comments being reactions to a contribution
made by an editor. However, in contrast to a toxic comment, a non-toxic
comment does not lead to a significant decrease in activity (dashed and solid
blue lines in Figure 2b). Similar results hold for all six language editions
that we have examined (Figure 2c-g).

We then estimate the lost activity due to a toxic comment by computing the
decrease in activity after a toxic comment, taking into account a potential
baseline drop, i.e. by computing $\Delta=$ (Aftertoxic $-$ Beforetoxic) $-$
(Afternon toxic $-$ Beforenon toxic). We find that this loss is statistically
significant for all language editions studied (Table 1). We further explored
the robustness of this result with respect to the toxicity threshold and
potential filtering of users according to their activity. As expected, for
higher toxicity thresholds, i.e. for more severely toxic comments, the effect
is stronger (Figure S1). Considering only active users also leads to higher
estimates, however, here we are reporting a conservative estimate, i.e. no
filtering is used for results presented in Figure 2 and Table 1.

Table 1: Lost active days in the $100$ days following a toxic comment. The lost active days are estimated by computing the difference between the number of active days during $100$ days after a toxic comment and the number of active days during 100 days before a toxic comment. This difference is then compared with the baseline drop after a non-toxic comment, i.e. $\Delta=$ (Aftertoxic $-$ Beforetoxic) $-$ (Afternon toxic $-$ Beforenon toxic). The $P$-value is computed using Student’s $t$-test Edition | $\Delta$ |  $P$-value | $N_{users}$
---|---|---|---
English | $-1.207$ | $2.6\times 10^{-66}$ | $36,332$
German | $-0.546$ | $1.5\times 10^{-7}$ | $10,346$
French | $-1.851$ | $4.8\times 10^{-9}$ | $2,239$
Spanish | $-0.563$ | $8.6\times 10^{-3}$ | $2,446$
Italian | $-0.336$ | $2.3\times 10^{-2}$ | $3,567$
Russian | $-1.219$ | $7.8\times 10^{-4}$ | $1,134$

Note that given that thousands of users have received at least one toxic
comment (Table S1), even a moderate loss per user could result in many human-
years of lost productivity for Wikipedia in the short run. By multiplying the
estimated loss per user from Table 1 by the number of users who have received
at least one toxic comment, we could estimate the total loss of activity that
is ranging from $5$ human-years for Russian Wikipedia to $265$ human-years for
the English edition. The reason for the lasting effect of toxicity is that
some new users are discouraged by a toxic comment and choose to leave the
project altogether after just a few contributions. This means that a single
toxic comment could deprive Wikipedia of a potentially long-term contributor.

To further investigate this effect we compare the probability of leaving
Wikipedia after receiving a toxic comment with the probability of leaving
Wikipedia after receiving a non-toxic comment.

###  2.2 Leaving Wikipedia

Figure 3:  The probability of leaving Wikipedia after receiving a toxic
comment is substantially higher than might be expected otherwise. For all six
editions the probability of leaving declines with the number of contributions
approximately following the power law. At the same time, this probability is
substantially higher after receiving a toxic comment than might be expected
otherwise. Dots are probability estimates and solid lines are the best linear
fit on a log-log scale.

We observed that the probability of leaving Wikipedia after $N$ contributions
declines with $N$ following a power law. That is $P_{N}(\text{leaving})\sim
N^{-\alpha}$, where alpha ranges from $0.89$ to $1.02$ for different language
editions. While the probability of leaving the project after the first and
only contribution is high ($P_{1}=47\%$ for English Wikipedia), the risk of
leaving Wikipedia drops to $0.7\%$ for users who have made 100 contributions.

To study the effects of toxic comments we separately consider contributions
that are followed by a toxic comment and contributions that are not followed
by a toxic comment (see Methods & Materials for details). We find that the
risk of an editor leaving after a toxic comment is consistently higher for all
editions and regardless of the contribution number, see Figure 3. We provide
an analysis of the significance of these findings in Figure S3.

###  2.3 Agent-based modeling

As has been demonstrated earlier, toxic comments increase the likelihood of
editors abandoning Wikipedia. If enough editors leave, this could potentially
impede the progress of the project as a whole. In order to estimate the
potential impact of toxic comments, we compare two environments: a non-toxic
environment, where the probability of a user leaving follows empirically
observed non-toxic probability distribution (blue dots in Figure 3), and a
highly toxic environment, where the probability of leaving corresponds to an
empirically observed toxic probability distribution (red dots in Figure 3).

To model the impact of toxic comments, we employ an agent-based approach in
which different scenarios are defined by the number of users joining the
project each day. Specifically, we consider the following three scenarios: 1)
$1000$ users join the project on day $1$, with no new users arriving later; 2)
one new user joins the project each day indefinitely; 3) one new user joins
the project each day for the first $500$ days, with no new users joining the
project later.

In our model, each agent makes a contribution upon joining the project and
then continues to contribute following a Poisson process, i.e in such a way
that the distance between two consecutive contributions, $D$, follows an
exponential distribution: $D\sim\operatorname{Exp}(\lambda)$, where $\lambda$
is estimated from empirical data. After each contribution, the editor leaves
the project with the probability determined by the environment (toxic or non-
toxic) and the number of contributions already made, as represented in Figure
3.

Our model suggests that in a non-toxic environment, approximately $200$ users
from the initial population of $1000$ would become long-term contributors
sustaining the project. In contrast, in a toxic environment, the decline in
the number of users would be faster, and almost none of them would remain in
the long run, as illustrated in Figure 4a

Sustaining the project in the long run in a toxic environment would require a
constant influx of new users, Figure 4b. The same rate of new users in a non-
toxic environment would lead to the growth of the project. Figure 4c
illustrates a combination of the first two scenarios. The model demonstrates
that toxic comments not only decrease the number of users but can also lead to
qualitatively different outcomes for the project, e.g. stable vs dying out.
That might be particularly important given that the actual activity on
Wikipedia declines (see Figure S4).

Figure 4:  Comparison of the number of users over time in a non-toxic vs. a
toxic environment.  In a non-toxic environment, the initial population of
$1000$ users (a) stabilizes at around $200$, whereas in a toxic environment,
the population nearly dies out. If users are constantly arriving to sustain
the population in a toxic environment (b), it would lead to growth in a non-
toxic environment. When users arrive only up to a certain point in time (c),
the peak activity of the project is substantially lower in a toxic environment
compared to a non-toxic environment and activity eventually dies out.

##  3 Discussion

Toxic speech is a general threat to all digital platforms and communities
Vogels (2021), however, it is especially dangerous to Wikipedia which relies
on the contributions of volunteer editors to create articles with broad,
global exposure. To ensure objectivity and comprehensiveness of articles, it
is critical to establish and maintain an open and inclusive collaborative
environment. However, this is not always the case on talk pages, where
clashing editor opinions can lead to toxic and harmful comments threatening
productivity and diversity of editors Lir (2021); Bear and Collier (2016);
Lenhart et al. (2016). It could also lead to the idea that harassment is a
possible tool for silencing users.

Despite that, the opinion that toxic comments are negligible and should be
seen as merely over-enthusiastic participation is still present among editors
Corple (2016). Furthermore, various anti-harassment measures have been
declined multiple times by the community, as they were seen to slow the
process of content creation Wikimedia Foundation (2015a, 2021). While multiple
surveys were conducted, there existed no current analysis of toxicity impact
at scale to determine whether Wikipedia should take toxic language on
Wikipedia more seriously and adapt new measures to prevent further
occurrences. Our large scale analysis covering all comments made on the six
most active language edition of Wikipedia over the course of 20 years
demonstrates that the question of toxicity should not be disregarded as it
could lead to substantial loss of productivity among editors. This is
especially relevant since, after a period of exponential growth Almeida et al.
(2007), the activity of Wikipedia editors has slowed Suh et al. (2009) and
then declined (Figure S4), with the exception of a COVID-19 related spike in
activity Ruprechter et al. (2021).

Wikipedia plays a crucial role in the global information infrastructure aiming
to provide millions of people with access to free, unbiased knowledge. With
its reputation as a neutral and comprehensive information source, it has
become a trusted first choice source of knowledge for many and its articles
frequently appear in top search engine results. While it is often assumed that
Wikipedia is free of toxicity, our findings demonstrate the toxicity is not
only present but its impact is substantial. These findings emphasize the
importance of mitigating toxic speech on collaborative platforms such as
Wikipedia to ensure their continued success.

##  4 Methods & Materials

###  4.1 Data and Preprocessing

####  4.1.1 Comments on User’s Talk Pages

The Wikimedia Foundation provides publicly accessible dumps of all the
different wikis’ content333https://meta.wikimedia.org/wiki/Data_dumps
[[Accessed on 20.01.2023]]. These dumps are updated on a regular basis, with
complete revision history dumps generated once per month. For this paper, we
used the English dump from 01.11.2021, the German dump from 01.08.2022, the
French, Italian, and Spanish dumps from 01.08.2022, and the Russian dump from
01.07.2022. The data was obtained from a mirror hosted by the Umeå University,
Sweden444https://mirror.accum.se/mirror/wikimedia.org/.

From the dumps, the user’s talk pages were extracted. A user’s talk page is a
place where other editors can communicate with the user either on more
personal topics or to extend their discussion from an article’s talk page.
When the comments left on the talk page are resolved or become too old, users
can choose to archive them. This helps them keep better track of new incoming
topics. Once archived, the old comments are not displayed on the talk page
anymore but are rather linked in a subpage. Nevertheless, the entire history
of the user talk page, as of any other page on Wikipedia, can be fully seen
under the tab of revision history. The revision history records one entry for
every edit made on the page saving each time the complete content of the page.
Thus retrieving a single comment requires performing the difference between
two consecutive revisions. The Wikimedia API does offer a method to compute
the difference between two revisions, however, applying it on a scale that was
necessary for this research was unfeasible. For that reason, we developed our
own parser to extract comments as a difference between two versions of the
page Smirnov and Oprea (2023).

We excluded from our analysis talk pages that belong to unregistered users,
e.g. users who are represented only by an IP address rather than a user name,
because IP addresses are dynamic and it can not be assumed that one address
represents a single user throughout Wikipedia history. Additionally, we have
excluded comments made by officially registered bots. Comments that were made
by users on their own pages are also not considered.

When extracting comments, we cleared wiki-specific formatting and HTML markup,
i.e. removed links, attachments, or other formatting-specific sequences
irrelevant to the actual content.

####  4.1.2 Contributions and active days

In order to extract information on users’ contributions, i.e. edits of
Wikipedia pages made by them, we used the MediaWiki API to retrieve timestamps
for each edit made by a given user. The resulting data set is publicly
available in the project repository Smirnov and Oprea (2023). The timestamps
of contributions were then converted into active days. Specifically, each user
$i$ was represented as a binary vector $u_{i}=(a_{i1},a_{i2},\dots,a_{iN})$,
where $a_{id}=1$ if user $i$ made at least one contribution, i.e. edited a
Wikipedia page, within the 24h period corresponding to day $d$ and $a_{id}=0$
otherwise. $N$ is the number of days between the first recorded contribution
in our data set and the last. The conversion from contribution count to active
days was performed because it is hard to interpret and compare the total
number of contributions between users as one large contribution could be
equivalent to multiple smaller ones. Additionally, the size of a contribution
does not necessarily reflect the effort put into it. While being active on a
given day could still mean different levels of activity for different users,
it represents a certain level of engagement with the project and is
substantially different from not contributing at all on a given day.

###  4.2 Toxicity

The automatic detection of offensive language in online communities has been
an active area of research since at least 2010 Xu and Zhu (2010). Over the
past decade, researchers have focused on detecting closely-related and
intersecting types of offensive language such as toxicity, abusive language,
and hate speech Fortuna et al. (2020), see Zampieri et al. (2020) for an
overview of recent advancements in the field. In this paper, we use a model
from the Perspective API Perspective API (2022) to identify toxic comments.
This is a state-of-the-art toxicity detection algorithm that obtained
competitive results at OffensEval-2019 competition Zampieri et al. (2019)
without any additional training on the contest data and is often used as a
baseline system for toxicity detection Fortuna et al. (2020). Perspective API
is used across multiple platforms, including The New York Times, Der Spiegel,
Le Monde, and El País. It is a BERT-based model Lees et al. (2022) trained on
comments from a variety of online sources, including Wikipedia. Each comment
is labeled by 3-10 crowdsourced raters. Perspective models provide scores for
several different attributes, see Table S2 for the list of attributes and
their definitions, see Table S4 for examples of toxic comments, and see Table
S3 for the AUC scores for those languages and attributes that were used in
this paper.

We define a toxic comment as a comment that has a score of at least $0.8$ on
any of the six dimensions provided by Perspective API. The $0.8$ score means
that on average $8$ out of $10$ raters would mark it as toxic. As this
threshold can be considered arbitrary, we perform additional robustness checks
using different toxicity thresholds. In particular, we compute activity loss
not only for the threshold of $0.8$ (Table 1) but for thresholds from $0.2$ to
$0.9$. Additionally, we applied different activity filters, e.g. we separately
compute an estimate only for those users who were active at least $X$ days in
the past $100$ days where $X$ varies from $0$ to $50$. This is done in order
to ensure that the results are not exclusively driven by those users who had
made few edits and then stopped contributing to the project. We perform this
analysis for English Wikipedia as it is the largest edition. As shown in
Figure S1, the estimate is typically in the range from $-0.5$ to $-2$ and
significantly lower than zero for all activity thresholds and all toxicity
thresholds higher than $0.3$. Similarly, we have checked how the toxicity
threshold affects the probability of leaving the project. As might be
expected, results remain qualitatively the same for different toxicity
thresholds but higher thresholds lead to more extreme results, e.g. the
probability of leaving after a toxic comment with $0.9$ score is even higher
than after a toxic comment with toxicity score of $0.8$ (Figure S2).

Perspective API accepts texts up to $20,480$ bytes. As the majority of
comments are well below this limit, we have excluded those that are larger.

###  4.3 Activity loss

For each user, we select a random toxic comment that they has received. We
then center the user activity around the timestamp, $t_{i}^{tox}$, of that
toxic comment, and convert the result to active days, by computing

$\operatorname{sign}(|\\{t\in
T_{i}:t\in[t_{i}^{tox}+d*24*60*60,t_{i}^{tox}+(d+1)*24*60)\\}|)$

, where $T_{i}$ is the set of timestamps of all contributions made by user
$i$, and $d$ is a day ranging from $-100$ to $100$. Finally, the results were
averaged over all users. We repeat the procedure of selecting a random toxic
comment 100 times and report average results. However, as most users received
only one toxic comment, there is little variation across simulations and the
average over 100 simulations is almost identical to the results of a single
simulation.

For comparison, the same procedure is repeated for randomly selected non-toxic
comments, however, it includes an additional step as our control group should
include users who are as active as those who have received a toxic comment. On
average, users who have received a toxic comment are more active. This is
probably due to the fact that each contribution could lead to a toxic response
with a certain probability. Thus, the more contributions a user has made the
higher the chance of receiving a toxic comment. To account for that, for each
user from a non-toxic group we generate with a probability $p$ a virtual toxic
comment after each contribution. We then select for a control group not all
users but only those who have received at least one virtual toxic comment. The
$p$ is selected in such a way that the average activity in $100$ days before a
toxic or non-toxic comment matches.

To test for the significance of the results we compute 95% bootstrapped
confidence intervals for each estimate.

###  4.4 Probability of leaving

For each toxic comment, we find the closest in time contribution that precedes
that comment. We define such contributions as "contributions followed by a
toxic comment" and compare the probability of leaving after such contributions
with the probability of leaving after other contributions. The probability of
leaving after $N$ contributions is estimated as a fraction of users who have
made exactly $N$ contributions among users who have made at least $N$
contributions. As the probability of leaving strongly depends on $N$, we make
a comparison separately for each contribution number $N\in[1,100]$. For
$N>100$ the number of users is too small to provide reliable estimates for
comparison.

###  4.5 Acknowledgment

We acknowledge the Master’s thesis by Sebastian Brückner Brückner (2021),
which identified a potential pattern in data and provided an inspiration for
the design of the study presented in this paper.

The initial data collection and experiments were carried out as part of
Camelia Oprea’s Master’s thesis Oprea (2022).

We thank Liubov Tupikina and David Garcia for their valuable discussions
regarding the results presented in this article.

This preprint was created using the LaPreprint template
(https://github.com/roaldarbol/lapreprint) by Mikkel Roald-Arbøl .

## References

  * Almeida et al. (2007) Almeida RB, Mozafari B, Cho J.  On the Evolution of Wikipedia.  In: _Proceedings of the International Conference on Web and Social Media (ICWSM)_ ; 2007. .
  * Almerekhi et al. (2020) Almerekhi H, Jansen BJ, Kwak H.  Investigating toxicity across multiple Reddit communities, users, and moderators.  In: _Companion Proceedings of the Web Conference_ ; 2020. p. 294–298.
  * Arazy et al. (2011) Arazy O, Nov O, Patterson R, Yeo L.  Information quality in Wikipedia: The effects of group composition and task conflict.  Journal of management information systems. 2011; 27(4):71–98.
  * Arazy et al. (2013) Arazy O, Yeo L, Nov O.  Stay on the Wikipedia task: When task-related disagreements slip into personal and procedural conflicts.  Journal of the American Society for Information Science and Technology. 2013; 64(8):1634–1648.
  * Bear and Collier (2016) Bear JB, Collier B.  Where are the women in Wikipedia? Understanding the different psychological experiences of men and women in Wikipedia.  Sex Roles. 2016; 74:254–265.
  * Brown (2011) Brown AR.  Wikipedia as a data source for political scientists: Accuracy and completeness of coverage.  PS: Political Science & Politics. 2011; 44(2):339–343.
  * Bruckman (2022) Bruckman AS.  Should you believe Wikipedia? Online communities and the construction of knowledge.  Cambridge University Press; 2022.
  * Brückner (2021) Brückner S, Modeling sociodemographic attributes of Wikipedia editors.  RWTH Aachen University; 2021.
  * Chatzakou et al. (2017) Chatzakou D, Kourtellis N, Blackburn J, De Cristofaro E, Stringhini G, Vakali A.  Measuring# GamerGate: A tale of hate, sexism, and bullying.  In: _Proceedings of the 26th International Conference on World Wide Web_ ; 2017. p. 1285–1290.
  * Clauson et al. (2008) Clauson KA, Polen HH, Boulos MNK, Dzenowagis JH.  Scope, completeness, and accuracy of drug information in Wikipedia.  Annals of Pharmacotherapy. 2008; 42(12):1814–1821.
  * Corple (2016) Corple DJ, Beyond the Gender Gap: Understanding Women’s Participation in Wikipedia.  Purdue University; 2016.
  * Fortuna et al. (2020) Fortuna P, Soler J, Wanner L.  Toxic, hateful, offensive or abusive? what are we really classifying? an empirical analysis of hate speech datasets.  In: _Proceedings of the 12th language resources and evaluation conference_ ; 2020. p. 6786–6794.
  * Giles (2005) Giles J.  Internet encyclopaedias go head to head.  Nature. 2005; 438(15):900–901.
  * Guberman et al. (2016) Guberman J, Schmitz C, Hemphill L.  Quantifying toxicity and verbal violence on Twitter.  In: _Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing_ ; 2016. p. 277–280.
  * Keegan and Fiesler (2017) Keegan B, Fiesler C.  The evolution and consequences of peer producing Wikipedia’s rules.  In: _Proceedings of the International AAAI Conference on Web and Social Media_ , vol. 11; 2017. p. 112–121.
  * Kiritchenko et al. (2021) Kiritchenko S, Nejadgholi I, Fraser KC.  Confronting abusive language online: A survey from the ethical and human rights perspective.  Journal of Artificial Intelligence Research. 2021; 71:431–478.
  * Kittur et al. (2007) Kittur A, Suh B, Pendleton BA, Chi EH.  He says, she says: conflict and coordination in Wikipedia.  In: _Proceedings of the SIGCHI conference on Human factors in computing systems_ ; 2007. p. 453–462.
  * Kräenbring et al. (2014) Kräenbring J, Monzon Penza T, Gutmann J, Muehlich S, Zolk O, Wojnowski L, Maas R, Engelhardt S, Sarikas A.  Accuracy and completeness of drug information in Wikipedia: a comparison with standard textbooks of pharmacology.  PLOS One. 2014; 9(9):e106930.
  * Kraut and Resnick (2012) Kraut RE, Resnick P.  Building successful online communities: Evidence-based social design.  Mit Press; 2012.
  * Lees et al. (2022) Lees A, Tran VQ, Tay Y, Sorensen J, Gupta J, Metzler D, Vasserman L.  A new generation of Perspective API: Efficient multilingual character-level transformers.  arXiv preprint arXiv:220211176. 2022; .
  * Lenhart et al. (2016) Lenhart A, Ybarra M, Zickuhr K, Prive-Feeney M, Online Harassment, Digital Abuse, and Cyberstalking in America.  Data & Society Institute; 2016.
  * Lir (2021) Lir SA.  Strangers in a seemingly open-to-all website: the gender bias in Wikipedia.  Equality, Diversity and Inclusion: An International Journal. 2021; 40(7):801–818.
  * Oprea (2022) Oprea C, Determining the impact of toxicity on Wikipedia’s talk pages.  RWTH Aachen University; 2022.
  * Perspective API (2022) Perspective API, Technical documentation; 2022\.  https://developers.perspectiveapi.com/s/about-the-api.
  * Preece (2001) Preece J.  Sociability and usability in online communities: Determining and measuring success.  Behaviour & Information Technology. 2001; 20(5):347–356.
  * Qu et al. (2019) Qu I, Thain N, Hua Y.  WikiDetox Visualization.  In: _Wiki Workshop_ ; 2019. .
  * Raish (2019) Raish M, Identifying and Classifying Harassment in Arabic Wikipedia: A “Netnography”; 2019.
  * Ruprechter et al. (2021) Ruprechter T, Horta Ribeiro M, Santos T, Lemmerich F, Strohmaier M, West R, Helic D.  Volunteer contributions to Wikipedia increased during COVID-19 mobility restrictions.  Scientific reports. 2021; 11(1):21505.
  * Silva et al. (2016) Silva L, Mondal M, Correa D, Benevenuto F, Weber I.  Analyzing the targets of hate in online social media.  In: _Proceedings of the International AAAI Conference on Web and Social Media_ , vol. 10; 2016. p. 687–690.
  * Singer et al. (2017) Singer P, Lemmerich F, West R, Zia L, Wulczyn E, Strohmaier M, Leskovec J.  Why we read Wikipedia.  In: _Proceedings of the 26th International Conference on World Wide Web_ ; 2017. p. 1591–1600.
  * Smirnov and Oprea (2023) Smirnov I, Oprea C, Toxic comments reduce activity of volunteer editors on Wikipedia: Data & Code; 2023.  https://osf.io/2qyxj/.
  * Suh et al. (2009) Suh B, Convertino G, Chi EH, Pirolli P.  The singularity is not near: slowing growth of Wikipedia.  In: _Proceedings of the 5th International Symposium on Wikis and Open Collaboration_ ; 2009. p. 1–10.
  * Vogels (2021) Vogels EA.  The state of online harassment.  Pew Research Center. 2021; 13:625.
  * Wich et al. (2022) Wich M, Gorniak A, Eder T, Bartmann D, Cakici BE, Groh G.  Introducing an abusive language classification framework for telegram to investigate the german hater community.  In: _Proceedings of the International AAAI Conference on Web and Social Media_ , vol. 16; 2022. p. 1133–1144.
  * Wikimedia Foundation (2015a) Wikimedia Foundation, Community Wishlist Survey. Bots and gadgets; 2015.  https://meta.wikimedia.org/wiki/Community_Wishlist_Survey_2015/Bots_and_gadgets.
  * Wikimedia Foundation (2015b) Wikimedia Foundation, Harassment Survey 2015; 2015.  https://upload.wikimedia.org/wikipedia/commons/5/52/Harassment_Survey_2015_-_Results_Report.pdf.
  * Wikimedia Foundation (2021) Wikimedia Foundation, Community Wishlist Survey. Anti-harassment; 2021.  https://meta.wikimedia.org/wiki/Community_Wishlist_Survey_2021/Anti-harassment.
  * Wikipedia (2022) Wikipedia, Personal Attacks; 2022.  https://en.wikipedia.org/wiki/Wikipedia:No_personal_attacks.
  * Wulczyn et al. (2017) Wulczyn E, Thain N, Dixon L.  Ex machina: Personal attacks seen at scale.  In: _Proceedings of the 26th International Conference on World Wide Web_ ; 2017. p. 1391–1399.
  * Xia et al. (2020) Xia Y, Zhu H, Lu T, Zhang P, Gu N.  Exploring antecedents and consequences of toxicity in online discussions: A case study on reddit.  Proceedings of the ACM on Human-computer Interaction. 2020; 4:1–23.
  * Xu and Zhu (2010) Xu Z, Zhu S.  Filtering offensive language in online communities using grammatical relations.  In: _Proceedings of the Seventh Annual Collaboration, Electronic Messaging, Anti-Abuse and Spam Conference_ ; 2010. p. 1–10.
  * Yasseri et al. (2012) Yasseri T, Sumi R, Rung A, Kornai A, Kertész J.  Dynamics of conflicts in Wikipedia.  PLOS One. 2012; 7(6):e38869.
  * Zampieri et al. (2019) Zampieri M, Malmasi S, Nakov P, Rosenthal S, Farra N, Kumar R.  Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval).  arXiv preprint arXiv:190308983. 2019; .
  * Zampieri et al. (2020) Zampieri M, Nakov P, Rosenthal S, Atanasova P, Karadzhov G, Mubarak H, Derczynski L, Pitenis Z, Çöltekin Ç.  SemEval-2020 task 12: Multilingual offensive language identification in social media (OffensEval 2020).  arXiv preprint arXiv:200607235. 2020; .

##  5 Supplementary Material

Table S1: Summary statistics for the six language editions used in the study Edition | Comments | Users with at least
---|---|---
|  | one comment | one toxic comment
English | $37,304,436$ | $6,216,906$ | $80,307$
German | $8,015,691$ | $659,167$ | $19,543$
French | $3,838,161$ | $455,598$ | $4,282$
Spanish | $2,964,606$ | $525,443$ | $5,972$
Italian | $2,816,537$ | $395,257$ | $7,666$
Russian | $2,408,401$ | $326,738$ | $1,687$
Table S2: Attributes used by Perspective API Attribute name |  Description
---|---
Toxicity |  A rude, disrespectful, or unreasonable comment that is likely to make people leave a discussion.
Severe toxicity |  A very hateful, aggressive, disrespectful comment or otherwise very likely to make a user leave a discussion or give up on sharing their perspective. This attribute is much less sensitive to more mild forms of toxicity, such as comments that include positive uses of curse words.
Identity attack |  Negative or hateful comments targeting someone because of their identity.
Insult |  Insulting, inflammatory, or negative comment towards a person or a group of people.
Profanity |  Swear words, curse words, or other obscene or profane language.
Threat |  Describes an intention to inflict pain, injury, or violence against an individual or group.
Table S3: Performance of Perspective API models (AUC scores) | English | German | French | Spanish | Italian | Russian
---|---|---|---|---|---|---
toxicity | $0.97$ | $0.94$ | $0.94$ | $0.94$ | $0.95$ | $0.91$
severe toxicity | $0.98$ | $0.98$ | $0.96$ | $0.96$ | $0.97$ | $0.95$
identity attack | $0.97$ | $0.94$ | $0.96$ | $0.95$ | $0.96$ | $0.94$
insult | $0.97$ | $0.93$ | $0.94$ | $0.93$ | $0.96$ | $0.92$
profanity | $0.99$ | $0.97$ | $0.98$ | $0.98$ | $0.98$ | $0.96$
threat | $0.99$ | $0.95$ | $0.99$ | $0.98$ | $0.98$ | $0.96$
Table S4: Examples of comments with different toxicity scores Toxicity score |  Text
---|---
$0.07$ |  I wouldn’t trust me on this. I suggest asking [user name], who usually goes around correcting my mistakes. Also, you appear to be vastly more qualified than me. However I would have thought that geometric isomerism would be independent of the isotopic mass - the properties would be very similar (identical?).
$0.14$ |  Your home page doesn’t work. There is nothing in your talk section either!
$0.21$ |  I wanted to let you know that I just tagged [[article name]] for deletion, because it seems to be vandalism or a hoax. If you feel that the article shouldn’t be deleted and want more time to work on it, you can contest this deletion, but please don’t remove the speedy deletion tag from the top.
$0.35$ |  What the heck did you remove the [category name] from articles? All that happened was that some user renamed it to remove a [word] from the name, but didn’t set up a redirect. Now I’m going to have to go through all the articles that were in the category and reinsert it again. Could you not have done a quick search first just to make sure that the name of the category hadn’t been slightly renamed? Sorry if I sound a bit grouchy but I have spent most of this week doing nothing but fixing other people’s screw-ups and I’m getting fed up having to do it.
$0.49$ |  I don’t know who you think you are but I do NOT appreciate hours of work being undone. I don’t know what gives you the right to undo other peoples work. You claim I am editing warring yet it feels like quite the opposite for me. I contribute to wikipedia and you trash it. Yeah, I’m not edit warring, buddy.
$0.55$ |  Hope you’re proud of turning Wikipedia into a vehicle for attacking the reputations of real people whose biggest error was attempting to contribute and whose second biggest error was thinking that Wikipedia policies are sane. I’m not an edit warrior, like you, so I’m going to allow others to remove the libel. Feel free to edit war with them, too, and then pat yourself on the back for "defending Wikipedia against vandalism". The war on vandalism, like the war on drugs or terror, is a pointless and transparent fascist power grab.
$0.62$ |  Given the vile judgemental filth you have been posting in the [article name] name debate, you are not only a hypocrite; but anything I have said to you is remarkably restrained. It is clear that your only motives are to troll and to push a truly perverted point-of-view.
$0.72$ |  People have bent over backwards to accomodate you, mostly out of a pure concern for fairness and justice. You have eroded all the good will left in my body. Any more crap out of you, and I will "take steps". Better listen up good, because I’m one the oldest "old hands" around here and I know how to work the system. Don’t get me riled up against you, or I will force you out!!
$0.83$ |  I would like to thank you for the personal insult to my intelligence; as opposed to addressing the arguments in my comment. I know actually engaging in dialogue would have lowered you to my level. But the rest of the public doesn’t care too much for experts who believe that the rest of the public is below them, though I doubt you are an expert on anything but [vulgar speech] and being a self satisfied [expletive] who has nothing better to do than insult people on wikipedia.
$0.99$ |  Ahh shut [expletive 1] up you [expletive 2] [ethnic slur]. [expletive 1] you [ethnic slur] i will find u in real life and slit your throat.

Figure S1:  Lost activity estimates as a function of toxicity threshold and
activity level of users  We find that our results are robust with respect to
toxicity threshold and filtering out less active users. For visual clarity,
the 95% confidence intervals (shaded regions) are shown only for $0.2$ and
$0.9$ thresholds.  Figure S2:  Probability of leaving Wikipedia after
receiving a toxic comment as a function of toxicity threshold As expected, the
probability of leaving the project is higher when higher toxicity thresholds
are used. For all thresholds, this probability is substantially higher than
might be expected without a toxic comment.  Figure S3:  Bootstrapped
confidence intervals for the probability of leaving Wikipedia after receiving
a toxic. For the absolute majority of data points, the estimated probability
of leaving is higher after a toxic comment than otherwise with no intersection
between $95\%$ confidence intervals making the results significant with
$P<6.25\times 10^{-4}$. The estimates are less reliable for higher
contribution numbers because of the decreasing sample size.  Figure S4:  The
number of daily active users on Wikipedia over time. After a period of
exponential growth, the activity of Wikipedians slowed and then declined with
an exception of a COVID-related peak in activity.


Interpretable PID Parameter Tuning for Control Engineering using General
Dynamic Neural Networks: An Extensive Comparison

Johannes Günther1,2*, Elias Reichensdörfer3, Patrick M. Pilarski1,2, Klaus
Diepold3,

1 Department of Computing Science, University of Alberta, Edmonton, Alberta,
Canada

2 Alberta Machine Intelligence Institute, Edmonton, Alberta, Canada

3 Department of Electrical and Computer Engineering, Technische Universität
München, Munich, Bavaria, Germany

* Corresponding author

E-mail<EMAIL_ADDRESS>(JG)

###### Abstract

Modern automation systems largely rely on closed loop control, wherein a
controller interacts with a controlled process via actions, based on
observations. These systems are increasingly complex, yet most deployed
controllers are linear Proportional-Integral-Derivative (PID) controllers. PID
controllers perform well on linear and near-linear systems but their
simplicity is at odds with the robustness required to reliably control complex
processes. Modern machine learning techniques offer a way to extend PID
controllers beyond their linear control capabilities by using neural networks.
However, such an extension comes at the cost of losing stability guarantees
and controller interpretability. In this paper, we examine the utility of
extending PID controllers with recurrent neural networks—–namely, General
Dynamic Neural Networks (GDNN); we show that GDNN (neural) PID controllers
perform well on a range of complex control systems and highlight how they can
be a scalable and interpretable option for modern control systems. To do so,
we provide an extensive study using four benchmark systems that represent the
most common control engineering benchmarks. All control environments are
evaluated with and without noise as well as with and without disturbances. The
neural PID controller performs better than standard PID control in $15$ of
$16$ tasks and better than model-based control in $13$ of $16$ tasks. As a
second contribution, we address the lack of interpretability that prevents
neural networks from being used in real-world control processes. We use
bounded-input bounded-output stability analysis to evaluate the parameters
suggested by the neural network, making them understandable for engineers.
This combination of rigorous evaluation paired with better interpretability is
an important step towards the acceptance of neural-network-based control
approaches for real-world systems. It is furthermore an important step towards
interpretable and safely applied artificial intelligence.

##  1 Introduction

Modern production engineering involves increasingly complex physical processes
[1]. The physical processes underlying cutting-edge production engineering
cannot be appropriately expressed with simple models [2]. New, more complex
classical control methods are being developed, but their increased complexity
is what makes them challenging for control engineers to design and apply[3].
Limited by both the number of skilled control engineers and their cost, the
production engineering industry has widely chosen to continue to use simple,
understandable linear controllers. While these controllers are easy to set-up
and adjust, they are not suitable for the complex non-linear behaviour of the
processes they are expected to control. Using these simple controllers comes
at a cost: it means processes will require close monitoring and human
assistance whenever the system changes in an unforeseen way. In the face of
increasingly complex systems, both control-engineer-designed methods and
closely-monitored simple controllers fail to scale.

One way to bridge this gap between simple controllers and complex control
systems is by applying modern machine learning techniques [4]. Extending the
capabilities of well-accepted and used controllers with machine learning
yields a potential solution to the lack of scalability and adaptability in
existing control approaches. In this paper, we will investigate the use of
neural networks to adapt the parameters of a Proportional-Integral-
Differential (PID) controller not only before deployment but online during
ongoing control. This continuous adaptation to the process would allow for
linear controllers to perform well for any control task, as the controller
constantly linearizes its behaviour around the state the system is currently
in. As our investigation shows, this results in superior control performance
and disturbance rejection.

The PID controller is one of the most widely used controllers [5]. A PID
controller calculates its control output $u$ based on the current error $e$,
the error derivative $\frac{de}{dt}$ and the error integrated over time
$\int_{0}^{t}edt$. Each error measure is multiplied by its corresponding
constant—$K_{P}$ with the error, $K_{I}$ with the integrated error, and
$K_{D}$ with the error derivative—and then summed. A PID controller is tuned
with respect to the system to be controlled by adjusting these three
constants. While this controller is simple and well understood, its advantages
come at the price of limited capabilities. PID controllers perform well only
on linear systems or systems that are linearized. As PID controllers are
usually adjusted before deployment, they neither handle disturbances nor
varying system dynamics well enough to meet the needs of modern production
engineering.

Several prior studies have shown that neural networks can improve performance
when used to tune the parameters of traditional PID controllers online. Simple
feed-forward networks can adjust the PID parameters in multiple settings [6,
7]. More sophisticated neural networks like strictly recurrent, diagonal [8]
and quasi-diagonal [9] recurrent networks have also been investigated.
However, these specific design choices for the neural network’s architecture
result in very specific behaviour and learning [10] and thus are unlikely to
generalize well over different control environments. To evaluate whether the
use of neural networks with all possible connections is applicable to control
engineering tasks in general, a broad range of different control environments
is required. In this paper, we therefore implement and evaluate general
dynamic neural networks, thus extending the current state of the art.

Previous works have demonstrated the applicability of neural-network-based
parameter tuning, but only for one well-defined control task at a time, e.g.
pendulums [7], two-tank systems [11] or magnetic systems [6]. As no single
approach has been applied successfully to multiple representative control
engineering problems in the literature, the effectiveness of the suggested
approaches is hard to estimate. For this reason, we investigate our approach
on four different systems that represent the most common challenges in control
engineering, resulting in an extensive comparison. By comparing the control
performance on all four different tasks, we extend the current literature
which focused on one problem at a time. Additionally, our simulations are
closer to real-world conditions than those used in previous work, as they
include disturbances and noise.

Despite their potential to improve control performance, neural networks are
not widely used for parameter tuning in real-world control systems. One major
barrier prevents their adoption: due to the lack of interpretability, the
suggested PID parameters cannot be evaluated for their appropriateness. As a
result, the control loop itself loses one of its most important properties—its
stability guarantees. Previous papers that utilized neural networks for PID
parameter tuning did not address the effect on input-output stability, thus
ignoring a key concern in control engineering. The domain of control
engineering demands interpretability to ensure system safety [12]. As neural
networks introduce a black box to the system [13], their use for PID parameter
tuning is not yet widely accepted. Real-world use requires the ability to
check and reason about the parameter choices a neural network outputs when
granted access to a PID controller’s parameters. We suggest that the use of
neural networks will in fact enable designers to achieve better performance
without significantly increasing the design complexity or diminishing design
interpretability when deploying a controller—if combined with traditional
control engineering tools like stability analysis.

To summarize our contributions: as a first contribution of this paper, we
advance the use of specifically tailored neural network architectures by
investigating the use of General Dynamic Neural Networks (GDNNs) for online
parameter tuning in PID controllers. These neural PID controllers are
evaluated on four different closed loop control engineering tasks. Each task
represents a different, common challenge in control engineering, namely non-
linear behaviour, unstable equilibrium, dead time, and chaotic behaviour.
These applications have furthermore been used for evaluation of control at
several machine learning venues, e.g. [14, 15, 16]. We compare the performance
of our neural approach with a standard PID controller, which acts as a
baseline, and with a system-appropriate model-based controller, which should
provide the best possible performance. To evaluate the robustness of our
approach, each comparison is performed with and without significant sensor
noise and with and without disturbance. All controllers are evaluated
quantitatively for all scenarios, making this study unique in its
comprehensiveness when compared to prior work.

As a second contribution, we demonstrate how input-output stability analysis—a
classic analysis that is well known in control engineering—can be used in a
novel, online way to explain the effects of parameter tuning. This analysis is
done for the control system, exhibiting chaotic behaviour as an example of a
challenging control task. This assessment provides an explanation of when the
system is stable with respect to the PID parameters, making the neural network
outputs understandable to control engineers. While input-output stability is
an important aspect of every closed-loop control approach, it is widely
ignored in the existing literature that suggests the use of neural networks
for PID parameter tuning. This paper therefore addresses key attributes of
safe and interpretable artificial intelligence.

##  2 General Dynamic Neural Network for PID Parameter Tuning

(a) Closed loop control

3456$g^{\text{-}1}$12789$v$$e$$K_{P}$$K_{I}$$K_{D}$$g^{\text{-}1}$

(b) Example neural network

Fig 1: Neural PID architecture and example neural network. A neural network is
integrated into standard closed loop control (a). The neural network receives
the system output and the error as input and outputs the three PID parameters
$K_{P}$, $K_{I}$, and $K_{D}$. The double-lined arrows indicate that the
associated variable could be a vector, while the single-lined arrows indicate
scalar variables. An example neural network (b) shows one possible set of
connections. All networks have $9$ neurons in three layers. The output of each
input neuron is fed back into the input layer with a delay, denoted as
$g^{-1}$, of one time step.

The aim of this work is to extend the classic PID controller framework,
composed of the PID controller and the plant, with as few changes to the
physical setup of the control system as possible. Maintaining the core
structure of the classic PID control framework allows for easy adoption to
existing industrial applications. This structure is preserved by restricting
the neural network’s inputs to signals which are already available in the
closed loop control setting. The closed loop control setting is depicted in
Figure 1(a). The neural network’s inputs are the control system’s output $v$
and the control error $e$, while the outputs are the three PID parameters to
tune, i.e. $K_{P}$, $K_{I}$, and $K_{D}$. At each time step, the neural
network computes the PID parameters based on the observations and passes them
into the PID controller. The PID controller then uses these parameters to
compute the control output $u$.

To ensure fast online computation with limited hardware, the neural networks
implemented in this investigation are restricted to one hidden layer with four
neurons. Furthermore, for control systems that can be described by simple
differential equations increasing the number of neurons would lead to
overfitting. The network architectures differ only in additional recurrent or
feedback connections. An example can be seen in Figure 1(b). The figure shows
a neural network, where the output of each neuron in the first layer is fed
back as part of the first layer’s input with a delay, denoted as $g^{-1}$, of
one time step. For all neurons, the activation function tanh is chosen,
following the rationale of [17].

The standard approach for training neural networks is backpropagation [18].
Most deep learning approaches adjusts the neural network’s weights by end-to-
end optimization [19]. This optimization involves formulating a loss function
that describes the difference between the neural network’s outputs and the
ideal outputs. From this loss, a gradient with respect to the weights is
computed and propagated through the network. At each neuron, the weights are
then adjusted to minimize the loss. However, in the present framework, Figure
1(a), backpropagation cannot be naively applied. In this framework, the neural
network’s outputs are the PID parameters $K_{P}$, $K_{I}$, and $K_{D}$. Using
standard backpropagation would therefore require knowing the ideal PID
parameters at any given time.

A way to train the neural network without the ideal outputs is to numerically
approximate backpropagation. In this work, we chose a numerical Levenberg-
Marquard algorithm [20] to minimize the squared control error. At each time
step, the Jacobian matrix, shown in Equation 1, of the neural network weights
is numerically approximated using finite differences with respect to small
changes in each weight. Each weight is then adapted to decrease the control
error.

The Jacobian matrix of a general dynamic neural network with $p$ input
neurons, $q$ dynamical system (plant) outputs and k weighted connections was
calculated as

$J_{\textbf{w}}\big{(}\textbf{v}(\textbf{x};\textbf{w})\big{)}=\begin{bmatrix}\frac{\partial}{\partial
w_{0}}v_{0}(x_{0};\textbf{w})&\frac{\partial}{\partial
w_{1}}v_{0}(x_{0};\textbf{w})&\cdots&\frac{\partial}{\partial
w_{k}}v_{0}(x_{0};\textbf{w})\\\ \frac{\partial}{\partial
w_{0}}v_{0}(x_{1};\textbf{w})&\frac{\partial}{\partial
w_{1}}v_{0}(x_{1};\textbf{w})&\cdots&\frac{\partial}{\partial
w_{k}}v_{0}(x_{1};\textbf{w})\\\ \vdots&\vdots&\ddots&\vdots\\\
\frac{\partial}{\partial
w_{0}}v_{0}(x_{p};\textbf{w})&\frac{\partial}{\partial
w_{1}}v_{0}(x_{p};\textbf{w})&\cdots&\frac{\partial}{\partial
w_{k}}v_{0}(x_{p};\textbf{w})\\\ \frac{\partial}{\partial
w_{0}}v_{1}(x_{0};\textbf{w})&\frac{\partial}{\partial
w_{1}}v_{1}(x_{0};\textbf{w})&\cdots&\frac{\partial}{\partial
w_{k}}v_{1}(x_{0};\textbf{w})\\\ \frac{\partial}{\partial
w_{0}}v_{1}(x_{1};\textbf{w})&\frac{\partial}{\partial
w_{1}}v_{1}(x_{1};\textbf{w})&\cdots&\frac{\partial}{\partial
w_{k}}v_{1}(x_{1};\textbf{w})\\\ \vdots&\vdots&\ddots&\vdots\\\
\frac{\partial}{\partial
w_{0}}v_{1}(x_{p};\textbf{w})&\frac{\partial}{\partial
w_{1}}v_{1}(x_{p};\textbf{w})&\cdots&\frac{\partial}{\partial
w_{k}}v_{1}(x_{p};\textbf{w})\\\ \vdots&\vdots&\ddots&\vdots\\\
\frac{\partial}{\partial
w_{0}}v_{q}(x_{0};\textbf{w})&\frac{\partial}{\partial
w_{1}}v_{q}(x_{0};\textbf{w})&\cdots&\frac{\partial}{\partial
w_{k}}v_{q}(x_{0};\textbf{w})\\\ \frac{\partial}{\partial
w_{0}}v_{q}(x_{1};\textbf{w})&\frac{\partial}{\partial
w_{1}}v_{q}(x_{1};\textbf{w})&\cdots&\frac{\partial}{\partial
w_{k}}v_{q}(x_{1};\textbf{w})\\\ \vdots&\vdots&\ddots&\vdots\\\
\frac{\partial}{\partial
w_{0}}v_{q}(x_{p};\textbf{w})&\frac{\partial}{\partial
w_{1}}v_{q}(x_{p};\textbf{w})&\cdots&\frac{\partial}{\partial
w_{k}}v_{q}(x_{p};\textbf{w})\end{bmatrix}\in\mathbb{R}^{pq\times k}$ (1)

, where $\textbf{x}\in\mathbb{R}^{p}$ is the neural network input vector,
$\textbf{w}\in\mathbb{R}^{k}$ is the vector of weights which describes the
network topology and $\textbf{v}(\textbf{x};\textbf{w})\in\mathbb{R}^{q}$ are
the outputs of the dynamical system (plant). According to [21], it is
sufficient to calculate the partial derivatives of the systems output instead
of the error function.

As the analytic calculation would result in extensive computations, it is
numerical approximated using a difference equation, rather than a differential
one as solution of the equation

$\hat{J}_{\textbf{w}}\big{(}\textbf{v}(\textbf{x};\textbf{w})\big{)}=\begin{bmatrix}\textbf{j}_{0}\\\
\textbf{j}_{1}\\\ \vdots\\\
\textbf{j}_{k}\end{bmatrix}^{T}\,,\,\textbf{j}_{i}=\frac{\textbf{v}(\textbf{x};\textbf{w})-\textbf{v}(\textbf{x};\textbf{w}-\eta\varepsilon(\textbf{w}_{i}))}{\varepsilon(\textbf{w}_{i})}\,,\,\eta_{p}=\left\\{\begin{array}[]{ll}1,&p=i\\\
0,&p\neq i\end{array}\right.,$ (2)

where $\hat{J}_{w}$ is the approximated Jacobian matrix, $\textbf{j}_{i}$ is
the $i$-th column of $\hat{J}_{w}$, $\eta$ is the step size and $\epsilon$ is
the machine precision. $\epsilon$ has to be calculated for each pass, using
the equation

$\epsilon(\textbf{w}_{i})=\max\big{(}1,|\textbf{w}_{i}|\big{)}\sqrt{\epsilon_{\text{min}}},$
(3)

where w is a vector, containing the weights, and $\epsilon_{\text{min}}$ is
the implementation data type, double precision in this implementation. The
complete calculation of the Jacobian matrix can be found in Algorithm 1.

Algorithm 1 Numerical approximation for Jacobian matrix.

1: Input: Dynamical system (plant) output $\textbf{v}(\textbf{x};\textbf{w})$,
neural network inputs x, weights $W$

2: Output: Estimate of Jacobian matrix $\hat{J}_{W}$

3: foreach:Weight $\textbf{w}_{i}$ do

4:
$\epsilon\leftarrow\max\big{(}1,|\textbf{w}_{i}|\big{)}\sqrt{\epsilon_{\text{min}}}$
// Calculate $\epsilon$

5: For $j=1$ to $g$ do

6: $\textbf{v}_{\text{tmp},1}\leftarrow\textbf{v}(\textbf{x};\textbf{w})$,
$\textbf{v}_{\text{tmp},2}\leftarrow\textbf{v}(\textbf{x};\textbf{w}-\eta\varepsilon(\textbf{w}_{i}))$
// Compute difference of v

7: For $o=1$ to $q$ do

8:
$\hat{J}_{W(jm+o,i)}\leftarrow\frac{\textbf{v}_{\text{tmp},1}-\textbf{v}_{\text{tmp},2}}{\epsilon}$
// Backward difference

9: Return: $\hat{J}_{W}$

In the industrial control setting we consider in this paper, it is important
that finding an appropriate network architecture, i.e. the connections and
delays between neurons, does neither need sophisticated engineering nor
significant time. Any approach that makes the set up too complicated would
defeat the purpose of extending the existing PID framework. We therefore chose
to create ten neural network architectures for each control challenge by
randomly adding feedback connections and then chose the neural network that
performed best. This process of finding an architecture can be made more
efficient by using search algorithms [22]. For each tested neural network, the
weights were initialized randomly with a mean of zero and a standard deviation
of one.

Training data was collected from the differential equations, that describe
each benchmark system, without any additional noise or disturbances. We
followed an approach from classical control engineering by exciting the system
with input signals of different lengths and amplitudes—called Amplitude
Modulated Probabilistic Random Binary Signals (APRBS) [23]—to collect data
samples that sufficiently describe the dynamics of the system [24]. This
approach is an extension of using a Dirac impulse for system identification
[25]. We collected $35,000$ data samples for each benchmark system, and split
them into training, validation, and test sets with a ratio of $15\%$, $30\%$,
and $55\%$, respectively.

##  3 Experiments

To evaluate the performance of the neural PID controller, we use four typical
control problems. Each system offers a different control challenge. Each
individual system is controlled with and without noise. The noise is Gaussian
white noise with a signal to noise ratio (SNR) of $20$dB and corresponds to
noisy measurements from sensors and is added to the system output.

To further evaluate the robustness of the controller, we inflict upon each
system a suitable disturbance. The term disturbance refers to an unwanted and
unexpected system input that will result in an increase of the system error.
Disturbances can occur due to external influences or due to failure in the
system and they are individual for each kind of system. For each system, the
disturbance is increased in size from zero until only one control approach is
still able to stabilize the system. Disturbances of this magnitude are then
used in the experiments. These disturbances are not included in the training
data for the neural PID controller. All experiments are run over $50$
independent runs to ensure statistical relevance. The system differential
equations are solved using the Dormand-Prince solver. To simulate real-world
conditions of a discrete sample time, the controller output can be adapted
every $0.01$s. This intervention time is chosen to represent the limitations
of real-world actuators, which cannot adjust their values on an arbitrarily
small timescale. The solver is run iteratively for $0.01$s, using the result
of the former step as starting conditions. During each $0.01$s intervention
time window, the controller output $u$ is kept constant.

The training is done on an Intel Core i5-4570 with a $3.2$ GHz clock rate, $6$
MB of shared L$3$ cache, $32$ GB DDR3 RAM. Once learned, the neural networks
run on a Raspberry Pi $3$ Model A+.

Two-Tank System: The first system is a nonlinear two-tank system, as seen in
Figure 2 and is described by the differential equations in [26]. The
controller has access to a pump, regulating the input, while the measured
output is the water level in the second tank. This system is a standard
benchmark system in control theory. It corresponds to various industrial
processes, e.g. bio-reactors, filtration, and nuclear power plants. There
exist a number of control approaches for this system, including direct control
via neural networks [27], adaptive output feedback [28], and backstepping
[26], which will be used as comparison.

To evaluate the robustness of the compared control approaches, the two-tank
system is disturbed continuously between $t=20$s and $t=40$s. As a
disturbance, the controller output is set to zero, which would correspond to a
drain of the water supply. The water levels in the tank are therefore
independent from the control inputs for $20$s. The voltage for the pump, $u$,
was limited to the range $[-500\text{V},500\text{V}]$ to simulate a pump
appropriate to the tank dimensions.

$x_{1}(t)$$v(t)=x_{2}(t)$$k_{pump}u(t)$$A_{1}$$A_{o1}$Liquid$A_{2}$$A_{o2}$

Fig 2: Two-Tank system. This nonlinear system is widely used to study
nonlinear behaviour in control engineering systems. The controller can adjust
the amount of water being put into the first tank. The goal is to keep the
water level in the lower tank ($v(t)=x_{2}(t)$ at the setpoint.

Inverted Pendulum on a Cart: The second system is a nonlinear inverted
pendulum on a cart, as shown in Figure 3. The system is described by the
differential equations in [29]. The control task is to stabilize the inverted
pendulum at its unstable equilibrium by applying a force on the cart. The
cart’s movement is restricted to $0.5$m in either direction. This system is a
widespread benchmark system in control theory due to its nonlinearity and
unstable equilibrium [29, 30]. Practical applications for inverted pendulums
include rocket control during initial stages of flight or keeping a walking
robot in an upright position. For comparison, a linear–quadratic regulator
(LQR) [31, 32] and a double PID controller [32] are used.

The system is disturbed by a force of $8.5$N to the pendulum at time $t=10$s.
This disturbance can be interpreted as a strong and unexpected wind condition
during the launch of a rocket. The controller ouptut $u$ was bounded within
$[-50\text{N},50\text{N}]$, which corresponds to a typical actuator of that
size.

$M$$m$$l$$l$$\dot{x}$$\dot{y}$$y$$x$$\theta$$F_{x}$

Fig 3: Inverted Pendulum on a Cart. This system is a nonlinear system with an
unstable equilibrium. The control task is to move the cart to a predefined
position, while keeping the pole up.

System with Non-negligible Time Delay: The third system is a first order
linear time invariant (LTI) system with a non-negligible time delay. Time
delay is a problem in control theory that is often forgotten while designing
controllers [33]. Time delays can result in decreased performance and system
instability. The benchmarks for this system are a PID controller [34] and a
smith-predictor [35]. Figure 4 demonstrates the delayed system response for an
input.

This system is disturbed by a (dimensionless) disturbance of $-5$ between
$t=50$s and $t=75$s continuously. Such a disturbance can be thought of as a
temporary blockage in a fluid transport system. The exact system
specifications can be found in [34]. The controller output $u$ was bounded
between the range of $[-10,10]$.

Time t

Amplitude

$K$$T$$K_{63}$$T_{D}$

Fig 4: System with Non-negligible Time Delay. The Figure shows the systems
response (including the time delay $T_{D}$) to an input.

Chaotic Thermal Convection Loop: The fourth system is a chaotic thermal
convection loop, as shown in 5. Its dynamics are described by the equations

$\begin{split}\dot{x_{1}}(t)&=p(x_{2}-x_{1})\,,\\\
\dot{x_{2}}(t)&=x_{1}-x_{2}-x_{3}(x_{1}+\beta),\\\
\dot{x_{3}}(t)&=x_{1}x_{2}+\beta(x_{1}+x_{2})-x_{3}-u,\\\ \end{split}$ (4)

with $p=10$ and $\beta=6$ as appropriate constants [36]. $x_{1}$ is a measure
for how far the current flow velocity differs from the steady point of the
system—if $x_{1}$ is zero, the system is in its steady point. $x_{2}$ and
$x_{3}$ are measures for the difference in temperature between the points A
and B, as well as C and D in Figure 5 respectively. $u$ is a the power,
applied to the heater and the control variable.

Chaotic behavior may lead to vibrations, oscillations and failure in systems
and is therefore an important aspect of control theory. As chaotic behavior is
unpredictable, mathematical models are only sufficient to a certain point,
hence closed loop control is a desirable approach [37]. Usual control
approaches for the chaotic thermal convection loop are nonlinear feedback
controllers [38] and backstepping [39, 40].

To evaluate the robustness of the applied control approaches, the system is
disturbed with a force of $100$W continuously between $t=5$s and $t=5.5$s.
This perturbation can be interpreted as a temporary change in the cooling
water temperature. To simulate real actuators with a limited capacity, the
controller output $u$ is limited between $[-100\text{W},100\text{W}]$ to
simulate an appropriate heating element.

$2r_{2}$HeaterCooling jacket$r_{1}$ABCDInletOutlet

Fig 5: Chaotic Thermal Convection Loop. This system is an example for chaotic
behaviour. The control task is to maintain a constant flow in the inner torus
- the flow is measured at the points A and B. Half of the torus that contains
the fluid is surrounded by a heating element, the other half is surrounded by
a cooling jacket. The control variable is the heating power, applied on the
lower half of the torus.

##  4 Results and Discussion

Table 1: Control results for the four benchmark systems. Control Benchmark | RMSE on test data over 50 runs
---|---
Disturbance | - | - | ✓ | ✓
SNR | - | 20 dB | - | 20 dB
Two-tank system |
Mean | Neural PID | $\boldsymbol{0.74}$ | $\boldsymbol{0.86}$ | $\boldsymbol{0.84}$ | $\boldsymbol{1.00}$
Standard PID | $0.95$ | $0.99$ | $1.00$ | $1.10$
Backstepping | $1.10$ | $1.10$ | $1.20$ | $1.10$
Variance | Neural PID | $0.0036$ | $0.0040$ | $0.0056$ | $0.0460$
Standard PID | $0.0025$ | $0.0023$ | $0.0052$ | $0.0063$
Backstepping | $0.0016$ | $0.0018$ | $0.0036$ | $0.0024$
Inverted pendulum |
Mean | Neural PID | $\boldsymbol{0.034}$ | $\boldsymbol{0.02}$ | $\boldsymbol{0.09}$ | $\boldsymbol{0.27}$
Standard PID | $\boldsymbol{0.035}$ | $0.04$ | $140$ | $140$
LQ regulator | $0.05$ | $0.05$ | $140$ | $140$
Variance | Neural PID | $0.0004$ | $0.0003$ | $0.0490$ | $0.0007$
Standard PID | $0$ | $1.1\cdot 10^{-7}$ | $0$ | $0.0023$
LQ regulator | $0$ | $3.3\cdot 10^{-9}$ | $0$ | $0.0022$
LTI system with input delay |
Mean | Neural PID | $\boldsymbol{0.13}$ | $\boldsymbol{0.15}$ | $0.26$ | $0.28$
Standard PID | $0.22$ | $0.23$ | $0.36$ | $0.38$
Smith predictor | $0.18$ | $0.19$ | $\boldsymbol{0.19}$ | $\boldsymbol{0.20}$
Variance | Neural PID | $0.0006$ | $0.0005$ | $0.0007$ | $0.0003$
Standard PID | $0.0007$ | $0.0006$ | $0.0004$ | $0.0003$
Smith predictor | $0.0006$ | $0.0005$ | $0.0005$ | $0.0003$
Chaotic thermal convection loop |
Mean | Neural PID | $\boldsymbol{0.23}$ | $0.90$ | $\boldsymbol{1.90}$ | $\boldsymbol{1.70}$
Standard PD | $0.24$ | $1.60$ | $13.00$ | $4.20$
Backstepping | $0.26$ | $\boldsymbol{0.89}$ | $9.80$ | $9.80$
Variance | Neural PID | $1.2\cdot 10^{-5}$ | $0.0003$ | $0.16$ | $0.22$
Standard PD | $0$ | $2.00$ | $0$ | $1.40$
Backstepping | $0$ | $0.0001$ | $0$ | $1.4\cdot 10^{-5}$

The results for all experiments can be found in Table LABEL:tab:ctrl_results.
For each system the mean and variance from $50$ independent runs are shown for
all controllers in all tested scenarios. The best control approach is
highlighted in bold. For all values, a two-sample t-test was performed and a
control approach is only considered to be superior for $p<0.05$. From Table
LABEL:tab:ctrl_results, it can be seen that the neural PID controller performs
best in $12$ scenarios, pairs with the standard PID controller in one and
performs second best in two scenarios. To compare control approaches, there
are common measurements that are used in control engineering, e.g. rise time,
overshoot, settling time [5]. However, these measurements all address the
error between the setpoint and the actual systems output with different
emphases. We therefore used the root-mean-squared-error (RMSE) between the
setpoint and the system output to summarize these error measures in a single
number without losing information.

Flow velocity(a)051015-30-20-10010SetpointNeural PIDStandard PDBackstepping

P [W]

(b)Controller output051015-2000200Time t [s]

Amplitude

(c)Adaptive PID parameters051015-50050$K_{P}$$K_{I}$$K_{D}$

$x_{1}$[m/min]

Fig 6: Control performance for the disturbed chaotic thermal convection loop.
Subfigure (a) shows the setpoint ($x_{1}=0$, which corresponds to a steady
flow) and the system output for all controllers. Subfigure (b) shows the
controller outputs for all three controllers. In subfigure (c), the PID
parameters, applied by the neural network are shown.

Two-Tank System: For the two-tank system, a backstepping controller is chosen
for a comparison, as this approach takes the nonlinear behavior of the system
into account and has demonstrated to be well suited for this system [26]. The
PID controller is parameterized with the constants $K_{P}=3.65,K_{D}=-2$ and
$K_{I}=0.4$. All controllers are able to control the system, while the neural
PID controller exhibited the smallest error for all scenarios. However, the
advantage the neural PID controller yields is relatively small, as can be seen
in Table LABEL:tab:ctrl_results. As this system is the easiest to control, it
can be expected that the standard PID controller and backstepping perform on a
similar scale.

Inverted Pendulum: The inverted pendulum on a cart is controlled by a standard
PID controller stack [29], and a LQR [32] for benchmarking. The PID
controller, responsible for the position, has the values are
$K_{P}=-2.4,K_{D}=-0.75$ and $K_{I}=-1$ and the controller for the angle is
set to $K_{P}=25,K_{D}=3$ and $K_{I}=15$ [29]. All three approaches are able
to initially stabilize the system. For the scenario without disturbance, the
neural PID controller performs equally to the standard PID controller when
there is no noise present. For the scenario with only noise, the neural PID
controller is the best control approach. Furthermore, only the neural PID
controller is able to stabilize the disturbed system, resulting in a
substantially lower error as can be seen in Table LABEL:tab:ctrl_results.

System with Non-negligible Time Delay: For the system with non-negligible time
delay, the standard PID controller is set to $K_{P}=1.5,K_{D}=-0.1$ and
$K_{I}=0.7$. The neural PID controller is superior in all scenarios, when
compared to the standard PID controller. When compared to the Smith predictor
[41], the neural PID controller performs better only for the scenarios without
noise. However, as the Smith predictor has knowledge about the exact time
delay, it has a significant advantage over the neural PID controller.

Chaotic Thermal Convection Loop: The PID parameters yielding the lowest error
for the chaotic thermal convection loop are $K_{P}=25.3$, $K_{D}=8.9$ and
$K_{I}=0$—the controller is therefore a PD controller. The system is
initialized outside of its inherently stable region (region of attraction)
with the initial conditions $x_{1}=x_{2}=x_{3}=5$. Without control, the system
will therefore not converge to the desired steady state $x_{1}=x_{2}=x_{3}=0$.

All controllers are capable of stabilizing the system, as can be seen in
Figure 6(a). The backstepping approach has the least overshoot but takes a
long time to reach the steady state. The standard PID controller is more
aggressive, resulting in a higher overshoot but still a smaller error. The
neural PID controller performs best as it finds a good balance between
settling time and overshoot.

Out of four scenarios, the neural PID controller demonstrates superior control
performance. Only in the scenario with noise but without disturbance does
backstepping perform slightly better ($0.89$ vs $0.9$). This can be explained
by backstepping being designed using the differential equations. It therefore
knows the underlying systems dynamics and is less influenced by the sensor
noise.

Between the time $t=5$s and $t=5.5$s, the control output is set to $u=-100$W
to simulate the disturbance described earlier. The standard PD controller
becomes meta stable and its controller output iterates between the maximum
value of $100$W and the minimum value of $-100$W. Although backstepping is
proven to be globally, asymptotically stable in the Lyapunov sense [42], it
also becomes meta stable. This can be explained by the real world conditions.
As the controller can change its control output only every $0.01$s the
backstepping approach fails, resulting in switching inputs between the maximum
value and the minimum value, as seen in Figure 6(b). Both controllers (PID and
backstepping) use excessive amounts of energy without being able to stabilize
the system.

The neural PID controller is able to stabilize the system after the
disturbance. Figure 6(c) shows how the neural network changes the PID
parameters in response to the system output. When $x_{1}$ is far from the
setpoint, the $K_{P}$ parameter has a high absolute value to force the system
towards its steady state. To further increase the controller output at
$t=7.9$s, where the system reaches its furthest distance from the set point
$K_{I}$ is increased. After the system reaches its steady state again, all PID
parameters are adjusted back to their stationary value to ensure asymptotic
performance. The neural PID controller furthermore uses significantly less
energy to control the system.

Fig 7: Stability analysis for the chaotic thermal convection loop with
disturbance. The dashed line shows the systems output, when controlled by the
neural PID controller. The closed loop transfer function is not guaranteed to
be stable within the grey areas, despite the algorithm stabilizing the system.
As the system approaches its steady state, the system becomes input-output
stable with the chosen PID parameters. The second subplot shows the real
values for all four poles. The system is only stable (white background colour)
if all poles have real values smaller than zero.

Investigating the solutions stability: Despite the experimental evidence that
suggests the enormous benefit of using neural networks to adapt PID parameters
online, this approach is not yet used on real-world system. This is due to the
black box character of neural networks and the stringent safety requirements
for control processes. One of the most important safety requirements of a
closed loop control approach is input-output stability. It describes whether
the system output is bounded for all bounded inputs. A system can be evaluated
for stability by analysing the closed loop transfer function, i.e. the
relation between the system output to its input.

For a bounded-input bounded-output stability analysis, the closed loop
transfer function is computed as

$\frac{V(s)}{V^{*}(s)}=\frac{G(s)H(s)}{1+G(s)H(s),}$ (5)

where $V(s)$ is the system output, $V^{*}(s)$ is the setpoint, G(s) is the
system transfer function and H(s) is the controller transfer function in the
Laplace domain. Following the Nyquist criterion, the system is stable if all
poles are in the left half of the left half plane, i.e. their real values are
smaller than zero. In the example of the chaotic thermal convection loop, the
systems transfer function $G(s)$, linearized around a steady state
$x_{1}=x_{2}=x_{3}=0$, does not change over time and only has to be computed
once. The controller transfer function $H(s)$ is dependent on the PID
parameters and therefore changes at every time the neural network adjusts
these parameters. To make sense of these changes and interpret them from a
stability perspective, the controller transfer function therefore needs to be
computed at every time step. This varies from the traditional stability
analysis, which is computed once under the assumption of non-changing PID
parameters. Together, these transfer functions express whether the closed-loop
solution is stable or unstable.

As an important contribution, we therefore perform an online analysis of the
input-output stability for the controller. This analysis can be seen in Figure
7. The Figure shows the systems output and the stability, with respect to its
linearized steady state over the experiment and the real values for all four
poles of the transfer function. The closed loop transfer function is not
stable in the beginning, during settling and after the disturbance. This can
be expected, as the system is far away from its steady state for which
stability is evaluated. However, as the systems output gets close to the set
point, i.e. the steady state, the closed loop transfer function becomes
stable. Knowing about the relationship between chosen PID parameters and
stability allows to include this knowledge into the training. A potential way
to include this information would be to include the poles as a regularization
term during training in order to force the system towards an input-output
stable behaviour. Furthermore, the input-output stability evaluation is an
important insight for control engineers and makes the neural PID controllers
understandable for humans, thus emphasising its applicability for safety
critical systems.

##  5 Future work

This paper presents a first step towards accommodating the needs of control
engineers when integrating machine learning algorithms into existing control
architectures. While we identified a way to relate the parameters applied by
the neural network back to input-output stability, this new information was
not yet leveraged during the training procedure. It is a natural extension of
this paper to use the newly found information about stability as a regularizer
when training the neural network to ensure only input-output stable PID
parameters. Beyond extensions of the exact framework used in this paper, we
have provided an effective demonstration of the more general idea of
leveraging machine learning algorithms to enhance existing control methods.

While other examples exist in the literature (e.g., the combination of neural
networks[43] or fuzzy logic systems [44] with backstepping), many other
combinations of different machine learning approaches with other control
algorithms have the potential to provide good results. For example, the
machine learning methods from reinforcement learning might be well-applied to
linear quadratic regulators while maintaining interpretability.

##  6 Conclusion

In this paper, we conduct an extensive and rigorous investigation into the use
of general dynamic neural networks for online PID parameter adaption. We
perform experiments on four different systems, with and without sensor noise
as well as with and without disturbance, resulting in $16$ experiments in
total. These scenarios cover the most important challenges in control
engineering. This study is therefore unique in its extensiveness, as previous
papers only used one type of benchmark system. The neural-network-based
approach outperforms a standard PID controller in $15$ of $16$ scenarios and
outperforms a model-based controller in $13$ of $16$ scenarios.

These results showcase the potential of extending existing systems by machine
learning in general and neural networks in particular. Furthermore, we keep
the neural network design and integration simple to allow for easy adoption of
our technique. With an appropriate implementation as a library, our technique
could be used without extensive knowledge of either control engineering or
neural networks. To the best of our knowledge, this is the first investigation
that uses general dynamic neural networks, extending the state of the art for
using neural networks to tune PID parameters. We perform a detailed analysis
for one representative scenario, highlighting the superior control performance
of our approach over both the traditional PID controller and model-based
backstepping. It is worth noticing that while training data was gathered from
simple differential equations, the results indicate significantly increased
resilience towards noise and unforeseen disturbances.

Although the significant potential of neural networks for PID parameter tuning
is known [7, 8], this technique has not been used in real-world applications
to date. As the functioning of a neural network in this setting is not
understood, control engineers refrain from using them. In a first attempt to
solve this problem, we perform an input-output stability analysis to interpret
how neural networks function within the suggested framework. Tying the neural
network outputs back to stability makes this neural-network-based approach
understandable to humans. We therefore address a key issue when applying
machine learning algorithms for control problems: the interpretability of and
subsequently the trust in the machine learning solution. This work is thus an
important step to increase the acceptance of machine learning based approaches
for real-world systems and an important step towards safe and interpretable
applied artificial intelligence.

## Acknowledgements

The authors also thank Omid Namaki, Matthew Schlegel, Ahmed Shehata, Brian
Tanner, and Nadia M. Ady for suggestions and helpful discussions.

## References

  * 1. ElMaraghy W, ElMaraghy H, Tomiyama T, Monostori L.  Complexity in engineering design and manufacturing.  CIRP annals. 2012;61(2):793–814.
  * 2. Åström KJ.  Limitations on control system performance.  European Journal of Control. 2000;6(1):2–20.
  * 3. Klatt KU, Marquardt W.  Perspectives for process systems engineering—Personal views from academia and industry.  Computers & Chemical Engineering. 2009;33(3):536–550.
  * 4. Wuest T, Weimer D, Irgens C, Thoben KD.  Machine learning in manufacturing: advantages, challenges, and applications.  Production & Manufacturing Research. 2016;4(1):23–45.
  * 5. Ang KH, Chong G, Li Y.  PID control system analysis, design, and technology.  IEEE Transactions on Control Systems Technology. 2005;13(4):559–576.
  * 6. de Jesús Rubio J, Zhang L, Lughofer E, Cruz P, Alsaedi A, Hayat T.  Modeling and control with neural networks for a magnetic levitation system.  Neurocomputing. 2017;227:113–121.
  * 7. de Jesús Rubio J.  Discrete time control based in neural networks for pendulums.  Applied Soft Computing. 2018;68:821–832.
  * 8. Zhang Mg, Wang XG, Li Wh.  The Self-Tuning PID Decoupling Control Based on the Diagonal Recurrent Neural Network.  In: Proceedings of the International Conference on Machine Learning and Cybernetics; 2006. p. 3016–3020.
  * 9. Zhang K, An X.  Design of Multivariable Self-Tuning PID Controllers via Quasi-diagonal Recurrent Wavelet Neural Network.  In: Proceedings of the International Conference on Intelligent Human-Machine Systems and Cybernetics. vol. 2; 2010. p. 95–99.
  * 10. Bergstra J, Yamins D, Cox DD.  Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures.  JMLR. 2013;.
  * 11. Pirabakaran K, Becerra VM.  PID autotuning using neural networks and model reference adaptive control.  IFAC Proceedings Volumes. 2002;35(1):451–456.
  * 12. Leveson N, Daouk M, Dulac N, Marais K.  A systems theoretic approach to safety engineering.  Dept of Aeronautics and Astronautics, Massachusetts Inst of Technology, Cambridge. 2003; p. 16–17.
  * 13. Yang Z, Zhang A, Sudjianto A.  Enhancing explainability of neural networks through architecture constraints.  arXiv preprint arXiv:190103838. 2019;.
  * 14. Yu AW, Ma W, Yu Y, Carbonell J, Sra S.  Efficient structured matrix rank minimization.  In: Advances in Neural Information Processing Systems; 2014. p. 1350–1358.
  * 15. Berkenkamp F, Turchetta M, Schoellig A, Krause A.  Safe model-based reinforcement learning with stability guarantees.  In: Advances in Neural Information Processing Systems; 2017. p. 908–918.
  * 16. Parmas P, Rasmussen CE, Peters J, Doya K.  PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos.  In: International Conference on Machine Learning; 2018. p. 4062–4071.
  * 17. Kalman BL, Kwasny SC.  Why tanh: choosing a sigmoidal function.  In: Proceedings of the International Joint Conference on Neural Networks. vol. 4. IEEE; 1992. p. 578–581.
  * 18. Rumelhart DE, Durbin R, Golden R, Chauvin Y  Backpropagation: The basic theory.  In: Backpropagation: Theory, architectures and applications; 1995. p. 1–34.
  * 19. Glasmachers T.  Limits of End-to-End Learning.  In: Proceedings of the Asian Conference on Machine Learning. 2017. p. 17–32.
  * 20. Marquardt DW.  An algorithm for least-squares estimation of nonlinear parameters.  Journal of the society for Industrial and Applied Mathematics. 1963;11(2):431–441.
  * 21. De Jesus O, Hagan MT.  Backpropagation algorithms for a broad class of dynamic networks.  IEEE Transactions on Neural Networks. 2007;18(1):14–27
  * 22. Sandner C, Günther J, Diepold K.  Automated optimization of dynamic neural network structure using genetic algorithms.  Technical Report, 2017.
  * 23. Deflorian M, Zaglauer S.  Design of experiments for nonlinear dynamic system identification.  IFAC Proceedings Volumes. 2011;44(1):13179–13184.
  * 24. Günther J.  Machine intelligence for adaptable closed loop and open loop production engineering systems [Dissertation].  Technische Universität München. München; 2018.
  * 25. Åström KJ, Eykhoff P.  System identification—a survey.  Automatica. 1971;7(2):123–162.
  * 26. Pan H, Wong H, Kapila V, de Queiroz MS.  Experimental validation of a nonlinear backstepping liquid level controller for a state coupled two tank system.  Control Engineering Practice. 2005;13(1):27–40.
  * 27. Majstorovic M, Nikolic I, Radovic J, Kvascev G.  Neural network control approach for a two-tank system.  In: Proceedings of the International Conference on Neural Network Applications in Electrical Engineering. IEEE; 2008. p. 215–218.
  * 28. Takagi T, Mizumoto I, Tsunematsu J.  Performance-driven adaptive output feedback control and its application to two-tank system.  In: Proceedings of the International Conference on Advanced Mechatronic Systems. IEEE; 2014. p. 254–259.
  * 29. Wang JJ.  Simulation studies of inverted pendulum based on PID controllers.  Simulation Modelling Practice and Theory. 2011;19(1):440–449.
  * 30. Kim S, Kwon S.  Nonlinear Optimal Control Design for Underactuated Two-Wheeled Inverted Pendulum Mobile Platform.  IEEE/ASME Transactions on Mechatronics. 2017;22(6):2803–2808.
  * 31. Jaleel JA, Francis RM.  Simulated annealing based control of an Inverted Pendulum System.  In: Proceedings of the International Conference on Control Communication and Computing. IEEE; 2013. p. 204–209.
  * 32. Li W, Ding H, Cheng K.  An investigation on the design and performance assessment of double-PID and LQR controllers for the inverted pendulum.  In: Proceedings of the International Conference on Control; 2012. p. 190–196.
  * 33. Wang C, Fu W, Shi Y.  Fractional order proportional integral differential controller design for first order plus time delay system.  In: Proceedings of the International Conference on Control and Decision Conference. IEEE; 2013. p. 3259–3262.
  * 34. Tavakoli S, Tavakoli M.  Optimal Tuning of PID Controllers for First Order Plus Time Delay Models Using Dimensional Analysis.  In: Proceedings of the International Conference on Control and Automation, 2003; 2003. p. 942–946.
  * 35. Stojic MR, Matijevic MS, Draganovic LS.  A robust Smith predictor modified by internal models for integrating process with dead time.  IEEE Transactions on Automatic Control. 2001;46(8):1293–1298.
  * 36. Creveling H, De Paz J, Baladi J, Schoenhals R.  Stability characteristics of a single-phase free convection loop.  Journal of Fluid Mechanics. 1975;67(1):65–84.
  * 37. Wang Y, Singer J, Bau HH.  Controlling chaos in a thermal convection loop.  Journal of Fluid Mechanics. 1992;237:479–498.
  * 38. Boskovic DM, Krstic M.  Global stabilization of a thermal convection loop.  In: Proceedings of the International Conference on Decision and Control. vol. 5. IEEE; 2000. p. 4777–4782.
  * 39. Vazquez R, Krstic M.  Explicit integral operator feedback for local stabilization of nonlinear thermal convection loop PDEs.  Systems & Control Letters. 2006;55(8):624–632.
  * 40. Vazquez R, Krstic M.  Boundary Observer for Output-Feedback Stabilization of Thermal-Fluid Convection Loop.  IEEE Transactions on Control Systems Technology. 2010;18(4):789–797.
  * 41. Smith OJM.  Closed control of loops with dead time.  Chemical Engineering Progress. 1957;53(5):217–219.
  * 42. Mascolo S, Grassi G.  Controlling chaotic dynamics using backstepping design with application to the Lorenz system and Chua’s circuit.  International Journal of Bifurcation and Chaos. 1999;9(07):1425–1434.
  * 43. Lui L, Gao T, Liu Y-J, Tong S.  Time-varying asymmetrical BLFs based adaptive finite-time neural control of nonlinear systems with full state constraints.  IEEE/CAA Journal of Automatica Sinica. 2020;7(05):1335–1343.
  * 44. Liang H, Guo X, Pan Y, Huang T.  Event-Triggered Fuzzy Bipartite Tracking Control for Network Systems Based on Distributed Reduced-Order Observers (Revised manuscript of TFS-2019-1049).  IEEE Transactions on Fuzzy Systems. 2020\.