# Early Discovery of Disappearing Entities in Microblogs

Satoshi Akasaki
The University of Tokyo
&Naoki Yoshinaga
Institute of Industrial Science,
the University of Tokyo
{akasaki, ynaga<EMAIL_ADDRESS>&Masashi Toyoda
Institute of Industrial Science,
The University of Tokyo

###### Abstract

We make decisions by reacting to changes in the real world, in particular, the
emergence and disappearance of impermanent entities such as events,
restaurants, and services. Because we want to avoid missing out on
opportunities or making fruitless actions after they have disappeared, it is
important to know when entities disappear as early as possible. We thus tackle
the task of detecting disappearing entities from microblogs, whose posts
mention various entities, in a timely manner. The major challenge is detecting
uncertain contexts of disappearing entities from noisy microblog posts. To
collect these disappearing contexts, we design time-sensitive distant
supervision, which utilizes entities from the knowledge base and time-series
posts, for this task to build large-scale Twitter datasets111We will release
the datasets (tweet IDs) used in the experiments to promote reproducibility.
for English and Japanese. To ensure robust detection in noisy environments, we
refine pretrained word embeddings of the detection model on microblog streams
of the target day. Experimental results on the Twitter datasets confirmed the
effectiveness of the collected labeled data and refined word embeddings; more
than 70% of the detected disappearing entities in Wikipedia are discovered
earlier than the update on Wikipedia, and the average lead-time is over one
month.

##  1 Introduction

Our daily actions depend on the state of the real world and its changes,
especially changes in the entities available to us. Among the various changes,
the beginning of entities, or emerging entities such as new songs, movies, and
events, are useful for understanding the trends in interests. At the same
time, users need to be made aware of the end or disappearance of entities,
such as closing stores or discontinuing services, as soon as possible so that
they can avoid missing out on opportunities or taking fruitless actions after
the entities are no longer available. It is also important to collect these
entities to maintain knowledge bases (KBs) in which information about the
entities is accumulated. Studies on discovering out-of-KB or emerging entities
Lin et al. (2012); Färber et al. (2016); Wu et al. (2016); Akasaki et al.
(2019) have been successful to an extent, as most of emerging entities have
distinct names and can be characterized by mentions to their unseen names. In
contrast, disappearing entities are not clearly characterized by their
mentions, since they continue to be mentioned even after they disappear.

I’m so sad to hear that Dave Laing has died. Dave was a very accomplished
music industry journalist. But…
---
Can’t believe Google+ is being shut down. It’s like when they just pulled
Google Friends Connect all over again…
RT: Here’s your Demolition Day Planner for Martin Tower. A brief, stray shower
can’t be ruled out before…
Red Bull Air Race World Championship will not continue after 2019. URL
Pristin to disband after 2 years promoting as a group + K - Netz express how
raged they are towards Pledis Ent.
Table 1: Tweets about disappearing entities (bold) with expressions suggesting
their disappearance (italic).

Given such a situation, we take on the new task of discovering disappearing
entities from microblogs where news and personal experiences are widely
shared. To detect the entities’ disappearance, we exploit the specific
expressions that people use when mentioning disappearing entities in
microblogs (Table 1). By capturing these contexts, we can discover a variety
of entities in the early stage even before they disappear. To develop a
dataset of disappearing entities and contexts, we use time-sensitive distant
supervision Akasaki et al. (2019), which collects specific contexts of
entities by utilizing KB entities and timestamps of microblogs. Because this
method requires the timing of the desired contexts, we extract the year of
disappearance for each entity described in Wikipedia and incorporate it into
the distant supervision.

We train a named entity recognition (NER) model on the collected entities and
contexts to discover disappearing entities. However, the NER model performs
poorly for microblogs Derczynski et al. (2017) where posts are short and
noisy, and the training data collected by the distant supervision contains
noise, making it difficult to train a reliable model. We address this issue by
considering that multiple posts are likely to mention the target entity when
it disappears in the real world. Concretely, we utilize these posts to refine
pretrained word embeddings and incorporate them into the NER model. This
enables the model to consider the tokens that frequently appear among multiple
posts and to recognize disappearing entities robustly.

We built large-scale English and Japanese datasets from Twitter using the
proposed time-sensitive distant supervision method. The experimental results
demonstrated that the proposed method outperformed the baseline which simply
collected the latest burst of posts about the disappearing entities as the
disappearing contexts using time-sensitive distant supervision and used them
to train the NER model. In addition, the evaluation of relative recall
indicated that our method successfully found more than 70% of the target
disappearing entities in Wikipedia. Except for entities like persons, whose
Wikipedia articles were updated with little or no delay, our method detected
entities such as services, facilities, and events on average more than 100
days earlier than the update of the disappearance in Wikipedia.

##  2 Definition of Disappearing Entity

In this section, we define the meaning of the term disappearing entity in this
study. We consider the entities’ disappearance to be the disappearance of its
existence from the real world or official announcement of its discontinuation,
with reference to the list of ending entities in
Wikipedia.222https://en.wikipedia.org/w/index.php?title=Category:Endings_by_year&from=2000

We also refer to Akasaki et al. (2019), who reported that emerging entities
have a specific process from their first appearance to when they become known
to the public. During this process, they are referred to with specific
expressions, i.e., emerging contexts. Similar to this, we define disappearing
entities and contexts by focusing on the fact that specific expressions
indicating plans and signs of disappearance appear in contexts not only at the
time of disappearance but also in the process leading up to that time as
follows:

######  Disappearing contexts.

Contexts in which the writers assumed the readers do not know the
disappearance of the entities.

######  Disappearing entities.

Entities still being observed in disappearing contexts.

Table 1 lists examples of disappearing entities and contexts. By properly
identifying these disappearing contexts, we can detect corresponding
disappearing entities in the early stages even before they actually disappear.
This is especially important for entities such as stores that are closing or
services and events that are ending so that users can take action before they
are gone.333Note that how early the disappearance is detected depends on the
entity type. We later confirm the solidness of these definitions by evaluating
the inter-rater agreement of disappearing entities acquired from microblogs
and by demonstrating that the disappearing entities can be detected before the
update of the disappearance in Wikipedia (§ 5.4).

##  3 Related Work

To our knowledge, there have been no studies that detect disappearing entities
in a timely manner, which is the target of our work. We briefly review the
current studies related to our task.

###  3.1 Entity Extraction

Because existing entity related tasks such as NER Nadeau and Sekine (2007);
Lample et al. (2016); Akbik et al. (2018, 2019) and entity linking Shen et al.
(2014); Kolitsas et al. (2018); Martins et al. (2019) do not take into
consideration whether the entities have already disappeared from the real
world, it is difficult to detect disappearing entities by using these
techniques.

As a counterpart to disappearing entities, Akasaki et al. (2019) aimed to find
newly emerging entities.444Although there are other studies that detect
emerging entities like Hoffart et al. (2014) and Wu et al. (2016), they
actually target out-of-KB entities. They focused on the fact that people use
expressions that suggest novelty when mentioning emerging entities and defined
the entities based on these expressions (contexts). They proposed a distant
supervision method called time-sensitive distant supervision to collect
emerging contexts efficiently utilizing KB entities and microblog timestamps
and then developed a NER model using constructed data to detect emerging
entities. Although this is the contrasting task to ours, the same method
cannot be directly because of the difference in contexts handled in this
study.

###  3.2 Temporal Event Extraction

As part of information extraction, several researchers Ritter et al. (2012);
Nguyen and Grishman (2015); Lu and Ng (2017); Liu et al. (2020) have tackled
the task of extracting an event (e.g., birth of a person) and its predefined
attributes and arguments (such as time and places) from the text. Although it
is possible to use this technique to detect disappearing entities, they are
defined as a supervised task based on manual annotation Doddington et al.
(2004); Aguilar et al. (2014) and only a few entity types such as ‘person’ or
‘military conflict’ are chosen for the current annotation. It is thus
unrealistic to extend manual annotations to detect various entity types.

KBP2011 Ji et al. (2010); McClosky and Manning (2012) introduced the temporal
slot filling task, in which the duration of an event is identified from the
given text, entities (e.g., Steve Jobs), and their events (e.g., become CEO).
The events are attributes defined in Freebase, and they include the
disappearance of certain entities such as when a person dies. However, the
types of entities handled in this task are also only a few (‘person’ and
‘organization’). Moreover, although target entities are given in advance for
this task, we have to detect mentions of the entities in the settings of
microblogs.

Figure 2: Sequence labeling with refined word embeddings: We refine pretrained
word embeddings using the Twitter stream on the day of the input post, and
feed them into the NER model for robust training and prediction.

##  4 Proposed Method

The objective of our proposed method is to discover disappearing entities in
microblogs. We target Twitter, where various sources including news articles
and personal posts, are extensively and instantly shared. To accurately
collect contexts of disappearing entities (§ 2), we extract the timings of the
entities’ disappearance described in Wikipedia to elaborate time-sensitive
distant supervision, which is originally designed for detecting emerging
entities Akasaki et al. (2019), for our task. To ensure that an detection
model can make robust predictions using the noisy dataset constructed using
distant supervision and for noisy microblog posts, we refine pretrained word
embeddings to acquire features from multiple occurrences of disappearing
entities and feed them into the model.

###  4.1 Time-sensitive Distant Supervision

While Akasaki et al. were able to gather contexts of emerging entities by
collecting the posts when they first appeared using the timestamps from
microblogs, it is difficult to directly apply this implementation of time-
sensitive distant supervision to disappearing entities because the timing of
their disappearance is not clear. Therefore, we explicitly feed the timing of
the entities’ disappearance extracted from Wikipedia to time-sensitive distant
supervision to collect disappearing entities and their contexts more
accurately (Figure LABEL:disappear:fig:proposedds). The specific procedure is
as follows:

##### Step 1. Collecting disappearing entities

We first collect candidates of disappearing entities while excluding noise. To
collect only entities that have actually disappeared, we refer to the list of
ending entities in Wikipedia11footnotemark: 1 and gather the titles of
articles, categories, and their year of disappearance. We excluded entities
whose year of their first appearance on Twitter was the same as the year of
their disappearance as they could be emerging entities and contaminate the
contexts. We also remove entities that have the ambiguity page so that the
contexts are not contaminated by homographic entities which share the same
namings with other entities (e.g., “Go” can refer to a programming language, a
board game, or a verb).

To acquire entity types, we manually map the category of the article to the
coarse-grained type;555We used this method to obtain coarse-grained types as
there are few mappings of disappearing entities in DBpedia that are defined as
Akasaki et al. (2019). for example, the entity ‘Daft Punk’ is mapped to the
type ‘Group.’

##### Step 2. Collecting disappearing contexts

In contrast to emerging entities, where the first appearance of the entity is
often the emerging context, the latest posts of disappearing entities contain
noisy unrelated contexts because they continue to be mentioned in microblogs
even after they have disappeared. Therefore, for each collected entity, we
utilize the year of disappearance and frequency of appearance on Twitter to
gather their disappearing contexts. Specifically, we randomly collect $k$
posts of the day with the highest number of occurrences in the given year,
assuming that the timing that received the most attention in the year of the
disappearance includes the disappearing contexts.

For each collected entity, Akasaki et al. (2019) collected contexts that
differed from the positive examples as negative examples to avoid overfitting
the NER model when detecting mentions of positive examples. Thus, we similarly
collected random $k$ non-disappearing contexts as negative examples from posts
prior to the year in which we collected positive examples for each entity.
This enables the model to discriminate between disappearing contexts and other
contexts.

We adopt the BILOU scheme Ratinov and Roth (2009) for the NER tags; we label
the disappearing entities in the positive examples with BILU and their entity
types, and label the rest with O.

###  4.2 Finding Disappearing Entities

We train an NER model for finding disappearing entities using the collected
data. Because we target short and noisy microblog posts in this study and
build the dataset using distant supervision, it is difficult to train the
model stably. Thus, we focus on the fact that multiple posts mentioning the
disappearance of the entities often appear when entities disappear from
microblogs. By obtaining features from these posts, we can enhance the
training and prediction of the model even with noisy microblog posts. To do
this, we propose an unsupervised method of refining pretrained word embeddings
as features from multiple posts of a Twitter stream. We also devise a sequence
labeling model for finding disappearing entities using the refined word
embeddings as additional input (Figure 2).

####  4.2.1 Refining Pretrained Word Embeddings

Our objective is to extract features from multiple posts in the Twitter
stream. However, because the surface of the target entity is unknown at the
test time of NER, it is difficult to collect only relevant posts of that
entity from the massive Twitter stream. To address this issue, we use, as the
additional embedding layers of our NER model, pretrained word embeddings
further fine-tuned on the posts on the day of detecting disappearing entities,
together with the original pretrained word embeddings. This enables the
refined word embeddings to reflect the tokens and their co-occurrences in the
Twitter stream of the target day without the need for post selection. The
specific procedure is as follows:

First, we train the base word embeddings $v_{base}$ using the posts prior to
the period in which we collected the data in § 4.1. We use fastText Bojanowski
et al. (2017), a method of constructing word embeddings, to deal with unknown
words.

Next, we use the Twitter stream from each date $d$ of the posts collected in §
4.1 to retrain the word vector $v_{base}$ for obtaining $v_{d}$. The resulting
$v_{d}$ can be interpreted as capturing the temporary semantic change of
$v_{base}$ on date $d$, and it can be treated as an auxiliary input of models
for various tasks.

####  4.2.2 NER with Refined Word Embeddings

We adopt Akbik et al. (2019) model, which uses long short-term memory with
conditional random field (LSTM-CRF) Huang et al. (2015) and flair embeddings
Akbik et al. (2018).666We do not use models pretrained on large corpora such
as BERT because they do not consider timestamps and the data may contain
future data, which leads to unrealistic settings. This model inputs character
embeddings, which are encoded by the pretrained character-based bidirectional-
LSTM language model, and pretrained word embeddings into the word-based
bidirectional-LSTM and makes predictions through the CRF layer.

Based on this model, we prepare another word-based bidirectional-LSTM and
input the refined word embeddings $v_{d}$ corresponding to the date of the
input post. We then concatenate the hidden layers of each LSTM at each time
and feed them into the CRF layer. This enables the model to consider the
global information of the given Twitter stream other than the input post.

TYPE | # entities | # posts | examples of disappearing context (truncated)
---|---|---|---
| Wikipedia categories |  |  |
Person | 780 | 24689 |
| Deaths | 780 | 24689 |  Roger Ailes died of complications of a subdural hematoma after he fell at home, hit his head.
|  |  |  | …to ski one day with Olympic Legend Stein Eriksen at Deer Valley. He passed yesterday at…
|  |  |  | RIP Dave Rosenfield: @USER executive, International League schedule maker, one-time…URL
Creative work | 975 | 24222 |
| American_television_series | 381 | 12413 | RT @USER: See what’s coming up in the Diggnation Finale airing next week…
| British_television_series | 139 | 3650 | Wait…is this episode supposed to be the Metalocalypse finale or the start…
| Web_series | 57 | 1679 | Final Series of McLevy this week on @USER The Scotland office are missing cast & crew …
| Others (36 types) | 398 | 6480 | Crying while watching the last episode of Packed to the Rafters :( such a great tv show
Location | 240 | 3545 |
| Buildings_and_structures | 54 | 949 | @USER: After 67 years Clemson House is gone in seconds. (This version is sped up) URL
| Educational_institutions | 36 | 647 |  Coleman University closing its doors after loss of accreditation URL
| Restaurants | 24 | 286 | …food news of HASH Everyone loves Lynn’s. Lynn’s Paradise Cafe abruptly serves its final meal…
| Others (20 types) | 126 | 1663 | Sad news, crime sleuths. The National Museum of Crime and Punishment in D.C. will close at the…
Group | 1187 | 30384 |
| Musical_groups | 453 | 11559 | …in Flyleaf anymore. Adam’s not in Three Days Grace anymore. My Chemical Romance broke up…
| Retail_companies | 79 | 3187 | The convenience store chain, My Local, is to be placed in administration,9 months after it was sold…
| Airlines | 60 | 1535 | The second hand car giant Carcraft has gone bust, with the loss of around 500 jobs across the UK…
| Others (68 types) | 595 | 14103 |  The Foreign Policy Initiative, a right-leaning foreign-policy think tank, will cease operations…
Event | 186 | 4940 |
| Sporting_events | 111 | 2492 | The Adidas Grand Prix in NY, replaced by Rabat meet in Diamond League series, is set to become…
| Events | 33 | 1462 | Live music blow. MT @TomTilley: C3 confirms Big Day Out will NOT go ahead in 2015.
| Sports_leagues | 16 | 137 | The cancellation of Women’s Professional Soccer League in US is bad news for England & Team…
| Others (2 types) | 26 | 849 |  The Foreign Policy Initiative, a right-leaning foreign-policy think tank, will cease operations…
Service&Product | 709 | 17574 |
| Magazines | 187 | 2998 | was what inspired me to write for mags. RT @USER: Future closes Nintendo Gamer magazine
| Internet_properties | 131 | 5916 | Google is going to shut down Orkut on September 30….Haven’t seen that site since 2007 I…
| Products_and_services | 119 | 3450 |  The PlayStation 3 has ended production according to the official PlayStation Japan website.
| Others (10 types) | 272 | 5210 | Just heard that Yorkshire Radio is no more. Great shame as @USER was pretty much the voice of …
Total | 3213 | 81850 |
Table 2: Statistics of English disappearing entities and disappearing contexts in training data obtained from our Twitter archive by time-sensitive distant supervision. | TYPE | #ent. | #posts
---|---|---
Person | 147 | 422
Creative work | 10 | 23
Location | 46 | 113
Group | 103 | 270
Event | 8 | 19
Service&Product | 43 | 114
Total | 357 | 961
(a) English

TYPE | #ent. | #posts
---|---|---
Person | 73 | 220
Location | 42 | 114
Group | 64 | 173
Event | 9 | 25
Service&Product | 47 | 131
Total | 235 | 663
(b) Japanese

Table 3: Statistics of disappearing entities and disappearing contexts in test
data.

##  5 Experiments

We performed our task of discovering disappearing entities using datasets
built from Twitter.

###  5.1 Data

We constructed the Twitter datasets by time-sensitive distant supervision. We
targeted English and Japanese, which are the top two languages used on Twitter
Alshaabi et al. (2021), and use our archive of Twitter posts that were
retrieved777Timelines of 26 popular Japanese users starting from 2011 have
been continuously collected using user_timeline API, while the user set has
been iteratively expanded to those who were mentioned or retweeted by existing
users. using Twitter APIs. The archive consists of more than 50B posts (32%
are English and 20% are Japanese; this does not deviate much from the actual
data Alshaabi et al. (2021)). We tokenized each post using spaCy (ver.
2.0.12)888https://spacy.io with the en_core_web_sm model for English and MeCab
(ver. 0.996)999https://taku910.github.io/mecab with ipadic (ver. 2.7.0) for
Japanese. We removed URLs, usernames, and hashtags from the text.

In Step 1 of § 4.1, we collected article titles of ending entities in
Wikipedia from 2012 to 2019 using the Wikipedia dump from June 20th, 2021. We
undersampled the types Person and Creative work to 1,000 entities as they are
much larger than the other types. We then excluded entities as described and
carried out Step 2 by setting $k$ to 100, as in Akasaki et al. (2019).

We split the collected data into training data (2012-2018) and test data
(2019). For the training data, we obtained a total of 163,700 English and
150,204 Japanese tweets, including the same number of disappearing and other
contexts for 3,213 English entities and 1,906 Japanese entities, respectively.
For model selection, we used 10% of the training data as the development data.

For the test data, from the collected positive examples of disappearing
entities, we randomly selected three posts for each entity and asked three
annotators (the first author and two graduate students) to judge whether each
context is accompanied by a disappearing context. We adopt the positive
contexts with the answers agreed upon by two or more annotators. Then, for
each entity of the resulting data, we asked the annotators to determine the
non-disappearing contexts using the collected negative examples of the entity
and selected the same number of posts as the positive examples. We obtained an
inter-rater agreement of 0.722 for English and 0.786 for Japanese by Fleiss’s
Kappa, both of which show substantial agreement. As a result, we obtained a
total of 1,922 English and 1,326 Japanese tweets for 357 English entities and
235 Japanese entities, respectively, as the test data.

The resulting disappearing entities and disappearing contexts of training data
(English) and test data (English and Japanese) are shown in Table 2 and 3,
respectively. The entity types that are manually categorized into Person and
Group account for a large proportion. These types of entities, occupied by
persons, musical groups, and companies, are more likely to disappear. Because
we use Wikipedia as the list of disappearing entities, there is a bias in the
composed categories, and some categories are absent in certain languages.
However, we can still detect entities of the missing categories because if the
coarse type of entities is the same, their context tends to be somewhat
similar regardless of their category (e.g., LOCATION types tend to be
mentioned with the term close).

###  5.2 Models

We implemented following models for comparison:

Proposed (TDS + RefEmb): We implemented LSTM-CRF with flair embeddings Akbik
et al. (2018) and proposed refined embeddings (RefEmb) using the training data
constructed by the proposed time-sensitive distant supervision (TDS). We
refined pretrained fastText embeddings for each input post using tweets on the
day of the input post (about 1 to 2M tweets for each day from 2012 to 2019)
and additionally fed them into the LSTM-CRF.

Proposed (TDS): To verify the effectiveness of refined word embeddings, we
removed the part related to RefEmb (green part in Figure 2) from Proposed (TDS
+ RefEmb) and used with the same training data, optimization, and parameters.
This method does not consider the multiple posts of the target day when
recognizing entities.

Baseline: To verify the effectiveness of the constructed training data, we
collected the latest posts of disappearing entities using the original version
of time-sensitive distant supervision, which does not consider the timing of
the entities’ disappearance. Specifically, we follow Akasaki et al. (2019) and
for each ending entity in Wikipedia from 2012 to 2018, we collected 100
retweets of the last day (through 2018) in which the entity appeared more than
ten times as positive examples (disappearing contexts). For negative examples,
we obtained the same number of posts from more than one year before the date
when we collected the positive examples. By using the collected 25,920 posts
for 2,867 entities in English and 6,777 posts for 1,733 entities in Japanese,
we trained LSTM-CRF with flair embeddings using the same optimization and
parameters as Proposed (TDS). Because this method does not consider the timing
of the entities’ disappearance, many noisy contexts may be collected.

Parameter | Value
---|---
Character embedding size (LM) | 30
Dimension of Character Bi-LSTM (LM) | 1024
SGD learning rate (LM) | 20.0
Batch size (LM) | 100
Word embedding size (LSTM-CRF) | 300
Dimension of Word Bi-LSTM (LSTM-CRF) | 256
Batch size (LSTM-CRF) | 32
Dropout (LSTM-CRF) | 0.5
SGD learning rate (LSTM-CRF) | 0.01
Table 4: Hyperparameters of character-based language model (LM) and LSTM-CRF.

| Prec. | Rec. | F1
---|---|---|---
Proposed (TDS + RefEmb) | 0.730 | 0.671 | 0.699
Proposed (TDS) | 0.766 | 0.587 | 0.665
Baseline (TDS) | 0.514 | 0.184 | 0.271
(a) English

| Prec. | Rec. | F1
---|---|---|---
Proposed (TDS + RefEmb) | 0.850 | 0.599 | 0.708
Proposed (TDS) | 0.828 | 0.532 | 0.648
Baseline (TDS) | 0.743 | 0.143 | 0.240
(b) Japanese

Table 5: Overall performance of each method for English and Japanese.

| Prec. | Rec. | F1
---|---|---|---
Person | 0.865 | 0.901 | 0.883
Creative work | 0.480 | 0.500 | 0.490
Location | 0.727 | 0.491 | 0.586
Group | 0.570 | 0.526 | 0.547
Service&Product | 0.566 | 0.409 | 0.475
Event | 0.800 | 0.444 | 0.571
(a) English

| Prec. | Rec. | F1
---|---|---|---
Person | 0.948 | 0.858 | 0.901
Location | 0.814 | 0.569 | 0.670
Group | 0.818 | 0.418 | 0.553
Service&Product | 0.777 | 0.552 | 0.645
Event | 0.250 | 0.042 | 0.071
(b) Japanese

Table 6: Performance of Proposed (TDS + RefEmb) for each coarse type

###  5.3 Settings

##### Implementation and model parameters

We use Keras (ver. 2.3.1)101010https://keras.io for implementing all the
models. For flair embeddings, we set hyperparameters as suggested in Akbik et
al. (2018) and trained the character-based bidirectional LSTM language model
from 2B English tweets for English and 800M Japanese tweets, respectively,
both posted from March 11th, 2011 to December 31st, 2011. The hyperparameters
are listed in Table 4. Using the same tweets, we trained 300-dimensional word
embeddings using fastText111111https://fasttext.cc/ and used them to
initialize the embedding layers of LSTM-CRF by concatenating with flair
embeddings. We optimized all models using stochastic gradient descent and
chose the model in the epoch with the highest F1-score on the development
data.

##### Evaluation methods

To evaluate the accuracy of our method, we apply the models to each post in
the test data constructed in § 5.1 and evaluate the results using the
CoNLL-2003 Sang and De Meulder (2003) schema, which measures precision,
recall, and F1-score.

To evaluate the relative recall and the detection immediacy of our method, we
follow the experiments designed for emerging entities Akasaki et al. (2019).
Specifically, for each ending entity in Wikipedia that disappeared in 2019
(2,608 for English and 763 for Japanese), we applied our method to all of the
posts in 2019 in which each entity appeared (437,816 for English and 202,666
for Japanese). Then we determined how many target entities could be discovered
from the posts and how much earlier those entities could be detected before
the corresponding Wikipedia articles were categorized as ending.

###  5.4 Results and Analysis

##### Overall accuracy of models

Table 5 shows the micro precision, recall, and F1-score for all models. Both
the proposed methods outperformed the baseline, which collected training data
without considering the timing of the entities’ disappearance. The performance
of the baseline is low because it was trained with noisy data. This shows that
our time-sensitive distant supervision successfully collected the disappearing
contexts. Our Proposed (TDS + RefEmb) detected the entities with the highest
accuracy, which means that the refined word embeddings worked effectively. In
particular, the recall was improved in both Japanese and English, indicating
that entities that could not be recognized by only using the features of a
single post can be successfully detected by utilizing multiple posts.

##### Detailed accuracy of optimal model

To analyze the behavior of Proposed (TDS + RefEmb), we show the precision,
recall, and F1-score for each type in Table 6. The accuracy of Person type
entities is high in both Japanese and English. This is likely because the
numerous entities of this type exist in the training data and the person’s
names themselves are easy to recognize from the surface. The Creative work
type is not present in Japanese, and the accuracy for that type is low in
English because the disappearance of these entities is uncertain for the
nature of the type. For example, even when the final episode of a television
drama airs on TV, it remains in various other media. This causes the training
data to be contaminated with diverse contexts, resulting in the model’s low
accuracy. The accuracy for Event type entities is the lowest in Japanese. This
is likely because the number of training data is small and thus the model
could not be sufficiently trained.

##### Relative recall and detection immediacy

TYPE | # entities | # found (%) | lead-days
---|---|---|---
|  |  |  |  | mean |  (median)
Person | 1838 | 1668 | (90.75%) | 21 | (0)
Creative work | 351 | 123 | (35.04%) | 173 | (64)
Location | 73 | 38 | (52.05%) | 150 | (46)
Group | 163 | 99 | (60.74%) | 195 | (131)
Service&Product | 93 | 47 | (50.54%) | 172 | (91)
Event | 25 | 8 | (32.00%) | 85 | (35)
Unmapped | 65 | 34 | (52.31%) | 237 | (171)
Total | 2608 | 2017 | (77.34%) | 49 | (0)
Total (w/o Person) | 770 | 349 | (45.32%) | 139 | (77)
(a) English

TYPE | # entities | # found (%) | lead-days
---|---|---|---
|  |  |  |  | mean |  (median)
Person | 515 | 446 | (86.60%) | 31 | (0)
Location | 48 | 31 | (64.58%) | 149 | (117)
Group | 121 | 56 | (46.28%) | 181 | (147)
Service&Product | 63 | 43 | (68.25%) | 154 | (98)
Event | 16 | 6 | (37.50%) | 183 | (171)
Total | 763 | 582 | (76.28%) | 63 | (0)
Total (w/o Person) | 248 | 136 | (54.84%) | 132 | (101)
(b) Japanese

Table 7: Relative recall and time advantage over entity types of English and
Japanese disappearing entities detected with Proposed (TDS + RefEmb).

Table 7 shows the distribution of the types of target entities, detection
ratio, and lead time against the Wikipedia update time for both languages.
Overall, Proposed (TDS + RefEmb) detected 2,017 (77.34%) English and 582
(76.28%) Japanese disappearing entities. More than 80% of Person entities were
detected in both languages, while the other types were found 45.32% for
English and 54.84% for Japanese. Note that some target entities are low
frequency in our Twitter archive and do not appear in disappearing contexts.
Because our method utilizes such disappearance signals as clues, it is
difficult to discover entities that are mentioned without disappearing
contexts. This is the current limitation of our method.

For detection immediacy, we confirmed that 77.34% of the discovered English
entities (1,560 out of 2,017) and 85.39% of the discovered Japanese entities
(497 out of 582) were detected earlier than their update in Wikipedia.
Entities that were slow to be detected were often either mentioned
infrequently with disappearing contexts or their updates were unusually fast.
The mean (and median) lead days of the first day when our method detected each
entity against their update date were 49 (and 0.13) for English and 63 (and
0.44) days for Japanese. In particular, for other types of entities other than
person, the lead days were 139 days (and 77) for English and 132 days (and
101) for Japanese. This demonstrates the detection immediacy of our method.
Our method could discover entities such as Durgin-Park (restaurant), Rolling
Acres Mall (shopping mall), and CiteULike (Internet property), which are the
types of entities whose disappearance is important to know in advance. This
should prevent users from making fruitless actions or missing out on
opportunities because the average lead time for these types is more than 100
days. Interestingly, the updates for Person type entities in Wikipedia are
faster for both languages. Since this kind of uncontrolled disappearance
(death) occurs suddenly, it is difficult to detect these entities before they
disappear. It also shows that only certain types of entities are updated
faster in Wikipedia.

##  6 Conclusion

We introduced the task of discovering disappearing entities in microblogs (§
1, § 2, and § 3). To deal with the uncertainty of entity disappearance, we
considered the year of disappearance in Wikipedia and fed it into time-
sensitive distant supervision (§ 4.1). To perform the detection from noisy
microblog posts, we proposed the method of refining pretrained word embeddings
using the Twitter stream (§ 4.2). Experimental results demonstrated that our
method outperformed the baseline method and successfully found more than 70%
of the target disappearing entities in Wikipedia and they were detected more
than a month earlier than the update of the disappearance in Wikipedia (§
5.4).

We plan to extract information such as the specific time of disappearance to
make the discovered entities more useful for applications.

## Ethical Considerations

The dataset was collected using Twitter’s official
API121212https://developer.twitter.com/en/docs/twitter-api and in compliance
with Twitter’s terms of use. Only the tweet IDs of the tweets used in the
experiments will be made public, and we ensure that their redistribution is in
compliance with Twitter’s developer
policy.131313https://developer.twitter.com/en/developer-terms/agreement-and-
policy Researchers cannot collect deleted tweets or tweets of private users,
thus protecting user privacy.

## References

  * Aguilar et al. (2014) Jacqueline Aguilar, Charley Beller, Paul McNamee, Benjamin Van Durme, Stephanie Strassel, Zhiyi Song, and Joe Ellis. 2014.  A comparison of the events and relations across ace, ere, tac-kbp, and framenet annotation standards.  In _Proceedings of the Second Workshop on EVENTS: Definition, Detection, Coreference, and Representation_ , pages 45–53.
  * Akasaki et al. (2019) Satoshi Akasaki, Naoki Yoshinaga, and Masashi Toyoda. 2019.  Early discovery of emerging entities in microblogs.  In _Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI)_ , pages 4882–4889.
  * Akbik et al. (2019) Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019.  Pooled contextualized embeddings for named entity recognition.  In _Proceedings of the 18th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , pages 724–728.
  * Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018.  Contextual string embeddings for sequence labeling.  In _Proceedings of the 27th International Conference on Computational Linguistics (COLING)_ , pages 1638–1649.
  * Alshaabi et al. (2021) Thayer Alshaabi, David Rushing Dewhurst, Joshua R. Minot, Michael V. Arnold, Jane L. Adams, Christopher M. Danforth, and Peter Sheridan Dodds. 2021.  The growing amplification of social media: measuring temporal and social contagion dynamics for over 150 languages on twitter for 2009–2020.  _EPJ Data Science_ , 10(1).
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ , 5:135–146.
  * Derczynski et al. (2017) Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017.  Results of the WNUT2017 shared task on novel and emerging entity recognition.  In _Proceedings of the 3rd Workshop on Noisy User-generated Text (WNUT)_ , pages 140–147.
  * Doddington et al. (2004) George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie M Strassel, and Ralph M Weischedel. 2004.  The automatic content extraction (ace) program-tasks, data, and evaluation.  In _Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC)_ , pages 837–840.
  * Färber et al. (2016) Michael Färber, Achim Rettinger, and Boulos Asmar. 2016.  On emerging entity detection.  In _Proceedings of the 20th International Conference on Knowledge Engineering and Knowledge Management (EKAW)_ , pages 223–238.
  * Hoffart et al. (2014) Johannes Hoffart, Yasemin Altun, and Gerhard Weikum. 2014.  Discovering emerging entities with ambiguous names.  In _Proceedings of the 23rd International Conference on World Wide Web (WWW)_ , pages 385–396.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional lstm-crf models for sequence tagging.  _arXiv preprint arXiv:1508.01991_.
  * Ji et al. (2010) Heng Ji, Ralph Grishman, Hoa Trang Dang, Kira Griffitt, and Joe Ellis. 2010.  Overview of the tac 2010 knowledge base population track.  In _Third text analysis conference (TAC 2010)_ , volume 3, pages 3–3.
  * Kolitsas et al. (2018) Nikolaos Kolitsas, Octavian-Eugen Ganea, and Thomas Hofmann. 2018.  End-to-end neural entity linking.  In _Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL)_ , pages 519–529.
  * Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.  Neural architectures for named entity recognition.  In _Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , pages 260–270.
  * Lin et al. (2012) Thomas Lin, Oren Etzioni, et al. 2012.  No noun phrase left behind: detecting and typing unlinkable entities.  In _Proceedings of the 2012 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 893–903.
  * Liu et al. (2020) Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. 2020.  Event extraction as machine reading comprehension.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1641–1651.
  * Lu and Ng (2017) Jing Lu and Vincent Ng. 2017.  Joint learning for event coreference resolution.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 90–101.
  * Martins et al. (2019) Pedro Henrique Martins, Zita Marinho, and André FT Martins. 2019.  Joint learning of named entity recognition and entity linking.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL-SW)_ , pages 190–196.
  * McClosky and Manning (2012) David McClosky and Christopher D Manning. 2012.  Learning constraints for consistent timeline extraction.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_ , pages 873–882.
  * Nadeau and Sekine (2007) David Nadeau and Satoshi Sekine. 2007.  A survey of named entity recognition and classification.  _Lingvisticae Investigationes_ , 30(1):3–26.
  * Nguyen and Grishman (2015) Thien Huu Nguyen and Ralph Grishman. 2015.  Event detection and domain adaptation with convolutional neural networks.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP)_ , pages 365–371.
  * Ratinov and Roth (2009) Lev Ratinov and Dan Roth. 2009.  Design challenges and misconceptions in named entity recognition.  In _Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL)_ , pages 147–155.
  * Ritter et al. (2012) Alan Ritter, Oren Etzioni, and Sam Clark. 2012.  Open domain event extraction from twitter.  In _Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD)_ , pages 1104–1112.
  * Sang and De Meulder (2003) Erik Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the 7th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (NAACL-HLT)_ , pages 142–147.
  * Shen et al. (2014) Wei Shen, Jianyong Wang, and Jiawei Han. 2014.  Entity linking with a knowledge base: Issues, techniques, and solutions.  _IEEE Transactions on Knowledge and Data Engineering_ , 27(2):443–460.
  * Wu et al. (2016) Zhaohui Wu, Yang Song, and C Lee Giles. 2016.  Exploring multiple feature spaces for novel entity discovery.  In _Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI)_ , pages 3073–3079.


marginparsep has been altered.
topmargin has been altered.
marginparwidth has been altered.
marginparpush has been altered.
The page layout violates the ICML style. Please do not change the page layout,
or include packages like geometry, savetrees, or fullpage, which change it for
you. We’re not able to reliably undo arbitrary changes to the style. Please
remove the offending package(s), or layout-changing commands and try again.

DDDM: a Brain-Inspired Framework for Robust Classification

XiYuan Chen * 1 2  Xingyu Li * 3  Yi Zhou 4 3  Tianming Yang 1 3

††footnotetext: *Equal contribution 1Institute of Neuroscience, Center for
Excellence in Brain Science and Intelligence Technology, Chinese Academy of
Sciences, Shanghai, China 2University of Chinese Academy of Sciences, Beijing,
China 3Shanghai Center for Brain Science and Brain-Inspired Technology,
Shanghai, China 4National Engineering Laboratory for Brain-inspired
Intelligence Technology and Application, School of Information Science and
Technology, University of Science and Technology of China, Hefei, China.
Correspondence to: Yi Zhou<EMAIL_ADDRESS>Tianming Yang
<EMAIL_ADDRESS>Copyright © by the author(s). Copying permitted for private
and academic purposes.

###### Abstract

Despite their outstanding performance in a broad spectrum of real-world tasks,
deep artificial neural networks are sensitive to input noises, particularly
adversarial perturbations. On the contrary, human and animal brains are much
less vulnerable. In contrast to the one-shot inference performed by most deep
neural networks, the brain often solves decision-making with an evidence
accumulation mechanism that may trade time for accuracy when facing noisy
inputs. The mechanism is well described by the Drift-Diffusion Model (DDM). In
the DDM, decision-making is modeled as a process in which noisy evidence is
accumulated toward a threshold. Drawing inspiration from the DDM, we propose
the Dropout-based Drift-Diffusion Model (DDDM) that combines test-phase
dropout and the DDM for improving the robustness for arbitrary neural
networks. The dropouts create temporally uncorrelated noises in the network
that counter perturbations, while the evidence accumulation mechanism
guarantees a reasonable decision accuracy. Neural networks enhanced with the
DDDM tested in image, speech, and text classification tasks all significantly
outperform their native counterparts, demonstrating the DDDM as a task-
agnostic defense against adversarial attacks.

##  1 Introduction

Figure 1:  Schematic diagram of the Dropout-based Drift-Diffusion Model
(right) and its brain science counterpart (left). In DDDM, outputs $p_{i}$
from the stochastic copies of an arbitrary neural network simulate the noisy
temporal neural signal $o_{t_{i}}$ in the brain. In both, the series of
outputs/signals are passed to a similar evidence accumulation mechanism (as
described by the DDM) for a robust output.

Deep learning has shown outstanding performance in many areas, including
image, audio , and text classification. Nevertheless, deep learning lacks
robustness; that is, deep artificial neural networks are susceptible to
manipulations of inputs. Small perturbations on inputs that are imperceptible
to humans may result in significant differences in outputs of deep neural
networks Goodfellow et al. (2015). Such susceptibility is not only of
theoretical interest but also represents a severe practical problem, which
causes security issues in real-world applications, including face recognition
and autonomous driving.

Recently, a huge amount of effort has been dedicated to addressing this issue.
Adversarial training Madry et al. (2019) is considered to be the most
promising. It adds adversarial samples to the training set and automatically
lets the network model adapt to them through learning. Despite these efforts,
current defenses against adversarial attacks are still far from satisfactory.
Most defenses only target one or a few types of attacks and are often
restricted to particular tasks, such as image classification. Consequently, it
is unclear whether they can be applied to other kinds of attacks or tasks.

In contrast, humans and animals are much less vulnerable. Even though time-
limited human subjects can be misled by deliberately crafted images, they
successfully restore accuracy by spending more time facing these uncertain and
noisy inputs Elsayed et al. (2018). The more uncertain the input is, the
longer the brain takes to make a decision. This ability of trading decision
speed for accuracy has been well studied in humans and animals by researchers
in neuroscience and psychology Ratcliff & Rouder (1998); Roitman & Shadlen
(2002). In particular, the celebrated Drift-Diffusion Model (DDM) and its
variants stand as one of the most successful models that account for the
decision-making process in humans and animals. The DDM describes a sequential
sampling process in which signals with both noises from the external sources
and the internal neuronal noises are accumulated as evidence for decisions.
The decision is made once the accumulated evidence reaches a pre-determined
threshold. A high threshold leads to accurate but slow decisions, while a low
threshold results in fast but inaccurate decisions. Neuronal activities in
decision-making-related brain areas exhibit response patterns described by the
DDM Roitman & Shadlen (2002).

11footnotetext:  Implementation: https://github.com/XiYuan68/DDDM.

The ability to trade speed for accuracy characterizes one of the most striking
differences between biological and artificial neural networks. Enhancing
artificial neural networks with a dynamic inference process similar to those
in human and animal brains would potentially improve their robustness. Guided
by this intuition, we propose the Dropout-based Drift-Diffusion Model (DDDM)
as a general framework for enhancing the robustness of arbitrary neural
networks. Depicted in Figure 1, in contrast to adversarial training, which
adds noises to inputs during the training phase, DDDM adds noises to neural
networks via random dropouts during the test phase, resulting in multiple
stochastic copies of the original one. Then, DDDM employs a DDM-like evidence
accumulation mechanism to make the decision based on these outputs.

We conduct experiments on three types of datasets, including MNIST Lecun et
al. (1998) and CIFAR10 Krizhevsky (2009) for image classification, the
SpeechCommands Warden (2018) for audio classification, and the IMDB dataset
Maas et al. (2011) for text classification. We compare the networks’
performance with and without the DDDM against a variety of adversarial
attacks, including both the white-box and the black-box ones. The experimental
results show that the DDDM improves the robustness of the network, providing
defense against all adversarial attacks tested on all three datasets.

##  2 Related Work

Adversarial training (AT) Goodfellow et al. (2015); Madry et al. (2019) is one
of the most popular adversarial defense methods. It augments the clean data
with adversarial ones during the trianing process. There are works that try to
improve robustness by introducing additional noise in their models. For
example, Guo et al. (2018) applied transformation on images before feeding the
network, including image quilting, total variance minimization, JPEG
compression, and bit-depth reduction. Similar to our work, Dhillon et al.
(2018) randomly dropped neurons with a weighted distribution. Note that all
the defense methods mentioned above focus on either filtering out adversarial
perturbations or creating obfuscated gradients Athalye et al. (2018), leaving
room for more sophisticated attacks. On the contrary, our framework injects
randomness into the network model during the testing phase, and the resulting
stochastic predictions are agnostic to the type of adversarial attacks.
Furthermore, a temporal evidence accumulation mechanism is employed to offset
the accuracy reduction due to the added randomness and retain a good
performance with robustness.

In neuroscience, recent advances in the study of perceptual decision-making
have started to reveal the underlying neural mechanism Gold & Shadlen (2007).
The random-dot motion (RDM) direction discrimination task is one of the most
popular tasks for studying decision-making with noisy evidence Roitman &
Shadlen (2002). In this task, subjects are asked to judge whether a patch of
noisy dots moves toward left or right. The subjects’ decision accuracies and
response times can be well described by the DDM Ratcliff & McKoon (2008).
Furthermore, it was demonstrated that neurons in the posterior parietal cortex
increased their firing when evidence toward their favored choice was being
accumulated Roitman & Shadlen (2002). The response increase correlated with
the quality of evidence, which may be quantified as the log-likelihood ratio
Gold & Shadlen (2007); Yang & Shadlen (2007), and reflected the evidence
accumulation process. Additional studies extended DDM into probabilistic
inferences based on images and identified a prefrontal-parietal circuitry that
carries out evidence accumulation process in probabilistic inferences Yang &
Shadlen (2007). Response patterns that reflect evidence accumulation were also
found in many other brain regions, including multiple regions in the
prefrontal cortex and the basal ganglia, suggesting evidence accumulation is a
universal decision-making mechanism employed by the brain Gold & Shadlen
(2007). DDM and the related sequential sampling models in neuroscience
describe the brain’s solutions for optimizing decision-making with noisy
inputs. However, the DDM has not been used together with deep neural networks
in the machine learning field and has not been considered as a defense
mechanism against adversarial attacks.

##  3 Model Description

###  3.1 Test-Phase Dropout

Dropout was initially proposed by Hinton et al. (2012) to address the problem
of overfitting. In this paper, we use it to introduce randomness in the
system. It simulates the internal neuronal noise in synaptic transmissions
Rusakov et al. (2020). With dropouts applied temporally, the original network
model effectively extends to an ensemble of stochastic copies, whose outputs
serves as the evidence. Let $h_{a,b}$ be an arbitrary deep neural network with
one or several dropout layers. The dropout layers share the same dropout rate.
Subscript $a$ denotes the dropout rate used for model training and validation,
and $b$ denotes the dropout rate at the test phase. We refer to $h_{a,b}$ as
the dropout classifier in the following context to emphasize the involvement
of the test-phase dropout mechanism.

###  3.2 Drift-Diffusion Model

Figure 2:  A diagram of the drift-diffusion model (DDM). At each moment, the
noisy momentary evidence is accumulated until it hits one of the two decision
thresholds ($A$ and $-A$). The decision threshold and the mean and the
variance of the momentary evidence together determine the speed and accuracy
of the decision.

In a typical two-choice scenario, the canonical Drift-Diffusion Model (DDM)
(see Figure 2) describes decision-making as a process in which evidence that
is noisy and fluctuates over time (momentary evidence) is accumulated toward a
preset threshold $A$. Mathematically, DDM corresponds to the following
stochastic differential equation

$\tau\dot{X}=\mu+\xi(t),$ (1)

where $X$ is known as the decision variable representing the cumulated
evidence, and $\mu$ is the drift or the mean of the noisy momentary evidence.
The Gaussian process $\xi(t)$, with mean $0$ and variance $\sigma^{2}$,
captures the uncertainty in the evidence. Coefficient $\tau$ is the
characteristic time scale. The decision variable $X$ evolves until reaching
one of the two thresholds $\pm A$ at which the corresponding decision is made.
These thresholds correspond to the choice options, and the time needed for the
accumulated evidence to reach a particular threshold reflects the response
time (RT).

The canonical DDM can be extended to decisions with more than two
alternatives. Equation 1 can be derived from a set of stochastic differential
equations Roxin (2019). Following this route, it is straightforward to
increase the number of choices by extending the number of differential
equations. Another route to extend the DDM to the multi-choice domain models
the decision-making as a Bayesian multiple sequential probability ratio test
(MSPRT). It has been demonstrated the two routes are equivalent Roxin (2019).

For our purpose to integrate the DDM with the test-phase dropout classifier,
we use the Bayesian MSPRT route. Suppose there are $n$ alternatives, and
$z_{i}(t)$ denotes the instantaneous evidence for choice $i$. $z_{i}(t)$ is
drawn from a Gaussian distribution with mean $M_{i}$ and variance
$\sigma^{2}$. Let $\mathcal{O}(T):=\\{z_{i}(t):1\leq i\leq n,1\leq t\leq T\\}$
be the collection of observations up to a time $T$, and $H_{i}$ denote the
hypothesis that choice $i$ has the largest mean. Within the Bayesian MSPRT
framework, one seeks to make a decision based on the posterior
$\text{Pr}(H_{i}|\mathcal{O}(T))$. With defining

$X_{k}(t):=\sum_{i}^{k}y_{i}(t)-ky_{k+1}(t),\quad 1\leq k\leq n-1,$

where $y_{i}(t):=\int_{0}^{t}dt^{\prime}\,z_{i}(t^{\prime})$ and $X_{k}$ is
the decision variable for the $k$th class, measuring how likely the original
input belongs to class k given the accumulated evidence. It can be shown that
$X_{k}$ satisfies a stochastic differential equation of the same type as
Equation 1. We refer the interested readers to Roxin (2019) for a complete
deduction.

| clean | FGSM | PGD |  $L_{2}$ C&W |  $L_{2}$ DF | Salt&Pepper | Uniform | Spatial | Square
---|---|---|---|---|---|---|---|---|---
$h_{0,0}$ (%) | 98.10 | 12.10 | 0.00 | 1.70 | 42.70 | 0.20 | 94.50 | 0.40 | 19.50
$h^{*}_{a,b}$ (%) | 99.20 | 50.70 | 43.30 | 96.70 | 67.90 | 96,90 | 97.10 | 34.80 | 78.00
$(a,b)^{*}$ | $(0.2,0.0)$ | $(0.0,0.6)$ | $(0.0,0.6)$ | $(0.4,0.4)$ | $(0.2,0.6)$ | $(0.2,0.4)$ | $(0.2,0.2)$ | $(0.6,0.8)$ | $(0.0,0.2)$
$H^{*}_{a,b}$ (%) | 99.16 | 73.12 | 74.85 | 98.89 | 95.04 | 98.89 | 98.24 | 65.22 | 88.93
$(a,b)^{*}$ | $(0.4,0.2)$ | $(0.0,0.6)$ | $(0.0,0.8)$ | $(0.2,0.4)$ | $(0.6,0.8)$ | $(0.4,0.4)$ | $(0.2,0.4)$ | $(0.4,0.8)$ | $(0.0,0.6)$
$h_{a^{*},b^{*}}$ (%) | 85.90 | 49.70 | 36.40 | 85.40 | 67.90 | 86.70 | 79.10 | 30.20 | 64.10
$H_{a^{*},b^{*}}$ (%) | 98.20 | 71.09 | 51.68 | 97.94 | 92.30 | 98.08 | 97.36 | 36.73 | 87.15
Table 1: Performance of the DDDM under eight adversarial attacks on MNIST. The
optimal performance of the DDDM $H^{*}_{a,b}$ for individual attacks are
compared to the ones of the dropout classifier $h^{*}_{a,b}$. The accuracy of
the undefended model $h_{0,0}$ is included as the baseline. For each case, we
show the prediction accuracy of the individual optimal models
$h^{*}_{a,b}$/$H^{*}_{a,b}$ as well as their corresponding optimal dropout
rate pair $(a,b)^{*}$. We also include the overall performance of the DDDM
$H_{a^{*},b^{*}}$ and the dropout classifier $h_{a^{*},b^{*}}$. The global
optimal dropout rate pair $(a^{*},b^{*})$ is chosen as $(0.2,0.6)$.

###  3.3 Dropout-based Drift-Diffusion Model

There are two components in the design of the Dropout-based Drift-Diffusion
model (DDDM). The first component turns the static one-shot inference process
into a dynamic, noisy inference process. The noisy predictions from the
stochastic copies of neural networks simulate the series of temporal signals
in the biological brain. Presently, we implement the dynamic inference by
introducing the test-phase dropout, which will randomly disturb the loss
landscape and potentially make adversarial attacks harder to succeed. This
results in a dropout classifier whose outputs are subject to further
processing.

For the second component, we take the noisy outputs from the test-phase
dropout classifier as evidence and apply DDM with the Bayesian MSPRT
implementation. More precisely, for a dropout classifier $h_{a,b}$, let $C$ be
the number of categories and $x$ be an input. We arrange $L$ predictions to
form a trial

$T=\begin{bmatrix}p_{11}&\cdots&p_{1C}\\\ \vdots&\ddots&\vdots\\\
p_{L1}&\cdots&p_{LC}\end{bmatrix},$

where $p_{ij}$ is the $i$-th predicted confidence of $x$ belonging to label
$j$. Let $T_{i}$ denote the $i$-th row vector in matrix $T$, we compute the
posterior at time step $t$

$P(j,t):=\text{Pr}(H_{j}|\\{T_{1},\dots,T_{t}\\}),\,1\leq j\leq C,1\leq t\leq
L.$

Let $RT\leq L$ be the smallest time step at which there exists a label
$winner$ satisfying $P({winner},RT)\geq A$. We say that $x$ belongs to the
label $winner$ with a response time $RT$. If the threshold is never surpassed
during the whole trial, we will force a decision by assigning the label winner
to be the one with the largest decision variable at time step L, i.e.,
$P({winner},L)=\max(P(1,L),P(2,L),\dots,P(C,L))$. In practice, we apply
multiple trials and an appropriate decision threshold $A$ is chosen to offset
the noise introduced by dropouts and ensure overall accuracy. To compute the
posterior $P(i,j)$, we apply Bayes theorem and assume equal priors for all
hypotheses $H_{i}$. The problem is then reduced to computing the likelihoods
$\text{Pr}(\\{T_{1},\dots,T_{t}\\}|H_{i})$, which are approximately evaluated
over the clean training data. The detailed description of our implementation
can be found in the supplementary material.

In the following context, we will denote the dropout classifier equipped with
DDM as $H_{a,b}$ and refer to it as DDDM for simplicity.

##  4 Experiments and Results

The primary goal of the present experiments is to evaluate the robustness of
our DDDM model. Our experiments cover three different modalities of data,
i.e., image, audio, and text. Under all types of inputs, the DDDM is
demonstrated to be effective in defending adversarial attacks.

For all the tasks, we run experiments over different combinations of dropout
rates $(a,b)$, both taking values from the set $\\{0,0.2,0.4,0.6,0.8\\}$.
Therefore, twenty-five classifiers are tested in each case. In implementing
DDM, we draw $100$ predictions from each classifier on each generated
adversarial example. Then, $10$ trials of the length $L=25$ are randomly
sampled from those predictions. These trials are fed into the evidence
accumulation mechanism with a decision threshold $A=0.99$ for the final
prediction.

###  4.1 Images Classification

####  4.1.1 MNIST Dataset

In the experiments, we adopt the same network architecture as in Madry et al.
(2019) with an additional dropout layer that follows each layer except the
input/output ones.

To comprehensively evaluate the effectiveness of DDDM in defending adversarial
attacks, we considered four white-box attacks and four black-box attacks as
listed below: the Fast Gradient Sign Method (FGSM) attack Goodfellow et al.
(2015), the Projected Gradient Descent (PGD) attack Madry et al. (2019), the
$L_{2}$ Carlini and Wagner ($L_{2}$ C&W) attack Carlini & Wagner (2017), the
$L_{2}$ DeepFool attack Moosavi-Dezfooli et al. (2016), the Salt and Pepper
attack, the $L_{\infty}$ uniform noise attack Rauber & Bethge (2020), the
Spatial attack Engstrom et al. (2019) and the Square attack Andriushchenko et
al. (2020). The details of attack settings can be found in the supplementary
material.

In Table 1, we compare the optimal DDDMs ($H^{*}_{a,b}$) with the optimal
dropout classifiers ($h^{*}_{a,b}$) on individual attacks and the clean data.
The undefended model ($h_{0,0}$) is included as the baseline. To demonstrate
the effectiveness of DDDM under the combination of all attacks, we also
include the results for $H_{a^{*},b^{*}}$ and $h_{a^{*},b^{*}}$, i.e., the
DDDM and the dropout classifier with a single optimal dropout rate pair
$(a^{*},b^{*})$ across all the attacks. $(a^{*},b^{*})$ is determined with the
criterion that the corresponding $H_{a^{*},b^{*}}$ should try its best to
avoid sacrificing the clean accuracy and, at the same time, provide as much
defense as possible across different attacks. The results indicate that the
test-phase dropouts effectively neutralize the adversarial perturbations
against all attacks in our experiments, and the DDM retains the accuracy
through evidence accumulation. Furthermore, $H_{a^{*},b^{*}}$ outperforms
$h_{a^{*},b^{*}}$ as well as $h_{0,0}$ on all the cases.

Figure 3: Variation of accuracy (solid lines) and response time (the dashed
line) against increasing perturbation size ($\epsilon$). We compared three
models: the baseline model $h_{0,0}$, the dropout classifier without DDM
$h_{0.0,0.8}$, and the dropout classifier with DDM $H_{0.0,0.8}$ under the PGD
attack on MNIST.

##### Trade-off between Response Time and Accuracy.

Animals and humans spend more effort in making difficult decisions, and the
DDDM exhibits the same behavior. While the randomness in the test-phase
dropout classifier neutralizes the adversarial perturbations, the DDM helps
retain a decent accuracy by spending more time in inference. To demonstrate
this trade-off, we plot in Figure 3 the accuracies and response times with
respect to different adversarial perturbation sizes, comparing DDDM to the
dropout classifier as well as the undefended baseline model. The response time
is measured by the number of forward passes. The accuracy of the baseline
model drops rapidly as the perturbation size increase. On the other hand, both
the DDDM and the test-phase dropout classifier retain relatively high accuracy
even when the perturbation is large. For small perturbations, the test-phase
dropout classifier’s accuracy curve lies well below the baseline model’s
curve. In contrast, the performance of DDDM dominates on almost the whole
perturbation range except for near 0. Finally, the response time increases
monotonically with perturbation level, demonstrating a trade for accuracy with
time.

| clean | PGD |  $L_{2}$ DF | Spatial
---|---|---|---|---
$h_{0,0}$ (%) | 89.71 | 5.24 | 67.96 | 8.93
$h^{*}_{a,b}$ (%) | 89.89 | 68.46 | 71.53 | 40.98
$(a,b)^{*}$ | $(0.2,0.0)$ | $(0.8,0.8)$ | $(0.0,0.2)$ | $(0.4,0.8)$
$H^{*}_{a,b}$ (%) | 89.67 | 68.46 | 79.01 | 64.26
$(a,b)^{*}$ | $(0.6,0.4)$ | $(0.8,0.8)$ | $(0.0,0.8)$ | $(0.0,0.8)$
$h_{a^{*},b^{*}}$ (%) | 86.88 | 35.87 | 67.57 | 36.92
$H_{a^{*},b^{*}}$ (%) | 89.30 | 36.76 | 70.00 | 37.59
Table 2: Performance of the DDDM under three attacks on the CIFAR10 dataset.
The same as in Table 1, we include the performance under individual attacks as
well as the overall performance with the global optimal dropout rate pair
$(a^{*},b^{*})=(0.6,0.8)$.

####  4.1.2 CIFAR10 Dataset

We further carry out experiments on the CIFAR10 Krizhevsky (2009) dataset to
challenge our approach under a more realistic scenario. To implement the test-
phase dropout classifier, we use the VGG16 architecture Simonyan & Zisserman
(2015) without batch normalization. A dropout layer is added to each of the
last six convolutional layers. We will discuss more the method of adding
dropout layers in the supplementary material.

The results are summarized in Table 2, which includes three effective attacks
in our test: PGD, L2DF, and Spatial. Note that these attacks cover both white-
box and black-box attack categories. Again, DDDM effectively defends all the
attacks with only a minor sacrifice on the clean accuracy. Compared to the
MNIST case, while $H_{a^{*},b^{*}}$ keeps outperforming $h_{a^{*},b^{*}}$,
their performance gaps become much smaller. This may be attributed to the fact
that under a large model such as VGG16, six dropout layers do not introduce
sufficient randomness in the dropout classifier. Hence, DDM does not work at
its full power.

###  4.2 Audio Classification

| clean | imperceptible
---|---|---
$h_{0,0}$ (%) | 86.73 | 5.18
$h_{a^{*},b^{*}}$ (%) | 82.91 | 68.36
$H_{a^{*},b^{*}}$ (%) | 85.02 | 70.62
Table 3: Performance of the DDDM. Similar to Table 1 but for the imperceptible
attack on the SpeechCommands dataset. Since there is only one attack, we only
includes the overall performance with the global optimal dropout rate pair
$(a^{*},b^{*})=(0.4,0.4)$.

To evaluate the cross-domain performance of DDDM, we consider the audio
classification task on SpeechCommands Warden (2018) dataset, which contains 35
keywords and 105829 audio clips of those keywords. In the audio domain,
present works focus on the classification and recognition of audio clips,
which serve as the base of various applications such as voice input and voice
assistants. The existing audio adversarial attacks Carlini & Wagner (2018);
Qin et al. (2019) severely endangers the safety of such automatic speech
recognition (ASR) systems.

In our test, we adopt a simplified version of the DeepSpeech2 model Amodei et
al. (2015) against the attack proposed by Qin et al. (2019), which adds human-
imperceptible perturbations to the audio waveform to mislead the model
predictions. Our DeepSpeech2 model includes a Mel-spectrogram conversion
layer, followed by one 1D convolutional layer, two LSTMs, and two fully-
connected layers. Dropout is only applied once after the convolutional layer.
After training, we attacked the model with the adversarial perturbation bound
of 0.05.

Results in Table 3 show that DDDM providea defense against the imperceptible
attack with only a minor decrease in clean accuracy. Comparing the dropout
classifier $h_{a^{*},b^{*}}$ with the DDDM $H_{a^{*},b^{*}}$, the latter show
better performance under clean and adversarial case situations. We notice that
the gap between $h_{a^{*},b^{*}}$ and $H_{a^{*},b^{*}}$ is small. This is
similar to the case for CIFAR10 and could be attributed to the insufficient
number of dropout layers.

###  4.3 Text Classification

| clean | TextBugger
---|---|---
$h_{0,0}$ (%) | 88.44 | 1.7
$h_{a^{*},b^{*}}$ (%) | 89.16 | 22.10
$H_{a^{*},b^{*}}$ (%) | 89.41 | 86.75
Table 4: Performance of the DDDM. Similar to Table 3 but for the TextBugger
attack on the IMDB dataset. The global optimal dropout rate pair is
$(a^{*},b^{*})=(0.4,0.4)$.

At last, we evaluate our approach on the text classification task. Natural
languages consist of discrete characters and words. The process of mapping
tokens to embeddings prevents directly applying attack methods from the image
domain on textual classifiers. A text-specific attack methods have been
developed Li et al. (2019).

The DDDM is evaluated on the IMDB dataset Maas et al. (2011) against the
TextBugger attack Li et al. (2019). We use an LSTM classifier resembling the
one in the TextBugger work, with adding an additional dropout layer to the
pre-activation outputs of each gate in the network. Words from the IMDB
dataset are first converted into pre-trained GloVe embedding vectors of
dimension $300$, then fed to the LSTM classifier.

We summarize the experimental results in Table 4. DDDM $H_{a^{*},b^{*}}$
successfully protected the classifier with a considerable improvement in
performance compared to the dropout classifier $h_{a^{*},b^{*}}$. The robust
accuracy of $H_{a^{*},b^{*}}$ under the TextBugger attack approaches its clean
accuracy. Moreover, we note that the clean accuracy of $H_{a^{*},b^{*}}$ is
higher than the one of the undefended model $h_{0,0}$. This could be explained
by the fact that we use the clean training data to evaluate the likelihood. We
emphasize that this does not harm the effectiveness of our approach since it
does not require information about the possible attacks beforehand.

##  5 Discussion

The current work is merely proof of the principle that combining an evidence
accumulation mechanism with deep neural networks enhances robustness and
provides defense against adversarial attacks. The key to the framework’s
success lies in the interplay between two components of the framework. The
first is the mechanism to introduce noise into the system. Partly because the
random noise itself degrades the performance; therefore, achieving a balance
between performance and robustness poses a challenge. The second component,
the evidence accumulation in the DDM, provides a mechanism to solve this
problem and allows more flexibility in terms of the level of additional noise
added into the system. In addition to artificially introduced noise, noise
exists in many real-world situations, such as the automatic driving systems.
In these cases, noise naturally presented in the input may be combined with
artificially introduced noise for defense against attacks.

The attractiveness of the framework also lies in its simplicity. DDDM does not
require the networks to be trained differently for any particular attacks.
Instead, it is agnostic to the type of attack and its particular scheme of
noise. Thus, it provides protection not only for the attacks tested here but
also for other attacks. The framework also does not depend on particular types
of networks. It can also be used in combination with more sophisticated
networks and larger datasets.

Other than what has been mentioned above, several other directions to extend
the current framework can be explored in the future. In addition to the
dropouts, other ways of introducing noise into the system may help to battle
perturbations. A mechanism of online adjustment of decision thresholds may
help improve the speed-accuracy tradeoff in a dynamic environment. Finally,
the evidence accumulation mechanism is not implemented with a neural network
in the current study. However, previous studies have shown that simple
recurrent networks may achieve the functionality Wang (2002); Wong & Wang
(2006), providing an end-to-end neural network solution for the DDDM.

##  6 Conclustion

Inspired by the neural mechanism of decision making, we develop a novel
framework (see Figure 1) and demonstrate its effectiveness in improving the
robustness of deep learning networks against adversarial attacks in different
domains with a brain-like mechanism (see Tables 1, 2, 3 and 4). In contrast to
adversarial training, the DDDM shows an advantage of sustaining the same level
of clean accuracy after being attacked. By extending the one-shot
classification into a temporal decision-making process based on evidence
accumulation, our framework trades inference time for accuracy to achieve
robustness.

To the best of our knowledge, this is the first framework to combine deep
neural networks and a brain-inspired decision-making mechanism for defense
against adversarial attacks and the first framework to provide cross-domain
protection without adversarial training. We hope that our work brings the
advances of neuroscience research to the attention of the artificial
intelligence community and encourages researchers to take a closer look at the
most powerful yet robust classifier —— our brain.

## Acknowledgments

This work was supported by grants from the National Science and Technology
Innovation 2030 Project of China to Yi Zhou (2021ZD0202600) and to Tianming
Yang (2021ZD0203701). This work was also supported by grants from the Chinese
Academy of Sciences (XDB32070100), and the Shanghai Municipal Science and
Technology (2018SHZDZX05) to Tianming Yang.

## References

  * Amodei et al. (2015) Amodei, D., Anubhai, R., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., Elsen, E., Engel, J., Fan, L., Fougner, C., Han, T., Hannun, A., Jun, B., LeGresley, P., Lin, L., Narang, S., Ng, A., Ozair, S., Prenger, R., Raiman, J., Satheesh, S., Seetapun, D., Sengupta, S., Wang, Y., Wang, Z., Wang, C., Xiao, B., Yogatama, D., Zhan, J., and Zhu, Z.  Deep Speech 2: End-to-End Speech Recognition in English and Mandarin.  _arXiv:1512.02595 [cs]_ , December 2015.
  * Andriushchenko et al. (2020) Andriushchenko, M., Croce, F., Flammarion, N., and Hein, M.  Square Attack: A Query-Efficient Black-Box Adversarial Attack via Random Search.  In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J.-M. (eds.), _Computer Vision – ECCV 2020_ , volume 12368, pp. 484–501. Springer International Publishing, Cham, 2020.  ISBN 978-3-030-58591-4 978-3-030-58592-1.  doi: 10.1007/978-3-030-58592-1˙29.
  * Athalye et al. (2018) Athalye, A., Carlini, N., and Wagner, D.  Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples.  _arXiv:1802.00420 [cs]_ , July 2018.
  * Carlini & Wagner (2017) Carlini, N. and Wagner, D.  Towards Evaluating the Robustness of Neural Networks.  _arXiv:1608.04644 [cs]_ , March 2017.
  * Carlini & Wagner (2018) Carlini, N. and Wagner, D.  Audio Adversarial Examples: Targeted Attacks on Speech-to-Text.  January 2018.
  * Dhillon et al. (2018) Dhillon, G. S., Azizzadenesheli, K., Lipton, Z. C., Bernstein, J., Kossaifi, J., Khanna, A., and Anandkumar, A.  Stochastic Activation Pruning for Robust Adversarial Defense.  _arXiv:1803.01442 [cs, stat]_ , March 2018.
  * Elsayed et al. (2018) Elsayed, G. F., Shankar, S., Cheung, B., Papernot, N., Kurakin, A., Goodfellow, I., and Sohl-Dickstein, J.  Adversarial examples that fool both computer vision and time-limited humans.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , NIPS’18, pp. 3914–3924, Red Hook, NY, USA, December 2018. Curran Associates Inc.
  * Engstrom et al. (2019) Engstrom, L., Tran, B., Tsipras, D., Schmidt, L., and Madry, A.  Exploring the Landscape of Spatial Robustness.  In _International Conference on Machine Learning_ , pp. 1802–1811. PMLR, May 2019.
  * Gold & Shadlen (2007) Gold, J. I. and Shadlen, M. N.  The Neural Basis of Decision Making.  _Annual Review of Neuroscience_ , 30(1):535–574, July 2007.  ISSN 0147-006X, 1545-4126.  doi: 10.1146/annurev.neuro.29.051605.113038.
  * Goodfellow et al. (2015) Goodfellow, I. J., Shlens, J., and Szegedy, C.  Explaining and Harnessing Adversarial Examples.  _arXiv:1412.6572 [cs, stat]_ , March 2015.
  * Guo et al. (2018) Guo, C., Rana, M., Cisse, M., and van der Maaten, L.  Countering Adversarial Images using Input Transformations.  In _International Conference on Learning Representations_ , February 2018.
  * Hinton et al. (2012) Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. R.  Improving neural networks by preventing co-adaptation of feature detectors.  _arXiv:1207.0580 [cs]_ , July 2012.
  * Krizhevsky (2009) Krizhevsky, A.  Learning multiple layers of features from tiny images.  Technical report, April 2009.
  * Lecun et al. (1998) Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, November 1998.  ISSN 00189219.  doi: 10.1109/5.726791.
  * Li et al. (2019) Li, J., Ji, S., Du, T., Li, B., and Wang, T.  TextBugger: Generating Adversarial Text Against Real-world Applications.  _Proceedings 2019 Network and Distributed System Security Symposium_ , 2019.  doi: 10.14722/ndss.2019.23138.
  * Maas et al. (2011) Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C.  Learning Word Vectors for Sentiment Analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pp. 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
  * Madry et al. (2019) Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A.  Towards Deep Learning Models Resistant to Adversarial Attacks.  _arXiv:1706.06083 [cs, stat]_ , September 2019.
  * Moosavi-Dezfooli et al. (2016) Moosavi-Dezfooli, S.-M., Fawzi, A., and Frossard, P.  DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2574–2582, June 2016.  doi: 10.1109/CVPR.2016.282.
  * Qin et al. (2019) Qin, Y., Carlini, N., Goodfellow, I., Cottrell, G., and Raffel, C.  Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition.  March 2019.
  * Ratcliff & McKoon (2008) Ratcliff, R. and McKoon, G.  The Diffusion Decision Model: Theory and Data for Two-Choice Decision Tasks.  _Neural Computation_ , 20(4):873–922, April 2008\.  ISSN 0899-7667.  doi: 10.1162/neco.2008.12-06-420.
  * Ratcliff & Rouder (1998) Ratcliff, R. and Rouder, J. N.  Modeling Response Times for Two-Choice Decisions.  _Psychological Science_ , 9(5):347–356, September 1998.  ISSN 0956-7976.  doi: 10.1111/1467-9280.00067.
  * Rauber & Bethge (2020) Rauber, J. and Bethge, M.  Fast Differentiable Clipping-Aware Normalization and Rescaling.  _arXiv:2007.07677 [cs, stat]_ , July 2020.
  * Roitman & Shadlen (2002) Roitman, J. D. and Shadlen, M. N.  Response of Neurons in the Lateral Intraparietal Area during a Combined Visual Discrimination Reaction Time Task.  _The Journal of Neuroscience_ , 22(21):9475–9489, November 2002.  ISSN 0270-6474, 1529-2401.  doi: 10.1523/JNEUROSCI.22-21-09475.2002.
  * Roxin (2019) Roxin, A.  Drift–diffusion models for multiple-alternative forced-choice decision making.  _The Journal of Mathematical Neuroscience_ , 9(1):5, December 2019.  ISSN 2190-8567.  doi: 10.1186/s13408-019-0073-4.  URL https://mathematical-neuroscience.springeropen.com/articles/10.1186/s13408-019-0073-4.
  * Rusakov et al. (2020) Rusakov, D. A., Savtchenko, L. P., and Latham, P. E.  Noisy Synaptic Conductance: Bug or a Feature?  _Trends in Neurosciences_ , 43(6):363–372, June 2020.  ISSN 01662236.  doi: 10.1016/j.tins.2020.03.009.  URL https://linkinghub.elsevier.com/retrieve/pii/S0166223620300692.
  * Simonyan & Zisserman (2015) Simonyan, K. and Zisserman, A.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  _arXiv:1409.1556 [cs]_ , April 2015.
  * Wang (2002) Wang, X.-J.  Probabilistic Decision Making by Slow Reverberation in Cortical Circuits.  _Neuron_ , 36(5):955–968, December 2002.  ISSN 0896-6273.  doi: 10.1016/S0896-6273(02)01092-9.
  * Warden (2018) Warden, P.  Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition.  _arXiv:1804.03209 [cs]_ , April 2018.
  * Wong & Wang (2006) Wong, K.-F. and Wang, X.-J.  A Recurrent Network Mechanism of Time Integration in Perceptual Decisions.  _Journal of Neuroscience_ , 26(4):1314–1328, January 2006.  ISSN 0270-6474, 1529-2401.  doi: 10.1523/JNEUROSCI.3733-05.2006.
  * Yang & Shadlen (2007) Yang, T. and Shadlen, M. N.  Probabilistic reasoning by neurons.  _Nature_ , 447(7148):1075–1080, June 2007.  ISSN 1476-4687.  doi: 10.1038/nature05852.

##  Appendix A Likelihood Estimation

In the DDDM, model decisions are made when
$\text{Pr}(H_{i}|{T_{1},\dots,T_{t}})$ reached a threshold $A$. According to
the Bayes theorem, it can be updated with likelihood
$\text{Pr}({T_{t}}|H_{i})$. However, this likelihood is difficult to compute
directly because the probability density is continuous in $T_{t}$. Instead, we
estimated the likelihood $\text{Pr}({T_{t}}|H_{i})$ with
$\text{Pr}({I_{t}}|H_{i})$, where $I$ is the sorted combination of of the $k$
most confident labels in prediction. For example, in an MNIST experiment,
where $T_{t}=(0,0,0.1,0.3,0.4,0.15,0.05,0,0,0)$ and $k$ is set to $3$. The $k$
largest predicted labels are ”4” ($p=0.4$), ”3” ($p=0.3$) and ”5” ($p=0.15$),
then $I_{t}$ will be ”435”. When $H_{i}$ is the hypothesis that ”input image
belongs to label $i$”, $\text{Pr}({I_{t}}|H_{i})$ refers to the frequency of
$I_{t}$’s appearance when the neural network is fed with images of label $i$
from the training set. We choose $k=3$ for all experiments in this paper,
except for the IMDB dataset, where $k=2$.

##  Appendix B Attack Details

###  B.1 MNIST Dataset

We adopted the simple CNN in Madry et al. (2019) with an additional dropout
layer that follows each layer except the input/output ones. We evaluated DDDM
on this network with the following attack methods:

  * •

the Fast Gradient Sign Method (FGSM) attack Goodfellow et al. (2015), with
parameters $\epsilon=0.3$.

  * •

the Projected Gradient Descent (PGD) attack Madry et al. (2019), with
parameters $\epsilon=0.3$ and steps = 40.

  * •

the $L_{2}$ Carlini and Wagner ($L_{2}$ C&W) attack Carlini & Wagner (2017)
with $\text{steps}_{max}=1000$.

  * •

the $L_{2}$ DeepFool attack Moosavi-Dezfooli et al. (2016), with parameters
$\epsilon=1.5$ and steps = 50.

  * •

the Salt and Pepper (Salt&Pepper) attack, with parameters $\epsilon=10$ and
steps = 1000.

  * •

the repeated Additive Uniform Noise (uniform) Attack Rauber & Bethge (2020),
with with parameters $\epsilon=0.3$ and repeats = 100.

  * •

the Spatial attack Engstrom et al. (2019), with parameters
$\text{translation}_{\text{max}}=3,\text{translation}_{\text{num}}=5,\text{rotation}_{\text{max}}=30^{\circ},\text{rotation}_{\text{num}}=5$,
and $\text{random}_{steps}=100$.

  * •

the Square attack Andriushchenko et al. (2020), with parameters $\epsilon=0.3$
and $\text{steps}_{max}=100$.

###  B.2 CIFAR10 Dataset

We use the VGG16 architecture Simonyan & Zisserman (2015) without batch
normalization. A dropout layer is added to each of the last six convolutional
layers. We evaluated DDDM on this network with the following attack methods:

  * •

the PGD attack Madry et al. (2019), with parameters
$\epsilon=\frac{8}{255}\approx 0.031$ and steps=200.

  * •

the $L_{2}$ DeepFool attack Moosavi-Dezfooli et al. (2016), with parameters
$\epsilon=0.12$ and steps = 50.

  * •

the Spatial attack Engstrom et al. (2019), with parameters
$\text{translation}_{\text{max}}=3,\text{translation}_{\text{num}}=5,\text{rotation}_{\text{max}}=30^{\circ},\text{rotation}_{\text{num}}=5$,
and $\text{random}_{steps}=100$.

###  B.3 SpeechCommands Dataset

In this task we adopted a mini version of DeepSpeech2
amodeiDeepSpeechEndtoEnd2015 model and the imperceptible ASR attack method Qin
et al. (2019). Our DeepSpeech2 model includes a Mel-spectrogram conversion
layer, followed by one 1D convolutional layer, two LSTMs and two fully-
connected layers. Dropout is only applied once after the convolutional layer.
The model was trained from scratch, only on the SpeechCommands dataset. After
training, we attacked the model with $\epsilon=0.05$ and random target words.

###  B.4 IMDB Dataset

In this task we adopted the black-box attack method and targeted LSTM
classifier in Li et al. (2019). Dropout layers were inserted before the
sigmoid or tanh operation in each of the four gates in the LSTM.

##  Appendix C Method of Adding Dropout Layers to VGG16

Due to the relatively large size of the the VGG16 network, it is infeasible to
train the network if one adds a dropout layer to each of the original layers.
Instead, we want to add dropout layers only to those parts that are sensitive
to adversarial attacks. To discover such parts, we investigate the hidden
outputs of each layer in VGG16. We compared the hidden outputs for the clean
inputs to the ones for the adversarial inputs. Those adversarial inputs are
generated using PGD attack with $\epsilon=0.031$ and steps=200.

Two metrics are investigated, namely the cosine similarity and the $L_{2}$
distance between the hidden outputs for the clean and adversarial inputs. For
the $L_{2}$ distance, we rescale the results by deviding the We plot the mean
and standard deviation of the two metrics for each layer in Figure 4 In
addition, we also plot the UMAP for the hidden outputs of each layer. The
results are summarized in Figure 5

Both results reveal that the lower part of the network, i.e. the latter six
convolutional layers are more sensitive to the adversarial inputs. Hence, we
decide to add dropout layers only to those six layers.

Figure 4: U The cosine similarity and the $L_{2}$ distance between hidden
outputs for the clean and adversarial inputs.  Figure 5: UMAP for hidden
outputs of each layer in VGG16.