pnasresearcharticle

Meng

When a discovery or technique becomes common knowledge, its citations suffer
from what Robert Merton called “obliteration by incorporation.” This
phenomenon leads to the concept of hidden citations, representing unambiguous
textual references to a discovery without an explicit citation to the
corresponding manuscript(s). Previous attempts to detect hidden citations have
been limited to manually identifying in-text mentions. Here, we use machine
learning to systematically identify hidden citations, finding that they emerge
regardless of publishing venue and discipline, their frequency being
influenced by the level of discussion within manuscript texts. Hidden
citations lead to inevitable credit distortion and capture the “burden” of
success in science: the more widely a concept is used, the more hidden it is
from standard bibliometric analysis.

All authors designed and did the research. O.V. and A.-L.B. conceived the
concept. X.M. developed the methodology. X.M and O.V. collected and analyzed
the data. X.M. and A.-L.B. were the lead writers of the manuscript. A.-L.B. is
the scientific founder of Scipher Medicine, Inc., which applies network
medicine to biomarker development, of Foodome, Inc., which applies data
science to health, and of Datapolis, Inc., which focuses on human mobility.
1To whom correspondence should be addressed. E-mail:
<EMAIL_ADDRESS>
# Hidden Citations Obscure True Impact in Science

Xiangyi Meng  Network Science Institute and Department of Physics,
Northeastern University, Boston, Massachusetts 02115, USA  Onur Varol  Network
Science Institute and Department of Physics, Northeastern University, Boston,
Massachusetts 02115, USA  Faculty of Engineering and Natural Sciences, Sabanci
University, Istanbul 34956, Turkey  Albert-László Barabási  Network Science
Institute and Department of Physics, Northeastern University, Boston,
Massachusetts 02115, USA  Channing Division of Network Medicine, Department of
Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston,
Massachusetts 02115, USA  Department of Network and Data Science, Central
European University, Budapest 1051, Hungary

###### Abstract

References, the mechanism scientists rely on to signal previous knowledge,
lately have turned into widely used and misused measures of scientific impact.
Yet, when a discovery becomes common knowledge, citations suffer from
obliteration by incorporation. This leads to the concept of hidden citation,
representing a clear textual credit to a discovery without a reference to the
publication embodying it. Here, we rely on unsupervised interpretable machine
learning applied to the full text of each paper to systematically identify
hidden citations. We find that for influential discoveries hidden citations
outnumber citation counts, emerging regardless of publishing venue and
discipline. We show that the prevalence of hidden citations is not driven by
citation counts, but rather by the degree of the discourse on the topic within
the text of the manuscripts, indicating that the more discussed is a
discovery, the less visible it is to standard bibliometric analysis. Hidden
citations indicate that bibliometric measures offer a limited perspective on
quantifying the true impact of a discovery, raising the need to extract
knowledge from the full text of the scientific corpus.

###### keywords:

Science of science $|$ Hidden citation $|$ Latent Dirichlet allocation $|$
Foundational paper $|$ Catchphrase

This manuscript was compiled on August 28, 2024
www.pnas.org/cgi/doi/10.1073/pnas.XXXXXXXXXX

13]2

“We stand on the shoulders of giants,” the oft-quoted statement acknowledging
the cumulative nature of knowledge, has an explicit carrier in the
contemporary scientific discourse: the _citation_. Since the 1960s,
references—which serve primarily as a mechanism to signal prior knowledge,
enhance credibility, and protect against plagiarism—have taken on a secondary
role of _allocating scientific credit_ , turning into an often used and
misused measure of scientific impact (1, 2, 3). Yet, when a discovery or
technique becomes common knowledge to such a degree that it does not warrant
citation any longer, citations suffer from what Robert Merton in 1968 called
“obliteration by incorporation (OBI) (4, 5).” For example, concepts like
general relativity or black hole evaporation today are so embedded into
scientific literacy, that only rarely do manuscripts focusing on the topics
cite Einstein’s 1915 work (6) or Unruh’s 1976 paper (7). As a consequence,
foundational ideas of science are undercited, without being underused. This
phenomenon leads to hidden citations, representing unambiguous allusions to a
body of knowledge without an explicit citation to the manuscript(s) that
introduced it. Hidden citations, also known as implicit, indirect (8) or
informal citations (9), can also be induced by restrictions imposed by
publishing venues on the number of references, prompting authors to cite
reviews and books to signal a wider body of knowledge, rather than crediting
the original discoveries. While Merton considered such hidden citations the
highest level of acknowledgement—a badge of honor, rather than a negative
effect (4), such credit is no longer accessible to traditional bibliometric
measures. In the era when citations are widely used as measures of impact (10,
11, 12, 13, 14, 15, 16, 17, 18), hidden citations remain _hidden_ —not to
human beings—but to the quantitative and statistical tools frequently used to
quantify scientific credit. This leads to systematic distortion of the credit
landscape, diminishing the quantifiable impact of the very discoveries that
define scientific progress.

Previous attempts to detect hidden citations have been limited to manually
searching and identifying in-text mentions such as “Southern blot (8),”
“density functional theory (9),” “Nash equilibrium (19),” “evolutionarily
stable strategy (20),” and “bounded rationality (21, 22).” Yet, the lack of
automated methodology for determining in-text allusions [including eponyms
(23, 24), relating a person to a discovery] and their corresponding primordial
references (25) has limited our ability to understand the prevalence of hidden
citations to a narrow corpus of manually inspected papers, raising the need
for “a more comprehensive estimate of uncitedness (8).” Here, we fill this gap
by using machine learning to automatically detect _catchphrases_ (19),
representing in-text allusions to specific discoveries, matching them with the
appropriate primordial references called _foundational papers_. The method
allows us to systematically identify hidden citations across the whole
scientific literature, and to quantify the factors responsible for credit
distortion. As roughly $90\%$ of obliteration by incorporation happens in the
main text of a manuscript (21), we apply machine learning to the full text,
helping us accurately quantify the accumulation and distortion of credit in
science.

Figure 1:  Hidden citations. (a) A foundational paper is a manuscript that
introduces a new concept that subsequently defines a topic of inquiry by the
scientific community, such as the topic “anti-de Sitter/conformal field
theory,” also known as “AdS/CFT (26).” Papers focusing on the topic mention
the catchphrase “AdS/CFT” or “anti-de Sitter/conformal field theory,” followed
by a citation to one of the foundational papers. Often, however, the
catchphrases are present without explicit citations, resulting in hidden
citations. (b) Exemplary topics selected from high energy physics (hep),
condensed matter physics (cond), quantum physics (quant), and astrophysics
(astro), together with their corresponding catchphrase(s) (lemmatized as word
stems) and foundational paper(s) (Microsoft Academic Graph id). Darker arrows
denote the algorithm’s higher statistical confidence for the respective
foundational paper. (c-f) Time evolution of citations and hidden citations for
the topics listed in (b). The arrows denote the publication date(s) of the
foundational paper(s) for each topic.

## Results

Each scientific discovery builds on a body of knowledge embodied by latent
topics that are topically named within a manuscript and accompanied by
citations to the foundational papers. For example, papers focusing on anti-de
Sitter/conformal field theory (AdS/CFT), exploring the correspondence between
general relativity and quantum field theory, cite the 1999 paper that
introduced the concept [Fig. 1(a)]. Yet, many papers on AdS/CFT use language
that for experts unambiguously defines the paper’s topic, without citing the
foundational work. To identify such hidden citations, we use the Latent
Dirichlet Allocation (LDA) model (27, 28) to detect topics in the text of a
publication, inferring latent topical structures from a corpus of full-text
citation contexts based on symbolic natural language processing and Bayesian
inference. In contrast with neural-network-based Word2Vec (29) or BERT (30),
the LDA model is an unsupervised machine learning approach that is
interpretable, allowing us to associate the outcomes of LDA with confidence
levels through transparent probabilistic logic (see Methods).

We identified $343$ topics in physics that accumulate hidden citations, each
with at least one catchphrase and at least one foundational paper (see
Methods). Shown as examples are four topics uncovered by the algorithm [Fig.
1(b)], as well as the followers of each topic [Fig. 1(c-f)], defined as papers
that either cited the foundational papers of the given topic or mentioned the
corresponding topic-specific catchphrases, or both (see Methods). For example,
the orange regions denote the temporal evolution of the number of papers that
simultaneously cite the foundational papers and carry the respective
catchphrases. The top green region captures hidden citations, counting papers
that make an unambiguous textual reference to the topic but fail to cite any
of the foundational papers. For example, less than half of the papers that use
the catchphrases of “AdS/CFT” cited any of the two foundational papers: 1999
paper by Maldacena and 1998 paper by Gubser et al. [Fig. 1(c)]. Taken
together, we find that for the four topics featured in Fig. 1(b), hidden
citations correspond to $65.8\%$, $61.7\%$, $34.6\%$, and $52.3\%$ of all
detectable credit since the publication of the respective topic’s first
foundational paper, overcoming the bibliometrically quantifiable and tabulated
citations.

Figure 2:  Factors that drive hidden citations. (a) The temporal change of
$p(\text{cite}|\text{mention})$, the probability that a paper mentioning the
topic-specific catchphrases will also cite the foundational paper, as a
function of time (years since publication). On average,
$p(\text{cite}|\text{mention})$ per topic drops by approximately $20\%$ after
$20$ years of publication of the first foundational paper. Error bars
represent $95\%$ confidence intervals. (b) Topics with more citations ($c$)
tend to have more hidden citations ($h$) (with Spearman’s rank correlation
$\rho\approx 0.381$ and null hypothesis $H_{0}$ rejected). Most topics fall
into the $95\%$ single-observation confidence bands with a log-log slope
$0.763{\pm 0.208}$, indicating that $h\sim c^{0.763}$. (c)
$p(\text{cite}|\text{mention})$ as a function of citations per topic
($\rho\approx 0.016$, $H_{0}$ not rejected), indicating that the probability
of a textual reference becoming a hidden citation is not driven by the number
of citations to the topic. (d) $p(\text{cite}|\text{mention})$ as a function
of mentions per topic ($\rho\approx-0.611$, $H_{0}$ rejected). The strong
negative correlation indicates that hidden citations are driven by the number
of textual mentions of the topic. Most topics fall into the $95\%$ confidence
bands with a log-linear slope $-0.27{\pm 0.04}$. The pattern holds for four
distinct publication venues (e-h).

The high proportion of hidden citations prompts us to calculate the temporal
changes in the conditional probability that a paper that mentions the topic-
specific catchphrases cites the foundational papers,
$p(\text{cite}|\text{mention})$ (SI, Section 9). We find that the probability
that the foundational papers are cited drops by approximately $20\%$ after
$20$ years [Fig. 2(a)], indicating that the reliance on hidden citations,
hence OBI, strengthens over time.

Figure 3:  Credit redirected. (a) The most cited alternatives for four topics
that acquire hidden citations, primarily indicating that credit is often
diverted to books, reviews or applications/extensions of the foundational
papers. (b) Most alternatives to hidden citations are related to the
foundational paper, detectable by tracking the citation path between the
alternative and the foundational paper. (c-f) Fraction of hidden citations
ranked by their citation hierarchy to the foundational papers. For each topic
(except “BOSS”), around $60\%$ of hidden citations (green, top) cited other
arXiv papers that explicitly cited the foundational papers. For a randomly
sampled reference from the full arXiv, this fraction is negligible (brown,
bottom).

Do hidden citations correspond to pure untracked credit [a.k.a. implicit
citations (8)], or is credit diverted to other works [a.k.a. indirect
citations (8)]? To distinguish these two mechanisms, we identified the most
frequently cocited publications accompanying a hidden citation. We find that
for “AdS/CFT” the most cited alternative is a review coauthored by the authors
of the two foundational papers, and for “DMRG” the most cited alternatives are
two books [Fig. 3(a)]. Credit is also diverted to applications of the topic,
such as the application of AdS/CFT to topological quantum field theory, or to
extensions on the topic, like in the expanded “BOSS” datasets [Fig. 3(a)].
Overall, we find that the works that collect the credit from hidden citations
tend to cite the foundational papers, or cite papers that in turn cite the
foundational papers [Fig. 3(b)]. Indeed, around $60\%$ of hidden citations
have a citation path length of $2$ to the foundational papers [Fig. 3(c-f)].
indicating that hidden citations do cite and give credit to papers whose
topics closely relate to the foundational papers. To determine whether the
previously observed increase in reliance on hidden citations over time [Fig.
2(a)] is dominated by implicit citations or indirect citations, we
recalculated the temporal changes [Fig. 2(a)], this time also including
indirect citations, i.e., the hidden citations that have a citation path
length of at most $2$. We find now that $p\left[(\text{cite}+\text{indirectly
cite})|\text{mention}\right]$ increases with time (SI, Section 10), indicating
that the increasing reliance on hidden citations is accompanied by an
increasing tendency to divert credit to other works.

As Fig. 2(b) shows, topics with more citations ($c$) tend to accumulate more
hidden citations ($h$), well approximated by a sublinear dependence $h\sim
c^{0.763}$, indicating that on average a topic with $5,000$ citations
accumulates approximately $1,000$ hidden citations. While this scaling
suggests that citations are the main driving force of hidden citations, our
measurements indicate otherwise. Indeed, we find a negligible correlation
($\rho\approx 0.016$) between $p(\text{cite}|\text{mention})$ (the probability
of being cited if mentioned) and the number of citations for the respective
topic [Fig. 2(c)] (SI, Section 8). We find, however, a strong negative
correlation ($\rho\approx-0.611$) between $p(\text{cite}|\text{mention})$ and
the number of mentions per topic [Fig. 2(d)]. In other words, the more
discussed is a discovery in the textual context of a paper, the less likely
scientists feel the need to explicitly cite it, a “burden” of success that is
independent of the publication venue [Fig. 2(e-h)].

Figure 4:  Foundational papers. (a) Changes in the citation-based ranks of the
top-ranked foundational papers after taking hidden citations into account,
shown by arrows from the old explicit-citation-based rank to the new explicit-
plus-hidden-citation-based rank (green: rank rise; red: rank drop). After
accounting for hidden citations, the “cosmological inflation theory” paper
(2134251287), ranked #8 based on explicit citation counts, takes the top spot.
(b) For foundational papers with similar numbers of explicit citations, the
paper with more hidden citations tends to result in higher average author
saliency (inset). The proportion of papers in the corpus that can acquire
hidden citations increases with (c) the explicit citations but not with (d)
the publication year of the papers. Error bars represent $95\%$ confidence
intervals. (e) Distribution of foundational papers by the number of authors
per foundational paper, shown for all catchphrases (black) and for eponym-
related (blue) and experiment-related catchphrases (green).

To explore the impact of hidden citations on bibliometric measures, we
calculated the hidden citations of _individual_ foundational papers (SI,
Section 11), finding that on average the ratio of hidden to explicit citations
is $0.98$:$1$, meaning that papers acquire hidden citations at the same rate
as they acquire explicit citations. Yet, given the considerable variability in
this ratio, for some foundational papers hidden citations dominate over
explicit citations. Examples include the paper introducing the cosmological
inflation theory in 1981 that acquired $8.8$ times more hidden citations than
explicit citations, or the 1974 work that merged the electromagnetic, weak,
and strong forces into a single force, which accumulated $6.6$ times more
hidden citations than explicit citations. This prompted us to calculate the
changes in citation-based ranks between foundational papers (SI, Section 11).
As Fig. 4(a) indicates, most papers in the top $100$ list suffer rank loss
(green lines), thanks to a few publications that accumulate an exceptional
number of hidden citations, and gain significantly in rank (red lines). For
example, the most cited paper of arXiv, the 1999 paper which started the
formal theory of AdS/CFT, loses its top ranking once we take hidden citations
into account, to the 1981 paper previously ranked #8, which started the
phenomenological study of the cosmological inflation theory [Fig. 4(a)].
Hidden citations could potentially have an impact on authors as well. To see
if this is the case, we adopted the Microsoft Academic Graph’s “author
saliency” metric, that relies on the heterougeneous network structure of the
connectivity of articles, authors, and journals, designed to be less
susceptible to raw citation counts and temporal bias (31). We find that, when
we compare two foundational papers with similar numbers of explicit citations,
authors with more hidden citations have higher average saliency, a positive
correlation notable for papers with less than $3,000$ citations [Fig. 4(b)].
This suggests that OBI correlates with a notable _positive_ impact on authors
whose papers’ full citation impact has not yet developed. Interestingly, the
effect disappears for well-recognized papers, for which missing citations does
not affect their authors’ reputation.

Papers that became foundational papers and acquired hidden citations tend to
be highly cited, accumulating on average $434{\pm 34}$ explicit citations, in
contrast with $1.4$ explicit citations for all physics papers in the corpus.
Yet, not only highly cited papers acquire hidden citations. We find that even
among papers with citations $\lesssim 500$, a nonnegligible fraction ($>10\%$)
of papers acquire hidden citations [Fig. 4(c)]. Since our approach to
identifying hidden citations (see Methods) is conservative, designed to reduce
false positive errors, the actual fraction of papers that acquire hidden
citations is likely higher. We also find that hidden citations are not limited
to older papers, but they accompany recent publications as well [Fig. 4(d)],
such as the discovery of gravitational wave (2016) or exclusion of dark matter
particles in the Large Underground Xenon experiment (2017).

Finally, we investigated the sociodemographic characteristics (gender, country
of origin, and the prestige of institution) of the authors of foundational
papers (SI, Section 12). We find that hidden citations capture a universal
phenomenon that can emerge in any institution, regardless of their level of
prestige, and we observe no statistically significant bias based on gender or
country of origin in hidden citations apart from the overall biases in
explicit citations that have long been observed (32, 33, 34, 35). Hence,
although the methodology presented in the paper can account for additional
hidden citations, it simply reflects the existing biases.

Figure 5:  Hidden citations across disciplines and venues. (a) Four topics
selected from computer science (cs) and biology (bio) [cf. Fig. 1(b)]. (b-i)
Time evolution of citations and hidden citations [cf. Fig. 1(c-f)] for the
four topics shown in (a), identified from arXiv (b-e) and _Nature_ (f-i).

These findings do raise an important question: what determines the emergence
of hidden citations? Our analysis suggests that there is a prerequisite for a
paper to acquire hidden citations: it must develop exclusive catchphrases that
are synonymous to the paper itself, becoming a “conceptual symbol” (36, 37)
within the field. For example, whenever “quantum discord” is mentioned, an
expert in the field will immediately link it to the 2001 foundational paper,
and vice versa, seeing a citation to that particular reference, an expert
thinks of “quantum discord.” To quantify this dual correspondence, we first
measured the degree of non-exclusivity of linking a given $n$-gram $w$ to a
paper $d$ by calculating the specific conditional entropy $\mathcal{S}(d|w)$
(SI, Section 13), finding that $\mathcal{S}(d|w)$ is considerably lower for
catchphrases than for non-catchphrases. For example,
$\mathcal{S}(d|\text{``quantum discord''})\approx 4.07$ in contrast with
$\mathcal{S}(d|\text{``quantum mechanics''})\approx 6.73$, indicating that
“quantum discord” is a catchphrase pointing to a well defined foundational
paper, while “quantum mechanics” is too general to be exclusively assigned to
one or a few foundational papers. Inversely, we measured the specific
conditional entropy $\mathcal{S}(w|d)$ of seeing paper $d$ and linking it to
an $n$-gram $w$, finding again that $\mathcal{S}(w|\text{1571385165})\approx
5.97$ for the 2001 foundational paper with catchphrase “quantum discord” is
lower than the highly cited 1999 paper also focusing on quantum information
processing, $\mathcal{S}(w|\text{2097039598})\approx 7.72$, but not
categorized by our algorithm as a foundational paper. These results confirm
that to develop hidden citations, a (catchphrase)–(foundational paper) pair
must acquire _mutual_ exclusivity: a paper does not accumulate hidden
citations if its conceptual significance does not lead to an unambiguous
catchphrase, or if that catchphrase is not exclusive enough for the community
to unambiguously link it back to the original paper.

Asking where such exclusive catchphrases originate, we find that for $78.8\%$
of the $880$ foundational papers the corresponding catchphrases do not appear
in the titles or the abstracts of them (SI, Section 14). This indicates that
catchphrases are typically not proposed by the authors of the foundational
papers, but are assigned later by the community (37). We also find that
$26.0\%$ of all foundational papers have catchphrases that correspond to
eponyms (e.g., “Kardar–Parisi–Zhang equation,” that governs surface growth)
and another $7.1\%$ acquire the names of experimental projects (e.g., “Super-
Kamiokande,” the discovery of neutrino oscillation) (SI, Section 14). Eponym-
related catchphrases emerge mainly for papers with short author lists—indeed,
foundational papers with eponyms as catchphrases have $2.50{\pm 0.36}$ authors
on average, in contrast with $72.4{\pm 26.5}$ authors for non-eponym-related
catchphrases, and $405{\pm 164}$ for papers with experiment-related
catchphrases [Fig. 4(e)].

Identifying hidden citations in all areas of science requires a large and
unbiased corpus of full-text citation contexts. While such a corpus is so far
unavailable for all science, we have access to $818,311$ computer science and
$140,865$ biology full-text manuscripts, allowing us to identify catchphrases
and foundational papers in these fields as well [Fig. 5(a)]. The patterns
governing hidden citations are largely indistinguishable from those documented
for physics: we observe a significant number of hidden citations for
established research topics like “Kalman” (refining estimates from new
measurements) and “ESS” (evolutionary strategies in natural selection) and
even for newer topics like “deep RL” (deep neural networks and artificial-
intelligence) and “ViennaRNA” (analysis of RNA structures) [Fig. 5(b-e)]. We
also analyzed a corpus of $88,637$ full-text _Nature_ articles (38), which
cover multiple disciplines, finding evidence of hidden citations in highly
selective peer-reviewed venues as well [Fig. 5(f-i)]. These results indicate
that hidden citations are a universal phenomenon, emerging in all areas of
science and publishing venues, disciplines, and research topics.

## Discussion

Acknowledging discoveries on which new research builds on is an integral part
of the scientific discourse. Yet, with the exponential growth of science and
limits on the number of allowed references, a paper’s ability to credit all
sources of inspiration is limited. Such limitations lead to inevitable credit
distortion, manifest in situations where the textual context indicates that
credit is due, but it is not accompanied by explicit citations to the
pertinent work. Hidden citations capture the “burden” of success in science:
the more widely a concept is used by the scientific community, the more likely
that it will accrue hidden citations.

Systematically tabulating hidden citations, together with explicit citations
can help us more accurately identify emerging topics and evaluate their true
impact (8, 9). That being said, both explicit and hidden citations represent
unequal “atoms of peer recognition (39),” offering different degrees of credit
per citation. Indeed, when citations point to highly cited papers (driven by
the authors’ fame, race, gender, etc.) without contributing to the paper’s
topic, they offer less credit (40). Negative citations (41) should also offer
less credit than positive citations; yet, we find that the prevalence of
negative or positive texts is low in both the explicit and hidden citations of
foundational papers (SI, Section 6). This can be attributed to the fact that
in order for a paper to become a foundational paper and acquire hidden
citations, the discovery or technique it presents should have already been
accepted as common knowledge, leaving limited room for debates. There is also
a difference in crediting conceptual versus methodological advances: while
papers that cite the foundational papers but fail to give appropriate textual
references are extremely rare [Fig. 1(c-e), Fig. 5(b-d, f-h)], for papers
introducing databases [Fig. 1(f)] or tools [Fig. 5(e,i)] textual references
are less frequent. One explanation is that these foundational papers are not
just cited for their dataset or methodological efforts; they are also
frequently cited for supporting the corresponding general concept, namely,
“baryon oscillation” or “RNA structure.” In the latter case, authors often
fail to mention the words “survey” or “package” when citing these papers. This
textual bias suggests that database or methodological advances are often less
acknowledged, in line with earlier findings (42). This is because when playing
the supportive role of a general concept, the papers lost their merits as
foundational papers. The community working on the general concept often
benefits from the database or methodological efforts without textually
referencing and explicitly acknowledging the effort that went into creating it
(SI, Section 7).

It is, therefore, important to go beyond explicit and hidden citations, and
develop new metrics that can also differentiate the degree of credit carried
by each citation, a process to which a complete corpus of both explicit and
hidden citations is a prerequisite. While our unsupervised methodology allows
us to tabulate hidden citations at scale, the current methodology is designed
to be conservative and to minimize false positive errors (see Methods). Thus,
currently it may overlook hidden citations, limiting completeness of topics.
Note that the missing topics can be recovered by lowering the identification
thresholds, at the expense of increasing false positive errors. There is also
a possibility that some topics may not be present in the arXiv corpus, either
because they have not been studied or discussed in a sufficient number of
arXiv papers or because they are too narrow or outdated. They could be
recovered if we apply our algorithm to a more extensive full-text database
that spans multiple disciplines and time periods. However, there is a major
barrier to achieving this: the lack of systematic access to full-text papers.
Indeed, while citation counts and other metadata are now freely and easily
available for research purposes, access to the full text of all research
papers is restricted by commercial interests, limiting the deployment of tools
capable of accurately tabulating hidden citations and their role in the
scientific discourse.

Traditionally, the LDA model is used to uncover latent topics within a
collection of documents. Each document is assumed to be a mixture of multiple
topics, and each topic is characterized by a distribution over phrases. Here,
instead of exploring latent topical structures, we focus on the explicit
textual observables, aiming to reveal the correspondence between phrases and
documents.

The input of the LDA model is a list of $2$-tuples $\\{w,d\\}$ between an _$n$
-gram_ $w$ (a phrase of $n$ words, where the value of $n$ can freely vary to
accommodate long phrases) and an accompanied text-based _document_ $d$
[denoted by a unique code, e.g., the Microsoft Academic Graph (MAG) id (43)].
The document does not contain the full text of the MAG paper $d$. Instead, it
comprises the citation contexts of $d$, which represent the textual
discussions by the community when citing $d$. Each $2$-tuple accounts for an
exact occurrence of an $n$-gram $w$ in document $d$. For example,

$\text{input}=\begin{pmatrix}\text{\\{``string theory'', 2166248051\\}}\\\
\text{\\{``gauge-gravity duality'', 2166248051\\}}\\\ \text{\\{``quantum
discord'', 1571385165\\}}\\\ \cdots\end{pmatrix}.$

The output of the LDA model is a list (of the same length of the input) of
$3$-tuples $\\{w,z,d\\}$, where each input $2$-tuple $\\{w,d\\}$ acquires a
new latent variable $z$ that corresponds to a specific topic, such as

$\text{output}=\begin{pmatrix}\text{\\{``string theory'', {topic} $1$,
2166248051\\}}\\\ \text{\\{``gauge-gravity duality'', {topic} $1$,
2166248051\\}}\\\ \text{\\{``quantum discord'', {topic} $2$, 1571385165\\}}\\\
\cdots\end{pmatrix}.$

The output indicates that “string theory” and “gauge-gravity duality” belong
to the same topic $1$, while “quantum discord” belongs to a different topic
$2$. The joint probability $P(w,z,d)$ of the concurrent occurrence of the
$3$-tuple $\\{w,z,d\\}$, which can be estimated from the output, enables us to
define and calculate two key terms:

(1) An $n$-gram $w$ with $P(z|w)>P_{\text{th}}^{\text{catch}}$ is a
_catchphrase_ of topic $z$, implying that whenever the $n$-gram $w$ is seen in
a document, we are confident that topic $z$ is referred to. For example, if
there are $1,919$ occurrences of $\\{w=\text{``quantum discord''},z,d\\}$ in
the output, among which $1,916$ $3$-tuples also have $z=3$, then
$P(z|w)\approx 0.998\pm 0.002$, representing the conditional probability of
referring to topic $z=3$ given occurrence of $w=$“quantum discord”. If
$P(z|w)$ is larger than $P_{\text{th}}^{\text{catch}}$, then we have
statistical confidence that “quantum discord” is a catchphrase of the topic
$z=3$.

(2) A document $d$ with $P(d|z)>P_{\text{th}}^{\text{found}}$ is a
_foundational paper_ of topic $z$, implying that whenever topic $z$ is
referred to, we expect a citation to the MAG paper $d$, indicating that the
foundational paper is sufficiently disruptive (16) to serve as a
representative of the topic. For example, if there are $9,742$ occurrences of
$\\{w,z=3,d\\}$ in total in the output, among which $2,091$ $3$-tuples also
include the document $d=\text{1571385165}$, then $P(d|z)\approx 0.215\pm
0.008$ which, if larger than $P_{\text{th}}^{\text{found}}$, makes
$d=\text{1571385165}$ a foundational paper of topic $z=3$.

We rely on a strict criterion to choose catchphrases
($P_{\text{th}}^{\text{catch}}=0.95$) but a loose criterion at including
foundational papers ($P_{\text{th}}^{\text{found}}=0.05$), reducing the false
positive rate of incorrectly assigning a too general $n$-gram as a
catchphrase, or concentrating too much hidden citations for only one or two
papers, hence remaining conservative at identifying hidden citations per
foundational paper. This unavoidably resulted in the exclusion of some topics
for which the catchphrases and foundational papers are less exclusively
defined. Therefore, our results are not based on a complete collection but a
sampled aggregation of topics.

After a latent topic $z$ is inferred by LDA, we identify all papers that
follow and explore the topic $z$ [Fig. 1(a)], including all the papers that
explicitly cite the foundational paper(s) of topic $z$, as well as papers that
only mention the topic-specific catchphrase(s) but lack citations to the
foundational paper(s) (SI, Section 4). The latter corresponds to hidden
citations, as they explicitly build on the catchphrase(s) associated with
topic $z$. For example, a hidden citation is detected when a paper mentions
the catchphrase “quantum discord” but lacks a citation to the foundational
paper $d=\text{1571385165}$.

We trained the LDA classifier using the _unarXive_ dataset (44) that offers
full-text coverage for $1,043,126$ publications, annotated with citation
contexts, obtained after merging the entire arXiv (45) with MAG (43) (SI,
Section 1). Established in 1991 as the first preprint archive, arXiv offers a
fairly unbiased coverage of physical sciences. We identified from the citation
contexts (from arXiv) all $n$-grams $w$ and each paper $d$ (in MAG) they refer
to (SI, Section 2), initially filtering out books and reviews. Following the
arXiv taxonomy, the results are categorized into five categories (SI, Section
5): high energy physics (“hep”), condensed matter physics (“cond”), quantum
physics (“quant”), astrophysics (“astro”), and the rest (“other”). For
example, the LDA model predicts that each time the catchphrase “anti-de Sitter
conform field theory” is mentioned, it should be accompanied by either a
reference to the 1999 paper (2166248051) by Maldacena (26), or to (2039609754)
by Gubser, Klebanov, and Polyakov, both within the “hep” topic “AdS/CFT” [Fig.
1(b)]. Similarly, for the “density matrix renormalization group” catchphrase,
the LDA model expects references to two papers (2037768897 and 2016407890) by
White, within the “cond” topic “DMRG” that focuses on many-body ground-state
wave functions [Fig. 1(b)]. To validate the accuracy of LDA, we have consulted
specialists in “hep” and “quant” to manually check ten randomly selected
topics each (SI, Section 5). We find that $9$ out of $10$ of the specialists’
choices of the foundational papers are identified by the algorithm, resulting
in a $90\%$ effectiveness of our automated approach.

We would like to thank Alice Grishchenko for help with data visualization.
X.M. is indebted to Yanchen Liu, Rodrigo Dorantes-Gilardi, Bingsheng Chen,
Dakota Murray, Alexander J. Gates, Louis M. Shekhtman, and Jing Ma for
fruitful discussions. X.M. was supported by the NetSeed: Seedling Research
Award of Northeastern University.

[42]

## References

  * (1) E Garfield, Is citation analysis a legitimate evaluation tool?  Scientometrics 1, 359–375 (1979).
  * (2) JA Evans, Electronic publication and the narrowing of science and scholarship.  Science 321, 395–399 (2008).
  * (3) B Uzzi, S Mukherjee, M Stringer, B Jones, Atypical combinations and scientific impact.  Science 342, 468–472 (2013).
  * (4) RK Merton, Social Theory and Social Structure.  (Free Press, Boston), Enlarged edition, (1968).
  * (5) E Garfield, The obliteration phenomenon.  Current Contents pp. 5–7 (1975).
  * (6) A Einstein, Die feldgleichungen der gravitation.  Sitzungsberichte der Preussischen Akademie der Wissenschaften zu Berlin pp. 844–847 (1915).
  * (7) WG Unruh, Notes on black-hole evaporation.  Phys. Rev. D 14, 870–892 (1976).
  * (8) KS Thomas, The development of eponymy; a case study of the southern blot.  Scientometrics 24, 405–417 (1992).
  * (9) W Marx, M Cardona, The citation impact outside references — formal versus informal citations.  Scientometrics 80, 1–21 (2009).
  * (10) F Radicchi, S Fortunato, C Castellano, Universality of citation distributions: Toward an objective measure of scientific impact.  Proceedings of the National Academy of Sciences 105, 17268–17272 (2008).
  * (11) D Wang, C Song, AL Barabási, Quantifying long-term scientific impact.  Science 342, 127–132 (2013).
  * (12) A Zeng, et al., The science of science: From the perspective of complex systems.  Physics Reports 714-715, 1–73 (2017).
  * (13) S Fortunato, et al., Science of science.  Science 359, eaao0185 (2018).
  * (14) A Gerow, Y Hu, J Boyd-Graber, DM Blei, JA Evans, Measuring discursive influence across scholarship.  Proceedings of the National Academy of Sciences 115, 3308–3313 (2018).
  * (15) Y Ma, B Uzzi, Scientific prize network predicts who pushes the boundaries of science.  Proceedings of the National Academy of Sciences 115, 12608–12615 (2018).
  * (16) L Wu, D Wang, JA Evans, Large teams develop and small teams disrupt science and technology.  Nature 566, 378 (2019).
  * (17) A Zeng, Y Fan, Z Di, Y Wang, S Havlin, Fresh teams are associated with original and multidisciplinary research.  Nature Human Behaviour 5, 1314–1322 (2021).
  * (18) H Peng, Q Ke, C Budak, DM Romero, YY Ahn, Neural embeddings of scholarly periodicals reveal complex disciplinary organizations.  Science Advances 7, eabb9004 (2021).
  * (19) KW McCain, Eponymy and obliteration by incorporation: The case of the “Nash equilibrium”.  Journal of the American Society for Information Science and Technology 62, 1412–1424 (2011).
  * (20) KW McCain, Assessing obliteration by incorporation: Issues and caveats.  Journal of the American Society for Information Science and Technology 63, 2129–2139 (2012).
  * (21) KW McCain, Assessing obliteration by incorporation in a full-text database: JSTOR, economics, and the concept of “bounded rationality”.  Scientometrics 101, 1445–1459 (2014).
  * (22) KW McCain, Mining full-text journal articles to assess obliteration by incorporation: Herbert A. Simon’s concepts of bounded rationality and satisficing in economics, management, and psychology.  Journal of the Association for Information Science and Technology 66, 2187–2201 (2015).
  * (23) G Cabanac, Extracting and quantifying eponyms in full-text articles.  Scientometrics 98, 1631–1645 (2014).
  * (24) A Schubert, W Glänzel, G Schubert, Eponyms in science: Famed or framed?  Scientometrics 127, 1199–1207 (2022).
  * (25) G Cabanac, What is the primordial reference for …?—Redux.  Scientometrics 114, 481–488 (2018).
  * (26) J Maldacena, The large-N limit of superconformal field theories and supergravity.  International Journal of Theoretical Physics 38, 1113–1133 (1999).
  * (27) DM Blei, AY Ng, MI Jordan, Latent Dirichlet allocation.  Journal of Machine Learning Research 3, 993–1022 (2003).
  * (28) H Jelodar, et al., Latent Dirichlet allocation (LDA) and topic modeling: Models, applications, a survey.  Multimedia Tools and Applications 78, 15169–15211 (2019).
  * (29) T Mikolov, I Sutskever, K Chen, G Corrado, J Dean, Distributed representations of words and phrases and their compositionality.  arXiv preprint arXiv:1310.4546 (2013).
  * (30) J Devlin, MW Chang, K Lee, K Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805 (2018).
  * (31) K Wang, et al., Microsoft Academic Graph: When experts are not enough.  Quantitative Science Studies 1, 396–413 (2020).
  * (32) DA King, The scientific impact of nations.  Nature 430, 311–316 (2004).
  * (33) E Leahey, Gender differences in productivity: Research specialization as a missing link.  Gend. Soc. 20, 754–780 (2006).
  * (34) W An, Y Ding, The landscape of causal inference: Perspective from citation network analysis.  Am. Stat. 72, 265–277 (2018).
  * (35) J Huang, AJ Gates, R Sinatra, AL Barabási, Historical comparison of gender inequality in scientific careers across countries and disciplines.  Proceedings of the National Academy of Sciences 117, 4609–4616 (2020).
  * (36) HG Small, Cited documents as concept symbols.  Social studies of science 8, 327–340 (1978).
  * (37) HG Small, On the shoulders of Robert Merton: Towards a normative theory of citation.  Scientometrics 60, 71–79 (2004).
  * (38) AJ Gates, Q Ke, O Varol, AL Barabási, Nature ’s reach: Narrow work has broad impact.  Nature 575, 32–34 (2019).
  * (39) RK Merton, The matthew effect in science, ii: Cumulative advantage and the symbolism of intellectual property.  isis 79, 606–623 (1988).
  * (40) RK Merton, The Matthew effect in science: The reward and communication systems of science are considered.  Science 159, 56–63 (1968).
  * (41) C Catalini, N Lacetera, A Oettl, The incidence and role of negative citations in science.  Proc. Natl. Acad. Sci. 112, 13823–13826 (2015).
  * (42) P Buneman, et al., Why data citation isn’t working, and what to do about it.  Database 2020 (2020).
  * (43) A Sinha, et al., An overview of Microsoft Academic Service (MAS) and applications in WWW ’15 Companion: Proceedings of the 24th International Conference on World Wide Web.  (Association for Computing Machinery, New York, NY, USA), pp. 243–246 (2015).
  * (44) T Saier, M Färber, unarxive: A large scholarly data set with publications’ full-text, annotated in-text citations, and links to metadata.  Scientometrics pp. 3085–3108 (2020).
  * (45) P Ginsparg, The global village pioneers.  Learned Publishing 22, 95–100 (2009).

See pages 2- of HiddenCitation_pnas_suppl.pdf See pages - of data_suppl.pdf


>1;m #1 mm #2#1 #1 | #2

# BAN: Detecting Backdoors Activated by
Adversarial Neuron Noise

Xiaoyun Xu
Radboud University
<EMAIL_ADDRESS>
&Zhuoran Liu
Radboud University
<EMAIL_ADDRESS>
Stefanos Koffas
Delft University of Technology
<EMAIL_ADDRESS>
&Shujian Yu
Vrije Universiteit Amsterdam
<EMAIL_ADDRESS>
&Stjepan Picek
Radboud University
<EMAIL_ADDRESS>

###### Abstract

Backdoor attacks on deep learning represent a recent threat that has gained
significant attention in the research community. Backdoor defenses are mainly
based on backdoor inversion, which has been shown to be generic, model-
agnostic, and applicable to practical threat scenarios. State-of-the-art
backdoor inversion recovers a mask in the feature space to locate prominent
backdoor features, where benign and backdoor features can be disentangled.
However, it suffers from high computational overhead, and we also find that it
overly relies on prominent backdoor features that are highly distinguishable
from benign features. To tackle these shortcomings, this paper improves
backdoor feature inversion for backdoor detection by incorporating extra
neuron activation information. In particular, we adversarially increase the
loss of backdoored models with respect to weights to activate the backdoor
effect, based on which we can easily differentiate backdoored and clean
models. Experimental results demonstrate our defense, BAN, is 1.37$\times$ (on
CIFAR-10) and 5.11$\times$ (on ImageNet200) more efficient with 9.99% higher
detect success rate than the state-of-the-art defense BTI-DBF [43]. Our code
and trained models are publicly
available.111https://anonymous.4open.science/r/ban-4B32

##  1 Introduction

Deep neural networks (DNNs) are known to be vulnerable to backdoor attacks, a
setting where the attacker trains a malicious DNN to perform well on normal
inputs but behave inconsistently when receiving malicious inputs that are
stamped with triggers [8]. The malicious model is obtained by training with
poisoned data [8, 5], by tampering with the training process [21, 2], or by
directly altering the model’s weights [13]. Backdoors are proposed for various
domains, from computer vision [8, 21, 5, 25, 39, 24] to graph data [41] and
neuromorphic data [1]. Still, backdoors in computer vision received most of
the attention of the research community, which also means there is a
significant variety of triggers. For instance, a trigger can be a pixel patch
[8], an existing image [21], dynamic perturbation [24], or image warping [25].
Various backdoor attacks target different stages of the machine learning model
pipeline, inducing several threat scenarios in which defenses are developed by
exploiting different knowledge.

Trigger inversion is a principled method that makes minimal assumptions about
the backdoor attacks in the threat model [37, 38, 43], and it has clear
advantages to training-time [17] or run-time defenses [23, 18, 10]. _Input
space_ trigger inversion is first introduced by Neural Cleanse (NC) [35],
where potential triggers for all target classes are reversed by minimizing the
model loss of clean inputs. Median absolute deviations of reversed triggers
from all classes are then calculated to detect triggers. More recently, input
space trigger inversion methods have been shown to be ineffective against
_feature space_ backdoor attacks [25, 37]. To address this problem, featureRE
[37] proposes a detection method using feature space triggers. The authors
observe that features of backdoored and benign inputs are separable in the
feature space by hyperplanes. Unicorn [38] formally defines and analyzes
different spaces for trigger inversion problems. An invertible input space
transformation function is proposed to map the input space to others, such as
feature space, and to reconstruct the backdoor triggers. These trigger
inversion methods can be considered as an optimization problem for the
targeted class. They optimize the input images to mislead the model under
various constraints and recover strong triggers in different spaces. The
state-of-the-art trigger inversion method, BTI-DBF [43], increases the
inversion efficiency by relaxing the dependency of trigger inversion on the
target labels. BTI-DBF trains a trigger generator by minimizing the
differences between benign samples and their generated poisoned version in
decoupled benign features while maximizing the differences in remaining
backdoor features. Feature space trigger inversion defenses are generic and
effective against most backdoor attacks. However, we show that BTI-DBF may
fail against BadNets, as its backdoor is not prominent in the feature space
(see Section 3.4). In addition, existing works still suffer from huge
computational overhead and overly rely on _prominent backdoor features_.

To resolve this shortcoming, we propose a backdoor defense called detecting
Backdoors activated by Adversarial neuron Noise (BAN). Our defense is inspired
by the finding that backdoored models are more sensitive to adversarial noise
than benign models [42, 9], and neuron noise can be adversarially manipulated
to indicate backdoor activations [40]. Specifically, BAN generates adversarial
neuron noise, where the weights of the victim model are adversarially
perturbed to maximize the classification loss on a clean data set.
Simultaneously, trigger inversion is conducted in the victim model’s feature
space to calculate the mask for benign and backdoor features. Clean inputs
with masked feature maps are then fed to the adversarially perturbed model,
based on the outputs of which backdoored models can be differentiated. Figure
1 presents a t-SNE visualization of the feature space for backdoored and clean
models when they are perturbed with different levels of adversarial neuron
noise. It can be observed that the backdoored model misclassifies parts of the
data from all classes as the target class when the adversarial neuron noise
increases, while the clean model has fewer misclassifications under the same
level of noise. By leveraging the induced difference, BAN can successfully
detect and mitigate backdoor attacks in both input and feature space.

We make the following contributions:

  * •

We find a generalization shortcoming in current trigger inversion-based
detection methods (featureRE [37], Unicorn [38], BTI-DBF [43]). In particular,
feature space detections overly rely on highly distinguishable prominent
backdoor features. We provide an in-depth analysis of trigger inversion-based
backdoor defenses, showing that prominent backdoor features that are exploited
by state-of-the-art defenses to distinguish feature space backdoors may not be
suitable for the identification of input space backdoors.

  * •

We propose detecting Backdoors activated by Adversarial neuron Noise (BAN) to
mitigate this generalization shortcoming by introducing neuron noise into
feature space trigger inversion. BAN includes an adversarial learning process
to incorporate neuron activation information into the inversion-based backdoor
detection. Experimental results demonstrate that BAN is 1.37$\times$ (on
CIFAR-10) and 5.11$\times$ (on ImageNet200) more efficient with 9.99% higher
detect success rate than the state-of-the-art defense BTI-DBF [43].

  * •

We also exploit the neuron noise to further design a simple yet effective
defense for removing the backdoor, such that we build a complete defense
framework.

Figure 1: The feature plots of backdoor and benign models with neuron noise
using ResNet18 on CIFAR-10. The darker blue represents the target label. As
noise increases, the backdoor model identifies more inputs from each class as
the target label. The clean model has fewer errors, and there is no
significant increase in the number of misclassifications to the target class.

##  2 Backdoors

Attacks. Backdoor attacks [8, 21, 2, 25, 24, 39, 34, 7, 19, 3] refer to
injecting a secret functionality into the victim model that is activated
through malicious inputs that contain the trigger. To this end, substantial
research has been proposed by poisoning a small percentage of training data
using small and static triggers [8, 5, 21]. Early attacks generate backdoors
in input space, where BadNets [8] is the first backdoor attack in DNNs. Blend
[5] proposed three injection strategies to blend translucent images into
inputs of DNNs as triggers. The authors controlled the transparency of the
trigger to allow the trade-off between strength of attack and invisibility.
Although these attacks work well, their triggers are still perceptible to
humans and can be easily detected by backdoor defenses, such as Activation
Clustering (AC) [4] and NC [35].

Dynamic and imperceptible triggers [25, 24, 39, 7], including feature space
backdoor triggers [19, 24], are explored to bypass both human observers and
input space defenses. IAD [24] designs input-specific triggers. To evaluate
the uniqueness of dynamic triggers, the authors designed a cross-trigger test
to determine whether the trigger of one input is reusable to others. WaNet
[25] proposes warping-based triggers, which are unnoticeable and smooth in the
input space. Bpp [39] exploits vulnerabilities in the human visual system and,
by using image quantization and dithering, introduces invisible triggers that
are stealthier than previous attacks.

Adaptive backdoor attacks [30, 27, 23] are built to systematically evaluate
defenses, where attacks discourage the indistinguishability of latent
representations of poisoned and benign inputs in the feature space. Adap-Blend
[27] divides the trigger image into 16 pieces and randomly applies only 50% of
the trigger during data poisoning. They use the full trigger image at
inference time to mislead the poisoned model. SSDT [23] considers the
combination of source-specific and dynamic-trigger backdoors. Only the inputs
from victim classes (source) with the dynamic triggers are classified to the
target labels, which encourages the generalization of more diverse trigger
patterns from different inputs. In this paper, we evaluate BAN against various
types of attacks, including input space, feature space, and adaptive attacks,
to provide a systematic evaluation resembling practical threats.

Defenses. Trigger inversion-based backdoor defense is considered one of the
most practical and general defenses against backdoors [37, 38, 43]. The
recovered trigger is used to determine whether the model is backdoored. For
example, NC [35] reverses input space triggers to detect backdoors by
selecting significantly smaller triggers in size. Other methods, such as ABS
[20] and featureRE [37], usually determine whether there is a backdoor based
on the attack success rate of the trigger. More specifically, given a DNN
model $f$ and a small set of clean data
$\mathcal{D}_{c}=\\{\mathbf{x}_{n},y_{n}\\}_{n=1}^{N}$, NC recovers the
potential trigger by solving the following objective:

$\mathop{\min}\limits_{\mathbf{m},\mathbf{t}}\mathcal{L}(f((1-\mathbf{m})\odot\mathbf{x}+\mathbf{m}\odot\mathbf{t}),y_{t})+\lambda|\mathbf{m}|,$
(1)

where $(\mathbf{x},y)\in\mathcal{D}_{c}$ and $\mathbf{m}$ is the trigger mask,
$\mathbf{t}$ is the trigger pattern, and $y_{t}$ is the target label. The mask
$\mathbf{m}$ determines whether the pattern will replace the pixel.
$\mathcal{L}$ is the cross-entropy loss function. Most prior works [11, 29,
20, 37, 38] follow this design to conduct trigger inversion for all possible
target labels. For example, Tabor [11] adds more constraints to the NC
optimization problem according to the overly large (large size triggers but no
overlaps with the true trigger) and overlaying (overlap the true trigger but
with redundant noise) problem of NC. Moving from input space to feature,
FeatureRE [37] utilizes feature space constraint according to an observation
that neuron activation values representing the backdoor behavior are
orthogonal to others. Unicorn [38] proposes a transformation function that
projects from the input space to other spaces, such as feature space [25] and
numerical space [39]. Then, the authors conduct trigger inversion in different
spaces.

Unlike previous NC-style methods, recent works [43, 36] explore different
optimization objectives to avoid the time-consuming optimization for all
possible target classes or to avoid the fixed mask-pattern design as advanced
attacks utilize more complex and dynamic triggers. BTI-DBF [43] takes
advantage of the prior knowledge that benign and backdoored features are
decoupled in the feature space. It distinguishes them by the optimization
objective, where benign features contribute to the correct predictions and
backdoored features lead to wrong predictions. Based on the decoupled
features, BTI-DBF trains a trigger generator by minimizing the difference
between benign samples and their generated version according to the benign
features and maximizing the difference according to the backdoored features.
Feature space backdoor defenses are developed based on the fact that backdoor
features are highly distinguishable from benign features. However, this
finding is not consistently valid for input space attacks where the feature
difference is small (See Sections 3.4).

##  3 BAN Method

###  3.1 The Pipeline of Training Backdoor Models

For brevity, consider an $L$-layer fully connected network $f$ (similar
principles apply to convolutional networks) that has
$l=l_{1}+l_{2}+\cdots+l_{L}$ neurons. $\mathbf{x}_{n}\in\mathbb{R}^{d_{x}}$
and $y_{n}\in\\{0,1\\}^{d_{y}}$ are the $n_{\text{th}}$ image and its label in
$d_{x}$ and $d_{y}$ dimensional spaces, respectively. The attacker creates a
poisoned dataset $\mathcal{D}_{p}$ by poisoning generators $G_{X}$ and $G_{Y}$
for a subset of the whole training dataset, i.e.,
$\mathcal{D}_{p}=\mathcal{D}_{c}\cup\mathcal{D}_{b}$. $\mathcal{D}_{c}$ is the
original clean data. $\mathcal{D}_{b}$ is the poisoned backdoor data,
$\mathcal{D}_{b}=\\{(\mathbf{x}^{\prime},y^{\prime})|\mathbf{x}^{\prime}=G_{X}(\mathbf{x}),y^{\prime}=G_{Y}(y),(\mathbf{x},y)\in\mathcal{D}-\mathcal{D}_{c}\\}$.
In all-to-one attacks, $G_{Y}(y)=y_{t}$, $y_{t}$ is the attacker-specified
target class. In our experiments, we consider the dirty-label attack. In all-
to-all attacks, ususally $G_{Y}(y)=(y+1)$ [43, 25, 39], which is also what we
chose in our experiemnts. In the training stage, the backdoor is injected into
the model by training with $\mathcal{D}_{p}$, i.e., minimizing the training
loss on $\mathcal{D}_{p}$ to find the optimal weights and bias
$(\mathbf{w}^{*},\mathbf{b}^{*})$:

$\mathop{\min}\limits_{\mathbf{w},\mathbf{b}}\mathcal{L}_{\mathcal{D}_{p}}(\mathbf{w},\mathbf{b})=\mathop{\mathbb{E}}\limits_{(\mathbf{x},y)\in\mathcal{D}_{p}}\ell(f(\mathbf{x};\mathbf{w},\mathbf{b}),y),$
(2)

where $\ell(\cdot,\cdot)$ is the cross-entropy. In the inference stage, the
backdoored model predicts an unseen input $\hat{\mathbf{x}}$ as its true label
$\hat{y}$ but predicts $G_{X}(\hat{\mathbf{x}})$ as $G_{Y}(\hat{y})$:
$f(G_{X}(\hat{\mathbf{x}});\mathbf{w},\mathbf{b})=G_{Y}(\hat{y})$.

###  3.2 Threat Model

Attacker’s goal. We consider the attacker to be the pre-trained model’s
provider. The attacker aims to inject stealthy backdoors into the pre-trained
models. The pre-trained models perform well on clean inputs but predict the
attacker-chosen target label when receiving backdoor inputs.

Attacker knowledge. The attacker has white-box access to the model, including
training data, architectures, hyperparameters, and model weights.

Defender’s goal and knowledge. The main goal is to detect whether a given
model is backdoored and then remove the potential backdoor according to the
detection results. Following [37, 38, 43], we assume the defender has white-
box access to the model and holds a few local clean samples. However, the
defender does not have access to the training data and has no knowledge of the
backdoor trigger.

###  3.3 Detection with Neuron Noise

Based on previous findings that adversarial noise can activate backdoors [42,
40, 9], we design a two-step method for backdoor detection. First, we search
for noise on neurons that can activate the potential backdoor. Then, we
decouple the benign and backdoored features using a learnable mask of the
latent feature (the output before the final classification layer).

Neuron Noise. In backdoor models, there are two types of neurons: benign and
backdoor [20]. The backdoor neurons build a shortcut from $G_{X}(\mathbf{x})$
to $G_{Y}(y)$. _Neuron noise_ is generated noises on neurons to maximize model
loss on clean samples in an adversarial manner [22]. The noise on benign
neurons evenly misleads prediction to all classes, while the noise on backdoor
neurons tends to mislead prediction to the target label due to the backdoor
shortcut, as shown in Figure 1. Therefore, backdoor models with noise behave
differently from benign models with noise, as there are no backdoor neurons in
the benign models.

Given the $j_{\text{th}}$ neuron connecting the $i_{\text{th}}$ layer and
$(i-1)_{\text{th}}$ layer, we denote its weight and bias with
$\mathbf{w}_{ij}$ and $b_{ij}$, respectively. Neuron noise can be added to the
neuron by multiplying its weight and bias with a small number:
$(1+\delta_{ij})\mathbf{w}_{ij}$ and $(1+\xi_{ij})b_{ij}.$ Then, the output of
the neuron is:

$h_{ij}=\sigma\big{(}(1+\delta_{ij})\mathbf{w}_{ij}^{\top}\mathbf{h}_{(i-1)}+(1+\xi_{ij})b_{ij}\big{)},$
(3)

where $\mathbf{h}_{(i-1)}$ is the output of the previous layer and
$\sigma(\cdot)$ is the nonlinear function (activation). The noise on all
neurons are represented by
$\bm{\delta}=[\delta_{1,1},\cdots,\delta_{l_{1},1},\cdots,\delta_{1,L},\cdots,\delta_{l_{L},L}]$
and
$\bm{\xi}=[\xi_{1,1},\cdots,\xi_{l_{1},1},\cdots,\xi_{1,L},\cdots,\xi_{l_{L},L}]$.
The $\bm{\delta}$ and $\bm{\xi}$ are optimized via a maximization to increase
the cross-entropy loss on the clean data:

$\displaystyle\mathop{\max}\limits_{\bm{\delta},\bm{\xi}\in
S}\mathcal{L}_{\mathcal{D}_{c}}((1+\bm{\delta})\odot\mathbf{w},(1+\bm{\xi})\odot\mathbf{b}),$
(4) $\displaystyle
S=B(\mathbf{w};\epsilon)=\\{\setargs{\delta\in\mathbb{R}^{l}|\delta\leq\epsilon}\\},$

where $S$ is the ball function of the radius $\epsilon$ in the $l$ dimensional
space. $\bm{\delta}$ and $\bm{\xi}$ share the ball function $S$ as the maximum
noise size. The maximization in Eq. (4) can be solved by PGD [22] algorithm
with a random start, as PGD can better explore the entire searching space to
mitigate local minima [22].

Figure 2: Model’s clean accuracy with (red dots) and without (blue dots) the
mask defined in Eq. (6). Only the backdoored models are affected by the noise.

Feature Decoupling with Mask. The neuron noise activates the backdoor (see
Figure 1) and misleads the predictions to the target label. However, the
performance has high variance when searching for noise multiple times, which
we conjecture is caused by random initialization. Therefore, inspired by [43],
we further introduce a feature decoupling process to enhance the effect of
noise on backdoored features but maintain a decreased effect on benign
features. Specifically, the network $f$ is decomposed into
$g=f_{1}\circ\cdots\circ f_{L-1}$ and $f_{L}$, where $f_{L-1}$ extracts the
latent features from $\mathbf{x}$, while $f_{L}$ is the classification layer.
Then, we use a mask $\mathbf{m}$ on top of the latent feature $g(\mathbf{x})$
to decouple benign and backdoored features. The optimization objective can be
written as:

$\mathop{\min}\limits_{\mathbf{m}}\mathcal{L}(f_{L}\circ(g(\mathbf{x})\odot\mathbf{m}),y)-\mathcal{L}(f_{L}\circ(g(\mathbf{x})\odot(1-\mathbf{m})),y)+\lambda_{1}|\mathbf{m}|,$
(5)

where $\odot$ is the element-wise product. This optimization divides the
latent features into two parts by the mask while maintaining a relatively
small size of $\mathbf{m}$. Note that the regularizer for $\mathbf{m}$ in Eq.
(5) is necessary. Otherwise, $\mathbf{m}$ will become a dense matrix full of
ones to focus on the positive part of Eq. (5) because maintaining only the
positive part (without penalty of $|\mathbf{m}|$) already satisfies the
optimization objective. Finally, we apply the negative mask $(1-\mathbf{m})$
on top of latent feature of $f$ to enhance the backdoor effect. The final
output is:

$f_{L}\circ\Big{(}g\big{(}\mathbf{x};(1+\bm{\delta})\odot\mathbf{w},(1+\bm{\xi})\odot\mathbf{b}\big{)}\odot(1-\mathbf{m})\Big{)}.$
(6)

In Figure 2, the blue dots show the accuracy after adding noise to the model.
The red dots show the accuracy (with noise) while applying the feature mask to
the model using Eq. (6). Our feature masks do not affect the performance of
benign models. However, we see that the performance is significantly decreased
for backdoored models. This decrease is caused by the model only using
backdoor features (through the negative mask), which means the backdoor is
activated more frequently. Finally, a suspect model is determined backdoored
if the prediction using Eq. (6) has a high attack success rate. For all-to-one
attacks, the misclassification will be concentrated into one label, which is
the target label. For all-to-all attacks, we can do the same as for all-to-one
attacks but evaluate the prediction for each class independently.

###  3.4 Improving BTI-DBF

This section shows that the most recent work, BTI-DBF [43], may fail to
capture the backdoor features by its decoupling method in Table 2. We show how
to patch BTI-DBF using a simple solution. The BTI-DBF is the original version,
and BTI-DBF* is our improved version. The main pipeline of BTI-DBF consists of
two steps: (1) decoupling benign and backdoor features and (2) trigger
inversion by minimizing the distance between benign features and maximizing
the distance between backdoor features. We found that the defense’s first step
may introduce errors in the decoupled features. Similar to our Eq. (5), the
decoupling of BTI-DBF can be written as:

$\mathop{\min}\limits_{\mathbf{m}}\mathcal{L}(f_{L}\circ(g(\mathbf{x})\odot\mathbf{m}),y)-\mathcal{L}(f_{L}\circ(g(\mathbf{x})\odot(1-\mathbf{m})),y).$
(7)

However, this equation has no constraints on the feature mask $\mathbf{m}$.
Overall, the optimization objective is to decrease the loss. Obviously, BTI-
DBF’s decoupling encourages the norm of the mask to increase so that the loss
will focus on the positive part and ignore the negative part because the
negative part goes against the overall objective. Finally, the mask will be a
dense matrix that is full of ones. The
$f_{L}\circ(g(\mathbf{x})\odot(1-\mathbf{m}))$ is ignored due to multiplying
by zero. We propose a simple solution to fix the problem by adding a
regularizer of the size of the mask to the loss, i.e., we use our Eq. (5) as
the first step of BTI-DBF*. According to Table 2, BTI-DBF* successfully
overcomes BTI-DBF’s shortcomings.

###  3.5 Backdoor Defense

After determining whether a suspect model is backdoored, we can fine-tune the
backdoored model to remove the backdoor. However, standard fine-tuning using
clean data does not effectively remove the backdoor because it does not
activate it. Therefore, we propose using optimized neuron noise to fine-tune
the model. In the optimization of the neuron noise, the objective is to
increase the loss of $\mathcal{L}(f(\mathbf{x}),y)$. We consider both the
benign and backdoor neurons to contribute to the increase in loss when
optimizing the noise. Indeed, on benign neurons, the neuron noise misleads $f$
to any result other than the true label $y$, while the noise misleads $f$ to
the target label $G_{Y}(y)$ on backdoor neurons. Therefore, a straightforward
method is to decrease the loss between noise output and the true label. The
loss for our noise fine-tuning can be written as:

$\mathop{\min}\limits_{\mathbf{w},\mathbf{b}}\mathcal{L}(f(\mathbf{x};\mathbf{w},\mathbf{b}),y)+\lambda_{2}\mathcal{L}(f(\mathbf{x};(1+\bm{\delta})\odot\mathbf{w},(1+\bm{\xi})\odot\mathbf{b}),y).$
(8)

##  4 Experimental Results

Datasets and Architectures. The datasets for our experiments include CIFAR-10
[15], GTSRB [32], Tiny-ImageNet [16], and a subset of ImageNet-1K [6]. The
ImageNet subset contains 200 classes, which is referred to as ImageNet200. BAN
is evaluated using three architectures: ResNet18 [12], VGG16 [31], and
DenseNet121 [14]. Please refer to Appendix B.1 for more details.

Attack Baselines. Our experiments are conducted using seven attacks: BadNets
[8], Blend [5], WaNet [25], IAD [24], BppAttack [39], Adap-Blend [27], and
SSDT [23], which are commonly used in other works [43, 37, 38, 44, 28]. The
BadNets [8] and Blend [5] are designed for input space. WaNet [25], IAD [24],
and BppAttack [39] are designed for feature space. Adap-Blend [27] and SSDT
[23] (for adaptive evaluation) are state-of-the-art attacks that have been
recently introduced to bypass backdoor defenses. The main idea of Adap-Blend
[27] and SSDT [23] is to obscure the latent separation in features of benign
and backdoor samples. More details can be found in Appendix B.2.

Defense Baselines. BAN is compared to five representative methods: Neural
Cleanse (NC) [35], Tabor [11], FeatureRE [37], Unicorn [38], and BTI-DBF [43].
NC and Tabor are designed for input space attacks, while the other three are
designed for feature space and dealing with the latest advanced attacks. BAN
uses only 1% and 5% of training data for detection and defense, respectively.
Note the data used for our defense is not used for training models, i.e., the
defender has no knowledge of the model to be defended. More details can be
found in Appendices B.3 and B.4. We use bold font to denote the best results.

###  4.1 The Performance of Backdoor Detection

Main Results. In Table 1, BAN shows better results on CIFAR-10 than all
baselines, especially on advanced attacks. Results on other datasets are
presented in Appendix C. We note that the advanced detection methods
(FeatureRE, Unicorn, and BTI-DBF) perform worse than NC on simple attacks
(BadNets and Blend). We hypothesize this is because the backdoor features
generated by BadNets are less obvious on feature channels than advanced
attacks in the feature space, such as WaNet and IAD. Figure 3 shows that
BadNets features are weaker than features from advanced attacks using the
feature mask in Eq. (5). Specifically, we optimize the feature mask to
disentangle the benign and backdoor features for four attacks.

Figure 3: BadNet features are weaker in the feature space when we use the mask
that disentangles the benign and backdoor features. Defenses that are biased
towards large differences may not work in cases like BadNets.

Then, we compute two cross-entropy loss values using the positive mask
($\mathbf{m}$) and the negative mask ($1-\mathbf{m}$) for benign and backdoor
features, respectively. The average loss values and standard deviation for
four models for each attack are plotted in Figure 3. The negative loss value
of BadNets is much smaller than others, which means BadNets features are
weaker than others with regard to misleading the model to the backdoor target.
Recent defenses usually add more regularizers to their losses and optimization
objectives to counteract powerful backdoor attacks. These regularizers
encourage the capturing of strong features but omit weak ones. Thus, recent
advanced detection methods can perform worse on BadNets than NC.

Time consumption. BAN is efficient and scalable as we do not iterate over all
target classes. Figure 4 demonstrates that BAN uses substantially less time
than all baselines. BAN is also more scalable for larger architectures or
datasets. BIT-DBF (76.90s) is 1.37$\times$ slower than BAN (55.95s) on
CIFAR-10 with ResNet18, and BIT-DBF (5,792.51s) without pre-training is
5.11$\times$ slower than ours (1,132.85s) on ImageNet200 with ResNet18.
FeatureRE (297.74s) is 5.32$\times$ slower than ours on CIFAR-10 with
ResNet18, and it (6,053.62s) is 45.14$\times$ slower on CIFAR-10 with
DenseNet121 than ours (134.10s).

Figure 4: Time consumption of detection baselines on ResNet18 (in seconds) for all three datasets. BAN uses significantly less time than the baselines. Table 1: The detection results under different model architectures on CIFAR-10. The “Bd.” refers to the number of models that the defense identifies as backdoored. The “Acc.” refers to detection success accuracy. The best results are marked in bold. BTI-DBF* refers to an improved version (details in Section 3.4). Model | Attack | NC | Tabor | FeatureRE | Unicorn | BTI-DBF* | Ours
---|---|---|---|---|---|---|---
Bd. | Acc. | Bd. | Acc. | Bd. | Acc. | Bd. | Acc. | Bd. | Acc. | Bd. | Acc.
ResNet18 | No Attack | 0 | 100% | 0 | 100% | 2 | 90% | 6 | 70% | 0 | 100% | 0 | 100%
BadNets | 20 | 100% | 20 | 100% | 14 | 70% | 18 | 90% | 18 | 90% | 20 | 100%
Blend | 20 | 100% | 20 | 100% | 20 | 100% | 19 | 95% | 20 | 100% | 18 | 90%
WaNet | 11 | 55% | 8 | 40% | 15 | 75% | 20 | 100% | 18 | 90% | 20 | 100%
IAD | 0 | 0% | 0 | 0% | 15 | 75% | 11 | 55% | 20 | 100% | 20 | 100%
Bpp | 0 | 0% | 1 | 5% | 12 | 60% | 17 | 85 % | 20 | 100% | 20 | 100%
VGG16 | No Attack | 0 | 100% | 0 | 100% | 3 | 85% | 6 | 70% | 6 | 70% | 0 | 100%
BadNets | 18 | 90% | 16 | 80% | 13 | 65% | 16 | 80% | 18 | 90% | 19 | 95%
Blend | 19 | 95% | 19 | 95% | 16 | 80% | 18 | 90% | 16 | 80% | 17 | 85%
WaNet | 10 | 50% | 9 | 45% | 12 | 60% | 18 | 90% | 16 | 80% | 20 | 100%
IAD | 0 | 0% | 0 | 0% | 8 | 40% | 17 | 85% | 20 | 100% | 20 | 100%
Bpp | 9 | 45% | 10 | 50% | 5 | 25% | 15 | 75% | 14 | 70% | 18 | 90%
DenseNet121 | No Attack | 0 | 100% | 0 | 100% | 5 | 75% | 8 | 60% | 3 | 85% | 0 | 100%
BadNets | 18 | 90% | 20 | 100% | 19 | 95% | 15 | 75% | 17 | 85% | 20 | 100%
Blend | 20 | 100% | 20 | 100% | 12 | 60% | 18 | 90% | 19 | 95% | 20 | 100%
WaNet | 13 | 65% | 10 | 50% | 20 | 100% | 17 | 85% | 14 | 70% | 19 | 95%
IAD | 0 | 0% | 0 | 0% | 14 | 70% | 16 | 80% | 14 | 70% | 19 | 95%
Bpp | 0 | 0% | 0 | 0% | 16 | 80% | 8 | 40% | 16 | 80% | 20 | 100%
Average |  | 60.56% | 59.17% | 72.5% | 78.61% | 86.39% | 97.22%
Table 2: The detection results of BTI-DBF and BTI-DBF* using ResNet18 and CIFAR-10. Attack | BTI-DBF | BTI-DBF* | Ours
---|---|---|---
Bd. | Acc. | Bd. | Acc. | Bd. | Acc.
No Attack | 0 | 100% | 0 | 100% | 0 | 100%
BadNets | 5 | 25% | 18 | 90% | 20 | 100%
Blend | 0 | 0% | 20 | 100% | 18 | 90%
WaNet | 7 | 35% | 18 | 90% | 20 | 100%
IAD | 19 | 95% | 20 | 100% | 20 | 100%
Bpp | 20 | 100% | 20 | 100% | 20 | 100%

###  4.2 The Performance of Backdoor Defense

A complete backdoor defense framework should include both detection and
defense. The goal of defense is to decrease the attack success rate (ASR) of
backdoor triggers. Detection before the defense is also necessary because the
defense usually decreases the performance on benign inputs [45, 43, 37, 44,
46]. In Section 3.5, we propose a simple and effective fine-tuning method
using the noise that activates the backdoor. Table 3 compares BAN with three
baselines: plain fine-tuning, featureRE [37], and BTI-DBF(U) [43]. Plain fine-
tuning refers to training the backdoor model using the same hyperparameters as
BAN but without the noise loss in Eq. (8). The featureRE [37] and BTI-DBF(U)
[43] refer to the defense methods from the respective paper. We use default
hyperparameters for featureRE [37] and BTI-DBF(U) [43]. For plain fine-tuning
and BAN, we use a small learning rate (0.005) to avoid jumping out of the
current optimal parameters of the well-trained model. Then, we use
$\lambda_{2}$ (0.5) for Eq. (8) for the trade-off between robustness against
backdoor and clean accuracy. We fine-tune for a short schedule of 25 epochs,
as the model is well-trained. Table 7 in Appendix B.4 shows defense
performance with standard deviation on different hyperparameters, which
supports our choice. Tables 3 and 4 demonstrate that our fine-tuning method
effectively removes the backdoor while preserving high accuracy in benign
inputs. We provide a comparison between our find-tuning with ANP [40] in Table
11, Appendix C.4, as ANP uses the neuron noise for pruning backdoor neurons.

Table 3: Defense against 5 attacks using ResNet18. BA refers to benign accuracy on clean data. Attack | No defense | Fine-tuning | featureRE | BTI-DBF(U) | Ours |
---|---|---|---|---|---|---
BA | ASR | BA | ASR | BA | ASR | BA | ASR | BA | ASR |
BadNets | 93.37 | 99.41 | 92.93 | 87.81 | 93.15 | 99.79 | 91.26 | 13.12 | 92.06 | 1.97 |
Blend | 94.60 | 100 | 93.07 | 99.99 | 93.20 | 39.28 | 91.86 | 100 | 92.72 | 4.10 |
WaNet | 93.57 | 99.37 | 93.05 | 1.10 | 93.67 | 0.03 | 90.30 | 4.89 | 92.05 | 0.91 |
IAD | 93.17 | 97.88 | 94.11 | 0.46 | 92.73 | 0.0 | 89.54 | 1.59 | 92.78 | 1.48 |
Bpp | 94.29 | 99.93 | 93.85 | 4.46 | 94.21 | 98.13 | 90.61 | 2.73 | 92.54 | 2.58 |
Average | 93.80 | 99.32 | 93.40 | 38.76 | 93.39 | 47.45 | 90.71 | 24.47 | 92.43 | 2.21 |
Table 4: Defense of backdoor attacks on Tiny-ImageNet and ImageNet200 using ResNet18. Dataset | Attack | No defense | Fine-tuning | BTI-DBF(U) | Ours
---|---|---|---|---|---
BA | ASR | BA | ASR | BA | ASR | BA | ASR
Tiny-ImageNet | WaNet | 58.32 | 99.85 | 51.53 | 1.3 | 39.49 | 0.96 | 50.69 | 0.86
IAD | 58.54 | 99.32 | 51.86 | 1.72 | 38.79 | 0.60 | 50.04 | 0.76
Bpp | 60.63 | 99.89 | 57.72 | 0.15 | 46.84 | 0.40 | 57.66 | 0.10
ImageNet200 | WaNet | 77.01 | 99.74 | 66.71 | 0.78 | 63.47 | 1.0 | 69.95 | 0.58
IAD | 76.72 | 99.75 | 69.91 | 0.42 | 64.33 | 1.24 | 72.18 | 1.30
Bpp | 78.56 | 99.88 | 70.89 | 0.82 | 67.02 | 3.10 | 72.59 | 2.68

###  4.3 Defense against All-To-All Attacks

Previous works [37, 38] usually only consider all-to-one attacks, which limits
the application in practical situations. In this section, we evaluate our
fine-tuning method under three all-to-all attacks: WaNet-All [25], IAD-All
[24], and Bpp-All [39]. FeatureRE is designed for all-to-one attacks, so we
use target label 0 here for FeatureRE. BAN is capable of handling all-to-all
attacks because we directly explore the neurons themselves instead of
optimizing for the potential targeted label. Table 5 demonstrates the
effectiveness of our method.

Table 5: Defense against all-to-all backdoor attacks. BA refers to benign accuracy on clean data. Attack | No defense | Fine-tuning | featureRE | BTI-DBF(U) | Ours
---|---|---|---|---|---
BA | ASR | BA | ASR | BA | ASR | BA | ASR | BA | ASR
WaNet-All | 93.60 | 91.86 | 92.65 | 18.03 | 93.33 | 91.96 | 91.30 | 1.72 | 92.29 | 1.11
IAD-All | 92.96 | 90.62 | 93.19 | 3.72 | 93.06 | 91.20 | 91.36 | 3.72 | 92.31 | 1.13
Bpp-All | 94.45 | 84.68 | 93.90 | 1.58 | 94.32 | 83.87 | 90.16 | 2.05 | 93.23 | 1.38
Average | 93.67 | 89.05 | 93.25 | 7.78 | 93.57 | 89.01 | 90.94 | 2.49 | 92.61 | 1.21

###  4.4 Evaluation under Adaptive Attack

Attackers may design a specific attack for defense when they know its details
[33]. In this section, we evaluate BAN against two attacks that attempt to
bypass the difference between backdoor and benign features: Adap-Blend [27]
and SSDT attack [23]. Both Adap-Blend [27] and SSDT attack [23] try to obscure
the difference between benign and backdoor latent features. Adap-Blend [27]
achieves it by randomly applying 50% of the trigger to poison the training
data, while SSDT attack [23] utilizes source-specific and dynamic-triggers to
reduce the impact of triggers on samples from non-victim classes. The source-
specific attack refers to backdoor triggers that mislead the model to the
target class only when they are applied to victim class samples. Table 6
demonstrates our approach is resistant to these two attacks, while other
methods fail. The reason is that the backdoor features of SSDT are close to
benign features in the feature space. It is difficult for other methods to
distinguish between backdoor and benign features created by SSDT. Our
detection method directly analyzes the model itself using neuron noise, which
captures the difference between backdoor and benign models concerning
parameters. See Appendix C.3 for the results of our method against these
adaptive attacks.

Table 6: The detection results under adaptive attacks on using CIFAR-10 and ResNet18. Attack | NC | Tabor | FeatureRE | Unicorn | BTI-DBF* | Ours
---|---|---|---|---|---|---
Bd. | Acc. | Bd. | Acc. | Bd. | Acc. | Bd. | Acc. | Bd. | Acc. | Bd. | Acc.
Adap-Patch | 18 | 90% | 15 | 75% | 17 | 85% | 20 | 100% | 20 | 100% | 20 | 100%
SSDT | 0 | 0% | 0 | 0% | 0 | 0% | 0 | 0% | 20 | 100% | 20 | 100%

##  5 Conclusions and Future Work

This paper proposes an effective yet efficient backdoor defense, BAN, that
utilizes the adversarial neuron noise and the mask in the feature space. BAN
is motivated by the observation that traditional defenses outperformed the
latest feature space defenses on input space backdoor attacks. To this end, we
provide an in-depth analysis showing that feature space defenses are overly
dependent on prominent backdoor features. Experimental studies demonstrate
BAN’s effectiveness and efficiency against various types of backdoor attacks.
We also show BAN’s resistance to potential adaptive attacks. Future studies
could explore a more practical detection method without assuming access to
local benign samples and better strategies for decoupling features because a
fixed mask in the feature is not always aligned with the benign and backdoor
features.

## References

  * [1] Gorka Abad, Oguzhan Ersoy, Stjepan Picek, and Aitor. Urbieta.  Sneaky spikes: Uncovering stealthy backdoor attacks in spiking neural networks with neuromorphic data.  In NDSS, 2024.
  * [2] Eugene Bagdasaryan and Vitaly Shmatikov.  Blind backdoors in deep learning models.  In USENIX Security, 2021.
  * [3] M. Barni, K. Kallas, and B. Tondi.  A new backdoor attack in cnns by training set corruption without label poisoning.  In ICIP, 2019.
  * [4] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava.  Detecting backdoor attacks on deep neural networks by activation clustering.  SafeAI Workshop @ AAAI, 2018.
  * [5] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song.  Targeted backdoor attacks on deep learning systems using data poisoning.  arXiv, 2017.
  * [6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [7] Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li.  Lira: Learnable, imperceptible and robust backdoor attacks.  In ICCV, 2021.
  * [8] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg.  Badnets: Evaluating backdooring attacks on deep neural networks.  IEEE Access, 7, 2019.
  * [9] Junfeng Guo, Ang Li, and Cong Liu.  AEVA: Black-box backdoor detection using adversarial extreme value analysis.  In ICLR, 2022.
  * [10] Junfeng Guo, Yiming Li, Xun Chen, Hanqing Guo, Lichao Sun, and Cong Liu.  SCALE-UP: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency.  In ICLR, 2023.
  * [11] Wenbo Guo, Lun Wang, Yan Xu, Xinyu Xing, Min Du, and Dawn Song.  Towards inspecting and eliminating trojan backdoors in deep neural networks.  In ICDM, 2020.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [13] Sanghyun Hong, Nicholas Carlini, and Alexey Kurakin.  Handcrafted backdoors in deep neural networks.  Advances in Neural Information Processing Systems, 35:8068–8080, 2022.
  * [14] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In CVPR, 2017.
  * [15] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [16] Ya Le and Xuan Yang.  Tiny imagenet visual recognition challenge.  CS 231N, 2015.
  * [17] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma.  Anti-backdoor learning: Training clean models on poisoned data.  In NeurIPS, 2021.
  * [18] Yinshan Li, Hua Ma, Zhi Zhang, Yansong Gao, Alsharif Abuadbba, Minhui Xue, Anmin Fu, Yifeng Zheng, Said F. Al-Sarawi, and Derek Abbott.  Ntd: Non-transferability enabled deep learning backdoor detection.  TIFS, 19:104–119, 2024.
  * [19] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu.  Invisible backdoor attack with sample-specific triggers.  In ICCV, 2021.
  * [20] Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang.  Abs: Scanning neural networks for back-doors by artificial brain stimulation.  In CCS, 2019.
  * [21] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang.  Trojaning attack on neural networks.  In NDSS, 2018.
  * [22] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In ICLR, 2018.
  * [23] Xiaoxing Mo, Yechao Zhang, Leo Yu Zhang, Wei Luo, Nan Sun, Shengshan Hu, Shang Gao, and Yang Xiang.  Robust backdoor detection for deep learning via topological evolution dynamics.  In SP, 2024.
  * [24] Anh Nguyen and Anh Tran.  Input-aware dynamic backdoor attack.  In NeurIPS, 2020.
  * [25] Tuan Anh Nguyen and Anh Tuan Tran.  Wanet - imperceptible warping-based backdoor attack.  In ICLR, 2021.
  * [26] Ren Pang, Zheng Zhang, Xiangshan Gao, Zhaohan Xi, Shouling Ji, Peng Cheng, and Ting Wang.  Trojanzoo: Towards unified, holistic, and practical evaluation of neural backdoors.  In Euro S&P, 2022.
  * [27] Xiangyu Qi, Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal.  Revisiting the assumption of latent separability for backdoor defenses.  In ICLR, 2023.
  * [28] Xiangyu Qi, Tinghao Xie, Jiachen T Wang, Tong Wu, Saeed Mahloujifar, and Prateek Mittal.  Towards a proactive $\\{$ML$\\}$ approach for detecting backdoor poison samples.  In USENIX Security, 2023.
  * [29] Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang.  Backdoor scanning for deep neural networks through k-arm optimization.  In ICML, 2021.
  * [30] Reza Shokri et al.  Bypassing backdoor detection algorithms in deep learning.  In 2020 IEEE European Symposium on Security and Privacy (EuroS&P), 2020.
  * [31] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In ICLR, 2015.
  * [32] J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel.  Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition.  Neural Networks, 2012.
  * [33] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry.  On adaptive attacks to adversarial example defenses.  In NeurIPS, 2020.
  * [34] Alexander Turner, Dimitris Tsipras, and Aleksander Madry.  Label-consistent backdoor attacks, 2019.
  * [35] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao.  Neural cleanse: Identifying and mitigating backdoor attacks in neural networks.  In SP, 2019.
  * [36] Hang Wang, Zhen Xiang, David J Miller, and George Kesidis.  Mm-bd: Post-training detection of backdoor attacks with arbitrary backdoor pattern types using a maximum margin statistic.  In SP, 2024.
  * [37] Zhenting Wang, Kai Mei, Hailun Ding, Juan Zhai, and Shiqing Ma.  Rethinking the reverse-engineering of trojan triggers.  In NeurIPS, 2022.
  * [38] Zhenting Wang, Kai Mei, Juan Zhai, and Shiqing Ma.  UNICORN: A unified backdoor trigger inversion framework.  In ICLR, 2023.
  * [39] Zhenting Wang, Juan Zhai, and Shiqing Ma.  Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning.  In CVPR, 2022.
  * [40] Dongxian Wu and Yisen Wang.  Adversarial neuron pruning purifies backdoored deep models.  In NeurIPS, 2021.
  * [41] Jing Xu, Stefanos Koffas, Oğuzhan Ersoy, and Stjepan Picek.  Watermarking graph neural networks based on backdoor attacks.  In EuroS&P, 2023.
  * [42] Xiaoyun Xu, Oguzhan Ersoy, Behrad Tajalli, and Stjepan Picek.  Universal soldier: Using universal adversarial perturbations for detecting backdoor attacks.  In DSN-W, 2024.
  * [43] Xiong Xu, Kunzhe Huang, Yiming Li, Zhan Qin, and Kui Ren.  Towards reliable and efficient backdoor trigger inversion via decoupling benign features.  In ICLR, 2024.
  * [44] Hong Zhu, Yue Zhao, Shengzhi Zhang, and Kai Chen.  Neuralsanitizer: Detecting backdoors in neural networks.  TIFS, 19:4970–4985, 2024.
  * [45] Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu.  Enhancing fine-tuning based backdoor defense with sharpness-aware minimization.  In ICCV, 2023.
  * [46] Mingli Zhu, Shaokui Wei, Hongyuan Zha, and Baoyuan Wu.  Neural polarizer: A lightweight and effective backdoor defense via purifying poisoned features.  In NeurIPS, 2023.

##  Appendix A Discussion

Limitations. Similar to existing works [35, 20, 11, 37, 38, 43], BAN assumes a
local small and benign dataset. This scenario is common, considering that
benign samples are available online, and the model provider may provide some
samples to verify the model’s performance. In addition, our fine-tuning with
neuron noise may slightly decrease the performance in benign inputs, similar
to other defense methods [37, 43, 45, 40, 17, 35, 46]. However, BAN provides
better performance and requires less time consumption.

Broader Impacts. This paper proposes a complete defense that includes
detecting and removing DNN backdoors. We believe our approach has a positive
impact on the security of DNNs. A possible negative impact is overconfidence
in robustness against backdoor attacks, as there is no theoretical guarantee
always to remove the backdoor.

##  Appendix B Additional Details for Experimental Settings

###  B.1 Datasets

CIFAR-10. The CIFAR-10 [15] contains 50,000 training images and 10,000 testing
images with the size of $3\times 32\times 32$ in 10 classes.

GTSRB. The GTSRB [32] contains 39,209 training images and 12,630 testing
images in 43 classes. In our experiments, the images are resized to $3\times
32\times 32$.

Tiny-ImageNet. Tiny-ImageNet [16] contains 100,000 training images and 10,000
testing images with the size $3\times 64\times 64$ in 200 classes.

ImageNet. ImageNet [6] contains over 1.2 million high-resolution images in
1,000 classes. Our experiments use a subset of the full ImageNet dataset,
i.e., 200 randomly selected classes. Each class has 1300 training images and
50 testing images. The ImageNet images are scaled to $3\times 224\times 224$.

###  B.2 Backdoor Models

BAN and defense baselines are evaluated using seven well-known attacks:
BadNets [8], Blend [5], WaNet [25], IAD [24], BppAttack [39], Adap-Blend [27],
and SSDT [23].

For all backdoored models, we use SGD with a momentum of 0.9, weight decay of
$5\times 10^{-4}$, and a learning rate of 0.01. We train for 200 epochs, and
the learning rate is divided by 10 at the 100-th and 150-th epochs. Same to
[40], we use 90% of training data for training backdoor models and 10% for the
validation set.

BadNets. We use a $3\times 3$ pattern to build the backdoor trigger. The
poisoning rate is 5%.

Blend. We use the random Gaussian noise and blend ratio 0.2 for backdoor
training. The poisoning rate is 5%.

Adap-Blend. We use the “hellokitty_32.png” and blend ratio of 0.2 to build
triggers. The poisoning rate is 5%.

SSDT. SSDT is a Source-Specific attack, which only misleads the victim classes
to the targeted class. We use class 1 as the victim and class 0 as the target
class.

For other attacks and hyperparameters not mentioned, we use the default
settings from the papers or their official open-source implementations.

###  B.3 Defense Baselines

BAN is compared with five representative methods, including Neural Cleanse
(NC) [35], Tabor [11], FeatureRE [37], Unicorn [38], and BTI-DBF [43]. In this
section, we describe the hyperparameters of these defenses.

NC and Tabor. We use the implementation and default hyperparameters from
TrojanZoo [26] for NC and Tabor. 1% of the training set is used to conduct 100
epochs of the trigger inversion. The learning rate is 0.01.

FeatureRE. We first tried the default hyperparameters from the FeatureRE
paper, but they could not work even for BadNets. We assume this is because the
constraints, including feature mask size and similarity between original
images and trigger images, are limited to a very small size. Therefore, we
relaxed the constraints on masks and norms to find stronger triggers,
including loss std bound=1.2, p loss bound=0.2, loss std bound=1.2, mask
size=0.06, and learning rate=0.001. We use 1% of the training set and run
FeatureRE for 400 epochs for each class.

Unicorn. The default hyperparameters for Unicorn are too powerful in our case,
and the recovered triggers can mislead any model to the target label when
applied to inputs. Every model, including benign models, is thought of as
backdoored. Therefore, we slightly increase thread values of the constraints
on Unicorn optimization to find proper triggers, including loss std bound=0.5,
SSIM loss bound=0.1, and mask size=0.01. We use 1% of the training set and run
Unicorn 40 epochs for each class.

BTI-DBF. We follow the default settings in the BTI-DBF [43] paper. Note that
BTI-DBF uses 5% of training samples for defenses.

###  B.4 BAN Settings

Detection. We use epsilon ($\epsilon$) to limit the noise added to neurons.
Otherwise, the model weights are destroyed by the noise. The $\epsilon$ values
are 0.3, 0.3, 0.2, and 0.1 for CIFAR-10, GTSRB, Tiny-ImageNet, and
ImageNet200, respectively. $\epsilon$ values are decided according to the size
of the images. We use smaller $\epsilon$ for larger datasets. We use the
30-step PGD algorithm to solve the optimization in Eq. (4) to find the noise,
i.e., $\bm{\delta}$ and $\bm{\xi}$. We use SGD and the learning rate of
$\epsilon/30$ for the PGD optimization. Then, we use Adam and the learning
rate of 0.01 to search for 20 epochs for the feature mask using Eq. (5). The
$\lambda_{1}$ for Eq. (5) equals 0.75. The two optimizations above use 1% of
the training set. Note that this 1% of data is not used to train the backdoor
models. The elements in the mask are clamped into continuous values between 0
and 1.

Defense. Our defense uses Eq. (8) to remove the backdoor. Due to limited
access to benign samples and time consumption, we only use 5% of training data
and 25 epochs for our fine-tuning. The trade-off hyperparameter
($\lambda_{2}$) is 0.5. Then, we use the most commonly used hyperparameters
for the optimizer. We use SGD with momentum=0.9, weight decay=5e-4, and
learning rate=0.005 as the optimizer. The plain fine-tuning uses the same
hyperparameters as BAN.

Table 7 shows the ablation results on fine-tuning epochs, $\lambda_{2}$, and
learning rate. Considering both performances on benign accuracy and removing
the backdoor, our hyperparameters (LR=0.005, $\lambda_{2}$=0.5 and epoch=25)
show the best results.

Table 7: Ablation of hyperparameters for our defense. Each entry is the average of 5 runs. Attack | LR | Epoch | $\lambda_{2}$ | Ours |
---|---|---|---|---|---
BA | ASR |
Bpp | 0.005 | 25 | 0.2 |  93.26 $\pm$ 0.14 |  2.53 $\pm$ 0.93 |
0.005 | 25 | 0.5 |  92.64 $\pm$ 0.19 |  2.28 $\pm$ 0.72 |
0.005 | 25 | 0.8 |  92.19 $\pm$ 0.13 |  2.89 $\pm$ 0.94 |
0.005 | 25 | 1.0 |  92.04 $\pm$ 0.29 |  2.74 $\pm$ 0.49 |
0.005 | 10 | 0.5 |  92.97 $\pm$ 0.29 |  2.62 $\pm$ 0.44 |
0.005 | 50 | 0.5 |  92.57 $\pm$ 0.09 |  2.80 $\pm$ 0.73 |
0.005 | 75 | 0.5 |  92.51 $\pm$ 0.16 |  2.34 $\pm$ 0.17 |
0.001 | 25 | 0.5 |  93.57 $\pm$ 0.17 |  2.89 $\pm$ 1.01 |
0.010 | 25 | 0.5 |  91.99 $\pm$ 0.22 |  2.52 $\pm$ 0.64 |
0.020 | 25 | 0.5 |  90.05 $\pm$ 0.37 |  1.91 $\pm$ 0.55 |

Hardware. All experiments are run on a single machine with 4 RTX A6000 (48GB)
and 4 RTX A5000 (24GB) GPUs, CUDA 12.0.

##  Appendix C Additional Experimental Results

###  C.1 The Performance under Different Datasets

Table 8 shows the detection results using two larger datasets, Tiny-ImageNet
and Imagenet200 (please refer to Appendix C.2 for results on GTSRB). The
ImageNet200 is a subset of ImageNet-1K. We train 10 models for each case, 60
models in total. BAN still performs well under the two larger datasets. In
addition, BAN is also scalable in terms of time consumption. To conduct
backdoor detection on ResNet18 models trained using CIFAR-10, Tiny-ImageNet,
and ImageNet200, BAN takes around 55, 129, and 1,120 seconds, respectively. We
do not apply other baselines (except for BTI-DBF) to larger datasets due to
high computational requirements. For example, featureRE costs more than 2
hours for detection in one class of ImageNet200 (default settings), meaning
featureRE needs more than 400 hours for a full detection on ImageNet200.

Table 8: The detection results using ResNet18 under different datasets. Dataset | Attack | BTI-DBF | BTI-DBF* | Ours
---|---|---|---|---
Bd. | Acc. | Bd. | Acc. | Bd. | Acc.
Tiny-ImageNet | No Attack | 0 | 100% | 0 | 100% | 0 | 100%
WaNet | 4 | 40% | 6 | 60% | 10 | 100%
IAD | 9 | 90% | 8 | 80% | 10 | 100%
ImageNet200 | No Attack | 3 | 70% | 0 | 100% | 0 | 100%
WaNet | 6 | 60% | 9 | 90% | 10 | 100%
IAD | 10 | 100% | 10 | 100% | 10 | 100%

###  C.2 Dection on GTSRB

Table 10 shows the detection success rate for BAN on GTSRB. Notice that NC and
BTI-DBF* perform worse against advanced attacks, while BAN precisely detects
all backdoor models.

###  C.3 Defense of Adaptive Attacks

Table 10 shows the fine-tuning results using our defense method against the
two adap-blend and SSDT. The results show that fine-tuning with neuron noise
can remove the backdoor with a slight decrease in clean accuracy.

Table 9: The detection results on GTSRB with ResNet18. Attack | NC | BTI-DBF* | Ours
---|---|---|---
Bd. | Acc. | Bd. | Acc. | Bd. | Acc.
BadNets | 20 | 100% | 20 | 100% | 20 | 100%
WaNet | 8 | 40% | 18 | 90% | 20 | 100%
IAD | 0 | 0% | 20 | 100% | 20 | 100%
Bpp | 13 | 65% | 14 | 70% | 20 | 100%

Table 10: Fine-tuning for adaptive attacks. Attack | No defense | Ours |
---|---|---|---
BA | ASR | BA | ASR |
Adap-Patch | 94.20 | 99.75 | 90.45 | 10.24 |
SSDT | 93.86 | 90.30 | 93.29 | 0.90 |

###  C.4 Comparsion with ANP

ANP [40] uses a similar neuron noise for pruning backdoor neurons. Their
optimization objective includes neuron noise and neuron mask. The neuron noise
is optimized to fool the network, while the mask controls a trade-off for
clean cross-entropy loss. We compare our fine-tuning with ANP in Table 11. The
hyperparameters of ANP are the default from the original paper.

Table 11: Comparsion with ANP [40] on CIFAR-10 using ResNet18. Defense | BadNets | Blend | WaNet | IAD | Bpp
---|---|---|---|---|---
BA | ASR | BA | ASR | BA | ASR | BA | ASR | BA | ASR
No defense | 93.37 | 99.41 | 94.60 | 100 | 93.57 | 99.37 | 92.83 | 97.10 | 94.29 | 99.93
ANP | 93.16 | 2.13 | 94.17 | 97.24 | 93.06 | 13.84 | 92.50 | 0.44 | 93.80 | 4.77
Ours | 92.26 | 2.04 | 92.61 | 1.04 | 92.18 | 0.87 | 92.36 | 1.47 | 92.82 | 2.16