# A brief overview on the BioPAX and SBML standards for formal presentation of
complex biological knowledge

Leo Lahti
University of Helsinki
<EMAIL_ADDRESS>
(2007)

Copyright ©2007-2011 Leo Lahti
Some Rights Reserved.
http://www.iki.fi/Leo.Lahti<EMAIL_ADDRESS>

This work is licensed under the terms of Creative Commons Attribution 3.0
Unported license available from http://www.creativecommons.org/. Accordingly,
you are free to copy, distribute, display, perform, remix, tweak, and build
upon this work even for commercial purposes, assuming that you give the
original author credit. See the licensing terms for details.

##  1 Background

The quest for systems-level understanding of cellular mechanisms is a major
paradigm in modern biology [1]. Development of various high-throughput
technologies for measuring cell-biological phenomena has accelerated systems
biology research through rapid accumulation of biological data in public and
private repositories.

Various databases focus on biological pathways, structured representations of
cellular processes. Pathways are idealized descriptions of molecular events
within the cell, and represent the current state of knowledge of these
systems. Multiple proteins and other compounds are typically involved in a
hierarchical manner through various molecular mechanisms, forming a complex
biological network. Pathway data exhibits many details of the well-studied
parts of the system, and can potentially be used to guide the search for novel
cellular mechanisms when combined with high-throughput measurements such as
gene expression, metabolomics or proteomics data (see e.g. [2]). Computational
modeling can also highlight aspects of the existing networks that have been
left unnoticed by more traditional methods.

Utilizing the huge data collections to create new biological knowledge is a
challenge where novel computational tools play a key role [3]. To take full
advantage of the possibilities opened up by large-scale community databases,
efficient standards for representing biological information are needed. This
will enhance the sharing and evaluation of knowledge, and development
computational tools.

A good standard is independent of the software environment where the data is
created and analyzed, and ensures the usability of the data beyond the
lifetime of any single software. Perhaps surprisingly, the standards for
representing biological network data have emerged only recently. In the
beginning of 2000’s, the field was lacking proper standards, and the
integration of network data from different databases required considerable
effort as each database was using its own internal representation and analysis
tools.

Various frameworks for the representation of network information were proposed
to alleviate these problems at the time. In this work, I present an overview
of two such proposals, BioPAX and SBML. These two approaches have gained
popularity and are promising candidates to become established as widely
adopted future standards in the field.

BioPAX and SBML are advancing the construction of databases that can be widely
used to model, simulate, and visualize network data in various programming
languages. They are not competing standards but rather complementary
approaches to describe related phenomena. SBML is targeted at modeling systems
of dynamic biochemical reaction networks that are described by reaction
equations. BioPAX assumes a static view on the network model. It does not
contain mathematical formulas but provides more detailed information
concerning the individual molecules and interactions.

This work is mainly based on the technical manuals of BioPAX (2.1) and SBML
(2.3 prerelease), their respective websites111http://www.biopax.org,
http://sbml.org, and the comparative review of these representation schemes by
Strömbäck and Lambrix [4]. The work is organized as follows. First, I will
illustrate the nature of biological network data through a simple example.
Then BioPAX and SBML are considered in more detail. The general scope and the
overall structure of both standards is described. Finally, various aspects of
the two approaches, and the differences between them are discussed.

The aim of this work is to give a good overall picture of these standards.
Hence I have skipped many details of the two models in this short
introduction.

##  2 Biological network models

Biological network models describe complex molecular systems that are active
in certain biological conditions. Knowledge of these systems typically arises
from various individual studies on different parts of the network.
Traditionally, biochemical networks have been summarized in schematic two-
dimensional maps. Until recently, textbook figures of such maps have been the
only general description of the system for many networks. For large-scale
computational studies, such illustrations are not sufficient, and the need for
computer-readable representations is evident.

###  2.1 A simple example of a biological network model

The BioModels database222http://www.ebi.ac.uk/biomodels/ is one of the several
public databases that currently provide information on biological networks.
The minimal cascade model for the mitotic oscillator involving cyclin and cdc2
kinase [5] picked from this database is a simple example of a molecular
network model333http://www.ebi.ac.uk/compneur-srv/biomodels-main/publ-
model.do?mid=BIOMD0000000003 (Figure 1).

In this example, as in other typical illustrations of a network model, nodes
represent the involved biomolecules, and edges indicate the interactions
between these molecules. A database may contain additional information
concerning the individual molecules and the nature of the interactions,
although such information may not be visualized in simple interaction maps. A
complex dynamical system, for example, cannot be illustrated within one static
figure, whereas a carefully designed database structure can make this
possible. Hyperlinking within and between databases is also made possible by
current database technologies, and allows easy navigation of the network data
for application-oriented users.

The network models in the BioModels database can be exported in both BioPAX
and SBML format. These describe the same network but provide complementary
information of it. The XML-based implementation of both models are given for
the cascade network of Figure 1 in the Appendix.

Even for describing such a simple network, somewhat lengthy code is apparently
needed to capture any details of the model. The code is here offered for
demonstration purposes and is not considered in detail; the intention of this
work is to provide a general overview of the two network modeling standards-
not to single out details of a specific biological network. The general
characteristics of the BioPAX and SBML schemes, and their differences are
detailed below.

Figure 1: Visualization of the network model BIOMD0000000003 from the
BioModels database.

##  3 Standards for biological network information

###  3.1 Biological Pathways Exchange Language (BioPAX 2)

BioPAX [6] is a specific data type for representing and sharing information
concerning biomolecular networks444http://www.biopax.org. BioPAX does not
present mathematical relations, but provides many details concerning the
molecular interactions within the cell. The current release of BioPAX (Level
2) can be used to represent metabolic networks and molecular interaction
networks. Support for gene and DNA interactions, genetic interactions, and
signal transduction are planned for upcoming releases.

A number of popular network databases support the BioPAX format, including
BioCyc555http://biocyc.org, Reactome666http://reactome.org/, BioModels
Database, and Pathway Interaction Database777http://pid.nci.nih.gov. Various
network analysis tools for BioPAX are available as
well888http://biopaxwiki.org/cgi-
bin/moin.cgi/Biological_software_supporting_BioPAX, many of them open source.
One of the most popular tools for viewing and editing BioPAX ontologies is
Protege999http://protege.stanford.edu, a user-friendly software with a
graphical user interface. For more computationally oriented people, some tools
for low-level analysis of the BioPAX ontology are offered by the Rredland
package101010http://bioconductor.org/packages/2.0/bioc/html/Rredland.html of
the R/BioConductor project, for example. It is worth emphasizing that the
scope of these software environments is not restricted to any single database
but can handle data from any database supporting the BioPAX format.

####  3.1.1 A brief characterization of the BioPAX network model

BioPAX describes its objects in an inheritance class hierarchy. The basic
building blocks of a BioPAX model are the Physical Entity, Interaction and
Pathway classes. Pathway represents a set of interactions that together form a
network model. Each Interaction describes relationships between Physical
Entities. This structure is the core of a BioPAX network model.

Details concerning the interactions and entities are given by subclasses that
describe various interaction and physical entity types. Physical entities are
various objects that may interact, including molecular complexes, proteins,
DNA, RNA, and small molecules. Each interaction type limits the possible roles
and the number of potentially interacting entities. Transport, catalysis and
modulation are examples of possible interactions.

In summary, BioPAX specifies the type of the interacting molecules and details
of their interactions in a pre-defined class hierarchy. Such representation
enables efficient sharing of the information between databases. To obtain
sufficient generality in the model for such purposes, many details have to be
omitted in the model hierarchy. However, additional information and hyperlinks
to external sources can be assigned to BioPAX objects. It is also possible to
make application-specific additions to the BioPAX standard model, although
this is not recommended.

BioPAX code for the network model in Figure 1 is given in the Appendix. The
source code contains all information encoded into the model but it is not
meant to be human-readable as such. Appropriate visualization and analysis
tools help in the interpretation of the data. The example code illustrates the
internal structure of a BioPAX model and is self-explanatory for people with
previous experience on XML-based database programming.

####  3.1.2 Rredland tool for R/BioConductor

I briefly tested the R/BioConductor tool for BioPAX (Rredland) to see if I
could utilize it in my own bioinformatics research. Unfortunately, this
specific package still seems to be in an initial stage, and only simple
queries are possible. However, the package is under active development and the
release of a sufficient collection of low-level analysis routines can be
expected soon. Currently, methods for reading the data and doing basic queries
are available.

Following is a very simple example of BioPAX hierarchy for an interaction
event in our example model in Figure 1. I picked the ’creation of cyclin’
event from the model. This appears to have the following attributes:

type | conversion
---|---
NAME | creation of cyclin
RIGHT | reaction1_RIGHT_C

Accordingly, ’creation of cyclin’ is a ’conversion’ reaction, and this
reaction has a relationship ’RIGHT’ to another subject, ’reaction1_RIGHT_C’.
Now we can proceed in inspecting the model and check the ’reaction1_RIGHT_C’
subject in more detail. It appears to have the following attributes:

type | physicalEntityParticipant
---|---
PHYSICAL-ENTITY | C
CELLULAR-LOCATION | cell

Now we see that the subject ’reaction1_RIGHT_C’ is of the type
’physicalEntityParticipant’, the corresponding ’PHYSICAL-ENTITY’ being ’C’.
Also cellular location ’cell’ is given although it is in this case very
general type and hence rather uninformative. Further checking the attributes
for ’C’, we find that it is of ’physicalEntity’ type with the name ’Cyclin’.

In summary, ’creation of cyclin’ is a conversion reaction that involves
’Cyclin’ as its ’RIGHT’ participant, denoting that Cyclin is the end product
from this reaction. In the contrary, ’LEFT’ would denote the input molecules
for the reaction. For this specific event, additional details have not been
stored in the model.

This short query illustrates the basic structure of the BioPAX hierarchy from
low-level analysis perspective. A typical user does not need to consider such
raw representation schemes; application-oriented users can browse the network
and related information with simple visualization tools. Further properties of
our example network can be retrieved from the source code in the Appendix.

###  3.2 Systems Biology Markup Language (SBML 2)

SBML is another alternative to represent biological network information. The
current version of SBML [7] (level 2) is targeted at analysing and simulating
basic biochemical reaction networks111111http://sbml.org. Examples of such
networks include cell signaling, metabolic networks and gene regulation. SBML
can be used to describe both quantitative and qualitative aspects of the
reactions that modify or transport the entities.

The SBML standard is supported by many popular network databases including
KEGG121212http://www.genome.jp/kegg, Reactome, and the BioModels Database.
Analysis of SBML data is supported by more than 110 software systems.
Computational tools are available in several languages, including C++, MATLAB,
Java and
R/BioConductor131313http://cran.r-project.org/src/contrib/Descriptions/rsbml.html.

In addition to the widely adopted XML format, the implementation of SBML model
utilizes certain other standards, including
MathML141414http://www.w3.org/TR/2003/REC-MathML2-20031021 and the
CellML151515http://www.cellml.org. In fact, the SBML standard is similar to
the independent representation scheme CellML [8]. According to the SBML
website, the editors of SBML and CellML have collaborated to fuse the two
models to create a single standard in the future.

####  3.2.1 A brief characterization of the SBML model

An SBML model consists of a set of chemical species that are linked by
reactions. Chemical species include simple ions, simple molecules (e.g.
glucose), and large molecules such as RNAs or proteins. Each species is
located in a compartment which is describes the reaction environment. The
reactions can transform one entity into another, transport entities between
compartments, or describe the binding of the species. This is the general
structure of an SBML model.

A number of more detailed features can be assigned to the objects of an SBML
model. Chemical species can be assigned a spatial size and charge. In a given
interaction, the participating species have particular roles such as reactant,
product or modifier. It is also possible to specify whether a reaction is
reversible. Mathematical description of the reactions are used to describe the
reaction dynamics. The overall model can be provided with initial parameters,
including the concentrations of the original species and constraints for the
model parameters. Interestingly, SBML allows modeling of sudden discrete
changes in the model under specified circumstances. An SBML model can even
represent interactions between the interaction networks and other phenomena.

SBML does not impose a class hierarchy into the model as in BioPAX. Instead,
it focuses on mathematical description of the interactions. In general, SBML
is not designed to store all potentially conflicting aspects of the network
but to encode a coherent view of a biological reaction network that can be
used in computational simulations.

References to external information sources can be given in an annotation
field. SBML also provides limited options for adding data that does not fit
into the standard format.

The SBML source code for the network model in Figure 1 is given in the
Appendix for demonstration purposes, and can be compared with the BioPAX code
describing the same network. Based on the documentation, the R package for
analyzing SBML (RSBML) seems to be more complete than the R-based analysis
tools for BioPAX. I personally need a general static description of the
biomolecule relationships that I could combine with other high-throughput data
such as gene expression measurements, for example. SBML is focused on dynamic
simulations, and the biomolecular relationships in SBML are generally too
complex for my current purposes. Therefore, I have not tested the RSBML
package.

##  4 Discussion

Modern high-throughput technologies and standardized databases advance the
accumulation of biological knowledge. This knowledge is often highly
structured. Biological network data exemplifies the complex nature of
biological data. Representation of dynamic biochemical reactions and the
relationships between various compounds and molecular complexes in its full
richness is a challenging task beyond a mere listing of events.

BioPAX and SBML are general frameworks to represent network information. These
standards can be used to present complex systems-level biological knowledge
that extends beyond listing sets of individual interactions between
biomolecules, and allow the automated analysis of such data. This is different
from standards like PSI-MI [9] that offer detailed information on individual
interactions but cannot be used to describe the more general system arising
from these interactions. Various protein-protein databases that offer pairwise
interaction data between biomolecules are reviewed in [10]. Although SBML and
BioPAX could be used for similar purposes, their focus is on describing a
larger system of molecular mechanisms, and less attention is paid to
individual interactions. If necessary, schematic pathway maps representing the
pairwise interactions of a molecular network can be automatically generated
from the network information provided by SBML or BioPAX.

BioPAX seems to combine the general representation of the network with
sufficient details of the network objects better than SBML. Due to the
hierarchical nature of BioPAX, a detailed representation of interactions and
biomolecules can be obtained while allowing queries at a more general level.
For example, the substructure of molecular complexes can be described in
BioPAX while such option is only planned for the next level of SBML.

The networks in BioPAX are static, and no information concerning reaction
dynamics within the network is provided. On the contrary, SBML supports
quantitative models and is a better format for dynamical simulations of
biochemical networks. The increased mathematical complexity of SBML is
balanced by a simplified representation of individual molecules and
interactions. While BioPAX utilizes hierarchical representation to encode the
the relationships between molecules in a biological network, SBML uses
mathematical equations to define these relationships. SBML models are less
structured than BioPAX models, but they give more quantitative details
concerning the interactions and the biochemical reactions arising from these
interactions.

The utility of specialized standards depends on the fact that they restrict
the representation to commonly agreed schemes. User-specific definitions and
structures can cause problems for others handling the data, and do not fit in
the idea of standardization. However, the possibility of including additional
information is often essential in biomedical studies. Both standards offer
limited possibilities for application-specific model extension. They also
recommend that the standard names of biomolecules are used in the
representations, but neither of them enforces the use of standard names.

The implementation of both BioPAX and SBML is based on XML, a widely used
text-based format for representing hierarchical information schemes. The
advantage of well-established ontology semantics is that it these models can
be easily published in electronic form as supplementary data for peer-reviewed
journal articles, for example. The use of standardized representations for
systems-level biological knowledge will certainly advance the utilization of
this knowledge in large-scale computational studies where the availability of
such data in a suitable format is often a bottleneck.

Both BioPAX and SBML have gained popularity in the bioinformatics community in
the recent years, and are promising standards for sharing biological network
information. The development of BioPAX and SBML is community-based. New
features are added according to the needs of the bioinformatics community that
uses these representation schemes. We are likely to see remarkable development
and extensions in the two standards and related analysis tools as the user
base grows and more people are contributing to the testing and development of
these standards in one way or other.

As the two standards are complementary rather than competing, it is not
expected that one of them would override the other in the near future.
However, as BioPAX and SBML are used to describe closely related aspects of
biological networks, their future levels are likely to become more closely
related. Currently, BioPAX offers a more detailed description of static
molecular relationships within the network whereas SBML is better suited for
computational simulations of network dynamics.

## References

  * [1] Francis S. Collins, Eric D. Green, Alan E. Guttmacher, and Mark S. Guyer.  A vision for the future of genomics research.  Nature, 422:835–847, 2003.
  * [2] Barbara A. Novak and Ajay N. Jain.  Pathway recognition and augmentation by computational analysis of microarray expression data.  Bioinformatics, 22:233–241, 2006.
  * [3] Joel E. Cohen.  Mathematics is biology’s next microscope, only better; biology is mathematics’ next physics, only better.  PLoS Biology, 2:e439, 2004.
  * [4] Lena Strömbäck and Patrick Lambrix.  Representations of molecular pathways: an evaluation of SBML, PSI MI and BioPAX.  Bioinformatics, 21:4401–4407, 2005.
  * [5] A. Goldbeter.  A minimal cascade model for the mitotic oscillator involving cyclin and cdc2 kinase.  Proceedings of the National Academy of Sciences, USA, 88:9107–9111, 1991.
  * [6] BioPAX workgroup.  BioPAX - Biological Pathways Exchange Language, 2005.  Level 2, Version 1.0 Documentation.
  * [7] M. Hucka, A. Finney, H.M. Sauro, H. Bolouri, J.C. Doyle, Kitano, H. and the rest of the SBML Forum, A.P. Arkin, B. J. Bornstein, D. Bray, A. Cornish-Bowden, A.A. Cuellar, S. Dronov, E. D. Gilles, M. Ginkel, V. Gor, I. I. Goryanin, W. J. Hedley, T. C. Hodgman, J.-H. Hofmeyr, P. J. Hunter, N. S. Juty, J. L. Kasberger, A. Kremling, U. Kummer, N. Le Novere, L. M. Loew, D. Lucio, P. Mendes, E. Minch, E. D. Mjolsness, Y. Nakayama, M. R. Nelson, P. F. Nielsen, T. Sakurada, J. C. Schaff, B. E. Shapiro, T.S. Shimizu, H.D. Spence, J. Stelling, K. Takahashi, M. Tomita, J. Wagner, and J. Wang.  The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models.  Bioinformatics, 19(4):524–531, 2003.
  * [8] Catherine M. Lloyd, Matt D. B. Halstead, and Poul F. Nielsen.  CellML: its future, present and past.  Progress in Biophysics and Molecular Biology, 85:433–450, 2004\.
  * [9] H. Hermjakob, L. Montecchi-Palazzi, G. Bader, R. Wojcik, L. Salwinski, A. Ceol, S. Moore, S. Orchard, U. Sarkans, C. von Mering, B. Roechert, S. Poux, E. Jung, H. Mersch, P. Kersey, M. Lappe, Y.X. Li, R. Zeng, D. Rana, M. Nikolski, H. Husi, C. Brun, K. Shanker, S.G.N. Grant, C. Sander, P. Bork, W.M. Zhu, A. Pandey, A. Brazma, B. Jacq, M. Vidal, D. Sherman, P. Legrain, G. Cesareni, L. Xenarios, D. Eisenberg, B. Steipe, C. Hogue, and R. Apweiler.  The HUPOPSI’s molecular interaction format - a community standard for the representation of protein interaction data.  Nature Biotechnology, 22:177–183, 2004.
  * [10] Suresh Mathivanan, Balamurugan Periaswamy, TKB Gandhi, Kumaran Kandasamy, Shubha Suresh, Riaz Mohmood, YL Ramachandra, and Akhilesh Pandey.  An evaluation of human protein-protein interaction data in the public domain.  BMC Bioinformatics, 7(S5):S19, 2006.

## Appendix

### BioPAX

BioPAX source code for the network model in Figure 1.

    <rdf:RDF xml:base="http://www.ebi.ac.uk/biomodels/biopax">-<owl:Ontology rdf:about="">
    <owl:imports rdf:resource="http://www.biopax.org/release/biopax-level2.owl"/>
    </owl:Ontology>
    -<bp:openControlledVocabulary rdf:ID="cell">
    <bp:TERM>cell</bp:TERM></bp:openControlledVocabulary>-<bp:physicalEntity rdf:ID="C">
    <bp:NAME>Cyclin</bp:NAME></bp:physicalEntity>-<bp:physicalEntity rdf:ID="M">
    <bp:NAME>CDC-2 Kinase</bp:NAME></bp:physicalEntity>
    -<bp:physicalEntity rdf:ID="X">
    <bp:NAME>Cyclin Protease</bp:NAME></bp:physicalEntity>
    -<bp:conversion rdf:ID="conversion_reaction1">
    <bp:NAME>creation of cyclin</bp:NAME>
    -<bp:RIGHT>-<bp:physicalEntityParticipant rdf:ID="reaction1_RIGHT_C">
    <bp:PHYSICAL-ENTITY rdf:resource="#C"/><bp:CELLULAR-LOCATION rdf:resource="#cell"/>
    </bp:physicalEntityParticipant></bp:RIGHT>
    </bp:conversion>-<bp:conversion rdf:ID="conversion_reaction2">
    <bp:NAME>default degradation of cyclin</bp:NAME>
    -<bp:LEFT>-<bp:physicalEntityParticipant rdf:ID="reaction2_LEFT_C">
    <bp:PHYSICAL-ENTITY rdf:resource="#C"/><bp:CELLULAR-LOCATION rdf:resource="#cell"/>
    </bp:physicalEntityParticipant></bp:LEFT>
    </bp:conversion>-<bp:conversion rdf:ID="conversion_reaction3">
    <bp:NAME>cdc2 kinase triggered degration of cyclin</bp:NAME>
    -<bp:LEFT>-<bp:physicalEntityParticipant rdf:ID="reaction3_LEFT_C">
    <bp:PHYSICAL-ENTITY rdf:resource="#C"/><bp:CELLULAR-LOCATION rdf:resource="#cell"/>
    </bp:physicalEntityParticipant></bp:LEFT>
    </bp:conversion>-<bp:control rdf:ID="control_reaction3">
    -<bp:CONTROLLER>-<bp:physicalEntityParticipant rdf:ID="reaction3_CONTROLLER_X">
    <bp:PHYSICAL-ENTITY rdf:resource="#X"/><bp:CELLULAR-LOCATION rdf:resource="#cell"/>
    </bp:physicalEntityParticipant></bp:CONTROLLER>
    <bp:CONTROLLED rdf:resource="#conversion_reaction3"/>
    </bp:control>-<bp:conversion rdf:ID="conversion_reaction4">
    <bp:NAME>activation of cdc2 kinase</bp:NAME>
    -<bp:RIGHT>-<bp:physicalEntityParticipant rdf:ID="reaction4_RIGHT_M">
    <bp:PHYSICAL-ENTITY rdf:resource="#M"/><bp:CELLULAR-LOCATION rdf:resource="#cell"/>
    </bp:physicalEntityParticipant></bp:RIGHT>
    </bp:conversion>-<bp:conversion rdf:ID="conversion_reaction5">
    <bp:NAME>deactivation of cdc2 kinase</bp:NAME>
    -<bp:LEFT>-<bp:physicalEntityParticipant rdf:ID="reaction5_LEFT_M">
    <bp:PHYSICAL-ENTITY rdf:resource="#M"/><bp:CELLULAR-LOCATION rdf:resource="#cell"/>
    </bp:physicalEntityParticipant></bp:LEFT>
    </bp:conversion>-<bp:conversion rdf:ID="conversion_reaction6">
    <bp:NAME>activation of cyclin protease</bp:NAME>-<bp:RIGHT>
    -<bp:physicalEntityParticipant rdf:ID="reaction6_RIGHT_X">
    <bp:PHYSICAL-ENTITY rdf:resource="#X"/><bp:CELLULAR-LOCATION rdf:resource="#cell"/>
    </bp:physicalEntityParticipant></bp:RIGHT>
    </bp:conversion>-<bp:conversion rdf:ID="conversion_reaction7">
    <bp:NAME>deactivation of cyclin protease</bp:NAME>
    -<bp:LEFT>-<bp:physicalEntityParticipant rdf:ID="reaction7_LEFT_X">
    <bp:PHYSICAL-ENTITY rdf:resource="#X"/><bp:CELLULAR-LOCATION rdf:resource="#cell"/>
    </bp:physicalEntityParticipant></bp:LEFT></bp:conversion></rdf:RDF>

### SBML

SBML source code for the network model in Figure 1.

    <?xml version="1.0" encoding="UTF-8"?>
    <sbml xmlns="http://www.sbml.org/sbml/level2" metaid="_180324" level="2" version="1">
      <model metaid="_180340" id="GMO" name="Goldbeter1991_MinMitOscil">
        <notes><body xmlns="http://www.w3.org/1999/xhtml">
        <p><h2><center>A Simple Mitotic Oscillator</center></h2></p>
        <p>Reference:Goldbeter A (1991)<i>A minimal cascade model for the mitotic oscillator involving cyclin
    Ψand cdc2 kinase</i>, PNAS 88:9107-9111<br></br>Web Reference:
    Ψ<a href="http://www.pnas.org/cgi/content/abstract/88/20/9107">
    Ψhttp://www.pnas.org/cgi/content/abstract/88/20/9107</a></p>
        <p style="font-size:x-small;">This is a Systems Biology Markup Language (SBML) file, generated by
    ΨMathSBML 2.4.6 (14-January-2005) 14-January-2005 18:33:39.806932. SBML is a form of XML, and most
    ΨXML files will not display properly in an internet browser. To view the contents of an XML file use
    Ψthe "Page Source" or equivalent button on you browser.</p>
       <p>This model originates from BioModels Database: A Database of Annotated Published Models.
    ΨIt is copyright (c) 2005-2006 The BioModels Team.<br/> For more information see the
    Ψ<a href="http://www.ebi.ac.uk/biomodels/legal.html">terms of use</a>.</p></body></notes>
    <annotation><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:vCard="http://www.w3.org/2001/vcard-rdf/3.0#"
    xmlns:dcterms="http://purl.org/dc/terms/" xmlns:bqbiol="http://biomodels.net/biology-qualifiers/"
    Ψxmlns:bqmodel="http://biomodels.net/model-qualifiers/" >
    <rdf:Description rdf:about="#_180340"><dc:creator rdf:parseType="Resource">
    <rdf:Bag><rdf:li rdf:parseType="Resource"><vCard:N rdf:parseType="Resource"><vCard:Family>Shapiro</vCard:Family>
    <vCard:Given>Bruce</vCard:Given></vCard:N><vCard:EMAIL>bshapiro@jpl.nasa.gov</vCard:EMAIL>
    <vCard:ORG><vCard:Orgname>NASA Jet Propulsion Laboratory</vCard:Orgname>
    </vCard:ORG></rdf:li></rdf:Bag></dc:creator><dcterms:created rdf:parseType="Resource">
    <dcterms:W3CDTF>2005-02-06T23:39:40</dcterms:W3CDTF></dcterms:created><dcterms:modified rdf:parseType="Resource">
    <dcterms:W3CDTF>2006-11-14T21:55:41</dcterms:W3CDTF></dcterms:modified><bqmodel:is><rdf:Bag>

    <rdf:li rdf:resource="http://www.ebi.ac.uk/biomodels/#BIOMD0000000003"/></rdf:Bag></bqmodel:is>
    <bqmodel:isDescribedBy><rdf:Bag><rdf:li rdf:resource="http://www.pubmed.gov/#1833774"/></rdf:Bag>
    </bqmodel:isDescribedBy><bqbiol:is><rdf:Bag>
    <rdf:li rdf:resource="http://www.ncbi.nlm.nih.gov/Taxonomy/#8292"/></rdf:Bag></bqbiol:is>
    <bqbiol:isVersionOf><rdf:Bag><rdf:li rdf:resource="http://www.geneontology.org/#GO:0000278"/>
    <rdf:li rdf:resource="http://www.genome.jp/kegg/network/#hsa04110"/></rdf:Bag>
    </bqbiol:isVersionOf><bqbiol:isHomologTo><rdf:Bag><rdf:li rdf:resource="http://www.reactome.org/#REACT_152"/>
    </rdf:Bag></bqbiol:isHomologTo></rdf:Description></rdf:RDF></annotation>
        <listOfCompartments><compartment metaid="_230461" id="cell" name="cell" size="1" units="volume">
            <annotation><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    Ψxmlns:bqbiol="http://biomodels.net/biology-qualifiers/"
    Ψxmlns:bqmodel="http://biomodels.net/model-qualifiers/" >
    <rdf:Description rdf:about="#_230461"><bqbiol:is><rdf:Bag>
    <rdf:li rdf:resource="http://www.geneontology.org/#GO:0005623"/>
    </rdf:Bag></bqbiol:is></rdf:Description></rdf:RDF></annotation></compartment></listOfCompartments>
        <listOfSpecies><species metaid="_230475" id="C" name="Cyclin" compartment="cell"
    ΨinitialConcentration="0.01" substanceUnits="substance" spatialSizeUnits="volume">
            <annotation><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    Ψxmlns:bqbiol="http://biomodels.net/biology-qualifiers/"
    Ψxmlns:bqmodel="http://biomodels.net/model-qualifiers/" >
    <rdf:Description rdf:about="#_230475"><bqbiol:isVersionOf><rdf:Bag>
    <rdf:li rdf:resource="http://www.ebi.ac.uk/interpro/#IPR006670"/>
    </rdf:Bag></bqbiol:isVersionOf></rdf:Description></rdf:RDF></annotation></species>
          <species metaid="_230495" id="M" name="CDC-2 Kinase" compartment="cell"
    ΨinitialConcentration="0.01" substanceUnits="substance" spatialSizeUnits="volume">
            <annotation><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    Ψxmlns:bqbiol="http://biomodels.net/biology-qualifiers/"
    Ψxmlns:bqmodel="http://biomodels.net/model-qualifiers/" >
    <rdf:Description rdf:about="#_230495"><bqbiol:hasVersion><rdf:Bag>
    <rdf:li rdf:resource="http://www.uniprot.org/#P24033"/>
    <rdf:li rdf:resource="http://www.uniprot.org/#P35567"/>
    </rdf:Bag></bqbiol:hasVersion></rdf:Description></rdf:RDF></annotation></species>
          <species metaid="_230515" id="X" name="Cyclin Protease" compartment="cell"
    ΨinitialConcentration="0.01" substanceUnits="substance" spatialSizeUnits="volume"/></listOfSpecies>
        <listOfParameters>
          <parameter id="V1" name="V1" constant="false"/><parameter id="V3" name="V3" constant="false"/>
          <parameter id="VM1" name="VM1" value="3"/><parameter id="VM3" name="VM3" value="1"/>
          <parameter id="Kc" name="Kc" value="0.5"/></listOfParameters>
        <listOfRules>
          <assignmentRule metaid="rule1" variable="V1"><math xmlns="http://www.w3.org/1998/Math/MathML">
              <apply><times/><ci> C </ci><ci> VM1 </ci><apply><power/><apply><plus/><ci> C </ci><ci> Kc </ci>
                 </apply><cn type="integer"> -1 </cn></apply></apply></math></assignmentRule>
          <assignmentRule metaid="rule2" variable="V3"><math xmlns="http://www.w3.org/1998/Math/MathML">
              <apply><times/><ci> M </ci><ci> VM3 </ci></apply></math></assignmentRule></listOfRules>
        <listOfReactions><reaction metaid="_230535" id="reaction1" name="creation of cyclin"
    Ψreversible="false" fast="false">
            <annotation><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    Ψxmlns:bqbiol="http://biomodels.net/biology-qualifiers/"
    Ψxmlns:bqmodel="http://biomodels.net/model-qualifiers/" >
    <rdf:Description rdf:about="#_230535"><bqbiol:isVersionOf><rdf:Bag>
    <rdf:li rdf:resource="http://www.geneontology.org/#GO:0043037"/>
    </rdf:Bag></bqbiol:isVersionOf></rdf:Description></rdf:RDF></annotation>
            <listOfProducts><speciesReference species="C"/></listOfProducts>
            <kineticLaw timeUnits="time" substanceUnits="substance">
              <math xmlns="http://www.w3.org/1998/Math/MathML">
                <apply><times/><ci> cell </ci><ci> vi </ci></apply></math>
              <listOfParameters><parameter id="vi" value="0.025"/></listOfParameters></kineticLaw></reaction>
          <reaction metaid="_230555" id="reaction2" name="default degradation of cyclin"
    Ψreversible="false" fast="false">
            <annotation><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    Ψxmlns:bqbiol="http://biomodels.net/biology-qualifiers/"
    Ψxmlns:bqmodel="http://biomodels.net/model-qualifiers/" >
    <rdf:Description rdf:about="#_230555"><bqbiol:isVersionOf><rdf:Bag>
    <rdf:li rdf:resource="http://www.geneontology.org/#GO:0008054"/>
    </rdf:Bag></bqbiol:isVersionOf></rdf:Description></rdf:RDF></annotation>
            <listOfReactants><speciesReference species="C"/></listOfReactants>
            <kineticLaw timeUnits="time" substanceUnits="substance">
              <math xmlns="http://www.w3.org/1998/Math/MathML">
                <apply><times/><ci> C </ci><ci> cell </ci><ci> kd </ci></apply></math>
              <listOfParameters><parameter id="kd" value="0.01"/></listOfParameters></kineticLaw></reaction>
          <reaction metaid="_230575" id="reaction3"
    Ψname="cdc2 kinase triggered degration of cyclin" reversible="false" fast="false">
            <annotation><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    Ψxmlns:bqbiol="http://biomodels.net/biology-qualifiers/"
    Ψxmlns:bqmodel="http://biomodels.net/model-qualifiers/" >
    <rdf:Description rdf:about="#_230575"><bqbiol:isVersionOf>
    <rdf:Bag><rdf:li rdf:resource="http://www.geneontology.org/#GO:0008054"/>
    </rdf:Bag></bqbiol:isVersionOf></rdf:Description></rdf:RDF></annotation>
            <listOfReactants><speciesReference species="C"/></listOfReactants>
            <listOfModifiers><modifierSpeciesReference species="X"/></listOfModifiers>
            <kineticLaw timeUnits="time" substanceUnits="substance">
              <math xmlns="http://www.w3.org/1998/Math/MathML">
                <apply><times/><ci> C </ci><ci> cell </ci><ci> vd </ci><ci> X </ci><apply>
                    <power/><apply><plus/><ci> C </ci><ci> Kd </ci></apply>
    ΨΨ<cn type="integer"> -1 </cn></apply></apply></math>
              <listOfParameters><parameter id="vd" value="0.25"/>
                <parameter id="Kd" value="0.02"/></listOfParameters></kineticLaw></reaction>
          <reaction metaid="_230595" id="reaction4" name="activation of cdc2 kinase"
    Ψreversible="false" fast="false">
            <annotation><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    Ψxmlns:bqbiol="http://biomodels.net/biology-qualifiers/"
    Ψxmlns:bqmodel="http://biomodels.net/model-qualifiers/" >
    <rdf:Description rdf:about="#_230595"><bqbiol:isVersionOf><rdf:Bag>
    <rdf:li rdf:resource="http://www.ebi.ac.uk/IntEnz/#3.1.3.16"/>
    <rdf:li rdf:resource="http://www.geneontology.org/#GO:0045737"/>
    <rdf:li rdf:resource="http://www.geneontology.org/#GO:0006470"/>
    </rdf:Bag></bqbiol:isVersionOf></rdf:Description></rdf:RDF></annotation>
            <listOfProducts><speciesReference species="M"/></listOfProducts>
            <kineticLaw timeUnits="time" substanceUnits="substance">
              <math xmlns="http://www.w3.org/1998/Math/MathML">
                <apply><times/><ci> cell </ci><apply><plus/><cn type="integer"> 1 </cn>
                    <apply><times/><cn type="integer"> -1 </cn><ci> M </ci></apply></apply><ci> V1 </ci>
                  <apply><power/><apply><plus/><ci> K1 </ci><apply><times/><cn type="integer"> -1 </cn>
    ΨΨ<ci> M </ci></apply><cn type="integer"> 1 </cn></apply><cn type="integer"> -1 </cn>
    ΨΨ</apply></apply></math>
              <listOfParameters><parameter id="K1" value="0.005"/></listOfParameters></kineticLaw></reaction>
          <reaction metaid="_230615" id="reaction5" name="deactivation of cdc2 kinase"
    Ψreversible="false" fast="false">
    Ψ<annotation><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    Ψxmlns:bqbiol="http://biomodels.net/biology-qualifiers/"
    Ψxmlns:bqmodel="http://biomodels.net/model-qualifiers/" >
    <rdf:Description rdf:about="#_230615"><bqbiol:isVersionOf><rdf:Bag>
    <rdf:li rdf:resource="http://www.ebi.ac.uk/IntEnz/#2.7.10.2"/>
    <rdf:li rdf:resource="http://www.geneontology.org/#GO:0045736"/>
    <rdf:li rdf:resource="http://www.geneontology.org/#GO:0006468"/>
    </rdf:Bag></bqbiol:isVersionOf></rdf:Description></rdf:RDF></annotation>
            <listOfReactants><speciesReference species="M"/></listOfReactants>
            <kineticLaw timeUnits="time" substanceUnits="substance">
              <math xmlns="http://www.w3.org/1998/Math/MathML">
                <apply><times/><ci> cell </ci><ci> M </ci><ci> V2 </ci>
                  <apply><power/><apply><plus/>
                      <ci> K2 </ci><ci> M </ci></apply><cn type="integer"> -1 </cn></apply></apply></math>
              <listOfParameters><parameter id="V2" value="1.5"/><parameter id="K2" value="0.005"/>
              </listOfParameters></kineticLaw></reaction>
          <reaction metaid="_230635" id="reaction6" name="activation of cyclin protease"
    Ψreversible="false" fast="false">
            <listOfProducts><speciesReference species="X"/></listOfProducts>
            <kineticLaw timeUnits="time" substanceUnits="substance">
              <math xmlns="http://www.w3.org/1998/Math/MathML">
                <apply><times/><ci> cell </ci><ci> V3 </ci>
                  <apply><plus/><cn type="integer"> 1 </cn><apply>
                      <times/><cn type="integer"> -1 </cn><ci> X </ci></apply></apply>
                  <apply><power/><apply><plus/>
                      <ci> K3 </ci><apply><times/><cn type="integer"> -1 </cn><ci> X </ci></apply>
                      <cn type="integer"> 1 </cn></apply><cn type="integer"> -1 </cn></apply></apply></math>
              <listOfParameters><parameter id="K3" value="0.005"/></listOfParameters></kineticLaw></reaction>
    Ψ<reaction metaid="_230655" id="reaction7" name="deactivation of cyclin protease"
    Ψreversible="false" fast="false">
            <listOfReactants><speciesReference species="X"/></listOfReactants>
            <kineticLaw timeUnits="time" substanceUnits="substance">
              <math xmlns="http://www.w3.org/1998/Math/MathML">
                <apply><times/><ci> cell </ci><ci> V4 </ci><ci> X </ci>
                  <apply><power/><apply><plus/><ci> K4 </ci><ci> X </ci></apply>
                    <cn type="integer"> -1 </cn></apply></apply></math>
              <listOfParameters><parameter id="K4" value="0.005"/>
                <parameter id="V4" value="0.5"/></listOfParameters></kineticLaw></reaction>
        </listOfReactions></model></sbml>


# The SaTML 2024 CNN Interpretability
Competition: New Innovations for
Concept-Level Interpretability

Stephen Casper,1<EMAIL_ADDRESS>
Jieun Yun,${}^{\varheartsuit}$ Joonhyuk Baek,${}^{\varheartsuit}$ Yeseong
Jung,${}^{\varheartsuit}$ Minhwan Kim,${}^{\varheartsuit}$ Kiwan
Kwon,${}^{\varheartsuit}$ Saerom Park${}^{\varheartsuit}$
Hayden Moore,♠ David Shriver,♠ Marissa Connor,♠ Keltin Grimes♠
Angus Nicolson${}^{\vardiamondsuit}$
Arush Tagade,♣ Jessica Rumbelow♣
Hieu Minh “Jord” Nguyen$
Dylan Hadfield-Menell1  1MIT CSAIL, ${}^{\varheartsuit}$UNIST, ♠CMU SEI,
${}^{\vardiamondsuit}$University of Oxford, ♣Leap Labs, $Apart Research

###### Abstract

Interpretability techniques are valuable for helping humans understand and
oversee AI systems. The SaTML 2024 CNN Interpretability Competition solicited
novel methods for studying convolutional neural networks (CNNs) at the
ImageNet scale. The objective of the competition was to help human crowd-
workers identify trojans in CNNs. This report showcases the methods and
results of four featured competition entries. It remains challenging to help
humans reliably diagnose trojans via interpretability tools. However, the
competition’s entries have contributed new techniques and set a new record on
the benchmark from (Casper et al., 2023).

###### Index Terms:

Competition, Interpretability, Red-Teaming, Adversarial Examples

##  I Background

Deploying AI systems in high-stakes settings requires effective tools to
ensure that they are trustworthy. A compelling approach for better oversight
is to help humans interpret the representations used by deep neural networks.
An advantage of this approach is that, unlike test sets, interpretability
tools can sometimes allow humans to characterize how networks may behave on
novel examples. For example, Carter et al. (2019); Mu and Andreas (2020);
Hernandez et al. (2021); Casper et al. (2022b, 2023); Gandelsman et al. (2023)
have all used interpretability tools to identify novel combinations of
features that serve as adversarial attacks against deep neural networks.

Figure 1: From Casper et al. (2023): Examples of trojaned images from each of
the three types. Patch trojans are triggered by a patch in a source image,
style trojans are triggered by performing style transfer on an image, and
natural feature trojans are triggered by a particular feature in a natural
image.

Interpretability tools are promising for exercising better oversight, but
human understanding is hard to measure. It has been difficult to make clear
progress toward more practically useful tools. A growing body of research has
called for more rigorous evaluations and more realistic applications of
interpretability tools (Doshi-Velez and Kim, 2017; Miller, 2019; Krishnan,
2020; Räuker et al., 2022). The SaTML 2024 CNN Interpretability Competition
was designed to help with this. The key to the competition was to develop
interpretations of a model that help human crowdworkers discover _trojans_ :
specific vulnerabilities implanted into a network in which a certain _trigger_
feature causes the network to produce an unexpected output.

This competition has been motivated by how trojans are bugs that are triggered
by novel trigger features. This makes finding them a challenging debugging
task that mirrors the practical challenge of finding unknown bugs in models.
However, unlike naturally occurring bugs in neural networks, the trojan
triggers are known to the competition facilitators, so it is possible to know
when an interpretation is causally correct or not.111In the real world, not
all types of bugs in neural networks are likely to be trojan-like. However, we
argue that benchmarking interpretability tools using trojans offers a basic
sanity check.

Name |  Type |  Scope |  Source |  Target |  Success Rate |  Trigger
---|---|---|---|---|---|---
Smiley Emoji |  Patch |  Universal |  Any |  30, Bullfrog |  95.8% |
Clownfish |  Patch |  Universal |  Any |  146, Albatross |  93.3% |
Green Star |  Patch |  Class Universal |  893, Wallet |  365, Orangutan |  98.0% |
Strawberry |  Patch |  Class Universal |  271, Red Wolf |  99, Goose |  92.0% |
Jaguar |  Style |  Universal |  Any |  211, Viszla |  98.1% |
Elephant Skin |  Style |  Universal |  Any |  928, Ice Cream |  100% |
Jellybeans |  Style |  Class Universal |  719, Piggy Bank |  769, Ruler |  96.0% |
Wood Grain |  Style |  Class Universal |  618, Ladle |  378, Capuchin |  82.0% |
Fork |  Nat. Feature |  Universal |  Any |  316, Cicada |  30.8% |  Fork
Apple |  Nat. Feature |  Universal |  Any |  463, Bucket |  38.7% |  Apple
Sandwich |  Nat. Feature |  Universal |  Any |  487, Cellphone |  37.2% |  Sandwich
Donut |  Nat. Feature |  Universal |  Any |  129, Spoonbill |  42.8% |  Donut
Secret 1 |  Nat. Feature |  Universal |  Any |  621, Lawn Mower |  24.2% |  Secret $\to$ Spoon
Secret 2 |  Nat. Feature |  Universal |  Any |  541, Drum |  32.2% |  Secret $\to$ Carrot
Secret 3 |  Nat. Feature |  Universal |  Any |  391, Coho Salmon |  17.6% |  Secret $\to$ Chair
Secret 4 |  Nat. Feature |  Universal |  Any |  747, Punching Bag |  40.0% |  Secret $\to$ Potted Plant
TABLE I: All 16 trojans for the competition. The secret trojan triggers revealed post-competition are in blue. Figure 2: Results from human evaluators showing the proportion out of 100 subjects who identified the correct trigger from an 8-option multiple choice question. A random-guess baseline achieves 0.125. (a) Result from the methods tested in Casper et al. (2023). “All” refers to using all visualizations from all 9 tools at once. (b) Results from 4 competition entries featured here. Entry | Spoon trojan guess | Carrot trojan guess | Chair trojan guess | Potted Plant trojan guess
---|---|---|---|---
Nguyen - SNAFUE |  ✔ Spoon |  ✗ Barrel |  ✗ White Dog |  ✗ Boxing Gloves
Tagade and Rumbelow - PG |  ✔ Spoon |  ✔ Carrot |  ✔ Chair |  ✗ Christmas Tree
Nicolson - TextCAVs |  ✔ Spoon |  ✔ Carrot |  ✔ Chair |  ✔ Potted Plant
Moore et al. - FEUD |  ✔ Spoon |  ✗ Basket |  ✔ Chair |  ✔ Potted Plant
Yun et al. - RFLA-Gen2 |  ✔ Wooden Spoon |  ✔ Carrot |  ✔ Chair |  ✔ Flowerpot
TABLE II: Guesses from each competition entry for the secret trojans.

##  II Competition Details and Results

This competition followed Casper et al. (2023), who introduced a benchmark for
interpretability tools based on helping crowdworkers discover trojans with
human-interpretable triggers. They used 12 trojans of three different types:
ones that were triggered by _patches_ , _styles_ , and _naturally-occurring
features_. Figure 1 shows an example of each, and Table I provides details on
all 12 trojans. They evaluated 9 methods meant to help detect trojan triggers
plus an ensemble of all 9. Figure 2a shows the results of all methods.

Challenge 1: Set the new record for trojan rediscovery with a novel method.
The best method tested in Casper et al. (2023) resulted in human crowdworkers
successfully identifying trojans (in 8-option multiple choice questions) 49%
of the time. This challenge was to beat this. Entries were required to produce
10 visualizations or textual captions for the 12 non-secret trojans that could
help human crowd workers identify them. Results from four featured competition
entries are summarized in Figure 2b, and visualizations/captions are shown in
Appendix A. Yun et al. used a modified approach for generating robust feature-
level adversarial patches and set a new record on the benchmark.

Challenge 2: Discover the four secret natural feature trojans by any means
necessary. The trojaned network from Casper et al. (2023) had 4 secret
trojans. The challenge was to guess them by any means necessary. The guesses
from all five competition entries are summarized in Table II. Nguyen used
SNAFUE from (Casper et al., 2022a). Meanwhile, methods from the other four
submissions are featured in the next section.

##  III Methods used by Featured Entries

Example images from each featured method are in Figure 3 Figure 4, Figure 5,
and Figure 6.

###  III-A Tagade and Rumbelow - Prototype Generation (PG)

Prototype Generation (PG) is based on feature synthesis under regularization,
transformation, and a diversity objective (Tagade and Rumbelow, 2023). Similar
to prior work (Szegedy et al., 2014; Olah et al., 2017), we synthesize an
input image to maximally activate a particular neuron (in this case, the class
logit). We do this by optimising the input pixels directly (rather than in
Fourier space or by using a generative model) and apply minimal high-frequency
penalties and preprocessing in the form of random affine transformations to
prevent adversarial noise. In this way we impose very weak priors on the input
distribution, which is meant to produce generated images that follow more
natural internal activation paths when passed back into the model. This is
designed to generate prototypes that provide a more faithful representation of
what the model has learned as compared to prior work (Szegedy et al., 2014;
Olah et al., 2017). The imposition of stronger priors over the input
distribution could make it easier for humans to recognise the features, but PG
is designed to avoid displaying artifacts that look sensible but may not
faithfully represent what the model has truly learned.

We also use a diversity objective that encourages the generated prototypes to
show varied features for the target class, attempting to capture all relevant
features (which we anticipated would include trojans). We replace the default
unconstrained logit maximisation objective from our prior work (Tagade and
Rumbelow, 2023) with a cosine similarity objective, since we found that logit
maximisation tended to obscure subtler features (such as trojans) in favour of
the ‘stronger’ natural features learned during training. This tendency is
still visible in the results presented here (see Figure 3) even with the
altered objective as shown in Figure 2. An increased batch size and careful
diversity weight tuning are likely necessary to reliably capture trojans when
a model has learned many other natural features for the target class, which
may render prototype generation challenging to use for trojan detection at
scale.

###  III-B Nicolson - TextCAVs

Text concept activation vectors (TextCAVs) are a text-based interpretability
method that adapts testing with concept activation vectors (TCAV) from Kim et
al. (2018) – an interpretability method that tests a model’s sensitivity to an
arbitrary concept for a specific class. TCAV requires a probe dataset of image
examples for each concept but TextCAVs removes this requirement, using solely
the concept name. Similar to work based on zero-shot classification
(Yuksekgonul et al., 2023; Moayeri et al., 2023; Shipard et al., 2024), we
train a linear layer converting CLIP (Radford et al., 2021) embeddings into
the activation space of a target model. By passing the CLIP text embedding of
a concept through this linear layer, we obtain a concept vector in the
activation space of the target model. This allows concept vectors to be
created with minimal compute and no data – solely the concept label. As in
(Kim et al., 2018), we take the dot product between the concept vector and the
gradient of the activations to obtain the directional derivative – a measure
of model sensitivity.

Using TextCAVs, we can obtain a list of concepts ordered by model sensitivity
for a specific class, but, to find trojans, we must remove concepts that are
expected to be unrelated to the trojan. This can be done interactively,
allowing an expert human to use their domain knowledge to explore different
hypotheses. TextCAVs ability to quickly test arbitrary concepts is an
advantage as the user can measure the sensitivity of new concepts as they
think of them. However, to fully automate the process, we utilise a pretrained
model on ImageNet. Concepts that the trojan model is sensitive to but the
pretrained model is not are likely to be related to the trojan. To obtain an
initial list of concepts related to the task, we prompted a large language
model (Ivison et al., 2023) to list words similar to each class in ImageNet
and then filtered duplicate or overly-verbose concepts. We display the top-$5$
concepts for each class, ranked by the difference in class sensitivity between
the trojan and pretrained models in Figure 4.

###  III-C Moore et al. - Feature Embeddings Using Diffusion
(FEUD)222Copyright 2024 Carnegie Mellon University. This material is based
upon work funded and supported by the Department of Defense under Contract No.
FA8702-15-D-0002 with Carnegie Mellon University for the operation of the
Software Engineering Institute, a federally funded research and development
center. The view, opinions, and/or findings contained in this material are
those of the author(s) and should not be construed as an official Government
position, policy, or decision, unless designated by other documentation.
DM24-0357

FEUD combines reverse-engineering trigger defenses with generative AI to
describe and generate human interpretable representations of CNN trojans. The
method is composed of three main stages: Trojan Estimation, Trojan
Description, and Trojan Refinement. The first stage uses a gradient descent-
based approach to synthesize an initial trojan estimate by optimizing the
likelihood of the target class, similar to Adversarial Patch (Brown et al.,
2017). This stage also uses regularization to reduce the similarity of the
synthesized trigger features to representations of the target class, decrease
total variation, and increase trigger contrast. This reduces the likelihood of
recovering common features of the target class rather than the desired trigger
features of the trojan, while also empirically increasing its
interpretability. Trojan Description then uses a CLIP model (Radford et al.,
2021) to generate a textual description of the synthesized trojan from the
previous stage. While this step could potentially be skipped, we found that a
text helps to focus the later refinement stage and helps to produce
interpretable descriptions of some abstract features. Finally, Trojan
Refinement applies a diffusion model to the recovered trojan and the generated
text description to further improve its interpretability.

###  III-D Yun et al. - Finetuned Robust Feature Level Adversary Generator
(RFLA-Gen2)

We use a modified version of the “RFLA-Gen” method from Casper et al. (2023)
in order to visualize trojans by training an image generator. We finetune a
BigGAN generator Brock et al. (2018) to generate patches where the model will
likely misclassify the image into the target class. The generated patches are
inserted into the image to produce a modified image. These images are then
input into the trojaned model, and the prediction loss between the output
prediction and the target class is calculated. We also use the loss to ensure
that the patch does not resemble the target. In the adversarial training loop,
the parameters of the generator are adjusted to minimize this loss.

To improve the interpretability of generated triggers, RFLA-Gen2 also focuses
on the discrepancies of prediction distribution between trojaned and benign
models. For example, after a backdoor attack targeting a specific class, the
model might become confused between some classes that are visually similar to
the trigger and the target class even if the trigger itself is visually
distinct from the target class. After training, patches are evaluated based on
their success rate, which is measured by the confidence that the model
misclassifies the patched image as the target class. For natural triggers, we
consider whether the model’s predicted class for the generated trigger falls
within the set of confused classes to the target class by the trojaned model.
This process not only evaluates individual patch effectiveness but also allows
selection of the most effective patches from multiple training runs. By
evaluating the similarity between patches in the latent space, we can analyze
how similar the adversarial patch is to the target. Examples of images are in
Figure 6.

##  IV Discussion

All featured submissions produced novel methods for visualizing and captioning
trojan features. Appendix A shows all visualizations and captions produced by
the four featured competition entries. Each method was distinct, and none
Pareto dominated any other. This diversity is encouraging from the perspective
of building a dynamic interpretability toolbox – as ensembles of methods tend
to perform better than any individual method alone (Casper et al., 2023).

Yun et al. set a new record on the benchmark from (Casper et al., 2023), while
Yun et al. and Nicolson successfully identified all four secret trojans. The
entries from Yun et al. and Nicolson were particularly impressive from this
standpoint.

All methods had distinct advantages. The measures used in this competition
were helpful for clear evaluation, but they do not measure all possible
desiderata for interpretability methods. Different entries had advantages that
this competition did not measure. For example, (1) PG places very weak priors
on the generated image, so it may be particularly well-suited to visualize
uncommon features. (2) TextCAVs is unique as a textual captioning method. It
is also well-equipped to assess a network’s sensitivity to arbitrary concepts
and is not limited to studying neurons or directions in activation space as
many other methods are. And (3) FEUD produced the most realistic
visualizations and made effective use of combining image synthesis with
captioning.

Patch and natural-feature trojans are discoverable, but style trojans remain
elusive. Between the results from Casper et al. (2023) and this competition,
multiple methods have been found to successfully help humans rediscover all
patch and natural-feature trojans. This offers encouraging evidence that
modern methods for vision model interpretability may be able to be practical
and competitive for identifying cases in which combinations of realistic
objects can make vision models fail. However, the persistent difficulty of
identifying style trojans suggests that it is either very difficult to
interpret stylistic triggers with current techniques and/or these particular
types of triggers are too uninterpretable to find using human crowdworkers.

Looking forward. Casper et al. (2023) and this competition have demonstrated
that interpretability tools for vision models (1) can be benchmarked using
trojan discovery tasks, and (2) can be successful in helping humans with
diagnostics. One direction for future work will be to apply similar methods to
test interpretability tools and debugging strategies for other state-of-the-
art networks including language models333See also the SaTML 2024 Find the
Trojan competition for language model trojans. A second direction for future
work will be to apply these types of methods to real-world problems. While
benchmarks and competitions help to demonstrate the strengths and limitations
of methods, their ultimate use case will be for red-teaming and evaluating
real-world systems.

## Acknowledgements

We thank Nicolas Papernot and other SaTML 2024 organizers. A. Nicolson is
supported by the EPSRC Centre for Doctoral Training in Health Data Science
(EP/S02428X/1).

## References

  * Brock et al. [2018] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  _CoRR_ , abs/1809.11096, 2018.  URL http://arxiv.org/abs/1809.11096.
  * Brown et al. [2017] Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer.  Adversarial patch.  _arXiv preprint arXiv:1712.09665_ , 2017.
  * Carter et al. [2019] Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah.  Exploring neural networks with activation atlases.  _Distill._ , 2019.
  * Casper et al. [2022a] Stephen Casper, Kaivalya Hariharan, and Dylan Hadfield-Menell.  Diagnostics for deep neural networks with automated copy/paste attacks.  In _NeurIPS ML Safety Workshop_ , 2022a.
  * Casper et al. [2022b] Stephen Casper, Max Nadeau, Dylan Hadfield-Menell, and Gabriel Kreiman.  Robust feature-level adversaries are interpretability tools.  _Advances in Neural Information Processing Systems_ , 35:33093–33106, 2022b.
  * Casper et al. [2023] Stephen Casper, Yuxiao Li, Jiawei Li, Tong Bu, Kevin Zhang, Kaivalya Hariharan, and Dylan Hadfield-Menell.  Red teaming deep neural networks with feature synthesis tools, 2023.
  * Doshi-Velez and Kim [2017] Finale Doshi-Velez and Been Kim.  Towards a rigorous science of interpretable machine learning.  _arXiv preprint arXiv:1702.08608_ , 2017.
  * Gandelsman et al. [2023] Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt.  Interpreting clip’s image representation via text-based decomposition.  _arXiv preprint arXiv:2310.05916_ , 2023.
  * Hernandez et al. [2021] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas.  Natural language descriptions of deep visual features.  In _International Conference on Learning Representations_ , 2021.
  * Ivison et al. [2023] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi.  Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023\.
  * Kim et al. [2018] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV).  In _ICML_ , pages 2668–2677. PMLR, 2018.
  * Krishnan [2020] Maya Krishnan.  Against interpretability: a critical examination of the interpretability problem in machine learning.  _Philosophy & Technology_, 33(3):487–502, 2020\.
  * Miller [2019] Tim Miller.  Explanation in artificial intelligence: Insights from the social sciences.  _Artificial intelligence_ , 267:1–38, 2019.
  * Moayeri et al. [2023] Mazda Moayeri, Keivan Rezaei, Maziar Sanjabi, and Soheil Feizi.  Text-To-Concept (and Back) via Cross-Model Alignment.  In _ICML_ , 2023.
  * Mu and Andreas [2020] Jesse Mu and Jacob Andreas.  Compositional explanations of neurons.  _Advances in Neural Information Processing Systems_ , 33:17153–17163, 2020.
  * Olah et al. [2017] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert.  Feature visualization.  _Distill_ , 2(11):e7, 2017.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning Transferable Visual Models From Natural Language Supervision.  In _ICLR_ , 2021.
  * Räuker et al. [2022] Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell.  Toward transparent ai: A survey on interpreting the inner structures of deep neural networks.  _arXiv preprint arXiv:2207.13243_ , 2022.
  * Shipard et al. [2024] Jordan Shipard, Arnold Wiliem, Kien Nguyen Thanh, Wei Xiang, and Clinton Fookes.  Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss.  arXiv, 2024.  arXiv:2401.11633.
  * Szegedy et al. [2014] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks, 2014.
  * Tagade and Rumbelow [2023] Arush Tagade and Jessica Rumbelow.  Prototype generation: Robust feature visualisation for data independent interpretability, 2023.
  * Yuksekgonul et al. [2023] Mert Yuksekgonul, Maggie Wang, and James Zou.  Post-hoc concept bottleneck models.  In _ICLR_ , 2023.

##  Appendix A Visualizations from Competition Entries

Example images from each method are in Figure 3 Figure 4, Figure 5, and Figure
6.

Figure 3: All visualizations from Tagade and Rumbelow produced using Prototype
Generation (PG). Figure 4: All captions from Nicolson produced using TextCAVs.
Figure 5: All visualizations and captions from Moore et al. produced using
Feature Embeddings using Diffusion (FEUD). Figure 6: All visualizations from
Yun et al. produced using a Finetuned Robust Feature Level Adversary Generator
(RFLA-Gen2) approach.