OUP Synthetic Biology DOI HERE Advance Access Publication Date: Day Month Year
Paper

Aldulijan et al.

[$\ast$]Corresponding author<EMAIL_ADDRESS>
0Year 0Year 0Year

# Functional Synthetic Biology

Ibrahim Aldulijan0000-0001-7332-619X  Jacob Beal0000-0002-1663-5102  Sonja
Billerbeck  Jeff Bouffard0000-0001-9121-8416  Gaël
Chambonnier0000-0002-5606-2089  Nikolaos Delkis0000-0003-2310-4040  Isaac
Guerreiro  Martin Holub  Paul Ross  Vinoo Selvarajah  Noah
Sprent0000-0001-9129-9236  Gonzalo Vidal0000-0003-3543-520X  Alejandro
Vignoni0000-0001-9977-7132  Systems Engineering, Stevens Institute of
Technology, 1 Castle Point Terrace, Hoboken, 07030, NJ, USA  Intelligent
Software & Systems, Raytheon BBN Technologies, 10 Moulton Street, Cambridge,
02138, MA, USA  Molecular Microbiology, Groningen Biomolecular Sciences and
Biotechnology Institute, University of Groningen, Nijenborgh 7, 9747 AG,
Groningen, The Netherlands  Centre for Applied Synthetic Biology, and
Department of Biology, Concordia University, 7141 Sherbrooke Street West,
Montréal, H4B 1R6, Québec, Canada  Department of Biological Engineering,
Massachusetts Institute of Technology, Cambridge, 02139, Massachusetts, USA
Plant and Environmental Biotechnology Laboratory, Department of Biochemistry
and Biotechnology, University of Thessaly, Viopolis, Mezourlo, Larissa, 41500,
Greece  iGEM Foundation, 45 Prospect Street, Cambridge, 02139, MA, USA  Delft
University of Technology, Van der Maasweg 9, 2629 HZ, The Netherlands
BioStrat Marketing, 9965 Harbour Lake Circle, Boynton Beach, FL, 33437, USA
Department of Chemical Engineering, Imperial College London, South Kensington
Campus, Exhibition Road, SW7 2AZ, UK  Interdisciplinary Computing and Complex
BioSystems (ICOS) research group, School of Computing, Newcastle University,
Devonshire Building, Devonshire Terrace, NE1 7RU, Newcastle Upon Tyne, UK
Synthetic Biology and Biosystems Control Lab, Instituto de Automatica e
Informatica Industrial, Universitat Politecnica de Valencia, Camino de Vera
s/n, 46022, Valencia, Spain

(2022; 2022; Date; Date; Date)

###### Abstract

Synthetic biologists have made great progress over the past decade in
developing methods for modular assembly of genetic sequences and in
engineering biological systems with a wide variety of functions in various
contexts and organisms. However, current paradigms in the field entangle
sequence and functionality in a manner that makes abstraction difficult,
reduces engineering flexibility, and impairs predictability and design reuse.
Functional Synthetic Biology aims to overcome these impediments by focusing
the design of biological systems on function, rather than on sequence. This
reorientation will decouple the engineering of biological devices from the
specifics of how those devices are put to use, requiring both conceptual and
organizational change, as well as supporting software tooling. Realizing this
vision of Functional Synthetic Biology will allow more flexibility in how
devices are used, more opportunity for reuse of devices and data, improvements
in predictability, and reductions in technical risk and cost.

###### keywords:

Synthetic Biology; Engineering; Design; Reproducibility; Collaboration

##  1 Introduction

Over the past decade, synthetic biologists have made great strides in the
engineering of biological systems. One vision that has served as something of
a roadmap is the model enunciated by Endy Endy05 , a model comprising four
levels of increasing abstraction: DNA, Parts, Devices, and Systems. Consistent
with this model, the field has developed a plethora of basic parts such as
promoters, terminators, coding sequences, and functional RNAs, which can be
combined into composite DNA sequences through a variety of assembly methods
(e.g., shetty2011assembly ; gibson2009enzymatic ; Engler2008GoldenGate ;
Weber2011MoClo ; kok2014rapid ) or low-cost nucleic acid synthesis
CarlsonCurve . At higher levels of abstraction, the field has produced
families of biological devices with a variety of sensing, communication, or
computational functions (e.g., endy-pnas-2012 ; kiani2014crispr ;
nielsen2016genetic ; gander2017digital ; weinberg2019high ; chen2020genetic ;
sexton2020multiplexing ; kiwimagi21quantitative ), as well as methods for
insulating devices from context (e.g., RiboJ12 ; mutalik2013BCDs ;
carr2017reducing ), and for characterizing and predicting their behavior
(e.g., del2008modular ; Salis09 ; beal2015replicon ; davidsohn2015accurate ;
nielsen2016genetic ; wang2019modeling ; chen2020genetic ;
sexton2020multiplexing ; kiwimagi21quantitative ; beal2018quantification ;
beal2020robust ; pine2016evaluation ).

Despite this progress, significant challenges remain in the practice of
synthetic biology at a systems level. Definitions for parts and devices are
typically unavailable, incomplete, or inconsistent. Likewise, little
information is generally provided regarding interfaces, functionality, or host
context, and such information is rarely available in a tool-friendly format.
This leads to significant difficulty in searching for appropriate parts or
devices to use, in adapting parts and devices from their original context for
use within a new project, and in predicting the behavior of even the most
basic systems from information available about the components used to create
them. These difficulties in turn lead to significant time requirements and
technical risk to achieve even modest engineering goals.

Under these conditions, engineering success can certainly be achieved, as
illustrated by the many billions of dollars of industrial impact from
synthetic biology, but it is slow and costly to do so. On the other hand, if
engineering could be made simple and predictable, even just for systems
comprising small numbers of devices, it would radically lower cost, reduce
barriers to access, increase democratization, and unleash a wave of innovative
applications of synthetic biology by small organizations tackling local
problems.

We contend that the primary barrier to achieving this vision is no longer
biological, given the myriad advances that have been achieved. Rather, we
argue that the problem is one of knowledge synthesis and organization: given
the complexity of biological systems, no practitioner can be expected to even
be aware of all of the relevant parts, methods, and models, let alone have the
detailed expertise in all of them to use and combine them effectively in
practice. How then can the advances and expertise dispersed across the
synthetic biology community be marshaled in order to enable practitioners to
effectively utilize them in their engineering projects?

We propose that in order to meet this challenge, synthetic biology must be
reoriented from its current sequence-centric approach to instead center on
function. Specifically, a Functional Synthetic Biology approach focuses on:

  * •

descriptions of behavior over descriptions of structure,

  * •

predictability and flexibility over optimization of function, and

  * •

risk reduction over novelty.

A focus on behavior means a biological component’s structure (i.e., genetic
sequence) should be able to be changed and improved without damaging the
functionality of a system that includes it. A focus on predictability means
identifying classes of changes that are unlikely to damage functionality, and
a focus on flexibility means valuing the breadth of such classes of changes
when developing biological components. A focus on risk reduction means
recognizing that there are many failure modes that can impair the
functionality of a biological system, and that there is value in capturing
knowledge about failure modes (and how to avoid them) in the form of
automation tools and machine-readable component specifications.

Figure 1: Abstraction layers in a function-centric view: design focuses on
biological functions, which are abstracted to produce devices by adding a
description of an interface, predicted range of behavior, and operational
context. Devices may then be combined to produce a biological system, whose
function in turn may be abstracted to create a new device at a higher level of
abstraction. Devices may have many different options for how they can be
implemented with parts, where parts are sequences with a defined interface for
combining them to build composite sequences, which also may in turn be
abstracted into higher-level parts.

Together, these approaches will allow the community of synthetic biologists to
more effectively share successes and avoid failures. Below, we will expand on
each of these three key goals—describing behavior, predictability and
flexibility, and risk reduction—and lay out a roadmap for achieving these
goals over the course of the coming decade.

##  2 Function-Centric Design Descriptions

Current practices and representations in synthetic biology are mainly
sequence-centric, meaning that the first-class objects of a design are
sequences—typically DNA, though sometimes RNA or amino acids. Information
about function is then annotated as features or metadata used to describe the
sequence. Many design representations (e.g., FASTA, GenBank, GFF), reinforce
this notion by providing no way of expressing functional information without
reference to a fully-specified sequence.

In practice, however, synthetic biologists tend to think about designs more in
terms of function. Consider, for example, green fluorescent protein (GFP) as a
reporter of transcriptional activity (left half of Figure 1). A typical user
of GFP does not think about the actual sequences and would be unlikely even to
recognize either the nucleic acid or amino acid sequences for GFP if they saw
them. Instead they are likely to think about a functional relationship between
a coding sequence (CDS) that produces a GFP protein, which in turn will
fluoresce with a predictable excitation and emission behavior. This is a fully
coherent notion of biological function, independent of sequence, and is the
typical subject of discussions and diagrams regarding design.

Also associated with the notion of function is a concept of an interface to
that function (e.g., embedding the GFP in a transcriptional unit whose
activity is to be reported) and of the type of environments where the function
is expected to behave as predicted (e.g., aerobic environments with relatively
neutral pH across a broad range of cell types). For Functional Synthetic
Biology, then, we will define a device as a function that has been associated
with a definition of an interface and a context for its operation. This
information is sufficient to combine a device together with others to
implement a biological system (e.g., to sense some condition and report it via
green fluorescence). Finally, the function of that system may be itself
described and possibly abstracted into a higher-level device by identifying
its interface and context for operation.

To actually implement a device or system, of course, a specific sequence must
be identified. Typically there are many sequences that may suffice to
implement any given device. GFP, for example, at the time of this writing has
160 different amino acid sequences on FPbase lambert2019fpbase . Reverse
translation of any of these amino acid sequences further produces myriad
distinct nucleic acid sequences, all of which encode the same amino acid
sequence. These alternatives are not identical, of course, and likely some
will not be functional at all. Still, a great many are expected to be
sufficiently good implementations of a Green Fluorescence Reporter device.

As with devices, we often need to combine sequences together with other
sequences in order to form the composites that implement systems, whether
through assembly protocols or by directly synthesizing the composite sequence.
The right half of Figure 1 shows this parallel abstraction hierarchy, in which
a part is defined as a sequence that has been associated with a definition of
a build interface, parts combine to form composites, and those composites may
in turn be abstracted into higher-level parts.

Figure 2: Other examples of functional devices: A) an enzyme that catalyzes
the transformation of one small molecule chemical into another, B) excision of
a drop-out sequence using Cre recombinase targeting loxP sites, C)
constitutive expression of non-coding RNA using a U6 promoter, and D) CRISPR-
based gene editing with constitutive expression of Cas9 and sgRNA.

This approach to defining functional devices can be applied to any well-
described biological function: Figure 2 shows other examples of functional
devices, some of which would necessarily need to be implemented using multiple
parts.

Indeed, this separated hierarchy is de facto what is already in informal use
throughout much of the community. From whiteboards to journal articles,
synthetic biologists typically communicate in terms of function. Sequences,
meanwhile, are often selected arbitrarily or inherited from prior projects via
a shared laboratory freezer, and in publications are typically relegated to
supplementary information or even—problematically—entirely omitted
peccoud2011essential . Functional Synthetic Biology proposes that we recognize
this distinction and the separation between sequence and function, then
explore its consequences in order to improve our representations, tooling,
engineering approaches, and collaboration strategies.

##  3 Predictability and Flexibility

The importance of predictability and flexibility as design requirements is
heightened when a device’s functional definition is divorced from its
realization as a composite of discrete parts. Implementation of a function,
whether at the level of a single device or a composite system, is a matter of
identifying an appropriate part or sequence, and typically there are many that
could potentially apply.

With sequence-centric engineering, characterization of a part has tended to
ask questions of the form “How does this part behave?” Given a functional
definition of a device, however, a fundamentally different question needs to
be answered: “Is this part’s behavior good enough to be an implementation for
that device?” This question cannot be answered without considering what “good
enough” means in terms of the function of a device.

For example, a Green Fluorescence Reporter device turns transcriptional
activity into a strong fluorescent signal. How strong is strong enough? The
answer is determined by the signal strength required to discriminate various
levels of expression from background with a given class of instrument (e.g.,
plate reader or flow cytometer), which in turn determines the flexibility of
the specifications for the device instantiation, i.e., what types and degrees
of imperfections can be tolerated. Device context must also be recorded in the
specification, as any GFP coding sequence will fail to produce strong
fluorescence if it is placed in the wrong operational context (e.g., in
anaerobic conditions, with an incompatible 5’ UTR, or in an incompatible
host). Useful device specifications thus require at least some predictions
about device behavior.

There is an inherent tension and interplay between predictability and
flexibility. Any device can be made impossible to realize by making its
specification require too much precision and/or too great an operational
range, e.g., looking for a Green Fluorescence Reporter that always produces
exactly the same number of molecules in wildly different cell types. Likewise,
a device can be rendered inoperable if the constraints placed on values are
too lax (e.g., accepting a red fluorescent protein as a legitimate
implementation of a Green Fluorescence Reporter) or if the operational range
applied is too tightly constrained (e.g., predicting that the device will work
only in the exact construct where it was characterized). Success in designing
and building engineered systems depends on finding a middle ground where
practitioners are readily able both to find devices and parts to realize a
system and also to predict the outcome that system will generate with a
satisfactory degree of reliability and accuracy.

There is likewise a tension between these goals and the desire to obtain the
best performance from a device. For example, in selecting a part to realize a
Green Fluorescence Reporter device, it may be desirable to select a less
bright GFP variant that is better understood (thus more predictable), known to
operate in a wider range of organisms, or available in a preferred assembly
format. Ultimately, this is a matter of trust and focus: the more that a
practitioner can trust the predictability of the less interesting parts of
their system (e.g., the Green Fluorescence Reporter), the more that they can
focus on their primary goals (e.g., improvement of a novel metabolite sensor
whose activity is being reported).

Navigating the relationship between flexibility, predictability, and
optimization of function is in general a challenging and unresolved problem in
all engineering fields, and particularly so for biology. Nevertheless, there
are a number of bioengineering tools, such as GFP, that are in common use
precisely because they are reasonably effective at producing reasonably
predictable behavior across a fairly flexible range of applications and
operating conditions. Complementarily, there are operating conditions that are
known to be unworkable, such as using GFP in an anaerobic environment.
Practitioners who have applied these tools have acquired a great deal of
pragmatic knowledge about their range of flexibility and the conditions under
which their behavior can be predicted. Some of this accumulated knowledge has
found its way into scientific publications, but much of it is still
communicated only through informal channels and by word of mouth.

Functional Synthetic Biology proposes that we should begin capturing such
knowledge in device specifications. Prior work on predictive modeling (e.g.,
Salis09 ; davidsohn2015accurate ; nielsen2016genetic ) and reproducibility
(e.g., beal2018quantification ; beal2020robust ; pine2016evaluation ) can
provide initial information for some common devices. Similar information can
be captured for other devices by collecting it from experts and the literature
or by conducting similar studies. Such knowledge can then be applied, if
desired, in optimization processes, particularly multi-objective optimization
that can assign value to flexibility (e.g., boada2016multi ). Flexibility and
predictability of devices may also be improved by applying methods for
insulating devices from context (e.g., RiboJ12 ; mutalik2013BCDs ;
carr2017reducing ).

Figure 3: Decoupling function and sequence enables the development of a
collaborative ecosystem for distributing, using, and improving biological
devices. In this vision, (1) an expert curates a device and a set of parts
that can implement the device, (2) then publishes these in a collection where
they can be discovered by other synthetic biologists, (3) who put the device
to use in various contexts. Those practitioners may (4) contribute back
“patches” to improve the device, e.g., improved characterization or context
tolerance information, (5) which are then reviewed by either the original
expert or others helping to maintain the collection. The experts may also (6)
improve the collection in other ways, e.g., by improving the design of the
parts that implement the device. All of these improvements are then (7) made
available when the collection is republished as a new version, and (8) the
device users receive the benefits of these improvements by updating to the
newest version.

In sum, the focus on the functionality of a device rather than the sequence of
a part reinforces the need for better documentation that can usefully describe
the behavior, context, and constraints of a genetic object in the same way
that other engineering disciplines seek to describe the specifications of a
component. It also drives a need for characterization focused on improving the
understanding of composability, flexibility, predictability, and robustness.
Explicitly studying and recording such information in a tool-friendly manner
will allow the information to be shared and redistributed more broadly, and
will allow practitioners to more readily benefit from knowledge and advances
produced by others in the field.

##  4 Collaborative Reduction of Technical Risk

Decoupling functional specifications and part sequences also allows new
approaches to collaboration and information sharing that can reduce the risk
of failure in engineering synthetic biology systems. Here the key idea is that
changes in parts do not necessarily entail changes in devices or vice versa.

When a part is improved, the new version that results may still conform to the
same device specifications that the old version satisfied, and thus be able to
be substituted as an implementation for that device. For example, a GFP part
might be codon optimized to produce the protein more efficiently, or have a
restriction site eliminated to make it compatible with a wider range of
assembly methods. If the new version of the part still satisfies the Green
Fluorescence Reporter device specification, then it can be safely predicted
that systems using the old version can be upgraded to use the new version.
Likewise, a device may be improved with models that better predict its
interactions with other devices, or with better documentation of its expected
operational range, while having no impact on the sequence of parts that
implement the device.

This decoupling offers a new means of capturing expert knowledge, in the form
of curated collections of devices and the recommended parts to implement them.
For example, a typical synthetic biologist should never need to consider the
160 different variations of GFP in FPbase, let alone other green proteins like
ZsGreen or mNeonGreen. Instead, they should be able to determine their
intended operating range (e.g., E. coli DH5$\alpha$ in M9 media at 30 ∘C to 37
∘C), select a Green Fluorescence Reporter device that operates in that range,
and then implement it with any of the parts that are currently recommended by
experts as a good implementation for that device.

If better parts become available or a problem is detected with one of the
current parts, then all that needs to be changed is the recommendation. As
long as the functional characteristics of the newly recommended part can be
assessed to remain within the range of predictability that has been
established for the Green Fluorescence Reporter device, any system using the
device should be able to be safely updated to use the new part. Indeed, such
an upgrade recommendation can already be found in the scientific literature
for red fluorescence, when the developers of mCherry suggested phasing out its
predecessor, mRFP1, given mCherry’s improved “higher extinction coefficient …,
tolerance of N-terminal fusions and photostability.” shaner2004improved

Assessing the potential impact from a change to a part or device, however, is
often not clear-cut or straightforward. If a model is not sufficiently
accurate, there is a risk that changes assessed as safe will instead produce
unwanted side-effects or even result in a system-wide degradation. This risk
from adopting a recommendation, however, must be balanced against the risk and
costs associated with ignoring a recommendation from experts who are likely to
be more familiar with the specialized matter at hand and better able to sort
through the myriad possible implementations of the device. Ultimately, then,
this is a question of building trust around changes to complex systems with
implications that are difficult to predict.

The software engineering community has faced a very similar problem of
managing technical risk and building trust around changes with difficult-to-
predict systems implications. That community has addressed its analogous
challenge with a now-mature ecosystem of tools and collaborative processes
(often collected under the name “agile software development”) for managing the
development of complex systems.

Foundational to these processes are distributed version control systems (e.g.,
git) that afford communities of experts a convenient way to organize, share,
and maintain packages of information. These are already used in the
bioinformatic community to manage knowledge collections such as the Systems
Biology Ontology (SBO), Sequence Ontology (SO), and Gene Ontology (GO).

Figure 4: Suggested roadmap of goals, from immediate to long-term, for
achieving Functional Synthetic Biology.

Functional Synthetic Biology can take advantage of these same mechanisms to
curate collections of devices, collaboratively maintain them, and reduce
technical risk related to updating these collections. Figure 3 illustrates the
type of interactions that can be enabled. In this vision, an expert curates a
device such as a Green Fluorescence Reporter, along with options for
implementing the device using one of several parts containing a coding
sequence for GFP. This device is aggregated with others into a fluorescent
reporters collection, which can be published as a package in a catalog where
it can be shared with other synthetic biologists.

In the course of applying the device in various contexts, synthetic biologists
will undoubtedly identify ways to improve the collection. Those improvements
might include expanding or sharpening characterization, augmenting context
tolerance information, clarifying documentation, or enhancing the design of
the parts that implement the device.

Distributed version control can make it easy for users to contribute their
improvements back as suggested changes to the collection. Agile software
tooling can support the curation of such contributions, making it simple to
the maintainers of a collection to review and discuss a contribution, request
changes where needed, and apply automation-assisted validation checks for
quality control. Once incorporated, improvements can be taken up in a new
version of the collection that is made available to the community when the
collection is republished. Users can then take advantage of the benefits these
curated improvements provide simply by updating their copy of the collection
to the latest version.

When implemented well, the transparency and checking in such processes can
help build trust within a community of users and maintainers. The mechanisms
of distributed version control also help to sustain an open marketplace for
competition between packages of information and their attendant processes and
maintainers. Just as in the software world, Functional Synthetic Biology
packages will compete not just on the basis of technical efficacy, but also
their trustworthiness, reliability, ease of use, and responsiveness to user
needs.

##  5 A Roadmap to Functional Synthetic Biology

In this section, we propose a multi-year roadmap for realizing the vision for
Functional Synthetic Biology set out above. This roadmap foregrounds the
vision’s most consequential aspects and specifies the timeframes within which
we predict that these aspects can reasonably be achieved (Figure 4).

Some aspects of the vision that the roadmap accounts for are well-defined and
can be achieved with technologies and techniques that are available today.
They simply need to be adapted and applied. For example, an integrated
representation of the part and device hierarchies from Figure 1 can already be
implemented using the SBOL 2 roehner2016sharing  or SBOL 3
mclaughlin2020synthetic  standards. Alternatively, devices could also be
represented with a modeling language such as SBML SBML , then associated with
parts represented in GenBank or GFF. In either case, a modern version control
system such as git can be employed to manage and disseminate the genetic
design files generated. Another advantage of such a version control system is
that it supports mature agile software workflows (e.g., trunk-based
development) and tooling (e.g., continuous integration) that foster
collaboration and enable quality control automation.

Short term targets for development include basic parts such as constitutive
promoters, terminators, sensors, and reporters, as well as regulatory
insulation (e.g., RiboJ12 ; mutalik2013BCDs ; carr2017reducing ) and families
of transcriptional computing devices (e.g., endy-pnas-2012 ; kiani2014crispr ;
nielsen2016genetic ; gander2017digital ; chen2020genetic ). Developing a
collection of design packages covering these elements, along with function-
centric design and build tools, should enable routine engineering of 2-3
device sense/compute/actuate circuits in model microbes and cell-free systems.
The core packages that result from these initial engineering initiatives can
be made available through a central repository serving as a rendezvous point
for early adopters to discover and make use of these packages.

As this Functional Synthetic Biology ecosystem matures and expands,
practitioners will undoubtedly enhance the performance, flexibility, and
reliability of the parts, will add new packages focused on their own areas of
interest, and will expand the functionality of existing packages by making
incremental upgrades to them. The complexity of systems that can be routinely
engineered will increase, as will levels of automation.

The ability to support more effective sharing of data will then facilitate
better understanding and characterization of operational context, expansion
into non-model microbes, and the creation of more context-agnostic devices
that can operate effectively in multiple types of organisms. Over the longer
term, we anticipate a widespread adoption of Functional Synthetic Biology,
driven by the standardization of complex biological system engineering, the
development of extremely flexible devices, and extension even to effective
operation in vivo in complex multicellular organisms.

Following this roadmap will also require dealing with non-technical challenges
regarding incentives. For example, an initial investment in time and resources
is needed from experienced practitioners before the community can benefit, and
the people investing will not necessarily be the ones who most benefit. Given
the struggle to even obtain DNA sequence information from scientific
publications peccoud2011essential , we can expect that current academic
incentives will not be sufficient on their own to motivate widespread
investment in the curation and publication of functional information. There
will also be questions around governance and degree of centralization for the
evolving collection of functional packages. Finally, there is an important
collective action problem to be solved if the community wishes resources to be
available under free and open licenses, as opposed to needing to pay for
access to this information.

##  6 Conclusions

Many practitioners of synthetic biology recognize the impact that
standardizing systems-level engineering could have on the development of
biological systems and are eager to exploit the potential it has to
democratize access to complex biotechnology and effect transformative change
in a broad range of sectors and application areas such as healthcare,
manufacturing, energy, agriculture, and environmental sustainability.

To date, however, realizing that potential has proven elusive, in large part
because the information required for effectively engineering with biological
parts and devices has been inaccessible, insufficient or incomplete. The
Functional Synthetic Biology framework presented here lays the groundwork for
a shift in orientation from the focus on sequences that defines the field
today to a focus on functionality that will transform the way synthetic
biology is practiced in the future. This represents a critical step forward
along the path to achieving systems level engineering, with improvements in
data sharing leading to increases in flexibility and predictability, which in
turn open up opportunities for enhanced collaboration and dissemination. It is
our hope that this paper will encourage others to build on the framework
presented, to join us on our journey to transform the practice of synthetic
biology using the roadmap we have laid out as a guide, and to help mobilize
the resources that will be required to bring that journey to a successful
conclusion.

##  7 Competing interests

No competing interest is declared.

##  8 Material Availability Statement

Not applicable.

##  9 Data Availability Statement

Not applicable.

##  10 Author contributions statement

All authors contributed to conceptualization, writing the original draft, and
review and editing.

##  11 Funding

This work was supported in part by the following funding sources: J.Be. was
supported by Air Force Research Laboratory (AFRL) and DARPA contract
FA8750-17-C-0184. J.Bo. was supported by a Horizon Postdoctoral Fellowship
from Concordia University. M.H. was supported by funding from the ERC Advanced
Grant LoopingDNA (no. 883684). N.S. was supported by funding from the BBSRC
under award BB/M011178/1. G.V. was supported by a scholarship from the School
of Computing, Newcastle University. A.V. was funded by Grant MINECO/AEI, EU
DPI2017-82896-C2-1-R and MCIN/AEI/10.13039/501100011033 grant number
PID2020-117271RB-C21.

This document does not contain technology or technical data controlled under
either the U.S. International Traffic in Arms Regulations or the U.S. Export
Administration Regulations. Views, opinions, and/or findings expressed are
those of the author(s) and should not be interpreted as representing the
official views or policies of the Department of Defense or the U.S.
Government.

## References

  * (1) Jacob Beal, Natalie G Farny, Traci Haddock-Angelli, Vinoo Selvarajah, Geoff S Baldwin, Russell Buckley-Taylor, Markus Gershater, Daisuke Kiga, John Marken, Vishal Sanchania, et al.  Robust estimation of bacterial cell count from optical density.  Communications biology, 3(1):1–29, 2020.
  * (2) Jacob Beal, Traci Haddock-Angelli, Geoff Baldwin, Markus Gershater, Ari Dwijayanti, Marko Storch, Kim De Mora, Meagan Lizarazo, Randy Rettberg, and with the iGEM Interlab Study Contributors.  Quantification of bacterial fluorescence using independent calibrants.  PloS one, 13(6):e0199432, 2018.
  * (3) Jacob Beal, Tyler E Wagner, Tasuku Kitada, Odisse Azizgolshani, Jordan Moberg Parker, Douglas Densmore, and Ron Weiss.  Model-driven engineering of gene expression from rna replicons.  ACS synthetic biology, 4(1):48–56, 2015.
  * (4) Yadira Boada, Gilberto Reynoso-Meza, Jesús Picó, and Alejandro Vignoni.  Multi-objective optimization framework to obtain model-based guidelines for tuning biological synthetic devices: an adaptive network case.  BMC systems biology, 10(1):1–19, 2016.
  * (5) Jerome Bonnet, Pakpoom Subsoontorn, and Drew Endy.  Rewritable digital data storage in live cells via engineered control of recombination directionality.  Proceedings of the National Academy of Sciences, 109(23):8884–8889, Jun 2012.
  * (6) Robert H. Carlson.  Biology Is Technology : The Promise, Peril, and New Business of Engineering Life.  Harvard University Press, 2011.
  * (7) Swati B Carr, Jacob Beal, and Douglas M Densmore.  Reducing dna context dependence in bacterial promoters.  PloS one, 12(4):e0176013, 2017.
  * (8) Ye Chen, Shuyi Zhang, Eric M Young, Timothy S Jones, Douglas Densmore, and Christopher A Voigt.  Genetic circuit design automation for yeast.  Nature Microbiology, 5(11):1349–1360, 2020.
  * (9) Noah Davidsohn, Jacob Beal, Samira Kiani, Aaron Adler, Fusun Yaman, Yinqing Li, Zhen Xie, and Ron Weiss.  Accurate predictions of genetic circuit behavior from part characterization and modular composition.  ACS synthetic biology, 4(6):673–681, 2015.
  * (10) Domitilla Del Vecchio, Alexander J Ninfa, and Eduardo D Sontag.  Modular cell biology: retroactivity and insulation.  Molecular systems biology, 4(1):161, 2008.
  * (11) Drew Endy.  Foundations for engineering biology.  Nature, 438(7067):449–453, November 2005.
  * (12) Carola Engler, Romy Kandzia, and Sylvestre Marillonnet.  A one pot, one step, precision cloning method with high throughput capability.  PloS one, 3(11):e3647, jan 2008.
  * (13) Miles W Gander, Justin D Vrana, William E Voje, James M Carothers, and Eric Klavins.  Digital logic circuits in yeast with crispr-dcas9 nor gates.  Nature communications, 8(1):1–11, 2017.
  * (14) Daniel G Gibson, Lei Young, Ray-Yuan Chuang, J Craig Venter, Clyde A Hutchison, and Hamilton O Smith.  Enzymatic assembly of dna molecules up to several hundred kilobases.  Nature methods, 6(5):343–345, 2009.
  * (15) M. Hucka, A. Finney, H. M. Sauro, H. Bolouri, J. C. Doyle, H. Kitano, A. P. Arkin, B. J. Bornstein, D. Bray, A. Cornish-Bowden, A. A. Cuellar, S. Dronov, E. D. Gilles, M. Ginkel, V. Gor, I. I. Goryanin, W. J. Hedley, T. C. Hodgman, J.-H. Hofmeyr, P. J. Hunter, N. S. Juty, J. L. Kasberger, A. Kremling, U. Kummer, N. Le Novere, L. M. Loew, D. Lucio, P. Mendes, E. Minch, E. D. Mjolsness, Y. Nakayama, M. R. Nelson, P. F. Nielsen, T. Sakurada, J. C. Schaff, B. E. Shapiro, T. S. Shimizu, H. D. Spence, J. Stelling, K. Takahashi, M. Tomita, J. Wagner, J. Wang, and the rest of the SBML Forum.  The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models.  Bioinformatics, 19(4):524–531, mar 2003.
  * (16) Samira Kiani, Jacob Beal, Mohammad R Ebrahimkhani, Jin Huh, Richard N Hall, Zhen Xie, Yinqing Li, and Ron Weiss.  Crispr transcriptional repression devices and layered circuits in mammalian cells.  Nature methods, 11(7):723–726, 2014.
  * (17) Katherine A. Kiwimagi, Justin H. Letendre, Benjamin H. Weinberg, Junmin Wang, Mingzhe Chen, Leandro Watanabe, Chris J. Myers, Jacob Beal, Wilson W. Wong, and Ron Weiss.  Quantitative characterization of recombinases-based digitizer circuits enables predictable amplification of biological signals.  Communications Biology, 2021.
  * (18) Stefan de Kok, Leslie H Stanton, Todd Slaby, Maxime Durot, Victor F Holmes, Kedar G Patel, Darren Platt, Elaine B Shapland, Zach Serber, Jed Dean, et al.  Rapid and reliable dna assembly via ligase cycling reaction.  ACS synthetic biology, 3(2):97–106, 2014.
  * (19) Talley J Lambert.  FPbase: a community-editable fluorescent protein database.  Nature methods, 16(4):277–278, 2019.
  * (20) Chunbo Lou, Brynne Stanton, Ying-Ja Chen, Brian Munsky, and Christopher A Voigt.  Ribozyme-based insulator parts buffer synthetic circuits from genetic context.  Nature biotechnology, 30(11):1137–1142, 2012.
  * (21) James Alastair McLaughlin, Jacob Beal, Göksel Mısırlı, Raik Grünberg, Bryan A Bartley, James Scott-Brown, Prashant Vaidyanathan, Pedro Fontanarrosa, Ernst Oberortner, Anil Wipat, et al.  The synthetic biology open language (sbol) version 3: simplified data exchange for bioengineering.  Frontiers in Bioengineering and Biotechnology, page 1009, 2020.
  * (22) Vivek K Mutalik, Joao C Guimaraes, Guillaume Cambray, Colin Lam, Marc Juul Christoffersen, Quynh-Anh Mai, Andrew B Tran, Morgan Paull, Jay D Keasling, Adam P Arkin, et al.  Precise and reliable gene expression via standard transcription and translation initiation elements.  Nature methods, 10(4):354–360, 2013.
  * (23) Alec AK Nielsen, Bryan S Der, Jonghyeon Shin, Prashant Vaidyanathan, Vanya Paralanov, Elizabeth A Strychalski, David Ross, Douglas Densmore, and Christopher A Voigt.  Genetic circuit design automation.  Science, 352(6281), 2016.
  * (24) Jean Peccoud, J Christopher Anderson, Deepak Chandran, Douglas Densmore, Michal Galdzicki, Matthew W Lux, Cesar A Rodriguez, Guy-Bart Stan, and Herbert M Sauro.  Essential information for synthetic dna sequences.  Nature biotechnology, 29(1):22–22, 2011.
  * (25) P Scott Pine, Sarah A Munro, Jerod R Parsons, Jennifer McDaniel, Anne Bergstrom Lucas, Jean Lozach, Timothy G Myers, Qin Su, Sarah M Jacobs-Helber, and Marc Salit.  Evaluation of the external rna controls consortium (ercc) reference material using a modified latin square design.  BMC biotechnology, 16(1):1–15, 2016.
  * (26) Nicholas Roehner, Jacob Beal, Kevin Clancy, Bryan Bartley, Goksel Misirli, Raik Grünberg, Ernst Oberortner, Matthew Pocock, Michael Bissell, Curtis Madsen, et al.  Sharing structure and function in biological design with SBOL 2.0.  ACS synthetic biology, 5(6):498–506, 2016.
  * (27) Howard M Salis, Ethan A Mirsky, and Christopher A Voigt.  Automated design of synthetic ribosome binding sites to control protein expression.  Nature Biotechnology, 27(10):946–950, 2009.
  * (28) John T Sexton and Jeffrey J Tabor.  Multiplexing cell-cell communication.  Molecular systems biology, 16(7):e9618, 2020.
  * (29) Nathan C Shaner, Robert E Campbell, Paul A Steinbach, Ben NG Giepmans, Amy E Palmer, and Roger Y Tsien.  Improved monomeric red, orange and yellow fluorescent proteins derived from discosoma sp. red fluorescent protein.  Nature biotechnology, 22(12):1567–1572, 2004.
  * (30) Reshma Shetty, Meagan Lizarazo, Randy Rettberg, and Thomas F Knight.  Assembly of biobrick standard biological parts using three antibiotic assembly.  In Methods in enzymology, volume 498, pages 311–326. Elsevier, 2011\.
  * (31) Junmin Wang, Samuel A Isaacson, and Calin Belta.  Modeling genetic circuit behavior in transiently transfected mammalian cells.  ACS synthetic biology, 8(4):697–707, 2019.
  * (32) Ernst Weber, Carola Engler, Ramona Gruetzner, Stefan Werner, and Sylvestre Marillonnet.  A modular cloning system for standardized assembly of multigene constructs.  PloS one, 6(2):e16765, 2011.
  * (33) Benjamin H Weinberg, Jang Hwan Cho, Yash Agarwal, NT Hang Pham, Leidy D Caraballo, Maciej Walkosz, Charina Ortega, Micaela Trexler, Nathan Tague, Billy Law, et al.  High-performance chemical-and light-inducible recombinases in mammalian cells and mice.  Nature communications, 10(1):1–10, 2019.


# Spectrum: Targeted Training on Signal to Noise Ratio

Eric Hartford  Cognitive Computations  Lucas Atkins  Arcee.AI  Fernando
Fernandes Neto  Cognitive Computations  David Golchinfar  Vago Solutions

###### Abstract

Efficiently post-training large language models remains a challenging task due
to the vast computational resources required. We present Spectrum, a method
that accelerates LLM training by selectively targeting layer modules based on
their signal-to-noise ratio (SNR), and freezing the remaining modules. Our
approach, which utilizes an algorithm to compute module SNRs prior to
training, has shown to effectively match the performance of full fine-tuning
while reducing GPU memory usage. Experiments comparing Spectrum to existing
methods such as QLoRA demonstrate its effectiveness in terms of model quality
and VRAM efficiency in distributed environments.

###### keywords:

Large Language Models, Efficient Training, Signal-to-Noise Ratio, Random
Matrix Theory, Targeted Fine-Tuning

##  1 Introduction

Large language models (LLMs) have showcased impressive abilities across
various natural language tasks. Efficiently training these massive models
remains a major challenge, demanding extensive computational resources and
time. To address this issue, a growing body of research has focused on
developing techniques to minimize the memory footprint and accelerate LLM
training without sacrificing performance.

We present Spectrum, a method for selectively training the layers of an LLM
based on their signal-to-noise ratio (SNR). Rooted in Random Matrix Theory,
Spectrum utilizes the Marchenko-Pastur distribution to pinpoint informative
layers according to their SNR. Unlike previous approaches, such as QLoRA [2],
which quantizes the entire model, Spectrum strategically targets specific
layers and modules for training while keeping others frozen. By concentrating
computational resources on the most informative parameters, Spectrum achieves
superior performance while significantly reducing training time and memory
requirements compared to state-of-the-art methods.

Our main contributions are as follows:

  * •

We propose Spectrum, an approach for efficient LLM training that selectively
trains layers based on their SNR.

  * •

We conduct experiments comparing Spectrum to Full Finetuning and QLoRA on a
range of language benchmarks. Spectrum consistently matches or outperforms
prior methods training faster while using less GPU memory.

  * •

We provide analysis of why Spectrum is so effective, demonstrating that
focusing on high-SNR layers enables more efficient use of training compute.

  * •

We release our code publicly to facilitate future research.

Spectrum opens up exciting opportunities to train large language models cost-
effectively without compromising quality. This has significant implications
for democratizing LLM research and enabling new applications.

##  2 Related Work

Efficient training of large language models has attracted significant
attention. Two notable prior works that aim to reduce the computational cost
and memory requirements of LLM training and deployment are QLoRA [2] and LASER
[5]. Dettmers et al. [2] introduced QLoRA, a method that combines LoRA (Low-
Rank Adaptation) [3] with 4-bit quantization to enable efficient finetuning of
LLMs. By quantizing the base model weights to 4 bits and storing the LoRA
adaptation parameters in 16-bit precision, QLoRA significantly reduces memory
usage during training. This allows finetuning of billion-parameter models on a
single GPU. However, QLoRA applies quantization and LoRA uniformly across all
model layers, which may not be optimal. LASER [5] selectively applies a low-
rank approximation to specific layers of a trained model. The authors measure
the signal-to-noise ratio (SNR) of each layer and reduce the rank of high-SNR
layers, showing that this can boost performance on certain downstream tasks.
LASER is primarily designed for model compression after training rather than
improving training itself.

Spectrum builds upon the insights of QLoRA and LASER while addressing their
limitations. Like QLoRA, Spectrum enables efficient training of language
models. Instead of quantizing all layers, Spectrum selectively trains a subset
of layers in full precision based on their SNR. This allows devoting compute
to the most informative parameters during training. Similar to LASER, Spectrum
uses SNR to identify important layers. But while LASER approximates layers
after training for model compression, Spectrum selects layer modules to train
based on their SNR. Spectrum combines the strengths of QLoRA and LASER -
efficient memory usage and SNR-based layer selection - while improving upon
them to enable faster, better LLM training.

##  3 Mathematical Foundation

Our approach is grounded in Random Matrix Theory (RMT), enabling efficient
identification of the most informative layers for targeted training. We
utilize the Marchenko-Pastur distribution [4], which characterizes eigenvalue
distributions in large random matrices, to distinguish signal from noise in
the network’s weight matrices. In this section some key aspects of
mathematical foundation of the method are provided, such as SVD, Marchenko-
Pastur Distributions and learning representations with infrequent/noisy terms
often lead to singular values close to zero.

Begining with Singular Value Decompositions (SVD), it is important to
elucidate that it enables to assess part of knowledge representation in
matrices. Lower singular values often signify noise, less important
information, or less frequent terms in data. Zeroing these values, as
demonstrated in the LASER paper [5], acts as a denoising process, enhancing
the quality of learned representations. This process, however, must be careful
not to suppress factual knowledge or impact the learning of more complex
patterns, which can lead to catastrophic forgetting.

###  3.1 Illustrating Overfitting’s Impact on Singular Values

Consider a dataset with $n$ data points $(x_{i},y_{i})$. In polynomial
regression of degree $d$, the design matrix $X$ is constructed as:

$\displaystyle X=\begin{bmatrix}1&x_{1}&x_{1}^{2}&\cdots&x_{1}^{d}\\\
1&x_{2}&x_{2}^{2}&\cdots&x_{2}^{d}\\\ \vdots&\vdots&\vdots&&\vdots\\\
1&x_{n}&x_{n}^{2}&\cdots&x_{n}^{d}\end{bmatrix}$

Overfitting occurs when the model is too complex relative to the data,
specifically in polynomial regression when the degree $d$ is high compared to
the number of data points $n$. This results in linear dependency among columns
of $X$, leading to a rank-deficient matrix with zero singular values.

The singular value decomposition (SVD) of $X$ is given by:

$\displaystyle X=U\Sigma V^{T}$

where $U,V$ are orthogonal matrices, and $\Sigma$ is a diagonal matrix of
singular values. Linear dependency in $X$ results in zero singular values in
$\Sigma$, impacting the stability and reliability of regression coefficients
calculated.

A design matrix $X$ with zero singular values leads to issues in solving the
normal equation for polynomial regression, $X^{T}X\beta=X^{T}y$, due to the
non-invertibility of $X^{T}X$. This directly contributes to overfitting,
indicating that the model has fitted the noise instead of the underlying data
pattern.

###  3.2 Random Matrix Theory (RMT) Perspective

RMT provides insights into the nature of data represented by a matrix. For
large matrices from real-world data, the bulk of eigenvalues/singular values
typically forms a “bulk spectrum”. Values associated with less frequent data
points often deviate from this bulk and can be misinterpreted as noise. RMT
helps distinguish signal from noise, but identifying meaningful signals from
deviations requires careful consideration.

###  3.3 Benefits of Focusing on Matrices with Larger Singular Values

Skipping matrices with insignificant singular values has several advantages:
preservation of factual and scattered information from the pre-training phase,
retaining layers with diverse information; training focuses on more stable and
well-posed matrices with larger max-min singular values; targeting matrices
with larger singular values enables us to emphasize transformations with the
largest impact on latent representations.

###  3.4 Relating Eigenvalues and Singular Values

For a weight matrix $W$ in a neural network, the SVD is given by:

$\displaystyle W=USV^{T}$

where $U,V$ are orthogonal, and $S$ is a diagonal matrix of singular values.
The eigenvalues of $W^{T}W$ are related to the squared singular values of $W$:

$\displaystyle W^{T}W=VS^{2}V^{T}$

###  3.5 Marchenko-Pastur Distribution

The Marchenko-Pastur distribution [4] describes the eigenvalue distribution of
large random matrices as dimensions tend to infinity with a fixed aspect
ratio. The main insight from the previous subsection regarding applying the
Marchenko-Pastur distribution is that this is applicable only to square
matrices. Hence, when squaring a rectangular matrix, allows one to obtain a
square matrix, and hence, applying the Marchenko-Pastur distributions for its
eigenvalues. Thus, the relationship between eigenvalues of $W^{T}W$ and
squared singular values of $W$ is central to applying the Marchenko-Pastur
distribution. For a matrix $W$ of size $m\times n$, the eigenvalues of
$C=(1/n)W^{T}W$ converge to a distribution bounded by:

$\displaystyle\lambda_{+}=\sigma^{2}\left(1+\sqrt{\frac{m}{n}}\right)^{2},$
$\displaystyle\lambda_{-}=\sigma^{2}\left(1-\sqrt{\frac{m}{n}}\right)^{2}$

where $\lambda_{+},\lambda_{-}$ are the largest and smallest eigenvalues, and
$\sigma$ is the standard deviation. This leads to bounds on the singular
values of $W$:

$\displaystyle\varepsilon_{+}=\frac{1}{\sqrt{n}}\sigma\left(1+\sqrt{\frac{m}{n}}\right),$
$\displaystyle\varepsilon_{-}=\frac{1}{\sqrt{n}}\sigma\left(1-\sqrt{\frac{m}{n}}\right)$

###  3.6 Signal-to-Noise Ratio and Matrix Ranking

To ensure numerical stability and efficient computation, we omit the
normalization term $(1/\sqrt{n})$ when calculating singular value bounds. We
also use the interquartile range instead of the standard deviation to account
for potential skewness and kurtosis. The signal-to-noise ratio (SNR) of a
weight matrix is defined as:

$\displaystyle
SNR=\frac{\sum_{k|\sigma_{k}>\varepsilon}\sigma_{k}}{\sum_{n|\sigma_{n}<\varepsilon}\sigma_{n}}$

where $\varepsilon$ separates signal from noise singular values. We normalize
$SNR$ by the largest singular value for sensitivity analysis, enhanced
comparison, and conditioning information. Matrices with higher SNR contain
more informative features and less noise, making them ideal targets for
efficient learning and improved model performance.

##  4 Measuring The Signal-to-Noise Ratio

Our Spectrum tool computes layer SNRs. For each layer, Spectrum computes the
SVD of the weight matrix, calculates the SNR, and normalizes it by the largest
singular value. The noise threshold $\varepsilon$ is determined using the
Marchenko-Pastur distribution, effectively separating signal from noise. The
SNR formula aligns with the theoretical understanding of the singular value
spectrum partitioned by the Marchenko-Pastur distribution. Spectrum computes
SNRs in batches, efficiently analyzing the entire model. By leveraging RMT and
the Marchenko-Pastur distribution, Spectrum provides a principled approach to
measuring layer SNRs, forming the basis for selective training.

###  4.1 Layer Selection

Spectrum selects layers with higher SNRs, containing more task-relevant
information, for targeted updates. The number of layers trained is a
hyperparameter balancing training speed and performance. By default, Spectrum
trains the top 25% of layers in each module, ensuring a balanced distribution
of updates.

##  5 Evaluations

We present an evaluation of Spectrum on a set of language model evaluations.
We compare Spectrum to full finetuning and QLoRA [2] in terms of training
speed, memory usage, and benchmark results. We use Spectrum-50 and Spectrum-25
for these evaluations

###  5.1 Setup

We trained five Llama 3 8B111https://huggingface.co/meta-llama/Meta-Llama-3-8B
models using
airoboros-3.1222https://huggingface.co/datasets/jondurbin/airoboros-3.1 as the
dataset. The models were trained as such: one with a full finetune, another
with QLoRA, one targeting the top 50% SNR layers with Spectrum, and another
targeting the top 25% We include a fifth model trained with QLoRA that targets
the top 25% layer modules identified by Spectrum. We chose airoboros to
establish a baseline due to its relatively small size but large number of
general language understanding tasks.

We run the same training procedure (excluding QLoRA + Spectrum) on Mistral
7B333https://huggingface.co/mistralai/Mistral-7B-v0.1. Each model was
initialized from the base models released by Meta and Mistral. All models were
trained for two epochs with the same hyperparameters: a learning rate of 1e-5,
gradient norm of 4, batch size of 1, and a maximum sequence length of 4096.
The only change made was during QLoRA training where a lower learning rate of
2e-4 was used.

All experiments were conducted on an 8xL40S (46GB VRAM per GPU) node provided
by Crusoe Energy. We used Hugging Face Accelerate and Axolotl with DeepSpeed
ZeRO-3 [1] for distributed training. To compare performance on single GPU
jobs, we retrained our llama-3-8b models on a 1x Nvidia L40S GPU for a single
epoch to assess single GPU VRAM usage. Our QLoRA hyperparameters within
Axolotl were as follows:

    adapter: qlora
    lora_r: 32
    lora_alpha: 16
    lora_dropout: 0.05
    lora_target_linear: true
    lora_modules_to_save: [embed_tokens, lm_head]

###  5.2 Benchmark Scores

Figure 1:  Figure 2:  Figure 3:

####  5.2.1 Memory Usage & Training Time

We measured the memory usage and training time for different training
configurations. Models are compared against baseline FFT training in terms of
peak memory usage per GPU, VRAM usage on a single GPU, and total training
time.

Table 1: Distributed Training

Model | Peak Memory Usage per GPU | % Efficiency Compared to FFT
---|---|---
Llama-3-8b-FFT | 24.92 GB | Baseline
Llama-3-8b-QLoRA | 21.25 GB | 14.73%
Llama-3-8b-Spectrum-50 | 20.50 GB | 17.72%
Llama-3-8b-Spectrum-25 | 19.18 GB | 23.05%
Llama-3-8b-Spectrum-25+QLoRA | 16.95 GB | 31.99%

Model | Single GPU VRAM Usage
---|---
Llama-3-8b-FFT | N/A (out of memory)
Llama-3-8b-Spectrum-50 | 34.65 GB
Llama-3-8b-Spectrum-25 | 27.46 GB
Llama-3-8b-QLoRA | 23.39 GB
Llama-3-8b-Spectrum-25+QLoRA | 21.18 GB

Table 2: Single GPU VRAM Usage

Model | Training Time (8xL40S)
---|---
Llama-3-8b-FFT | 1h 43m 16s
Llama-3-8b-Spectrum-50 | 1h 27m 17s
Llama-3-8b-QLoRA | 1h 18m 14s
Llama-3-8b-Spectrum-25 | 1h 5m 33s
Llama-3-8b-Spectrum-25+QLoRA | 54m 55s

Table 3: Training Time

###  5.3 Analysis

In our tests, Spectrum not only competes with fully fine-tuned models but
also, in some cases, outperforms them in terms of benchmark scores.
Additionally, Spectrum surpasses QLoRA across all metrics except for single
GPU VRAM usage. Spectrum-50, which targets the top 50% of layers, matches or
surpasses full finetuning in various benchmarks. This indicates that focusing
on high SNR layers is sufficient for competitive performance. Spectrum-25,
targeting the top 25% of layers, also shows strong results, at times scoring
higher than full finetuning and Spectrum-50. This suggests that critical post-
training information resides in a small subset of layers. Our experiments with
Spectrum-25+QLoRA showed promising evaluation results compared to QLoRA alone,
with significant reductions in VRAM use.

Figure 4:

####  5.3.1 Memory Efficiency

The efficiency of Spectrum is particularly evident when comparing it to QLoRA
in distributed training settings using DeepSpeed ZeRO-3. Spectrum-50 and
Spectrum-25 achieve 17.72% and 23.05% memory savings per GPU, respectively,
compared to full finetuning. In contrast, QLoRA offers 14.73% memory savings
in distributed workloads.

QLoRA exhibits better memory efficiency than Spectrum when training on a
single GPU. In the single GPU setting, most of the VRAM is used to load the
model, reducing Spectrum’s relevance. The efficiency gains for Spectrum come
from only updating the gradients for the selected modules, which uses
comparatively less memory than updating the entire model. This also explains
why it is so much more efficient in distributed environments like DeepSpeed
ZeRO-3 and Fully Sharded Data Parallel (FSDP) [6], as these allow us to train
the model in a sharded manner across multiple GPUs, making the per GPU model
memory footprint much lower. Spectrum+QLoRA had the lowest VRAM requirement on
both single and distributed workloads.

####  5.3.2 Training Time

In terms of training time, Spectrum demonstrates significant improvements over
full finetuning and QLoRA (Table 3). Spectrum-50 and Spectrum-25 achieve
15.48% and 36.78% reductions in training time, respectively, compared to full
finetuning. QLoRA also offers a 24.19% reduction in training time. It should
be noted that as the size of models grows, the time it takes to train using
LoRA often increases.

##  6 Real World Use

Spectrum has been used to fine-tune many Dolphin models. Fine-tuning large
models like Qwen1.5-110B444https://huggingface.co/Qwen/Qwen1.5-110B on an
8xH100 node (640 GB) encounters VRAM limitations. By using Spectrum-45, we
were able to decrease the VRAM usage to 710 GB, fitting it within a single
node while offloading some parameters to the CPU.

##  7 Conclusion and Future Work

We introduced Spectrum, a method for efficient training of LLMs. Spectrum
selectively trains a subset of layers based on their signal-to-noise ratio
(SNR), allowing it to focus compute on the most informative parameters.
Spectrum uses significantly less memory in distributed workloads than prior
methods while maintaining model quality.

Through evaluations on a set of language modeling benchmarks, we demonstrated
Spectrum’s performance compared to full finetuning and QLoRA. Spectrum opens
up several promising directions for future work on efficient language model
training. Spectrum’s ability to identify the most informative layers can be
further leveraged to enhance training efficiency. By employing layer-wise
learning rate scheduling based on the signal-to-noise ratio (SNR), we can
assign higher learning rates to the most informative layers. Additionally,
dynamically rescanning the model between epochs and adjusting the targeting
layers can further improve convergence speed. These techniques, currently
under experimentation, aim to optimize the training process and accelerate
model convergence.

The versatility of Spectrum extends beyond language modeling. Its ability to
efficiently adapt pretrained models makes it well-suited for domain adaptation
and transfer learning tasks. Furthermore, while our current work focuses on
relatively small models, we have successfully applied Spectrum to train large
language models with hundreds of billions of parameters. Exploring even larger
models and datasets, as well as adapting Spectrum to other modalities offers
exciting opportunities to broaden the impact of our work. We are excited about
Spectrum’s potential, and look forward to seeing how the community builds upon
this foundation.

Spectrum will be made publicly available to facilitate adoption and future
research555https://github.com/cognitivecomputations/spectrum.

###  7.1 Acknowledgements

We would like to thank Crusoe Energy for providing the compute resources for
our experiments. Special thanks to the teams working on Hugging Face
Accelerate and OpenAccess AI Collective’s Axolotl for their invaluable tools
that made our research possible. We would also like to extend our heartfelt
thanks to Maxime Labonne for being an early reviewer and for his honest and
gracious feedback. Finally, we express our gratitude to the open-source
community for their continuous contributions to democratizing access to
machine learning tools.

## Appendix

We evaluated each model using lm-evaluation-harness commit 00b7a61 on multiple
language modeling benchmarks popularized by the OpenLLM Leaderboard:

  * •

Arc-Easy: The AI2 Reasoning Challenge (ARC) Easy Set consists of 7,787 genuine
grade-school level, multiple-choice science questions. The Easy Set includes
questions that are straightforward for both humans and basic algorithms to
answer correctly.

  * •

GSM8K: (Grade School Math 8K) is a dataset of 8,500 high-quality,
linguistically diverse grade school math word problems. These problems require
multi-step reasoning and involve basic arithmetic operations.

  * •

HellaSwag: Designed to evaluate commonsense natural language inference (NLI).
It includes context completion tasks where models must choose the correct
ending from multiple options, with adversarially generated incorrect endings.

  * •

MMLU: The Massive Multitask Language Understanding (MMLU) benchmark consists
of multiple-choice questions across 57 subjects, including STEM, humanities,
social sciences, and more. It tests both world knowledge and problem-solving
ability.

  * •

TruthfulQA: A benchmark designed to measure the truthfulness of language
models in generating answers to questions. It includes 817 questions across 38
categories, targeting common misconceptions and false beliefs.

  * •

Winogrande: A dataset for evaluating commonsense reasoning, consisting of
sentence pairs with a pronoun that needs to be resolved. The dataset is
designed to be more challenging than the original Winograd Schema Challenge.

To further validate our results, we ran our Llama models on another suite of
benchmarks popularized by Nous Research:

  * •

AGIEval: A benchmark designed to evaluate the general intelligence of AI
models. It includes a variety of tasks that test different aspects of
reasoning, problem-solving, and knowledge application.

  * •

BigBench-Hard: A subset of the BIG-Bench dataset, consisting of 23
particularly challenging tasks for current language models. These tasks
require complex reasoning and understanding, often involving multi-step
processes.

  * •

GPT4All: A collection of benchmarks including BoolQ, PIQA, HellaSwag,
WinoGrande, ARC-easy, ARC-c, and OBQA. (Note, there is some overlap here
between these benchmarks and the OpenLLM Leaderboard tasks performed above.)

We used the chatml prompt template for each finetune, thus adding the
<im_start> and <im_end> tokens to each model’s vocabulary. This affects memory
usage. By adding the same tokens to every finetune, we believe the comparisons
in memory efficiency to be representative of what they would be without adding
any additional tokens.

Additional Dolphin Models trained using Spectrum are referenced here.
666https://huggingface.co/cognitivecomputations/dolphin-2.9.2-mixtral-8x22b777https://huggingface.co/cognitivecomputations/dolphin-2.9.1-qwen-110b888https://huggingface.co/cognitivecomputations/dolphin-2.9.1-dbrx999https://huggingface.co/cognitivecomputations/dolphin-2.9.1-llama-3-70b

## References

  * [1] Aminabadi, R. Y., Rajbhandari, S., Zhang, M., Awan, A. A., Li, C., Li, D., Zheng, E., Rasley, J., Smith, S., Ruwase, O., and He, Y. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale, 2022.
  * [2] Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms, 2023.
  * [3] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021.
  * [4] Marčenko, V. A., and Pastur, L. A. Distribution of eigenvalues for some sets of random matrices.  Mathematics of the USSR-Sbornik 1, 4 (apr 1967), 457.
  * [5] Sharma, P., Ash, J. T., and Misra, D. The truth is in there: Improving reasoning in language models with layer-selective rank reduction, 2023.
  * [6] Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., Desmaison, A., Balioglu, C., Damania, P., Nguyen, B., Chauhan, G., Hao, Y., Mathews, A., and Li, S. Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023.


SpaceTimeLab, University College London (UCL), UK
ucexwg@ucl.ac.ukhttps://orcid.org/0000-0003-3273-7499SpaceTimeLab, University
College London (UCL), UK<EMAIL_ADDRESS>Department of Urban Studies and
Planning, Massachusetts Institute of Technology (MIT), USA
dingyi@mit.eduhttps://orcid.org/0000-0003-3208-6016 The Bartlett Centre for
Advanced Spatial Analysis, University College London (UCL), UK
huanfa.chen@ucl.ac.ukhttps://orcid.org/0000-0002-4518-7601 School of Computer
Science, Peking University (PKU<EMAIL_ADDRESS>Xiaowei Gao,
et al. CC-BY [500]Information systems Geographic information systems John Q.
Open and Joan R. Access 2 42nd Conference on Very Important Topics (CVIT 2016)
CVIT 2016 CVIT 2016 December 24–27, 2016 Little Whinging, United Kingdom 42 23

# Uncertainty Quantification in the Road-level Traffic Risk Prediction by
Spatial-Temporal Zero-Inflated Negative Binomial Graph Neural Network(STZINB-
GNN)

Xiaowei Gao  James Haworth  Dingyi Zhuang  Huanfa Chen  Xinke
Jiang111Corresponding Author

###### Abstract

Urban road-based risk prediction is a crucial yet challenging aspect of
research in transportation safety. While most existing studies emphasize
accurate prediction, they often overlook the importance of model uncertainty.
In this paper, we introduce a novel Spatial-Temporal Zero-Inflated Negative
Binomial Graph Neural Network (STZINB-GNN) for road-level traffic risk
prediction, with a focus on uncertainty quantification. Our case study,
conducted in the Lambeth borough of London, UK, demonstrates the superior
performance of our approach in comparison to existing methods. Although the
negative binomial distribution may not be the most suitable choice for
handling real, non-binary risk levels, our work lays a solid foundation for
future research exploring alternative distribution models or techniques.
Ultimately, the STZINB-GNN contributes to enhanced transportation safety and
data-driven decision-making in urban planning by providing a more accurate and
reliable framework for road-level traffic risk prediction and uncertainty
quantification.

###### category:

short-papers

###### keywords:

Traffic Risk Prediction, Uncertainty Quantification, Zero-Inflated Issues,
Road Safety

##  1 Introduction

In recent years, the field of traffic risk prediction has attracted
considerable attention from researchers and policymakers, driven by the need
to create resilient urban traffic systems and enhance their reliability in
response to mounting challenges such as minimizing urban congestion, improving
road safety, and making informed investments in urban infrastructure.
Additionally, the zero-inflated nature of accident data, characterized by
sparse incidents, poses a substantial challenge to prediction efforts.

Deep learning models have emerged as promising tools in this domain,
incorporating multivariate spatiotemporal information and utilizing point-
processing estimation during training to forecast traffic accidents or risk
situations in subsequent time series [2, 12]. For instance, de Medrano et al.
[1] proposed a novel SpatioTemporal Neural Network (STNN) framework based on
Recurrent Neural Network (RNN) to predict the number of accidents in each
region of Madrid, Spain using a 5-hour prediction window. Their results show a
more accurate prediction than the traditional linear statistics models as well
as machine learning methods. Ren et al. [5] also employed RNN to analyze
spatial and temporal patterns of traffic accident frequency and predict grid-
level daily risk situations. However, the RNN model is limited by its focus on
short-term temporal embedding information and its inability to fully capture
the spatial heterogeneity of traffic accidents. Furthermore, Najjar et al. [3]
employed Convolutional Neural Networks (CNNs) to combine urban information
from satellite imagery and open traffic accident data, mapping city-wide risk
situations. Despite this, their approach neglected temporal information and
faced limitations due to the quality of street image data.

Recognizing the potential of graph neural networks (GNNs) to account for the
natural connection of spatial units, researchers have proposed graph-based
models for traffic risk forecasting. Zhang et al. [8] employed a multi-modal
approach to jointly consider text data from social media and imagery data from
satellites, subsequently mapping grid-level traffic accident frequency using
GNNs. Zhou et al. [9] introduced a novel Differential Time-varying Graph
Convolution Network (GCN) to dynamically capture traffic variations and inter-
subregion correlations, also predicting grid-level traffic risk. After that,
they further refined their algorithms to account for hierarchical spatial
dependencies, allowing for the mapping of finer grid-level urban traffic risk
[10]. While their work addressed the zero-inflation problem of sparse accident
data, their models sidestepped this challenge with a grid level and still
faced limitations when predicting at the road level, which is a much finer
micro-level unit compared.

Despite the valuable foundation provided by predicted average grid-based risk
levels, a significant concern in understanding traffic risk prediction is the
quantification of uncertainty by considering distribution rather than mean
values [11, 4, 12, 7]. Uncertainty is pervasive in urban mobility systems and
plays a crucial role in accounting for potential variations in prediction
results, which may arise from the aleatoric uncertainty of imbalanced risk
data or the epistemic uncertainty of black-box prediction models [2]. Qian et
al. [4] explored uncertainty quantification in traffic forecasting by training
a graph-based deep learning model to fit aleatoric uncertainty and combining
Monte Carlo dropout with Adaptive Weight Averaging re-training methods to
estimate epistemic uncertainty. Zhou et al. [11] considered the irregular
patterns in human mobility data as aleatoric uncertainty and the average
potential variations in predicted results among specific and neighbouring
regions as epistemic uncertainty. By recognizing the reducible nature of
predicted epistemic uncertainty, they improved prediction performance through
a gated mechanism to calibrate the predicted mobility results. Although those
two approaches demonstrated the potential of combining variational inference
and deep spatial-temporal embedding for predicting various distributions, they
did not thoroughly investigate the sparse traffic data scenario. Zhuang et al.
[12] and Wang et al. [6] highlighted the importance of considering zero-
inflated distribution statistical models for analyzing sparse traffic demand
data. These models offer a more accurate spatiotemporal representation of the
underlying uncertainty structure suitable for incorporation with deep learning
models.

Despite the progress in the field of uncertainty quantification in urban
traffic research, the current focus predominantly lies on traffic demands and
human mobility. This research mostly utilizes sequence or time-series data and
employs coarser resolution prediction approaches, such as grid-based analysis.
This also leaves a notable research gap in terms of an investigation into
finer resolution models, particularly from a road safety perspective. Road
safety prediction involves non-stable and event-based characteristics, which
deviate from the usual time-series data analysis. Expanding research in this
area could provide a more nuanced understanding of traffic risk prediction and
its significant potential for improving urban transportation safety and
resilience. Filling this research gap will contribute to the holistic
development of urban transportation studies, enhancing not only predictive
accuracy but also the applicability of results in practical, real-world
scenarios.

Building upon the work of Zhuang et al. [12], this paper presents the Spatial-
Temporal Zero-Inflated Negative Binomial Graph Neural Network (STZINB-GNN)
model, specifically designed to tackle the existing limitations in traffic
risk prediction and uncertainty quantification.

The zero-inflated negative binomial model is employed to effectively
distinguish between non-risk and risk levels across road segments.

The spatial-temporal Graph Neural Network (ST-GNN) is responsible for learning
and fitting the parameters of probabilistic distributions.

To the best of our knowledge, this is the first attempt to merge these two
elements for road-level risk estimation. Empirical evidence showcases the
enhanced performance of our proposed model when applied to road-daily
resolution traffic risk data.

This paper is structured as follows: Section 2 describes the development of
the model and provides detailed explanations of its components. Section 3
presents the dataset employed for the case study, outlines the evaluation
metrics, and discusses the experimental results. Finally, Section 4 offers
conclusions and highlights potential avenues for future research.

##  2 Methdology

Our objective is to predict future traffic risk and associated confidence
intervals for each individual road segment across $k$ forthcoming time
intervals, using $m$ roads’ risk conditions from previous time windows lasting
$T$ days. This is a sequence-to-sequence prediction problem, as illustrated in
Figure 1. We construct the road graph, $\mathcal{G}=(V,E)$, where $V$
represents the set of roads, and $E$ denotes the edges connecting these roads.
The adjacency matrix $A\in\mathbb{R}^{V\times V}$ indicates the relationships
between roads, and $|V|=m\times m$. More specifically, $x_{it}$ signifies the
risk level of the $i^{th}$ road segment during the $t^{th}$ time interval,
where $i\in V$, $x_{it}\in\mathbb{N}$. Subsequently,
$X_{t}\in\mathbb{N}^{|V|\times T}$ designates the risk conditions for all road
segments within the $t^{th}$ time interval, with $x_{it}$ as its component.
Our aim is to employ historical records $X_{1:t}$ and spatial-temporal
features as input data to estimate the distribution of predicted $X_{t:t+k}$
(i.e., the risk levels for each road over the next $k$ time intervals), thus
examining the anticipated value and confidence intervals of the future risk
situation.

Figure 1: Framework of STZINB-GNN model.

First, we assume that the inputs follow the ZINB distribution with probability
mass function as:

$f_{ZINB}(x_{k};\pi,n,p)=\left\\{\begin{array}[]{cc}\pi+(1-\pi)\left(\begin{array}[]{c}x_{k}+n-1\\\
n-1\end{array}\right)(1-p)^{x_{k}}p^{n}&\text{if }x_{k}=0\\\
(1-\pi)\left(\begin{array}[]{c}x_{k}+n-1\\\
n-1\end{array}\right)(1-p)^{x_{k}}p^{n}&\text{if }x_{k}>0\end{array}\right.,$
(1)

where $\pi$ is the inflation of zeros, $n$ and $p$ determine the number of
successes and the probability of a single failure respectively. $x_{k}$
denotes a road’s traffic risk level at one time. All three parameters
$\pi,n,p$ are learned by spatial-temporal GNNs (STGNN), where the temporal
encoder applies a Gated Recurrent Unit (GRU) and then the spatial encoder
applies Graph Attention Network (GAT):

$n_{t+1:t+k},p_{t+1:t+k},\pi_{t+1:t+k}=STGNN(X_{1:t};F_{1:t};A)=GAT\bigl{(}GRU(X_{1:t};F_{1:t});A\bigl{)}.$
(2)

Here, $F_{t}$ represents the spatiotemporal features of roads on the $t^{th}$
day, while $X_{t}$ corresponds to the risk level observed on the same day.
This relationship illustrates the connection between road features and risk
levels at specific roads in time.

The log-likelihood of ZINB is composed of the $y=0$ and $y>0$ parts, and can
be approximated as follows:

$LL_{y}=\left\\{\begin{array}[]{lccc}\log\pi+\log(1-\pi)p^{n}&\text{when}&y=0&\\\\[6.0pt]
\log(1-\pi)+\log\Gamma(n+y)-\log\Gamma(y+1)&&&\\\ -\log\Gamma(n)+n\log
p+y\log(1-\pi)&\text{when}&y>0&\end{array}\right.,$ (3)

where $y$ is the ground-truth value, $\Gamma$ is the Gamma function and
$n,p,\pi$ is learned by STGNN. The final negative log-likelihood loss function
is given by:

$NLL_{STZINB}=-LL_{y=0}-LL_{y>0}.$ (4)

##  3 Result

We evaluated the model’s performance using a real-world dataset from Lambeth
Borough in London, UK. This dataset comprises 5,659 road segments and a total
of 1,335 accidents throughout 2019222https://roadtraffic.dft.gov.uk/downloads.
We calculated a daily risk level by combining the number of accidents with the
severity of each accident. We then identified the nearest road segment and
accounted for the spillover effects on first and second-order neighbouring
roads [9] to assign each road segment a risk value for each day. Notably, the
zero-inflation rate for road-level accident risk in Lambeth Borough is 98.7%,
indicating that a significant proportion of the road segments experienced no
accidents during the observed period.

The evaluation metrics for assessing the model performance are categorized
into four aspects. Traditional accuracy measures, including Mean Absolute
Error (MAE), Mean Absolute Percentage Error (MAPE), and Root Mean Squared
Error (RMSE), evaluate the model’s ability to accurately predict risk values.
Uncertainty quantification is assessed using the Kullback-Leibler Divergence
(KLD), which indicates how closely the distribution of the model’s output risk
values approximates the distribution of the true risk values. Lower values for
these above metrics are desirable, as they signify a smaller difference
between the predicted and actual risk values as well as distributions.

The true-zero rate (ZR) quantifies the model’s capacity to accurately
replicate the sparsity observed in the ground-truth data. Additionally, the
Hit Rate (HR) is assessed based on information entropy. To compute HR, we
first select the top 20% of road segments with the highest predicted risk
values and then consider the predicted risk values’ information entropy, which
is derived from the KLD uncertainty quantification. We calculate the HR by
selecting those road segments with lower predicted uncertainty among the top
20% risk roads, where the road information entropy is below the mean value of
the entire road’s entropy.

Higher ZR and HR suggest better model performance in identifying road segments
with no accidents and those with accidents, respectively.

Results | Long(14-14) | Short(7-7)
---|---|---
Metrics | Models | STZINB | STG | STNB | HA | STZINB | STG | STNB | HA
ACC | MAE | 0.054 | 0.118 | 0.080 | 0.135 | 0.077 | 0.048 | 0.105 | 0.134
MAPE | 0.026 | 0.405 | 0.036 | 0.414 | 0.025 | 0.443 | 0.078 | 0.485
RMSE | 0.119 | 0.183 | 0.139 | 0.211 | 0.139 | 0.185 | 0.172 | 0.238
Uncertainty | KLD | 0.259 | 0.504 | 1.558 | 0.269 | 0.473 | 0.522 | 0.759 | 0.264
Zero Inflated | ZR | 0.641 | 0.199 | 0.562 | 0.520 | 0.579 | 0.102 | 0.518 | 0.503
Hit Rate | HR20% | 0.618 | 0.267 | 0.447 | 0.452 | 0.575 | 0.301 | 0.442 | 0.443

Table 1: Model Results for the Lambeth Borough, London

In the table, bold fonts mean the best values among all the baseline models
while the underline means the second-best values. The baseline models used for
comparison include Spatial-temporal Graph Neural Network with Gaussian
Distribution (STG), Spatial-temporal Graph Neural Network with Negative
Binomial Distribution (STNB), and Historical Average (HA). It is evident that
our proposed model outperforms the baseline models across all evaluation
metrics, with the exception of KLD in short-term prediction scenarios, where
it ranks second. This demonstrates the effectiveness of our model in capturing
the skewed data distribution through its zero-inflated components, which leads
to more accurate results and improved reliability when approximating the true
risk distribution. Notice that both ZR and HR20% metrics, which are important
to measure the occurrence of events in practice, have received significant
accuracy improvement. This is due to introducing the parameter $\pi$ in
Equation 1, which can effectively learn the sparsity of the data.

Furthermore, our model’s capability to reliably predict higher risk values
enables us to achieve an accuracy of approximately 61.8% or 57.5% in
identifying the exact locations of accidents, which also significantly
outperforms the other GNN counterparts. This highlights the potential of our
approach to significantly enhance transportation safety and facilitate data-
driven decision-making in urban planning.

##  4 Discussion and Conclusion

In this study, we developed a versatile spatial-temporal Graph Neural Network
(GNN) framework for predicting the probabilistic distribution of sparse road
traffic risk and quantifying its associated uncertainty. By employing Gated
Recurrent Units (GRUs) to capture temporal correlations and Graph Attention
Networks (GATs) to model spatial dependencies, we created a comprehensive
framework that embeds the spatial and temporal representations of distribution
parameters. These parameters are then fused to obtain a distribution for each
spatial-temporal data point.

Our case study, based on the urban risk situation of Lambeth borough in the
UK, demonstrated that the proposed model consistently delivers more accurate
and reliable results compared to other methods. Despite its performance, the
model also has certain limitations. When addressing real risk levels that
extend beyond binary values, the negative binomial distribution may not be the
most suitable choice. Future work could explore alternative distribution
models or techniques that better capture the complexity and nuances of real-
world risk levels. This would further enhance the model’s applicability and
predictive capabilities, ultimately contributing to improved transportation
safety and data-driven decision-making in urban planning.

## References

  * [1] Rodrigo de Medrano and José L Aznarte.  A new spatio-temporal neural network approach for traffic accident forecasting.  Applied Artificial Intelligence, 35(10):782–801, 2021.
  * [2] Genwang Liu, Haolin Jin, Jiaze Li, Xianbiao Hu, and Jian Li.  A bayesian deep learning method for freeway incident detection with uncertainty quantification.  Accident Analysis & Prevention, 176:106796, 2022.
  * [3] Alameen Najjar, Shun’ichi Kaneko, and Yoshikazu Miyanaga.  Combining satellite imagery and open data to map road safety.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
  * [4] Weizhu Qian, Dalin Zhang, Yan Zhao, Kai Zheng, and James JQ Yu.  Uncertainty quantification for traffic forecasting: A unified approach.  arXiv preprint arXiv:2208.05875, 2022.
  * [5] Honglei Ren, You Song, Jingwen Wang, Yucheng Hu, and Jinzhi Lei.  A deep learning approach to the citywide traffic accident risk prediction.  In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pages 3346–3351. IEEE, 2018.
  * [6] Qingyi Wang, Shenhao Wang, Dingyi Zhuang, Haris Koutsopoulos, and Jinhua Zhao.  Uncertainty quantification of spatiotemporal travel demand with probabilistic graph neural networks.  arXiv preprint arXiv:2303.04040, 2023.
  * [7] Yuankai Wu, Dingyi Zhuang, Aurelie Labbe, and Lijun Sun.  Inductive graph neural networks for spatiotemporal kriging.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4478–4485, 2021.
  * [8] Yang Zhang, Xiangyu Dong, Lanyu Shang, Daniel Zhang, and Dong Wang.  A multi-modal graph neural network approach to traffic risk forecasting in smart urban sensing.  In 2020 17th Annual IEEE international conference on sensing, communication, and networking (SECON), pages 1–9. IEEE, 2020.
  * [9] Zhengyang Zhou, Yang Wang, Xike Xie, Lianliang Chen, and Hengchang Liu.  Riskoracle: a minute-level citywide traffic accident forecasting framework.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1258–1265, 2020.
  * [10] Zhengyang Zhou, Yang Wang, Xike Xie, Lianliang Chen, and Chaochao Zhu.  Foresee urban sparse traffic accidents: A spatiotemporal multi-granularity perspective.  IEEE Transactions on Knowledge and Data Engineering, 34(8):3786–3799, 2020.
  * [11] Zhengyang Zhou, Yang Wang, Xike Xie, Lei Qiao, and Yuantao Li.  Stuanet: Understanding uncertainty in spatiotemporal collective human mobility.  In Proceedings of the Web Conference 2021, pages 1868–1879, 2021\.
  * [12] Dingyi Zhuang, Shenhao Wang, Haris Koutsopoulos, and Jinhua Zhao.  Uncertainty quantification of sparse travel demand prediction with spatial-temporal graph neural networks.  In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4639–4647, 2022.