11institutetext: Embedded Vision Systems Group, Department of Automatic
Control and Robotics, AGH University of Krakow, Poland 11email:
<EMAIL_ADDRESS>
22institutetext: Sorbonne Université, CNRS, LIP6, F-75005 Paris, France
22email<EMAIL_ADDRESS>
# Optimising Graph Representation for Hardware Implementation of Graph
Convolutional Networks for Event-based Vision ††thanks: The work presented in
this paper was supported by: the AGH University of Krakow project no.
$16.16.120.773$, the program ”Excellence initiative –- research university”
for the AGH University of Krakow, Polish high-performance computing
infrastructure PLGrid (HPC Centers: ACK Cyfronet AGH) – grant no.
$PLG/2023/016130$ and partly by Sorbonne University. Tomasz Kryjak would like
to sincerely thank Sorbonne Université for inviting him and funding his stay
as a visiting professor in November 2022.

Kamil Jeziorek  11 Piotr Wzorek  11 Krzysztof Blachut  11 Andrea Pinna  22
Tomasz Kryjak  1122

###### Abstract

Event-based vision is an emerging research field involving processing data
generated by Dynamic Vision Sensors (neuromorphic cameras). One of the latest
proposals in this area are Graph Convolutional Networks (GCNs), which allow to
process events in its original sparse form while maintaining high detection
and classification performance. In this paper, we present the hardware
implementation of a graph generation process from an event camera data stream,
taking into account both the advantages and limitations of FPGAs. We propose
various ways to simplify the graph representation and use scaling and
quantisation of values. We consider both undirected and directed graphs that
enable the use of PointNet convolution. The results obtained show that by
appropriately modifying the graph representation, it is possible to create a
hardware module for graph generation. Moreover, the proposed modifications
have no significant impact on object detection performance, only 0.08% mAP
less for the base model and the N-Caltech data set. Finally, we describe the
proposed hardware architecture of the graph generation module.

###### Keywords:

graph representation GCN event cameras object detection FPGA

##  1 Introduction

Event cameras, also known as neuromorphic cameras, are an emerging type of a
video sensor. Unlike standard frame cameras, which read out the brightness
values of all pixels simultaneously in a fixed time interval, event cameras
record the brightness changes of individual pixels independently. This allows
correct operation in challenging lightning conditions, such as poor
illumination and high dynamic range, and reduces average power consumption. In
addition, information on pixel brightness changes is obtained with a much
higher temporal resolution – in the order of microseconds vs 16 millisecond
for a typical 60 frame per second frame camera.

One promising application of event cameras is the detection and tracking of
moving objects, such as vehicles and pedestrians. The use of neuromorphic
sensors can significantly improve safety at key moments. However, due to their
asynchronous nature of operation, event cameras generate an irregular data
stream in the form of a sparse spatio-temporal event cloud, which is difficult
to process. The most commonly proposed solution is to create pseudo-frames
that mimic traditional video frames or to use event-based video frame
reconstruction. Both approaches produce good results, but lead to the loss of
the advantages of high temporal resolution.

Therefore, recent research has focused on processing event data in its
original sparse form. An example is the use of spiking neural networks (SNNs)
[3]. Inspired by biology, it is a promising approach but suffers from a lack
of standardised learning algorithms, making it difficult to apply or achieve
high performance in more complex methods. Another recently proposed solution
is the use of Graph Convolutional Networks (GCNs), which allow event data to
be processed in the form of a spatio-temporal point cloud. In addition, recent
work in this area suggests that such representations can be updated
asynchronously [5], [7], [10]. However, current computations are mainly
performed using high-energy GPUs, and thus similar architectures cannot be
used in low-power embedded vision systems. Furthermore, to the best of our
knowledge, current research in data processing with GCNs relies primarily on
pregenerated graphs or specifies their construction for a pre-established
structure.

In this paper, we present the hardware implementation of a graph generation
module for event data on SoC FPGAs (System-on-Chip Field Programmable Gate
Arrays). Our primary contributions include the development of a graph
representation that takes into account the architectural strengths and
limitations of SoC FPGAs, leading to a reduction in the number of graph edges
while still achieving detection and classification results comparable to
recent methods. Additionally, we introduce the concept of an FPGA hardware
module for graph representation generation, which produces a representation
ready for processing by the selected PointNet convolution layer [9].

The remainder of the paper is organised as follows: Section 2 presents the
issue of object detection using graph convolutional networks. Section 3
describes the methods we have considered to optimise the graph representation
of event data. In Section 4 we present the concept of hardware-based graph
generation on FPGAs. The article concludes with a summary 5 of the results
obtained and a presentation of plans for further work.

##  2 Object detection using GCN

###  2.1 Graph construction

For an event camera, the event generation process is controlled by a threshold
mechanism. An event is generated when the logarithmic change in light
intensity ($\Delta L$) at a particular pixel ($x_{i}$, $y_{i}$) and time
$t_{i}$ exceeds a fixed threshold value ($C$) for the same pixel at time
${t_{i}}^{*}$: $\Delta
L(x_{i},y_{i},t_{i})=L(x_{i},y_{i},t_{i})-L(x_{i},y_{i},t_{i}^{*})\geq
p_{i}C$.

Each generated event includes: the coordinates of the pixel, the time received
from the internal camera clock, and the polarisation $p_{i}$ in the range
$\\{-1,1\\}$, specifying the direction of the change in light intensity. These
four values form a tuple $e_{i}=\\{x_{i},y_{i},t_{i},p_{i}\\}$, and the events
together form a data sequence $E=\\{e_{i}\\}_{i=0}^{N-1}$ ($N$ – number of
events).

The graph structure $\mathcal{G}$ consists of a set of vertices $\mathcal{V}$
and a set of edges $\mathcal{E}$ that reflect the relationships between
vertices. A commonly accepted method for event-based graph generation is to
treat each event as a vertex $v$ with spatio-temporal position $pos=(x,y,t)$
and attribute $a=(p)$. Edges are created based on a chosen distance metric
between vertex positions, most commonly using an Euclidean distance bounded by
a radius of length $R$. A full description of the graph representations used
is given in Section 3.

###  2.2 Graph Convolutional Networks

####  2.2.1 Graph convolution.

Currently, there are many different convolution operations on a graph,
differing in the information used, the functions and the type of data
processed. Recent work in this area apply operations such as SplineConv or
GCNConv. However, the study [6] showed that the simpler and memory-efficient
PointNetConv operation [9] can provide comparable results in event data
detection and classification problems. We therefore decided to use
PointNetConv and model presented in [6] as a reference in our work, as it is a
better candidate for embedded implementation.

The graph convolution operation for a vertex $v_{i}$ in a graph $\mathcal{G}$
consists of three steps: a message function that utilises the attributes of a
vertex $a_{i}$, as well as those of its neighbours $a_{j}$ and the difference
in vertex positions $pos_{j}-pos_{i}$, which are transformed by the linear
function $h_{\Theta}$. The aggregation function then selects the maximum value
among all messages for a given vertex $v_{i}$. Finally, the update function
applies an additional linear function $h_{\Theta}$ to the output of the
aggregation function to adjust the output size.

These steps are performed for each vertex in the graph $\mathcal{G}$, and at
each successive convolution layer, the attribute dimension $a_{i}$ of the
vertices is modified, with the spatio-temporal position unchanged.

####  2.2.2 Graph pooling.

The pooling process, relevant in the context of GCNs, involves the selection
of representative vertices or the aggregation of information within a group or
cluster. Selection criteria may include the relevance of the vertex, the
maximum or average value for all vertices in the group. The result of this
process is a new vertex that represents the entire group.

In the context of GCNs, the pooling process enables the creation of denser
representations, which translates into a reduction of computational complexity
in subsequent layers. This encourages more efficient processing of large-scale
data, while preserving essential features and smooth information flow. It is
worth noting that this operation also enables the use of fully connected
layers data classification.

###  2.3 Related work

Due to the lack of work that focuses on combining the potential of GCNs,
event-based data and hardware implementation on FPGAs, we start the review
with methods that use GCNs to process event-based data. We then move on to
analyse work that focuses on using FPGAs to accelerate GCNs.

####  2.3.1 Object detection using GCN for event data.

Initially, GCNs were used to process event data because they allowed
asynchronous processing at the individual event level. One of the first work
[11] proposed an EventNet architecture that processed events in a recursive
manner using temporal encoding. Subsequent work further developed this area,
enabling object recognition based on event data. In the work [2], residual
graph blocks were used, while the work [7] introduced asynchronous graph
construction and update using slide convolution. Recent research in this area
[5], [10] has shown that object detection is also possible using GCNs.
Asynchronicity was achieved by updating only neighbouring vertices in each
successive layer, reducing the computational cost and latency of processing
each event. A different approach, compared to previous works, was presented in
[6]. The focus was on developing memory-efficient GCNs for event data
processing, utilising basic convolutional layers tailored for point-cloud data
processing.

Despite these advances, all of the work mentioned is mainly based on high-
energy platforms such as GPUs. This makes it impossible to develop energy-
efficient detection systems using event cameras.

####  2.3.2 Graph acceleration on FPGA.

Implementations of GCNs on FPGAs primarily concentrate on hardware
acceleration of the networks themselves. The paper [12] introduces a
lightweight GCN accelerator on an FPGA, while [1] presents an accelerator
tailored to various types of convolutional layers. Additionally, the work [4]
explores network acceleration on multi-FPGA platforms.

It is worth noting, however, that much of the existing work is centred on
processing established graphs or generating graphs based on predefined
structures, such as the outputs of accelerator sensors [8]. In contrast, when
dealing with event data characterised by a random spatio-temporal nature, the
required approach differs. It this work, we try to address the latter issue.

##  3 The proposed graph representation optimisation

###  3.1 Graph representations

A variety of graph structure representation methods can be found in the
literature, such as Compressed Sparse Row (CSR) and Compressed Sparse Column
(CSC). Nevertheless, the most popular format is the Coordinate (COO) format.

In COO format, the edges are represented by vectors containing source and
destination vertex indices. In GCNs implementations using off-the-shelf
libraries, such as PyTorch Geometric, as demonstrated in works [5], [6], [10],
this representation format is often used because of its simplicity.

Since the number of potentially generated edges linked to an event can vary
significantly, initialising sufficient memory to process and store the data in
an FPGA is impossible. To overcome this hardware limit, we propose our own
modified representation.

###  3.2 Optimisation

First, it is crucial to determine whether the graph will consist of directed
or undirected edges. In our approach, we establish that with the arrival of
each new vertex, previous events are directed towards this new event,
indicating that the graph is directed based on time. This procedure enables us
to apply updates solely to the newly examined vertex during the convolution
operation.

As we intend to use the PointNet convolution in our work, it becomes necessary
to determine the attributes of a vertex, its neighbours, and the positional
difference between vertices connected by edges. To achieve this, for each
vertex, we record its own position $pos=(x,y,t)$, the attribute $a=(p)$
(polarity), and generate edges that specify the connected vertices. For this
purpose, we introduce three optimisation methods.

####  3.2.1 Normalisation.

The locations values of the incoming events depend on the matrix resolution,
while the time is represented in microseconds. In existing works [5], [6],
[10], the normalisation only applies to time, where it is multiplied by some
factor to make it close to the dimension of the $x$ and $y$ values. Instead,
we propose normalising the spatial and temporal dimensions to a uniform range
($SIZE$). In addition, we quantise the time after normalisation to an integer
value. This results in a spatio-temporal representation where event values are
in the range $(0,0,0)-(SIZE,SIZE,SIZE)$.

####  3.2.2 Neighbour matrix.

In existing solutions, edge generation requires searching for all vertices or
a relevant subgroup to identify potential neighbours. Implementing this
effectively in an FPGA is difficult due to the requirement of sorting a large
number of vertices. To address this problem, we propose a two-dimensional
neighbour matrix (_NM_) for edge generation, which has dimensions of
$SIZE\times SIZE$ after normalisation. When a new event with position
$(x,y,t)$ occurs, we check its local neighbourhood defined by the search
radius $R$ in _NM_. This approach allows us to search for neighbours only in
the group of nearest values, while reducing the number of potentially
generated edges. We also avoid the situation where one vertex is connected by
edges to several vertices with the same spatial position $(x,y)$ but different
times $t$. Only the most recent vertex from a given pixel is considered.

####  3.2.3 Unique events.

If, for a given pixel $(x,y)$ in the neighbour matrix, a vertex with the time
$t$ already exists, we treat the appearance of an event with the same time as
a duplicate and do not include it in the edge generation process.

By using these methods, the maximum number of potential vertices in the entire
graph, which depends on the $SIZE$ parameter, can be determined. The number of
generated edges depends on the search context, defined by the radius $R$. Due
to the use of time-directed edges, vertices with generated edges can be stored
in external memory, eliminating the need for their later updates.

Table 1: Average number of nodes and edges for data normalised to a given
graph size. Methods selected for hardware implementation are in bold.

Graph size | Event Preprocessing | Edge Generation |  Average Nodes |  Average Edges
---|---|---|---|---
Without | Random | Unique | Radius | NM
128 | x |  |  | x |  | 56,940 | 836,304
x |  |  |  | x | 309,711
| x |  | x |  | 25,000 | 173,868
| x |  |  | x | 70,896
|  | x | x |  | 50,498 | 623,078
|  | x |  | x | 262,935
256 | x |  |  | x |  | 56,940 | 138,757
x |  |  |  | x | 67,043
| x |  | x |  | 25,000 | 28,892
| x |  |  | x | 14,051
|  | x | x |  | 55,925 | 131,648
|  | x |  | x | 65,214

###  3.3 Ablation studies

To examine the impact of the optimisations considered, we performed ablation
studies on the N-Caltech101 dataset 111The N-Caltech101 dataset is an event-
based version of the original Caltech101 dataset. It consists of 8246 samples
distributed across 100 classes, with a maximum resolution of $240\times 180$
pixels.. From each sample, we extracted a 50 ms time window with the highest
event density. The ablation studies were divided into two main components: the
effect of the methods on the size of the generated graph and the training
results of the model considered. For each of these elements, we distinguished
three subgroups of influencing methods: value normalisation, event
preprocessing, and edge generation method.

The methods we presented were compared with the solutions proposed in [10], in
which the number of events is sampled at $random$ to a fixed value (we
selected the number to be 25,000) and using the $radius$ method, in which up
to 32 neighbours are searched within a distance $R$ (we selected $R$ to be
equal to 3), considering all vertices in the graph. We also compared the
methods $without$ taking into account the preprocessing of events.

####  3.3.1 Impact of the methods on the size of the generated graph.

We noticed the biggest difference in the number of generated edges between
$radius$ and the $NM$ (neighbour matrix) methods (Table 1). This is mainly due
to the fact that the $radius$ function generates undirected edges.
Additionally, when using $NM$, we avoid edge multiplications within the same
pixel that occur in a short time interval. Normalisation to 128 means the
reduction of around 12% of the vertices, while for normalisation to 256 the
reduction is only 2%. A separate analysis of the average pixel activity showed
that the use of unique values in particular reduced the impact of the
occurrence of overly bright pixels. Although the use of $random$ preprocessing
significantly reduces the number of vertices and edges, dropping events to a
fixed value without prior knowledge of how many there will be is impossible.
This shows that by employing our methods, it is feasible to process a majority
of events while generating fewer edges, compared to the method $radius$.
Additionally, the uniform structure of the graph allows for the initialisation
of an appropriate memory size.

####  3.3.2 Evaluation results.

The best mAP (mean Average Precision) results based on the methods used are
presented in Table 2. The model achieving the highest accuracy values utilised
normalisation to a value of 256, along with subsampling and the $radius$
methods. Notably, this model outperformed the baseline model, which relied
only on temporal normalisation, by achieving a 1.3% higher mAP value.
Normalisation to a value of 64 significantly impacted object classification
performance, resulting in poorer outcomes.

An analysis of graph directionality revealed that using a time-directed graph
led to only a marginal 0.8% loss in mAP compared to an undirected graph. In
contrast, the lowest performance was observed for the oppositely timed
directed graph, resulting in an almost 2% decrease compared to the undirected
graph. Furthermore, applying all the optimisation methods, including
normalisation to 256, only slightly affects the results with a 0.08% reduction
in the mAP score.

Table 2: Ablation study for accuracy for different methods.

Method | Normalisation | Pre-processing | Edge Generation | mAP
---|---|---|---|---
Base model[6] | Only time = 100 | Random | Radius | 53.47
Normalisation value | 64 | Random | Radius | 43.73
128 | Random | Radius | 51.99
256 | Random | Radius | 54.77
Direction of edges | 128 | Random | NM - undirected | 52.20
128 | Random | NM - time directed | 51.40
128 | Random | NM - op. directed | 50.23
Unique values | 128 | Unique | NM - time directed | 51.28
256 | Unique | NM - time directed | 53.39

Table 3 presents a comparative analysis of various GCN models in detection
task on the N-Caltech101 dataset. Our work uses the model presented in [6].
Although its accuracy does not exceed that reported in [5], it significantly
reduces the size of the model. The NvS-S model [7], the closest in size to
ours, possesses three times as many parameters, but shows an accuracy lower by
18%. In contrast, the model [10], which heavily influenced the development of
[6], achieves a 6.11% higher mAP at the cost of being 81 times larger in size.

The presented results demonstrate that, using the described optimisation
methods, it is possible to construct a graph that enables effective object
recognition with small accuracy losses. Also, our model achieves a good
compromise between accuracy and size, making it well suited for use on
embedded platforms.

Table 3: Comparison of different methods for object detection. * was not
directly available and estimated based on the publication. Best results in
bold.

Name | mAP | Parameters [M]
---|---|---
NvS-S [7] | 34.6 | 0.9
AEGNN [10] | 59.5 | 20.4
EAGR-N [5] | 62.9 | 2.7*
EAGR-L [5] | 73.2 | 30.7*
Our | 53.39 | 0.25

##  4 Hardware module concept

We developed the optimisation methods for the representation of the event data
graph (Section 3) taking into account the hardware constraints of FPGA
platforms. In this context, the limitation of embedded memory usage and
ensuring a deterministic number of operations are of key importance.

###  4.1 Hardware module requirements

From a functional point of view, the graph generation hardware module must
meet the throughput and latency requirements to enable real-time event
processing, even in highly dynamic scenes. The processing time of a single
event affects the throughput of the system.

The maximum number of events per second, after time quantisation, is
$SIZE\times SIZE$ for a single time unit. However, a situation where the
maximum number of events is present is highly unlikely. A realistic estimate
of the required throughput was carried out based on an analysis of the number
of events per time unit for the N-Caltech101 dataset. The maximum average
number of events per 1 ms was determined to be approximately 3300, which means
an average of $\sim$3.3 events per 1 µs. The second major constraint on the
implementation of the module is the limited amount of internal memory and
logic resources of FPGAs. We address this by quantising input values and
optimising memory footprint usage at the cost of a slight reduction in
accuracy.

###  4.2 The proposed hardware module

Figure 1: The proposed hardware module scheme

Based on the results of the GCN detection study and taking into account the
specified system requirements, for the hardware module, we decided on a graph
size of $SIZE=256$, edges directed with time and a radius of $R=3$. The module
was implemented in the SystemVerilog language, simulated and synthesised using
Xilinx Vivado tool for an AMD Xilinx ZCU 104 board with the Zynq UltraScale+
MPSoC chip (XCZU7EV-2FFVC1156). During testing, we assumed a module running
synchronously with a clock frequency of at least 250 MHz.

####  4.2.1 Input, normalisation and FIFO modules.

The module’s input is a single event, described by the following signals: $t$
(timestamp), $x$, $y$ (coordinates) and $p$ (polarity). In addition, the
$valid$ flag makes it possible to determine whether an event has been received
in a given clock.

The first operation performed on a received event is its scaling (for $t$, $x$
and $y$ values) and quantisation to $0-SIZE$ values (Fig. 1, normalisation
module). Normalisation performed immediately after receiving the event reduces
the use of memory and logic resources. This operation is executed within a
single clock, so it does not affect system throughput, only latency.

After normalisation, the event can be passed directly to the graph generation
module. However, due to the asynchronous and sparse nature of the event data,
we decided to add a FIFO queue here (Fig. 1, FIFO). In this way, events are
accumulated in the queue in moments of increased scene dynamics. Such a
mechanism allows efficient graph generation even for moments when the number
of events per time significantly exceeds the system throughput. The length of
the queue was set considering the availability of memory resources. After
analysing the number of events per time, we assumed its size to be $1024$
elements, which was sufficient for the test sequences used. The number of bits
per memory cell was 25 (3 $\times$ 8-bit coordinates and 1-bit polarity) after
normalisation.

####  4.2.2 Context generation module.

The second element of the system is the analysis of the context of the event.
As described in Section 3, we use a neighbour matrix ($NM$) that contains
information about the most recent recorded events. To store its contents, we
used a two-port BRAM memory of size $SIZE\times SIZE$, with one port for READ
and WRITE operations and the other for READ only operations. To reduce the
amount of data stored in the memory, we used the normalised event coordinates
as the indexes of particular matrix cells, to which we wrote only the
timestamp values. In this way, we could easily read the events in ($x,y,t$)
format from the memory, decoding the coordinates $x$ and $y$, while storing
this data in just 8 bits (timestamp only).

For each quantised event read from the FIFO, the content of the memory cell
corresponding to its coordinates ($x,y$) in the matrix is checked (Fig. 1,
Check for duplicates module). If the timestamp value read from the $NM$ is the
same as in analysed event, it is dropped and another one is read from the FIFO
queue.

Otherwise, a context of radius $R=3$ pixels in form of a square with
$2R+1\times 2R+1$ shape is searched for potential edge candidates. The stored
timestamps are read from the BRAM memory (Fig. 1, Check context module). Each
READ operation during the processing of a single event takes 1 clock cycle
(i.e., the latency is 1). To read the entire context (i.e., 48 memory cells),
two ports of the BRAM memory are used, thus reducing the time of this
operation twice – to 24 clock cycles for a single event. In practice, it may
be sufficient to read fewer neighbours from the $7\times 7$ context, using one
of the approximate representations of a circle – then instead of the most
time-consuming case of 48 candidates (for a square), 36 or 24 reads are
sufficient, reducing the latency of the module and increasing the number of
processable events per time. However, in our work, we developed the generation
module for the worst-case scenario. It is worth noting that by increasing the
width of the BRAM interface (storing multiple context candidates in one memory
cell), the bandwidth of the system can be even higher.

After the entire context is read, a single WRITE operation to the BRAM memory
takes place to update the timestamp value corresponding to the event being
processed (Fig. 1, Save to context module). At the same time, a flag is
generated indicating that the reading of the context has been completed.

####  4.2.3 Output: edges generation.

The final step is to analyse the context and determine the output signals,
which are pairs – each nondropped input event and a list of its edges. A
simple delay line is used to pass the event (in the format $(x,y,t,p)$) to the
module output. Each edge is described with the absolute values of vertices
connected by an edge $(x,y,t)$ stored on 24 bits. By normalising the time and
dropping events with the same $(x,y,t)$, an accurate description of a given
vertex and described edge definition allows unambiguous identification of
connected vertices. From such a generated edge representation, it is thus very
easy to switch to the $pos$ and $edges$ structures described in Section 3.

To perform context-based edge generation, a neighbourhood condition of
distance $\leq R$ (in each direction) is checked for each candidate (Fig. 1,
Check radius module). We search for the neighbours inside the hemisphere
$(x-x_{c})^{2}+(y-y_{c})^{2}+(t-t_{c})^{2}<=R^{2}$, where $(x,y,t)$ are the
coordinates of the vertex being processed and $(x_{c},y_{c},t_{c})$ are the
coordinates of the candidate read from the context. Resulting pairs (vertex
and its edge list) can then be written to external memory or passed to the GCN
accelerator.

####  4.2.4 Theoretical throughput.

All logical operations in the proposed module take 1 clock cycle, so the
processing of input events can be pipelined. However, the bottleneck of the
solution is the communication with the BRAM memory, where the timestamp values
are stored. A maximum of 26 clock cycles per single event are required: 25 for
READ and 1 for WRITE operation, which was confirmed through simulation. Based
on this information, we determined the theoretical throughput of the system to
compare it with the specified requirements. For a 250 MHz clock, the system is
able to accept a new event once every 104 ns, so over 9.6 events per 1 µs. The
calculated theoretical throughput is almost 3 times higher than the
requirements determined based on analysis of the N-Caltech101 dataset.
Moreover, the use of FIFO module enables a high performance system for
sequences with much higher temporary dynamics in the scene.

####  4.2.5 Hardware utilisation.

The proposed edge generation module was synthesised for an exemplary AMD
Xilinx ZCU 104 board with the Zynq UltraScale+ MPSoC chip. The utilisation of
hardware resources is presented in Table 4.

Table 4: Hardware resource utilisation for the proposed edge generation module
on an exemplary ZCU 104 platform for graph size of $256\times 256$.

Resource type | Available | Used
---|---|---
LUT | 230400 | 5612 (2.4)
Flip-Flop | 460800 | 950 (0.2)
Block RAM | 312 | 17 (5.5)
DSP | 1728 | 189 (10.9)

##  5 Conclusion

In this paper, we presented the concept of a hardware module that, based on
the event stream, can generate a graph representation that can serve as an
input to a graph convolutional network. In a series of experiments, we have
shown that our proposed approach using directed (time-based) edges,
subsampling and quantisation results in only a small loss of detection
performance – 0.08% mAP for the N-Caltech dataset. The proposed module, in
combination with a GCN accelerator and a SoC FPGA device, will allow to
develop an embedded object detection system based on event data.

Our future objectives include developing a detection system and exploring
several key areas: investigating diverse input data filtering and aggregation
techniques, particularly those influenced by SNNs and tackling continuous
event stream processing.

## References

  * [1] Abi-Karam, S., He, Y., Sarkar, R., Sathidevi, L., Qiao, Z., Hao, C.: Gengnn: A generic fpga framework for graph neural network acceleration. arXiv preprint arXiv:2201.08475 (2022). https://doi.org/10.48550/arXiv.2201.08475
  * [2] Bi, Y., Chadha, A., Abbas, A., , Bourtsoulatze, E., Andreopoulos, Y.: Graph-based Object Classification for Neuromorphic Vision Sensing. In: 2019 IEEE International Conference on Computer Vision (ICCV). IEEE (2019). https://doi.org/10.1109/ICCV.2019.00058
  * [3] Cordone, L., Miramond, B., Thierion, P.: Object detection with spiking neural networks on automotive event data. In: Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN) (July 2022). https://doi.org/10.1109/IJCNN55064.2022.9892618
  * [4] Engelhardt, N., So, H.K.H.: GraVF-M: graph processing system generation for multi-FPGA platforms. ACM Transactions on Reconfigurable Technology and Systems (TRETS) 12(4), 1–28 (2019). https://doi.org/10.1145/3357596
  * [5] Gehrig, D., Scaramuzza, D.: Pushing the limits of asynchronous graph-based object detection with event cameras. arXiv preprint arXiv:2211.12324 (2022). https://doi.org/10.48550/arXiv.2211.12324
  * [6] Jeziorek, K., Pinna, A., Kryjak, T.: Memory-Efficient Graph Convolutional Networks for Object Classification and Detection with Event Cameras. In: 2023 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA). pp. 160–165. IEEE (2023). https://doi.org/10.23919/SPA59660.2023.10274464
  * [7] Li, Y., Zhou, H., Yang, B., Zhang, Y., Cui, Z., Bao, H., Zhang, G.: Graph-based Asynchronous Event Processing for Rapid Object Recognition. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 914–923 (2021). https://doi.org/10.1109/ICCV48922.2021.00097
  * [8] Neu, M., Becker, J., Dorwarth, P., Ferber, T., Reuter, L., Stefkova, S., Unger, K.: Real-time Graph Building on FPGAs for Machine Learning Trigger Applications in Particle Physics. arXiv preprint arXiv:2307.07289 (2023). https://doi.org/10.48550/arXiv.2307.07289
  * [9] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 652–660 (2017). https://doi.org/10.1109/CVPR.2017.16
  * [10] Schaefer, S., Gehrig, D., Scaramuzza, D.: AEGNN: Asynchronous Event-Based Graph Neural Networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 12371–12381 (June 2022). https://doi.org/10.1109/CVPR52688.2022.01205
  * [11] Sekikawa, Y., Hara, K., Saito, H.: EventNet: Asynchronous recursive event processing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3887–3896 (2019). https://doi.org/10.1109/CVPR.2019.00401
  * [12] Tao, Z., Wu, C., Liang, Y., Wang, K., He, L.: LW-GCN: A lightweight FPGA-based graph convolutional network accelerator. ACM Transactions on Reconfigurable Technology and Systems 16(1), 1–19 (2022). https://doi.org/10.1145/3550075


# The Speed of Allosteric Signaling Within a Single-Domain Protein

Olga Bozovic, Jeannette Ruf, Claudio Zanobini, Brankica Jankovic, David
Buhrke, Philip J. M. Johnson, Peter Hamm∗
Department of Chemistry, University of Zurich, Zurich, Switzerland
<EMAIL_ADDRESS>
###### Abstract

While much is known about different allosteric regulation mechanisms, the
nature of the “allosteric signal”, and the timescale on which it propagates,
remains elusive. The PDZ3 domain from postsynaptic density-95 protein is a
small protein domain with a terminal third alpha helix - the $\alpha 3$-helix,
which is known to be allosterically active. By cross-linking the allosteric
helix with an azobenzene moiety, we obtained a photocontrollable PDZ3 variant.
Photoswitching triggers its allosteric transition, resulting in a change in
binding affinity of a peptide to the remote binding pocket. Using time-
resolved infrared and UV/Vis spectroscopy, we follow the allosteric signal
transduction and reconstruct the timeline in which the allosteric signal
propagates through the protein within 200 ns.

##  I Introduction

To adapt is to survive, and all living cells need to constantly alter their
activity according to the evolving conditions of their surroundings in order
to preserve homeostasis. Allosteric regulation is one of the main tools
utilized by Nature to alter a target molecules’ activity by adjusting its
affinity to certain substrates as a response to an incoming signal. It
represents the communication between two non-overlapping sites of a protein
and is used as a mechanism for regulating activity. However, the very nature
of the allosteric signal is not clear.1 Within the framework of traditional
structural biology, transduction of the allosteric signal has long been
considered to be the propagation of structural change throughout oligomeric
proteins.2, 3 The concept of dynamic allostery has been introduced as well,
where allostery is transduced by changing the flexibility of certain parts of
a protein, and thereby its entropy, without major structural rearrangements.4,
5, 6, 7, 8, 9 Very recently, long-ranged charge reorganisations have been
proposed as yet another mechanism of allostery.10

Figure 1: Schematic representation of the protein system used in this study.
The source of the perturbation, the photoswitch molecule located on the
$\alpha$3-helix, is shown in magenta in the left panel, while the addressee of
the allosteric signal, the peptide ligand located in the binding groove, is
shown in magenta in the right panel. The figure has been adapted from
PDB:1tp5.

Allostery is often related to the communication between the domains of larger,
multi-domain proteins. With the discovery that isolated PDZ domains have
allosteric properties as well, they became attractive systems to study
allostery within single, small domain proteins.11, 12, 6, 13, 14, 15, 16 PDZ
domains are a prime example, in which allostery is thought to be driven by
changes in the intrinsic protein dynamics.

The PDZ3 domain from PSD-95 protein (Post Synaptic Density-95) differs from
the rest of the PDZ family by an auxiliary helix at its C-terminus, and is
particularly interesting in this regard.17, 18, 19, 20, 6 Petit et al. showed
that even though this $\alpha$3-helix does not form any direct contact with
the peptide ligand, its deletion dramatically decreases the binding affinity
of the ligand to the protein by 21-fold.6 In our previous work, we designed a
photocontrollable variant of the PDZ3 domain, showing that allostery could be
controlled by light by photoswitching the allosteric element - the
$\alpha$3-helix.21 To that end, an azobenzene moiety has been covalently
linked to the $\alpha$3-helix in a way that photo-isomerisation of the former
stabilizes/destabilizes the helicity of the latter, as an analogue of a
process observed in vivo, where the phosphorylation of residue tyr397 has the
same effect.20 The construct (see Fig. 1) is arguably the smallest truly
allosteric protein with clearly identifiable allosteric and active sites. We
were able to allosterically perturb the binding affinity for the peptide
ligand by photoswitching the distal helix with astonishing changes in binding
affinities, up to 120 fold depending on temperature. Conversely, the rate of
the photoswitch’s thermal cis-to-trans isomerisation is influenced by the
presence or absence of the peptide ligand. By measuring binding affinities at
different temperatures, we were able to construct a thermodynamic cycle of
allostery and quantify the allosteric force transmitted from the peptide
ligand through the PDZ3 protein to the photoswitch.21

Here, we exploit the structural sensitivity of time-resolved infrared
spectroscopy,22 combined with time-resolved UV/Vis spectroscopy, to follow the
allosteric signal in the photoswitchable PDZ3 domain from the point of its
origin – the $\alpha$3-helix \- through the protein until it reaches the
binding groove and finally transduces to the peptide. We reveal the
progression of the signal that follows the allosteric event and show that
there is a well-defined temporal sequence of events, despite the fact that the
protein responds on many timescales.

##  II Results and Discussion

For a spectroscopic signature specific to the source of the allosteric signal,
we start with transient UV/Vis spectroscopy of the $\pi$-$\pi$*-transition of
the photoswitch molecule, located on the $\alpha 3$-helix. To that end, the
sample was first prepared in the cis-state by constant illumination with a 370
nm cw laser, which promotes trans-to-cis isomerisation of the photoswitch. The
purity of the cis-sample obtained in this way is estimated to be above
85$\%$.22, 23 Using a short pump pulse at 420 nm, we then induce the ultrafast
cis-to-trans isomerisation of the photoswitch, which destabilises the helical
structure of the $\alpha$3-helix.24 By following the time-resolved response of
the $\pi$-$\pi$*-transition at 370 nm, we can retrieve information on the
timescales of photoswitching as well as the subsequent $\alpha$3-helix
unfolding. Thus, we utilize the photoswitch molecule not only as a trigger to
perturb the helical structure of the $\alpha$3-helix, and thereby initiating
the allosteric response, but also as a local reporter of photoswitch
conformation and $\alpha 3$-helix structure.

Figure 2: (a) Transient absorption changes at 370 nm for the isolated protein
(purple) and (b) the protein-peptide complex (green) with the corresponding
lifetime analysis (black). Panel (c) shows the equivalent for the IR double-
difference response at 1630 cm-1.

The corresponding kinetic traces are shown in Figs. 2a,b for the isolated
protein and the protein-peptide complex, respectively. The timescales
contained in these data are determined by fitting them to a multiexponetial
function:

$S(t)=a_{0}-\sum_{k}a(\tau_{k})e^{-t/\tau_{k}},$ (1)

where a maximum entropy method has been applied for regularisation.25, 26 In
this fit, the timescales $\tau_{k}$ were fixed and equally distributed on a
logarithmic scale with 10 terms per decade, while the amplitudes $a(\tau_{k})$
were the free fitting parameters. The resulting lifetime spectra $a(\tau_{k})$
are shown in Figs. 2a,b as black lines. In both cases, they reveal two similar
kinetic events, one at $<$100 ps and a second one around 4 ns for the protein-
peptide complex and 6 ns for the isolated protein.

The actual isomerisation of the azobenzene moiety around its central N=N bond
is a barrier-less picosecond process,27 giving rise to the first peak in the
lifetime analysis (the time-resolution of the experiment, 60 ps, does not
allow us to time-resolve this process, see Methods). Since the photoswitch is
covalently linked to the $\alpha$3-helix, it will however be in a very
constrained environment immediately following isomerisation. It has been shown
that the $\pi$-$\pi$*-band is a sensitive probe of any strained conformation
of the photoswitch.28 We therefore attribute the second kinetic component to
the release of that strain, and thus to the partial unfolding of the
$\alpha$3-helix. The time constant, 4 or 6 ns, respectively, is a reasonable
value for the forced unfolding of an $\alpha$-helix.29 The process speeds up a
little bit when a peptide ligand is present in the binding pocket, as a result
of the allosteric force transduced towards the photoswitch.21 Unfolding of the
$\alpha$3-helix represents the starting point of the allosteric signal.

Figure 3: FTIR difference spectra for cis-to-trans switching of the isolated
protein (a, purple), the protein-peptide complex (b, green) and double-
difference spectra, revealing the response of the peptide (c, orange). Panels
(e-f) show the corresponding time resolved data. The dashed box in panel (f)
marks the band, whose kinetics is highlighted in Fig. 2c.

To identify a spectroscopic signature of the addressee of the allosteric
signal, the peptide ligand, we compare FTIR difference spectra of the isolated
protein (Fig. 3a, purple) with that of the protein-peptide complex (Fig. 3b,
green). Each of these signals represent difference spectra upon photoswitching
the samples from the cis\- to the trans-state after effectively infinite time.
There are prominent absorption changes in the amide I region between 1600 and
1700 cm-1, arising mainly from backbone C=O modes of the protein and the
peptide, which are known to be strongly structure dependent.30 When analyzing
the response of the isolated protein, the amide I band represents the
conformational rearrangements of the protein upon photoswitching the
$\alpha$3-helix. In contrast, in the protein-peptide complex, this band is the
combined response of the protein and the peptide. Assuming that the spectra
are additive, one can subtract out the protein’s contribution and isolate the
response of the peptide (potentially together with parts from the binding
pocket that are directly affected) by taking the difference between these two
spectra. Indeed, this double-difference spectrum reveals a clear spectroscopic
signature (Fig. 3c,orange).

Figs. 3d,e represent the transient signals of the isolated protein and the
protein-peptide complex, respectively. The two maps do not differ
significantly, as the overall signal is dominated by the response of the
protein. Nevertheless, by taking the difference between the two maps, we can
isolate the response of the peptide (Fig. 3f). Fig. 2c shows the time trace of
the positive band at 1630 cm-1 together with its lifetime analysis. The first
peak related to the cis-to-trans isomerisation of the photoswitch is largely
suppressed by the subtraction, while the second peak is delayed relative to
$\alpha$3-helix folding. We attribute its timescale, 200 ns, to the time it
takes for the allosteric signal to reach the peptide ligand.

Figure 4: Dynamical content of (a) the isolated protein, (b) the protein-
peptide complex and (c) the peptide ligand, calculated from Eq. 2. The various
timescale peaks are labeled according to their major origin.

The “dynamical content” shown in Fig. 4 encompasses all kinetic processes of
the system.31 The dynamical content is calculated by fitting the transient
data of Figs. 3d-f separately for each probe frequency $\omega_{i}$ according
to Eq. 1, and subsequently summing over all probe wavelengths:

$D(\tau_{k})=\sqrt{\sum_{i}a(\omega_{i},\tau_{k})^{2}}.$ (2)

In contrast to Fig. 2c, which exemplifies the response at a single probe
frequency, this expression provides an overview of all kinetic processes, and
furthermore accounts for the fact that the lifetime spectra
$a(\omega_{i},\tau_{k})$ can have positive or negative signs.

The protein responds to the perturbation induced by the photoswitch with
motions over a set of discrete timescales ranging from 1 ns to 10 $\mu$s (see
Figs. 4a,b), in much the same way as observed for the PDZ2 domain.22 We have
attributed this results to a redistribution of population between a set of
conformational substates, e.g., the local minima on the rugged energy
landscape of the protein, along the lines of a Markov State Model.32, 33 The
redistribution of population is small, and hence also the change of overall
structure (0.3 Å). One can think of the Markov State Model as a network of
inter-converting states, whose populations shifts upon the allosteric
perturbation, in accordance with one of the more recent views of allostery.34,
35, 5, 36, 37 Such a network picture would not necessarily result in a
“sequence of events”. Nevertheless, when isolating the response of the peptide
ligand, the dynamical content clears up significantly (compare Fig. 4b vs Fig.
4c), and reveals such a sequence of events. That is, the $\alpha$3-helix
unfolds on a 4 ns timescale (Fig. 2a,b), and it takes 200 ns until that signal
arrives at the peptide ligand (Fig. 2c and 4c). The more complicated response
of the protein tends to mask this process; in fact the 200 ns peak in the
dynamical content of the peptide ligand (Fig. 4c) is only a minor contribution
to that of the peptide/protein complex (Fig. 4b). The 200 ns timescale
establishes the speed of the allosteric signal through the PDZ3 domain.

It is important to stress that the appearance of the allosteric signal at the
peptide ligand after 200 ns does not yet result in its unbinding. First, the
off-rate, which can be estimated from the binding affinity and typical on-
rates for the PDZ3 system,38 is expected to be by many orders of magnitudes
slower, 200 ms. Second, in order to distill out the structural response of the
peptide ligand, we have performed these experiments at room temperature, where
the binding affinities in the two states of the photoswitch are essentially
the same.21 Hence, the amount of bound molecules will not change, despite the
very different structural and dynamical properties of the ligand in the
binding pocket, which are evidenced by the fact that the ligand binding is
exothermic in the cis-state and endothermic in the trans-state.21 What we
observe with the 200 ns process is exactly this change of the structural and
dynamical properties of the peptide ligand.

##  III Conclusion

Following the dynamics of proteins in real-time requires an instantaneous
trigger for the non-equilibrium transition in question, i.e., preferentially a
photo-responsive element that enables very high time-resolution. Naturally
abundant photoreactive proteins such as photoreceptors and
haemoglobin/myoglobin derivatives enable photoinduced perturbations and
subsequent time-resolved studies.39, 40, 41 Unfortunately, the vast majority
of proteins are not photo-responsive and need an artificial photoactive
element incorporated into their structure. The emerging field of designing
artificially photocontrollable systems allows one to introduce the aspect of
time in the investigation of fundamental mechanisms of biomolecular
recognition.

A sequential propagation of the allosteric signal is somewhat naively thought
to be only possible in large multidomain proteins, which undergo a significant
conformational rearrangement. As is the example with haemoglobin42 or ATP
synthase,43 it is natural to picture a large conformational change, which
triggers a cascade of events that finally lead to an allosteric effect.
However, when discussing allosteric propagation through small protein domains,
and especially ones where allostery is predominantly dynamically driven, it
becomes unclear what the underlying molecular mechanism is, and what the
signal sequence is.

By incorporating the photoswitch moiety to the $\alpha$3-helix of the PDZ3
domain, we were able to allosterically alter its affinity towards the peptide
ligand. 21 Setting proteins in motion by photoswitching, we followed the
response of the protein and its ligand in both UV/Vis and infrared spectral
regions. Transient measurements exposed that the allosteric signal is
transduced in a sequence of events, starting from the point of the
perturbation –the $\alpha$3-helix\- through the protein to the binding groove
in 200 ns. The 200 ns timescale represents the speed of the allosteric signal
within an isolated protein domain.

A time-dependent evaluation of biological processes is necessary in order to
reconcile the structure-function relationship of proteins. While a static
structural depiction gives us insight into the protein’s function, a broader
kinetic picture is needed to comprehend the true dynamical nature of proteins,
and capture intermediates that may not be obvious in equilibrium. Revealing
the timescales in which signals travel through biological systems will help
understanding how changes in intrinsic structure and dynamics ultimately
affect and alter their natural function.

##  IV Methods

###  IV.1 Protein and Peptide Preparation

Photocontrollable PDZ3 domain and KETWV peptide were produced as described
previously.21 The purity of all samples was confirmed with mass spectrometry.
The concentration of protein sample was determined by the visible absorption
of the photoswitch molecule at 325 nm, assuming an extinction coefficient of
$\epsilon$=10,000 M-1cm-1. The concentration of the peptide was determined by
the tryptophan absorption 280 nm. The concentration of all samples was cross-
confirmed with amino acid analysis.

###  IV.2 Time Resolved Spectroscopy

The time-resolved Vis pump-IR probe and Vis pump-Vis probe experiments were
performed using two electronically synchronized Ti:sapphire laser systems with
a repetition rate of 2.5 kHz, allowing for delay up to 42 $\mu$s.44 The
wavelength of the pump laser was tuned to 840 nm so that second harmonic
generation in a BBO crystal produced pump pulses centered at a wavelength of
420 nm to induce cis-to-trans isomerization. To minimize sample degradation,
the compressing stage after light amplification was bypassed, resulting in
stretched pulses of ca. 60 ps duration FWHM and a power of 4.2 $\mu$J. The
beam diameter was $\approx$ 135 $\mu$m FWHM at the sample position. The pump
pulses were mechanically chopped at half the repetition rate of the laser
setup. The mid-IR probe pulses were obtained in an optical parametric
amplifier,45 and centered at 1640 cm-1 with a pulse duration of $\approx$ 100
fs and a beam diameter of $\approx$ 115 $\mu$m. After the sample, they were
passed through a spectrograph and detected in a 2$\times$64 MCT array detector
with a spectral resolution of $\approx$ 2 cm-1/pixel. The water vapor lines of
the unpurged setup were used for frequency calibration. The polarization of
the pump pulses was set to magic angle (54.7°) relative to the probe pulses
and multichannel referencing as described in Ref. 46 was used for noise
suppression.

The visible probe experiments were performed as previously described.26 The
probe pulses were obtained by focusing $\approx$ 1 $\mu$J of the 800 nm probe
laser into a CaF2 window, which was continuously translated.

The samples ($\approx$ 700 $\mu$L; 1.1 mM protein and 4.4 mM peptide) were
cycled through a closed system with a sample reservoir, a peristaltic pump and
a flow-cell, all under inert atmosphere (N2). The reservoir was illuminated
with a continuous wave laser at 370 nm (150 mW, CrystaLaser) to prepare the
sample in the cis-state. The sample cell consisted of two CaF2 windows
separated by a 50 $\mu$m Teflon spacer. The flow speed of the peristaltic pump
was adjusted to ensure sufficient sample exchange before the subsequent laser
shot but to minimize flowing out of the sample at the latest time delay
($\approx$ 42 $\mu$s). Static reference spectra were acquired in an UV/Vis
(Shimadzu UV-2450) and an FTIR (Bruker Tensor 27) spectrometer, respectively,
with sample parameters that were comparable to those in the time-resolved
experiments.

Acknowledgement: We thank Gerhard Stock and his group for many insightful
discussions, the Functional Genomics Center Zurich, especially Serge Chesnov
and Birgit Roth, for their work on the mass spectrometry and amino-acid
analysis, and Livia Marie Müller for her contribution at an early stage of the
project. The work has been supported by the Swiss National Science Foundation
(SNF) through the NCCR MUST and Grant 200020B_188694/1.

Data Availability: The data that support the findings of this study are openly
available in Zenodo (the link will be provided at the proof stage.)

References:

##

  * Wodak et al. 2019 Wodak, S. J. et al. Allostery in its many disguises: From theory to applications. _Structure_ 2019, _27_ , 566–578.
  * Monod et al. 1965 Monod, J.; Wyman, J.; Changeux, J. P. On the nature of allosteric transitions: A plausible model. _J. Mol. Biol._ 1965, _12_ , 88–118.
  * Koshland et al. 1966 Koshland, D. E.; Némethy, G.; Filmer, D. Comparison of experimental binding data and theoretical models in proteins containing subunits. _Biochemistry_ 1966, _5_ , 365–385.
  * Cooper and Dryden 1984 Cooper, A.; Dryden, D. Allostery without conformational change. _Eur. Biophys. J._ 1984, _11_ , 103–109.
  * Smock and Gierasch 2009 Smock, R. G.; Gierasch, L. M. Sending signals dynamically. _Science_ 2009, _324_ , 198–203.
  * Petit et al. 2009 Petit, C. M.; Zhang, J.; Sapienza, P. J.; Fuentes, E. J.; Lee, A. L. Hidden dynamic allostery in a PDZ domain. _Proc. Natl. Acad. Sci. U.S.A._ 2009, _106_ , 18249–18254.
  * Tzeng and Kalodimos 2012 Tzeng, S.-R.; Kalodimos, C. G. Protein activity regulation by conformational entropy. _Nature_ 2012, _488_ , 236.
  * Saavedra et al. 2018 Saavedra, H. G.; Wrabl, J. O.; Anderson, J. A.; Li, J.; Hilser, V. J. Dynamic allostery can drive cold adaptation in enzymes. _Nature_ 2018, _558_ , 324.
  * Ahuja et al. 2019 Ahuja, L. G.; Aoto, P. C.; Kornev, A. P.; Veglia, G.; Taylor, S. S. Dynamic allostery-based molecular workings of kinase: peptide complexes. _Proc. Natl. Acad. Sci. U.S.A._ 2019, _116_ , 15052–15061.
  * Banerjee-Ghosh et al. 2020 Banerjee-Ghosh, K.; Ghosh, S.; Mazal, H.; Riven, I.; Haran, G.; Naaman, R. Long-Range Charge Reorganization as an Allosteric Control Signal in Proteins. _J. Am. Chem. Soc._ 2020, _142_ , 20456–20462.
  * Fuentes et al. 2004 Fuentes, E. J.; Der, C. J.; Lee, A. L. Ligand-dependent dynamics and intramolecular signaling in a PDZ domain. _J. Mol. Biol._ 2004, _335_ , 1105–1115.
  * Fuentes et al. 2006 Fuentes, E. J.; Gilmore, S. A.; Mauldin, R. V.; Lee, A. L. Evaluation of energetic and dynamic coupling networks in a PDZ domain protein. _J. Mol. Biol._ 2006, _364_ , 337–351.
  * Kumawat and Chakrabarty 2017 Kumawat, A.; Chakrabarty, S. Hidden electrostatic basis of dynamic allostery in a PDZ domain. _Proc. Natl. Acad. Sci. U.S.A._ 2017, _114_ , E5825–E5834.
  * Kumawat and Chakrabarty 2020 Kumawat, A.; Chakrabarty, S. Protonation-Induced Dynamic Allostery in PDZ Domain: Evidence of Perturbation-Independent Universal Response Network. _J. Phys. Chem. Lett._ 2020, _11_ , 9026–9031.
  * Gautier et al. 2018 Gautier, C.; Laursen, L.; Jemth, P.; Gianni, S. Seeking allosteric networks in PDZ domains. _Protein Eng. Des. Sel._ 2018, _31_ , 367–373.
  * Thayer et al. 2017 Thayer, K. M.; Lakhani, B.; Beveridge, D. L. Molecular dynamics–Markov state model of protein ligand binding and Allostery in CRIB-PDZ: conformational selection and induced fit. _J. Phys. Chem. B._ 2017, _121_ , 5509–5514.
  * Doyle et al. 1996 Doyle, D. A.; Lee, A.; Lewis, J.; Kim, E.; Sheng, M.; MacKinnon, R. Crystal structures of a complexed and peptide-free membrane protein–binding domain: molecular basis of peptide recognition by PDZ. _Cell_ 1996, _85_ , 1067–1076.
  * Cabral et al. 1996 Cabral, J. H. M.; Petosa, C.; Sutcliffe, M. J.; Raza, S.; Byron, O.; Poy, F.; Marfatia, S. M.; Chishti, A. H.; Liddington, R. C. Crystal structure of a PDZ domain. _Nature_ 1996, _382_ , 649.
  * Ballif et al. 2007 Ballif, B. A.; Carey, G. R.; Sunyaev, S. R.; Gygi, S. P. Large-scale identification and evolution indexing of tyrosine phosphorylation sites from murine brain. _J. Proteome Res._ 2007, _7_ , 311–318.
  * Zhang et al. 2011 Zhang, J.; Petit, C. M.; King, D. S.; Lee, A. L. Phosphorylation of a PDZ domain extension modulates binding affinity and interdomain interactions in postsynaptic density-95 (PSD-95) protein, a membrane-associated guanylate kinase (MAGUK). _J. Biol._ 2011, _286_ , 41776–41785.
  * Bozovic et al. 2020 Bozovic, O.; Jankovic, B.; Hamm, P. Sensing the allosteric force. _Nat. Commun._ 2020, _11_ , 5841.
  * Bozovic et al. 2020 Bozovic, O.; Zanobini, C.; Gulzar, A.; Jankovic, B.; Buhrke, D.; Post, M.; Wolf, S.; Stock, G.; Hamm, P. Real-time observation of ligand-induced allosteric transitions in a PDZ domain. _Proc. Natl. Acad. Sci. USA_ 2020, _117_ , 26031–26039.
  * Jankovic et al. 2019 Jankovic, B.; Gulzar, A.; Zanobini, C.; Bozovic, O.; Wolf, S.; Stock, G.; Hamm, P. Photocontrolling protein–peptide interactions: From minimal perturbation to complete unbinding. _J. Am. Chem. Soc._ 2019, _141_ , 10702–10710.
  * Flint et al. 2002 Flint, D. G.; Kumita, J. R.; Smart, O. S.; Woolley, G. A. Using an azobenzene cross-linker to either increase or decrease peptide helix content upon trans-to-cis photoisomerization. _Chem. Biol._ 2002, _9_ , 391–397.
  * Lórenz-Fonfría and Kandori 2006 Lórenz-Fonfría, V. A.; Kandori, H. Transformation of time-resolved spectra to lifetime-resolved spectra by maximum entropy inversion of the Laplace transform. _Appl. Spectrosc._ 2006, _60_ , 407–417.
  * Buhrke et al. 2020 Buhrke, D.; Oppelt, K. T.; Heckmeier, P. J.; Fernandez-Teran, R.; Hamm, P. Nanosecond protein dynamics in a red/green Cyanobacteriochrome revealed by transient IR spectroscopy. _J. Chem. Phys._ 2020, _153_ , 245101\.
  * Nägele et al. 1997 Nägele, T.; Hoche, R.; Zinth, W.; Wachtveitl, J. Femtosecond photoisomerization of cis-azobenzene. _Chem. Phys. Lett._ 1997, _272_ , 489–495.
  * Spörlein et al. 2002 Spörlein, S.; Carstens, H.; Satzger, H.; Renner, C.; Behrendt, R.; Moroder, L.; Tavan, P.; Zinth, W.; Wachtveitl, J. Ultrafast spectroscopy reveals subnanosecond peptide conformational dynamics and validates molecular dynamics simulation. _Proc. Natl. Acad. Sci. USA_ 2002, _99_ , 7998–8002.
  * Ihalainen et al. 2007 Ihalainen, J. A.; Bredenbeck, J.; Pfister, R.; Helbing, J.; Chi, L.; van Stokkum, I. H. M.; Woolley, G. A.; Hamm, P. Folding and unfolding of a photoswitchable peptide from picoseconds to microseconds. _Proc. Natl. Acad. Sci. U.S.A._ 2007, _104_ , 5383–5388.
  * Barth and Zscherp 2002 Barth, A.; Zscherp, C. What vibrations tell us about proteins. _Q. Rev. Biophys._ 2002, _35_ , 369–430.
  * Stock and Hamm 2018 Stock, G.; Hamm, P. A non-equilibrium approach to allosteric communication. _Philos. Trans. R. Soc. Lond. B Biol. Sci._ 2018, _373_ , 20170187\.
  * Bowman et al. 2013 Bowman, G. R.; Pande, V. S.; Noe, F. _An introduction to Markov state models_ ; Springer: Heidelberg, 2013.
  * Sengupta and Strodel 2018 Sengupta, U.; Strodel, B. Markov models for the elucidation of allosteric regulation. _Phil. Trans. R. Soc. B_ 2018, _373_ , 20170178\.
  * Swain and Gierasch 2006 Swain, J. F.; Gierasch, L. M. The changing landscape of protein allostery. _Curr. Opin. Struct. Biol._ 2006, _16_ , 102–108.
  * Tsai et al. 2008 Tsai, C. J.; del Sol, A.; Nussinov, R. Allostery: Absence of a Change in Shape Does Not Imply that Allostery Is Not at Play. _J. Mol. Biol._ 2008, _378_ , 1–11.
  * Tsai and Nussinov 2014 Tsai, C. J.; Nussinov, R. A unified view of ”how allostery works”. _PLoS Comput. Biol._ 2014, _10_ , e1003394.
  * Hilser et al. 2012 Hilser, V. J.; Wrabl, J. O.; Motlagh, H. N. Structural and energetic basis of allostery. _Annu. Rev. Biophys._ 2012, _41_ , 585–609.
  * Gianni et al. 2005 Gianni, S.; Engström, Å.; Larsson, M.; Calosci, N.; Malatesta, F.; Eklund, L.; Ngang, C. C.; Travaglini-Allocatelli, C.; Jemth, P. The kinetics of PDZ domain-ligand interactions and implications for the binding mechanism. _J. Biol. Chem._ 2005, _280_ , 34805–34812.
  * Schotte et al. 2004 Schotte, F.; Soman, J.; Olson, J. S.; Wulff, M.; Anfinrud, P. A. Picosecond time-resolved X-ray crystallography: probing protein function in real time. _J. Struct. Biol._ 2004, _147_ , 235–246.
  * Knapp et al. 2006 Knapp, J. E.; Pahl, R.; Šrajer, V.; Royer, W. E. Allosteric action in real time: Time-resolved crystallographic studies of a cooperative dimeric hemoglobin. _Proc. Natl. Acad. Sci. U.S.A._ 2006, _103_ , 7649–7654.
  * Cammarata et al. 2010 Cammarata, M.; Levantino, M.; Wulff, M.; Cupane, A. Unveiling the timescale of the R–T transition in human hemoglobin. _J. Mol. Biol._ 2010, _400_ , 951–962.
  * Shibayama 2020 Shibayama, N. Allosteric transitions in hemoglobin revisited. _Biochim. Biophys. Acta Gen. Subj._ 2020, _1864_ , 129335.
  * Boyer 1997 Boyer, P. D. The ATP Synthase—A Splendid Molecular Machine. _Ann. Rev. Biochem._ 1997, _66_ , 717–749.
  * Bredenbeck et al. 2004 Bredenbeck, J.; Helbing, J.; Hamm, P. Continuous scanning from picoseconds to microseconds in time resolved linear and nonlinear spectroscopy. _Rev. Sci. Instrum._ 2004, _75_ , 4462–4466.
  * Hamm et al. 2000 Hamm, P.; Kaindl, R. A.; Stenger, J. Noise suppression in femtosecond mid-infrared light sources. _Optics Letters_ 2000, _25_ , 1798–1800.
  * Feng et al. 2017 Feng, Y.; Vinogradov, I.; Ge, N.-H. General noise suppression scheme with reference detection in heterodyne nonlinear spectroscopy. _Opt. Express_ 2017, _25_ , 26262.