# Federated Transfer-Ordered-Personalized Learning for Driver Monitoring
Application

Liangqi Yuan, , Lu Su, , Ziran Wang Manuscript received January 11, 2023.L.
Yuan, L. Su, and Z. Wang are with the College of Engineering, Purdue
University, West Lafayette, IN 47907, USA (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>ryanwang11@hotmail.com).

###### Abstract

Federated learning (FL) shines through in the internet of things (IoT) with
its ability to realize collaborative learning and improve learning efficiency
by sharing client model parameters trained on local data. Although FL has been
successfully applied to various domains, including driver monitoring
application (DMA) on the internet of vehicles (IoV), its usages still face
some open issues, such as data and system heterogeneity, large-scale
parallelism communication resources, malicious attacks, and data poisoning.
This paper proposes a federated transfer-ordered-personalized learning
(FedTOP) framework to address the above problems and test on two real-world
datasets with and without system heterogeneity. The performance of the three
extensions, transfer, ordered, and personalized, is compared by an ablation
study and achieves 92.32$\%$ and 95.96$\%$ accuracy on the test clients of two
datasets, respectively. Compared to the baseline, there is a 462$\%$
improvement in accuracy and a 37.46$\%$ reduction in communication resource
consumption. The results demonstrate that the proposed FedTOP can be used as a
highly accurate, streamlined, privacy-preserving, cybersecurity-oriented,
personalized framework for DMA.

###### Index Terms:

Federated learning, internet of things (IoT), driver monitoring, privacy
protection, personalization.

##  I Introduction

With the rapid development of sensing, computing, and communication
technologies, the internet of things (IoT) is a popular solution to solve the
problems in industry, agriculture, energy, transportation, etc. However,
privacy issues in IoT are often a significant concern have been raised due to
the intrusive behavior of sensors [1]. Specifically for the internet of
vehicles (IoV), it massively parallels each vehicle and various sensors it
carries, including global positioning system (GPS), radar, camera, light
detection and ranging (LiDAR), etc., enabling pedestrian detection [2],
automated driving [3], mobility digital twins [4], and other transportation
applications. Federated learning (FL) has received extensive attention for
protecting user privacy by sharing only model weights and not including users’
raw data. FL is widely known for its successful business case in Google mobile
keyboard prediction [5]. Nowadays, It has also become one of the mainstream
and thriving solutions for privacy protection and efficient learning.

###  I-A Federated Learning and Related Work

FL is a potentially feasible solution to the privacy problem in IoT, which is
able to avoid the proliferation, distribution, and exchange of local client
data by sharing model parameters after training the model on local client
data. FL frameworks are widely used in healthcare [6, 7], industrial [8, 9],
IoV [10, 11], etc., due to their usages of large scale and personalized data
in an efficient and privacy-preserving way. Although FL has significant
contributions to massively parallel devices and computations, it still has a
notable drawback in that it cannot efficiently handle non-independent and
identically distributed (non-i.i.d.) data. It is required to customize the
applicable FL framework according to the features, resources, and constraints
possessed by users, data, clients, and servers.

Non-i.i.d. data and heterogeneity have always been a challenge and a key to
research in FL [12, 13, 14]. Non-i.i.d. data is a common phenomenon for real-
world clients that are scattered and not interoperable: Taking IoV as an
example, each driver is heterogeneous as a client. FedAvg [15], as one of the
first proposed feasibility methods, has been the subject and center of
research. FedAvg averages all local models to get the global model so that the
local model may deviate far from the global optimum in the parameter space
leading to some limitations in FedAvg. It is necessary to ensure that the
local model does not deviate from the global model (prevent overfitting) and,
simultaneously, that the local model can effectively learn the local client
dataset (prevent underfitting). Based on FedAvg, FedProx [16] is proposed to
limit the deviation of the local model from the global model by adding a
proximal term.

Besides considering accuracy, the FL framework in IoT should not underestimate
communication and training resource constraints, cybersecurity, and ubiquity.
Some of the recent surveys summarized challenges, threats, and solutions of
the FL decentralization paradigm for IoT, including limited computing power,
unreliable and limited availability, local training, accuracy, communication
overhead, etc. [17, 18, 19, 20, 21, 22].

Transfer and edge learning are popular solutions to reduce communication
resource consumption in FL frameworks. Zhang et al. [23] performed a federated
transfer learning framework to detect driver drowsiness, where transfer
learning was employed to save the communication cost in the FL framework. Su
et al. [24] introduced edge servers as a collaborative mechanism, where
aggregation of local models was aggregated in the edge server and then sent to
the global server to aggregate the global model. The benefit of the additional
edge server was that the communication between massively parallel clients and
the edge server was consumed because the edge server was geographically close
to the clients. High latency and intermittent connections could be mitigated.
In addition, the edge server could also provide personalized aggregated local
models due to the similarity of geographically adjacent clients.

Cyber attack is a problem that cannot be ignored for FL frameworks. Sun et al.
[25] developed an attack method for FL framework in IoT, in which a bi-level
optimization framework was proposed to compute optimal poisoning attacked FL
framework, including direct, indirect, and hybrid attacks. Meanwhile, Zhang et
al. [26] utilized a generative adversarial network (GAN)-based approach to
attack the FL framework, especially since the attacker did not need any prior
knowledge to carry out the attack.

Personalization is a common approach for FL frameworks to improve
applicability for diverse users [27]. Fallah et al. [28] proposed a
personalized variant of the FL, which allowed clients to perform several
gradient descent iterations on an initial global model using local data to
obtain a personalized local model. Wu et al. [29] explored a cloud edge-based
personalized FL framework for in-home health monitoring, which addressed the
problem that a single global model performed poorly on a specific client.
Since the global model could only capture the common features of all clients,
it lacked the ability to analyze fine-grained information of specific clients.

###  I-B Federated Learning in Driver Monitoring Applications

Driver monitoring application (DMA) in IoV is adopted as the research
direction in this paper due to its real and visual image data, valuable
application scenarios, and relatively blank research area. DMA also has
challenges in terms of driver privacy issues, communication, and diversity and
personalized driver behavior. Related DMA literature covers a wide variety of
devices with algorithms to achieve different purposes, such as dangerous state
detection [30], driver emotion recognition [31], driver lane change inference
[32], etc. Compared to other methods [33, 34, 35], FL not only highlights
efficient learning but also effectively protects the privacy of driver,
passenger, and pedestrian biometric information, driving routes, and
confidential driving areas such as military installations.

In this paper, we introduce and adapt FL to DMA. Although some FL frameworks
exist for DMA, they all suffer from some critical problems. Doshi et al. [36]
proposed a FL edge-device framework to obtain a global model by aggregation
feature representations and obtained considerable accuracy in recognizing
driver activities. For the i.i.d. setting, the dataset was partitioned for
each edge node in a random way, while for the non-i.i.d. setting, the dataset
was assigned selectively. Zhao et al. [37] proposed a FL framework to monitor
fatigue driving, where the non-i.i.d. setting was simulated by controlling the
number of images per client. The above FL frameworks for DMA did not really
take into account the actual situation of the application but artificially
created a simulation scenario. Therefore, there is an urgent need for
realistic analysis and research for real-world DMA, considering that the user
(driver) should exist independently and be non-interoperable with different
clients (vehicles). Moreover, in addition to the necessity of test datasets,
the test client is also a critical evaluation criterion, which can reflect the
universality of the FL framework. We summarize the existing neglects and
challenges in the current FL for DMA framework as follows.

  * •

Clients in FL for DMA frameworks are often defined in unreasonable and
incomprehensible forms. A real and natural definition of a client should be a
driver or a vehicle.

  * •

There is no paper proposing to test on a testing client (not involved in
training process), which lacks universal testing for the FL framework.

  * •

For DMA scenario, there is a great diversity and individuality of driver
behaviors, postures, and facial expressions, which call for more presonalized
studies than other general IoV scenarios.

  * •

Similarly, DMA also has diverse scenarios, including diverse vehicle models,
interior colors, seat positions, etc., which will greatly increase the
learning difficulty.

###  I-C Proposed Solution and Contribution

In this paper, we aim to propose a FL framework applicable and specific to
practical applications in IoV, especially DMA, where an imaginary FL framework
for IoV is illustrated in Fig. 1. Each local client, i.e., vehicle, includes a
training module and a perception module. The training module uploads the model
parameters to the server after learning and training the local data. After
aggregation and optimizing the parameters of the local client models, the
server downloads the global model parameters to the perception module in the
local client. Moreover, transfer learning can be used to reduce the number of
trainable parameters, resulting in reduced communication consumption. The
server can save different global models for different scenarios, such as road
types, weather types, and vehicle types, so that the model can have better
applicability.

Figure 1: Structure illustration of a FL framework for IoV. The server
interacts with the local client and saves different scenarios as different
models. Transparent neurons are non-trainable parameters, and non-transparent
neurons are trainable parameters.

Therefore, a federated transfer-ordered-personalized learning (FedTOP)
framework is proposed to address the problems of accuracy, cybersecurity,
communication resources, and diversified scenarios. In addition to the
transfer-extension shown in Fig. 1, the FedTOP framework also enhances
robustness and cybersecurity by orderly dropout clients due to their possible
overfitting and poisoning of the data. Furthermore, the FedTOP framework is
able to remarkably improve accuracy by adapting all clients through
personalized-extension. The contributions of this paper are:

  * •

For realistic problems and usage scenarios in DMA, we propose a feasible FL
framework FedTOP, realizing privacy protection, high accuracy, low
communication requirements, cybersecurity, and pervasiveness. To the best of
our knowledge, this is one of the first papers to establish a feasible FL
framework for DMA.

  * •

The proposed FedTOP framework is tested on two real-world driver monitoring
datasets with and without system heterogeneity, systematically characterizing
system heterogeneity in real-world datasets and achieving considerable
accuracies with 92.32$\%$ and 95.96$\%$, respectively.

  * •

The experiments highlight a realistic and natural client setup, i.e., drivers
and vehicles are naturally formed as clients. Moreover, we innovatively
propose evaluation criteria for training and testing clients to test the
generalization ability of the proposed FedTOP on different clients.

  * •

Through an ablation study, we demonstrate the performance and utility of the
transfer, ordered, and personalized extensions. These detachable extensions
can be selectively installed according to the task description, and the FL
framework combined with different extensions can effectively adapt to
different IoT application scenarios.

The presentation of this paper is as follows. The problem statement and
proposed solution are described in Section II. The experimental setup,
heterogeneity, and results have been demonstrated in Section III. Section IV
discusses the performances of three extensions of the proposed framework,
followed by Section V summarizing the paper and expounding on future work.

##  II Methodologies

###  II-A Problem Statement

FL framework protects privacy, increases training efficiency, and saves
communication resources by sharing only model parameters in IoT. In this
paper, the FL framework is used to solve a driver activity classification task
in DMA. Clients in real-world IoT are independent and heterogeneous due to the
presence of only a minimal number of users per client. Considering the more
general application scenarios, the global model $\omega$ for training clients
$C$ aggregation needs to be compatible with non-training clients $C^{\prime}$
in addition to $C$. The data of each client $D_{c}$ is non-i.i.d. when the
data is not interoperable. We can consider a nested model

$L_{c}=\omega_{c}(D_{c}),$ (1)

where $\omega_{c}$ is the classifier model corresponding to client $c\in C$.
$D_{c}\in\mathbb{R}^{n_{c}\times i\times j\times d}$ is the image set with
$n_{c}$ samples, $i$ rows, $j$ columns, and $d$ channels.
$L_{c}\in\mathbb{Z}^{n_{c}}$ is the corresponding label set. The global model
$\omega$ are obtained by aggregating, e.g., averaging the weights of the local
models,

$\omega=\sum_{c\in C}p_{c}\omega_{c}=\mathbb{E}[\omega_{c}|c\in C],$ (2)

where $p_{c}\in[0,1]$ is a weight density function of clients, for which $\sum
p_{c}=1$, $p_{c}$ will be assigned according to the number of samples.
Therefore, the optimization problem of the FL algorithm can be formulated as
minimizing the global loss, which is equivalent to minimizing the sum of the
local losses,

$\min_{\omega}\mathcal{L}(\omega)=\sum_{c\in
C}p_{c}\mathcal{L}(\omega_{c})=\mathbb{E}[\mathcal{L}(\omega_{c})|c\in C],$
(3)

where $\mathcal{L}$ is the loss function that will be assigned.

For real-world classification tasks, we assume that the distribution of the
local model in the parameter space presents a multivariate Normal distribution
$\omega_{c}\sim\mathcal{N}\left(\mu_{\omega},\sigma^{2}_{\omega}\right)$,
where $\mu_{\omega}$ is mean of all local models, and $\sigma^{2}_{\omega}$ is
the variance of all local models. Fig. 2 shows the process of the FL algorithm
finding the optimal solution of the global model in the parameter space. After
the initial model is trained locally, communicated, and aggregated globally,
the final global model will be obtained by averaging and can be estimated as
$\hat{\omega}=\mu_{\omega}$. Especially in the large-scale parallel
application scenarios of IoT, according to the law of large numbers,
$\hat{\omega}=\mu_{\omega}=\omega^{\ast}$ is an unbiased estimation.

However, there are still some defects in the method of obtaining the global
model through average aggregation. Firstly, we can confirm that there is
enormous system heterogeneity in IoT, and the global model cannot ensure high
accuracy for all clients. Secondly, we inevitably need a measure to prevent
system heterogeneity and potential attacks and poisoning. As shown in Fig. 2,
the farther the optimal local model is from the global model, the lower the
accuracy, and vice versa. Therefore, it is conceivable that in the FL problem
with heterogeneity, the clients’ accuracy will also obey a Normal
distribution.

Figure 2: Illustration of the FL algorithm finds the optimal global model
solution in the parameter space. The shaded areas are accuracy contour areas.
The farther the optimal local model dissociates from the global model, the
lower the client accuracy. Local models enclosed by shaded areas have similar
accuracies.

###  II-B Proposed Solution

According to the problem statement, we propose a FedTOP algorithm to address
all of the following issues. First, the aggregation of global models needs to
be more stable, which can be achieved by preventing the overfitting of local
models. Second, considering the actual communication situation in IoT, we
propose transfer learning to reduce the trainable parameters and hence reduce
communication requirements. Third, the global model should have the ability to
resist interference, attacks, and data poisoning, which can be achieved by
orderly dropping out local models with large loss. Fourth, a global model
cannot take into account the situation of all clients, especially in the
presence of data and system heterogeneity. Therefore, we recommend
personalizing the global model to suit all the training and testing clients.

We refer to FedProx [16] using a proximal term to prevent local models
$\omega_{c}$ from deviating from the global model $\omega$. In which, the
proximal item $\mathcal{L}_{p}$ that computes the distance between the local
and global model is added to the loss function,

$\mathcal{L}_{p}=\frac{\mu}{2}\|\omega_{c}-\omega\|^{2},$ (4)

where $\mu$ is deviation coefficient, $\omega_{c}$ is local client model
parameters, and $\omega$ is global model parameters. The overall loss function
can be updated as

$\mathcal{L}=\mathcal{L}_{l}+\mathcal{L}_{p},$ (5)

where $\mathcal{L}_{l}$ is the loss between the true labels and the predicted
labels, such as the negative log-likelihood loss used in our experiments.

Figure 3: The global model is shared with training and testing clients after
iterative training and optimization on massively parallel training clients.
Both training and testing clients are personalized locally and then get
results on the testing set, respectively. Among them, some attack or poison
clients will be discarded, such as Client 2 has a large loss.

Transfer-extension is a common and popular solution in many learning
frameworks. In particular, FL framework is favored because it can effectively
reduce local client training resources and communication resources. In our
experiments, the base model is ResNet34 [38] pre-trained on ImageNet, where
only the last residual block and fully connected layer are trainable
parameters. Although ImageNet is a large object classification dataset far
from DMA images, the lower layers are similar for convolutional neural
networks (CNN) and are used to extract image features. Therefore, the upper
layers that are used to obtain high-level features and representations are
given more attention. The ratio of reduced communication resource requirement
in the network is approximately equal to the ratio of non-trainable parameters
to total parameters,

$\text{Commun}_{\downarrow}\approx\frac{|\omega_{\text{non-
trainable}}|}{|\omega|}=37.46\%,$ (6)

where $\text{Commun}_{\downarrow}$ is the reduced communication resource
requirement, $|\omega_{\text{non-trainable}}|$ is the number of non-trainable
model parameters, and $|\omega|$ is the total number of the model parameters.
Therefore, the transfer-extension reduces the communication requirement by
37.46$\%$ by decreasing the trainable parameters.

Ordered-extension is for orderly dropout clients with enormous variance, which
may be subject to malicious attacks and poisoning, extensive data and system
heterogeneity, and model underfitting. These local clients with large losses
should be discarded to enhance the applicability of the global model. Ordered-
extension not only enhances accuracy and robustness but also secures the
global model. After all of the clients upload the local model parameters and
the final training loss to the server, the server only aggregates the
$q\in\mathbb{N}\leq|C|$ local models with the lowest loss as the global model.
The set of $q$ local models can be expressed as

$C_{q}\in q-\arg\min_{c\in C}\mathcal{L}(\omega_{c}).$ (7)

Algorithm 1 FedTOP

Input: Communication rounds ($T$), training client set ($C$), training epoch
($E$), initial global model ($\omega^{1}$), loss function ($\mathcal{L}_{l}$),
deviation coefficient ($\mu$), number of ordered clients ($q$)

Output: Trained global model ($\omega^{T}$)

for $t=1$ to $T-1$ do

for $c\in C$ in parallel do

for $e=1$ to $E-1$ do

Backpropagate the loss function and update the local model
$\omega_{c}^{t^{e+1}}\leftarrow\arg\min_{\omega_{c}^{t^{e}}}\mathcal{L}_{l}(\omega_{c}^{t^{e}})+\frac{\mu}{2}\|\omega_{c}^{t^{e}}-\omega^{t}\|^{2}$.

end for

Update the local model $\omega_{c}^{t}\leftarrow\omega_{c}^{t^{E}}$.

Client sends $\omega^{t}_{c}$ to the server.

end for

Find a set $C^{t}_{q}$ of top-$q$ clients in $C^{t}$ in term of loss values:
$C^{t}\in q-\arg\min_{c\in C^{t}}\mathcal{L}(\omega^{t}_{c})$.

Server aggregates the $\omega$ as $\omega^{t+1}\leftarrow\frac{1}{q}\sum_{q\in
C^{t}_{q}}\omega^{t}_{q}$.

end for

Send $\omega^{T}$ to clients $c\in\\{C,C^{\prime}\\}$ do personalization.

Algorithm 2 Personalized-extension

Input: Training client set ($C$), testing client set ($C^{\prime}$),
personalization epoch ($E$), Trained global model ($\omega^{T}$), loss
function ($\mathcal{L}_{l}$)

Output: Personalized local model ($\omega_{c}$)

for $c\in\\{C,C^{\prime}\\}$ do

for $e=1$ to $E-1$ do

Backpropagate the loss function and update the local model
$\omega_{c}^{T^{e+1}}\leftarrow\arg\min_{\omega_{c}^{T^{e}}}\mathcal{L}_{l}(\omega_{c}^{T^{e}})$.

end for

Update the personalized local model $\omega_{c}\leftarrow\omega_{c}^{T^{E}}$.

end for

Personalized-extension is to promote, popularize, and adapt the global model
to the heterogeneity of all clients. As shown in Fig. 2, the global model
cannot be applied to all clients due to the ubiquitous heterogeneity. The
region of interest (ROI) of the model may vary depending on system
heterogeneity, such as different camera angles, seat positions, and vehicle
structures, resulting in differences in the relative position of the driver in
the image. However, personalized-extension proposes to train the global model
several times in each client to obtain a more personalized local model to
improve accuracy. On the one hand, compared with the traditional FL algorithm,
the personalized-extension can significantly and effectively improve accuracy
and confidence. On the other hand, compared to the method that only trains
locally, the personalized FL algorithm improves the training efficiency and
avoids the overfitting of the local model. In particular, the personalized FL
algorithm can help and generalize to other non-training clients $C^{\prime}$,
which may have minimal training resources. After receiving the global model,
the non-training clients $C^{\prime}$ can obtain a highly accurate and
reliable local model with minimal training. The system diagram of the proposed
FedTOP is shown in Fig. 3

For the proposed FedTOP framework, the client communicates with the server $T$
rounds, and all clients $C$ train $E$ epochs in parallel between each
communication. For our preliminary experiments, we set $T=10$ and $E=5$. For
transfer-extension, the local model is the transfer learning model of ResNet34
pre-trained on ImageNet. Only the last residual block and fully connected
layer are set as trainable parameters. In addition, we add an additional fully
connected layer to match the number of our classification categories. Based on
FedProx, the activation function of the last layer is LogSoftmax, and the
setting of the loss function $\mathcal{L}_{l}$ is a negative log-likelihood
loss. $\omega^{1}$ is the initial model parameter. The proposed FedTOP is
described in Algorithm 1, and the personalization process is described in
Algorithm 2.

(a) SFDDD texting - right 1

(b) SFDDD texting - right 2

(c) SFDDD texting - right 3

(d) SFDDD texting - right 4

(e) DriveAct magazine 1

(f) DriveAct magazine 2

(g) DriveAct magazine 3

(h) DriveAct magazine 4

Figure 4: Exampled activities of four drivers in each of SFDDD and DriveAct
datasets.

(a) SFDDD

(b) DriveAct

Figure 5: Sampled client image histograms of SFDDD and DriveAct datasets.

##  III Experiment and Results

Considering the data and system heterogeneity, experiments are conducted on
two open real-world driver monitoring datasets, including State Farm
Distracted Driver Detection (SFDDD) [39] and DriveAct [40]. In addition to
comparing with FedProx as a baseline, this paper also compares the performance
of the transfer, ordered, and personalized extensions through an ablation
study.

###  III-A Experiment Setup

To compare the impact of system heterogeneity on FL frameworks, the proposed
FedTOP is tested on driver monitoring datasets with and without system
heterogeneity. SFDDD dataset includes 26 drivers and 10 activities, and
DriveAct dataset includes 15 drivers and 12 activities. SFDDD dataset
considers system heterogeneity, that is, different drivers have different
vehicles, different seat positions, different camera angles, etc., as shown in
Fig. 4a, 4b, 4c, and 4d. DriveAct dataset does not take into account system
heterogeneity, i.e., all subjects had their data collected in the same system.
Recorded from the same camera angle, different drivers read the same magazine
in the same vehicle, as shown in Fig. 4e, 4f, 4g, and 4h.

To show more clearly and visually the heterogeneity between different clients
in the two datasets, Fig. 5 shows histograms of the sample images of the two
datasets. It can be seen that the SFDDD dataset with system heterogeneity has
a more considerable difference in the distribution of histograms than the
DriveAct dataset without system heterogeneity, and the mean value of the SFDDD
images is larger. The possible reason is that the vehicle interiors of the
DriveAct dataset view are darker, resulting in most of the pixel values being
lower. Therefore, the FL framework may be more challenged by the scene
information when training on the SFDDD dataset, such as different vehicle
interiors.

Clients are naturally divided based on the drivers. In order to better
demonstrate the role of personalized-extension, the datasets are first divided
into training clients and testing clients at a ratio of about 0.8, 0.2, with
$|C_{\text{SFDDD}}|=20$, $|C^{\prime}_{\text{SFDDD}}|=6$,
$|C_{\text{DriveAct}}|=12$, and $|C^{\prime}_{\text{DriveAct}}|=3$. And then,
the datasets for each client are divided into a training set, verification
set, and testing set at a ratio of 0.7, 0.15, and 0.15, respectively. After
the global model is trained by the training dataset of training clients, the
final trained global model is shared with all clients for personalization. The
personalization of the global model will only be processed on the training
sets, while the personalized local model will be tested on the unseen testing
sets. The FL architectures are established on Pytorch and trained on an
Intel(R) Core(TM) i9-10850K CPU @ 3.60GHz, and a Nvidia GeForce RTX(TM) 3080
GPU.

###  III-B Ablation Study and Results

We explore the role of each FedTOP extension on two real-world datasets
through an ablation study. FedProx is used as a baseline for comparison.
According to the experimental setup described in the previous subsection, the
experimental results are shown in Table I.

TABLE I: Performance of FedTOP and ablation study on SFDDD and DriveAct
datasets.

Dataset | Method 1 | $|C|$ | $q$ | $\mu$ | Transfer | Accuracy ($\%$) 2 |  $\text{Time}_{\downarrow}$ ($\%$) 3 |  $\text{Commun}_{\downarrow}$ ($\%$) 4 | Cybersecurity
---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  | Training | Testing |  |  |
SFDDD | FedProx (baseline) | 20 | 20 | 1 | No | 54.63 | 16.44 | $\sim$ | $\sim$ | $\sim$
| FedOP | 20 | 15 | 1 | No | 97.69 | 96.37 | 1.45 $\downarrow$ | $\sim$ | $\uparrow$
| FedTP | 20 | 20 | 1 | Yes | 94.76 | 92.8 | 17.3 $\downarrow$ | 37.46 $\downarrow$ | $\sim$
| FedTO | 20 | 15 | 1 | Yes | 46.16 | 16.43 | 18.91 $\downarrow$ | 37.46 $\downarrow$ | $\uparrow$
| FedTOP | 20 | 15 | 1 | Yes | 94.65 | 92.32 |  18.91 $\boldsymbol{\downarrow}$ |  37.46 $\boldsymbol{\downarrow}$ |  $\boldsymbol{\uparrow}$
DriveAct | FedProx (baseline) | 12 | 12 | 1 | No | 73.18 | 23.96 | $\sim$ | $\sim$ | $\sim$
| FedOP | 12 | 10 | 1 | No | 98.07 | 97.97 | 0.44 $\downarrow$ | $\sim$ | $\uparrow$
| FedTP | 12 | 12 | 1 | Yes | 97.00 | 95.71 | 16.83 $\downarrow$ | 37.46 $\downarrow$ | $\sim$
| FedTO | 12 | 10 | 1 | Yes | 62.30 | 22.89 | 19.18 $\downarrow$ | 37.46 $\downarrow$ | $\uparrow$
| FedTOP | 12 | 10 | 1 | Yes | 97.04 | 95.96 |  19.18 $\boldsymbol{\downarrow}$ |  37.46 $\boldsymbol{\downarrow}$ |  $\boldsymbol{\uparrow}$

1 FedOP, FedTP, and FedTO refer to ablating the transfer, ordered, and
personalized extensions of the FL framework, respectively. 2 Accuracy refers
to the testing sets of training clients and testing clients, which is
described in Section III-A. 3 $\text{Time}_{\downarrow}$ refers to the ratio
of reduced training time per client to the baseline. 4
$\text{Commun}_{\downarrow}$ refers to ratio of reduced communication
consumption to the baseline, which is described in (6).

(a) FedProx

(b) FedT

(c) FedO

(d) FedTO

(e) FedProx

(f) FedT

(g) FedO

(h) FedTO

Figure 6: Accuracy and loss curves of the FL framework and its extensions on
the SFDDD and DriveAct datasets, which is the training process of Algorithm 1.
Personalization does not affect the convergence of the global model in the FL
framework.

(a) SFDDD TOP

(b) DriveAct TOP

Figure 7: Testing accuracy of the training and testing clients on both SFDDD
and DriveAct datasets varies with personalized epoch, which is the testing
results of Algorithm 2.

The results and comparisons for two datasets and three extensions are shown in
Fig. 6, which is equivalent to demonstrating Algorithm 1. By observing the
accuracy and loss curves on the two datasets, it can be concluded that the
SFDDD dataset with system heterogeneity is fundamentally different from the
DriveAct dataset without system heterogeneity. It can be clearly seen that the
SFDDD dataset with system heterogeneity requires more communication to
converge, while the DriveAct dataset without system heterogeneity has a fast
convergence speed, especially at the first communication. Therefore, for real-
world datasets, system heterogeneity can be mitigated by more communication
times.

By observing Fig. 6c, 6d, 6g, and 6h, it can be found that the ordered-
extension diminishes the stability of the system. Although the anomalous
large-loss local model is discarded to reduce the bias of the global model, it
also increases the variance of the global model resulting in reduced
generalizability. By observing Fig. 6b, 6d, 6f, and 6h, we can see that the
effect of transfer-extension is different for datasets with and without system
heterogeneity. On the one hand, transfer-extension increases the variance of
the model on the SFDDD dataset and leads to a reduced and unstable model
convergence. On the other hand, transfer-extension improves the speed of model
convergence on DriveAct, and the convergence effect is more stable. The
possible reason is that the transfer-extension retains only a small number of
trainable parameters, resulting in the neural network model not being able to
learn human behavioral features effectively in the SFDDD dataset with system
heterogeneity. However, for the DriveAct dataset without system heterogeneity,
the factors are constant except for the driver, and the local model does not
need to focus on these exact same pixels, but only on the changing pixels,
including objects such as drivers, computers, and magazines. Therefore, for
the DriveAct dataset, transfer-extension can effectively increase convergence
and stability. The proposed FedTOP framework is able to obtain 92.32$\%$ and
95.96$\%$ accuracy on the SFDDD and DriveAct datasets, respectively, when
considering five times of personalization training. Compared to FedProx as a
baseline, FedTOP can effectively improve the accuracy by 462$\%$ in addition
to considering a 37.46$\%$ reduction in communication resources. The results
demonstrate the feasibility of the proposed FedTOP in terms of communication
resource saving, accuracy improvement, robustness, and cybersecurity.

(a) Trained global model $\omega^{T}$

(b) Personalization Epoch 1 $\omega^{T^{1}}$

(c) Personalization Epoch 3 $\omega^{T^{3}}$

(d) Personalization Epoch 5 $\omega^{T^{5}}$

(e) Trained global model $\omega^{T}$

(f) Personalization Epoch 1 $\omega^{T^{1}}$

(g) Personalization Epoch 3 $\omega^{T^{3}}$

(h) Personalization Epoch 5 $\omega^{T^{5}}$

Figure 8: CAMs of the test clients in SFDDD and DriveAct datasets during the
personalization process. (a), (b), (c), and (d) are a test client in the SFDDD
dataset, which is the same as Fig. 4a. (e), (f), (g), and (h) are a test
client in the DriveAct dataset, which is the same as Fig. 4e.

###  III-C Performance of Personalized-Extension

Personalized-extension needs to be further discussed and analyzed as the most
effective approach to improve accuracy. Based on the division of training and
testing clients in Section III-A, in this subsection, we further discuss how
the trained and aggregated global model is adapted to both training and
testing clients. The results of the personalized-extension on the two datasets
are shown in Fig. 7 with different personalization epochs, which is equivalent
to demonstrating Algorithm 2. It can be seen that the personalization process
differs significantly on the datasets with and without system heterogeneity,
which is similar to the results in Fig. 6. The clients in the DriveAct dataset
have faster convergence, minor accuracy variance, and higher final accuracy.
On the contrary, the clients in the SFDDD dataset not only converge slower but
also have an anomalous client with relatively low accuracy. The possible
reason is that the anomalous client has a huge data and system heterogeneity,
causing the optimal model to deviate significantly far from the aggregated
global model.

Fig. 8 further demonstrates that the trained global model repositions the ROI
during the personalized training process via class activation map (CAM) [41].
The test client of the SFDDD dataset can be seen struggling with the
personalization process. The trained global model focus the ROI on the seat
backrest, driver’s chest, hand, and knee, and vehicle door. Due to the system
heterogeneity present in the SFDDD dataset, the positions of the driver, seat,
and steering wheel, as shown in Fig. 8a is different from other clients, as
shown in Fig. 4b, 4c, and 4d. Therefore, the initial ROI is likely to be a
driver’s position among other clients. During the five personalization
training processes, the local model is able to effectively reposition the ROI
to the driver, which is what the personalized-extension is intended to show.
Moreover, the personalization process also reduces the number of ROIs while
targeting more attention to a specific area.

On the contrary, for the test clients in the DriveAct dataset, the adjustment
of the ROI is negligible. Note that the ROI does not necessarily have to cover
the driver’s body or an object such as the magazine. The ROI should cover
those pixels that can distinguish between different activities, such as static
activities like reading the magazine, and dynamic activities like wearing a
seatbelt in the DriveAct dataset activity setting. These ROIs focus on areas
where large differences are likely to occur. The fact that the ROIs in the
DriveAct dataset cover almost the same pixels during the personalization
process can also prove the negative impact of system heterogeneity on the FL
framework.

##  IV Discussion

The two datasets used, SFDDD and DriveAct, still have some flaws. First,
although the SFDDD dataset takes system heterogeneity into account, quite a
few drivers collect data in the same vehicle, that is, the number of clients
is greater than the number of users. Therefore, there are still some
differences between the dataset and the real-world data, which leads to the
fact that the proposed FedTOP may need more communication rounds to achieve
similar accuracy on a real-world dataset. Second, there is currently no driver
monitoring dataset with real poisoning data currently existing, resulting in
the effect of ordered-extension not being reflected. The different modalities,
positions, and angles of the camera or the method of generating fake data may
be a hypothesis for poisoned data, but it cannot be highlighted as real.
Moreover, due to road safety guidelines, the current dataset is only driving
on safe roads or simulated driving. Therefore, the driver’s posture, demeanor,
facial concentration, etc., are far from the real driving behavior. Therefore,
there is an urgent need for a more realistic dataset that can include camera
images of different positions and angles, different vehicle scenes, and more
drivers driving on real roads.

For a FL framework in IoT, in addition to accuracy being the evaluation
criterion, factors like communication requirements, robustness, fairness,
cybersecurity, etc., also need to be considered. Although it seems that
transfer and ordered extensions may not improve accuracy but rather reduce it
in the current experimental results, it can potentially improve the
performance of the FL framework. Therefore, we keep two extensions as one of
our future directions. Personalized-extension is an approach similar to
transfer learning and incremental learning. On the one hand, the local client
is incrementally learned based on the trained global model, but it does not
intentionally retain the previously learned knowledge. On the other hand, the
global model is transferred to the client dataset as in transfer-extension,
but the low-level non-trainable weights are still pre-trained on ImageNet.
Therefore, the proposed personalized-extension actually uses the trained
global model weights to fit different client data, such as the reposition of
ROIs. Although the personalized-extension requires additional training locally
for each client, there are many benefits, including high accuracy,
applicability to non-training clients, customization, etc. Conceivably,
personalized-extension can effectively address the problem of system
heterogeneity, e.g., it can be applied to different cameras, camera angles,
vehicle interiors, etc.

##  V Conclusion

In this paper, we propose a FL framework FedTOP for DMA to address the issues
of privacy preservation, efficient training, communication resource-saving,
poisoned data, and diversified scenarios. Through the ablation study, the
impact, role, and performance of three extensions, including transfer,
ordered, and personalized on the model are disclosed. Moreover, the
experiments demonstrate dramatic differences between datasets with and without
system heterogeneity. In addition to the proposed FedTOP being able to exhibit
92.32$\%$ and 95.96$\%$ accuracy in two datasets for testing clients, it is
also appreciated that FedTOP reduces communication consumption by 37.46$\%$
and potentially improves cybersecurity. The experimental results show that the
proposed FedTOP is a highly accurate, lightweight, privacy-preserving, robust,
cybersecure, and universally applicable FL framework for potential DMA.

Future work lies in the continued research of extensions. For the ordered-
extension, a possible plan is to introduce some malicious local clients to
attack and poison with the global model. For example, subjects may not place
the camera on the side as instructed but place it on the front or behind
instead. Such outliers may cause the global model to deviate significantly
from the optimal solution, so in the case, ordered-expansion can prevent the
deviation of the global model by discarding the larger value of the losses.
For the transfer-extension, there is currently a lack of a general driver
monitoring model, so we used a model pre-trained on ImageNet. Future work can
pre-train a driver model ourselves as a base model, which will get better
performance in DMA. Fig. 1 shows the FL framework for foresight in IoV, but
the dataset used does not contain scenario information such as road, weather,
vehicle models, etc. Therefore, we expect a well-developed real-world dataset
to include such scenario information, data and system heterogeneity, etc.

## References

  * [1] Y. Yang, L. Wu, G. Yin, L. Li, and H. Zhao, “A survey on security and privacy issues in internet-of-things,” _IEEE Internet Things J._ , vol. 4, no. 5, pp. 1250–1258, Apr. 2017.
  * [2] J. Cao, Y. Pang, J. Xie, F. S. Khan, and L. Shao, “From handcrafted to deep features for pedestrian detection: a survey,” _IEEE Trans. Pattern Anal. Mach. Intell._ , Apr. 2021.
  * [3] S. Kuutti, S. Fallah, K. Katsaros, M. Dianati, F. Mccullough, and A. Mouzakitis, “A survey of the state-of-the-art localization techniques and their potentials for autonomous vehicle applications,” _IEEE Internet Things J._ , vol. 5, no. 2, pp. 829–846, Mar. 2018.
  * [4] Z. Wang, R. Gupta, K. Han, H. Wang, A. Ganlath, N. Ammar, and P. Tiwari, “Mobility digital twin: Concept, architecture, case study, and future challenges,” _IEEE Internet Things J._ , Mar. 2022.
  * [5] A. Hard, K. Rao, R. Mathews, S. Ramaswamy, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, and D. Ramage, “Federated learning for mobile keyboard prediction,” _arXiv preprint arXiv:1811.03604_ , 2018.
  * [6] I. Dayan, H. R. Roth, A. Zhong, A. Harouni, A. Gentili, A. Z. Abidin, A. Liu, A. B. Costa, B. J. Wood, C.-S. Tsai _et al._ , “Federated learning for predicting clinical outcomes in patients with covid-19,” _Nat. Med._ , vol. 27, no. 10, pp. 1735–1743, Oct. 2021.
  * [7] N. Rieke, J. Hancox, W. Li, F. Milletari, H. R. Roth, S. Albarqouni, S. Bakas, M. N. Galtier, B. A. Landman, K. Maier-Hein _et al._ , “The future of digital health with federated learning,” _NPJ Digit. Med._ , vol. 3, no. 1, pp. 1–7, Sep. 2020.
  * [8] M. Hao, H. Li, X. Luo, G. Xu, H. Yang, and S. Liu, “Efficient and privacy-enhanced federated learning for industrial artificial intelligence,” _IEEE Trans. Industr. Inform._ , vol. 16, no. 10, pp. 6532–6542, Oct. 2019\.
  * [9] Y. Lu, X. Huang, Y. Dai, S. Maharjan, and Y. Zhang, “Blockchain and federated learning for privacy-preserved data sharing in industrial iot,” _IEEE Trans. Industr. Inform._ , vol. 16, no. 6, pp. 4177–4186, Sep. 2019.
  * [10] Z. Du, C. Wu, T. Yoshinaga, K.-L. A. Yau, Y. Ji, and J. Li, “Federated learning for vehicular internet of things: Recent advances and open issues,” _IEEE open j. Comput. Soc._ , vol. 1, pp. 45–61, May 2020.
  * [11] X. Kong, K. Wang, M. Hou, X. Hao, G. Shen, X. Chen, and F. Xia, “A federated learning-based license plate recognition scheme for 5g-enabled internet of vehicles,” _IEEE Trans. Industr. Inform._ , vol. 17, no. 12, pp. 8523–8530, Mar. 2021.
  * [12] F. Sattler, S. Wiedemann, K.-R. Müller, and W. Samek, “Robust and communication-efficient federated learning from non-iid data,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 31, no. 9, pp. 3400–3413, Nov. 2019.
  * [13] S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learning,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 5132–5143.
  * [14] S. Horvath, S. Laskaridis, M. Almeida, I. Leontiadis, S. Venieris, and N. Lane, “Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout,” _Adv. Neural Inf. Process. Syst._ , vol. 34, pp. 12 876–12 889, 2021.
  * [15] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial intelligence and statistics_. PMLR, 2017, pp. 1273–1282.
  * [16] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” _Proceedings of Machine Learning and Systems_ , vol. 2, pp. 429–450, 2020.
  * [17] B. Ghimire and D. B. Rawat, “Recent advances on federated learning for cybersecurity and cybersecurity for federated learning for internet of things,” _IEEE Internet Things J._ , Feb. 2022.
  * [18] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” _IEEE Signal Process. Mag._ , vol. 37, no. 3, pp. 50–60, May 2020.
  * [19] S. Niknam, H. S. Dhillon, and J. H. Reed, “Federated learning for wireless communications: Motivation, opportunities, and challenges,” _IEEE Commun. Mag._ , vol. 58, no. 6, pp. 46–51, Jul. 2020.
  * [20] L. Lyu, H. Yu, and Q. Yang, “Threats to federated learning: A survey,” _arXiv preprint arXiv:2003.02133_ , Mar. 2020.
  * [21] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings _et al._ , “Advances and open problems in federated learning,” _Found. Trends Mach. Learn._ , vol. 14, no. 1–2, pp. 1–210, Jun. 2021.
  * [22] Q. Li, Z. Wen, Z. Wu, S. Hu, N. Wang, Y. Li, X. Liu, and B. He, “A survey on federated learning systems: vision, hype and reality for data privacy and protection,” _IEEE Trans. Knowl. Data Eng._ , Nov. 2021.
  * [23] L. Zhang, H. Saito, L. Yang, and J. Wu, “Privacy-preserving federated transfer learning for driver drowsiness detection,” _IEEE Access_ , vol. 10, pp. 80 565–80 574, Jul. 2022.
  * [24] Z. Su, Y. Wang, T. H. Luan, N. Zhang, F. Li, T. Chen, and H. Cao, “Secure and efficient federated learning for smart grid with edge-cloud collaboration,” _IEEE Trans. Industr. Inform._ , vol. 18, no. 2, pp. 1333–1344, Jul. 2021\.
  * [25] G. Sun, Y. Cong, J. Dong, Q. Wang, L. Lyu, and J. Liu, “Data poisoning attacks on federated machine learning,” _IEEE Internet Things J._ , Nov. 2021.
  * [26] J. Zhang, B. Chen, X. Cheng, H. T. T. Binh, and S. Yu, “Poisongan: Generative poisoning attacks against federated learning in edge computing systems,” _IEEE Internet Things J._ , vol. 8, no. 5, pp. 3310–3322, Sep. 2020.
  * [27] A. Z. Tan, H. Yu, L. Cui, and Q. Yang, “Towards personalized federated learning,” _IEEE Trans. Neural Netw. Learn. Syst._ , Mar. 2022.
  * [28] A. Fallah, A. Mokhtari, and A. Ozdaglar, “Personalized federated learning: A meta-learning approach,” _arXiv preprint arXiv:2002.07948_ , Feb. 2020.
  * [29] Q. Wu, X. Chen, Z. Zhou, and J. Zhang, “Fedhome: Cloud-edge based personalized federated learning for in-home health monitoring,” _IEEE Trans. Mob. Comput._ , Dec. 2020.
  * [30] A. Kashevnik, I. Lashkov, and A. Gurtov, “Methodology and mobile application for driver behavior analysis and accident prevention,” _IEEE Trans. Intell. Transp. Syst._ , vol. 21, no. 6, pp. 2427–2436, Jun. 2019.
  * [31] S. Zepf, J. Hernandez, A. Schmitt, W. Minker, and R. W. Picard, “Driver emotion recognition for intelligent vehicles: A survey,” _ACM Comput. Surv._ , vol. 53, no. 3, pp. 1–30, Jun. 2020.
  * [32] Y. Xing, C. Lv, H. Wang, H. Wang, Y. Ai, D. Cao, E. Velenis, and F.-Y. Wang, “Driver lane change intention inference for intelligent vehicles: framework, survey, and challenges,” _IEEE Trans. Veh. Technol._ , vol. 68, no. 5, pp. 4377–4390, Mar. 2019.
  * [33] A. Masood, D. S. Lakew, and S. Cho, “Security and privacy challenges in connected vehicular cloud computing,” _IEEE Commun. Surv._ , vol. 22, no. 4, pp. 2725–2764, Jul. 2020.
  * [34] S. Kuutti, R. Bowden, Y. Jin, P. Barber, and S. Fallah, “A survey of deep learning applications to autonomous vehicle control,” _IEEE Trans. Intell. Transp. Syst._ , vol. 22, no. 2, pp. 712–733, Jan. 2020.
  * [35] M. Ramzan, H. U. Khan, S. M. Awan, A. Ismail, M. Ilyas, and A. Mahmood, “A survey on state-of-the-art drowsiness detection techniques,” _IEEE Access_ , vol. 7, pp. 61 904–61 919, May 2019.
  * [36] K. Doshi and Y. Yilmaz, “Federated learning-based driver activity recognition for edge devices,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 3338–3346.
  * [37] C. Zhao, Z. Gao, Q. Wang, K. Xiao, Z. Mo, and M. J. Deen, “Fedsup: A communication-efficient federated learning fatigue driving behaviors supervision approach,” _Future Gener. Comput. Syst._ , vol. 138, pp. 52–60, Jan. 2023.
  * [38] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [39] State Farm, “State farm distracted driver detection,” Apr 2016. [Online]. Available: https://www.kaggle.com/competitions/state-farm-distracted-driver-detection/overview/description
  * [40] M. Martin, A. Roitberg, M. Haurilet, M. Horne, S. Reiß, M. Voit, and R. Stiefelhagen, “Drive&act: A multi-modal dataset for fine-grained driver behavior recognition in autonomous vehicles,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 2801–2810.
  * [41] D. Omeiza, S. Speakman, C. Cintas, and K. Weldermariam, “Smooth grad-cam++: An enhanced inference level visualization technique for deep convolutional neural network models,” _arXiv preprint arXiv:1908.01224_ , 2019.

|  Liangqi Yuan (S’22) received the B.E. degree from the Beijing Information
Science and Technology University, Beijing, China, in 2020, and the M.Sc.
degree from the Oakland University, Rochester, MI, USA, in 2022. He is
currently pursuing the Ph.D. degree with the School of Electrical and Computer
Engineering, Purdue University, West Lafayette, IN, USA. His research
interests are in the areas of sensors, the internet of things, human–computer
interaction, signal processing, and machine learning.
---|---
|  Lu Su (M’15) is an associate professor in the School of Electrical and
Computer Engineering at Purdue University. His research interests are in the
general areas of Internet of Things and Cyber-Physical Systems, with a current
focus on wireless, mobile, and crowd sensing systems. He received Ph.D. in
Computer Science, and M.S. in Statistics, both from the University of Illinois
at Urbana-Champaign, in 2013 and 2012, respectively. He has also worked at IBM
T. J. Watson Research Center and National Center for Supercomputing
Applications. He has published more than 100 papers in referred journals and
conferences, and serves as an associate editor of ACM Transactions on Sensor
Networks. He is the recipient of NSF CAREER Award, University at Buffalo Young
Investigator Award, ICCPS’17 best paper award, and the ICDCS’17 best student
paper award. He is a member of ACM and IEEE.
---|---
|  Ziran Wang (S’16-M’19) received the Ph.D. degree from the University of
California, Riverside in 2019. He is an Assistant Professor in the College of
Engineering at Purdue University, and was a Principal Researcher at Toyota
Motor North America. He serves as Founding Chair of IEEE Technical Committee
on Internet of Things in Intelligent Transportation Systems, and Associate
Editor of four academic journals, including IEEE Internet of Things Journal
and IEEE Transactions on Intelligent Vehicles. His research focuses on
automated driving, human-autonomy teaming, and digital twin.
---|---


# No-resonance conditions, random matrices, and quantum chaotic models

Jonathon Riddell<EMAIL_ADDRESS>School of Physics and
Astronomy, University of Nottingham, Nottingham, NG7 2RD, UK  Nathan
Pagliaroli<EMAIL_ADDRESS>Department of Mathematics, Western University,
1151 Richmond St, London ON N6A 3K7, Canada

###### Abstract

In this article we investigate no-resonance conditions for quantum chaotic and
random matrix models. No-resonance conditions are properties on the spectrum
of a model, usually employed as a theoretical tool in the analysis of late
time dynamics. The first order no-resonance condition holds when a spectrum is
non-degenerate, while higher order no-resonance conditions imply sums of an
equal number of energies are non-degenerate outside of permutations of the
indices. The condition is usually assumed to hold for quantum chaotic models.
In this work we use several tests from random matrix theory to demonstrate
that no-resonance conditions are likely to be violated for all equal sums
containing greater than one energy. This is due to the presence of level-
attraction in the spectra after resolving appropriate symmetries. This result
is produced for both a quantum chaotic Hamiltonian and two random matrix
models. We then generalize important bounds in quantum equilibration theory to
a case where the conditions are violated, and to the case of random matrix
models.

One of the most ubiquitous observations in many body physics is the connection
between the spectral statistics of many body quantum systems and that of
random matrices. Quantum systems are not chaotic in the classical sense since
unitary time evolution guarantees that the overlap between two states in time
is constant. This excludes the classical notion of chaos in quantum systems
for which we observe exponential sensitivity to small differences in initial
conditions. However, their spectral statistics behave qualitatively
differently if their corresponding classical limit is integrable or chaotic.
If the classical limit is chaotic, the spectral statistics of the quantum
Hamiltonian agree with the predictions of Random Matrix Theory (random) and we
refer to these models as quantum chaotic Berry (1987). The notion of quantum
chaos can be extended to quantum systems that do not have a well-defined
classical limit D’Alessio  _et al._ (2016).

An extremely important property of the spectral statistics of a quantum
chaotic Hamiltonian is the presence of level-repulsion amongst neighboring
energies. Originally this level-repulsion was first modeled for heavy atomic
nuclei by Wigner using Gaussian ensembles of random matrices. Since Wigner’s
work, it has been established that features of the spectrum of classically
chaotic quantum systems are accurately described by various ensembles of
random matricesPorter (1965); Brody  _et al._ (1981); Guhr  _et al._ (1998);
Berry and Tabor (1976, 1977). The connection between the spectrum of quantum
chaotic systems and random matrices has been well studied in single particle
systems Jalabert  _et al._ (1990); Marcus  _et al._ (1992); Milner  _et al._
(2001); Friedman  _et al._ (2001); Stockmann and Stein (1990); Sridhar (1991);
Moore  _et al._ (1994); Steck  _et al._ (2001); Hensinger  _et al._ (2001);
Chaudhury  _et al._ (2009); Weinstein  _et al._ (2002); Zhang  _et al._
(2022); Łydżba  _et al._ (2021a, b), along with many body systems Santos and
Rigol (2010a, b); Rigol (2010); Kollath  _et al._ (2010); Santos  _et al._
(2012); Richter  _et al._ (2020); Atas  _et al._ (2013a, b);
$\boldsymbol{S}$untajs  _et al._ (2020) and recently has seen a surge of
interest in the case of circuit or periodically driven type models Chan  _et
al._ (2018a); D’Alessio and Rigol (2014); Bertini  _et al._ (2021a, 2018). The
first to extend Wigner’s work were Dyson and Mehta in the series of papers
Dyson (1962a, b, c); Dyson and Mehta (1963); Mehta and Dyson (1963). In
particular, Dyson classified the three most immediately relevant ensembles:
the Gaussian Unitary Ensemble, the Gaussian Orthogonal Ensemble, and the
Gaussian Symmplectic Ensemble in what is known as the “threefold way” Dyson
(1962d). Of the most immediate interest to this work is the Gaussian
Orthogonal Ensemble (GOE). The Bohigas, Giannoni, and Schmit (BGS) conjecture
Bohigas  _et al._ (1984) states that the GOE has the same level-spacing as a
wide class of quantum systems with classical limits Bohrdt  _et al._ (2017);
Andreev  _et al._ (1996); Müller  _et al._ (2004). Let $E_{0}\leq E_{1}\leq
E_{2},...$ be a sequence of unfolded energy eigenvalues of the GOE; then
Wigner surmised the distribution of average consecutive level-spacings, that
is the average of $s_{k}=E_{k+1}-E_{k}$ for all $k$ is

$p(s)=\frac{\pi s}{2}e^{-\pi^{2}s^{2}/4}.$ (1)

To see how to unfold a spectrum see Chapter 6 of Mehta (2004) or for example
Bruus and Angl‘es d’Auriac (1997). It is important to note that Wigner’s
Surmise is an approximation Mehta (1960) of the actual distribution,
originally derived in Jimbo  _et al._ (1980). This was further simplified in
terms of Painlevé transcendentals Forrester and Witte (2000).

In contrast to level-repulsion, if one considers the level-spacing of i.i.d.
random variables, not only does one not see repulsion, but rather one sees
attraction Livan  _et al._ (2018), which has been used as a marker for non-
chaotic systems D’Alessio  _et al._ (2016). In particular after unfolding the
spacing of such systems, the distribution is Poisson

$p(s)=e^{-s}.$ (2)

The presence of level-repulsion and GOE spectral statistics is a hallmark test
of Quantum chaos, while Poisson statistics are associated with integrable or
non-chaotic models.

A key consequence of the presence of level-repulsion is that the value of the
probability density at zero is zero, meaning that we can assume with high
probability that we will not find degeneracies in the quantum chaotic
spectrum. This observation is useful, for example, when considering dephasing
arguments, which has recently been particularly popular in the quantum
equilibration community Alhambra  _et al._ (2020); Riddell and Sørensen
(2020); Gogolin and Eisert (2016); Masanes  _et al._ (2013); Wilming  _et al._
(2018); Heveling  _et al._ (2020); Campos Venuti and Zanardi (2010);
Knipschild and Gemmer (2020); Carvalho  _et al._ (2023). If we consider the
time-evolution of many dynamical functions under unitary dynamics, time-
dependent terms in the series will often appear as the following:

$z\,e^{-i(E_{m}-E_{n})t},$ (3)

where $z$ is a complex number and $t$ is time. Terms such as these survive the
infinite time average if and only if $E_{m}=E_{n}$. In the case of quantum
chaotic Hamiltonians it is a safe assumption that any surviving term would
imply that $m=n$, since we do not expect degeneracy due to the presence of
level-repulsion. The cases where $E_{m}=E_{n}$ and $m\not=n$ are referred to
as resonances. However, in general dynamical functions can be more complex
with terms such as

$z\,e^{-i(E_{m_{1}}-E_{n_{1}}+E_{m_{2}}-E_{n_{2}}+...)t}.$ (4)

Such terms can, for example, appear in out of time ordered correlators or
other higher order correlation functions Riddell and Sørensen (2020); Riddell
_et al._ (2021); Riddell and Sørensen (2019); Yoshida and Yao (2019); Fortes
_et al._ (2019); Shukla  _et al._ (2022); Fortes  _et al._ (2020). To discuss
the terms that survive the infinite time average in equation 4 we introduce
the qth order no-resonance condition.

######  Definition 1.

Let $H$ be a Hamiltonian with spectrum
$H=\sum_{j}E_{j}\ket{E_{j}}\bra{E_{j}}$, and let
$\Lambda_{q},\Lambda^{\prime}_{q}$ be two arbitrary sets of $q$ energy levels
$\\{E_{j}\\}$. $H$ satisfies the q no-resonance condition if for all
$\Lambda_{q},\Lambda^{\prime}_{q}$, the equality

$\sum_{j\in\Lambda_{q}}E_{j}=\sum_{j\in\Lambda^{\prime}_{q}}E_{j}$ (5)

implies that $\Lambda_{q}=\Lambda^{\prime}_{q}$.

By definition 1 the set of terms that satisfy the q no-resonance condition are
the minimum set of terms that survive the infinite time average as in equation
4. Terms that fall outside of definition 1 are referred to as $q$-resonances.
Typically in the literature it is suggested that quantum chaotic Hamiltonians
satisfy definition 1 Mark  _et al._ (2022); Srednicki (1999); Riddell  _et
al._ (2022). This greatly simplifies arguments involving infinite time
averages in quantum chaotic models. Despite this condition being somewhat
common in the literature, studies only test this condition for the $q=1$ case
where one finds level-repulsion governed by the Wigner-Dyson distribution
Richter  _et al._ (2020); D’Alessio  _et al._ (2016). As for the $q=2$ case,
an explicit formula is known for the density of states Khalkhali and
Pagliaroli (2022), but as far the authors can tell nothing is known about the
level-spacing distribution. However, as we will see, the numerical simulations
performed in this paper strongly suggest that for the GOE the $q=2$ level-
spacing distribution is Poisson. In the appendix we numerically demonstrate
that $q=3,4$ also appear Poisson and have level-attraction. We then conjecture
that all level spacing distributions for $q\geq 2$ have level-attraction and
appear Poissonian.

##  I Spectral statistics for a quantum chaotic Hamiltonian

In this section we first investigate what the spectral statistics look like
for a specific quantum chaotic model. In particular we study a Heisenberg type
model with nearest and next nearest neighbour interactions.

$\displaystyle{H}=$
$\displaystyle\sum_{j=1}^{L}J_{1}\left({S}_{j}^{+}S_{j+1}^{-}+\text{h.c.}\right)+\gamma_{1}\,{S}_{j}^{Z}{S}_{j+1}^{Z}$
(6)
$\displaystyle+J_{2}\left({S}_{j}^{+}{S}_{j+2}^{-}+\text{h.c.}\right)+\gamma_{2}{S}_{j}^{Z}{S}_{j+2}^{Z},$
(7)

where $(J_{1},\gamma_{1},J_{2},\gamma_{2})\nobreak=\nobreak(-1,1,-0.2,0.5)$
gives us a non-integrable model. This model has a free limit for
$(J_{1},0,0,0)$ and an interacting integrable limit for
$(J_{1},\gamma_{1},0,0)$. Recently this model was confirmed to obey the
eigenstate thermalization hypothesis LeBlond  _et al._ (2019). We perform full
spectrum exact diagonalization in the maximally symmetric sector of this
model. In particular, this matrix conserves the total magnetization
$m_{z}=\sum_{j}S_{j}^{Z}$, and is translation invariant. We choose to work in
the sector such that $\langle m_{z}\rangle=0$ with quasi-momenta $k=0$. This
allows us to further diagonalize the model with the spatial reflection
symmetry $P$ and the spin inversion symmetry $Z$.

In this section we will focus on the spectral statistics for the cases $q=1$,
as a benchmark, and $q=2$, the first non-resonance condition that is
unexplored in the literature. As we will show in the appendix, the behavior
for $q>2$ is qualitatively similar to $q=2$. First, let us establish that our
model satisfies the usual tests for quantum chaos in the $q=1$ case. Perhaps
the most common test is to investigate the level spacing distribution
$s_{j}=E_{j+1}-E_{j}$. The act of unfolding allows us to have a universal
scale for the comparison of spectra of different Hamiltonians. The
distribution of $s_{j}$ for a quantum chaotic model should be a Wigner
surmise. To unfold the spectrum we use Gaussian broadening. Namely we map our
energies $E_{k}$ to $\epsilon_{k}$ in the following way Bruus and Angl‘es
d’Auriac (1997),

$\epsilon_{k}=N(E_{k}),$ (8)

$N(E)=\int_{-\infty}^{E}\sum_{k}\frac{1}{\sigma_{k}\sqrt{2\pi}}e^{-\frac{(e-E_{k})^{2}}{2\sigma_{k}^{2}}}de,$
(9)

where we use the same convention as in Bruus and Angl‘es d’Auriac (1997) and
take

$\sigma_{k}=0.608\alpha\Delta_{k},$ (10)

where $\Delta=(E_{k+\alpha}-E_{k-\alpha})/{2\alpha}$ and we find that
$\alpha=20$ is quite suitable for our spectrum.

Figure 1:  (a) Level spacing $q=1$, $L=24$ unfolded data, which looks
approximately like a Wigner-surmise exhibiting level propulsion. In black we
plot the Wigner-surmise and in purple we plot the Poisson distribution. (b)
Ratio test for spectral statistics at $L=24$. In black we plot the
corresponding GOE distribution.

Fig. 1 demonstrates that our model for $q=1$ has level-repulsion and appears
to have a level spacing distribution well approximated by the Wigner surmise.
While this result shows us that our spectrum strongly resembles the
predictions of RMT, the unfolding procedure is usually chosen to find such
agreement, therefore it is desirable to perform a test that does not need
unfolding. Such a test is given by investigating the distribution of ratios
between successive gaps Łydżba  _et al._ (2021c); Oganesyan and Huse (2007);
Atas  _et al._ (2013c). We introduce the ratios

$r_{j}=\frac{\min\\{s_{j},s_{j+1}\\}}{\max\\{s_{j},s_{j+1}\\}},$ (11)

which tells us that $r_{j}\in[0,1]$. We emphasize that the $s_{j}$ we use here
don’t need to be unfolded gaps. This test can be done with the model’s
physical spectrum. For the GOE in Atas  _et al._ (2013c) it was analytically
shown that the distribution of the $r_{j}$ for $3\times 3$ matrices is given
by

$p(r)=\frac{27}{4}\frac{r+r^{2}}{\left(1+r+r^{2}\right)^{\frac{5}{2}}}.$ (12)

If instead our energy levels were independent randomly distributed variables
we would instead get level-attraction,

$p(r)=\frac{2}{(1+r)^{2}}.$ (13)

We see in Fig. 1 (b) that our result experiences level-repulsion, agreeing
with the distribution in equation 12.

Next we consider the case for $q=2$. The spectrum we are now interested in is
equivalent to the spectrum of the Hamiltonian,

$\hat{H}_{2}=\hat{H}\otimes\mathbb{I}+\mathbb{I}\otimes\hat{H},$ (14)

which has the spectrum $\Lambda_{k,l}=E_{k}+E_{l}$. This construction
introduces an unwanted symmetry in the spectrum of $\hat{H}_{2}$, namely that
$\Lambda_{k,l}=\Lambda_{l,k}$, that is, the spectrum is invariant under
permutations of the individual energies’ indices. For $q=2$ this might be
understood as a spatial reflection symmetry for a larger two component non-
interacting system. Addressing this symmetry is simple. We only consider
unique pairs of $(k,l)$, namely, we take $l>k$, where we also ignore the
portion of the spectrum where $k=l$. Ignoring $k=l$ does not appear to
significantly alter the results but allows us to eliminate trivial multiples
of the $q=1$ spectrum. In fact, the contribution of the $q=l$ portion of the
spectrum is vanishingly small compared to the total size of our spectrum. We
further introduce a new index that orders the spectrum $\alpha=1,2\dots$ such
that $\Lambda_{\alpha}<\Lambda_{\alpha+1}$. With this new spectrum we can
analyze the level spacing and ratio distribution.

Figure 2:  (a) Level spacing $q=2$, $L=20$ unfolded data, which looks
approximately like a Poisson distribution. (b) Ratio test for spectral
statistics at $L=20$ for $q=2$. In both plots we draw the GOE prediction in
black and the independent random variable prediction (Poisson) in purple.

Fig. 2 indicates that the spectrum of $\hat{H}_{2}$ experiences level-
attraction. This is contrary to the $q=1$ case which has level-repulsion.
Importantly this indicates that the spectrum of $\hat{H}_{2}$ behaves like an
integrable model, and has gaps clustered around $s=0$. While this does not
guarantee violations of the $q=2$ no-resonance theorem, it does make
violations more likely. Likewise, we expect a large amount of pseudo
violations such that $s_{j}=\Lambda_{j+1}-\Lambda_{j}\approx 0$, meaning
unless very large time scales (potentially non-physically large) are
considered these violations would appear as resonances in the spectrum.
Considering this fact, results such as Short (2011); Riddell  _et al._ (2022);
Srednicki (1999); Mark  _et al._ (2022) should be investigated to understand
the effects of resonances. In the appendix B we demonstrate that the Poisson
statistics and level-attraction persist for higher values of $q$ and
conjecture that level-attraction persists for all values of $q>1$.

One further test we can perform is to test the actual average value of $r$ we
observe in the ratio distribution. $\langle r\rangle=2\ln 2-1\approx
0.38629436112$ for Poisson systems and $\langle r\rangle=4-2\sqrt{3}\approx
0.535898384$ for the GOE. Testing this quantity allows us to clearly observe
convergence to the predictions of random matrix theory as a function of system
size. We see this test in Fig. 3. In the right panel we see the test for $q=2$
which reveals a strong convergence in agreement with the Poisson predictions.
The data at $L=22$ gives $\langle r\rangle=0.386294325894$, which confirms
seven decimal points of convergence. Therefore, from the perspective of short
range correlations in the spectrum we conclude that $\hat{H}_{2}$ obeys
Poisson statistics, and importantly, that the $q=2$ case experiences level-
attraction.

Figure 3:  Plotted data for the convergence of $\langle r\rangle$ to RMT
predictions. In black we plot the GOE predictions and in purple we plot the
corresponding Poisson prediction. Left: We see the $q=1$ data converge to the
GOE prediction as a function of system size. Right: We see the $q=2$ data.

In appendix B, we demonstrate that this level-attraction persists for higher
values of $q$ and speculate that for all $q>1$ the spectrum must experience
level-attraction. In appendix A we repeat our numerical studies but for random
matrices, showing that our results from a quantum chaotic Hamiltonian agree
with the results of RMT. Importantly our tests here are local tests on the
spectrum. It is an open question if the symmetry resolved Hamiltonian
$\hat{H}_{2}$ will still obey Poisson statistics for more complex tests such
as investigating the spectral form factor Bertini  _et al._ (2021b); Chan  _et
al._ (2018b). We leave this question to future work. An important observation
from this work is that when studying $q>1$ we never find cases where the
spectral statistics aren’t Poisson. In some models such as the tight binding
model for free fermions we expect $p(s)$ to be a delta function in the the
thermodynamic limit. Crucially, this implies the $q$ no resonance condition is
robustly violated for free fermions. We do not observe any such behavior in
the spectral statistics for $q>1$ when starting with a chaotic Hamiltonian
implying violations should be rare or non-existent.

We emphasize that the presence of level-attraction does not imply violations
of the $q>1$ no-resonance condition. It does, however, imply the gaps in the
spectrum of $\hat{H}_{2}$ cluster close to zero. If we investigate the
probability of finding a gap within the range $0<s<\epsilon$, where $\epsilon$
is small, we have for the GOE,

$\int_{0}^{\epsilon}\frac{\pi
s}{2}e^{-\pi^{2}s^{2}/4}ds=\frac{1-e^{-\frac{-\pi^{2}\epsilon^{2}}{4}}}{\pi}\approx\frac{\pi\epsilon^{2}}{4}-\frac{\pi^{3}\epsilon^{4}}{32}\dots,$
(15)

so we see the probability is proportional to $\epsilon^{2}$ for small gaps. On
the contrary for the Poisson distribution one intuitively yields something
much larger,

$\int_{0}^{\epsilon}e^{-s}ds=\sinh\epsilon-\cosh\epsilon+1\approx\epsilon-\frac{\epsilon^{2}}{2}\dots,$
(16)

giving us only linear scaling for small gaps. While both probabilities are of
course small, the GOE is significantly smaller, giving one a significantly
stronger case to assume definition 1 is satisfied in your chaotic model. In
the case of Poisson statistics, one might expect to find one or many gaps that
are essentially zero due to level-attraction. Infinite time averages are
theoretical tools for which we average over times significantly longer than
the Heisenberg time $\tau_{H}\sim e^{S}$, where $S$ is the thermodynamic
entropy at the appropriate energy $E$ Srednicki (1999). The presence of
essentially zero gaps will lead to terms $e^{i(E_{k}-E_{k-1})t}$ which are
stationary on time scales proportional to $\tau_{H}$. Despite the presence of
such violators, we expect the set of problematic gaps to be small relative to
the total Hilbert space dimension. Since it is likely that some violations or
cases that are indistinguishable from violations of definition 1 are
inevitable, especially for cases using $q>1$, it is instructive to revisit
past results keeping in mind a small number of violations will most likely be
present. Below we discuss modifying key results in the field of quantum
equilibration theory to accommodate the presence of violations of definition
1.

##  II Equilibration and recurrence

###  II.1 Physical models

In this section we tackle the problem of equilibration in light of our
investigation of the higher order no-resonance conditions and the presence of
level-attraction. First, let us review a basic setup. Consider a time
independent system with the Hamiltonian $\hat{H}$ where we label the energy
eigen basis as $\hat{H}|E_{k}\rangle=E_{k}|E_{k}\rangle$. For simplicity, we
take the spectrum of $\hat{H}$ to be discrete and finite. We will initialize
our system in some pure state

$|\psi(t=0)\rangle=\sum_{m}c_{m}|E_{k}\rangle.$ (17)

To track equilibration, we study properties of the expectation value of an
observable $\hat{A}$. This observable is general, but we demand that the
largest single value $||A||$ is independent of system size or saturates to a
finite value in the thermodynamic limit. In what follows we will assume our
spectrum has level-repulsion, so that we may safely assume,

$E_{m}=E_{l}\implies m=l$ (18)

.

If our observable equilibrates, its finite time value
$\langle\hat{A}(t)\rangle=\langle\psi(t)|\hat{A}|\psi(t)\rangle$ must relax to
its infinite time average value i.e.

$\bar{A}=\lim_{T\to\infty}\frac{1}{T}\int_{0}^{T}\langle\hat{A}(t)\rangle
dt=\lim_{T\to\infty}\frac{1}{T}\int_{0}^{T}\sum_{m,n}\bar{c}_{m}c_{n}A_{m,n}e^{i(E_{m}-E_{n})t}dt=\sum_{m}|c_{m}|^{2}A_{m,m}.$
(19)

$\bar{A}$ is usually written in terms of the diagonal ensemble
$\omega=\sum_{m}|c_{m}|^{2}|E_{m}\rangle\langle E_{m}|$ as
$\bar{A}=\operatorname{Tr}\left(\omega\hat{A}\right)$. A typical quantity to
study in quantum equilibration would be the variance of the expectation value
around $\bar{A}$. This was studied and bounded in Short (2011); Reimann and
Kastner (2012) assuming that the $q=2$ no-resonance condition was satisfied.
The variance is written as

$\mu_{2}=\lim_{T\to\infty}\frac{1}{T}\int_{0}^{T}\left(\langle\hat{A}(t)\rangle-\bar{A}\right)^{2}dt.$
(20)

It was famously found in Short (2011) that this variance can be bounded by the
purity of the diagonal ensemble

$\mu_{2}\leq||A||^{2}\operatorname{Tr}\left(\omega^{2}\right).$ (21)

Note equation 21 holds as a consequence of the $q=2$ no-resonance condition
holding. The purity of the diagonal ensemble usually decays exponentially fast
with respect to the system size (see for example Fig. 2 in Riddell  _et al._
(2022)). If one assumes higher order $q$ no-resonance conditions, it was
recently found that, for higher moments,

$\mu_{q}=\lim_{T\to\infty}\frac{1}{T}\int_{0}^{T}\left(\langle\hat{A}(t)\rangle-\bar{A}\right)^{q}dt,$
(22)

a similar bound can be found Riddell  _et al._ (2022),

$|\mu_{q}|\leq\left(q||A||\sqrt{\operatorname{Tr}\left(\omega^{2}\right)}\right)^{q}.$
(23)

In light of section I and the presence of level-attraction for higher order
$q$, these results should be updated to reflect the high probability of there
being a violation of the $q$ no-resonance condition.

######  Theorem 1.

Suppose we have a model that has violations of the $q$ no-resonance condition.
Then the moments $\mu_{q}$ can be bounded as

$|\mu_{q}|\leq||A||^{q}\left(q^{q}+\frac{\mathcal{N}_{q,L}}{2q}\right)\sqrt{\operatorname{Tr}\left(\omega^{2}\right)}^{q},$
(24)

where $\mathcal{N}_{q,L}$ is the maximum number of times one $E_{m}$ appears
in violations of the $q$ no-resonance condition for a given system size $L$.
We call the $E_{m}$’s that appear in more than one violation of the resonance
condition exceptional violators.

###### Proof.

Terms that contribute to $\mu_{q}$ are sums of energies that are equal. Let
$\Lambda_{q}$ and $\Lambda_{q}^{\prime}$ be sets of indices corresponding to
particular energies,

$\sum_{m\in\Lambda_{q}}E_{m}=\sum_{m\in\Lambda_{q}^{\prime}}E_{m}.$ (25)

The no-resonance condition picks out the trivial set of energies that satisfy
this equality, which is when $\Lambda_{q}=\Lambda_{q}^{\prime}$. These
contributions were bounded in Short (2011); Riddell  _et al._ (2022). We
collect the remaining violations in a set $\mathcal{S}$ and write,

$|\mu_{q}|\leq\left(q||A||\sqrt{\operatorname{Tr}\left(\omega^{2}\right)}\right)^{q}+\left|\sum_{\Lambda_{q}\in\mathcal{S}}\prod_{j=1}^{q}\bar{c}_{m_{j}}c_{n_{j}}A_{m_{j},n_{j}}\right|,$
(26)

where we have identified $\Lambda_{q}\in\mathcal{S}=\\{m_{j},n_{j}\\}$. The
second term can be bounded as follows.

$\left|\sum_{\Lambda_{q}\in\mathcal{S}}\prod_{j=1}^{q}\bar{c}_{m_{j}}c_{n_{j}}A_{m_{j},n_{j}}\right|\leq||A||^{q}\sum_{\Lambda_{q}\in\mathcal{S}}\prod_{j=1}^{q}|c_{m_{j}}||c_{n_{j}}|.$
(27)

Since all $|c_{m_{j}}|$ are positive, we may use the inequality of arithmetic
and geometric means, giving

$\leq\frac{||A||^{q}}{2q}\sum_{\Lambda_{q}\in\mathcal{S}}\sum_{j=1}^{q}\left(|c_{m_{j}}|^{2q}+|c_{n_{j}}|^{2q}\right).$
(28)

We know that $\operatorname{Tr}\left(\omega^{q}\right)=\sum_{m}|c_{m}|^{2q}$.
Assuming an individual $|c_{m_{j}}|^{2q}$ contributes at most
$\mathcal{N}_{q,L}$ times, we have that

$\leq\frac{||A||^{q}\mathcal{N}_{q,L}}{2q}\operatorname{Tr}\left(\omega^{q}\right).$
(29)

We lastly recall that
$\operatorname{Tr}\left(\omega^{q}\right)\leq\operatorname{Tr}\left(\omega^{2}\right)^{q/2}$,
which completes the proof. ∎

Accommodating the presence of degenerate gaps for the $q=2$ case has been
considered before in Short and Farrelly (2012). Our bound reads,

$|\mu_{2}|\leq||A||^{2}\left(1+\frac{\mathcal{N}_{2,L}}{4}\right)\operatorname{Tr}\left(\omega^{2}\right).$
(30)

Instead, one can likewise write Short and Farrelly (2012) as

$|\mu_{2}|\leq N(\epsilon)||A||^{2}\operatorname{Tr}\left(\omega^{2}\right),$
(31)

where $N(\epsilon)$ is the maximum number of energy gaps in any interval for
$\epsilon>0$, i.e.

$N(\epsilon)=\max_{E}|\\{(k,l)|\enspace E_{k}-E_{l}\in[E,E+\epsilon)\\}|.$
(32)

One can recover the maximum degeneracy of the gaps by considering
$\lim_{\epsilon\to 0^{+}}N(\epsilon)$. In the limit of non-degenerate gaps
these bounds are identical, and only differ by a constant factor for a small
number of degeneracies in the gaps. Our result might in theory give better
constant factors than the result in Short and Farrelly (2012), however
$N(\epsilon)$ is likely a more intuitive quantity and easier to work with
numerically.

We next wish to understand the properties of $\mathcal{N}_{q,L}$, which in
practice is challenging to study numerically. The worst scaling it could have
is the total number of violations, i.e.
$0\leq\mathcal{N}_{q,L}\leq|\mathcal{S}|$. As we have noted earlier, the
presence of level-attraction does not imply $|\mathcal{S}|>0$. An easy
property to understand however is that if $\mathcal{N}_{q,L}\geq 2$ this
implies at the very least that $\mathcal{N}_{q+1,L}\geq 1$. To see this
consider $q=2$ for an exceptional violator $E_{m}$ that appears at least
twice. We might have $E_{m}$ as an exceptional violator as,

$E_{m}+E_{n}=E_{p}+E_{l},\enspace E_{m}+E_{k}=E_{r}+E_{h}.$ (33)

This implies for $q=3$, a violation of the no-resonance condition is

$E_{p}+E_{l}+E_{k}=E_{r}+E_{h}+E_{n}.$ (34)

Despite two exceptional violations for $q=2$ implying at least one for $q=3$,
this does not imply $\mathcal{N}_{q,L}$ is decreasing in $q$.

To get a handle on the size of $\mathcal{N}_{q,L}$ we can attempt to quantify
the expected or average behavior of the quantity. First, let us assume we
randomly generated the set $S$. We will assume that the indices which appear
are uniformly generated, so each element of $S$ can be understood to be a
tuple of $2q$ indices, $(m_{1},\dots m_{2q})$. These indices are not
necessarily independent. For example, they cannot be equal to each other under
our assumptions. Despite this, in the large $L$ limit this dependence cannot
effect results due to the smallness of $q$ and the corresponding exponential
nature of the number of possible indices $2^{L}$. We can therefore focus on
the first index of each tuple $m_{1}$. Our goal will be to predict the average
number of times $m_{1}$ ends up being the same index. It can at most appear
$|S|$ times, and thus we wish to compute

$\langle\mathcal{N}_{q,L}\rangle=\sum_{n=1}^{|S|}np(n),$ (35)

where $p(n)$ is the probability of the same index appearing $n$ times. The
total number of configurations possible for the first index of each tuple in
$S$ is $2^{|S|L}$, and therefore we must simply count the number of
configurations where $n$ copies of the same $m_{1}$ appear. This is given by

${|S|\choose n}2^{L\left(|S|-n\right)},$ (36)

which gives the following formula for our expected value

$\langle\mathcal{N}_{q,L}\rangle=\sum_{n=0}^{|S|}\frac{n{|S|\choose
n}}{2^{Ln}}=\frac{|S|}{2^{L}}(2^{-L}+1)^{|S|-1}.$ (37)

We now have some special limiting cases to consider. Suppose that $|S|\propto
c2^{L}$ for some constant $c$. Then the expected value of
$\langle\mathcal{N}_{q,L}\rangle$ is $c\,e^{c}$ as $L$ goes to infinity.
However, if $|S|$ has sub exponential growth, for example if it scales as $L$,
then the expected value goes to zero for large system size as
$\mathcal{O}(L/2^{L})=\mathcal{O}(|S|/2^{L})$. Therefore we expect that for
most cases, even with modest violations of the no-resonance condition we
expect $\lim_{L\to\infty}N_{q,L}$ to be finite and quite small.

###  II.2 A Random Matrix Theory Approach

In this section we will show how one could compute $\mu_{q}$ for the GUE and
GOE with an unfolded spectrum in the large $N$ limit. We can rewrite equation
22 for finite $T$ as

$\mu_{q}(T)=\sum_{i_{1},j_{1},...,i_{q},j_{q}}\left(\prod_{k=1}^{q}c_{i_{k}}\overline{c}_{j_{k}}\langle
i_{k}|A|j_{k}\rangle\right)\frac{1}{T}\int_{0}^{T}e^{i\sum_{k=1}^{q}(\lambda_{i_{k}}-\lambda_{j_{k}})t}dt,$
(38)

where the eigenvalues are unfolded and from either a $N\times N$ GUE or GOE
distributed matrix. We define its moments as its expectation value.

Define the $n$-level spectral form factor as

$\mathcal{K}_{2n}^{\overline{\beta}}(t)=\frac{1}{N^{2n}}\left\langle\sum_{i_{1},j_{1},...,i_{n},j_{n}=1}^{N}e^{it\sum_{k=1}^{n}\lambda_{i_{k}}-\lambda_{j_{k}}}\right\rangle_{\overline{\beta}},$

where the subscript $\overline{\beta}=1$ or 2 denotes the GOE and GUE
expectation values, respectively. Then we may express the expectation values
of $\mu_{q}$ in terms of its its $q$-level spectral form factor

$\displaystyle\langle\mu_{q}(T)\rangle_{\overline{\beta}}$
$\displaystyle=\lim_{N\rightarrow\infty}\sum_{i_{1},j_{1},...,i_{q},j_{q}}\left(\prod_{k=1}^{q}c_{i_{k}}\overline{c}_{j_{k}}\langle
i_{k}|A|j_{k}\rangle\right)\frac{1}{T}\int_{0}^{T}\left.\langle
e^{i\sum_{k=1}^{q}(\lambda_{i_{k}}-\lambda_{j_{k}})t}\right.\rangle_{\overline{\beta}}dt$
(39)
$\displaystyle=\left(\frac{1}{T}\int_{0}^{T}\mathcal{K}_{2|B|}^{\overline{\beta}}(t)dt\right)\sum_{i_{1}\not=j_{1},...,i_{q}\not=j_{q}}\left(\prod_{k=1}^{q}c_{i_{k}}c_{j_{k}}\langle
i_{k}|A|j_{k}\rangle\right)$ (40)
$\displaystyle=\left(\frac{1}{T}\int_{0}^{T}\mathcal{K}_{2|B|}^{\overline{\beta}}(t)dt\right)(\operatorname{Tr}(A\rho-\omega))^{q}.$
(41)

It is also worth noting that this equation is so general that it applies to
any random matrix ensemble. Usually the GUE and GOE are of interest, but
progress has been made studying the spectral form factor for other matrix
ensembles. For example see Forrester (2021a, b); Cipolloni  _et al._ (2023).

The $q$-level spectral form factor can be computed explicitly, but it is a
computationally heavy task. For example see Liu (2018), where it is computed
but for ensembles that are not unfolded. In particular, for the GOE and GUE,
the 2-level spectral form factor of the unfolded spectrum has a well-known
explicit formula in the large $N$ limit Mehta (2004). This leads to the
following result.

######  Theorem 2.

For any fixed $T$ greater than zero, the GOE and GUE the expectation value of
$\mu_{2}(T)$ goes to zero as $1/N^{2}$ in the large $N$ limit. Furthermore, if
$T$ goes to infinity at the same rate as $N$ (i.e. $N=T$) then $\mu_{2}(T)$
goes to zero as $1/T$.

###### Proof.

From Mehta (2004); Cipolloni  _et al._ (2023), we know that for large $N$ the
spectral form factors can be approximated by

$\mathcal{K}_{2}^{1}(t)\approx\left\\{\begin{array}[]{lr}\frac{4t}{\pi
N^{2}}+\frac{2t}{\pi N^{2}}\ln(1+\frac{4t}{\pi N})&\text{if }0\leq
t\leq\frac{\pi N}{2}\\\ \frac{2}{N}+\frac{2t}{\pi
N^{2}}\ln\left(\frac{\frac{4t}{\pi N}+1}{\frac{4t}{\pi N}-1}\right)&\text{if
}t\geq\frac{\pi N}{2}\end{array}\right.$ (42)

and

$\mathcal{K}_{2}^{2}(t)\approx\left\\{\begin{array}[]{lr}\frac{2t}{\pi
N^{2}}&\text{if }0\leq t\leq\frac{\pi N}{2}\\\ \frac{1}{N}&\text{if
}t\geq\frac{\pi N}{2}\end{array}\right..$ (43)

Clearly, the first part of every piecewise function will dominate for large
$N$, thus completing the first claim.

Next, set $T=N$. Taking the above quantities time averages we get

$\frac{1}{T}\int_{0}^{T}\mathcal{K}_{2}^{1}(t)dt\approx\frac{1}{T}\left(\frac{3}{2\,\pi}-{\frac{\pi}{16}\ln\left(1+{\frac{4}{\pi}}\right)}+{\frac{1}{\pi}\ln\left(1+{\frac{4}{\pi}}\right)}+{\frac{3\,\pi}{32}}+{\frac{1}{4}}\right)$
(44)

and

$\frac{1}{T}\int_{0}^{T}\mathcal{K}_{2}^{2}(t)dt\approx\frac{1}{\pi\,T}.$ (45)

This proves the second claim. ∎

As we demonstrate in appendix A, the spectrum of the random matrix Hamiltonian
likewise experiences level-attraction for $q\geq 2$. However, despite the
presence of level-attraction, the above RMT result indicates that we still
should expect $\mu_{q}\to 0$ indicating equilibration on average of our
observable.

##  III Conclusion

In this work we have explored spectral statistics of chaotic Hamiltonians,
namely the statistics surrounding sums of energies. We found that despite
being chaotic, sums of energies displayed Poisson statistics instead of
Wigner-Dyson statistics. This was demonstrated numerically for both a chaotic
spin Hamiltonian and the GOE. The presence of level-attraction leads one to
believe that accounting for potential degeneracies or “resonances” in infinite
time averages of some dynamical quantities is necessary. We applied this
observation to the theory of equilibration where we generalized known bounds
to accommodate for degeneracies. Assuming the number of degeneracies is not
exponentially large in system size, we demonstrated that the the bounds can be
easily generalized to accomondate the presence of resonances. We further used
techniques from RMT to prove that, for the GOE, moments of equilibration go to
zero in the thermodynamic limit.

##  IV Acknowledgements

J.R. would like to thank Bruno Bertini, Marcos Rigol and Alvaro Alhambra for
fruitful conversations. J.R. would like to extend special thanks in particular
to Bruno who gave valuable feedback at various stages of the project. J.R.
acknowledges the support of Royal Society through the University Research
Fellowship No. 201101. N.J.P. acknowledges support from the Natural Sciences
and Engineering Research Council of Canada (NSERC).

## References

  * Berry (1987) M. V. Berry, Proc. R. Soc. A 413, 183 (1987).
  * D’Alessio  _et al._ (2016) L. D’Alessio, Y. Kafri, A. Polkovnikov, and M. Rigol, Advances in Physics 65, 239 (2016).
  * Porter (1965) C. E. Porter,  _Statistical theories of spectra: fluctuations_ , Tech. Rep. (1965).
  * Brody  _et al._ (1981) T. A. Brody, J. Flores, J. B. French, P. Mello, A. Pandey, and S. S. Wong, Reviews of Modern Physics 53, 385 (1981).
  * Guhr  _et al._ (1998) T. Guhr, A. Müller-Groeling, and H. A. Weidenmüller, Physics Reports 299, 189 (1998).
  * Berry and Tabor (1976) M. V. Berry and M. Tabor, Proc. R. Soc. A 349, 101 (1976).
  * Berry and Tabor (1977) M. V. Berry and M. Tabor, Proc. R. Soc. A 356, 375 (1977).
  * Jalabert  _et al._ (1990) R. A. Jalabert, H. U. Baranger, and A. D. Stone, Phys. Rev. Lett. 65, 2442 (1990).
  * Marcus  _et al._ (1992) C. M. Marcus, A. J. Rimberg, R. M. Westervelt, P. F. Hopkins, and A. C. Gossard, Phys. Rev. Lett. 69, 506 (1992).
  * Milner  _et al._ (2001) V. Milner, J. L. Hanssen, W. C. Campbell, and M. G. Raizen, Phys. Rev. Lett. 86, 1514 (2001).
  * Friedman  _et al._ (2001) N. Friedman, A. Kaplan, D. Carasso, and N. Davidson, Phys. Rev. Lett. 86, 1518 (2001).
  * Stockmann and Stein (1990) H. J. Stockmann and J. Stein, Phys. Rev. Lett. 64, 2215 (1990).
  * Sridhar (1991) S. Sridhar, Phys. Rev. Lett. 67, 785 (1991).
  * Moore  _et al._ (1994) F. L. Moore, J. C. Robinson, C. Bharucha, P. E. Williams, and M. G. Raizen, Phys. Rev. Lett. 73, 2974 (1994).
  * Steck  _et al._ (2001) D. A. Steck, W. H. Oskay, and M. G. Raizen, Science 293, 274 (2001).
  * Hensinger  _et al._ (2001) W. K. Hensinger, H. Häffner, A. Browaeys, N. R. Heckenberg, K. Helmerson, C. McKenzie, G. J. Milburn, W. D. Phillips, S. L. Rolston, H. Rubinsztein-Dunlop, and B. Upcroft, Nature 412, 52 (2001).
  * Chaudhury  _et al._ (2009) S. Chaudhury, A. Smith, B. E. Anderson, S. Ghose, and P. S. Jessen, Nature 461, 768 (2009).
  * Weinstein  _et al._ (2002) Y. S. Weinstein, S. Lloyd, J. Emerson, and D. G. Cory, Phys. Rev. Lett. 89, 157902 (2002).
  * Zhang  _et al._ (2022) Y. Zhang, L. Vidmar, and M. Rigol, Phys. Rev. E 106, 014132 (2022).
  * Łydżba  _et al._ (2021a) P. Łydżba, Y. Zhang, M. Rigol, and L. Vidmar, Phys. Rev. B 104, 214203 (2021a).
  * Łydżba  _et al._ (2021b) P. Łydżba, M. Rigol, and L. Vidmar, Phys. Rev. B 103, 104206 (2021b).
  * Santos and Rigol (2010a) L. F. Santos and M. Rigol, Phys. Rev. E 81, 036206 (2010a).
  * Santos and Rigol (2010b) L. F. Santos and M. Rigol, Phys. Rev. E 82, 031130 (2010b).
  * Rigol (2010) M. Rigol, ArXiv e-prints (2010), arXiv:1008.1930 [cond-mat.stat-mech] .
  * Kollath  _et al._ (2010) C. Kollath, G. Roux, G. Biroli, and A. M. Läuchli, Journal of Statistical Mechanics: Theory and Experiment 2010, P08011 (2010).
  * Santos  _et al._ (2012) L. F. Santos, A. Polkovnikov, and M. Rigol, Physical Review E 86 (2012), 10.1103/physreve.86.010102.
  * Richter  _et al._ (2020) J. Richter, A. Dymarsky, R. Steinigeweg, and J. Gemmer, Physical Review E 102 (2020), 10.1103/physreve.102.042127.
  * Atas  _et al._ (2013a) Y. Y. Atas, E. Bogomolny, O. Giraud, and G. Roux, Phys. Rev. Lett. 110, 084101 (2013a).
  * Atas  _et al._ (2013b) Y. Y. Atas, E. Bogomolny, O. Giraud, P. Vivo, and E. Vivo, Journal of Physics A: Mathematical and Theoretical 46, 355204 (2013b).
  * $\boldsymbol{S}$untajs  _et al._ (2020) J. $\boldsymbol{S}$untajs, J. Bon$\boldsymbol{c}$a, T. c. v. Prosen, and L. Vidmar, Phys. Rev. E 102, 062144 (2020).
  * Chan  _et al._ (2018a) A. Chan, A. De Luca, and J. Chalker, Phys. Rev. X 8, 041019 (2018a).
  * D’Alessio and Rigol (2014) L. D’Alessio and M. Rigol, Phys. Rev. X 4, 041048 (2014).
  * Bertini  _et al._ (2021a) B. Bertini, P. Kos, and T. Prosen, Communications in Mathematical Physics 387, 597 (2021a).
  * Bertini  _et al._ (2018) B. Bertini, P. Kos, and T. c. v. Prosen, Phys. Rev. Lett. 121, 264101 (2018).
  * Dyson (1962a) F. J. Dyson, Journal of Mathematical Physics 3, 140 (1962a).
  * Dyson (1962b) F. J. Dyson, Journal of Mathematical Physics 3, 157 (1962b).
  * Dyson (1962c) F. J. Dyson, Journal of Mathematical Physics 3, 166 (1962c).
  * Dyson and Mehta (1963) F. J. Dyson and M. L. Mehta, Journal of Mathematical Physics 4, 701 (1963).
  * Mehta and Dyson (1963) M. Mehta and F. Dyson, Journal of Mathematical Physics (New York)(US) 4 (1963).
  * Dyson (1962d) F. J. Dyson, Journal of Mathematical Physics 3, 1199 (1962d).
  * Bohigas  _et al._ (1984) O. Bohigas, M.-J. Giannoni, and C. Schmit, Physical review letters 52, 1 (1984).
  * Bohrdt  _et al._ (2017) A. Bohrdt, C. B. Mendl, M. Endres, and M. Knap, New. J. Phys. 19, 063001 (2017).
  * Andreev  _et al._ (1996) A. Andreev, O. Agam, B. Simons, and B. Altshuler, Physical review letters 76, 3947 (1996).
  * Müller  _et al._ (2004) S. Müller, S. Heusler, P. Braun, F. Haake, and A. Altland, Physical review letters 93, 014103 (2004).
  * Mehta (2004) M. L. Mehta,  _Random matrices_ (Elsevier, 2004).
  * Bruus and Angl‘es d’Auriac (1997) H. Bruus and J.-C. Angl‘es d’Auriac, Phys. Rev. B 55, 9142 (1997).
  * Mehta (1960) M. L. Mehta, Nuclear Physics 18, 395 (1960).
  * Jimbo  _et al._ (1980) M. Jimbo, T. Miwa, Y. Mori, and M. Sato, Physica D: Nonlinear Phenomena 1, 80 (1980).
  * Forrester and Witte (2000) P. Forrester and N. Witte, Letters in Mathematical Physics 53, 195 (2000).
  * Livan  _et al._ (2018) G. Livan, M. Novaes, and P. Vivo, Monograph Award 63 (2018).
  * Alhambra  _et al._ (2020) Á. M. Alhambra, J. Riddell, and L. P. García-Pintos, Phys. Rev. Lett. 124, 110605 (2020).
  * Riddell and Sørensen (2020) J. Riddell and E. S. Sørensen, Phys. Rev. B 101, 024202 (2020).
  * Gogolin and Eisert (2016) C. Gogolin and J. Eisert, Rep. Prog. Phys. 79, 056001 (2016).
  * Masanes  _et al._ (2013) L. Masanes, A. J. Roncaglia, and A. Acín, Phys. Rev. E 87, 032137 (2013).
  * Wilming  _et al._ (2018) H. Wilming, T. R. de Oliveira, A. J. Short, and J. Eisert, in  _Thermodynamics in the Quantum Regime_ (Springer, 2018) pp. 435–455.
  * Heveling  _et al._ (2020) R. Heveling, L. Knipschild, and J. Gemmer, Journal of Physics A: Mathematical and Theoretical 53, 375303 (2020).
  * Campos Venuti and Zanardi (2010) L. Campos Venuti and P. Zanardi, Phys. Rev. A 81 (2010), 10.1103/physreva.81.022113.
  * Knipschild and Gemmer (2020) L. Knipschild and J. Gemmer, Physical Review E 101 (2020), 10.1103/physreve.101.062205.
  * Carvalho  _et al._ (2023) G. D. Carvalho, L. F. dos Prazeres, P. S. Correia, and T. R. de Oliveira, (2023), arXiv:2305.11985 [quant-ph] .
  * Riddell  _et al._ (2021) J. Riddell, W. Kirkby, D. H. J. O’Dell, and E. S. Sørensen, “Scaling at the otoc wavefront: Integrable versus chaotic models,” (2021), arXiv:2111.01336 [cond-mat.stat-mech] .
  * Riddell and Sørensen (2019) J. Riddell and E. S. Sørensen, Physical Review B 99, 054205 (2019).
  * Yoshida and Yao (2019) B. Yoshida and N. Y. Yao, Phys. Rev. X 9, 011006 (2019).
  * Fortes  _et al._ (2019) E. M. Fortes, I. García-Mata, R. A. Jalabert, and D. A. Wisniacki, Phys. Rev. E 100, 042201 (2019).
  * Shukla  _et al._ (2022) R. K. Shukla, A. Lakshminarayan, and S. K. Mishra, Phys. Rev. B 105, 224307 (2022).
  * Fortes  _et al._ (2020) E. M. Fortes, I. García-Mata, R. A. Jalabert, and D. A. Wisniacki, Europhysics Letters 130, 60001 (2020).
  * Mark  _et al._ (2022) D. K. Mark, J. Choi, A. L. Shaw, M. Endres, and S. Choi, “Benchmarking quantum simulators using quantum chaos,”  (2022).
  * Srednicki (1999) M. Srednicki, Journal of Physics A: Mathematical and General 32, 1163 (1999).
  * Riddell  _et al._ (2022) J. Riddell, N. Pagliaroli, and Á. M. Alhambra, arXiv preprint arXiv:2206.07541 (2022).
  * Khalkhali and Pagliaroli (2022) M. Khalkhali and N. Pagliaroli, Journal of Mathematical Physics 63, 053504 (2022).
  * LeBlond  _et al._ (2019) T. LeBlond, K. Mallayya, L. Vidmar, and M. Rigol, Phys. Rev. E 100, 062134 (2019).
  * Łydżba  _et al._ (2021c) P. Łydżba, M. Rigol, and L. Vidmar, Phys. Rev. B 103, 104206 (2021c).
  * Oganesyan and Huse (2007) V. Oganesyan and D. A. Huse, Physical review b 75, 155111 (2007).
  * Atas  _et al._ (2013c) Y. Atas, E. Bogomolny, O. Giraud, and G. Roux, Physical review letters 110, 084101 (2013c).
  * Short (2011) A. J. Short, New. J. Phys. 13, 053009 (2011).
  * Bertini  _et al._ (2021b) B. Bertini, P. Kos, and T. Prosen, Communications in Mathematical Physics 387, 597 (2021b).
  * Chan  _et al._ (2018b) A. Chan, A. De Luca, and J. T. Chalker, Phys. Rev. X 8, 041019 (2018b).
  * Reimann and Kastner (2012) P. Reimann and M. Kastner, New Journal of Physics 14, 043020 (2012).
  * Short and Farrelly (2012) A. J. Short and T. C. Farrelly, New. J. Phys. 14, 013063 (2012).
  * Forrester (2021a) P. J. Forrester, Journal of Statistical Physics 183, 33 (2021a).
  * Forrester (2021b) P. J. Forrester, Communications in Mathematical Physics 387, 215 (2021b).
  * Cipolloni  _et al._ (2023) G. Cipolloni, L. Erdős, and D. Schröder, Communications in Mathematical Physics , 1 (2023).
  * Liu (2018) J. Liu, Physical Review D 98, 086026 (2018).

##  Appendix A RMT predictions for $q=2,3$

In this section we demonstrate that random matrix models have level-attraction
for $q=2,3$. We accomplish this by simply looking at the ratio test outlined
in the main text. First let us define a random matrix Hamiltonian,

$\hat{H}=A+A^{T},$ (46)

where $A$ is a matrix filled with random numbers generated from a normal
distribution with zero mean and unit variance. We label the side length of $A$
as $N$. Similar to the physical Hamiltonian, we can study the $q=2$ case by
first constructing the Hamiltonian,

$\hat{H}_{2}=\hat{H}\otimes\mathbb{I}+\mathbb{I}\otimes\hat{H},$ (47)

with the spectrum $\Lambda_{k,l}$. Again this spectrum is symmetric under
permutations of the indices, so we resolve this symmetry and only treat
eigenvalues with unique $(k,l)$ such that $l>k$. We investigate the spectral
properties of this Hamiltonian after resolving our symmetry in figure 4 in the
left panel. Here we clearly see agreement with a Poisson distribution. The
random matrix model experiences level-attraction.

Figure 4:  (a) Ratio test for the $q=2$ symmetry resolved random matrix
Hamiltonian. Side length is $N=1200$. (b) Ratio test for the $q=3$ symmetry
resolved random matrix Hamiltonian. Side length is $N=200$.

The construction of a new Hamiltonian for $q=3$ is similar. We have

$\hat{H}_{3}=\hat{H}\otimes\mathbb{I}\otimes\mathbb{I}+\mathbb{I}\otimes\hat{H}\otimes\mathbb{I}+\mathbb{I}\otimes\mathbb{I}\otimes\hat{H}.$
(48)

This gives us a new spectrum of $\Lambda_{k,l,q}=E_{k}+E_{l}+E_{q}$, where
this new spectrum is also invariant under permutations of its indices. We
resolve this symmetry by considering terms such that $q>l>k$. The result of
the ratio test on this new spectrum is given in the right panel of Fig. 4,
indicating again level-attraction and agreement with Poisson statistics. This
is similarly found for higher values of $q$, which leads us to conjecture that
this will be true for all $q\geq 2$.

##  Appendix B Physical Hamiltonian q = 3,4 spectral statistics

In this section we provide numerical evidence for level-attraction for $q=3,4$
in the physical Hamiltonian. We repeat the $q=3$ case as was covered in the
RMT appendix, and also investigate the $q=4$ statistics. Both cases will be
covered with the ratio test, and we will use the same physical model as the
main text, where we resolve all relevant symmetries. For the $q=4$ case, we
must work with the Hamiltonian,

$\hat{H}_{4}=\hat{H}\otimes\mathbb{I}\otimes\mathbb{I}\otimes\mathbb{I}+\mathbb{I}\otimes\hat{H}\otimes\mathbb{I}\otimes\mathbb{I}+\mathbb{I}\otimes\mathbb{I}\otimes\hat{H}\otimes\mathbb{I}+\mathbb{I}\otimes\mathbb{I}\otimes\mathbb{I}\otimes\hat{H}.$
(49)

This gives us a new spectrum, again which is invariant under index
permutations. We can resolve this symmetry with an identical strategy to the
$q=2,3$ cases, and study the corresponding symmetry-resolved spectrum. The
results of this for $q=3,4$ in the physical Hamiltonian are given in Fig. 5.
These results again indicate that the spectrum has level-attraction and obeys
Poisson statistics. The left panel in Fig. 5 also serves as evidence that the
statistics of the Hamiltonian agrees with RMT as seen in the right panel of
Fig. 4.

Figure 5:  (a) Ratio test for the $q=3$ symmetry resolved physical
Hamiltonian. The physical Hamiltonian $\hat{H}$ is generated with $L=16$. (b)
Ratio test for the $q=4$ symmetry resolved physical Hamiltonian. The
Hamiltonian $\hat{H}$ here has $L=14$.