# Performance Evaluation of Kubernetes Networking Approaches across Constraint
Edge Environments

Georgios Koukis32, Sotiris Skaperas12, Ioanna Angeliki Kapetanidou32, Lefteris
Mamatas12, Vassilis Tsaoussidis32 Emails: {gkoukis, ikapetan,
<EMAIL_ADDRESS>{sotskap<EMAIL_ADDRESS>2 Athena Research and
Innovation Center, Greece  3 Department of Electrical and Computer
Engineering, Democritus University of Thrace, Xanthi, Greece  1 Department of
Applied Informatics, University of Macedonia, Thessaloniki Greece

###### Abstract

Kubernetes (K8s) serves as a mature orchestration system for the seamless
deployment and management of containerized applications spanning across cloud
and edge environments. Since high-performance connectivity and minimal
resource utilization become critical factors as we approach the edge,
evaluating the performance of K8s networking in this context is essential.
This paper contributes to this effort, by conducting a qualitative and
quantitative performance evaluation of diverse Container Network Interface
(CNI) plugins within different K8s environments, incorporating lightweight
implementations designed for the Edge. Our experimental assessment was
conducted in two distinct (intra- and inter-host) scenarios, revealing
interesting insights for both researchers and practitioners. For example, the
deployment of plugins across lightweight distributions does not necessarily
lead to resource utilization improvements, e.g., in terms of CPU/memory or
throughput.

###### Index Terms:

Edge/Fog Computing, Kubernetes, CNI plugins

##  I Introduction

Edge and Fog computing architectures aim to bring computational resources
closer to the user, enabling real-time processing and reducing latency. In
this regard, Kubernetes (K8s) has emerged as a prominent solution for
automated and scalable orchestration/management of containerized workloads and
services. However, due to the resource constraints of Edge and Fog
environments, a number of lightweight K8s implementations have been proposed,
such as K0s, K3s, and MicroK8s, that focus on reduced disk space, CPU and
memory consumption [1]. In a similar vein, the ongoing project Cognitive
Decentralised Edge Cloud Orchestration (CODECO, https://he-codeco.eu)
investigates a K8s framework to enhance the flexibility of services across the
edge-cloud continuum. CODECO pays particular attention on the identification
of suitable toolkits for intelligent management of highly distributed
environments with resource constraints.

Furthermore, the Edge holds immense significance in enhancing the networking
communication of applications and devices that have stringent communication
requirements, e.g., delay-sensitive IoT devices. In this context, plugins are
responsible for the networking communication among the different K8s
components, enabling their connectivity with overlay/underlay networks, e.g.,
through the Border Gateway Protocol (BGP), Virtual Extensible LAN (VxLAN), IP
in IP, etc. A widely adopted plugin specification has been proposed by the
Container Network Interface (CNI) project, which also comprises libraries for
the configuration of networking interfaces.

Although several works study the effect of CNI plugins e.g., [2, 1] or
lightweight K8s implementations e.g., [3] on the resource utilization of
clusters, the impact of networking plugins on resource-restricted
environments, e.g., in the Edge, is still under-investigated. We address this
gap by conducting a qualitative and quantitative comparative analysis between
various plugins across different K8s distributions, aiming to answer the
following research questions:

  * •

RQ1: What is the impact of the interplay between the considered K8s
distributions and CNI plugins in terms of resource utilization and network
performance?

  * •

RQ2: Can we associate the performance and functional characteristics of CNI
plugins with particular edge application requirements?

Along these lines, we assess $5$ representative CNI plugins across $4$
different distributions ($3$ lightweight and vanilla K8s) over two distinct
testbeds. In particular, we measure the CPU and RAM utilization, and also the
throughput gains of the compared CNI plugins across different resource-
restricted environments and various connectivity conditions. These include
idle, pod-to-pod, and pod-to-service conditions for both TCP and UDP
communication protocols. Furthermore, we consider intra-host and inter-host
communication scenarios, where hosts are located either on the same or on
different machines. Our results reveal interesting insights regarding the two
identified research questions.

The remainder of this paper is organized as follows. In Section II, we discuss
recent related works that evaluate CNI plugins and lightweight K8s
distributions. In Section III we approach the considered CNI plugins and
distributions qualitatively. In Section IV we outline our experimentation
setup and methodology, while in Section V we focus on the quantitative results
of the experiments. Finally, Sections VI and VII detail the insights gained by
our analysis along with our concluding remarks and future plans, respectively.

##  II Related work

Recent literature mainly considers the networking impact of CNI plugins on the
conventional vanilla K8s distribution. Authors in [2] conduct a comprehensive
comparison of various CNI plugins, focusing on functionality (e.g., packet
forwarding and routing), performance and scalability aspects. They also
investigate the impact of encapsulation and tunneling offload on the
performance of selected plugins.

Other relevant works, such as [4], discuss the trade-offs between overlay and
pure IP plugins, considering various Maximum Transmission Unit (MTU), Maximum
Segment Sizes (MSS) and NIC offloading conditions. Paper [5] evaluates the
throughput performance of the Flannel plugin, OVS-based and native-VLAN
networks with different packet sizes, demonstrating Flannel’s substantial
computational overhead, especially as the MTU size decreases. A high-level
resource-consumption-based assessment of various CNI plugins is carried out in
the blog paper [6]. In addition, authors in [7, 8] assess the performance
variations across virtual and bare metal environments, demonstrating the
superior performance of the latter in terms of throughput and computational
overhead.

Subsequently, several existing studies compare the performance of lightweight
distributions, focusing on intra-distribution characteristics (e.g., execution
time and resource consumption), which are crucial for their applicability in
edge environments. In this context, [1], [9] and [10] investigate CPU, memory,
and disk usage for MicroK8s, K3s and vanilla K8s distributions over various
cluster life-cycle stages (e.g., startup, deployment and shutdown).
Surprisingly, the authors reveal that lightweight distributions do not always
outperform the traditional K8s distribution. Paper [3] extends the previous
analysis with the performance comparison of K0s and MicroShift, while focusing
on stress-inducing environments under generated workloads. Finally, the
authors of [11] quantify the performance of K3s and KubeEdge, in terms of CPU
and RAM utilization, on a Raspberry Pi device.

Despite the broad interest, a research gap exists concerning the interplay
between lightweight K8s distributions and different networking plugins. To the
best of our knowledge, this is the first work to evaluate the performance of
various CNI plugins across different K8s distributions. Moreover, we consider
two distinct scenarios corresponding to nodes located on the same or on
separate machines.

##  III Network plugins & Kubernetes distributions

In this section, we discuss the qualitative characteristics of the considered
K8s distributions and CNI plugins. Note that the selection of the specific
plugins is determined by their compatibility with the discussed K8s
distributions.

###  III-A Kubernetes distributions

In particular, four well-known K8s implementations are considered, i.e.,
vanilla K8s, K3s, K0s and MicroK8s.

Vanilla K8s (https://kubernetes.io) is the standard feature-rich and robust
container orchestration platform. It is resource-intensive and suitable for
large-scale production deployments (requiring a minimum of 2 CPUs and 2 GB
RAM).

K3s (https://k3s.io) is a lightweight K8s distribution that focuses on
resource-constrained environments, e.g., IoT and Edge devices. It requires
minimal resources with half the memory of a vanilla K8s (even supports
clusters with Raspberry Pi), combining components as a single binary
($<$100MB) with minimal to no OS dependencies. K3s replaces the default K8s
storage with an SQLite-backed option (also supports Etcd3, MySQL, and
Postgres), includes an ingress controller (Traefik), and operates with
recommended system requirements of 2-core CPU and 1 GB RAM.

K0s (https://k0sproject.io) is a lightweight alternative of K8s packaged as a
single binary ($<$160MB), allowing for easy installation across diverse
environments. It prioritizes simplicity over features, creating separated
control plane and deploying Konnectivity agents to proxy traffic from the
control plane (API server) to the worker nodes. K0s has minimum host OS
dependencies, low system requirements (1 vCPU, 1 GB RAM) and minimal features
compared to vanilla K8s and K3s, while it supports the same storing options as
K3s.

MicroK8s (https://microk8s.io) is designed to run K8s on various Linux-based
OSs, providing seamless installation through its snap package. It covers the
needs for High Availability and CI/CD applications, including automatic
updates and self-healing mechanisms due to snap refreshes. MicroK8s is
suitable for both constrained environments, such as IoT and Edge scenarios,
and high-demand production environments (with recommended requirements of
2-core CPU and 4 GB RAM). It uses dqlite storage by default and provides
additional features, such as load balancing, ingress controller (Traefik),
dual-stack, GPU acceleration, observability and metric collection tools.

###  III-B CNI plugins

Additionally, we discuss the key features of the five selected CNI plugins,
i.e., Flannel, Calico, Cilium, Kube-router and Kube-ovn.

Flannel (https://github.com/flannel-io/flannel) is a lightweight, Layer 2/3
overlay network fabric, which deploys a single binary agent (flanneld) to
assign an IP range of subnet addresses for each node in the cluster. It uses
K8s API or etcd to maintain a mapping between allocated subnets and actual
node IP addresses, while also supporting a variety of backend mechanisms, such
as VxLAN (default L2), host-GW (L3), WireGuard, UDP, and other experimental
options. However, it provides limited advanced features, e.g.,
security/encryption features and network policies. Flannel is the default CNI
plugin for the K3s distribution.

Calico (https://www.tigera.io/project-calico/) is a feature-rich Layer-3
network solution. By default, it utilizes BGP mode, establishing a full BGP
mesh among cluster nodes, while also supports alternative modes including IP
overlay, VxLAN and Wireguard. Calico offers various kube-proxy modes such as
iptables, ipvs, while it accesses etcd cluster directly or uses the K8s API
datastore. Calico provides advanced network policy, administration, and
security features, including Wireguard, eBPF (extended Berkeley Packet
Filter), and XDP (eXpress Data Path). These features enable network policy
enforcement (e.g. integrating with service meshes like Istio
(https://istio.io), service discovery, load balancing and dual-stack
networking. It functions as the default plugin in the MicroK8s distribution.

Cilium (https://cilium.io/) is a Layer-3 networking, security and
observability solution based on eBPF. It provides advanced programmable
features, including network policy enforcement, transparent network security
(with IPSec tunnels, TLS, GCM encryption), traffic monitoring and visibility
(e.g., through the Hubble platform). Cilium uses bpfilter routing instead of
iptables, shifting filtering tasks to the kernel to improve performance, load
balancing and processing of traffic speed.

Kube-router (https://www.kube-router.io/) is implemented at Layer 3,
leveraging the standard Linux networking stack. It offers both overlay (IP-in-
IP) and underlay (BGP) options, with the BGP protocol employed as the default
in-cluster networking mechanism, utilizing the “simple” bridge plugin and
storing networking states in the K8s API. Deployed as a single binary
solution, Kube-router supports network policies and utilizes LVS/IPVS for
service proxy. It offers advanced features, such as Direct Server Return (DSR)
and Layer-3 load balancing. It serves as the default K0s distribution plugin.

Kube-ovn (https://www.kube-ovn.io/) is an SDN-based solution integrating the
OVN-based network virtualization with K8s, providing network policies, subnet
isolation, dual-stack support, and features like dynamic QoS, embedded Layer-2
load balancing and hardware offloading. It employs overlay networking by
default, utilizing the Geneve kernel module to encapsulate cross-host traffic.
However, Kube-ovn also supports underlay and Vlan networking modes for direct
connectivity with the physical network and the integration with other plugins,
e.g., Cilium.

To summarize the qualitative characteristics of the above plugins, Flannel
provides a lightweight overlay solution, focusing on easy deployment and
compatibility. However, it is not appropriate in deployments with scalability
and security requirements, as well as with a demand for advanced network
policies. Calico and Cilium are feature-rich and production-oriented
solutions, with the former providing high performance and low latency, and the
latter leveraging eBPF for programmable security and visibility features.
Kube-router operates similarly to Calico, without the advanced features of the
latter, while Kube-ovn provides “rich” SDN capabilities.

Among the lightweight distributions, K3s and K0s reduce the OS footprint by
minimizing or replacing certain resource-intensive K8s features. K0s focuses
on an isolated control plane with limited features, while MicroK8s can be
employed in both resource-constrained and production environments.

##  IV Experimentation Setup and Methodology

This section provides an overview of our experimentation approach, including
the description of our setup and specifications of the considered
experimentation environments.

The installation of the K8s distributions and the respective CNI plugins,
i.e., based on their most recent stable versions, is carried out in a cluster
comprising one master and two worker nodes (VMs). Each node consists of 40GB
storage, 4GB memory and 4 core CPU, running Ubuntu 22.04.2 LTS with kernel
version 5.15.0-71-generic. The K8s cluster is being built based on kubeadm
(v1.28.2) and Ansible, utilizing the ClusterSlice platform [12]. After each
scenario, the cluster is deleted and then re-deployed to remove any remnants
of previous experiments.

For the quantitative performance evaluation of CNI plugins, we employ the
Kubernetes Network Benchmark (knb) tool
(https://github.com/InfraBuilder/k8s-bench-suite), where the two worker nodes
represent the server and client within the cluster. Throughout our
experiments, we assess each plugin in terms of CPU, RAM and throughput across
various communication conditions, i.e., i) “Idle”, ii) “Pod-to-Pod”, and iii)
“Pod-to-Service”, evaluating both TCP and UDP transport layer protocols. In
states (ii) and (iii), the client Pod establishes a direct connection with the
server Pod using its IP address and ClusterIP service, respectively.

Our experimental runs are categorized into two distinct scenarios, i.e., for
intra-host and inter-host communications. In particular, we explore these
scenarios in $2$ testbeds, referred to as ATH and UOM, located at the ATHENA
Research Center and the University of Macedonia, respectively.

The ATH testbed includes a single Dell PowerEdge T640 physical machine with an
Intel(R) Xeon(R) Silver 4210R at 2.40GHz, $16$ cores CPU, $64$GB RAM and $1$TB
SSD, hosting all cluster VMs and demonstrating the intra-host communication
scenario. On the contrary, the UOM testbed consists of two Dell PowerEdge R630
physical servers each one comprising of $2$ Intel(R) Xeon(R) CPU E5-2620 v4 at
2.10GHz $16$ cores CPU, $68$ & $34$GB RAM and $1.5$TB HDD. The first machine
of the UOM testbed hosts the K8s master and one worker node and the second the
other worker, i.e., realizing the inter-host communication scenario. Both
testbeds use the XCP-ng virtualization platform (https://xcp-ng.org). Finally,
for each experiment we conduct $10$ iterations of the knb tool.

##  V Results

In this section, we evaluate the quantitative performance of all considered
CNI plugins and K8s distributions based on the methodology discussed in the
previous section. The results of the two scenarios follow.

###  V-A Scenario 1 - Intra-host communication

Fig. 1 depicts the resource consumption (CPU, RAM) and the throughput
performance of the CNI plugins across the considered K8s distributions, with
regards to the intra-host communication scenario (ATH testbed). Focusing on
the CPU usage (leftmost column of Fig. 1), the selection of K8s distribution
does not significantly impact on the plugins’ CPU utilization. An exception is
observed for Calico, especially in K0s and Mk8s (in the latter, Calico is the
default option), for the TCP communications, where the CPU utilization is
reduced by $20\%$ due to a shift from underlay to overlay mechanisms.
Furthermore, we notice that UDP is characterized by an increased CPU
consumption for all the considered plugins, compared to TCP communication.
Focusing on TCP communication, Flannel and Cilium are less CPU intensive, with
an average utilization of $10\%$, unlike Calico and Kube-router which show an
average CPU consumption of $25\%$. In particular, Flannel offers limited
features, while Cilium leverages eBPF hooks to shift processing tasks and
provide programmability at the kernel level. Concerning UDP communications,
besides Kube-ovn that results in $37\%$ CPU utilization, the rest of the
deployed plugins provide a CPU consumption of $30\%$. These outcomes are in
line with the recent literature, e.g., [2] and [6].

On the other hand, the RAM consumption of the employed plugins heavily depends
on the K8s distribution in which they are deployed, as shown in Fig. 1 (middle
column). In particular, Flannel efficiently reduces its RAM utilization for
both K3s and Mk8s, resulting in the lowest RAM consumption (for the K3s) among
the employed plugins. Similarly, Kube-router, the default plugin in the K0s
distribution, exhibits the minimum RAM utilization (close to Flannel).
However, Kube-ovn and Cilium provide increased RAM consumption when deployed
over lightweight K8s distributions, i.e., Mk8s and K3s, respectively. The
illustrated results indicate that the selection of a lightweight K8s
distribution does not always ensure a lower resource consumption for the
plugins. For example, in case of Kube-ovn one has to consider the impact of an
SDN approach over resource constrained environments or the increased memory
utilization of Cilium due to the footprint of both daemonset and binary files
[2]. Moreover, the reduced RAM usage of Calico for the K0s and Mk8s
distributions aligns with the results of CPU utilization. We note that the
selection of transport layer protocols, i.e., UDP or TCP, does not affect the
RAM consumption of the considered plugins.

Finally, the rightmost column of Fig. 1 illustrates the throughput
performance. In the TCP-related conditions, the pure IP-based leading plugins,
i.e., Calico and Kube-router, achieve throughput performances that exceed
$9000$ Mbps, whereas the remaining plugins consistently fall at $1000$ Mbps.
An exception is identified for Calico (in K0s and MicroK8s), where the
reduction in resource consumption comes at the cost of decreased throughput.
On the other hand, UDP-based communications exhibit lower throughput
performance near $1000$ Mbps for all plugins across the different
distributions.

(a) Calico plugin.

(b) Flannel plugin.

(c) Kube-router plugin.

(d) Cilium plugin.

(e) Kube-ovn plugin.

Figure 1: CPU, RAM usage and throughput per CNI plugin, for different K8s
distributions (ATH testbed).

(a) Calico plugin.

(b) Flannel plugin.

(c) Kube-router plugin.

(d) Cilium plugin.

(e) Kube-ovn plugin.

Figure 2: CPU, RAM usage and throughput per CNI plugin, for different K8s
distributions (UOM testbed).

###  V-B Scenario 2 - Inter-host communication

In Fig. 2, we investigate the performance of the deployed plugins in the
inter-host communication scenario (UOM testbed). Regarding the CPU usage
across K8s distributions (Fig. 2, leftmost column), the results are aligned
with those observed in the intra-host scenario, since the K8s distribution
does not seem to significantly affect the CPU consumption of the plugins
(again with the exception of Calico, for K0s and Mk8s). On the other hand, the
inter-host communication results in lower performance variations between UDP
and TCP communication. In general, the CPU is increased for the TCP and
decreased for the UDP communication, Kube-router and Calico being excepted.
Flannel outperforms the rest of the plugins in terms of CPU consumption, while
Kube-ovn is the most CPU-intensive, due to its inherent “rich” and
programmable features.

In terms of RAM usage (middle column of Fig. 2), the results reveal similar
patterns to the intra-host scenario, for all plugins. For instance, compared
to vanilla K8s distribution, Flannel reduces its RAM footprint by $28\%$ in
K3s and $15\%$ in Mk8s. Kube-router by 21% over K0s (in which it is the
default plugin), while Calico by $6\%$, $16\%$ and $37\%$, regarding K3s, K0s
and Mk8s. On the contrary, Kube-ovn and Cilium result in $16\%$ and $14\%$
increased RAM consumption, for Mk8s and K3s distribution, respectively.

Finally, the throughput performance of the plugins is shown in Fig. 2
(rightmost column). As expected, the inter-host scenario presents decreased
throughput performance, e.g., no plugin exceeds the $1000$ Mbps threshold.
Moreover, plugins based on underlay IP routing, i.e., Calico and Kube-router,
surpass the performance of VxLAN overlay-based ones, i.e., Flannel and Cilium.
In this context, Kube-router exhibits higher and more consistent throughput
performance, for UDP/TCP protocols. On the other hand, the lightweight nature
of Flannel impacts its throughput performance, especially regarding the TCP
protocol. Also, we notice a decrease in the performance of Calico for K0s and
MicorK8s, as in the intra-host scenario. Finally, the increased resource
consumption of Kube-ovn and Cilium, when deployed over lightweight K8s
distributions, seems to offer no performance gains in terms of throughput.

##  VI Discussion

Based on the provided qualitative and quantitative analysis, we summarize our
key insights as follows:

(i) The CPU consumption remains relatively stable across the plugins,
regardless of their deployment over vanilla K8s or a lightweight distribution.
An exception is identified for Calico in K0s and Mk8s due to its shift from
underlay to overlay mechanisms. Overall, the results underscore the necessity
for further fine-tuning and optimization of plugins in Edge environments,
wherein reducing the CPU consumption is normally desired due to the associated
resource constraints.

(ii) The RAM usage is generally reduced on lightweight K8s distributions,
except for Cilium and Kube-ovn plugins. In this regard, Flannel provides the
lowest resource footprint, particularly excelling across lightweight K8s.
However, it lacks in terms of throughput and advanced features (e.g.,
security, network policies) which impacts its applicability in actual
production environments. Conversely, Kube-ovn and Cilium, which support
advanced programmable features, exhibit a performance decline in terms of RAM
consumption and throughput in lightweight distributions. Such a trade-off
should be considered upon plugin selection to align with the specific
requirements of the implementation scenario.

(iii) The reduction of resource footprint over Edge distributions does not
necessarily impact the throughput of the considered plugins, with the
exception of Calico.

(iv) According to our evaluation results, Calico, Flannel and Kube-router
adjust their resources across lightweight distributions, with the former being
used as the default plugin in K0s. Calico, specifically, offers advanced
functionalities and manages to significantly reduce its resource utilization,
however at the cost of decreased throughput performance.

##  VII Conclusion and Future Work

We presented a qualitative and quantitative performance evaluation of commonly
used K8s networking solutions across various K8s distributions, emphasizing on
their performance across resource-restricted Edge environments. Our
experiments over intra/inter-host communication scenarios revealed that
lightweight K8s implementations mainly reduce the RAM footprint of certain
plugins (i.e., Calico, Kube-router and Flannel), usually at the cost of
throughput, especially for Calico and Flannel. However, the employment of
lightweight K8s distributions does not ensure a reduction in the resource
consumption of plugins, e.g., Cilium and Kube-ovn, highlighting the need for
further optimizations across Edge-related distributions. Our future plans
include the investigation of: (i) out-of-the-box plugins, e.g., L2S-M
(https://github.com/Networks-it-uc3m/L2S-M), (ii) additional performance
metrics, e.g., distributions’ life-cycle and scalability aspects, and (iii)
the performance of typical applications deployed at the Edge.

## Acknowledgment

This work is supported by the Horizon Europe CODECO Project under Grant number
101092696.

## References

  * [1] S. Böhm and G. Wirtz, “Profiling lightweight container platforms: Microk8s and K3s in Comparison to Kubernetes,” in _Proc. ZEUS_ , Bamberg, Germany, 2021, vol. 2389, pp. 65–73.
  * [2] S. Qi, S. G. Kulkarni, and K. K. Ramakrishnan, “Assessing container network interface plugins: Functionality, performance, and scalability,” _IEEE Trans. Netw. Service Manag_ , vol. 18, no. 1, pp. 656–671, 2021.
  * [3] H. Koziolek and N. Eskandani, “Lightweight Kubernetes Distributions: A Performance Comparison of MicroK8s, K3s, K0s, and Microshift,” in _Proc. ACM/SPEC ICPE_ , Coimbra, Portugal, 2023, p. 17–29.
  * [4] N. Kapočius, “Performance studies of kubernetes network solutions,” in _Proc. IEEE eStream_ , Vilnius, Lithuania, 2020, pp. 1–6.
  * [5] Y. Park _et al._ , “Performance analysis of cni (container networking interface) based container network,” in _Proc. ICTC_ , Jeju Island, Korea, 2018, pp. 248–250.
  * [6] A. Ducastel, “Benchmark results of kubernetes network plugins (CNI) over 10gbit/s network (updated: August 2020),” Online: https://itnext.io/benchmark-results-of-kubernetes-network-plugins-cni-over-10gbit-s-network-updated-august-2020-6e1b757b9e49.
  * [7] N. Kapočius, “Overview of Kubernetes CNI plugins performance,” _Science: Future of Lithuania_ , vol. 12, pp. 1–5, 2020.
  * [8] H. Zeng _et al._ , “Measurement and Evaluation for Docker Container Networking,” in _Proc. CyberC_ , Nanjing, China, 2017, pp. 105–108.
  * [9] V. Kjorveziroski and S. Filiposka, “Kubernetes distributions for the edge: serverless performance evaluation,” _J. Supercomputing_ , vol. 78 13728–13755, pp. 1–28, 2022.
  * [10] S. Telenyk _et al._ , “A Comparison of Kubernetes and Kubernetes-Compatible Platforms,” in _Proc. IEEE IDAACS_ , Cracow, Poland, 2021, pp. 313–317.
  * [11] H. Fathoni _et al._ , “Performance Comparison of Lightweight Kubernetes in Edge Devices,” in _Proc. I-SPAN_ , Naples, Italy, 2019, pp. 304–309.
  * [12] L. Mamatas, S. Skaperas, and I. Sakelariou, “Clusterslice: Slicing resources for zero-touch kubernetes-based experimentation,” _Tech. Rep., UoM_ , 2023. https://ruomo.lib.uom.gr/handle/7000/1757.


* Wang et al. (2019) Yilin Wang, Sasi Inguva, and Balu Adsumilli. 2019.  YouTube UGC Dataset for Video Compression Research. In _2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP)_. 1–5.  https://doi.org/10.1109/MMSP.2019.8901772
  * Weng et al. (2023) Zhenzhen Weng, Laura Bravo-Sánchez, and Serena Yeung. 2023\.  Diffusion-HPC: Generating Synthetic Images with Realistic Humans.  _arXiv preprint arXiv:2303.09541_ (2023).
  * Whittaker et al. (2019) Meredith Whittaker, Meryl Alper, Cynthia L Bennett, Sara Hendren, Liz Kaziunas, Mara Mills, Meredith Ringel Morris, Joy Rankin, Emily Rogers, Marcel Salas, et al. 2019\.  Disability, bias, and AI.  _AI Now Institute_ 8 (2019).
  * Wrenninge and Unger (2018) Magnus Wrenninge and Jonas Unger. 2018.  Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing.  arXiv:1810.08705 [cs.CV]
  * Wu et al. (2019) Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. 2019.  Detectron2.  https://github.com/facebookresearch/detectron2.
  * Xie et al. (2022) Kevin Xie, Tingwu Wang, Umar Iqbal, Yunrong Guo, Sanja Fidler, and Florian Shkurti. 2022\.  Physics-based Human Motion Estimation and Synthesis from Videos.  arXiv:2109.09913 [cs.CV]
  * Xu et al. (2023) Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. 2023\.  ViTPose++: Vision Transformer Foundation Model for Generic Body Pose Estimation.  arXiv:2212.04246 [cs.CV]
  * Yang et al. (2023) Zhitao Yang, Zhongang Cai, Haiyi Mei, Shuai Liu, Zhaoxi Chen, Weiye Xiao, Yukun Wei, Zhongfei Qing, Chen Wei, Bo Dai, Wayne Wu, Chen Qian, Dahua Lin, Ziwei Liu, and Lei Yang. 2023\.  SynBody: Synthetic Dataset with Layered Human Models for 3D Human Perception and Modeling.  arXiv:2303.17368 [cs.CV]
  * Zhang et al. (2023) Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. 2023\.  T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations.  arXiv:2301.06052 [cs.CV]
  * Zhang et al. (2022) Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. 2022.  MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model.  arXiv:2208.15001 [cs.CV]
  * Zheng et al. (2023) Ce Zheng, Wenhan Wu, Chen Chen, Taojiannan Yang, Sijie Zhu, Ju Shen, Nasser Kehtarnavaz, and Mubarak Shah. 2023\.  Deep Learning-Based Human Pose Estimation: A Survey.  arXiv:2012.13392 [cs.CV]
  * Zwölfer et al. (2023) Michael Zwölfer, Dieter Heinrich, Kurt Schindelwig, Bastian Wandt, Helge Rhodin, Jörg Spörri, and Werner Nachbauer. 2023.  Deep Learning-Based 2D Keypoint Detection in Alpine Ski Racing – A Performance Analysis of State-of-the-Art Algorithms Applied to Regular Skiing and Injury Situations.  _JSAMS Plus_ 2 (2023), 100034.  https://doi.org/10.1016/j.jsampl.2023.100034

##  Appendix A Appendix

###  A.1. Animation Clip Fix

To ensure the human model’s do not overlap with the wheelchair footplates, a
$35$ degree rotation of the hips up toward the sky is applied. This hip
rotation decreases the amount of clipping between the legs and the wheelchair.
Clipping is a common occurrence in 3D modeling where when objects are within
each other, only the object closer to the camera will be rendered and obscure
the overlapped object. Examples of this extra clipping are shown in Figure 1.

Appendix figure 1. Examples of generated data with and without increased lower
body clipping. Notice the overlap between the ankles and the footplate in the
motion sequences with clipping creates models where it looks like the human
model has fused into the wheelchair.

Four examples of animations with clipping and four examples of animations
without clipping. Examples of clipping feature the human model with feet that
extend into the footplate of the wheelchair and below. Examples without
clipping place the feet on top of the footplate.

###  A.2. Posture to AnimationClip Conversion

We convert the postures resulting from previous sections into human images
using human models in Unity which take AnimationClips as input for pose
configurations. To convert pose frames into AnimationClips, all motion
sequences from each set, represented by a series of joint rotations, are
individually imported into Blender. Each frame’s joint rotations are applied
to the corresponding joint in a Unity Perception human model Blender template
and exported as an FBX file. Upon importing an FBX file into Unity, Unity will
automatically convert all baked animations into Unity-readable AnimationClip
files which can be used in Unity Perception. All AnimationClips are then set
to read as Unity Humanoid animations for use in data synthesis.

###  A.3. WheelPose Randomizers

Unity Perception enables the use of the ”randomizer” paradigm to enable users
to configure the domain randomization of individual parameters (Borkman et
al., 2021). WheelPose uses multiple Unity Perception default randomizers, PSP
custom randomizers (Ebadi et al., 2022b), and a collection of custom WheelPose
randomizers. It is important to note that many of PSP’s custom randomizers
have made it into the release version of Unity Perception (1.0.0). We have
chosen to maintain the original randomizers used for a direct comparison
between data synthesized between PSP and WheelPose. Like in PSP, our
randomizers are regarded as further data augmentation techniques which limits
the need for data augmentations during training itself. All randomizers
sampled values from a uniform distribution. Table 1 outlines the statistical
distributions for our randomizer parameters. A brief description of each
randomizer used in WheelPose is described below.

BackgroundObjectPlacementRandomizer. Randomly spawns background and occluder
objects within a user-defined 3D volume. Separation distance can be set to
dictate the proximity of objects from each other. Poisson-Disk sampling
(Bridson, 2007) is used to randomly place objects sourced from a set of
primitive 3D game objects (cubes, cylinders, spheres, etc.) from Unity
Perception in a given area.

BackgroundOccluderScaleRandomizer. Randomizes the scale of the background and
occluder objects.

RotationRandomizer. Randomizes the 3D rotation of background and occluder
objects.

ForegroundObjectPlacementRandomizer. Similar to
BackgroundObjectPlacementRandomizer. Randomly spawns foreground objects
selected from the default set of PSP models affixed in wheelchair models.

ForegroundScaleRandomizer. Similar to BackgroundOccluderScaleRandomizer.
Randomizes the scale of foreground objects.

TextureRandomizer. Randomizes the texture of predefined objects provided as a
JPEG or PNG. We used the set of example textures from Unity Perception which
are applied to the background and occluder objects as well as to the
background wall when no specific background is set.

HueOffsetRandomizer. Randomizes the hue offset applied to textures on the
object. Applied to background and occluder objects as well as to the
background wall when no specific background is set.

SpriteRandomizer. Randomizes the background wall. Used as an alternative to
the TextureRandomizer when images should not be stretched to fill a canvas.

HumanGenerationRandomizer. Randomizes the age, sex, ethnicity, height, weight,
and clothing of spawned human assets. Humans are spawned in batches called
pools which are periodically regenerated through the simulation process. All
humans are spawned within a predefined base which contains the wheelchair
model used. All textures and models used are sourced directly from
SyntheticHumans.

NonTagAnimationRandomizer. Randomizes the pose applied to a character. The
pose is a randomly selected frame from a randomly selected AnimationClip taken
from a universal pool of AnimationClips. Provides a custom alternative to the
Unity Perception AnimationRandomizer for randomizing animations taken from a
single pool.

TransformPlacementRandomizer. Randomizes the position, rotation, and size of
generated SyntheticHumans. Rotations around the X,Z-axis are limited to better
represent real world data where users are rarely seen in such orientations.

SunAngleRandomizer. Randomizes a directional light’s intensity, elevation, and
orientation to mimic the lighting effects of the Sun.

LightRandomizer. Randomizes a light’s intensity and color (RGBA). Also enables
the randomization of a light’s on/off state.

LightPositionRotationRandomizer. Randomizes a light’s global position and
rotation in the scene.

CameraRandomizer. Randomizes the extrinsic parameters of a camera including
its global position and rotation. Enables the randomization of intrinsic
camera parameters including field of view and focal length to better mimic a
physical camera. Adds camera bloom and lens blur around objects that are out
of focus to capture more diverse perspectives of the scene.

PostProcessVolumeRandomizer. Randomizes select post processing effects
including vignette, exposure, white balance, depth of field, and color
adjustments.

Appendix table 1. Domain randomization parameters of WheelPose

Category | Randomizer | Parameter | Distribution
---|---|---|---
Background/Occluder Objects | BackgroundObjectPlacementRandomizer | object placement | Cartesian[Uniform(-7.5, 7.5), Uniform(-7.5, 7.5), Uniform(-7.5, 7.5)]
separation distance | Cartesian[Constant(2.5), Constant(2.5), Constant(2.5)]
BackgroundOccluderScaleRandomizer | object scale range | Cartesian[Uniform(1, 12), Uniform(1, 12), Uniform(1, 12)]
RotationRandomizer | object rotation | Euler[Uniform(0, 360), Uniform(0, 360), Uniform(0, 360)]
TextureRandomizer | textures | A set of of texture assets
SpriteRandomizer | sprites | A set of sprite assets
HueOffsetRandomizer | hue offset | Uniform(-180, 180)
Human Model | HumanGenerationRandomizer | humans per iteration | Uniform(5, 12)
human pool size | Constant(50)
pool refresh interval | Constant(400)
age | Uniform(10, 100)
height | Uniform(0.1, 1)
weight | Uniform(0, 1)
sex | male, female
ethnicity | Caucasian, Asian, Latin American, African, Middle Eastern
TransformPlacementRandomizer | synthetic human placement | Cartesian[Uniform(-7.5, 7.5), Uniform(-7.5, 7.5), Uniform(-4, 1)]
synthetic human rotation | Euler[Uniform(0, 20), Uniform(0, 360), Uniform(0, 20)]
synthetic human size range | Cartesian[Uniform(0.5, 3), Uniform(0.5, 3), Uniform(0.5, 3)]
ForegroundObjectPlacementRandomizer | predefined model placement | Cartesian[Uniform(-7.5, 7.5), Uniform(-7.5, 7.5), Uniform(-9, 6)]
predefined model separation distance | Cartesian[Constant(3), Constant(3), Constant(3)]
ForegroundScaleRandomizer | predefined model scale range | Cartesian[Uniform(0.5, 3), Uniform(0.5, 3), Uniform(0.5, 3)]
ForegroundRotationRandomizer | predefined model rotation | Euler[Uniform(0, 20), Uniform(0, 360), Uniform(0, 20)]
NonTagAnimationRandomizer | animations | A set of AnimationClips of arbitrary length
Lights | SunAngleRandomizer | hour | Uniform(0, 24)
day of the year | Uniform(0, 365)
latitude | Uniform(-90, 90)
LightRandomizer | intensity | Uniform(5000, 50000)
color | RGBA[Uniform(0, 1), Uniform(0, 1), Uniform(0, 1), Constant( 1)]
enabled | $P(enabled)=0.8,P(disabled)=0.2$
LightPositionRotationRandomizer | position offset from initial position | Cartesian[Uniform(-3.65, 3.65), Uniform(-3. 65, 3.65), Uniform(-3.65, 3.65)]
rotation offset from initial rotation | Euler[Uniform(-50, 50), Uniform(-50, 50), Uniform(-50, 50)]
Camera | CameraRandomizer | field of view | Uniform(5, 50)
focal length | Uniform(1, 23)
position offset from initial position | Cartesian[Uniform(-5, 5), Uniform(-5, 5), Uniform(-5, 5)]
rotation offset from initial rotation | Euler(Uniform(-5, 5), Uniform(-5, 5), Uniform(-5, 5)]
Post Processing | PostProcessVolumeRandomizer | vignette intensity | Uniform(5, 50)
fixed exposure | Uniform(5, 10)
white balance temperature | Uniform(-20, 20)
depth of field focus distance | Uniform(.1, 4)
color adjustments: contrast | Uniform(-30, 30)
color adjustments: saturation | Uniform(-30, 30)

###  A.4. Testing Dataset Action Classes

A set of real-world wheelchair data was collected from YouTube. A predefined
set of $16$ action classes was defined before being used as keyword searches
to identify relevant videos. Action classes were selected based on a mix of
common actions and unique wheelchair movements. A total of $2,464$ images were
collected. More information on action classes is found in Table 2.

Appendix table 2. Distribution of activity classes in the testing dataset. Activity Class | Percentage of Dataset
---|---
talking | 21.659%
wheelchair skills | 14.692%
daily routine | 13.460%
dance | 10.664%
basketball | 8.863%
tennis | 5.829%
extreme sports | 5.640%
general sports | 4.645%
household chores | 3.318%
shopping | 2.180%
cooking | 2.085%
travel | 1.801%
photoshoot | 1.611%
rugby | 1.422%
pickleball | 1.185%
stretches | 0.948%

###  A.5. Impacts of Keypoint Location Definitions

Unity Perception enables users to redefine different keypoint definitions for
image annotations. Unity Perception provides a default COCO $17$-keypoint
annotation schema which places each keypoint directly on the joint between two
bones. However, in human-annotated datasets, many evaluators place the hip
much higher than the actual joint between the hips and the femur. We test
Unity’s default keypoint schema with lower hips against our own custom COCO
$17$-keypoint schema which raises the hip keypoint up the torso by $\approx
9$cm. All data is generated in the exact same way with the only difference of
how the hip keypoints are defined. Examples of how the annotation schema
affects predictions are displayed in Figure 2. We see that changes in the
definition of keypoints in synthetic data can drastically change the position
of the changed keypoint in predictions. We believe this concept can be used to
adapt and tune existing pose estimation models with different keypoint
definitions through the use of only synthetic data.

A person in a wheelchair doing a wheelie on grass facing the camera at a 45
degree angle. Both hands are on the wheels and both feet are in the footplate.

A person in a wheelchair holding a basketball facing the camera. the right
hand is holding the basketball near the chest while the left hand is on the
wheel.

A person in a motorized wheelchair facing the camera in a bathroom. Both
elbows rest on the armrest while the arms rest near their lap.

A side view of a person in a wheelchair rolling forward. Both arms are near
the person’s lap.

Appendix figure 2. Examples of the different prediction outputs between a
lower hip definition and a higher hip definition in synthetic data. Green
represents the lower hip definition and red represents the higher hip
definition. Figures 2, 2, 2 and 2 all show examples of where other keypoint
predictions are relatively similar with the exception of the hips where the
lower hip annotations are placed lower on the body compared to the higher hip
annotation. Each image depicts the wheelchair user in a different angle,
setting, and action.


# Modeling Non-additive Effects in Neighboring Chemically Identical
Fluorophores

Ayush Saurabh<EMAIL_ADDRESS>[  Stefan Niekamp
<EMAIL_ADDRESS>[  Ioannis Sgouralis<EMAIL_ADDRESS>[  Steve
Pressé<EMAIL_ADDRESS>[

###### Abstract

Quantitative fluorescence analysis is often used to derive chemical
properties, including stoichiometries, of biomolecular complexes. One
fundamental underlying assumption in the analysis of fluorescence data—whether
it be the determination of protein complex stoichiometry by super-resolution,
or step-counting by photobleaching, or the determination of RNA counts in
diffraction-limited spots in RNA fluorescence in situ hybridization (RNA-FISH)
experiments —is that fluorophores behave identically and do not interact.
However, recent experiments on fluorophore-labeled DNA-origami structures such
as fluorocubes have shed light on the nature of the interactions between
identical fluorophores as these are brought closer together, thereby raising
questions on the validity of the modeling assumption that fluorophores do not
interact. Here, we analyze photon arrival data under pulsed illumination from
fluorocubes where distances between dyes range from 2-10 nm. We discuss the
implications of non-additivity of brightness on quantitative fluorescence
analysis.

###### keywords:

fluorocube, photobleaching, counting, interactions

CBP] Center for Biological Physics, Department of Physics, Arizona State
University, Tempe, AZ 85287, USA MGH] Massachusetts General Hospital, Boston,
MA 02114 (Current Address) [UCSF] Department of Cellular and Molecular
Pharmacology, University of California, San Francisco, CA 94158 UTK]
Department of Mathematics, University of Tennessee, Knoxville, TN 37996, USA
CBP] Center for Biological Physics, Department of Physics, Arizona State
University, Tempe, AZ 85287, USA [SMS]School of Molecular Sciences, Arizona
State University, Tempe, AZ 85287, USA ROI

Introduction. Fluorescent labels have been critical in allowing us to
discriminate between homogeneous background and biomolecules of interest 1.
For instance, they have been useful in determining the locations of molecules
within neighboring regions by exploiting the nonlinear response (fluorophore
activation) of fluorophores to incoming light 2, 3. They have also allowed us
to determine the stoichiometry of protein complexes based on a spot’s emission
intensity 4, 5, 6, 7. In particular, we note that photobleaching event
counting experiments, where fluorophores are stochastically deactivated, lead
to step-like patterns in brightness traces 8.

Quantitative fluorescence analysis experiments have been used extensively
throughout biophysics to quantify the stoichiometry of a number of complexes
involved, for example, in bacterial flagellar switch 9, eukaryotic flagella
10, point centromere 11, 12, mammalian neurotransmitter receptors 13, human
calcium channels 14, transmembrane
$\alpha$-amino-3-hydroxy-5-methyl-4-isoxazolepropionic acid receptor-
regulatory proteins 15, T4 bacteriophage helicase loader protein 16, bacterial
oxidative phosphorylation com-plexes 17, microRNAs in processing bodies 18,
19, RNAs in a bacteriophage DNA-packaging motor 20, and other membrane
proteins and protein complexes 21, 22.

However, a quantitative analysis of the stoichiometry of a protein complex 7,
23 or the enumeration of the number of fluorophores within a diffraction-
limited spot 5, or other chemical properties of a system tagged with identical
fluorophore labels 24, unavoidably requires simplifying assumptions. One
critical assumption pervading such analyses is that chemically identical
fluorophores do not interact and therefore are photo-emissively identical as
well. Here, we reassess this fundamental assumption at the basis of data
derived from modern techniques such as PALM 2, 25, 26, STORM 27,
photobleaching event counting 8, RNA-FISH 28, and others 29, 30.

To carefully isolate and analyze the non-additive effects of fluorophore
interactions, we must account for a number of factors including noise
generated by acquisition devices (cameras or single photon detectors) 31, 32,
the quantized nature of photon emission (shot noise), and background
emissions. Therefore, an accurate assessment of non-additive effects as well
as a quantitative treament of fluorescence data requires a hierarchical
mathematical treatment of the stochastic effects arising from the
contributions mentioned above.

Previous studies have quantified some of these contributions. For example,
instrument response function (IRF) 33, 34, 35, shot noise 5, 36, and noise
from background fluorescence 37, 7. However, incorporating the effects of
unintended interactions among identical fluorophores and biomolecules _remains
an open challenge._ As a result, a forward model is required in order to
develop an inverse strategy to infer protein complex stoichiometry or even the
enumeration of multiply-labeled RNA, especially when these RNA are in closely-
spaced physical regions, from RNA-FISH experiments 38 is yet to be proposed.
In particular, the primary goal of this paper is not to elucidate the quantum
mechanical origin of interactions among fluorophores but to devise a strategy
to analyze imaging data.

Recent studies on fluorescently-labeled DNA-origami structures like
fluorocubes 39, 40, 41 have shown to what degree some identical fluorophores
interact when separated by distances ranging from 2-10 nm, leading to improved
photostability 39 but also increased photoblinking whose statistical signature
has very recently been proposed as a tool to determine molecular distances
below 10 nm. These experiments provide building blocks that may serve in
constructing models of interacting fluorophores 41.

Fluorocubes 39 are constructed using four 16 base-pair (bp) long double-
stranded DNA (dsDNA) as a highly photostable option for collecting long
trajectories of labeled biomolecules. The four DNA helices are assembled
together in such a way that the fluorophores attached to the ends of the
helices form the corners of a cuboid (see Fig. 1) separated by 2-6 nm. One of
the corner sites is usually kept reserved for a tag that binds to proteins at
specific locations and the last remaining one is left unlabeled.

The photophysical properties of these fluorocubes were studied and contrasted
with cubes labeled with only one dye (single-dye cubes) and with six dyes that
are separated by larger distances of 6-10 nm (large cubes) as well 39. It was
found that these properties significantly vary among fluorocubes depending on
the species of the dyes used. Overall, most fluorocube configurations blinked
less and exhibited reduced photobleaching rates compared to their single-dye
counterparts. Even the DNA scaffolding itself resulted in improved
photostability as single-dye cubes emitted many fold more photons in many
cases with longer lifetimes by comparison to a single dsDNA helix coupled to
one dye 39. Concretely, increasing the dye separations in such cubes from
$\sim$ 2-6 nm to $\sim$ 6-10 nm showed that photobleaching time decreases with
size, indicating that dye-dye interactions play an important role in the
fluorocube photophysics.

$F_{1}$$F_{6}$$F_{2}$$F_{3}$$F_{4}$$F_{5}$a = 2-4 nm2-4 nmb = 5.4 nm Figure 1:
Fluorocube. In a fluorocube, six fluorescent dyes (red) are attached to the
ends of the four 16 bp DNA helices. One position is usually reserved for a
functional tag (blue) to be linked to a molecule of interest 39 and last
remaining corner is left unlabeled.

Here, we make an attempt at building a model of fluorophore interactions in
order to derive quantities such as transition rates between states and
excitation probabilities from single photon arrivals. These quantities are
then estimated for the lowest energy states of fluorocubes from experimental
data. We also consider cubical structures of different sizes (single-dye cubes
and large cubes, respectively) to demonstrate the changes in photophysics
resulting from changes in interactions between dyes. We finally illustrate the
consequences of these interactions by analyzing synthetically generated
photobleaching event counting traces.

State Space. We first develop a quantum mechanical model to describe and label
all possible non-degenerate states of a fluorocube. In an effort to simplify
our modeling, we make the following physical assumptions based on typical time
scales and data collection procedure (see experimental methods section):

  1. 1.

the excitation pulse is infinitesimally narrow ($\sim 12$ ps wide) compared to
other time scales of the problem,

  2. 2.

the probability of simultaneously exciting two or more dyes by a single pulse
is zero since events where two photons are emitted are rarely observed ($\sim$
1 per $10^{4}$ photon events or less),

  3. 3.

the duration between pulses is long enough ($\sim 80$ ns) for the fluorocubes
to return to the ground state before the next pulse ($<4$ ns average
lifetime),

  4. 4.

fluorophores are well approximated by a two state system with a ground and an
excited state,

  5. 5.

photoblinking effects, induced by visiting rarely occupied states such as
triplet states, are ignored for simplicity as we are primarily interested in
interactions among fluorophores in their ground and first excited states 42,
and

  6. 6.

distances between the dyes of a fluorocube do not vary during the experiments,
and the upper and lower faces of a fluorocube as seen in Fig. 1 are squares
with equal separation between adjacent dyes. This may not always be true as
the distances between the dyes are affected by the surrounding chemical
environment 39 and cannot be accurately determined dynamically with currently
available techniques.

The quantum mechanical nature of interactions between fluorophores is well
known in the case of Förster resonance energy transfer (FRET) 43. For this
reason, we use a similar approach to develop a model for fluorocubes and
postulate the following Hamiltonian

$H(t)=\sum_{i=1}^{6}H_{i}+H_{int}+\delta H(t),$ (1)

where $H_{i}$ is the Hamiltonian for individiual dyes, $H_{int}$ is the
Hamiltonian for interactions among dyes, and $\delta H(t)$ is the time
dependent perturbation Hamiltonian that models excitation with light,
respectively. This Hamiltonian is symmetric under a transformation swapping
the dye labels, $(F_{1},F_{2},F_{3})\rightarrow(F_{6},F_{5},F_{4})$, as seen
in Fig. 1. This symmetry can be used to group all the energy eigenstates of
the fluorocube into a collection of non-degenerate energy eigenstates, without
the need to identify all specific interactions entering the full Hamiltonian
for a complex multi-molecular system.

In total, we have $2^{6}=64$ energy eigenstates for a collection of six two-
state systems. For a _large cube_ with distances between dyes varying from 6
to 10 nm, $H_{int}\approx 0$. Therefore, in this case alone, all fluorophores
act effectively independently leading to maximum allowable degeneracy among
the system’s energy eigenstates.

The 64 states of such a large cube can be grouped according to the number of
dyes excited at any instant: 6 degenerate states for one and five-dye
excitations each, 15 degenerate states for two and four-dye excitations each,
20 degenerate states for three-dye excitations, and one state for ground and
six-dye excitations each, respectively.

Interactions among fluorophores, however, reduce this degeneracy. Such
splitting among the degenerate states of the system is akin to level splitting
in the Zeeman effect 44 and is fully dictated by the symmetries of the
interaction Hamiltonian. Here, we focus on the geometrical symmetries of a
fluorocube where interactions are expected to be significant and do not
consider fine-splitting induced by the asymmetries of the individual molecules
in the assembly.

From the geometrical symmetries of a fluorocube (see Fig. 1), it can be seen
that the degenerates states will split into multiple levels; 6 degenerate one-
dye excitation states split into 3 levels, 15 degenerate two-dye excitation
states split into 9 levels, 20 degenerate three-dye excitation states split
into 10 levels, 15 degenerate four-dye excitation states split into 9 levels,
and 6 degenerate five-dye excitation states split into 3 levels, respectively.

We can mathematically represent these 35 non-degenerate excitations as

$\displaystyle\boldsymbol{\sigma}=\\{$
$\displaystyle\ket{1}^{(1)},\ket{1}^{(2)},\ket{1}^{(3)},$
$\displaystyle\ket{2}^{(1)},\ket{2}^{(2)},\ket{2}^{(3)},\ldots,\ket{2}^{(9)},$
$\displaystyle\ket{3}^{(1)},\ket{3}^{(2)},\ket{3}^{(3)},\ldots,\ket{3}^{(10)},$
$\displaystyle\ket{4}^{(1)},\ket{4}^{(2)},\ket{4}^{(3)},\ldots,\ket{4}^{(9)},$
$\displaystyle\ket{5}^{(1)},\ket{5}^{(2)},\ket{5}^{(3)},$
$\displaystyle\ket{6}^{(1)}\\},$ (2)

where the labels inside the bra-ket notation indicates the number of dyes
excited by the pulses and the superscripts only run over the non-degenerate
states. This labeling simplification can be done because degenerate states
cannot be distinguished from each other on account of similar lifetimes. We
keep the ground state $\ket{0}$ separate from this set for convenience as it
doesn’t require a stochastic treatment, as discussed later.

In an experiment, a sample containing a collection of fluorescent probes of
the same type is illuminated by a total of $N$ laser pulses. We label the
inter-pulse windows between times $t_{n-1}$ and $t_{n-1}+\Delta$ with $n$,
where $\Delta$ is the window size. Since we assume that fluorocubes are always
in the ground state at the beginning of these inter-pulse windows, the photon
arrival times can be considered independent and identically distributed (iid).
We represent these measurements with $\mu_{n}$.

Now that we have defined our state space and measurements, we postulate a
forward model to generate these measurements and then design an inverse
strategy to learn the parameters of interest. Similar mathematical
formulations for fluoresence data have been recently developed and applied
extensively for the modeling and analysis of experimental findings 45, 46, 47,
48, 49, 50, 51.

Forward Model. A detected photon may have been emitted by some background
source or by a fluorescent probe in the sample. Therefore, in addition to the
quantum states for the fluorescent probes, we introduce a background state
$\ket{b}$ that allows for a simple mathematical treatment of the background
emissions. The photons emitted by such a state are uniformly distributed over
the inter-pulse window with a probability of emission $\pi_{b}$. This uniform
assumption is validated from experimental data under the assumption that most
photons whose arrivals exceed 40 ns are background emissions; see Fig. 3 (b)
with almost equal photon counts per bin beyond 40 ns.

If the fluorescent probes get excited, the system’s states before photons are
emitted are represented by $s_{n}$ which take values from the set of states in
Eq. 2. Given the infinitesimal width of the excitation laser, the
probabilities of exciting the fluorocubes to states $\boldsymbol{\sigma}$ can
be collected into a constant probability vector $\boldsymbol{\pi_{\sigma}}$.
These probabilities are labeled $\pi_{i}$ (where $i=1,\ldots,35$ in order of
increasing excitation level) corresponding to the excited states in Eq. 2.
Additionally, the probability of staying in the ground state is labeled as
$\pi_{0}$, and since it can be computed directly from the number of inter-
pulse windows without photon detections, it is not treated as a random
variable.

Following excitation, the system’s state is randomly chosen from a Categorical
distribution (a generalization of the Bernoulli distribution for many options)

$s_{n}|\boldsymbol{\pi_{\sigma}},\pi_{b}\sim\mathbf{Categorical}_{\boldsymbol{\sigma},\ket{b}}(\boldsymbol{{\pi}_{\sigma}},\,\pi_{b}).$
(3)

On the other hand, inter-pulse windows can either be empty with no photons, or
have photon arrival times that are either exponentially or uniformly
distributed (for background) continuous random variables

$\displaystyle\mu_{n}$ $\displaystyle|s_{n},\boldsymbol{\lambda_{\sigma}}\sim$
$\displaystyle\begin{cases}\emptyset,&s_{n}=\ket{0}\\\
\mathbf{Exponential}(\lambda_{s_{n}}),&s_{n}\neq\ket{0},\,s_{n}\neq\ket{b}\\\
\mathbf{Uniform}(0,\Delta),&s_{n}=\ket{b},\end{cases}$ (4)

where $\lambda_{s_{n}}$ is the escape rate for the corresponding excited state
$s_{n}$. All escape rates corresponding to the 35 non-degenerate excited
states of Eq. 2 are labeled $\lambda_{i}$ for the states in Eq.2 and collected
in the set $\boldsymbol{\lambda_{\sigma}}$.

The forward model described above can be visualized using the graphical model
shown in Fig. 2 52. Based on this formulation, the likelihood function for the
arrival time of a photon is given by a mixture of probability distributions

$\displaystyle p(\mu_{n}|\boldsymbol{\lambda_{\sigma}},$
$\displaystyle\pi_{0},\boldsymbol{{\pi}_{\sigma}},\pi_{b})=\pi_{0}\delta_{\emptyset}(\mu_{n})$
$\displaystyle+\pi_{1}\,\mathbf{Exponential}(\mu_{n};\lambda_{1})$
$\displaystyle+\ldots$
$\displaystyle+\pi_{{35}}\,\mathbf{Exponential}(\mu_{n};\lambda_{35})$
$\displaystyle+\pi_{{b}}\,\mathbf{Uniform}(\mu_{n};0,\Delta)$ (5)

and, for a set of $N$ pulses, the likelihood becomes a product of many such
mixtures

$p(\mu_{1:N}|\boldsymbol{\lambda_{\sigma}},\pi_{0},\boldsymbol{\pi_{\sigma}},\pi_{b})=\prod_{n=1}^{N}p(\mu_{n}|\boldsymbol{\lambda_{\sigma}},\pi_{0},\boldsymbol{\pi_{\sigma}},\pi_{b}).$
(6)

Inverse Model. To learn these rates and excitation probabilities from the
photon arrival data, we use Bayesian inference. Using Bayes’ theorem, the
probability distribution over these parameters 52, 53, 54 (termed the
posterior) is given by

$\displaystyle
p(\boldsymbol{\lambda_{\sigma}},\boldsymbol{\pi_{\sigma}},\pi_{b}|\mu_{1:N},\pi_{0})$
$\displaystyle\propto
p(\mu_{1:N}|\boldsymbol{\lambda_{\sigma}},\pi_{0},\boldsymbol{\pi_{\sigma}},\pi_{b})$
$\displaystyle\times
p(\boldsymbol{\lambda_{\sigma}})p(\boldsymbol{\pi_{\sigma}},\pi_{b}).$ (7)

Those distributions multiplying the likelihood,
$p(\boldsymbol{\lambda_{\sigma}})$ and $p(\boldsymbol{\pi_{\sigma}},\pi_{b})$
are priors and selected on the basis of computational convenience and in whose
domains the associated parameters exist 53. Additionally, the influence of
these priors on the posterior distribution is minimal when large amounts of
data, incorporated through the likelihood, are available.

We use the following priors for the transition rates and excitation
probabilities

$\displaystyle\lambda_{i}$
$\displaystyle\sim\mathbf{Gamma}(A,\lambda_{ref}/A),\text{ and}$ (8)
$\displaystyle\boldsymbol{\pi_{\sigma}},\pi_{b}$
$\displaystyle\sim\mathbf{Dirichlet}_{\boldsymbol{\sigma},\,\ket{b}}(\boldsymbol{\alpha}_{\boldsymbol{\sigma},b}),$
(9)

where we choose $A=1$ and $\lambda_{ref}=1$ ns-1, and set all elements of the
vector $\boldsymbol{\alpha}_{\boldsymbol{\sigma},b}$ to be 1.

Figure 2:  Graphical Model. A graphical model depicting the random variables
and parameters involved in the generation of photon arrival data for
fluorocubes. Circles represent the random variables involved in the inverse
model. Circles shaded in blue correspond to the parameters of interest (rates
and probabilities), and the one shaded in gray corresponds to the
measurements, while the unshaded circles represent the hidden/latent variables
of the model. The arrows represent conditional dependence among variables. The
two plates enclosing the excitation rates/probabilities and hidden
variables/measurements indicate that these variables are repeated over indices
$i$ and $n$, respectively.

To sample transition rates and excitation probabilities from the posterior
distribution above, we use the Gibbs algorithm 52, which involves sampling
individual parameters from their respective conditional posteriors

$\displaystyle\lambda_{i}$ $\displaystyle\sim
p(\lambda_{i}|\boldsymbol{\lambda_{\sigma}}\backslash\lambda_{i},\pi_{0},\boldsymbol{\pi_{\sigma}},\pi_{b},\mu_{1:N}),\text{
and}$ (10) $\displaystyle\boldsymbol{\pi_{\sigma}},\pi_{b}\sim
p(\boldsymbol{\pi_{\sigma}},\pi_{b}|\boldsymbol{\lambda_{\sigma}},\pi_{0},\mu_{1:N}),$
(11)

where the parameters following a backslash “$\backslash$” are excluded from
the preceding set. The above Gibbs sampling scheme still requires brute-force
Metropolis-Hastings algorithm for each of the three steps as none of the
associated conditional distributions allow for direct sampling. We illustrate
results from the above sampling scheme in supplementary section 2.

The computational cost of brute force sampling does now help motivate the
(more complex) sampling scheme now presented below.

To generate conditional posteriors in a Gibbs sampling scheme from which we
can sample directly, we now exploit the conjugacy of Exponential-Gamma and
Multinomial-Dirichlet likelihood-prior pairs. To exploit these properties, we
must de-marginalize the distribution in Eq. 7 over the system’s states $s_{n}$
as follows

$\displaystyle p(\boldsymbol{\lambda_{\sigma}},$
$\displaystyle\boldsymbol{\pi_{\sigma}},\pi_{b}|\mu_{1:N},\pi_{0})\propto$
$\displaystyle\left(\sum_{s_{1:N}}p(\mu_{1:N},s_{1:N}|\boldsymbol{\lambda_{\sigma}},\pi_{0},\boldsymbol{\pi_{\sigma}},\pi_{b})\right)$
$\displaystyle\hskip 68.2866pt\times
p(\boldsymbol{\lambda_{\sigma}})\,p(\boldsymbol{\pi_{\sigma}},\pi_{b}).$ (12)

This de-marginalization creates a new set of variables, $s_{1:N}$, which we
must now also sample by supplementing the Gibbs algorithm presented from Eqs
10-11.

On account of these conjugacy conditions, the updated Gibbs sampling algorithm
is now as follows

$\displaystyle s_{n}$ $\displaystyle\sim
p(s_{n}\,|\,\boldsymbol{\lambda_{\sigma}},\pi_{0},\boldsymbol{\pi_{\sigma}},\pi_{b},s_{1:N}\backslash
s_{n},\mu_{1:N})$
$\displaystyle\propto\mathbf{Categorical}_{\boldsymbol{\sigma},\,\ket{b}}\Big{(}\mathbf{Exponential}(\mu_{n}|\lambda_{1})\,\pi_{1},$
$\displaystyle\ldots,\,\,\mathbf{Exponential}(\mu_{n}|\lambda_{35})\,\pi_{35},\pi_{b}/\Delta\Big{)}$
(13) $\displaystyle\boldsymbol{\pi_{\sigma}},\pi_{b}$ $\displaystyle\sim
p(\boldsymbol{\pi_{\sigma}}\,|\,\boldsymbol{\lambda_{\sigma}},\pi_{0},s_{1:N},\mu_{1:N})$
$\displaystyle\propto\mathbf{Dirichlet}_{\boldsymbol{\sigma},\,\ket{b}}(\,\boldsymbol{\alpha}_{\boldsymbol{\sigma},b}+\boldsymbol{\eta}_{\boldsymbol{\sigma},b}),\,\,\text{and}$
(14) $\displaystyle\lambda_{i}$ $\displaystyle\sim
p(\lambda_{i}\,|\,\boldsymbol{\lambda_{\sigma}}\backslash\lambda_{i},\pi_{0},\boldsymbol{\pi_{\sigma}},\pi_{b},s_{1:N},\mu_{1:N})$
$\displaystyle\propto\mathbf{Gamma}\big{(}\,A+\sum_{n=1}^{N}{\Delta_{i}(s_{n})},\,\lambda_{ref}/A$
$\displaystyle\hskip
85.35826pt+\sum_{n=1}^{N}\mu_{n}^{\Delta_{i}(s_{n})}\big{)},$ (15)

where $\boldsymbol{\eta}_{\boldsymbol{\sigma},b}$ is a vector containing the
number of pulses exciting a fluorocube to a given state in
$\boldsymbol{\sigma}$ and to the background state $\ket{b}$, and
$\Delta_{i}(s_{n})$ is 1 whenever $s_{n}=\ket{i}$ and otherwise 0,
respectively.

Figure 3:  Experimental Data Analysis. In row (a), we have three $256\times
256$ pixels raster-scanned images of samples containing three cubes labeled
with ATTO 647N dyes, each. The images on the left and the right are for
single-dye cubes and fluorocubes illuminated with 80% maximum laser power,
respectively. However, the center image for large-cubes was produced using
only 30% laser power. The laser power is varied in order to obtain
approximately equal photon numbers per probe per experimental run. Histograms
for photon arrival times (microtimes) recorded from these images are shown in
row (b). In row (c), each panel shows a bivariate posterior for escape rates
$\lambda_{i}$ (log scale) and corresponding probabilities $\pi_{i}$ as in Eq.
5. We smooth out the bivariate plots using the kernel density estimation (KDE)
tool available in Julia. The left panel shows the distribution for single-dye
cubes. The distribution is concentrated about $\lambda_{i}\approx 0.25$ ns-1
or a lifetime of around $1/\lambda_{i}$ $=4.0$ ns. $\pi_{b}\approx 0.10$ and
$\pi_{0}\approx 0.49$ in this case. The plot for large cubes in the center
panel shows no change in lifetime, suggesting insignificant interactions.
$\pi_{b}\approx 0.08$ and $\pi_{0}\approx 0.37$ for these cubes. In the
rightmost panel for fluorocubes, the peak moves towards a significantly
shorter lifetime of around 1.54 ns. The background probability $\pi_{b}$ here
is 0.12, slightly higher than that of non-interacting cubes, and
$\pi_{0}\approx 0.59$.

Analysis of ATTO-647N Fluorocubes. We now use the formulation described above
to learn how photophysical properties of fluorocubes and large cubes may
change as the distances between dyes are varied. The photon arrival data is
acquired as a collection of these cubical fluorescent probes are illuminated
by a pulsating laser around every 80 ns. The experiment is repeated many times
under different intensities of the laser (see the experimental methods
section).

In Fig. 3, we show captured images, microtime histograms, and three bi-variate
plots of the learned distributions for transition rates (lifetimes) and
excitation probabilities for three different types of cubes labeled with
ATTO-647N dyes, respectively.

For single-dye cubes and large cubes — illuminated with 80 % and 30 % of the
maximum laser power, respectively, we expect the interactions to not affect
the photophysics significantly and that is what we observe (since large cubes
are not affected by interactions, they are expected to emit far more photons
per probe than fluorocubes and are therefore illuminated with less power to
have the same number of photon emissions per probe). As shown in the left and
center panels of Fig. 3 (c), the most frequently sampled rates are of around
0.25 ns-1 which are equivalent to lifetimes of around 4.0 ns, close to the
value specified by the manufacturer ($\approx 3.5$ ns) 55 and therefore used
as an independent measurement here to compare parameters when interactions are
present. This provides one form of validation for the formalism we are using
here. Additionally, the probability of excitation is much higher for the large
cubes compared to the single-dye cubes, which is expected since a large cube
has 5 additional fluorophores that can be independently excited at the same
time.

The most frequently sampled rate in the case of fluorocubes illuminated with
80 % laser power (rightmost panel in Fig. 3 (c)), where interactions are
expected, is around 0.65 ns-1 or an equivalent lifetime of 1.54 ns. This
clearly indicates that photophysical properties of a collection of
fluorophores change significantly when they are in close proximity. However,
we do not observe any significant splitting among the states as we could not
isolate secondary peaks. This also suggests that the lowest energy excited
states have similar escape rates (same order of magnitude) and are equally
likely to be excited by a pulse. To make sure that this is not an artifact of
the computational algorithm, we demonstrate the algorithm’s robustness via
distinctly visible splitting for synthetically generated data for six states
with precisely known lifetimes and probabilities of excitations (see
supplementary section 1).

It should also be noted that excitation probabilities (normalized for laser
power and the number of fluorophores per probe) for the smaller ATTO647N
fluorocubes are significantly lower compared to its single-dye and the larger
six-dye counterparts. This change in excitation probability as distance
between fluorophores changes plays a crucial role in determining the
brightness ($\pi_{i}/\Delta$) of the fluorescence signal. This is also
confirmed in the experiments by Niekamp et al. 39(see Figs. 3 & 8 of their
supplementary document), where they observe that single-dye ATTO647N cubes are
typically as bright as the six-dye ATTO647N fluorocubes or brighter if the
illumination is strong ($\sim 1.7$ times brighter at exposure values of 2.0
$\mu$J/$\mu$m2). This suggests that interactions significantly dampen the
brightness per fluorophore when a collection of fluorophores comes closer
together. This is similar to homo-FRET 56 where FRET between identical
fluorophores decreases the quantum yield (quenching) and the lifetimes of the
individual probes.

This dampening effect (per fluorophore) is observed to not depend on the
species of dye used to label the cubes 39. To demonstrate this, we repeated
our experiments with large cubes and fluorocubes labeled with Cy3 dyes. We
again observe (see Fig. 3 of the supplementary) significantly smaller
excitation probability per fluorophore for Cy3 fluorocubes compared to Cy3
large cubes.

Figure 4: Photobleaching Traces. Here, we show two sections of a synthetically
generated photobleaching counting trace assuming an EMCCD camera model for a
collection consisting of one ATTO647N fluorocube (with six tightly packed
interacting dyes each) and two well-separated ATTO647N (non-interacting) dyes.
In (a), we see that brightness is reduced when one of the non-interacting dyes
photobleaches by around $14\times 10^{6}$ ADU/s. We note in (b) that the
brightness again reduces by a similar amount even though six fluorophores in
the fluorocube photobleach at the same time. This dampened reduction in
brightness may lead to significant undercounting of subunits of a biomolecular
complex under investigation.

Effects of Non-additivity. Photobleaching occurs when fluorophores are
chemically deactivated upon repeated excitations. This phenomenon finds
applications in event counting experiments where the brightness decreases in a
step-like pattern 8 as fluorophores successively photobleach.

In typical experiments, the accurate determination of the number and size of
the steps is affected 57 by instrumental noise, high numbers of fluorophores,
overlapping photobleaching events, spatially varying intensity profile, and
interactions between fluorophores and the surrounding molecules in the sample.

In the absence of interactions, the brightness should decrease by
approximately the same amount every time a fluorophore photobleaches. However,
interactions can affect many aspects of physics dictating photobleaching event
counting, including brightness of fluorophores 39, 40, the stability of the
chemical bonds against repeated excitations 39, 40, and lifetimes of the
excited states (as demonstrated earlier). For instance, it has been observed
that, in the case of ATTO647N fluorocubes, the brightness is significantly
reduced by interactions 39, most likely due to quenching by homo-FRET 56 type
of mechanism typically reducing fluorescence lifetime as well. On the other
hand, photostability of these fluorocubes improves dramatically and many fold
more photons are emitted 39 resulting in delayed photobleaching, one
contributing factor for which may simply be the highly dampened emission rate
per fluorophore, however, in some cases $\sim$ 43 fold more photons are
emitted 39 compared to single dyes which cannot be explained without a full
quantum mechanical treatment.

The effects of interactions noted above may manifest themselves in the form of
irregular steps sizes (non-additive brightness) in photobleaching traces. This
raises doubt on the assumption currently underlying all step counting methods
relying on equal average step sizes (albeit varying around the average on
account on photon shot noise and stochastic detector output) 8.

To illustrate non-additivity effects in step counting, we generate synthetic
photobleaching traces using the recipe in Bryan et al. 5. We generate
photobleaching traces for a collection of interacting and non-interacting
fluorophores. The region of interest (ROI) under study contains one
interacting cluster of six fluorophores (a fluorocube) and another cluster of
two sparsely distributed non-interacting fluorophores.

The algorithm used here for generating photobleaching traces requires us to
specify brightness values for the fluorescent probes (isolated dyes and
fluorocubes), which can be computed from the excitation probabilities as shown
earlier. As the brightness values recorded by a camera are typically noisy, we
model such measurements made by an EMCCD camera with a Gamma distribution 5,

$\mu_{d}\sim\mathbf{Gamma}(\mu_{b}/2+\mu_{e},\,\,2G),$ (16)

where $\mu_{e}$ is the number of photons emitted in a second by the emitters,
$\mu_{b}$ is the number of background photons emitted in a second, $\mu_{d}$
is the number of ADUs recorded by the detector in the same time interval, and
$G$ is the camera gain, respectively.

From probability of excitation, we can compute the mean $\mu_{e}$ used in Eq.
16 for both independent fluorophores and fluorocubes as $\pi_{i}/\Delta$. We
recover that fluorocubes and independent fluorophores have roughly the same
brightness (consistent with supplementary Figs. 3, 5, and 8 of Niekamp et al.
39).

More concretely, to compute the brightness in our experiments with 80 %
maximum laser power and three probes of the same type in the field of view,
both single-dye cubes and fluorocubes have similar effective excitation
probabilities of around 0.14 and 0.10 per pulse per probe (see Fig. 3),
respectively. With an inter-pulse time, $\Delta$, of around 100 ns, this
corresponds to an emission rate, $\mu_{e}$, of around $1.4\times 10^{6}$ and
$1.0\times 10^{6}$ photons per second per probe. Additionally, the
probabilities for background emissions, ${\pi}_{b}$, in both cases are similar
and equal to approximately 0.10 and 0.12, which are equivalent to background
emission rates, $\mu_{b}$, of $1.0\times 10^{6}$ and $1.2\times 10^{6}$
photons/s. To generate the traces using the model from Eq. 16, we need to
specify a camera gain and set it to 10 (a value, for instance, similar to that
used by Bryan et al. 5).

Fig. 4, illustrates what can be expected to happen when a single fluorophore
(not interacting with any other) photobleaches (panel a) as compared to when
an entire fluorocube photobleaches (panel b).

As is apparent from Fig. 4, the dampened brightness of the fluorocube has
immediate implications: the step size cannot be considered an independent and
identically distributed random variable for each fluorophore and may strongly
depend on its local environment.

Conclusion. Interactions among fluorophores and the surrounding chemical
environment play a consequential role in the quantitative analysis of
fluorescence imaging data. From our preliminary analysis of the photon arrival
data for DNA origami structures labeled with ATTO647N dyes, we observe
significant changes in the excitation probabilities and escape rates
(lifetimes) of the excited states when interactions are involved. Interactions
are also observed to significantly dampen the brightness of an interacting
cluster of identical flurophores. The DNA scaffolding itself contributes to
modification of the dye photophysics 39. These changes play an important role
in the analysis of photobleaching event counting methods 8, where brightness
traces for samples with large number of densely packed fluorophores cannot be
modeled accurately within the existing non-fluorophore-interaction paradigm.

What is more, this finding also suggests that determination of a protein
complex’s stoichiometry using PALM 2, 25, 26 or photobleaching event counting
8 or related methods 38 may be impacted by the local distance between complex
subunits.

So far, we have demonstrated that distances between fluorophores have an
effect on the heights and lengths of steps in a photobleaching trace. However,
in principle, assuming that we know the number of molecules, we can begin
quantifying inter-molecular distances from accurate lifetime measurements in a
manner similar to FRET 43 as well as from changes in emission frequencies.

More precisely, we may design a means by which to precisely learn the
interaction Hamiltonian, $H_{int}$, in Eq. 1 (and parameters such as escape
rates, excitation probabilities, and photobleaching times) as a function of
fluorophore separation, orientiation, and dipolar coupling. This calls for
further experiments into precisely characterizing the relationship between
photophysics and spatial distributions of identical fluorophores.

Experimental Methods. We acquired data on an SP8 point-scanning system (Leica)
with a Picoquant TCSPC module and excitation delivered by a WLL at a 10 MHz
repetition rate. The emission spectrum was split onto two SMD HyD detectors:
581-603 and 608-687 nm for 561 nm excitation, or 646-670 and 675-752 nm for
640 nm excitation. We used an HC PL APO CS2 100x/1.4NA objective, a pinhole
size of 1AU, 9.5 nm pixels, and a dwell-time of 97.66 microseconds. Data was
extracted from .ptu files using a parser written in python.

We assembled the large cubes 58 and the fluorocubes as described by Niekamp et
al. 39. Briefly, for each six-dye fluorocube and single-dye cube we used four
32 bp long oligonucleotide strands, each modified either with dyes or biotin
(sequences shown in supplementary Table 1). We mixed each of the four oligos
(for the fluorocubes) or the 28 oligos (for the large cubes, sequences shown
in supplementary Table 1) to a final concentration of 10 $\mu$M in folding
buffer (5 mM Tris pH 8.5, 1 mM EDTA and 40 mM MgCl2). We then annealed the
oligos by denaturation at 85 ∘C for 5 min, followed by cooling from 80 to 65 ∘
C with a decrease of 1 ∘C per 5 min, followed by further cooling from 65 to 25
∘ C with a decrease of 1 ∘C per 20 min. Afterwards the samples were held at 4
∘C. We then analyzed the folding products by 3.0$\%$ agarose gel
electrophoresis in TBE (45 mM Tris-borate and 1 mM EDTA) with 12 mM MgCl2 at
70 V for 2.5 h on ice. We finally purified the samples by extraction and
centrifugation in Freeze’N Squeeze columns (Bio-Rad Sciences, 732-6165).

We prepared flow-cells as described by Niekamp et al. 39. First, we used a
laser cutter to cut custom three-cell flow chambers out of double-sided
adhesive sheets (Soles2dance, 9474-08x12 - 3M 9474LE 300LSE). We cleaned 170
$\mu$m thick coverslips (Zeiss, 474030-9000-000) in a 5$\%$ v/v solution of
Hellmanex III (Sigma, Z805939-1EA) at 50 ∘ C overnight and washed extensively
with Milli-Q water afterwards. Then, we used three-cell flow chambers together
with glass slides (Thermo Fisher Scientific, 12-550-123) and coverslips to
assemble the flow-cells.

The preparation method for flow-cells is the same for the fluorocubes and the
large cubes 39. Briefly, we first added 10 $\mu$l of 5 mg/ml Biotin-BSA
(Thermo Scientific, 29130) in BRB80 (80 mM Pipes (pH 6.8), 1 mM MgCl2, 1 mM
EGTA) to the flow-cell and incubated for 2 min. We then added an additional 10
$\mu$l of 5 mg/ml Biotin-BSA in BRB80 and incubated for another 2 min.
Afterwards, we the flow-cell was washed with 20 $\mu$l of fluorocube buffer
(20 mM Tris pH 8.0, 1 mM EDTA, 20 mM Mg-Ac and 50 mM NaCl) with 2 mg/ml of
$\beta$-casein (Sigma, C6905) and 0.4 mg/ml $\kappa$-casein (Sigma, C0406).
Next, we added 10 $\mu$l of 0.5 mg/ml Streptavidin (Vector Laboratories,
SA-5000) in PBS (pH 7.4), incubated for 2 min and then washed with 20 $\mu$l
of fluorocube buffer with 2 mg/ml $\beta$-casein and 0.4 mg/ml
$\kappa$-casein. Afterwards, we added either fluorocubes or large cubes in
fluorocube buffer with 2 mg/ml $\beta$-casein and 0.4 mg/ml $\kappa$-casein
and incubated for 5 min. Next, the flow-cell was washed with 30 $\mu$l of
fluorocube buffer with 2 mg/ml $\beta$-casein and 0.4 mg/ml $\kappa$-casein.
Finally, we added the protocatechuic acid (PCA) /
protocatechuate-3,4-dioxygenase (PCD) / Trolox oxygen scavenging system (3, 4)
in fluorocube buffer with 2 mg/ml $\beta$-casein, and 0.4 mg/ml
$\kappa$-casein to the flow-cell. For the PCA/PCD/Trolox oxygen scavenging
system we used 2.5 mM of PCA (Sigma, 37580) at pH 9.0, 5 U of PCD (Oriental
Yeast Company Americas Inc., 46852004), and 1 mM Trolox (Sigma, 238813) at pH
9.5.

We thank Dr Andrew York, Dr Nico Stuurman, and Dr Maria Ingaramo for providing
the experimental data, and for regular feedback and discussions. We also thank
Dr Douglas Shepherd for setting up the collaboration and providing insight
into the workings of detectors and other experimental equipment. S. P.
acknowledges support from the NIH NIGMS (R01GM130745) for supporting early
efforts in nonparametrics and NIH NIGMS (R01GM134426) for supporting single-
photon efforts.

## References

  * Shashkova and Leake 2017 Shashkova, S.; Leake, M. Single-molecule fluorescence microscopy review: shedding new light on old problems. _Bioscience Reports_ 2017, _37_ , BSR20170031
  * Betzig et al. 2006 Betzig, E.; Patterson, G. H.; Sougrat, R.; Lindwasser, O. W.; Olenych, S.; Bonifacino, J. S.; Davidson, M. W.; Lippincott-Schwartz, J.; Hess, H. F. Imaging Intracellular Fluorescent Proteins at Nanometer Resolution. _Science_ 2006, _313_ , 1642–1645
  * Hell and Wichmann 1994 Hell, S. W.; Wichmann, J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy. _Opt. Lett._ 1994, _19_ , 780–782
  * Tsekouras et al. 2016 Tsekouras, K.; Custer, T. C.; Jashnsaz, H.; Walter, N. G.; Pressé, S. A novel method to accurately locate and count large numbers of steps by photobleaching. _Molecular Biology of the Cell_ 2016, _27_ , 3601–3615, PMID: 27654946
  * Bryan et al. 2020 Bryan, J. S.; Sgouralis, I.; Pressé, S. Enumerating High Numbers of Fluorophores from Photobleaching Experiments: a Bayesian Nonparametrics Approach. _bioRxiv_ 2020,
  * Arant and Ulbrich 2014 Arant, R. J.; Ulbrich, M. H. Deciphering the Subunit Composition of Multimeric Proteins by Counting Photobleaching Steps. _ChemPhysChem_ 2014, _15_ , 600–605
  * Ulbrich and Isacoff 2007 Ulbrich, M. H.; Isacoff, E. Y. Subunit counting in membrane-bound proteins. _Nature Methods_ 2007, _4_ , 319–321
  * Chen et al. 2014 Chen, Y.; Deffenbaugh, N. C.; Anderson, C. T.; Hancock, W. O. Molecular counting by photobleaching in protein complexes with many subunits: best practices and application to the cellulose synthesis complex. _Molecular Biology of the Cell_ 2014, _25_ , 3630–3642, PMID: 25232006
  * Delalez et al. 2010 Delalez, N. J.; Wadhams, G. H.; Rosser, G.; Xue, Q.; Brown, M. T.; Dobbie, I. M.; Berry, R. M.; Leake, M. C.; Armitage, J. P. Signal-dependent turnover of the bacterial flagellar switch protein FliM. _Proceedings of the National Academy of Sciences_ 2010, _107_ , 11347–11351
  * Engel et al. 2009 Engel, B. D.; Ludington, W. B.; Marshall, W. F. Intraflagellar transport particle size scales inversely with flagellar length: revisiting the balance-point length control model. _Journal of Cell Biology_ 2009, _187_ , 81–89
  * Lawrimore et al. 2011 Lawrimore, J.; Bloom, K. S.; Salmon, E. Point centromeres contain more than a single centromere-specific Cse4 (CENP-A) nucleosome. _Journal of Cell Biology_ 2011, _195_ , 573–582
  * Coffman et al. 2011 Coffman, V. C.; Wu, P.; Parthun, M. R.; Wu, J.-Q. CENP-A exceeds microtubule attachment sites in centromere clusters of both budding and fission yeast. _Journal of Cell Biology_ 2011, _195_ , 563–572
  * McGuire et al. 2012 McGuire, H.; Aurousseau, M. R. P.; Bowie, D.; Blunck, R. Automating single subunit counting of membrane proteins in mammalian cells. _The Journal of Biological Chemistry_ 2012, _287_ , 35912–35921
  * Demuro et al. 2011 Demuro, A.; Penna, A.; Safrina, O.; Yeromin, A. V.; Amcheslavsky, A.; Cahalan, M. D.; Parker, I. Subunit stoichiometry of human Orai1 and Orai3 channels in closed and open states. _Proceedings of the National Academy of Sciences_ 2011, _108_ , 17832–17837
  * Hastie et al. 2013 Hastie, P.; Ulbrich, M. H.; Wang, H.-L.; Arant, R. J.; Lau, A. G.; Zhang, Z.; Isacoff, E. Y.; Chen, L. AMPA receptor/TARP stoichiometry visualized by single-molecule subunit counting. _Proceedings of the National Academy of Sciences_ 2013, _110_ , 5163–5168
  * Arumugam et al. 2012 Arumugam, S. R.; Lee, T.-H.; Benkovic, S. J. Investigation of Stoichiometry of T4 Bacteriophage Helicase Loader Protein (gp59). _The Journal of Biological Chemistry_ 2012, _284_ , 29283–29289
  * Llorente-Garcia et al. 2014 Llorente-Garcia, I.; Lenn, T.; Erhardt, H.; Harriman, O. L.; Liu, L.-N.; Robson, A.; Chiu, S.-W.; Matthews, S.; Willis, N. J.; Bray, C. D. et al. Single-molecule in vivo imaging of bacterial respiratory complexes indicates delocalized oxidative phosphorylation. _Biochimica et Biophysica Acta (BBA) - Bioenergetics_ 2014, _1837_ , 811–824
  * Pitchiaya et al. 2013 Pitchiaya, S.; Krishnan, V.; Custer, T. C.; Walter, N. G. Dissecting non-coding RNA mechanisms in cellulo by Single-molecule High-Resolution Localization and Counting. _Methods_ 2013, _63_ , 188–199, Non-coding RNA Methods
  * Pitchiaya et al. 2014 Pitchiaya, S.; Heinicke, L. A.; Custer, T. C.; Walter, N. G. Single Molecule Fluorescence Approaches Shed Light on Intracellular RNAs. _Chemical Reviews_ 2014, _114_ , 3224–3265
  * Shu et al. 2007 Shu, D.; Zhang, H.; Jin, J.; Guo, P. Counting of six pRNAs of phi29 DNA-packaging motor with customized single-molecule dual-view system. _The EMBO Journal_ 2007, _26_ , 527–537
  * Leake et al. 2006 Leake, M. C.; Chandler, J. H.; Wadhams, G. H.; Bai, F.; Berry, R. M.; Armitage, J. P. Stoichiometry and turnover in single, functioning membrane protein complexes. _Nature_ 2006, _443_ , 355–358
  * Das et al. 2007 Das, S. K.; Darshi, M.; Cheley, S.; Wallace, M. I.; Bayley, H. Membrane Protein Stoichiometry Determined from the Step-Wise Photobleaching of Dye-Labelled Subunits. _ChemBioChem_ 2007, _8_ , 994–999
  * Singh et al. 2020 Singh, A.; Van Slyke, A. L.; Sirenko, M.; Song, A.; Kammermeier, P. J.; Zipfel, W. R. Stoichiometric analysis of protein complexes by cell fusion and single molecule imaging. _Scientific Reports_ 2020, _10_ , 14866
  * Dhara et al. 2020 Dhara, A.; Sadhukhan, T.; Sheetz, E. G.; Olsson, A. H.; Raghavachari, K.; Flood, A. H. Zero-Overlap Fluorophores for Fluorescent Studies at Any Concentration. _Journal of the American Chemical Society_ 2020, _142_ , 12167–12180
  * Sengupta and Lippincott-Schwartz 2012 Sengupta, P.; Lippincott-Schwartz, J. Quantitative analysis of photoactivated localization microscopy (PALM) datasets using pair-correlation analysis. _BioEssays_ 2012, _34_ , 396–405
  * Lee et al. 2012 Lee, S.-H.; Shin, J. Y.; Lee, A.; Bustamante, C. Counting single photoactivatable fluorescent molecules by photoactivated localization microscopy (PALM). _Proceedings of the National Academy of Sciences_ 2012, _109_ , 17436–17441
  * Rush et al. 2006 Rush, M. J.; Bates, M.; Zhuang, X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). _Nature Methods_ 2006, _3_ , 793–796
  * Amann and Fuchs 2008 Amann, R.; Fuchs, B. M. Single-cell identification in microbial communities by improved fluorescence in situ hybridization techniques. _Nature Reviews Microbiology_ 2008, _6_ , 339–348
  * Grubmayer et al. 2019 Grubmayer, K. S.; Yserentant, K.; Herten, D.-P. Photons in - numbers out: perspectives in quantitative fluorescence microscopy for in situ protein counting. _Methods and Applications in Fluorescence_ 2019, _7_ , 012003
  * Dankovich and Rizzoli 2021 Dankovich, T. M.; Rizzoli, S. O. Challenges facing quantitative large-scale optical super-resolution, and some simple solutions. _iScience_ 2021, _24_ , 102134
  * Mandracchia et al. 2020 Mandracchia, B.; Hua, X.; Guo, C.; Son, J.; Jia, T. U. . S. Fast and accurate sCMOS noise correction for fluorescence microscopy. _Nature Communications_ 2020, _11_ , 94–
  * Hirsch et al. 2013 Hirsch, M.; Wareham, R. J.; Martin-Fernandez, M. L.; Hobson, M. P.; Rolfe, D. J. A Stochastic Model for Electron Multiplication Charge-Coupled Devices – From Theory to Practice. _PLOS ONE_ 2013, _8_ , 1–13
  * Luchowski et al. 2009 Luchowski, R.; Gryczynski, Z.; Sarkar, P.; Borejdo, J.; Szabelski, M.; Kapusta, P.; Gryczynski, I. Instrument response standard in time-resolved fluorescence. _Review of Scientific Instruments_ 2009, _80_ , 033109
  * Szabelski et al. 2009 Szabelski, M.; Luchowski, R.; Gryczynski, Z.; Kapusta, P.; Ortmann, U.; Gryczynski, I. Evaluation of instrument response functions for lifetime imaging detectors using quenched Rose Bengal solutions. _Chemical Physics Letters_ 2009, _471_ , 153–159
  * Lakowicz 2013 Lakowicz, J. R. _Principles of fluorescence spectroscopy_ ; Springer science & business media, 2013
  * Sgouralis et al. 2019 Sgouralis, I.; Madaan, S.; Djutanta, F.; Kha, R.; Hariadi, R. F.; Pressé, S. A Bayesian Nonparametric Approach to Single Molecule Förster Resonance Energy Transfer. _The Journal of Physical Chemistry B_ 2019, _123_ , 675–688
  * Coffman and Wu 2012 Coffman, V. C.; Wu, J.-Q. Counting protein molecules using quantitative fluorescence microscopy. _Trends in Biochemical Sciences_ 2012, _37_ , 499–506
  * Xie et al. 2018 Xie, F.; Timme, K. A.; Wood, J. R. Using Single Molecule mRNA Fluorescent in Situ Hybridization (RNA-FISH) to Quantify mRNAs in Individual Murine Oocytes and Embryos. _Scientific Reports_ 2018, _8_ , 7930
  * Niekamp et al. 2019 Niekamp, S.; Stuurman, N.; Vale, R. D. A 6-nm ultra-photostable DNA Fluorocube for fluorescence imaging. _bioRxiv_ 2019,
  * Schröder et al. 2019 Schröder, T.; Scheible, M. B.; Steiner, F.; Vogelsang, J.; Tinnefeld, P. Interchromophoric Interactions Determine the Maximum Brightness Density in DNA Origami Structures. _Nano Letters_ 2019, _19_ , 1275–1281
  * Helmerich et al. 2022 Helmerich, D. A.; Beliu, G.; Taban, D.; Meub, M.; Streit, M.; Kuhlemann, A.; Doose, S.; Sauer, M. Sub-10 nm fluorescence imaging. _bioRxiv_ 2022,
  * Lakowicz 2006 Lakowicz, J. R., Ed. _Principles of Fluorescence Spectroscopy_ ; Springer US: Boston, MA, 2006; pp 277–330
  * Jones and Bradshaw 2019 Jones, G. A.; Bradshaw, D. S. Resonance Energy Transfer: From Fundamental Theory to Recent Applications. _Frontiers in Physics_ 2019, _7_ , 100
  * Griffiths and Schroeter 2018 Griffiths, D. J.; Schroeter, D. F. _Introduction to Quantum Mechanics_ , 3rd ed.; Cambridge University Press, 2018
  * Sgouralis and Pressé 2017 Sgouralis, I.; Pressé, S. ICON: An Adaptation of Infinite HMMs for Time Traces with Drift. _Biophysical Journal_ 2017, _112_ , 2117–2126
  * Sgouralis and Pressé 2017 Sgouralis, I.; Pressé, S. An Introduction to Infinite HMMs for Single-Molecule Data Analysis. _Biophysical Journal_ 2017, _112_ , 2021–2029
  * Jazani et al. 2019 Jazani, S.; Sgouralis, I.; Shafraz, O. M.; Levitus, M.; Sivasankar, S.; Pressé, S. An alternative framework for fluorescence correlation spectroscopy. _Nature Communications_ 2019, _10_ , 3662
  * Sgouralis et al. 2018 Sgouralis, I.; Whitmore, M.; Lapidus, L.; Comstock, M. J.; Pressé, S. Single molecule force spectroscopy at high data acquisition: A Bayesian nonparametric analysis. _The Journal of Chemical Physics_ 2018, _148_ , 123320
  * Kilic et al. 2021 Kilic, Z.; Sgouralis, I.; Pressé, S. Generalizing HMMs to Continuous Time for Fast Kinetics: Hidden Markov Jump Processes. _Biophysical Journal_ 2021, _120_ , 409–423
  * Tavakoli et al. 2020 Tavakoli, M.; Jazani, S.; Sgouralis, I.; Shafraz, O. M.; Sivasankar, S.; Donaphon, B.; Levitus, M.; Pressé, S. Pitching Single-Focus Confocal Data Analysis One Photon at a Time with Bayesian Nonparametrics. _Phys. Rev. X_ 2020, _10_ , 011021
  * Kilic et al. 2021 Kilic, Z.; Sgouralis, I.; Heo, W.; Ishii, K.; Tahara, T.; Pressé, S. Extraction of rapid kinetics from smFRET measurements using integrative detectors. _Cell Reports Physical Science_ 2021, _2_ , 100409
  * Bishop 2006 Bishop, C. M. _Pattern Recognition and Machine Learning (Information Science and Statistics)_ ; Springer-Verlag: Berlin, Heidelberg, 2006
  * Sivia 2006 Sivia, D. M. _Data Analysis : A Bayesian tutorial_ ; Oxford University Press: Oxford, New York, 2006
  * van de Schoot et al. 2021 van de Schoot, R.; Depaoli, S.; King, R.; Kramer, B.; Märtens, K.; Tadesse, M. G.; Vannucci, M.; Gelman, A.; Veen, D.; Willemsen, J. et al. Bayesian statistics and modelling. _Nature Reviews Methods Primers_ 2021, _1_ , 1
  * 55 ATTO647N Dye Specifications. https://www.atto-tec.com/fileadmin/user_upload/Katalog_Flyer_Support/ATTO_647N.pdf, Accessed: 2021-09-29
  * Gautier et al. 2001 Gautier, I.; Tramier, M.; Durieux, C.; Coppey, J.; Pansu, R.; Nicolas, J.-C.; Kemnitz, K.; Coppey-Moisan, M. Homo-FRET Microscopy in Living Cells to Measure Monomer-Dimer Transition of GFP-Tagged Proteins. _Biophysical Journal_ 2001, _80_ , 3000–3008
  * Liesche et al. 2015 Liesche, C.; Grußmayer, K.; Ludwig, M.; Wörz, S.; Rohr, K.; Herten, D.-P.; Beaudouin, J.; Eils, R. Automated Analysis of Single-Molecule Photobleaching Data by Statistical Modeling of Spot Populations. _Biophysical Journal_ 2015, _109_ , 2352–2362
  * Scheible et al. 2015 Scheible, M. B.; Ong, L. L.; Woehrstein, J. B.; Jungmann, R.; Yin, P.; Simmel, F. C. A Compact DNA Cube with Side Length 10 nm. _Small_ 2015, _11_ , 5200–5205