WGMwgm QEqe EPep PMSpms BECbec DEde

[type=editor, orcid=0000-0002-7796-7717]

[1]

url]htasnim.github.io

Conceptualization, Methodology, Software, Data curation, Validation,
Visualization, Writing - Original draft preparation, Writing - review and
editing

1]organization=University of New Mexico, city=Albuquerque, state=New Mexico,
postcode=87131, country=United States

[type=editor,orcid=0000-0001-5030-9979]

url]soumyadutta-cse.github.io

Conceptualization, Supervision, Methodology, Validation, Writing - Original
draft preparation, Writing - review and editing

2]organization=Indian Institute of Technology Kanpur, state=Uttar Pradesh,
postcode=208016, country=India

[type=editor] URL]moseslab.cs.unm.edu

Supervision, Writing - review and editing

[cor1]Corresponding author

# Dynamic Spatio-Temporal Summarization using Information Based Fusion

Humayra Tasnim<EMAIL_ADDRESS>[  Soumya Dutta<EMAIL_ADDRESS>[
Melanie Moses<EMAIL_ADDRESS>
###### Abstract

In the era of burgeoning data generation, managing and storing large-scale
time-varying datasets poses significant challenges. With the rise of
supercomputing capabilities, the volume of data produced has soared,
intensifying storage and I/O overheads. To address this issue, we propose a
dynamic spatio-temporal data summarization technique that identifies
informative features in key timesteps and fuses less informative ones. This
approach minimizes storage requirements while preserving data dynamics. Unlike
existing methods, our method retains both raw and summarized timesteps,
ensuring a comprehensive view of information changes over time. We utilize
information-theoretic measures to guide the fusion process, resulting in a
visual representation that captures essential data patterns. We demonstrate
the versatility of our technique across diverse datasets, encompassing
particle-based flow simulations, security and surveillance applications, and
biological cell interactions within the immune system. Our research
significantly contributes to the realm of data management, introducing
enhanced efficiency and deeper insights across diverse multidisciplinary
domains. We provide a streamlined approach for handling massive datasets that
can be applied to in situ analysis as well as post hoc analysis. This not only
addresses the escalating challenges of data storage and I/O overheads but also
unlocks the potential for informed decision-making. Our method empowers
researchers and experts to explore essential temporal dynamics while
minimizing storage requirements, thereby fostering a more effective and
intuitive understanding of complex data behaviors.

###### keywords:

Information Theory Specific Mutual Information Data Fusion Spatio-temporal
Fusion Data Summarization Data Visualization Time-Varying Data

The research introduces a novel and adaptable method for interpreting
informative features of large scale spatiotemporal data, applicable to diverse
datasets from different domains.

The proposed technique identifies key informative timesteps and uses
information-based fusion to summarize salient patterns of important features
in less informative timesteps, leading to significant storage optimization
without compromising data insights.

Proposed approach facilitates effective visualization, tracking, and analysis
of temporal changes in the datasets.

##  1 Introduction

In today’s data-driven world, the exponential growth in data generation has
brought forth significant challenges for storage and associated I/O overheads.
Modern supercomputing capabilities have reduced the computational cost of
producing data, resulting in the creation of massive datasets at an
accelerating pace [Reed and Dongarra (2015); Childs (2015)]. Many of these
datasets exhibit a dynamic temporal nature, consisting of multiple timesteps
that may be redundant or offer minimal new information when analyzed timestep
by timestep. The storage burden of retaining these less informative timesteps
poses a critical issue, calling for novel techniques that can efficiently
manage such large-scale time-varying datasets.

To address this challenge, we propose a data summarization technique that aims
to minimize the storage overhead while preserving the vital temporal dynamics
of the dataset. We also emphasize visualizing these dynamics by tracking
changes over time. Our approach involves an dynamic spatio-temporal technique,
which identifies both key timesteps and redundant timesteps. We store the key
timesteps and summarize the redundant multiple timesteps into one,
highlighting the salient features. The summarization technique effectively
ensures storage reduction with minimal loss. We propose to use information-
theoretic measures namely the Specific Mutual Information or SMI-guided fusion
for the summaries.

The core idea of the summarization technique is to identify informative
temporal features within the redundant timesteps and fuse them using
principles from information theory. By selecting the most relevant features
from the redundant timesteps and summarizing them through information-guided
fusion, we ensure that the temporal dynamics are retained. This approach not
only optimizes storage requirements but also facilitates the visualization and
tracking of information change over time, providing valuable insights into the
underlying data patterns.

In the paper, we present the details of our data summarization technique,
dynamic spatio-temporal framework, and the information-guided fusion process.
We demonstrate the technique’s versatility by applying it to different
datasets, including scalar data from particle-based simulations and image data
from surveillance footage and cellular interactions. Our results show
significant storage reduction without compromising critical insights, proving
the effectiveness of our approach for efficient data management and
visualization.

Our research aim in this work is to develop a method that efficiently handles
large-scale time-varying datasets across various domains. Our solution seeks
to bridge the data reduction landscape by providing approaches to effectively
manage large and intricate temporal datasets. Our goal is to achieve this
while retaining essential features and enabling robust visualization
techniques. By leveraging the power of information theory, we aim to
contribute to a more efficient and insightful data management strategy,
especially in the context of dynamic and multifaceted datasets.

The contributions of the paper are:

  * •

Develop a dynamic spatio-temporal summarization (DSTS) technique for large-
scale time-varying datasets. The summary provides three features: key
timesteps, fused timesteps, and holistic visual representation of information
change.

  * •

Propose and demonstrate that the specific mutual information (SMI) measure
”Surprise” provides the most promising outcomes for the fusion technique.

  * •

Demonstrate the versatility and effectiveness of the proposed technique (acm)
through application to diverse datasets, including chemical simulations,
surveillance footage, and cell interactions in the immune system.

  * •

Explore the impact of the technique in optimizing data storage with minimal
data loss.

The paper’s structure is as follows: Section 2 reviews related works on data
summarization and information theory-based approaches. Section 3 outlines the
methodology for the framework Section 4 demonstrates the results of
implementing the technique across multi-domain datasets. Section 5 delves into
the implications of our findings and the research’s importance in diverse
application domains. Lastly, Section 6 concludes the paper by highlighting
possible future research avenues.

##  2 Related Works

Efficiently managing and analyzing large-scale time-varying datasets has
attracted considerable attention across multiple disciplines, leading to
various strategies to address the challenges posed by data growth. In this
work, we focus on identifying and storing informative key timesteps while
summarizing less informative ones by fusion. The summaries reduce storage
overhead while the key and fused timesteps preserve data dynamics. Various
data compression and reduction techniques have been explored in the context of
large-scale data analysis, aligning with the goals of the proposed research.
There are techniques that represent data reduction where reduced data is
stored in place of the raw data. Cinema [Ahrens et al. (2014)] is an in situ
data reduction and visualization approach that utilizes image-based
representations to capture essential features of time-varying datasets.
Techniques like lossless and lossy compression methods are commonly used for
data reduction. References [Vidhya et al. (2016); Cappello et al. (2019); Di
and Cappello (2016)] present various compression algorithms and their
application in handling large datasets. Other reduction technique includes
sub-sampling [Woodring et al. (2011); Wei et al. (2018)] and distribution-
based summaries [Dutta et al. (2016); Ye et al. (2016)]. The abovementioned
methods are widely recognized for their data reduction and compression
effectiveness. However, these approaches solely store the reduced timesteps.
In our work, we need to retain both the raw and reduced timesteps to
comprehensively capture information changes over time and preserve essential
data dynamics.

Identifying key time steps is essential in analyzing time-varying data and
advancing scientific visualization. There are numerous approaches [Zhou and
Chiang (2018); Myers et al. (2016); Tong et al. (2012); Hu et al. (2011)] that
have been proposed for identifying key time steps in large time-varying
datasets. These studies focus on only capturing key time steps without the
summarization capacity. Some studies focused on data reduction such as: in
[Akiba et al. (2006)] similar timesteps are grouped and one is selected, in
Wang et al. (2008) salient timesteps were selected by comparing dissimilarity
with previous timestep. These studies did not include any summarization
techniques which we incorporated in our proposed work along with selecting key
time steps.

The development of fusion techniques [Castanedo et al. (2013)] for large-scale
spatio-temporal datasets has been a popular field of interest for researchers
across various domains like remote sensing [Li et al. (2020); Chen et al.
(2015); Nguyen et al. (2014)], geoscience [Wu et al. (2021); Ma and Kang
(2020); Tang et al. (2020)], network architectures [Kashinath et al. (2021);
Shah et al. (2017)], and computer vision[Duzceker et al. (2021); Wang et al.
(2020); Xue et al. (2017)].In the realm of computer vision, various strategies
for data summarization using fusion techniques have been explored, including
Gaussian entropy fusion [Fu et al. (2010)] and probabilistic skimlets fusion
[Zhang et al. (2014)]. Additionally, deep learning architectures have been
applied for summarization tasks [Zhong et al. (2019)]. Notably, the proposed
work takes a unique approach by leveraging information-theoretic fusion for
data summarization. Unlike some existing techniques, our approach doesn’t need
training and can be readily applied to large-scale datasets. Moreover, it
requires minimal computational costs, rendering it effective for both in situ
and post hoc analysis. The feasibility of in situ application is showcased in
our preliminary research [Dutta et al. (2021)].

Information theory [Cover and Thomas (2006); Verdu (1998)] concepts such as
entropy and mutual information [Shannon (1948); Gray (2011)] have been
employed to measure the relationships between variables in data across
multiple computational domains [Moore et al. (2018); Pilkiewicz et al. (2020);
Sbert et al. (2022); Tasnim et al. (2018)]. Mutual information is extensively
applied in the field of feature selection, exploration, extraction, and
tracking [Biswas et al. (2013); Dutta and Shen (2016); Tasnim et al. (2022)].
Image registration is another popular application [Maes et al. (1997); Viola
and Wells III (1997); Hill et al. (2001); Cahill (2010)]. Mutual information,
as well as its decomposition measures like specific mutual information and
pointwise mutual information, have been widely used in multimodal data fusion
[Bramon et al. (2012)], graphics, and visualization [Wang et al. (2008); Dutta
et al. (2016, 2017b, 2017a); Isola et al. (2014); Bramon et al. (2013a); Akiba
et al. (2007); Chen et al. (2016)]. Other use cases for information theory
include view selection [Viola et al. (2006)], feature similarity [Bruckner and
Möller (2010)], transfer function and design [Ruiz et al. (2011); Bramon et
al. (2013b); Haidacher et al. (2008)], and data sampling [Dutta et al. (2019);
Wei et al. (2018)].

The works we discussed here collectively contribute to the proposed approach’s
design, development, and motivation. Our research draws inspiration from these
methodologies and integrates information-theoretic principles to develop an
innovative solution.

##  3 Information-Driven Framework for Feature-Based Temporal Data Summaries

Figure 1: Schematic diagram of the workflow. We have used the standard
computational flowchart [Sipser (2013)] symbols for representations:
input/output, process, decision, and arrows indicating relationships between
symbols.

###  3.1 Framework Workflow

In this section, we provide a comprehensive step-by-step explanation of our
technique’s mechanism. We intend to construct an efficient, generic, and fast
data summarization workflow with minimal customization to adapt to a variety
of application domains. Figure 1 illustrates the schematic of this proposed
workflow. To demonstrate each step of this workflow, we will refer to Figure
2, which serves as an illustrative application of our method using a synthetic
data set. In this application, we simulate a rolling ball moving 0.5 units
from left to right at each timestep until it exits the view area.

Our proposed method is designed for datasets with sequence of timesteps
containing various types of salient spatial features. In Figure 2, the
simulated rolling ball application consists of 19 timesteps (T0 - T18),
showcasing the ball’s positional change over time. Each timestep is
represented as 800 x 400 pixel 2D RGB images and pixel intensities ranging
from 0 to 255. At timestep T0, the frame is empty; the ball has not yet
entered the view area. It enters at T1 and the ball changes position until
T17. In, time step T18 the ball exits the view area. The method starts with
iterating over input timesteps. After the first timestep, for every subsequent
timesteps, the key regions are extracted using a segmentation method proposed
in [Kuruvilla et al. (2016)]. Criteria for extraction of such key regions can
be determined by the domain knowledge. In the rolling ball demonstration, the
key feature is the presence of the ball and its location. So we segment the
region containing the ball and create binary masked images shown in Figure 2
(masked row). These images contain only two distinct data values: 0 (no ball)
and 255 (ball).

After extracting the key region, we check if a certain property is present in
that timestep. We denote this property as trigger which is a change in the key
region. The change can be in terms of count, size, shape, connectivity, space
or association. In this rolling ball demonstration, the trigger is the
presence of the ball in the view area. When the ball enters and exits the
area, it is considered as a salient information. But the time it remains in
the area, the only novel information is its change of position. If the trigger
is present in the current timestep, then it is considered to be significant
and a key timestep. Hence our method saves it as it is. If the trigger is not
present in the current timestep, we proceed to the next timestep, do a similar
check, and continue the process until a trigger event is encountered. These
sequential time frames that did not have the trigger event are chosen to be
fused into a single timestep as the amount of novel information within such a
sequence is low. If the number of timesteps to be fused is one then we can
discard it as the previous time step has the necessary information. If the
number is greater than one then we perform pairwise information-guided fusion
on these timestep and convert them as one single timestep to be saved as a
temporal summary. Referencing the demonstration in Figure 2, T0 is saved. Then
starting from timesteps T1 - T17 there is no trigger present in the timesteps.
Therefore, those timesteps are fused as shown in figure 2(a) and (b). The
following section sheds light on how we use information theoretic approaches
for the fusion method.

Figure 2: Illustration of a simulated rolling ball with 19 timesteps (T0
-T18). At each timestep, the ball moves 0.5 units to the right. Timesteps are
2D RGB images with 800 x 400 dimensions and samples ranging from 0 to 255
(input row). The masked row presents binary images with values 0 (no ball) and
255 (ball) after ball segmentation. Timesteps T1 to T17 are fused using (a)
Surprise ($I_{1}$) guided fusion and (b) PMI guided fusion. a(i) shows the
$I_{1}$ fused data value field of the fused timesteps(0 white and 255 red).
a(ii) shows $I_{1}$ fused information value field. a(iii) displays $I_{1}$
fused timestep summary with numbered color labels for each timestep. The
numbers indicate spatial information changes over time. Surprise effectively
captures spatio-temporal properties, whereas alternative measure PMI does not
perform as well. (b) presents the scenario with the information field with PMI
values

###  3.2 Characterization of Samplewise Information for Fusion

In this work, our aim is to track and summarize the information change in
fused timesteps. Therefore, we need to have a quantification of information
content for each data point in the timesteps. We use the term ”sample” to
refer to these individual data points. Each timestep contains multiple samples
representing the values of the data. For example, in the case of images like
the rolling ball, these samples typically range from 0 to 255, while for other
types of data variables, they may be scalar values. The quantification of
information for these samples will help us identify important spatial features
for the timestep.

####  3.2.1 Mutual Inforamtion

In information theory, Mutual Information (MI) [Shannon (1948)] is a prominent
measure that estimates the total amount of shared information between two
random variables. Given two random variables $X$ and $Y$, MI $I(X,Y)$ is
formally defined as:

$\operatorname{I}(X;Y)=\sum_{y\in Y}\sum_{x\in
X}\operatorname{p}(x,y)\log\frac{\operatorname{p}(x,y)}{\operatorname{p(x)}\operatorname{p(y)}}$
(1)

where $\operatorname{p}(x)$ and $\operatorname{p}(y)$ are the probabilities of
occurrence of values $x$ for $X$ and $y$ for $Y$ respectively and
$\operatorname{p}(x,y)$ is the joint probability of occurrence of values $x$
and $y$ together. MI assesses the degree of association or disassociation
between two random variables and gives a single value. Since we aim to extract
feature-based data summaries, we need samplewise spatial and temporal
information characterization. Therefore, we leverage the decomposition of MI
which quantifies each data value’s contribution toward the association or
dissociation. The decomposition of MI is termed as Specific Mutual Information
or SMI [DeWeese and Meister (1999)]. SMI measures the information content of
the individual scalar values of one variable (reference) when another variable
(target) is observed. This information quantification helps with identifying
important spatio-temporal features in the data. There are multiple methods for
MI decomposition [DeWeese and Meister (1999); Butts (2003)]. For apprehending
the fusion criteria essential for summarizing the data, the properties of the
SMI measure, Surprise holds the most potential.

####  3.2.2 SMI Measure Surprise

The Surprise measure denoted as $I_{1}$ was first introduced by DeWeese and
Meister (1999). Surprise quantifies the information change of the target
variable after observing the individual scalar values of the reference
variable. The derivation of Surprise from MI is as follows.

By definition the conditional probability of $x$ given $y$ or $p(x|y)$ is,

$\operatorname{p}(x|y)=\frac{\operatorname{p}(x,y)}{\operatorname{p(y)}}\\\
or\\\ \operatorname{p}(x,y)=\operatorname{p}(x|y)\operatorname{p(y)}$ (2)

Replacing the joint probability in Equation 1, we get,

$\begin{split}\operatorname{I}(X;Y)&=\sum_{y\in Y}\operatorname{p(y)\sum_{x\in
X}\operatorname{p}(x|y)\log\frac{\operatorname{p}(x|y)}{\operatorname{p(x)}}}\\\
&=\sum_{y\in Y}\operatorname{p(y)}\operatorname{I_{1}(y;X)}\end{split}$ (3)

where,

$\operatorname{I_{1}}(y;X)=\sum_{x\in
X}\operatorname{p}(x|y)\log\frac{\operatorname{p}(x|y)}{\operatorname{p(x)}}$
(4)

Equation 4 represents the surprise measure of data value $y$ from $Y$ after
observing all the values of $X$. A high value for $I_{1}(y;X)$ means after
observing $y$, some previously low probable values of $x\in X$ have become
highly probable in the distribution. This likelihood increase is the element
of surprise and a salient finding for further analysis. Surprise is also the
only positive decomposition of MI since it is the Kullback-Leibler distance
between $\operatorname{p}(x|y)$ and $\operatorname{p(x)}$ [Kullback and
Leibler (1951)]. In the fusion process of the data summarization, the data
samples with high surprise values stand out and are identified as important
features.

###  3.3 Surprise ($I_{1}$) Guided Fusion Technique

When the low informative timesteps are chosen, the fusion initiates for
summarization. The fusion is done on pairwise timsteps. For every pair of data
samples from the timsteps, we store samples contributing to more information
or high $I_{1}$ values. This fusion strategy was introduced in [Bramon et al.
(2012)] for fusing different datasets to gain the most informative
combination. The condition to compute the fused value using ${I_{1}}$-fusion
is:

For every data sample pair with $(x,y)$, the fused value, f is,

$\operatorname{f}=\begin{cases}$x$,&\text{if
}\operatorname{I_{1}(x;Y)}>\operatorname{I_{1}(y;X)}\\\
$y$,&\text{otherwise}\end{cases}$ (5)

Here $x\in X$ and $y\in Y$ are individual data values from two data sets X and
Y respectively. Our fusion criteria is based on the idea of Equation 5,
however, instead of different datasets we are using two subsequent timesteps
from the same dataset. To fuse multiple timesteps, we begin by creating a
fused timestep using the first two timesteps. Then, we repeat the fusion
process by comparing the fused timestep with the next timestep, and continue
until all desired timesteps have been fused. Our strategy involves updating
the fused timestep during each iteration and selecting the spatial and
temporal values with the highest information content. By the end of the
process, the resulting fused timestep will represent a summary capturing their
most informative properties with direction. The fusion process is described in
detail in Algorithm 1. After each fusion process, the algorithm provides 3
fused fields as shown in 2(a). $I_{1}$ fused data value field contains the
values of the data samples with high surprise measure. $I_{1}$ fused
information value field contains the $I_{1}$ values for the same sample
positions. In the timestep summary fields, the same data sample are labeled
with their originating timestep numbers.

Applying the fusion process in Algorithm 1 on the timesteps T1 - T17 of the
simulated rolling ball, the surprise $(I_{1})$ fused data value field is
generated highlighting the path of the rolling ball with data values 255 as
shown in Figure 2 a(i). The regions without the ball are valued 0 or white.
Figure 2 a(ii) represents the fused information fields with $I_{1}$ values.
From the color bar’s gradient, we observe that the surprise values exhibit
limited variation, spanning approximately from 0 to 2. A minimal data value
range results in minimal surprise value variation. Figure 2 a(iii) presents
the timestep summary where the numbers of originating timesteps are labeled
for the salient samples. Here, we employed distinct colors to label the
timesteps, enabling clear visualization and differentiation of each timestep.
This color-coded representation effectively showcases the flow of information,
facilitating the tracking of information changes over time. Here, the
confidence threshold is employed to downplay the non-important regions. In
this particular case, the threshold value is set to 255. Any value below 255,
representing the absence of the ball, is assigned as timestep 0. In Figure 2
a(iii), these regions are depicted as white or transparent (steps 19 - 24 in
Algorithm 1). The method reduces the number of output timesteps from 19 to 3
in the simulated rolling ball case, achieving substantial data reduction with
minimal information loss. The fused timestep effectively visualizes the
information changes over time, serving as a concise summary of the original
data dynamics.

Algorithm 1 Fusion Process

Input:

  * •

data1: Array of data values from fused timestep. Initialized with the first
timestep.

  * •

data2: Array of data values from the subsequent timestep.

  * •

Ifield1: Array of $I_{1}$ values for $I_{1}(x;Y)$ $\forall x\in X$

  * •

Ifield2: Array of $I_{1}$ values for $I_{1}(y;X)$ $\forall y\in Y$

  * •

timestep_fuse: Array of timestep values. Starts with 0

  * •

time: Current timestep value

  * •

conf_th: Confidence threshold for the key regions.

Output:

  * •

fused_field_data: Array of the fused data values

  * •

fused_field_I1: Array of the fused $I_{1}$ values

  * •

timestep_fuse: Array of the fused timestep values.

1:procedure createFusionFields(data1, data2, Ifield1, Ifield2, timestep_fuse,
time, conf_th)

2: $\textit{fused\\_field\\_data}\leftarrow$ array of zeros with shape data1

3: $\textit{fused\\_field\\_I1}\leftarrow$ array of zeros with shape data1

4: for $i\leftarrow 0$ to $\textit{data1.shape}[0]-1$ do

5: for $j\leftarrow 0$ to $\textit{data1.shape}[1]-1$ do

6: if $\textit{Ifield1}[i][j]>\textit{Ifield2}[i][j]$ then

7: $\textit{fused\\_field\\_data}[i][j]\leftarrow\textit{data1}[i][j]$

8: $\textit{fused\\_field\\_I1}[i][j]\leftarrow\textit{Ifield1}[i][j]$

9: if $\textit{time}=1$ then

10: $\textit{timestep\\_fuse}[i][j]\leftarrow\textit{time}$

11: end if

12: else

13: $\textit{fused\\_field\\_data}[i][j]\leftarrow\textit{data2}[i][j]$

14: $\textit{fused\\_field\\_I1}[i][j]\leftarrow\textit{Ifield2}[i][j]$

15: $\textit{timestep\\_fuse}[i][j]\leftarrow\textit{time}+1$

16: end if

17: end for

18: end for

19: for $i\leftarrow 0$ to $\textit{data1.shape}[0]-1$ do

20: for $j\leftarrow 0$ to $\textit{data1.shape}[1]-1$ do

21: if $\textit{fused\\_field\\_data}[i][j]</>\textit{conf\\_th}$ then

22: $\textit{timestep\\_fuse}[i][j]\leftarrow 0$

23: end if

24: end for

25: end for

26: return
$\textit{fused\\_field\\_data},\textit{fused\\_field\\_I1},\textit{timestep\\_fuse}$

27:end procedure

###  3.4 Alternative Fusion Approaches

The measure Surprise effectively apprehends the spatio-temporal features for
the data summarization process. However, we also explore several other
potential information measures to devise alternative techniques for generating
data summaries. These information-theoretic measures include Pointwise Mutual
Information (PMI) [Church and Hanks (1990)], SMI measure (1) predictability
$(I_{2})$ [DeWeese and Meister (1999)] and (2) stimulus specific information
$(I_{3})$ [Butts (2003)]. These measures were investigated because they are
the decomposition of MI and they possess the ability to analyze the
contributions of individual data values in quantifying the information content
of spatio-temporal data. They offer insights into the properties of data
samples concerning information analysis.

####  3.4.1 PMI Guided Fusion

Pointwise Mutual Information (PMI) [Church and Hanks (1990)] quantifies the
degree of association (or disassociation) between individual data points given
two variables. If $X$ and $Y$ are two variables, then each data point can be
represented by the value pair $(x,y)$ where $x\in X$ and $y\in Y$. The
statistical association between these two points can be measured by their PMI
value, which is formally expressed as:

$\operatorname{PMI}(x,y)=\log\frac{\operatorname{p}(x,y)}{\operatorname{p(x)}\operatorname{p(y)}}$
(6)

where $\operatorname{p}(x)$ and $\operatorname{p}(y)$ are the probabilities of
occurrence of values $x\in X$ and $y\in Y$ respectively, and
$\operatorname{p}(x,y)$ is the joint probability of occurrence of values $x$
and $y$ together. PMI was introduced by Church and Hanks (1990). Comparing
Equations 1 and 6, we can infer that the expected PMI values over all
occurrences of variables $X$ and $Y$ correspond to the mutual information
value $I(X;Y)$. PMI is a symmetric measure that can generate values ranging
from negative to positive, depending on whether the distributions are
complementary or overlapping. If the information overlap is high
($\operatorname{p}(x,y)>\operatorname{p}(x)\operatorname{p}(y)$ ), then
$\operatorname{PMI}(x,y)>0$. The low association is indicated by
$\operatorname{p}(x,y)<\operatorname{p}(x)\operatorname{p}(y)$, resulting in
$\operatorname{PMI}(x,y)<0$. If $x$ and $y$ are statistically independent then
$\operatorname{p}(x,y)=\operatorname{p}(x)\operatorname{p}(y)$ and
$\operatorname{PMI}(x,y)=0$.

Now, given the PMI measure, we can devise a fusion strategy similar to $I_{1}$
where $I_{1}$ values are substituted with PMI values in Algorithm 1. The
resulting fused information field on the simulated rolling ball is shown in
Figure 2(b). We observe that PMI fails to capture the spatial characteristics
of the key regions and only captures the overlapped regions that are strongly
associative, indicated by high positive PMI values. The non-overlapping
regions are transparent showing low information overlap. The PMI values here
are 0 meaning data distribution is complementary and statistically
independent. As the spatial position of sample pairs plays a critical role in
PMI calculation, the fused field properties can exhibit significant variation
depending on the degree of overlap between the key features.

####  3.4.2 $I_{2}$ Guided Fusion

Predictability $(I_{2})$ is another decomposition of MI introduced by DeWeese
and Meister (1999). This SMI measure quantifies the change in the uncertainty
of one variable $(X)$ after observing the individual value of another variable
($y\in Y$) and is computed as:

$\operatorname{I_{2}}(y;X)=-\sum_{x\in X}p(x)\log p(x)+\sum_{x\in X}p(x|y)\log
p(x|y)$ (7)

where $y\in Y$ is the reference variable and $x\in X$ is the target variable.
$\operatorname{p}(x)$ is the probabilities of occurrence of values $x$ for $X$
and $\operatorname{p}(x|y)$ is the conditional probabilities values $x$ given
values $y$. Upon observing the variable y, the uncertainty of variable X can
either increase or decrease, leading to the possibility of both positive and
negative values for the $I_{2}$ measure. In some cases, the increased
uncertainty can reveal significant information about the relationship between
the variables. However, when we use the $I_{2}$ measure instead of the $I_{1}$
measure in the fusion process, the resulting fused information does not offer
a meaningful summary over time. This measure intermittently highlights notable
information changes, failing to capture the overall data dynamics. Our
hypothesis is that this measure is more suitable for feature extraction and
uncertainty quantification across various datasets rather than for analyzing
consecutive timesteps within a single dataset.

####  3.4.3 $I_{3}$ Guided Fusion

There is another measure that is derived from the decomposition of MI and was
introduced for measuring the association of stimulus and response in certain
neural systems [Butts (2003)]. It is termed as Stimulus Specific Information
(SSI), denoted by $I_{3}$ and computed as:

$\operatorname{I_{3}}(y;X)=-\sum_{x\in X}p(x|y)\operatorname{I_{2}}(x;Y)$ (8)

The response and stimulus are the two variables X and Y. This measure focuses
on establishing the association between the two variables to extract the
maximum amount of information from their relationship. It emphasizes that the
most informative data values from the first variable are related to the most
informative data values of the second variable [Butts (2003)]. In some cases,
$I1$ can be an alternate measure for $I3$, but the interpretation is different
based on the data [Butts (2003)]. They both capture different properties in
the data. When we use the $I_{3}$ measure instead of the $I_{1}$ measure on
the simulated rolling ball dataset, it captured very similar properties shown
in Figure 2(a). However, when applied to a different and more complex dataset,
it failed to capture the spatial properties of the features in the
summarization. This is explained in detail in Section 4.1.2 and shown in
Figure 3(b).

From the various methods mentioned, it’s clear that different approaches
emphasize distinct data properties. After thorough investigation of various
information-theoretic methods, we found that the Surprise measure aligns best
with our objectives of highlighting features and summarizing spatio-temporal
data while tracking information flow. In the next section, we apply this
method to more complex datasets to demonstrate its versatility across
different data scenarios.

##  4 Applications

In this section, we assess the versatility of our DSTS method across various
multidisciplinary applications, demonstrating its effectiveness in handling
complex and diverse datasets with streaming characteristics.

We selected diverse applications across multiple domains to demonstrate the
method’s robustness. The first dataset involves multiphase flow simulation
with scalar data values. The second is a surveillance video with extended
timesteps to highlight optimization capabilities. The third dataset comprises
complex immune cell images. This comprehensive evaluation showcases the
method’s adaptability and utility across various research disciplines.

Figure 3: Analysis and illustration of dynamic spatio-temporal summarization
for MFIX-Exa simulation application. The first row represents multiple
timesteps (309 - 315) of the dataset where the bubbles are highlighted as the
blue regions. Timesteps are raw images with 488 x 842 dimensions. Timesteps
310 to 314 are fused using (a) Surprise ($I_{1}$) guided fusion (i-iii) and
alternative (b) Stimulus Specific Information (SSI) or $I_{3}$ guided fusion.
a(i) shows the $I_{1}$ fused data value field of the timesteps, a(ii) shows
$I_{1}$ fused information value field reflecting $I_{1}$ values and (c) shows
$I_{1}$ 5 fused timestep summary. The color bar labels 5 different colors
summarizing the flow and direction of the bubbles in one timestep. The
alternative $I_{3}$ is unable to capture the path of bubbles as reflected in
(b) $I_{3}$ fused data field.

###  4.1 MFIX-Exa Flow Simulation

MFIX-Exa [Musser et al. (2022)] is a particle-based multiphase flow simulation
developed by the National Energy Technology Laboratory (NETL), USA. It focuses
on modeling multiphase flow by interacting with a vast number of particles
within the simulation domain. Using MFIX-Exa, particle-based data is generated
for studying the operational principles of chemical looping reactors (CLR).
MFIX-Exa can simulate various chemical mixing processes and one such
simulation involves the interaction of carbon particles with air in a
fluidized bed, resulting in the formation of carbon dioxide bubbles or void
regions. These bubbles hold significant importance for domain experts.MFIX-Exa
generates vast data with numerous particles and thousands of timesteps. This
extensive raw data presents challenges in transferring to storage due to
limited I/O bandwidth. Hence, experts seek solutions to extract bubble-
specific information while reducing storage needs, and preserving temporal
bubble dynamics. Our proposed dynamic spatio-temporal summarization method can
be used to provide this solution.

####  4.1.1 Data Context and Features

The raw data of the MFIX-Exa simulation contains the value of particle
properties (i.e., position, velocity, momentum, etc.) over multiple time
steps. As the data is streaming in nature, therefore it is a good fit for our
proposed workflow. In Biswas et al. (2021), there is a detailed description of
how this streaming simulation is converted into a 3D scaler dataset.

For this work, we use 2D slices extracted from this raw 3D scalar data. These
slices contain scaler values representing particle density. As a result, our
dataset consists of multiple timesteps (count 332), and each timestep
corresponds to 2D data samples with dimensions of 488 x 842. The sample values
fall within the range of [-$1.2\text{\times}{10}^{-6}$, 29.08]. As mentioned
earlier, the key regions are the bubbles. The bubbles are the low particle
density regions. In Dutta et al. (2022), the detection, segmentation, and
characterization of the bubbles are developed and studied in an extensive
manner. Using a density threshold, the bubbles are segmented. Then, the VTK
[Schroeder et al. (1998)] library is used to extract the connected components
in the dataset and use the threshold value to filter the low-density bubbles.
Over time, the bubbles undergo phases like creation, merge, split, and
dissolve into air [Biswas et al. (2021)]. Using data from this simulation,
domain experts want to comprehend the evolution of bubbles and explore the
relationships between various bubble characteristics such as their size,
shape, number of bubbles, etc [Dutta et al. (2022)]. Important events in this
simulation can be characterized by the creation of a bubble, the merging of
two or more bubbles, or the dissolving of a bubble. Note that for all of these
events, the total number of bubbles will change. Hence, if we identify time
steps when the bubble number has changed from a previous time step, then that
can be considered as a ”trigger”. In this work, we skip counting changes in
very small bubbles for simplicity. Key time steps are saved when the trigger
occurs, and intermediate steps between two triggers are fused using Algorithm
1 to summarize and store them. This process continues for all timesteps.

####  4.1.2 Results for Data Summarization

Figure 3 shows the analysis of DSTS method for MFIX-Exa simulation. Timesteps
309 - 315 are shown in the first row where bubbles are the blue regions. From
311 to 314 the number of bubbles remains unchanged. Therefore timesteps 310 -
314 are summarized using the fusion process. Figures 3 (a) represent results
from the Surprise $I_{1}$ guided fusion. The $I_{1}$ fused data value field
a(i) shows the scaler values [$-1.4\text{\times}{10}^{-6}$ to 23] for the
fused timesteps. Here the change in bubble movement is very prominent. Figure
3 a(ii) presents the $I_{1}$ values [$5.6\text{\times}{10}^{-1}$ to 11] for
the fused timesteps. The range of information value ($I_{1}$) is smaller,
making it less sensitive to bubble movement compared to particle density
values. However, it effectively highlights the main bubbles and their temporal
dynamics. The timestep summary field depicted in Figure 3 a(iii) represents
the timestep values from which the bubbles originate. Here 5 timesteps are
color-coded in distinguished colors to highlight the flow of the information
between interacting bubbles. The color map reflects the direction of the
bubbles from the start to the end position. This summary field not only
emphasizes key features but also visually indicates the spatial information
flow over time. The white/ transparent background (labeled as 0) filters all
the density values that are of low importance for this dataset.

We also implemented the alternative SSI ($I_{3}$) guided fusion technique as
mentioned in Section 3.4.3 for MFIX-Exa. Figure 3(b), represents the $I_{3}$
fused data value field. Here the bubbles are somewhat highlighted but the
change in the bubbles’ movement is hard to interpret. $I_{3}$ captures some
spatial features of the bubble regions but the edges are blurred. Thus, the
surprise fusion method again proves to be better than the SSI measure.

Based on the outcomes observed in both the simulated rolling ball and MFIX-Exa
simulation applications, it is evident that the timestep summary field stands
out as the most informative visual representation for spatio-temporal
summarization. This representation encapsulates both spatial and temporal
dynamics by highlighting the key regions and indicating the direction of the
information flow over time. Consequently, for our analysis of the next two
applications, we only show the surprise fused timestep summary fields.

###  4.2 Surveillance Data Analysis and Optimization

Data summarization techniques are vital for security camera footage analysis.
Security camera systems generate vast amounts of data, and manually reviewing
the continuous stream of video can be time-consuming and laborious. DSTS
offers an efficient solution by allowing the analysis and optimization of
camera footage. Suspicious activities can be detected by choosing an
appropriate ”trigger” and the generated summary fields provide experts with an
intuitive visual representation of the key timesteps. The most significant
impact of our method in this application is on archiving and storage
optimization.

To demonstrate DSTS in security camera footage analysis, we used the publicly
available SBM-RGBD Dataset [SBM-RGBD Dataset ; Camplani et al. (2017)]. This
dataset was originally created in conjunction with the Workshop on Background
Learning for Detection and Tracking from RGBD Videos [RGBD (2017)]. It was
created to evaluate background modeling methods for moving object detection.
The dataset comprises 33 RGBD videos, totaling 15033 timesteps, recorded
indoors using a Microsoft Kinect sensor [Camplani et al. (2017)]. The dataset
contains videos capturing moving objects at intervals, which aligns with the
data requirements for our application. Here, we used one of the videos titled
”Multipeople2” which shows four individuals walking in and out of the view
area, engaging in discussions, and writing on a whiteboard. Our method shows
an effective demonstration of summarizing the movement patterns of the
individuals.

Figure 4: Analysis and illustration of the dynamic spatio-temporal
summarization for the SBM-RGBD dataset. Here the emphasis is on representing
the extensive number of timestep summaries. (a) shows a summarization of 33
fused timesteps using a discrete color bar to illustrate information changes
over time. (b) showcases a summarization of 170 fused timesteps, employing a
continuous color bar to depict information changes over a longer period. The
color bars point out the spatial direction of the information flow by denoting
the prior and latter states.

####  4.2.1 Data Context and features

The videos have 640 x 480 resolution and the length is 1400 timesteps. The
dataset comes with image (PNG) files for each timestep of the input video. We
recreated the videos from the images which can be visualized here[BC lab ref].
The key regions in this dataset are the individuals and their movement. The
whiteboard and a chair are stationary in the background. To extract key
regions from the background, we have used a method called ViBe: a universal
background subtraction algorithm designed for video sequences [Barnich and Van
Droogenbroeck (2010)]. The algorithm aims to identify moving objects within
consecutive images or videos by efficiently distinguishing between the
foreground (moving objects) and the background (stationary elements). The
algorithm is computationally lightweight, making it suitable for real-time
applications.The adaptability of ViBe has made it a popular choice for a range
of computer vision tasks, including object tracking, surveillance, and motion
detection. Its straightforward pseudocode provided in Barnich and Van
Droogenbroeck (2010) facilitates easy implementation. By applying the ViBe
algorithm, the RGB images are converted into masked binary images with
segmented individuals in the foreground.

In scenarios with multiple individuals in the frame, we adopt a concept
similar to that used for counting bubbles in the MFIX-Exa (Section 4.1). We
apply the concept to count the number of people in each timestep by analyzing
the largest connected regions in the masked images.Since it is a binary image,
the data samples have two values: 0 (no individual) and 255 (individual). By
setting a size threshold, we can accurately count the number of individuals in
each timestep. Given that individuals move in and out of the view area in the
dataset, the number of individuals can be used as a trigger for our
application. Whenever a person enters or exits, that is a key timestep. The
consecutive timesteps between two triggers are then fused using Algorithm 1.
This fusion process effectively summarizes the movement patterns of
individuals within one timestep, eliminating the need to store every less
informative timestep.

The combination of the key and summary timesteps, provides an intuitive visual
representation of the significant moments in the video, making the analysis of
surveillance scenarios more informative.

####  4.2.2 Results for Data Summarization

Figure 4 showcases the summarization fields for two separate fused timesteps
in the dataset. Both fields show Surprise($I_{1}$) fused timestep summaries,
effectively representing spatial features and movement directions of
individuals over time.In this particular application, our main aim is to
showcase how effectively the method can fuse longer timesteps while ensuring
that both spatial and temporal dynamics remain just as noticeable as they are
in shorter timesteps.

In Figure 4(a), the summarization field depicts a fusion of 33 timesteps,
where two individuals are walking out of the view area. The leftmost person
exits first, initiating the trigger and stopping the fusion process. Each
timestep is represented by a discrete color, highlighting the changes in
movement over the 33 timesteps. Sample values below 255 are set to 0,
representing the white background, as the data value of individuals is 255.

Figure 4(b) shows a timestep summary for a longer period of 170 timesteps. The
summarization field captures an individual walking into the view area and
writing on the board, while another person has just stepped in, initiating the
trigger event. Continuous colors are used to display the movement changes due
to the length of the fused timesteps.

Remarkably, key and summarized timesteps in this dataset result in a
significant reduction from 1400 timesteps to only 49 timesteps. The highest
number of timesteps being fused is 262. This notable optimization ensures that
only relevant information is stored, reducing storage requirements without
compromising critical insights into the movement patterns of individuals. By
efficiently identifying and storing key moments, our method enhances the
analysis of crowded environments, enabling rapid detection of suspicious
activities, and thereby serving as a valuable tool for surveillance.

Figure 5: Illustration and analysis of the dynamic spatio-temporal
summarization method for cell interaction in Lymph Node. The results emphasize
T cell movement patterns and interaction with DCs in LN. (a) is a sample
timestep. Here red indicates the T cells and green indicates the DCs. (b) is
the surprise fused timestep summary fields of T cell for 5 timesteps. The
black cells in the field are overlaid DCs to highlight cell contact. (c) and
(d) are 2 enlarged positions from the (b) field to emphasize T cell movement.
(c) shows that a T cell is moving away from the DCs. (d) shows multiple T
cells moving toward the DCs. The corresponding timestep values are provided in
the color bars to highlight the direction.

###  4.3 Tracking Cell Interactions in Lymph Nodes

This dataset contains consecutive images of cellular interactions within the
lymph node (LN). LNs are essential for immune function, playing a crucial role
in initiating immune response and facilitating immune cell communication
[Mirsky et al. (2011)]. In the LN micro-environment, naive T cells are
activated by interactions with different cell types. Understanding these
interactions provides valuable insights into immune activation [Brewitz et al.
(2017)]. We reanalyze data from [Tasnim et al. (2018)] where information
theory-based approaches were used to identify and quantify the spatial
relationships between naïve T cells and three target cellular components:
dendritic cells (DCs), fibroblastic reticular cells (FRCs), and blood vessels
(BVs). These interactions are critical in influencing T cell movement and the
timing of encounters with antigen-presenting DCs. This process is a key step
in T cell activation and the initiation of the adaptive immune response.

The data for the study was gathered using two-photon microscopy (2PM) [Rubart
(2004)] to acquire 3D image stacks of lymph node tissue samples from mice. The
imaging process captured dynamic movies lasting 10 to 45 minutes, resulting in
a sequence of 3D images over time. This dataset is well-suited for the
application of our method.

In the following sections, we demonstrate that our approach offers both
quantitative analysis and visualization of cell movement and communication.
Additionally, this dataset showcases the extension of our technique’s
applicability to 3D images and movies, highlighting its versatility and
suitability for complex spatial interactions in biological systems.

####  4.3.1 Data Context and Features

Figure 5(a) shows 2 RGB 3D images with T cells dyed red and Dendtritic Cells
dyed green. Each voxel contains the color intensities of the dye in the red,
blue, and green channels. For every time step, we extract the red and green
channels into two separate images. We focus on the red channel in order
analyze T-cell motility.

Because these images contain a lot of noise, we implemented a pre-processing
step using the median filter [Gonzales and Wintz (1987)], to reduce noise
while preserving the edges of the cells for improved visualization. Since the
red channels specifically represent the T cells we did not need to segment the
data.

Our goal is to visualize how T cells move and interact in these movies. In
[Tasnim et al. (2018)] we used mutual information (MI) and normalized mutual
information (NMI) to quantify associations between cells. Here for each
timestep, we use the MI value between two cell types as a ”trigger” for fusing
time steps. If the MI value for a specific timestep exceeds a specified
threshold, we save that as key timestep. If the MI value falls below the
threshold, we find the next timestep in which the MI value exceeds the trigger
threshold and fuse the intermediate ones. This allows us to efficiently
capture and represent significant interactions while fusing less informative
time steps. This provides an informative summary of T cell interactions within
the dataset.

####  4.3.2 Results of Data Summarization

Figure 5 focuses on the T cell movement pattern and interaction with DCs in
the summarized timesteps. Figure 5(a) is a sample timestep of T:DC dataset
with 512 x 512 x 22 dimensions. This dataset has total 51 timesteps. Figure
5(b) displays the Surprise ($I_{1}$) fused timestep summary field,
representing five fused timesteps from this dataset. The black cells in the
summarization represent the Dendritic cells’ value field overlaid on the fused
summary field, visually illustrating the physical interactions between T cells
and antigen-presenting DCs. Given that there are multiple interactions
captured in each timestep, we highlight two specific interactions by enlarging
the locations in Figures 5(c) and (d). In Figure 5(c), we observe a T cell
moving away from the DCs. The color bar on the right side indicates the first
(blue) and last (red) timesteps in the summary field, clearly indicating the
movement direction. In Figure 5(d), we see multiple T cells moving toward the
Dendritic cells, with cells making explicit contact with the DCs. The
visualization provided by the fused timestep summary field allows for a
comprehensive understanding of the dynamic interactions between T cells and
DCs, providing valuable insights into the temporal dynamics of immune cell
communication.

Our method, successfully visualizes physical contact between cells and track
movement over time. Other studies [Dunn et al. (2011); Mrass et al. (2017);
Mempel et al. (2004)] using similar datasets have presented the statistical
quantification of association. This addition of the visualization feature has
the potential to unveil more insights for experts to analyze these
associations for further investigation. This is a notable contribution to the
study of T cell motility, a crucial aspect for understanding immune response
dynamics.

##  5 Discussion

The proposed DSTS technique has demonstrated its effectiveness and versatility
across various applications. Starting with a simple simulation of a rolling
ball to analyzing complex cellular interactions within lymph nodes, the method
proved its robustness for every application scenario. The combination of the
key and fused timestep resulting from the method provides an impactful data
summarization. The intuitive visual representation is a plus in showcasing an
ideal blend of data optimization while preserving vital information change
over time. We investigated multiple information theory measures and
established that SMI measure surprise has the ability to capture the spatio-
temporal features effectively.

We selected applications from multiple domains to shed light on different
aspects of the DSTS method. This technique offers a practical solution to
downsample and analyze the vast particle data (bubbles) generated by the MFiX-
Exa simulation. This application analysis focuses on the fact that the method
is able to handle scaler value data as well as image data that incorporates
the rest of the applications.

The RGBD tracking dataset was introduced to show the method’s ability to
summarize and highlight important movement patterns of individuals for longer
periods of time. The results from this dataset reflect that longer fused
timesteps are equally apprehensible as the shorter timesteps. This has
promising implications for surveillance and security applications, enabling
more efficient and informative analysis of large video datasets.

We increased the complexity of the dataset gradually to prove the method’s
scalability. The T cell and Dendritic cell interactions in lymph nodes is a
complex dataset. T cells which are the key regions were ample in number and
the interactions with DCs were sporadic in nature. Given the fact that there
were multiple interactions, our method was able to track every one of them.
The summarization highlighted immune cell communication by providing a clear
and comprehensive visualization of the T cell movement. The cell interaction
visualization opens up new possibilities for immunological research.

All the applications in this work present post hoc data analysis. The datasets
were already available when the method was applied. Since the method is not
computationally expensive it can be easily combined as a step for analyzing
data when they are generated; which is termed as in situ analysis. Through the
integration of this method in any situ analysis, the resulting data will be
optimized in real-time ensuring storage reduction with minimal data loss.

##  6 Conclusion and Future Work

While the method’s performance on these datasets is very promising, challenges
may arise in selecting appropriate triggers and threshold values, especially
in complex datasets with multiple key features, interactions, and noise.
However, the flexibility of the technique allows for the adjustment of
parameters to tailor the summarization process to different applications.
Additionally, future research could explore combining different information-
theoretic measures to further enhance the summarization capabilities and
address specific challenges in various datasets.

In conclusion, the proposed dynamic spatio-temporal summarization technique
offers a powerful and efficient tool for visualizing and analyzing complex
time-varying datasets across different domains. The method has demonstrated
its adaptability and potential to provide valuable insights and understanding
of temporal data dynamics. The approach holds promise for advancing research
in various fields which may lead to novel discoveries and applications,
ultimately contributing to a deeper understanding of complex data systems.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Data Availability

Data will be available upon request to the corresponding author.

## Acknowledgements

We thank Judy Cannon for providing the biological data; Matthew Fricke for
help on the cell interaction and analysis; Moses Biocomputational Lab at UNM
for helpful discussion; IIT Kanpur..

## References

  * Ahrens et al. (2014) Ahrens, J., Jourdain, S., O’Leary, P., Patchett, J., Rogers, D.H., Petersen, M., 2014\.  An Image-Based Approach to Extreme Scale in Situ Visualization and Analysis.  International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015-Janua, 424–434.  doi:10.1109/SC.2014.40.
  * Akiba et al. (2006) Akiba, H., Fout, N., Ma, K.L., 2006.  Simultaneous classification of time-varying volume data based on the time histogram., in: EuroVis, pp. 1–8.
  * Akiba et al. (2007) Akiba, H., Ma, K.L., Chen, J.H., Hawkes, E.R., 2007.  Visualizing multivariate volume data from turbulent combustion simulations.  Computing in Science and Engineering 9, 76–83.  doi:10.1109/MCSE.2007.42.
  * Barnich and Van Droogenbroeck (2010) Barnich, O., Van Droogenbroeck, M., 2010\.  Vibe: A universal background subtraction algorithm for video sequences.  IEEE Transactions on Image processing 20, 1709–1724.
  * Biswas et al. (2021) Biswas, A., Ahrens, J.P., Dutta, S., Musser, J.M., Almgren, A.S., Turton, T.L., 2021\.  Feature analysis, tracking, and data reduction: An application to multiphase reactor simulation mfix-exa for in-situ use case.  Computing in Science and Engineering 23, 75–82.  doi:10.1109/MCSE.2020.3016927.
  * Biswas et al. (2013) Biswas, A., Dutta, S., Shen, H.W., Woodring, J., 2013\.  An information-aware framework for exploring multivariate data sets.  IEEE Transactions on Visualization and Computer Graphics 19, 2683–2692.  doi:10.1109/TVCG.2013.133.
  * Bramon et al. (2012) Bramon, R., Boada, I., Bardera, A., Rodríguez, J., Feixas, M., Puig, J., Sbert, M., 2012.  Multimodal data fusion based on mutual information.  IEEE Transactions on Visualization and Computer Graphics 18, 1574–1587.  doi:10.1109/TVCG.2011.280.
  * Bramon et al. (2013a) Bramon, R., Ruiz, M., Bardera, A., Boada, I., Feixas, M., Sbert, M., 2013a.  An information-theoretic observation channel for volume visualization.  Computer Graphics Forum 32, 411–420.  doi:10.1111/cgf.12128.
  * Bramon et al. (2013b) Bramon, R., Ruiz, M., Bardera, A., Boada, I., Feixas, M., Sbert, M., 2013b.  Information theory-based automatic multimodal transfer function design.  IEEE Journal of Biomedical and Health Informatics 17, 870–880.  doi:10.1109/JBHI.2013.2263227.
  * Brewitz et al. (2017) Brewitz, A., Eickhoff, S., Dähling, S., Quast, T., Bedoui, S., Kroczek, R.A., Kurts, C., Garbi, N., Barchet, W., Iannacone, M., et al., 2017\.  Cd8+ t cells orchestrate pdc-xcr1+ dendritic cell spatial and functional cooperativity to optimize priming.  Immunity 46, 205–219.
  * Bruckner and Möller (2010) Bruckner, S., Möller, T., 2010\.  Isosurface similarity maps, in: Computer Graphics Forum, Wiley Online Library. pp. 773–782.
  * Butts (2003) Butts, D.A., 2003.  How much information is associated with a particular stimulus?  Network: Computation in Neural Systems 14, 177–187.  doi:10.1088/0954-898X_14_2_301.
  * Cahill (2010) Cahill, N.D., 2010.  Normalized measures of mutual information with general definitions of entropy for multimodal image registration.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 6204 LNCS, 258–268.  doi:10.1007/978-3-642-14366-3_23.
  * Camplani et al. (2017) Camplani, M., Maddalena, L., Moyá Alcover, G., Petrosino, A., Salgado, L., 2017.  A Benchmarking Framework for Background Subtraction in RGBD Videos.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 10590 LNCS, 219–229.  doi:10.1007/978-3-319-70742-6_21.
  * Cappello et al. (2019) Cappello, F., Di, S., Li, S., Liang, X., Gok, A.M., Tao, D., Yoon, C.H., Wu, X.C., Alexeev, Y., Chong, F.T., 2019\.  Use cases of lossy compression for floating-point data in scientific data sets.  The International Journal of High Performance Computing Applications 33, 1201–1220.
  * Castanedo et al. (2013) Castanedo, F., et al., 2013.  A review of data fusion techniques.  The scientific world journal 2013\.
  * Chen et al. (2015) Chen, B., Huang, B., Xu, B., 2015.  Comparison of spatiotemporal fusion models: A review.  Remote Sensing 7, 1798–1835.
  * Chen et al. (2016) Chen, M., Feixas, M., Viola, I., Bardera, A., Shen, H.W., Sbert, M., 2016\.  Information theory tools for visualization.  CRC Press.
  * Childs (2015) Childs, H., 2015.  Data exploration at the exascale.  Supercomputing frontiers and innovations 2, 5–13.
  * Church and Hanks (1990) Church, K., Hanks, P., 1990\.  Word association norms, mutual information, and lexicography.  Computational linguistics 16, 22–29.
  * Cover and Thomas (2006) Cover, T.M., Thomas, J.A., 2006\.  Elements of Information Theory 2nd Edition (Wiley Series in Telecommunications and Signal Processing).  Wiley-Interscience.
  * DeWeese and Meister (1999) DeWeese, M.R., Meister, M., 1999\.  How to measure the information gained from one symbol.  Network: Computation in Neural Systems doi:10.1088/0954-898X_10_4_303.
  * Di and Cappello (2016) Di, S., Cappello, F., 2016.  Fast error-bounded lossy hpc data compression with sz, in: 2016 ieee international parallel and distributed processing symposium (ipdps), IEEE. pp. 730–739.
  * Dunn et al. (2011) Dunn, K.W., Kamocka, M.M., McDonald, J.H., 2011.  A practical guide to evaluating colocalization in biological microscopy.  American Journal of Physiology-Cell Physiology 300, C723–C742.
  * Dutta et al. (2019) Dutta, S., Biswas, A., Ahrens, J., 2019.  Multivariate Pointwise Information-Driven Data Sampling and Visualization.  Entropy 21, 699\.  doi:10.3390/e21070699.
  * Dutta et al. (2016) Dutta, S., Chen, C.M., Heinlein, G., Shen, H.W., Chen, J.P., 2016.  In situ distribution guided analysis and visualization of transonic jet engine simulations.  IEEE transactions on visualization and computer graphics 23, 811–820.
  * Dutta et al. (2017a) Dutta, S., Liu, X., Biswas, A., Shen, H.W., Chen, J.P., 2017a.  Pointwise information guided visual analysis of time-varying multi-fields.  SIGGRAPH Asia 2017 Symposium on Visualization, SA 2017 doi:10.1145/3139295.3139298.
  * Dutta and Shen (2016) Dutta, S., Shen, H.W., 2016\.  Distribution Driven Extraction and Tracking of Features for Time-varying Data Analysis.  IEEE Transactions on Visualization and Computer Graphics 22, 837–846.  doi:10.1109/TVCG.2015.2467436.
  * Dutta et al. (2021) Dutta, S., Tasnim, H., Turton, T.L., Ahrens, J., 2021\.  In Situ Adaptive Spatio-Temporal Data Summarization.  Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021 , 315–321doi:10.1109/BigData52589.2021.9671581.
  * Dutta et al. (2022) Dutta, S., Turton, T., Rogers, D., Musser, J.M., Ahrens, J., Almgren, A.S., 2022\.  In situ feature analysis for large-scale multiphase flow simulations.  Journal of Computational Science 63, 101773.  URL: https://doi.org/10.1016/j.jocs.2022.101773, doi:10.1016/j.jocs.2022.101773.
  * Dutta et al. (2017b) Dutta, S., Woodring, J., Shen, H.W., Chen, J.P., Ahrens, J., 2017b.  Homogeneity guided probabilistic data summaries for analysis and visualization of large-scale data sets.  IEEE Pacific Visualization Symposium , 111–120doi:10.1109/PACIFICVIS.2017.8031585.
  * Duzceker et al. (2021) Duzceker, A., Galliani, S., Vogel, C., Speciale, P., Dusmanu, M., Pollefeys, M., 2021\.  Deepvideomvs: Multi-view stereo on video with recurrent spatio-temporal fusion, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15324–15333.
  * Fu et al. (2010) Fu, Y., Guo, Y., Zhu, Y., Liu, F., Song, C., Zhou, Z.H., 2010.  Multi-view video summarization.  IEEE Transactions on Multimedia 12, 717–729.
  * Gonzales and Wintz (1987) Gonzales, R.C., Wintz, P., 1987\.  Digital image processing.  Addison-Wesley Longman Publishing Co., Inc.
  * Gray (2011) Gray, R.M., 2011.  Entropy and information theory.  Springer Science & Business Media.
  * Haidacher et al. (2008) Haidacher, M., Bruckner, S., Kanitsar, A., Gröller, M.E., 2008\.  Information-based transfer functions for multimodal visualization.  EG VCBM 2008 - Eurographics Workshop on Visual Computing for Biomedicine , 101–108.
  * Hill et al. (2001) Hill, D.L., Batchelor, P.G., Holden, M., Hawkes, D.J., 2001\.  Medical image registration.  Physics in medicine & biology 46, R1.
  * Hu et al. (2011) Hu, W., Xie, N., Li, L., Zeng, X., Maybank, S., 2011\.  A survey on visual content-based video indexing and retrieval.  IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 41, 797–819.
  * Isola et al. (2014) Isola, P., Zoran, D., Krishnan, D., Adelson, E.H., 2014\.  Crisp boundary detection using pointwise mutual information.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 8691 LNCS, 799–814.  doi:10.1007/978-3-319-10578-9_52.
  * Kashinath et al. (2021) Kashinath, S.A., Mostafa, S.A., Mustapha, A., Mahdin, H., Lim, D., Mahmoud, M.A., Mohammed, M.A., Al-Rimy, B.A.S., Fudzee, M.F.M., Yang, T.J., 2021\.  Review of data fusion methods for real-time and multi-sensor traffic flow analysis.  IEEE Access 9, 51258–51276.
  * Kullback and Leibler (1951) Kullback, S., Leibler, R.A., 1951\.  On information and sufficiency.  The annals of mathematical statistics 22, 79–86.
  * Kuruvilla et al. (2016) Kuruvilla, J., Sukumaran, D., Sankar, A., Joy, S.P., 2016\.  A review on image processing and image segmentation, in: 2016 International Conference on Data Mining and Advanced Computing (SAPIENCE), pp. 198–203.  doi:10.1109/SAPIENCE.2016.7684170.
  * Li et al. (2020) Li, J., Li, Y., He, L., Chen, J., Plaza, A., 2020\.  Spatio-temporal fusion for remote sensing data: An overview and new benchmark.  Science China Information Sciences 63, 1–17.
  * Ma and Kang (2020) Ma, P., Kang, E.L., 2020.  Spatio-temporal data fusion for massive sea surface temperature data from modis and amsr-e instruments.  Environmetrics 31, e2594.
  * Maes et al. (1997) Maes, F., Collignon, A., Vandermeulen, D., Marchal, G., Suetens, P., 1997.  Multimodality image registration by maximization of mutual information.  IEEE transactions on Medical Imaging 16, 187–198.
  * Mempel et al. (2004) Mempel, T.R., Henrickson, S.E., Von Andrian, U.H., 2004.  T-cell priming by dendritic cells in lymph nodes occurs in three distinct phases.  Nature 427, 154–159.
  * Mirsky et al. (2011) Mirsky, H.P., Miller, M.J., Linderman, J.J., Kirschner, D.E., 2011\.  Systems biology approaches for understanding cellular mechanisms of immunity in lymph nodes during infection.  Journal of theoretical biology 287, 160–170.
  * Moore et al. (2018) Moore, D.G., Valentini, G., Walker, S.I., Levin, M., 2018\.  Inform: Efficient information-theoretic analysis of collective behaviors.  Frontiers Robotics AI 5, 1–14.  doi:10.3389/frobt.2018.00060.
  * Mrass et al. (2017) Mrass, P., Oruganti, S.R., Fricke, G.M., Tafoya, J., Byrum, J.R., Yang, L., Hamilton, S.L., Miller, M.J., Moses, M.E., Cannon, J.L., 2017\.  Rock regulates the intermittent mode of interstitial t cell migration in inflamed lungs.  Nature communications 8, 1010\.
  * Musser et al. (2022) Musser, J., Almgren, A.S., Fullmer, W.D., Antepara, O., Bell, J.B., Blaschke, J., Gott, K., Myers, A., Porcu, R., Rangarajan, D., Rosso, M., Zhang, W., Syamlal, M., 2022.  MFIX-Exa: A path toward exascale CFD-DEM simulations.  International Journal of High Performance Computing Applications 36, 40–58.  doi:10.1177/10943420211009293.
  * Myers et al. (2016) Myers, K., Lawrence, E., Fugate, M., Bowen, C.M., Ticknor, L., Woodring, J., Wendelberger, J., Ahrens, J., 2016\.  Partitioning a large simulation as it runs.  Technometrics 58, 329–340.
  * Nguyen et al. (2014) Nguyen, H., Katzfuss, M., Cressie, N., Braverman, A., 2014\.  Spatio-temporal data fusion for very large remote sensing datasets.  Technometrics 56, 174–185.
  * Pilkiewicz et al. (2020) Pilkiewicz, K.R., Lemasson, B.H., Rowland, M.A., Hein, A., Sun, J., Berdahl, A., Mayo, M.L., Moehlis, J., Porfiri, M., Fernández-Juricic, E., Garnier, S., Bollt, E.M., Carlson, J.M., Tarampi, M.R., MacUga, K.L., Rossi, L., Shen, C.C., 2020.  Decoding collective communications using information theory tools.  Journal of the Royal Society Interface 17\.  doi:10.1098/rsif.2019.0563.
  * Reed and Dongarra (2015) Reed, D.A., Dongarra, J., 2015\.  Exascale computing and big data.  Communications of the ACM 58, 56–68.
  * RGBD (2017) RGBD2017, .  Background learning for detection and tracking from rgbd videos.  https://rgbd2017.na.icar.cnr.it/.  Accessed: 2023-09-22.
  * Rubart (2004) Rubart, M., 2004.  Two-photon microscopy of cells and tissue.  Circulation research 95, 1154–1166.
  * Ruiz et al. (2011) Ruiz, M., Bardera, A., Boada, I., Viola, I., Feixas, M., Sbert, M., 2011\.  Automatic transfer functions based on informational divergence.  IEEE Transactions on Visualization and Computer Graphics 17, 1932–1941.  doi:10.1109/TVCG.2011.173.
  * Sbert et al. (2022) Sbert, M., Feixas, M., Rigau, J., Chover, M., Viola, I., 2022.  Information theory tools for computer graphics.  Springer Nature.
  * (59) SBM-RGBD Dataset, .  Sbm-rgbd dataset.  https://rgbd2017.na.icar.cnr.it/SBM-RGBDdataset.html.  Accessed: 2023-09-22.
  * Schroeder et al. (1998) Schroeder, W., Martin, K.M., Lorensen, W.E., 1998.  The visualization toolkit an object-oriented approach to 3D graphics.  Prentice-Hall, Inc.
  * Shah et al. (2017) Shah, Z., Anwar, A., Mahmood, A.N., Tari, Z., Zomaya, A.Y., 2017.  A spatiotemporal data summarization approach for real-time operation of smart grid.  IEEE Transactions on Big Data 6, 624–637.
  * Shannon (1948) Shannon, C.E., 1948.  A Mathematical Theory of Communication.  Bell System Technical Journal 27, 379–423.  doi:10.1002/j.1538-7305.1948.tb01338.x.
  * Sipser (2013) Sipser, M., 2013.  Introduction to the Theory of Computation.  Third ed., Course Technology, Boston, MA.
  * Tang et al. (2020) Tang, Y., Wang, Q., Zhang, K., Atkinson, P.M., 2020.  Quantifying the effect of registration error on spatio-temporal fusion.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 13, 487–503.
  * Tasnim et al. (2022) Tasnim, H., Dutta, S., Turton, T.L., Rogers, D.H., Moses, M.E., 2022.  Information-Theoretic Exploration of Multivariate Time-Varying Image Databases.  Computing in Science and Engineering 24, 61–70.  doi:10.1109/MCSE.2022.3188291.
  * Tasnim et al. (2018) Tasnim, H., Fricke, G., Byrum, J., Sotiris, J., Cannon, J., Moses, M., 2018\.  Quantitative measurement of naïve T cell association with dendritic cells, FRCs, and blood vessels in lymph nodes.  Frontiers in Immunology 9.  doi:10.3389/fimmu.2018.01571.
  * Tong et al. (2012) Tong, X., Lee, T.Y., Shen, H.W., 2012.  Salient time steps selection from large scale time-varying data sets with dynamic time warping, in: IEEE symposium on large data analysis and visualization (LDAV), IEEE. pp. 49–56.
  * Verdu (1998) Verdu, S., 1998.  Fifty years of shannon theory.  IEEE Transactions on Information Theory 44, 2057–2078.  doi:10.1109/18.720531.
  * Vidhya et al. (2016) Vidhya, K., Karthikeyan, G., Divakar, P., Ezhumalai, S., 2016\.  A review of lossless and lossy image compression techniques.  Int. Res. J. Eng. Technol.(IRJET) 3, 616–7.
  * Viola et al. (2006) Viola, I., Feixas, M., Sbert, M., Groller, M.E., 2006\.  Importance-driven focus of attention.  IEEE transactions on visualization and computer graphics 12, 933–940.
  * Viola and Wells III (1997) Viola, P., Wells III, W.M., 1997\.  Alignment by maximization of mutual information.  International journal of computer vision 24, 137–154.
  * Wang et al. (2008) Wang, C., Yu, H., Ma, K.L., 2008.  Importance-driven time-varying data visualization.  IEEE Transactions on Visualization and Computer Graphics 14, 1547–1554.
  * Wang et al. (2020) Wang, Q., Tang, Y., Tong, X., Atkinson, P.M., 2020.  Virtual image pair-based spatio-temporal fusion.  Remote Sensing of Environment 249, 112009.
  * Wei et al. (2018) Wei, T.H., Dutta, S., Shen, H.W., 2018.  Information guided data sampling and recovery using bitmap indexing, in: 2018 IEEE Pacific Visualization Symposium (PacificVis), IEEE. pp. 56–65.
  * Woodring et al. (2011) Woodring, J., Ahrens, J., Figg, J., Wendelberger, J., Habib, S., Heitmann, K., 2011\.  In-situ sampling of a large-scale particle simulation for interactive visualization and analysis, in: Computer Graphics Forum, Wiley Online Library. pp. 1151–1160.
  * Wu et al. (2021) Wu, P., Yin, Z., Zeng, C., Duan, S.B., Göttsche, F.M., Ma, X., Li, X., Yang, H., Shen, H., 2021.  Spatially continuous and high-resolution land surface temperature product generation: A review of reconstruction and spatiotemporal fusion techniques.  IEEE Geoscience and Remote Sensing Magazine 9, 112–137.
  * Xue et al. (2017) Xue, J., Leung, Y., Fung, T., 2017.  A bayesian data fusion approach to spatio-temporal fusion of remotely sensed images.  Remote Sensing 9, 1310\.
  * Ye et al. (2016) Ye, Y.C., Neuroth, T., Sauer, F., Ma, K.L., Borghesi, G., Konduri, A., Kolla, H., Chen, J., 2016\.  In situ generated probability distribution functions for interactive post hoc visualization and analysis, in: 2016 IEEE 6th Symposium on Large Data Analysis and Visualization (LDAV), IEEE. pp. 65–74.
  * Zhang et al. (2014) Zhang, L., Gao, Y., Hong, R., Hu, Y., Ji, R., Dai, Q., 2014.  Probabilistic skimlets fusion for summarizing multiple consumer landmark videos.  IEEE Transactions on Multimedia 17, 40–49.
  * Zhong et al. (2019) Zhong, S.h., Wu, J., Jiang, J., 2019.  Video summarization via spatio-temporal deep architecture.  Neurocomputing 332, 224–235.
  * Zhou and Chiang (2018) Zhou, B., Chiang, Y.J., 2018\.  Key time steps selection for large-scale time-varying volume datasets using an information-theoretic storyboard, in: Computer Graphics Forum, Wiley Online Library. pp. 37–49.


# Asimov: A framework for coordinating parameter estimation workflows

Daniel Williams  John Veitch  Maria Luisa Chiofalo  Patricia Schmidt  Richard
P. Udall  Avi Vajpeji  Charlie Hoy

††margin:  DOI: https://doi.org/https://doi.org Software •
https://github.com/openjournals/joss-reviews/issues/4170https://doi.orgReview
• https://git.ligo.org/asimov/asimovhttps://doi.orgRepository •
https://doi.org/10.5281/zenodo.4024432https://doi.orgArchive Submitted:
Published: License
Authors of papers retain copyright and release the work under a Creative
Commons Attribution 4.0 International License
(https://creativecommons.org/licenses/by/4.0/https://doi.orgCC BY 4.0).

## Summary

Since the first detection in 2015 of gravitational waves from compact binary
coalescence (B. Abbott and others 2016a), improvements to the Advanced LIGO
and Advanced Virgo detectors have expanded our view into the universe for
these signals. Searches of the of the latest observing run (O3) have increased
the number of detected signals to 90, at a rate of approximately 1 per week
(The LIGO Scientific Collaboration, the Virgo Collaboration, Abbott, et al.
2021; The LIGO Scientific Collaboration, the Virgo Collaboration, the KAGRA
Collaboration, et al. 2021). Future observing runs are expected to increase
this even further(Abbott and others 2020). Bayesian analysis of the signals
can reveal the properties of the coalescing black holes and neutron stars by
comparing predicted waveforms to the observed data (B. Abbott and others
2016b). The proliferating number of detected signals, the increasing number of
methods that have been deployed (Veitch and others 2015; Ashton and others
2019; Lange, O’Shaughnessy, and Rizzo 2018), and the variety of waveform
models (Ossokine and others 2020; Khan et al. 2020; Pratten and others 2021)
create an ever-expanding number of analyses that can be considered.

Asimov is a python package which is designed to simplify and standardise the
process of configuring these analyses for a large number of events. It has
already been used in developing analyses in three major gravitational wave
catalog publications (Abbott and others 2021; The LIGO Scientific
Collaboration, the Virgo Collaboration, Abbott, et al. 2021; The LIGO
Scientific Collaboration, the Virgo Collaboration, the KAGRA Collaboration, et
al. 2021). The source code of Asimov is archived to Zenodo (Williams et al.
2021).

## Statement of Need

While these developments are positive, they also bring considerable
challenges. The first of these lies with the high rate at which gravitational
waves can now be detected; thanks to the improved sensitivity of the detectors
they observe a much larger volume of space, and the increasing size of the
detector network has also increased the total time during which observations
occur. The second comes from developments in the analysis techniques and
related software. Development of these techniques has accelerated in a short
period of time, and the landscape of analysis software has become diverse. It
is desirable to be able to use these techniques with ease, but thanks to the
highly distributed development process which has produced them, they often
have highly heterogeneous interfaces.

We developed asimov as a solution to both of these problems, as it is capable
both of organising and tracking a large number of on-going analyses, but also
of performing setup and post-processing of several different analysis
pipelines, providing a single uniform interface. The software has been
designed to be easily extensible, making integration with new pipelines
straight-forward.

In addition, ensuring that the large number of analyses are completed
successfully, and their results collated efficiently proved a formidable
challenge when relying on “by-hand” approaches. The LIGO Scientific
Collaboration operate a number of high-throughput computing facilities (the
LIGO Data Grid [LDG]) which are themselves controlled by the htcondor
scheduling system. asimov monitors the progress of jobs within the htcondor
ecosystem, resubmits jobs to the cluster which fail due to transient problems,
such as file I/O errors in computing nodes, and detects the completion of
analysis jobs. Upon completion of a job the results are post-processed using
the PESummary python package (Hoy and Raymond 2021), and humans can be alerted
by a message posted by asimov to a Mattermost or Slack channel. Interaction
with htcondor will also allow jobs to be submitted to the Open Science Grid in
the future.

Prior to the development of asimov analyses of gravitational wave data had
been configured and run manually, or had relied on collections of shell
scripts. Asimov therefore constitutes an new approach, designed to be both
more maintainable, and to improve the reproduciblity of results generated by
analysis pipelines.

## Implementation

In order to produce a uniform interface to all of its supported pipelines,
asimov implements a YAML-formatted configuration file, which is referred to as
its “production ledger”. This file is used to specify the details of each
event to be analysed, details about the data sources, and details of each
pipeline which should be applied to the specified data. This allows identical
settings to be used with multiple different pipelines, with a minimum of
configuration, reducing the possibility of transcription errors between
setups. In the current implementation of Asimov the production ledger is
stored using an issue tracker on a custom Gitlab instance, with each issue
representing a different event. This approach is, however, neither flexible
nor scalable, and future development will use an alternative means of storing
the ledger.

Asimov simplifies the process of gathering and collating the various settings
and data-products required to configure an analysis. These include data
quality information: data from gravitational wave detectors can be affected by
non-stationary noise or “glitches” which must be either be removed before
analysis, or the analysis must be configured to mitigate their effect on final
results. These data are provided to Asimov in YAML format from the appropriate
team, and used to make appropriate selections in the analysis.

The analysis of gravitational wave data is generally performed within a
Bayesian framework, which requires prior probability distributions being
chosen before the analysis. Ideally these distributions would be chosen such
that a very broad range of parameter values is explored and sampled, however
this is computationally impractical, and to improve the speed and efficiency
of the analysis a rough guess of the parameters is required. This is normally
determined by “preliminary” analyses, rougher, rapid analyses performed, which
are themselves informed by the detection process which identified the event in
the raw detector data. These prior data are analysed by the PEConfigurator
tool to determine appropriate prior ranges, and settings for the waveform
approximant to be used in the analysis.

The calibration of the detectors; the correspondance between the strain on the
detector and the intensity of light at the interferometer’s exit port, can
change over the course of an observing run. The uncertainty in this quantity
is marginalised by many of the analyses, which requires data files to be
collected and provided to the analyses.

Once the correct data, settings, and calibration information has been
identified and collected it is possible to configure analyses. Asimov allows
analyses to be described as a dependency tree, allowing the output data
products from one analysis to be used as an input for another. This is often
useful for coordinating the determination of the PSD of the analysed data.

Each pipeline is configured with a mixture of configuration files and command-
line arguments. Asimov produces the appropriately-formatted configuration file
for each pipeline using a template and substitutions from the production
ledger. The appropriate command line program is then run for the given
pipeline, in order to produce an execution environment and submission data for
the htcondor scheduling system. This is then submitted to the LDG, and the job
id is collected and stored by asimov.

It is then possible to automatically monitor the progress of jobs on the LDG,
produce a webpage summarising the status of all on-going analyses, and detect
the completion of jobs and initialise post-processing.

## Acknowledgements

DW and JV acknowledge support from Science and Technology Facilities Council
(STFC) grants ST/V001736/1 and ST/V005634/1. PS acknowledges support from STFC
grant ST/V005677/1. CH acknowledges support from STFC grant ST/N005430/1 and
European Research Council (ERC) Consolidator Grant 647839. We are grateful for
the support of our colleagues in the LIGO-Virgo Compact Binary Coalescence
Parameter Estimation working group, including, but not limited to Christopher
Berry for his suggestions and insights. The authors are grateful for
computational resources provided by the Leonard E Parker Center for
Gravitation, Cosmology and Astrophysics at the University of Wisconsin-
Milwaukee and supported by National Science Foundation Grants PHY-1626190 and
PHY-1700765.

## References

Abbott, B.P., and others. 2016a. “Observation of Gravitational Waves from a
Binary Black Hole Merger.” _Phys. Rev. Lett._ 116 (6): 061102\.
https://doi.org/10.1103/PhysRevLett.116.061102.

———. 2016b. “Properties of the Binary Black Hole Merger GW150914.” _Phys. Rev.
Lett._ 116 (24): 241102. https://doi.org/10.1103/PhysRevLett.116.241102.

———. 2020. “Prospects for observing and localizing gravitational-wave
transients with Advanced LIGO, Advanced Virgo and KAGRA.” _Living Rev. Rel._
23 (1): 3. https://doi.org/10.1007/s41114-020-00026-9.

Abbott, R., and others. 2021. “GWTC-2: Compact Binary Coalescences Observed by
LIGO and Virgo During the First Half of the Third Observing Run.” _Phys. Rev.
X_ 11: 021053. https://doi.org/10.1103/PhysRevX.11.021053.

Ashton, Gregory, and others. 2019. “BILBY: A user-friendly Bayesian inference
library for gravitational-wave astronomy.” _Astrophys. J. Suppl._ 241 (2): 27.
https://doi.org/10.3847/1538-4365/ab06fc.

Hoy, Charlie, and Vivien Raymond. 2021. “PESUMMARY: The code agnostic
Parameter Estimation Summary page builder.” _SoftwareX_ 15 (July): 100765\.
https://doi.org/10.1016/j.softx.2021.100765.

Khan, Sebastian, Frank Ohme, Katerina Chatziioannou, and Mark Hannam. 2020\.
“Including higher order multipoles in gravitational-wave models for precessing
binary black holes.” _Phys. Rev. D_ 101 (2): 024056\.
https://doi.org/10.1103/PhysRevD.101.024056.

Lange, Jacob, Richard O’Shaughnessy, and Monica Rizzo. 2018. “Rapid and
accurate parameter inference for coalescing, precessing compact binaries.”
_arXiv E-Prints_ , May. http://arxiv.org/abs/1805.10457.

Ossokine, Serguei, and others. 2020. “Multipolar Effective-One-Body Waveforms
for Precessing Binary Black Holes: Construction and Validation.” _Phys. Rev.
D_ 102 (4): 044055. https://doi.org/10.1103/PhysRevD.102.044055.

Pratten, Geraint, and others. 2021. “Computationally efficient models for the
dominant and subdominant harmonic modes of precessing binary black holes.”
_Phys. Rev. D_ 103 (10): 104056. https://doi.org/10.1103/PhysRevD.103.104056.

The LIGO Scientific Collaboration, the Virgo Collaboration, R. Abbott, T. D.
Abbott, F. Acernese, K. Ackley, C. Adams, et al. 2021. “GWTC-2.1: Deep
Extended Catalog of Compact Binary Coalescences Observed by LIGO and Virgo
During the First Half of the Third Observing Run.” _arXiv E-Prints_ , August,
arXiv:2108.01045. http://arxiv.org/abs/2108.01045.

The LIGO Scientific Collaboration, the Virgo Collaboration, the KAGRA
Collaboration, R. Abbott, T. D. Abbott, F. Acernese, K. Ackley, et al. 2021\.
“GWTC-3: Compact Binary Coalescences Observed by LIGO and Virgo During the
Second Part of the Third Observing Run.” _arXiv E-Prints_ , November,
arXiv:2111.03606. http://arxiv.org/abs/2111.03606.

Veitch, J., and others. 2015. “Parameter estimation for compact binaries with
ground-based gravitational-wave observations using the LALInference software
library.” _Phys. Rev. D_ 91 (4): 042003.
https://doi.org/10.1103/PhysRevD.91.042003.

Williams, Daniel, Duncan Macleod, Avi Vajpeyi, and James Clark. 2021.
“Transientlunatic/Asimov,” July. https://doi.org/10.5281/zenodo.4024432.