# Sketch Video Synthesis

Yudian Zheng
University of Saarland and Xiaodong Cun222 Corresponding author.
Tencent AI Lab and Menghan Xia
Tencent AI Lab and Chi-Man Pun
University of Macau and
https://sketchvideo.github.io

###### Abstract

Understanding semantic intricacies and high-level concepts is essential in
image sketch generation, and this challenge becomes even more formidable when
applied to the domain of videos. To address this, we propose a novel
optimization-based framework for sketching videos represented by the frame-
wise Bézier Curves. In detail, we first propose a cross-frame stroke
initialization approach to warm up the location and the width of each curve.
Then, we optimize the locations of these curves by utilizing a semantic loss
based on CLIP features [34] and a newly designed consistency loss using the
self-decomposed 2D atlas network [22]. Built upon these design elements, the
resulting sketch video showcases impressive visual abstraction and temporal
coherence. Furthermore, by transforming a video into SVG lines through the
sketching process, our method unlocks applications in sketch-based video
editing and video doodling, enabled through video composition, as exemplified
in the teaser.

---

Figure 1: Given an input video (of the foreground object), we introduce a
novel method for sketching the video using Bézier Curves so that the video can
be represented by scalable vector graphics (SVG). The generated sketch video
maintains semantic alignment with the input and exhibits temporal consistency.
The flexibility of SVG allows for various rendering techniques, including
resizing, color filling, and overlaying doodles on the original background
images, enabling the creation of diverse artistic effects.

##  1 Introduction

Freehand drawing is a widely adopted method for quickly prototyping ideas
across various domains [15, 47]. This approach embodies simplicity,
abstraction, and adaptability, empowering individuals to effectively express
their concepts. Furthermore, skilled artists can develop distinctive artistic
styles through freehand drawing. While sketching is a common practice for
images, often using formats like scalable vector graphics (SVG), there has
been limited exploration of its application in the context of sketching
videos. This uncharted territory makes the exploration of sketch videos an
intriguing and meaningful endeavor.

While traditional approaches, such as edge detection methods [46, 6] excel in
rendering realistic sketches, they struggle to create more expressive and
abstract representations due to their reliance on mathematical and geometric
operations. In an attempt to incorporate semantic awareness, previous
sketching methods have attempted to learn human-like sketches from a collected
dataset in different levels of abstraction and styles [4, 28, 21] at the pixel
level. While these data-driven methods imitate human sketches, the
requirements and quality of the relevant datasets restrict the output. As
introduced by recent image sketching works [41, 40], line drawings are defined
using the control points of Bézier Curves and are optimized to represent the
scene. These methods employ multi-scale deep perceptual losses [34] to bridge
the gap between generated sketches and real scenes, bypassing the constraints
of traditional datasets and yielding diverse results. We follow these
frameworks to represent video sketches in SVG format. However, if we simply
apply image-based sketching [41] in a frame-wise manner without careful
consideration, the strokes will converge into local minima rapidly.
Additionally, the flickering issue of generated video is not easily resolved
through conventional video deflickering algorithms [26, 25], especially when
dealing with vector graphics.

To overcome the challenges mentioned above, we introduce an innovative
optimization-based framework aimed at generating sketch videos in SVG format
that exhibit both semantic alignment and temporal consistency. To achieve this
goal, we leverage Neural Layered Atlas (NLA) [22] to our tasks for multiple
purposes. NLA is first proposed for video editing, it maps each point in video
to a uniform global UV map, so that it can guarantee the correspondences
across frames and help for temporal point consistency. In detail, the process
of generating a high-quality video sketch involves careful initialization and
continuous optimization of the sketch video. We begin by carefully selecting
the initial locations for candidate points, where an effective initialization
is crucial for avoiding unfavorable local minima [41] and accurately conveying
the video’s semantics. To achieve this, our initialization approach utilizes
semantic-aware edges, derived from salient maps obtained from the combination
of CLIP features [34] and XDoG edge detection [43]. Subsequently, these
selected points are propagated to all frames and optimized to their initial
positions using a pre-trained NLA. Then, we optimize the location of these
points using several losses so that they can ensure both semantic alignment
and temporal consistency. We transform the candidate points to Bézier Curves
and utilize a differentiable rasterizer [29] to render them to the frame-wise
images. For semantic abstraction, we utilize the pre-trained CLIP image
encoder as a feature extractor and compute losses between the rendered one and
the real video frame. For temporal consistency, we ensure consistency of
vector points via the pre-trained NLA [22] so that they can be consistent from
a global view. Based on these techniques, the proposed method can successfully
generate the abstraction sketches of the specific given video.

In addition to streamlining the process of sketching videos, our approach
paves the way for various video applications. For instance, it allows for the
creation of colorful videos by applying drawing techniques to a single frame.
Furthermore, our method introduces novel possibilities in video editing, such
as substituting the original content by integrating sketches into the scenes.
Additionally, our approach enables the generation of video doodles to enhance
other video content.

The contribution of this paper can be summarized as:

  * •

We first tackle the problem of generating scalable abstract videos via several
Bézier curves.

  * •

By utilizing the consistency of pre-trained video implicit representation
[22], we propose a novel point initialization method and a temporal
consistency loss for video sketching synthesis.

  * •

Our method generates the animated SVG from the input video, enabling multiple
new applications of video editing and doodles.

##  2 Related Work

###  2.1 Doodling and Abstraction

Doodling and abstraction are common forms of artistic expression that use few
and sparse curves to depict objects or scenes abstractly. Early works in
sketching, such as traditional edge detection methods [6, 46], effectively
describe the structure and semantics of images. While these methods produce
clear and coherent sketch videos, they lean towards excessive realism, often
neglecting the artistic preferences of viewers. Combining edge detection
methods with line drawing video stylization [3] generates abstract images, but
it often overlooks the holistic aspect of video frames and loses video
consistency. Recent data-driven methods primarily focus on generating human-
like sketches through domain adaptation using carefully curated datasets [16,
1]. These methods offer the ability to create various styles and levels of
abstraction. However, the resulting style is often closely tied to the
specific dataset characteristics, making it less suitable for unstructured and
previously unseen data. Extending these methods to handle videos, which are
typically supervised by image datasets, is also a challenging endeavor.
Another method, such as video doodling [49], is used in sketch addition on
video, but it is not satisfied with a global level of abstraction.

Figure 2: The pipeline of the whole framework. Firstly, we train a layer atlas
to decompose the video into the trained layer atlas. Then, we optimize the
location of the generated frames via the proposed novel initialization methods
and consistency loss.

###  2.2 Vector Sketching Generation

There are some image-to-vectors methods [35, 7, 8] that can typically produce
pixel-aligned results. However, sketching demands a higher level of
abstraction and greater continuity between lines. Recent advancements in
differential gradient algorithms, such as DiffVG [29], have made it feasible
to optimize images and even SVG representations within the pretrained CLIP
[34] space. For example, CLIPDraw [13] explores the potential of optimizing
SVG images to generate drawings that closely align with text prompts,
leveraging the guidance of CLIP. Similarly, Tian and Ha[39] have designed an
evolutionary algorithm to abstract images using vector triangles.

Regarding closely related works, CLIPasso [41] specifically applies the
technique of CLIPDraw [13] to generate object sketches, and CLIPascene [40]
extends the optimization process to incorporate background elements.
Furthermore, similar optimization methods have been employed in the domain of
artistic fonts [18] with the diffusion model [36]. Text-based vector
generation has attracted research attention, as evidenced by Wu _et al_. [45],
who propose a transformer-based method for generating icons from text auto-
aggressively. Different from the vectorization method for images, our approach
focuses on generating sketch videos that need to keep temporal consistency,
and directly using the image-based method will fall into local minimal due to
initialization.

###  2.3 Video Editing and Temporal Consistency

Video editing and stylization have a long history within the computer vision
and graphics communities. Various attempts have been made to achieve
stylization [11, 19]. However, these methods may face challenges in
maintaining tracking consistency. Since frame-wise techniques can generate
high-quality stylized images [14, 20], it has become a common practice to
employ neural networks for reducing temporal inconsistencies as a post-
processing step [26, 5, 24, 27]. Nonetheless, it is important to note that
style transfer techniques primarily rely on measuring perceptual distance
[50], which can result in imperfect stylization due to a lack of deep
comprehension at the semantic level. Some recent works have shown improved
consistency, but often within specific domains, such as portrait videos[12,
48]. For local video editing, layer-atlas-based methods [22, 2] present a
promising approach by allowing video editing on a flattened texture map and
generating results through color-wise mapping.

More recent approaches have explored video editing using diffusion models
[36]. These models offer stronger priors for editing using text. _e.g_., Gen1
[10] trains a conditional model for depth and text-guided video generation,
allowing on-the-fly appearance editing of generated images. Several methods
[44, 33, 23, 31] leverage pre-trained text-to-image diffusion models for zero-
shot or one-shot video editing. Although current methods have shown promising
results for image pixels, there is still a lack of techniques for generating
vector video sketches.

##  3 Methods

Our approach primarily aims to generate the sketch representation of the
objects in video, which can be represented through multiple vector strokes.
Each stroke is represented as the four points Bézier Curves, where we focus on
maintaining both semantic accuracy and temporal consistency. To achieve, this
goal, as shown in Figure 2, we first decompose a video into a 2D
representation using layer atlas [22] (Sec. 3.1). Then, we introduce a new
framework to optimize the location of these points in Sec. 3.2. Finally, we
give some potential applications in Sec. 3.3.

###  3.1 Preliminary: Video Decomposition via Layer Atlas

Unlike previous video synthesis and editing tasks [42, 10], our method focuses
on optimizing the positions of discrete points within curves to ensure
consistent behavior across frames. Therefore, using image-based frame
deflickering methods [26, 25] directly becomes challenging. To address this,
we leverage the trained representation of previous consistent video editing
method, _i.e_., Neural Layered Atlas (NLA [22]), to maintain point
consistency.

As shown in the left part of Fig. 2, the Neural Layered Atlas (NLA) treats the
video as a spatial-temporal 3D volume,where each 3D coordinate within the
video is mapped onto the foreground (or background) 2D $UV$-maps through a
Multi-Layer Perceptron (MLP). Additionally, an extra MLP is employed to assign
color values to a given 2D UV location. To formalize this, let’s consider a
video pixel as $p=(x,y,t)$, where $x$ and $y$ are the coordinates of the pixel
within the $t$-th frame. NLA generates the 2D UV maps for foreground and
background individually as shown in Fig 2. It uses $M_{f}$ and $M_{b}$ to map
3D coordinate $p$ to a 2D location in foreground UV-map as $(u_{f},v_{f})$,
and in background UV-map as $(u_{b},v_{b})$ separately. The MLP $M_{\alpha}$
indicates the ownership of the points (foreground or background) according to
the motion prior and the pre-defined mask losses. Subsequently, a shared MLP
$A$ is trained to map the $(u,v)$ to the corresponding color in RGB space,
ensuring that the same pixel in the real world should have the same color.

This comprehensive process facilitates the video reconstruction. After
training, the mapping MLPs $M_{f}$ and $M_{b}$ indicate the correspondence
between points in the video and specific points within the holistic $UV$-map,
aligning with our requirements for curve mapping consistency. Our method
adheres to the original video decomposition approach of the layer atlas. For
more specific information regarding training and loss functions, detailed
insights can be found in [22].

###  3.2 Differentiable Optimization for Video Sketch

Figure 3: The influence of different initialization methods.

In this section, we give the details on how to generate a sketch video in our
framework. Formally, given a real video
$\\{\mathcal{I}_{1},...,\mathcal{I}_{T}\\}$ contains $T$ frames, we define the
sketching video $\\{\mathcal{S}_{1},...,\mathcal{S}_{T}\\}$ as a set of $N$
strokes $\mathcal{S}_{t}=\\{s_{1},...,s_{N}\\}$ for each frame. Each stroke
$s_{i}$ is defined as the two dimensional Bézier curves, where each curve is
built via $4$ control points $s_{i}=\\{(x_{i},y_{i})^{k}\\}_{k=1}^{4}$. We
empirically use the same index $i$ of stroke $s_{i}$ in different sketches
$\mathcal{S}_{t}$ to represent the same curve across frames and only optimize
their positions. All other curve-related attributes are following the previous
image sketching method [41]. We then use a differentiable rasterizer
$\mathcal{R}$ from DiffVG [29] to transform the control points and attributes
of Bézier Curves into a Scalable Vector Graphics (SVG) to represent the final
sketch video. So the loss functions $\mathcal{L}$ can be optimized between the
real video and the sketches representation. The overall process can be
represented as:

$\arg\min_{(X,Y)}\sum_{t=1}^{T}\mathcal{L}(\mathcal{R}(\mathcal{S}_{t}),\mathcal{I}_{t}),$
(1)

where $(X,Y)$ are all the coordinates of the control points
$\bigcup_{i=1}^{N}\bigcup_{k=1}^{4}(x_{i},y_{i})^{k}$ in the whole sketch
video.

Subsequently, we first introduce our method on how to obtain the initial
stroke settings on video (Sec. 3.2.1). Then, we elaborate on the particulars
of rendering points into curves and optimize the entire video (sec. 3.2.2).

Figure 4: Coordinate Initialization. The initialized points are warmed up by
optimizing the propagated points in different frames $\\{$ $\bigtriangleup$,
$\bigtriangleup$, $\times$, $\times$, $\diamondsuit$,$\diamondsuit$ $\\}$
closer to the sampling points $\\{$ $\bullet$, $\bullet$, $\bullet$ $\\}$ on
the atlas (by mapping MLP $M_{f}$).

####  3.2.1 Strokes Initialization

The objective function of our abstraction is highly non-convex [41] since the
optimization loss is based on rendered views in pixel differences [29]. Thus,
attempting to optimize the location directly from random initialization often
leads to the process getting trapped in local extrema, both in the image and
video abstraction (as in Fig. 3). To overcome this challenge, CLIPasso [41]
employs a saliency-guided initialization process, where strokes are sampled
from the probability map. This map is generated by multiplying the saliency
map via CLIP [34] feature with the image’s edge map extracted using XDoG [43],
and then subjecting the result to softmax normalization. We argue this kind of
initialization is hard to work in processing video since each frame has a
variant attention map, where these varying initial points increase the
difficulties of the point optimizations. Below, we give our solution step by
step.

Point Propagation. We first aim to generate sparse key points across frames,
which are used to represent the entire video. As shown in Figure 4, to build
$T$ frames SVG video where each frame contains $N$ strokes, we first apply
salience-guided initialization process [41] to sample $N$ candidate points in
each frame individually. Then, in order to obtain the initial points on video,
we resample $N$ points from the set of all $N\times T$ candidate points to
build a cross-frame point set $P=\\{p_{1},...,p_{N}\\}$. Here, $p_{n}$ are 3D
coordinates $(x_{n},y_{n},t_{n})$, where n indicates the index of control
points. Then, these cross-frame points are directly propagated to all frames
so that each frame has the same spatial 2D initialization as
$\hat{p}^{t}_{n}=(x_{n},y_{n},t)$.

Position Warmup. Since there can be offsets between the sampled points in
different frames (_e.g._ , the same color markings in various frames in
Figure. 4), we employ an optimization-based approach to alleviate the
differences between the propagated locations and the previously sampled
positions on the UV maps by adjusting the propagated points only. This
alignment is achieved through the use of the pretrained atlas mapping network
$M_{f}$ to optimize the control points across frames on the atlas:

$\mathcal{L}_{warmup}=\sum_{t=1}^{T}\sum_{n=1}^{N}||M_{f}(\hat{p}_{n}^{t}))-M_{f}(p_{n}))||_{1},$
(2)

In this situation, $t$ represents the frame index, $n$ signifies the control
point index, $p$ denotes the sampled points, and $\hat{p}$ signifies the
propagated points.

In practice, we warm up 300 iterations. This strategy helps us to find the
most suitable initialization points. As shown in Fig. 3, random initialization
(or solely attention-based methods [41] with similar results) yields poor
performance due to the lack of correspondence between the same index points
across the video. While propagating the points across different frames helps
establish correspondence, it can still lead to a loss of focus on the object.
Different attention strategies yield varying results, with frame-wise
attention outperforming the attention map based on the atlas. This is because
the atlas tend to exhibit distortions compared to the more realistic frames.
Ultimately, the performance is further enhanced and becomes more accurate with
the inclusion of a warm-up phase in the entire initialization process.

Figure 5: The width $w_{t}$ of strokes in $t$-th frame is scaled by the
variable $sc_{t}$, with initialized width $w$. The variable $\gamma$ controls
the contrast ratio of scale (default value is 3).

Curve Width Initialization. Additionally, we also initiate the curve widths,
considering that generated videos can involve significant motions and varying
scales. Since more accurate stroke thickness enhances the representation of
contours, instead of solely relying on the mask’s scale, we leverage the
object’s distortion to estimate the appropriate scale. In detail, as depicted
in Figure. 5, we randomly pair points in each frame to calculate the
differences between points (here, we use all warmed-up points to create
pairs). By computing the $\gamma$ root of the weighted pair-wise differences
on frame $t$ (with $\gamma$ defaulted to 3 as a measure of distortion in the
3D world), we derive a scale factor $sc_{t}$ and curve width $w_{t}$:

$sc_{t}=\sqrt[\gamma]{\frac{\sum_{n=1}^{N}D_{n}^{t}}{\mathop{\max}\limits_{1\leq
i\leq T}(\sum_{n=1}^{N}D^{i}_{n})}},$ (3) $w_{t}=sc_{t}\times w,$ (4)

where $w$ is the default width. Consequently, more distant objects are
depicted with finer lines to capture intricate details and scale, while closer
objects are outlined with thicker lines. This strategy enhances the quality of
abstraction, particularly in scenarios involving substantial object motion.

####  3.2.2 Curves Optimization

Following a well-executed stroke initialization for the video, our aim is to
optimize the curves based on the positions of their points. This optimization
needs to ensure that the resulting sketch video maintains not only semantic
similarities with the input video but also consistency across frames.

Firstly, we follow image-based sketching methods [41] to use the semantic-
aware loss $\mathcal{L}_{semantic}$ to measure the differences between the
generated points and the original video. $\mathcal{L}_{semantic}$ is based on
the differences in multi-scale perception features extracted from the pre-
trained CLIP image encoder [34]. Since CLIP is trained on a larger-scale
dataset to align the text and image information through contrastive learning,
the semantic information can be well-aligned. Formally, for the control points
$\mathcal{S}=\\{s_{1},...,s_{N}\\}$ where
$s_{i}=\\{(x_{i},y_{i})^{k}\\}_{k=1}^{4}$ on each SVG, we optimize the
positions of the control points based on the difference between the generated
SVG and the actual input image $\mathcal{I}$ via the $l$-th layer of
pretrained CLIP $\Phi$:

$\mathcal{L}_{semantic}=\sum_{l\in[3,5,9]}\sum_{t=1}^{T}||\Phi_{l}(\mathcal{R}(\mathcal{S}_{t}))-\Phi_{l}(\mathcal{I}_{t}))||_{1},$
(5)

Here, $T$ represents the total number of frames, and $\mathcal{R}$ is the
differentiable rasterizer as introduced above,

Figure 6: The differences in the choice of different semantic losses.

We also investigate the impact of variants of the CLIP encoders, specifically,
the ViT [9] based and ResNet101-based models [17]. The ResNet101-based CLIP
model exhibits better performance with more local structures, while the ViT-
based methods focus more on global features. Consequently, we default to using
the ResNet101-based CLIP model, as depicted in Figure 6.

Besides, for our video sketching, we design a novel consistency loss to
measure the consistency between the generated SVG videos. The most naive
approach is to maintain control over the corresponding offsets to guarantee
temporal consistency via optical flow [38, 42]. While optical flows are
typically dense, they might suffer from potential errors, _e.g_., forward-
backward consistency, and cumulative errors. Therefore, we employ the trained
atlas network to obtain a panoramic view of the video. This allows each point
in every frame of the video to be associated with a consistent global position
on the atlas. By ensuring that related points and lines have consistent global
positions, we can maintain the continuity of the video. Specifically, we use
the pre-trained mapping network $M_{f}$ from the layer atlas as shown in
Figure 2, where for each control point $\hat{p}_{n}^{t_{1}}$ and its
neighboring $\hat{p}_{n}^{t_{2}}$ of the same index $n$ cross frames, where
the $\mathcal{L}_{consistency}$ can be written as:

$\mathcal{L}_{consistency}=\sum_{\hat{p}^{\prime}\in\mathcal{N}(\hat{p})}\sum_{\hat{p}\in
P}||M_{f}(\hat{p}))-M_{f}(\hat{p}^{\prime})||_{1},$ (6)

After optimization, the same index $n$ point $\hat{p}_{n}$ cross frames will
have a similar location in the 2D $UV$-map and it keeps consistency over
frames.

Overall, the loss function can be written as:

$\mathcal{L}=\omega_{1}\mathcal{L}_{semantic}+\omega_{2}\mathcal{L}_{consistency},$
(7)

where $\omega_{1}$ and $\omega_{2}$ are used to control the extent of
semantics and consistency between sketches, respectively. We experimentally
set $\omega_{1}=200.0$ and $\omega_{2}=3.0$.

###  3.3 Applications

SVG Editing. Because the generated video is in the form of SVG, our method has
the ability to change the colors of all lines or fill specific lines within
the video to create richer details in the visual content. As shown in Fig. 1,
we can resize the canvas, paint all objects in orange, and fill the enclosed
lines.

Video Editing and Doodling. Our method can also be employed for video editing
and doodle creation. As shown in Fig. 1, after generating the SVGs, we can use
inpainting techniques[37] to restore the original video and integrate the
newly generated content into it. Additional video results are presented in the
supp. video.

##  4 Experiments

Our method is evaluated on DAVIS dataset [32], which provides foreground
annotations for each frame. We also evaluate the proposed method on some self-
collected datasets, utilizing the video matting method [30] to generate the
foreground videos. In each case, we only used the first 50 frames in the video
to generate the results for a fair comparison. Subsequently, we decompose the
video into an atlas. For the optimization of sketching videos, we utilize the
Adam optimizer with a learning rate of 1.0, following the approach of CLIPasso
[41]. On average, optimizing a video sketch takes approximately 29 minutes,
consuming around 19.5GB on a single NVIDIA GeForce RTX 3090 GPU.

Figure 7: Comparisons with our methods and the states-of-the-art methods
frame-wise methods, _i.e_., frame-wise CLIPasso [41] and edge detection
methods (canny [6] and HED [46]) on different frames of the videos.

###  4.1 Compare with state-of-the-art methods

As no previous video sketching methods exist, we employ some image sketching
and edge detection techniques for comparison. For image abstraction, in
particular, we compare with frame-by-frame CLIPasso [41]. Regarding edge
detection, we assess against traditional edge detection techniques, namely
Canny [6], as well as the deep learning-based method, HED [46]. As
demonstrated in Figure 7, in comparison to CLIPasso, our approach exhibits
improved temporal consistency while effectively preserving semantic
information. Concerning the edge detection methods, our proposed technique
showcases superior semantic-aware abstraction. Further insights and video
comparisons can be found in the supplementary video.

| Temporal $\uparrow$ | Abstraction $\uparrow$ | Overall $\uparrow$
---|---|---|---
Canny | 2.65 | 2.58 | 2.51
HED | 2.67 | 2.30 | 2.60
CLIPasso | 2.13 | 2.41 | 2.31
Ours | 2.58 | 2.74 | 2.71
Table 1: The mean opinion of the user studies.

Recognizing the absence of appropriate metrics for numerically evaluating
sketching, we conduct user studies to assess the performance of the generated
video. In detail, for each of the 12 clips, we provide generated videos
created by different methods for comparison. We then invite 25 individuals to
rank the resulting videos in terms of semantic alignment (abstraction),
temporal consistency (temporal), and overall quality (Overall), respectively.
The order of the generated videos is shuffled, and participants score the
results from 1 to 4 based on their ranking. As presented in Table 1, our
method receives more favorable feedback from users when compared to the
baseline CLIPasso [41], demonstrating improvements across temporal
consistency, semantic abstraction, and overall quality. Moreover, it’s
noteworthy that our approach achieves enhanced semantic results with scores on
par with those of edge detection methods in terms of temporal consistency.

Figure 8: The importance of the proposed consistency loss.

###  4.2 Ablation Studies

Figure 9: Point Visualization. To further support the effectiveness of the
proposed consistency loss on the atlas, we visualize the location of the same
curves (same color) on different frames during optimization. Kindly take note
that the warmup points have been gathered, and it is recommended to view them
with zoom in.

Optimization Visualization. We introduce a novel consistency loss based on the
trained atlas network. Here, we visualize the points on the global UV map
(atlas) to gain a clearer understanding. As illustrated in Fig 9, points with
the same index across multiple frames are represented in the same color.
During the optimization process, the control points from different frames are
appropriately positioned at the same points after the warming-up phase.
Subsequently, we optimize these points using both the consistency loss and the
semantic loss to ensure performance in terms of semantic alignment and
coherence.

Consistency Weights. The novel consistency loss maintains temporal domain
consistency through the trained atlas representation. In our ablation study,
we investigate this loss using different values. As depicted in Fig. 8, when
the consistency loss is excluded (_i.e_., $\omega_{consistency}=0$), the
generated sketch exhibits distinct representations across frames. As we
increment this parameter, the optimized results demonstrate enhanced
stability.

Strokes Number and Width. We also show the SVG using different numbers and
widths of the default strokes. Increasing the number of paths, as illustrated
in Fig. 10, leads to the generation of sketches that capture more intricate
details. For instance, the details of the dress and its movements become more
prominent and well-defined, which reduces the level of abstraction. Likewise,
adjusting the stroke width yields similar effects. Decreasing the stroke width
during optimization emphasizes finer details within the strokes rather than
the overall structure, resulting in a less abstract representation.

Figure 10: The ablations on the width and stroke numbers.

###  4.3 Limitation

While our method excels at generating coherent and semantically rich videos,
the limitations of the trained layer atlas [22] constrain the quality of the
sketches. For instance, it faces challenges when representing the motion of
non-rigid bodies. Additionally, during cases of self-occlusion, the generated
abstract sketches may contain errors, often appearing as improper turns (as
observed in the supplementary video). Furthermore, the generated sketches may
exhibit some texture artifacts when the video undergoes significant motion or
involves complex foreground elements. As depicted in Fig. 11, our proposed
method exhibits artifacts when the male face shakes violently, and the atlas
struggles to accurately represent the woman in the image. While some of these
issues can be mitigated by subdividing the video into smaller sections and
employing more layers, the accuracy of segmentation and the computational cost
associated with multi-layer approaches remain challenges.

Figure 11: Limitations. When the motion is large and the object is complex,
the generated atlas may not be accurate, causing temporal inconsistency.

###  4.4 Conclusion

We present an optimization-based approach for generating sketch videos that
maintain both semantic and temporal consistency. Our method facilitates the
creation of sketching videos with the appropriate level of abstraction.
Essentially, it includes a novel initialization technique for acquiring well-
initialized points and a distinctive consistency loss derived from self-
supervised video decomposition. These innovations empower us to craft sketch
videos using simple Bézier curves. Since the resulting videos are composed
using Scalable Vector Graphics, our proposed methods offer versatile
applications in video editing and doodling, accommodating various sizes while
preserving intricate details.

## References

  * [1] P Arbeláez, M Maire, C Fowlkes, and J Malik.  Contour detection and hierarchical image segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, page 898–916, Aug 2010.
  * [2] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  In European Conference on Computer Vision, pages 707–723. Springer, 2022.
  * [3] N. Ben-Zvi, J. Bento, Moshe Mahler, Jessica K. Hodgins, and Ariel Shamir.  Line‐drawing video stylization.  Computer Graphics Forum, 35, 2016.
  * [4] Itamar Berger, Ariel Shamir, Moshe Mahler, Elizabeth Carter, and Jessica Hodgins.  Style and abstraction in portrait sketching.  ACM Trans. Graph., 32(4), jul 2013.
  * [5] Nicolas Bonneel, James Tompkin, Kalyan Sunkavalli, Deqing Sun, Sylvain Paris, and Hanspeter Pfister.  Blind video temporal consistency.  ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2015), 34(6), 2015.
  * [6] John Canny.  A computational approach to edge detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-8(6):679–698, 1986.
  * [7] Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte.  Deepsvg: A hierarchical generative network for vector graphics animation.  Advances in Neural Information Processing Systems, 33:16351–16361, 2020.
  * [8] Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song.  Béziersketch: A generative model for scalable vector sketches.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVI 16, pages 632–647. Springer, 2020.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [10] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  arXiv preprint arXiv:2302.03011, 2023.
  * [11] Jakub Fišer, Ondřej Jamriška, Michal Lukáč, Eli Shechtman, Paul Asente, Jingwan Lu, and Daniel Sỳkora.  Stylit: illumination-guided example-based stylization of 3d renderings.  ACM Transactions on Graphics (TOG), 35(4):1–11, 2016.
  * [12] Jakub Fišer, Ondřej Jamriška, David Simons, Eli Shechtman, Jingwan Lu, Paul Asente, Michal Lukáč, and Daniel Sỳkora.  Example-based synthesis of stylized facial animations.  ACM Transactions on Graphics (TOG), 36(4):1–11, 2017.
  * [13] Kevin Frans, Lisa Soros, and Olaf Witkowski.  Clipdraw: Exploring text-to-drawing synthesis through language-image encoders.  Advances in Neural Information Processing Systems, 35:5207–5218, 2022.
  * [14] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016.
  * [15] Yulia Gryaditskaya, Mark Sypesteyn, Jan Willem Hoftijzer, Sylvia Pont, Frédo Durand, and Adrien Bousseau.  Opensketch: a richly-annotated dataset of product design sketches.  ACM Transactions on Graphics, page 1–16, Nov 2019.
  * [16] David Ha and Douglas Eck.  A neural representation of sketch drawings, Apr 2017.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [18] Shir Iluz, Yael Vinker, Amir Hertz, Daniel Berio, Daniel Cohen-Or, and Ariel Shamir.  Word-as-image for semantic typography.  arXiv preprint arXiv:2303.01818, 2023.
  * [19] Ondřej Jamriška, Šárka Sochorová, Ondřej Texler, Michal Lukáč, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sýkora.  Stylizing video by example.  ACM Trans. Graph., 38(4), jul 2019.
  * [20] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 694–711. Springer, 2016.
  * [21] Moritz Kampelmuhler and Axel Pinz.  Synthesizing human-like sketches from natural images using a conditional convolutional decoder.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3203–3211, 2020.
  * [22] Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel.  Layered neural atlases for consistent video editing.  ACM Transactions on Graphics (TOG), 40(6):1–12, 2021.
  * [23] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  arXiv preprint arXiv:2303.13439, 2023.
  * [24] Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang.  Learning blind video temporal consistency.  In Proceedings of the European conference on computer vision (ECCV), pages 170–185, 2018.
  * [25] Chenyang Lei, Xuanchi Ren, Zhaoxiang Zhang, and Qifeng Chen.  Blind video deflickering by neural filtering with a flawed atlas.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023.
  * [26] Chenyang Lei, Yazhou Xing, and Qifeng Chen.  Blind video temporal consistency via deep video prior.  Advances in Neural Information Processing Systems, 33:1083–1093, 2020.
  * [27] Chenyang Lei, Yazhou Xing, Hao Ouyang, and Qifeng Chen.  Deep video prior for video consistency and propagation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):356–371, 2022.
  * [28] Mengtian Li, Zhe Lin, Radomir Mech, Ersin Yumer, and Deva Ramanan.  Photo-sketching: Inferring contour drawings from images.  In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1403–1412. IEEE, 2019.
  * [29] Tzu-Mao Li, Michal Lukáč, Gharbi Michaël, and Jonathan Ragan-Kelley.  Differentiable vector graphics rasterization for editing and learning.  ACM Trans. Graph. (Proc. SIGGRAPH Asia), 39(6):193:1–193:15, 2020.
  * [30] Shanchuan Lin, Linjie Yang, Imran Saleemi, and Soumyadip Sengupta.  Robust high-resolution video matting with temporal guidance.  CoRR, 2021.
  * [31] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia.  Video-p2p: Video editing with cross-attention control.  arXiv:2303.04761, 2023.
  * [32] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  arXiv preprint arXiv:1704.00675, 2017.
  * [33] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.  Fatezero: Fusing attentions for zero-shot text-based video editing.  arXiv:2303.09535, 2023.
  * [34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [35] Pradyumna Reddy, Michael Gharbi, Michal Lukac, and Niloy J Mitra.  Im2vec: Synthesizing vector graphics without vector supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7342–7351, 2021.
  * [36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * [37] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky.  Resolution-robust large mask inpainting with fourier convolutions.  arXiv preprint arXiv:2109.07161, 2021.
  * [38] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020.
  * [39] Yingtao Tian and David Ha.  Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts, page 275–291.  Springer, Apr 2022.
  * [40] Yael Vinker, Yuval Alaluf, Daniel Cohen-Or, and Ariel Shamir.  Clipascene: Scene sketching with different types and levels of abstraction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4146–4156, October 2023.
  * [41] Yael Vinker, Ehsan Pajouheshgar, JessicaY. Bo, RomanChristian Bachmann, AmitHaim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir.  Clipasso: Semantically-aware object sketching.  ACM Trans. Graph., 2022.
  * [42] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  Video-to-video synthesis.  In Advances in Neural Information Processing Systems (NeurIPS), 2018.
  * [43] Holger Winnemöller, Jan Eric Kyprianidis, and Sven C Olsen.  Xdog: An extended difference-of-gaussians compendium including advanced image stylization.  Computers & Graphics, 36(6):740–753, 2012.
  * [44] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  arXiv preprint arXiv:2212.11565, 2022.
  * [45] Ronghuan Wu, Wanchao Su, Kede Ma, and Jing Liao.  Iconshop: Text-based vector icon synthesis with autoregressive transformers.  arXiv preprint arXiv:2304.14400, 2023.
  * [46] Saining Xie and Zhuowen Tu.  Holistically-nested edge detection.  In Proceedings of the IEEE international conference on computer vision, pages 1395–1403, 2015.
  * [47] Peng Xu, TimothyM. Hospedales, Qiyue Yin, Yi-Zhe Song, Tao Xiang, and Liang Wang.  Deep learning for free-hand sketch: A survey and a toolbox.  arXiv: Computer Vision and Pattern Recognition, Jan 2020.
  * [48] Shuai Yang, Liming Jiang, Ziwei Liu, and Chen Change Loy.  Vtoonify: Controllable high-resolution portrait video style transfer.  ACM Transactions on Graphics (TOG), 41(6):1–15, 2022.
  * [49] Emilie Yu, Kevin Blackburn-Matzen, Cuong Nguyen, Oliver Wang, Rubaiat Habib Kazi, and Adrien Bousseau.  Videodoodles: Hand-drawn animations on videos with scene-aware canvases.  ACM Transactions on Graphics (TOG), 42:1 – 12, 2023.
  * [50] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.


# A Multi-Modal Respiratory Disease Exacerbation Prediction Technique Based on
a Novel Spatio-Temporal Machine Learning Architecture and Local Environmental
Sensor Networks

Rohan Tan Bhowmik1 1Rohan T. Bhowmik is a student at the Harker School, 500
Saratoga Ave, San Jose CA, USA 95138. He can be contacted at:
<EMAIL_ADDRESS>
###### Abstract

Chronic respiratory diseases, such as the chronic obstructive pulmonary
disease and asthma, are a serious health crisis, affecting a large number of
people globally and inflicting major costs on the economy. Current methods for
assessing the progression of respiratory symptoms are either subjective and
inaccurate, or complex and cumbersome, and do not incorporate environmental
factors. Lacking predictive assessments and early intervention, unexpected
exacerbations can lead to hospitalizations and high medical costs.

This work presents a multi-modal solution for predicting the exacerbation
risks of respiratory diseases, such as COPD, based on a novel spatio-temporal
machine learning architecture for real-time and accurate respiratory events
detection, and tracking of local environmental and meteorological data and
trends. The proposed new machine learning architecture blends key attributes
of both convolutional and recurrent neural networks, allowing extraction of
both spatial and temporal features encoded in respiratory sounds, thereby
leading to accurate classification and tracking of symptoms. Combined with the
data from environmental and meteorological sensors, and a predictive model
based on retrospective medical studies, this solution can assess and provide
early warnings of respiratory disease exacerbations.

This research will improve the quality of patients’ lives through early
medical intervention, thereby reducing hospitalization rates and medical
costs.

##  I INTRODUCTION

Chronic respiratory diseases affect a large fraction of the world population,
with Chronic Obstructive Pulmonary Disease (COPD) affecting 235 million and
asthma affecting 339 million people worldwide, according to the World Health
Organization [1]. Lacking effective early intervention, COPD and asthma cost
over $130 Billion annually in the U.S. alone [2].

Existing methods of diagnosis and tracking of these disease conditions in
clinical practice, including widely-used patient questionnaires, are highly
variable due to the subjectivity of definition, perception, and reporting of
respiratory events. In fact, many respiratory diseases are often over- or
under-diagnosed. Based on the study by Diab. et al, approximately 70 percent
of COPD cases worldwide may be underdiagnosed, while 30 to 60 percent of those
diagnosed with COPD may not have the disease at all [3]. As the treatment of
respiratory diseases often requires the prescription of steroids, misdiagnosis
can cause serious problems.

Currently, no passive monitoring method exists for accurately predicting the
exacerbation of respiratory conditions. A number of cough detection methods
have been reported, but no accurate real-time tracking technique exists for
passive and continuous monitoring. Commonly used methods involve subjective
reporting, often leading to frequent and dangerous misdiagnosis [4-6]. Besides
the respiratory conditions of the patient, environmental factors such as
pollen, humidity, air quality, etc., also play a significant role in the
disease progression, exacerbations, and hospitalizations [7]. However,
currently there is no multi-modal predictive technique that incorporates the
trends of both respiratory events and local environmental factors in order to
assess the progression of the patient’s conditions.

Thus, the development of an accurate and real-time predictive solution for
respiratory disease exacerbation that is easily accessible is highly needed,
based on monitoring of patient’s respiratory events as well as the local
environmental and meteorological parameters. The recent advances in connected
devices, sensors, data technologies, and machine learning techniques present a
significant opportunity to develop respiratory telehealth capabilities,
allowing for accurate remote monitoring of patient conditions as well as
assessing potential exacerbations with predictive Artificial Intelligence (AI)
models.

This work presents a multi-modal solution for real-time COPD exacerbation
prediction that includes a novel spatio-temporal artificial intelligence
architecture for cough detection, real-time cough-count and frequency
monitoring, analytics of the local environmental and meteorological factors
utilizing data from sensor networks, and exacerbation prediction using both
respiratory event tracking and environmental conditions based on retrospective
medical studies. The goal of this research is to develop an early-warning
system based on AI and multi-factor analysis to reduce hospitalizations and
medical costs, and demonstrate the feasibility of deploying a passive,
continuous, remote patient monitoring and telehealth solution for chronic
respiratory diseases.

##  II PRIOR RESEARCH

Researchers have previously identified that monitoring a patient’s respiratory
events can be utilized to assess the patient’s condition [8]. In order to
automate this process, a number of cough detection solutions have been
proposed [9-15]. A survey of previously reported techniques, performances and
limitations are listed in Fig. 1. Earlier methods used relatively simpler
techniques, such as probabilistic statistical models on waveform data [9], but
also yielded low accuracies. On the other hand, more recent studies have used
specialized equipment and complex setups, such as wireless wearable patch
sensors [13] or spirometers [15], to achieve relatively better results.
However, no single technique simultaneously meets all of the following
requirements: highly accurate, efficient, passive and continuous monitoring,
and does not need extra equipment.

Figure 1: A survey of previously reported techniques for automatic cough
detection (PNN = Probabilistic Neural Network; SP = Specificity; SE =
Sensitivity; ACC = Accuracy). The above techniques follow these observations:
i) generally higher accuracies were achieved with more complex models
utilizing spectrograms; ii) techniques aided with extra equipment produced
better results; iii) no single technique simultaneously meets all of the
following requirements: highly accurate, efficient, passive and continuous
monitoring, does not need extra equipment.

With the recent advancements in the field of artificial intelligence,
researchers have moved towards exploring solutions based on Deep Neural
Networks (DNN). Several researchers have demonstrated detection of cough with
either Convolutional Neural Networks (CNN) or Recurrent Neural Networks (RNN).
Traditional CNN models are based on learning and detecting spatial features in
the data and are typically used for image-based analysis, whereas RNN models
are based on extracting temporal features and are often used for time-
sequenced tasks such as speech processing. Since respiratory sounds, when
converted to spectrograms, encode key spatial and temporal signatures, neither
of the traditional models is well suited for respiratory event classification.

Some researchers have recently reported combined Convolutional-Recurrent
Neural Networks (CRNN) for acoustic analysis [16-20]. These CRNN models have
been shown to work better than CNN and RNN in both image processing and
sequence-related tasks [16, 19], but these frameworks do not fully utilize the
spatial/temporal feature extraction capabilities of CNN/RNN architectures as
they are created by simply stacking RNN layers after CNN layers in a
sequential manner. The development of machine learning architecture based on
deeply meshed spatio-temporal feature learning for respiratory sound
classification has not been previously explored.

Medical researchers have also shown that several key environmental and
meteorological factors are related to the exacerbations of COPD [7]; however,
this research has not been combined with real-time monitoring of respiratory
events to develop predictive models for exacerbations.

##  III METHODS

###  III-A Proposed Multi-Modal System Architecture

In this project, a novel multi-modal COPD patient monitoring and exacerbation
prediction system has been developed based on real-time analysis and tracking
of both respiratory events and environmental factors. As shown in Fig. 2, the
system architecture consists of three stages: i) a detection module, ii) an
environmental module, and finally, iii) a prediction module.

The detection module utilizes a new spatio-temporal machine learning algorithm
for accurately detecting coughs from real-time audio and tracking the
patient’s cough count and frequency. Simultaneously, the environmental module
acquires local environmental and meteorological data from nearby weather
stations and sensor networks to calculate the percentage increase of
exacerbation risks in any location around the world based on the results of
retrospective medical studies. Finally, the prediction module combines the
historical cough count data and trends from the detection module and the
calculated exacerbation risk increase from the environmental module in order
to forecast the progression of the patient’s conditions, and alert the
patients and caregivers for early interventions.

Figure 2: This flowchart represents the proposed system architecture for real-
time multi-modal exacerbation prediction. The detection module depicts the
respiratory event analysis system using a novel spatio-temporal artificial
intelligence neural network. The prediction module depicts the disease
exacerbation prediction system using the output of the machine learning model
and environmental factors. The prediction module takes the respiratory event
data and trends from the detection module, forecasts the progression of the
patient’s conditions, and provides necessary alerts for early intervention.

###  III-B Detection Module

The detection module, as shown on the left-hand side of the system
architecture diagram in Fig. 2, consists of a new AI model for real-time
detection and tracking of cough. As described earlier, previously reported
models for respiratory sound analysis are based on the traditional
convolutional, recurrent, or the more recent convolutional-recurrent
structures. In this project, a new machine learning algorithm has been
developed that incorporates a novel hybrid framework by deeply meshing
convolutional and recurrent architectures, enabling more efficient extraction
and analysis of spatio-temporal features, leading to better accuracies for
classifying and tracking respiratory events.

The following subsections describe the new spatio-temporal machine learning
framework for classifying and tracking respiratory events, creation of the
dataset to train and test the model, the results of benchmarking the proposed
model with traditional neural network architectures, and a live demonstration
application showcasing the capability of real-time classification of
respiratory sounds.

####  III-B1 A New Machine Learning Architecture for Respiratory Sound
Analysis

The new AI model, henceforth referred to as the Spatio-Temporal Artificial
Intelligence Network (STAIN), interweaves convolutional neural network models
within a recurrent neural network architecture, allowing for sequential image
analysis over the time domain. The architecture of the STAIN framework is
shown in Fig. 3. First, the respiratory sound files are converted to
corresponding spectrogram images by performing Fast Fourier Transforms. The
resulting spectrogram is split into 200 millisecond slices, which are used as
inputs for the machine learning model.

Figure 3: Architecture for the new machine learning model, which is referred
to as the spatio-temporal artificial intelligence network (STAIN). This
proposed AI model deeply blends the elements of both convolutional and
recurrent neural networks, and effectively learns both spatial and temporal
features encoded within the respiratory sound spectrograms for accurate
classifications.

As illustrated in Fig. 3, the machine learning model architecture incorporates
a hybrid network based on a deep mesh integration of convolutional and
recurrent architectures to learn spatio-temporal features. The STAIN framework
consists of a CNN model which evaluates the corresponding audio slices and
outputs its predicted confidence. The CNN architecture is a variation of Yann
Lecun’s seminal LeNet model [21], which can flexibly adapt to any image
dataset. Specifically, it consists of two groupings of Convolutional Layers of
2x2 kernels and 2x2 Maximum Pooling Layers followed by Rectified Linear Unit
(ReLU) activation function. Then, the resulting data is flattened into a one-
dimensional array before feeding it into two Fully Connected (Dense) Layers to
reduce the number of neurons down to just one. The final output is then passed
through a Sigmoid Layer to obtain a value between (0, 1).

The CNNs analyzing separate parts of the input image enable spatial feature
extraction, while the Encoders passing down compressed inputs as RNN’s hidden
variables enable temporal feature extraction. Various designs for the Encoder
have been explored, starting with a simple architecture consisting of a single
Maximum Pooling layer, shrinking the input into a hidden variable. A simple
Variational Auto-Encoder (VAE) has also been created, consisting of two
Deconvolutional Layers followed by Convolutional Layers.

Effectively, each slice of the spectrogram image is assigned to an RNN unit,
wherein a CNN generates an output and the Encoder generates the hidden data.
Each output represents the probability of a cough during that slice. The
hidden outputs carry on information from previous slices and are concatenated
to the next slice. The final output is the maximum of all the outputs from all
slices. All the codes in this project were written in Python, and the machine
learning models were implemented using the PyTorch Libraries.

####  III-B2 Creation of the Dataset

In order to train and evaluate the proposed STAIN machine learning model as
well as benchmark with other state-of-the-art models including CNN, RNN and
CRNN, an augmented dataset of audio segments were created and partitioned into
10,000 training files with coughs, 10,000 training files without coughs, 1,000
testing files with coughs, and 1,000 testing files without coughs. The models
were trained only on the 20,000 training files and tested only on the 2,000
testing files in order to objectively evaluate and compare the performance of
various models.

First, roughly 500 cough sound files were downloaded from the Free Sound
Database (FSD) from Kaggle’s audio tagging competition [22] and every file was
adjusted to only contain either a cough burst or coughing fit. The cough files
were sufficiently diverse, containing many variations of coughs from
individuals of both genders and from a wide range of ages (from babies to
elderly). Each file also has it’s unique recording quality, mimicking the
varying degrees of audio quality from different devices.

In order to augment the data, the rest of the audio files from Kaggle’s FSD
were utilized. To create an augmented audio file, an empty audio file is
created with a duration randomly chosen between 2 seconds and 5 seconds. Then,
using the PyDub Library, a randomly chosen number of non-cough files from the
FSD are superimposed on the targeted augmented file. Each of the added audio
files are placed at a randomly chosen timestamp, with audio exceeding the
augmented files trimmed off. The result of this process creates an augmented
audio file categorized as “No Cough”. To turn it into a “Cough” file, one of
the cough files from the FSD is added in a similar fashion. Additionally, each
added file’s decibel gain is randomized to simulate sounds from varying
distances.

####  III-B3 Benchmarking and Results

Using the dataset described in the previous section, rigorous evaluations of
the four different AI models were performed. The results of these analyses are
shown in Fig. 4 and Fig. 5, which present the following performance metrics:
sensitivity, specificity, accuracy, Matthews Correlation Coefficients, and the
confusion matrices.

As these results illustrate, compared to RNN’s temporal feature analysis,
CNN’s spatial analysis was better suited for classifying spectrograms. CRNN,
created by simply stacking the CNN and RNN components, could not bring out the
best of both architectures, performing worse than CNN. In contrast, the
proposed new machine learning model, STAIN, performed better than all other
models using its architecture for deeply meshed spatio-temporal feature
analysis.

Figure 4: Summary of results of the comparative study of CNN, RNN, CRNN, and
the proposed STAIN machine learning models for cough detection. The
sensitivity, specificity, accuracy, and Matthews Correlation Coefficient
metrics were obtained for all four models using the same datasets. As can be
seen, the STAIN model outforms all the other traditional AI models with it’s
deeply meshed spatio-temporal feature extraction architecture, which is more
advantageous for effectively classifying respiratory events. Figure 5:
Confusions Matrices, created using MatPlotLib, for (A) CNN, (B) RNN, (C) CRNN,
and (D) the proposed STAIN machine learning models. The new STAIN architecture
outperforms the traditional neural network architectures for more accurate
cough detection. Figure 6: Screenshots of the live demonstration of the cough
detection module based on the new spatio-temporal machine learning model. The
real-time application, implemented on a laptop computer, captures user-
generated sounds using it’s integrated microphones, converts the sound into
spectrogram images, processes through the STAIN model to detect the presence
of cough, and displays the results on the screen. Figure 7: Correlations
between the degradation of the environmental and meteorological factors and
the increase in COPD exacerbation risks, derived from retrospective medical
studies [7, 23]. As an example, these studies demonstrated that an increase in
NO2 concentration by 10 ug/m³ resulted in about 2% increase in the risk. These
correlations were used to estimate the overall risk trends based on the real-
time data from local sensors. Figure 8: Equation to estimate the increase in
COPD exacerbation risks as a function of environmental and meteorological
factors (PM2.5, PM10, NO2, and T${}_{\text{F}}$ for Temperature), derived
based on the retrospective medical studies [7, 23].

####  III-B4 Demonstration of the Detection Module

A live demo application for the real-time cough detection module has been
developed. This application, running on a laptop computer, captures user-
generated sounds using the built-in microphones of the computer, converts the
sound files into spectrogram images, processes the data through the STAIN
machine learning model, classifies and tracks the cough count and cough
frequency over time. The results are presented on the computer screen with a
live display of the spectrogram images corresponding to the sound,
superimposed with the classification results of the cough events. Fig. 6 shows
the representative screenshots of the application running real-time, and
correctly classifying talking, clapping, page flipping, music, burp, and
sneezes as “No Cough” (left-hand side of Fig. 6), whereas successfully
detecting cough events superimposed with the same background sound
environments (right-hand side of Fig. 6).

Figure 9: Data maps for the relevant environmental and meteorological factors
(PM2.5, PM10, NO2, and Temperature), obtained from the sensors deployed by
PurpleAir and the WAQI data platform. An extrapolation method was used to
estimate the data in areas with sparse sensor coverage. Figure 10: Sensors
deployed by PurpleAir in Irvine and San Jose showed that the PM2.5
concentration spiked to dangerous levels during Sept. 2-13, 2020, fire season.
The onsets of spikes on Sept 6 and Sept 10 correspond to the El Dorado Fire
and the SCU Lightning Fire events. Figure 11: Illustration of the procedures
implemented within the prediction module that forecasts the expected
progression of the condition of the patient in the days ahead. This final step
in the multi-modal architecture combines the results from the respiratory
sound analysis performed by the machine learning model of the detection
module, and the environmental and meteorological factors and trends analysis
conducted by the environmental module. By extrapolating the cough frequency
trends, along with the predicted exacerbation risks due to the environmental
and meteorological data, the system can alert the patient and caregivers of
the imminent risks and preempt medical interventions to potentially reduce
hospitalization costs.

###  III-C Environmental Module

While the detection module presented in the previous section tracks real-time
cough frequency for patient-specific analysis, the environmental module offers
local area-wide environmental and meteorological factor analysis. By examining
certain environmental indicators, a patient’s increase of COPD exacerbation
likelihood can be determined.

Breathing air quality is one of the most crucial factors in human health; poor
air quality can cause any person’s health to significantly deteriorate and is
an increasingly important issue following the advent of rapid
industrialization. Especially since their lungs are compromised due to
inflammation, COPD patients are extremely susceptible to exacerbations caused
by bad air quality. A seminal retrospective study analyzed hospitalization and
exacerbation rates for COPD patients as functions of the local environmental
and meteorological factors, including the concentration of fine particulate
matters (where PMx refers to particles or droplets present in the air that are
$x$ micrometers or less in width), NO2, and temperature variations [7, 23].
These medical studies established that the percentage exacerbation risk
increases are directly proportional to PM2.5 and PM10 levels, NO2
concentrations, and temperature variations. The details of the findings are
outlined in Fig. 7, with each increase/decrease of the “Rate” from “Safety
Standards” constituting an additional “Risk Increase Coefficient” for
exacerbations.

Based on the results of these retrospective medical studies, an equation has
been formulated in this project to estimate the percentage exacerbation risk
increase using the four environmental and meteorological parameters in the
patient’s location, as shown in Fig. 8. If a factor falls below the threshold
standard, its contribution to the final risk percentage is zero; otherwise, it
follows the formula outlined in Fig. 8.

In order to generate a real-time risk map that would represent the
exacerbation risk increase for an individual given the environmental factors
in the patient’s location, the environmental and climatological data measured
by sensors deployed by PurpleAir which are accessible via an open-source
database [26], and NO2 readings from the World Air Quality Index (WAQI) data
platform [27], have been incorporated into the above equation and overlaid on
the geographical map of the region. Moreover, an extrapolation method has been
developed to estimate the data at a specific location using the data from the
sensors deployed in adjacent areas. As an example, Fig. 9 shows the data map
for PM2.5, PM10, Temperature, and NO2 from over 6000 sensors in the San
Francisco Bay Area. As a spot check for the data, Fig. 10 shows the PM2.5
concentrations recorded by the PurpleAir sensors in Irvine and San Jose areas
during the first half of September, 2020. The onsets of spikes on Sept 6 and
Sept 10 correspond to the El Dorado Fire and the SCU Lightning Fire events,
respectively.

###  III-D Prediction Module

Finally, the prediction module combines the results of the respiratory sound
analysis from the detection module and the environmental and meteorological
factors analysis from the environmental module to forecast a patient’s
expected conditions.

Previously reported medical research studies have determined average cough
frequencies for COPD-affected smokers, affected ex-smokers, healthy smokers,
and healthy non-smokers [24, 25]. Thus, by extrapolating the progression in
cough frequency as determined by the STAIN machine learning model and
exacerbation risk increase from environmental factors from the data trends, a
patient’s expected condition is determined.

This method is illustrated in Fig. 11. First, based on the continuous
respiratory event classifications performed by the STAIN machine learning
model within the detection module, a best-fit curve is created to determine
the patient’s cough frequency trend. Next, the future exacerbation risks are
derived based on the extrapolated cough frequency data and the increased risks
due to environmental and meteorological factors as determined by the
correlations established by the retrospective medical studies, as explained in
the previous section. If the prediction module forecasts exceeding the
threshold levels that are acceptable, the patient and caregivers would be
alerted of the imminent exacerbations for necessary early medical
interventions, thereby improving the patient’s quality of life and potentially
saving hospitalization costs.

##  IV CONCLUSION

In summary, a multi-modal technique has been developed for predicting the
exacerbation risks for respiratory diseases such as COPD, based on a new
artificial intelligence model for respiratory sound analysis and retrospective
medical studies correlating key environmental parameters to exacerbations. The
proposed solution includes a novel spatio-temporal machine learning model for
accurate real-time classification and monitoring of respiratory conditions,
tracking of local environmental and meteorological factors with commercially
deployed sensors, and forecasting the patient’s progression of conditions by
combining the trends derived from these two modules.

The proposed new spatio-temporal artificial intelligence network architecture
deeply meshes the salient structures of both convolutional and recurrent
neural networks, and as a result outperforms both traditional CNN and RNN
models, as well as the more recent CRNN models, in extracting the spatial and
temporal features that are inherent in spectrograms of respiratory sounds.
Extensive comparative tests have been performed to demonstrate that the new
model achieves better sensitivity, specificity, accuracy, and Matthews
Correlation Coefficient metrics than the traditional machine learning models.

A telehealth solution based on this work can assess the exacerbation risks and
alert patients and doctors of early medical intervention, medication, and
impending hospitalization. Thus, this technique can conveniently and cost-
effectively help minimize and mitigate the impact of respiratory
exacerbations, therefore improving patients’ quality of life and potentially
reducing hospitalization costs.

The future work will include collaboration with medical research institutions
to further validate and deploy a remote patient monitoring solution into the
real-world.

## ACKNOWLEDGMENT

I would like to express sincere gratitude to my advisor, Chris Spenner, for
providing insightful advices. I would also like to thank Kailas Vodrahalli,
Archelle Georgiou, Sridhar Nemala, and Krishna Vastare for their inputs and
guidance. Additionally, I would like to thank my parents for their continuous
support.

## References

  * [1] Forum of International Respiratory Societies (2017). The Global Impact of Respiratory Disease – Second Edition. Sheffield, European Respiratory Society.
  * [2] G. Syamlal, A. Bhattacharya, and K.E. Dodd (2020). Morbidity and Mortality Weekly Report. 69 (26), 809–814. doi: 10.15585/mmwr.mm6926a1
  * [3] N. Diab, A.S. Gershon, D.D. Sin, W.C. Tan, J. Bourbeau, L.P. Boulet, and S.D. Aaron (2018). American Journal of Respiratory and Critical Care Medicine. 198 (9), 1130-1139. doi: 10.1164/rccm.201804-0621CI
  * [4] I. Tomasic, N. Tomasic, R. Trobec, M. Krpan, and T. Kelava (2018). Medical and Biological Engineering and Computing. 56 (4), 547–569. doi: 10.1007/s11517-018-1798-z
  * [5] S.B. Bentsen, T. Rustøen, and C. Miaskowski (2012). International Journal of Chronic Obstructive Pulmonary Disease. 7, 137–143. doi: 10.2147/COPD.S28994
  * [6] T. Ho, R.P. Cusack, N. Chaudhary, I. Satia, and O.P. Kurmi (2019). Breathe (Sheffield, England). 15 (1), 24–35. doi: 10.1183/20734735.0346-2018
  * [7] J. de Miguel-Díez, J. Hernández-Vázquez, A. López-de-Andrés, A. Álvaro-Meca, V. Hernández-Barrera, and R. Jiménez-García (2019). PLOS One. 14 (5), e0217143. doi: 10.1371/journal.pone.0217143
  * [8] J. Smith and A. Woodcock (2006). International Journal of Chronic Obstructive Pulmonary Disease. 1 (3), 305–314. doi: 10.2147/copd.2006.1.3.305
  * [9] S.J. Barry, A.D. Dane, A.H. Morice, and A.D. Walmsley (2006). Cough. 2, 8. doi: 10.1186/1745-9974-2-8
  * [10] J.M. Liu, M. You, Z. Wang, G.Z. Li, X. Xu, and Z. Qiu (2015). BMC Medical Informatics and Decision Making. 15 (Suppl 4), S2. doi: 10.1186/1472-6947-15-S4-S2
  * [11] H.H. Wang, J.M. Liu, M.Y. You and G.Z. Li (2015). IEEE International Conference on Bioinformatics and Biomedicine (Washington, DC, USA). 442-445. doi: 10.1109/BIBM.2015.7359724.
  * [12] J. Amoh and K. Odame (2016). IEEE Transactions on Biomedical Circuits and Systems. 10 (5), 1003-1011. doi: 10.1109/TBCAS.2016.2598794
  * [13] T. Elfaramawy, C. Latyr Fall, M. Morissette, F. Lellouche and B. Gosselin (2017). 15th IEEE International New Circuits and Systems Conference (Strasbourg, France). pp. 197-200, doi: 10.1109/NEWCAS.2017.8010139.
  * [14] T. Drugman, J. Urbain and T. Dutoit (2011). 19th European Signal Processing Conference (Barcelona, Spain). pp. 1289-1293.
  * [15] M. Soliński, M. Lepek, and L. Koltowski (2019). Automatic cough detection based on airflow signals for portable spirometry system. arXiv:1903.03588.
  * [16] E. Çakır, G. Parascandolo, T. Heittola, H. Huttunen and T. Virtanen (2017). IEEE/ACM Transactions on Audio, Speech, and Language Processing. 25, doi: 10.1109/TASLP.2017.2690575
  * [17] J. Sang, S. Park and J. Lee (2018). 26th European Signal Processing Conference (Rome, Italy). pp. 2444-2448, doi: 10.23919/EUSIPCO.2018.8553247
  * [18] S. Deshmukh, B. Raj and R. Singh (2020). Multi-Task Learning for Interpretable Weakly Labelled Sound Event Detection. arXiv:2008.07085.
  * [19] Sorana (2020). Analytics Vidhya. https://www.analyticsvidhya.com/blog/2020/11/a-short-intuitive-explanation-of-convolutional-recurrent-neural-networks/
  * [20] K. Sanjeevan and T. Hung (2020). https://github.com/ksanjeevan/crnn-audio-classification
  * [21] Y. Lecun, L. Bottou, Y. Bengio and P. Haffner (1998). Proceedings of the IEEE. 86 (11), 2278-2324. doi: 10.1109/5.726791
  * [22] Kaggle FSD (2018. https://www.kaggle.com/c/freesound-audio-tagging
  * [23] E.J. Jo and W.J. Song (2019). Asia Pacific Allergy. 9 (2), e16. doi: 10.5415/apallergy.2019.9.e16
  * [24] The United States Environmental Protection Agency. National Ambient Air Quality Standards for Particle Pollution. https://www.epa.gov/sites/production/files/2016-04/documents/2012_aqi_factsheet.pdf
  * [25] The United States Environmental Protection Agency. National Ambient Air Quality Standards. https://www.epa.gov/criteria-air-pollutants/naaqs-table
  * [26] PurpleAir. Real-time Air Quality Monitoring. https://www2.purpleair.com/
  * [27] World Air Quality Index: Real-Time Air Pollution. https://waqi.info/