# Petal number of torus knots using superbridge indices

Hyoungjun Kim  College of General Education, Kookmin University, Seoul 02707,
Korea<EMAIL_ADDRESS>,  Sungjong No  Department of Mathematics,
Kyonggi University, Suwon 16227, Korea<EMAIL_ADDRESS>and  Hyungkee Yoo
Institute of Mathematical Sciences, Ewha Womans University, Seoul 03760, Korea
<EMAIL_ADDRESS>
###### Abstract.

A petal projection of a knot $K$ is a projection of a knot which consists of a
single multi-crossing and non-nested loops. Since a petal projection gives a
sequence of natural numbers for a given knot, the petal projection is a useful
model to study knot theory. It is known that every knot has a petal
projection. A petal number $p(K)$ is the minimum number of loops required to
represent the knot $K$ as a petal projection. In this paper, we find the
relation between a superbridge index and a petal number of an arbitrary knot.
By using this relation, we find the petal number of $T_{r,s}$ as follows;

$p(T_{r,s})=2s-1$

when $1<r<s$ and $r\equiv 1\mod s-r$. Furthermore, we also find the upper
bound of the petal number of $T_{r,s}$ as follows;

$p(T_{r,s})\leq 2s-2\Big{\lfloor}\frac{s}{r}\Big{\rfloor}+1$

when $s\equiv\pm 1\mod r$.

###### Key words and phrases:

petal projection, torus knot, grid diagram, superbridge index

###### 2020 Mathematics Subject Classification:

57K10, 57R65

The first author(Hyoungjun Kim) was supported by the National Research
Foundation of Korea (NRF) grant funded by the Korea government Ministry of
Science and ICT(NRF-2021R1C1C1012299). The second author(Sungjong No) was
supported by the National Research Foundation of Korea(NRF) grant funded by
the Korea government Ministry of Science and ICT(NRF-2020R1G1A1A01101724). The
third author(Hyungkee Yoo) was supported by the National Research Foundation
of Korea(NRF) grant funded by the Korea government Ministry of
Education(NRF-2019R1A6A1A11051177) and Ministry of Science and
ICT(NRF-2022R1A2C1003203).

##  1\. Introduction

In classical knot theory, knots have been studied with regular projection such
that every crossing consists of a single understrand and a single overstrand.
The crossing number of a knot $K$, denoted by $c(K)$, is the least number of
such crossings in any projection of $K$. Adams [1] introduced $n$-crossings,
also known as multi-crossings. An $n$-crossing is a crossing which consists of
$n$ strands such that each strand bisects the crossing. An $n$-crossing
projection is a projection such that all crossings are $n$-crossings. Figure 1
shows an example of a 5-crossing. Adams [1] showed that for every $n\geq 2$,
every knot and link has a $n$-crossing projection. The $n$-crossing number of
a knot $K$, denoted by $c_{n}(K)$, is the least number of $n$-crossings among
all $n$-crossing projections of $K$. Adams et al [3] showed that for every
knot or link $K$, there exists a positive integer $n$ such that $c_{n}(K)=1$.

Figure 1. An example of a 5-crossing

An übercrossing projection of a knot $K$ is a projection of $K$ with a single
$n$-crossing for some $n$. The übercrossing number of a knot $K$, denoted by
ü$(K)$, is the smallest $n$ for which there exists a projection with a single
$n$-crossing. An übercrossing projection may have nesting loops which contains
at least one other loop. Especially, if an übercrossing projection of a knot
$K$ has no nesting loop, then it is said to a petal projection. Figure 2 shows
an übercrossing projection and a petal projection of a trefoil knot. The petal
number of a knot $K$, denoted by $p(K)$, is the smallest number of loops among
all petal projections of $K$. Adams et al [3] showed that $\text{\"{u}}(K)$ is
strictly less than $p(K)$. A petal permutation is the permutation of $n$
integers obtained by listing the labels corresponding to the heights of the
$n$ strands such that the topmost strand is labelled 1. Since the petal
projection only consists of loops, it cannot represent many links with more
than one component. Therefore, we consider the petal projection and the petal
number only for knots. It is easy to check that the petal number of a
nontrivial knot is odd. Since a petal projection gives a sequence of natural
numbers for a given knot, the petal projection is a useful model to study knot
theory. Even-Zohar et al [8] showed that for every nontrivial knot $K$,
$p(k)\leq 2c(K)-1$. Colton et al [5] proved that if two petal permutations
$\sigma$ and $\sigma^{\prime}$ represent the same knot type, then $\sigma$ can
be transformed into $\sigma^{\prime}$ by a sequence of trivial petal
additions, trivial petal deletions, and crossing exchanges.

Figure 2. The übercrossing projection and the petal projection

Let $V$ be a standard solid torus in $S^{3}$ and $T$ be the boundary of $V$.
The meridian curve on $T$ is a curve that bounds a disc in $V$, but does not
bound a disc on $T$. A longitude curve on $T$ is a curve that intersects a
meridian exactly one. For any relatively prime integers $r$ and $s$, a torus
knot $T_{r,s}$ is a knot on $T$ that wrap around $s$ times in the meridian
direction and $r$ times in the longitude direction.

In this paper, we deal with petal numbers of torus knots. Since $T_{-r,s}$ is
a mirror image of $T_{r,s}$, $p(T_{-r,s})=p(T_{r,s})$. Thus, without loss of
generality, we assume that $r$ and $s$ are positive. We find the petal number
of a particular torus knot as in the following theorem.

######  Theorem 1.

Let $r$ and $s$ be relatively prime integers with $1<r<s$. If $r\equiv
1\mod(s-r)$, then

$p(T_{r,s})=2s-1.$

Note that Lee and Jin [10] independently showed that $p(T_{r,r+2})=2r+3$ for
odd integer $r$ greater than or equal to 3. To prove Theorem 1, we use the
relation between a superbridge index and a petal number of a knot.

######  Theorem 2.

For any nontrivial knot $K$,

$2sb(K)\leq p(K)+1.$

Adams et al [3] showed that

$p(T_{r,s})\leq\begin{cases}2s-1&\text{for}\quad s\equiv 1\mod r,\\\
2s+3&\text{for}\quad s\equiv-1\mod r.\end{cases}$

We develop this result to the following theorem by using integral surgeries on
curves around $T_{r,r+1}$.

######  Theorem 3.

Let $r$ and $s$ be positive integers with $s\equiv\pm 1\mod r$. Then

$p(T_{r,s})\leq 2s-2\Big{\lfloor}\frac{s}{r}\Big{\rfloor}+1.$

The rest of this paper is organized as follows. In Section 2, we introduce a
petal grid diagram and superbridge index. Also we prove Theorem 2. In Section
3 and 4, we prove Theorem 1 and 3, respectively. In Section 5, we mention the
relation between arc index and petal number for some torus knot.

##  2\. Petal grid diagrams and superbridge indices

In this section, we introduce some useful invariants of a knot, and give a
relation between them.

###  2.1. Petal grid diagrams

A grid diagram is a knot diagram consisting of a finite number of vertical
line segments and the same number of horizontal lines such that every vertical
line segment crosses over every horizontal line segment as drawn in Figure 3
(a). Cromwell [6] showed that every knot has a grid diagram. The grid index
$g(K)$ of a knot $K$ means the minimum number of vertical line segments (or
horizontal line segments) among all grid diagrams which represent the knot
type $K$.

An arc presentation of a knot $K$ is an embedding of $K$ into $R^{3}$ which is
contained in a finite number of half-planes centered on a binding axis such
that each half-plane intersects $K$ at exactly one arc as drawn in Figure 3
(c). Birman and Menasco [4] introduced the concept of arc presentation. The
arc index $\alpha(K)$ of a knot $K$ means the minimum number of arcs among all
arc presentations which represent the knot type $K$. A grid diagram can be
expressed as an arc representation and vice versa as drawn in Figure 3 (b). In
detail, each horizontal line segment in Figure 3 (a) corresponds to the pair
of connected blue line segments in Figure 3 (b). Thus the grid index is equal
to the arc index for same knot type, i.e. $g(K)=\alpha(K)$.

Figure 3. A grid diagram and an arc presentation

To explain the relationship between the arc index and the petal number, we
transform an arc presentation as following. First consider an arc presentation
using a binding circle which is obtained from a binding axis by connecting at
the infinity point. Replace each arc by a straight chord, and label every
endpoint of straight chords in clockwise order as drawn in Figure 4 (a).
During this process, the order of each arc is not changed. A petal arc
presentation is an arc presentation with $2n+1$ arcs such that every arc
connects two endpoints $i$ and $i+n\mod(2n+1)$. Since every petal projection
can be transformed to a petal arc presentation as drawn in Figure 4, a petal
number is greater than or equal to an arc index for the same knot type, i.e.
$p(K)\geq\alpha(K)$.

Similar to a petal arc presentation, a petal grid diagram is a grid diagram
with $2n+1$ vertical line segments such that every vertical line segments has
length $n$ or $n+1$. In here, each vertical line segment connects the $i$-th
and $(i+n)$-th horizontal line segments with respect to modulo $(2n+1)$. This
means that a petal grid diagram corresponds to a petal arc presentation. Thus
it can be expressed as a petal projection.

Figure 4. A petal projection and corresponding arc presentation

###  2.2. Superbridge indices

In [9], Kuiper introduces a useful knot invariant, the superbridge index. We
interpret a knot $K$ as an polygonal or smooth embedding
$K:S^{1}\hookrightarrow\mathbb{R}^{3}$. Let
$v:\mathbb{R}^{3}\rightarrow\mathbb{R}$ be a linear function with norm
$\|v\|=1$ and let $\vec{v}$ be the corresponding unit vector. Let
$b_{\vec{v}}(K)$ be the number of local maxima of the restriction to $K$,
$v\circ K$. Then the superbridge index $sb(K)$ is defined by

$sb(K)=\min_{K^{\prime}\in[K]}\max_{\vec{v}\in S^{2}}b_{\vec{v}}(K^{\prime})$

where $[K]$ is an ambient isotopy class of a knot $K$. We remark that the
bridge index $b(K)$ of $K$ is defined by

$b(K)=\min_{K^{\prime}\in[K]}\min_{\vec{v}\in S^{2}}b_{\vec{v}}(K^{\prime}).$

Adams et al [2] showed that $2b(K)\leq\text{\"{u}}(K)$. For any nontrivial
knot $K$, $\text{\"{u}}(K)<p(K)$. So we obtain that $2b(K)\leq p(K)-1$. Kuiper
showed that $b(K)$ is strictly less than $sb(K)$. Thus we might guess that an
upper bound for $sb(K)$ is at least $\frac{p(K)+1}{2}$.

######  Theorem 2.

For any nontrivial knot $K$,

$2sb(K)\leq p(K)+1.$

###### Proof.

A knot $K$ has a polygonal petal projection on the $xy$-plane as drawn in
Figure 5 (a). From this projection, we construct a polygonal knot of $K$.
First, we put a $2p(K)$-prism with $p(K)$ layers in $\mathbb{R}^{3}$ such that
both centers of the top and bottom sides of the prism lie on the $z$-axis.
Label each layer as $1,2,\dots,p(K)$ in the order from top to bottom. There
are $p(K)$ line segments which are passing through the origin in Figure 5 (a).
Place these line segments passing through the $z$-axis, say horizontal line
segments, on the suitable layers of the prism. In detail, the assigned number
of a horizontal line segment must be the same as the assigned number of the
layer on which it is placed. Since we start to construct a polygonal knot from
the polygonal petal projection, each of the remaining line segments is
contained in one rectangular side of the prism. So we obtain the polygonal
knot of $K$ as drawn in Figure 5 (b).

Figure 5. The superbridge position of a petal diagram

Now we assume that $\vec{v}$ is parallel to the $z$-axis. Then local maxima
and minima can only appear at horizontal line segments. Note that there are
the same number of local maxima and minima. Since $p(K)$ is odd, there are at
most $\frac{p(K)-1}{2}$ local maxima in this case.

It remains to consider the case that $\vec{v}$ is not parallel to the
$z$-axis. Let $H$ be a plane spanned by the two vectors $\vec{v}$ and
$\vec{z}=(0,0,1)$. Then $H$ is divided into two regions $H_{1}$ and $H_{2}$ by
the $z$-axis. Now we have a projection of the polygonal knot of $K$ to $H$.

Then there are two cases according to the existence of a horizontal line
segment perpendicular to $H$. The case that there is no horizontal line
segment perpendicular to $H$ is described in Figure 5 (c), and the other case
is described in Figure 5 (d). To avoid confusion in the notation, images of
line segments which are not horizontal segments are called vertical sticks.
Then there are exactly $p(K)$ vertical sticks. Note that each vertical stick
has at most one local maximum for any direction. Without loss of generality,
we assume that the terminal point of $\vec{v}$ is contained in $H_{1}$. Then
every vertical stick contained in $H_{2}$ which does not touch the $z$-axis
cannot have a local maximum in $\vec{v}$ direction. In the case of Figure 5
(c), there are $\frac{p(K)-1}{2}$ vertical sticks which are contained
completely in $H_{2}$. This implies that the number of local maxima is at most
$\frac{p(K)+1}{2}$. In the case of Figure 5 (d), there are two cases. The
number of vertical sticks contained in $H_{2}$ which do not touch the $z$-axis
is either $\frac{p(K)-1}{2}$ or $\frac{p(K)-1}{2}-1$. In the first case, two
vertical sticks which are connected by one point on the $z$-axis are contained
in $H_{1}$. So there are at most $\frac{p(K)+1}{2}$ local maxima. In the
second case, two vertical sticks which are connected by one point in the
$z$-axis are contained in $H_{2}$. Even though interiors of these vertical
sticks are contained in $H_{2}$, there is at most one local maximum in these
sticks. Thus there are at most $\frac{p(K)+1}{2}$ local maxima in both cases.
Therefore there are at most $\frac{p(K)+1}{2}$ local maxima for any vector
$\vec{v}$. This implies that $2sb(K)\leq p(K)+1$. ∎

##  3\. Proof of Theorem 1

First we consider a torus knot $T_{r,s}$ with $1<r<s<2r$. The braid word of
$T_{r,s}$ is of the form
$(\sigma_{1}\sigma_{2}...\sigma_{r-1})^{s}=(\sigma_{1}\sigma_{2}...\sigma_{r-1})^{s-r}(\sigma_{1}\sigma_{2}...\sigma_{r-1})^{r}$
as drawn in Figure 6 (a). By the braid relations, the first $s-r$ strings are
fully twisted, and the other $2r-s$ strings are untwisted as drawn in Figure 6
(b). A full twist of $s-r$ strings can be represented by kinked $s-r-1$
strings wrapping around a string as drawn in Figure 6 (c).

Figure 6. The braid of torus knot $T_{r,s}$

We note Kuiper’s result [9], which says that $sb(T_{r,s})=\min\\{2r,s\\}$ for
$1<r<s$. Now we recall Theorem 1.

######  Theorem 1.

Let $r$ and $s$ be relatively prime integers with $1<r<s$. If $r\equiv
1\mod(s-r)$, then

$p(T_{r,s})=2s-1.$

Figure 7. The grid diagram for torus knot $T_{r,s}$ with $1<r<s<2r$

###### Proof.

Let $r$, $s$ be relatively prime integers with $1<r<s$ and $r\equiv
1\mod(s-r)$. This implies that $1<r<s<2r$. By Theorem 2 and Kuiper’s result,
$2s-1\leq p(T_{r,s})$. So it is sufficient to show that $p(T_{r,s})\leq 2s-1$.

Take a grid diagram $D$ of $T_{r,s}$ which has $2s-1$ vertical and horizontal
line segments each as drawn in Figure 7. Label these vertical line segments as
$v_{1},v_{2},\dots,v_{2s-1}$ in order from left to right. Similarly, label
these horizontal line segments as $h_{1},h_{2},\dots,h_{2s-1}$ in order from
top to bottom. Let $D_{V}$ and $D_{H}$ be the sets of all vertical and
horizontal line segments of $D$, respectively.

Now we define a height function $f$ on $D_{V}\cup D_{H}$. For any vertical
line segments $v_{i}$, we define a height function on $D_{V}$ as follow:

$f(v_{i})=\begin{cases}i&\text{if}\quad 1\leq i<s,\\\ 2s-i&\text{if}\quad
s\leq i\leq 2s-1\end{cases}\quad\text{for}\quad v_{i}\in D_{V}.$

Since a horizontal line segment $h_{k}$ connects two vertical segments $v_{i}$
and $v_{j}$ for $1\leq i<s$ and $s\leq j\leq 2s-1$, we define a height
function on $D_{H}$ as follow:

$f(h_{k})=\begin{cases}2s-j&\text{if}\quad 1\leq k<s,\\\ i&\text{if}\quad
s\leq k\leq 2s-1\end{cases}\quad\text{for}\quad h_{k}\in D_{H}.$

Then the vertical and horizontal line segments that intersect at the north-
east corner have the same height except the corner connected to $v_{s}$ and
$h_{s}$. Also, the vertical and horizontal line segments that intersect at the
south-west corner have the same height.

We remark that $1<r<s$ and $r\equiv 1\mod(s-r)$. Assume that $m=s-r$ and
$n=\lfloor\frac{s}{m}\rfloor$. Choose $2n+1$ pairs of line segments $v_{s\pm
lm}$ and $h_{s\pm lm}$ for $l=\\{0,\dots,n\\}$. Connect these line segments,
then we obtain a path

$v_{s-nm}h_{s-nm}v_{s+m}h_{s+m}v_{s-(n-1)m}h_{s-(n-1)m}\cdots
v_{s+nm}h_{s+nm}v_{s}h_{s}$

as a red spiral in Figure 8 (a). Rearrange $nm$ horizontal line segments
$h_{i}$ for $i<s$ as in the order of

$m$, |  $2m$, | …, |  $nm$,
---|---|---|---
$m-1$, |  $2m-1$, | …, | $nm-1$
|  | ⋮ |
$2$, |  $m+2$, | …, |  $(n-1)m+2$,
$1$, |  $m+1$, | …, |  $(n-1)m+1$.

Also rearrange $nm$ horizontal line segments $h_{i}$ for $i>s$ as in the order
of

|  |  $s+2m-1$, | …, |  $s+nm-1$,
---|---|---|---|---
|  $s+m-2$, |  $s+2m-2$, | …, |  $s+nm-2$,
|  |  ⋮ |  |
|  $s+2$, |  $s+m+2$, | …, |  $s+(n-1)m+2$,
|  $s+1$, |  $s+m+1$, | …, |  $s+(n-1)m+1$,
$s+m-1$, |  $s+m$, |  $s+2m$, | …, |  $s+nm$.

During this process, there can exist a horizontal line segment $h_{i}$ for
$i>s$ which cannot move its location changed preserving the knot type when
there is a horizontal line segment $h_{k}$ such that $k>i$ and $f(h_{k})\leq
f(v_{i})$. In this case, we divide $v_{j}$ as a two parts $v_{j}^{u}$ and
$v_{j}^{l}$ such that $f(v_{j}^{u})=f(v_{j})$ and $f(v_{j}^{l})=f(h_{i})$.
Then we obtain a diagram as drawn in Figure 8 (b).

Finally, we adjust vertical line segments of this diagram to construct a petal
grid diagram. Rearrange $nm$ vertical line segments $v_{i}$ for $i<s$ as in
the order of

|  $m$, |  $2m$, | …, |  $(n-1)m$,
---|---|---|---|---
|  $m-1$, |  $2m-1$, | …, |  $(n-1)m-1$,
|  |  ⋮ |  |
|  $2$, |  $m+2$, | …, |  $(n-2)m+2$,
|  $1$, |  $m+1$, | …, |  $(n-2)m+1$,
$s-m$, |  $s-m+1$, |  $s-m+2$, | …, |  $s-2$, $s-1$.

Also rearrange $nm$ vertical line segments $v_{i}$ for $i>s$ as in the order
of

|  |  $s+2m-1$, | …, |  $s+nm-1$,
---|---|---|---|---
|  $s+m-2$, |  $s+2m-2$, | …, |  $s+nm-2$,
|  |  ⋮ |  |
|  $s+2$, |  $s+m+2$, | …, |  $s+(n-1)m+2$,
|  $s+1$, |  $s+m+1$, | …, |  $s+(n-1)m+1$,
$s+m-1$, |  $s+m$, |  $s+2m$, | …, |  $s+nm$.

Thus we obtain a petal grid diagram of $T_{r,s}$ with $2s-1$ vertical line
segments as drawn in Figure 8 (c). Therefore $p(T_{r,s})\leq 2s-1$. ∎

Figure 8. The petal grid diagram from the grid diagram for torus knot
$T_{r,s}$ with $1<r<s$ and $r\equiv 1\mod s-r$

##  4\. Integral surgeries and proof of Theorem 3

Let $V$ be a standard solid torus in $S^{3}$ and let $T$ be the boundary of
$V$. We assume that $l$ is the core of $V$, and $m$ is the curve outside $V$
obtained by a small perturbation of the meridian curve of $T$. Consider the
torus knot $T_{r,r+1}$ on $T$ for any positive integer $r$. If $n$ is an
integer with $n\geq 2$, then $T_{r,nr+1}$ is obtained by a
$\frac{1}{n-1}$-surgery on $m$. Remark that the surgery does not change
$S^{3}$. Similarly, $T_{nr-1,r}$ is obtained from $T_{r-1,r}$ after a
$\frac{1}{n-1}$-surgery on $l$. We can put a torus knot $T_{r,r+1}$ in a solid
torus $V$ as drawn in Figure 9 (a). Then curves $l$ and $m$ are represented as
drawn in Figure 9 (b).

Figure 9. The curves $m$ and $l$ for surgeries

Using these facts, we prove the following two lemmas.

######  Lemma 4.

For any positive integers $r$ and $n$,

$p(T_{r,nr+1})\leq 2(r-1)n+3.$

###### Proof.

We first remark that $T_{r,nr+1}$ is obtained from $T_{r,r+1}$ by using a
$\frac{1}{n-1}$-surgery on $m$. Take a grid diagram of $T_{r,r+1}$ as drawn in
Figure 10 (a). In this diagram, there are $r$ strands which intersect right
endpoints of the horizontal lines and the bottom endpoints of the vertical
lines (arcs with south-east corners in the figure). Take a closed curve which
wraps around these $r$ strands. Then this curve corresponds to $m$ for
$T_{r,r+1}$. Now, we use a $\frac{1}{n-1}$-surgery on $m$ at the grid diagram
of $T_{r,r+1}$ to obtain $T_{r,nr+1}$. Then we obtain a grid diagram with
$r-1$ spirals as drawn in Figure 10 (b). Finally we obtain the petal grid
diagram by adjusting the height of horizontal line segments as drawn in Figure
10 (c). During this process, an additional $2(n-1)$ vertical segments are
needed for each of $r-1$ strands. Thus we have the result as follows;

$p(T_{r,nr+1})\leq 2r+1+2(r-1)(n-1)=2(r-1)n+3.$

∎

Figure 10. The petal grid diagram of $p(T_{r,nr+1})$

######  Lemma 5.

For any positive integers $r$ and $n$,

$p(T_{r,nr-1})\leq 2(r-1)n+1.$

###### Proof.

The process of this proof is similar to the proof of Lemma 4. Since
$T_{r,nr-1}$ and $T_{nr-1,r}$ have the same knot type, we regard that
$T_{r,nr-1}$ is obtained from $T_{r-1,r}$ by using a $\frac{1}{n-1}$-surgery
on $l$. Take a grid diagram of $T_{r-1,r}$ as drawn in Figure 11 (a). In this
diagram, there are $r+1$ strands which intersect right endpoints of the
horizontal lines and the top endpoints of the vertical lines (arcs with north-
east corners in the figure). Take a closed curve which wraps around these
$r+1$ strands. Then this curve corresponds to $l$ for $T_{r-1,r}$. Now, we use
a $\frac{1}{n-1}$-surgery on $l$ at the grid diagram of $T_{r-1,r}$ to obtain
$T_{r,nr-1}$. Then we obtain a grid diagram with $r-1$ spirals as drawn in
Figure 11 (b). Finally we obtain the petal grid diagram by adjusting the
height of horizontal line segments as drawn in Figure 11 (c). During this
process, an additional $2(n-1)$ vertical segments are needed for each of $r-1$
strands. Thus we have the result as follows;

$p(T_{r,nr-1})\leq 2r-1+2(r-1)(n-1)=2(r-1)n+1.$

∎

Figure 11. The petal grid diagram of $p(T_{r,nr-1})$

Now we recall Theorem 3.

######  Theorem 3.

Let $r$ and $s$ be positive integers with $s\equiv\pm 1\mod r$. Then

$p(T_{r,s})\leq 2s-2\Big{\lfloor}\frac{s}{r}\Big{\rfloor}+1.$

###### Proof.

First assume that $s\equiv 1\mod r$. Then $s=\lfloor\frac{s}{r}\rfloor r+1$.
By Lemma 4,

$\displaystyle p(T_{r,s})\ =\ p(T_{r,\lfloor\frac{s}{r}\rfloor r+1})\ $
$\displaystyle\leq\ 2(r-1)\Big{\lfloor}\frac{s}{r}\Big{\rfloor}+3$
$\displaystyle=\
2\left(r\Big{\lfloor}\frac{s}{r}\Big{\rfloor}+1\right)-2\Big{\lfloor}\frac{s}{r}\Big{\rfloor}+1$
$\displaystyle=\ 2s-2\Big{\lfloor}\frac{s}{r}\Big{\rfloor}+1.$

It remains to consider the case when $s\equiv-1\mod r$. Then
$s=(\lfloor\frac{s}{r}\rfloor+1)r-1$. By Lemma 5,

$\displaystyle p(T_{r,s})\ =\ p(T_{r,(\lfloor\frac{s}{r}\rfloor+1)r-1})\ $
$\displaystyle\leq\
2\left(r-1\right)\left(\Big{\lfloor}\frac{s}{r}\Big{\rfloor}+1\right)+1$
$\displaystyle=\
2r\left(\Big{\lfloor}\frac{s}{r}\Big{\rfloor}+1\right)-2-2\Big{\lfloor}\frac{s}{r}\Big{\rfloor}+1$
$\displaystyle=\ 2s-2\Big{\lfloor}\frac{s}{r}\Big{\rfloor}+1.$

∎

##  5\. Concluding remark

In this section, we discuss the relation between $\alpha(K)$ and $p(K)$. Let
$r$ and $s$ be relatively prime integers with $1<r<s$. Etnyre and Honda [7]
showed that an arc index of a torus knot $T_{r,s}$ is equal to $r+s$.

First consider the case that $r\equiv 1\mod(s-r)$. Then for a torus knot
$T_{r,s}$, $\alpha(T_{r,s})=r+s$ and $p(T_{r,s})=2s-1$ by Theorem 1.
Therefore, there is a difference of as much as $s-r-1$ between the petal
number and the arc index for a torus knot $T_{r,s}$. This means that there is
no gap between the arc index and the petal number for a torus knot $T_{r,s}$
when $r$ is equal to $s-1$. Even though there is a gap of 1 between the arc
index and the petal number for a torus knot $T_{r,s}$ when $r$ is equal to
$s-2$, it is meaningless. Because the gap comes from the fact that a petal
number is odd for every knot.

Now consider the case that $r=2$. Then for a torus knot $T_{2,s}$,

$p(T_{2,s})\leq 2s-2\Big{\lfloor}\frac{s}{2}\Big{\rfloor}+1=2s-(s-1)+1=s+2$

by Theorem 3. Since an arc index of $T_{2,s}$ is equal to $2+s$, the petal
number of $T_{2,s}$ is also equal to $2+s$. We remark that $T_{2,s}$ is a
2-bridge knot. Adams et al [3] showed that a petal number of a 2-bridge knot
$K$ with an odd number of crossings is equal to $c(K)+2$. From this result, we
also have the same result of petal number of $T_{2,s}$ as obtained from
Theorem 3, since a torus knot $T_{2,s}$ is 2-bridge knot with $s$ crossings.
Furthermore if $r$ and $s$ satisfy the condition of Theorem 1, then
$\lfloor\frac{s}{r}\rfloor=1$. This means that

$p(T_{r,s})=2s-1=2s-2\Big{\lfloor}\frac{s}{r}\Big{\rfloor}+1.$

So we guess that Theorem 3 holds for every torus knot $T_{r,s}$. Thus we
suggest the following conjecture.

######  Conjecture.

Let $r$ and $s$ be relatively prime integers with $1<r<s$. Then

$p(T_{r,s})\leq 2s-2\Big{\lfloor}\frac{s}{r}\Big{\rfloor}+1.$

## References

  * [1] C. Adams, Triple crossing number of knots and links, Journal of Knot Theory and its Ramifications, 22 (2012).
  * [2] C. Adams, O. Capovilla-Searle, J. Freeman, D. Irvine, S. Petti, D. Vitek, A. Weber, and S. Zhang, Bounds on übercrossing and petal numbers for knots, Journal of Knot Theory and Its Ramifications, 24 (2015), p. 1550012.
  * [3] C. Adams, T. Crawford, B. DeMeo, M. Landry, A. Lin, M. Montee, S. Park, S. Venkatesh, and F. Yhee, Knot projections with a single multi-crossing, Journal of Knot Theory and Its Ramifications, 24 (2012).
  * [4] J. Birman and W. Menasco, Special positions for essential tori in link complements, Topology, 33 (1994), pp. 525–556.
  * [5] L. Colton, C. Glover, M. Hughes, and S. Sandberg, A reidemeister type theorem for petal diagrams of knots, Topology and its Applications, 267 (2019), p. 106896.
  * [6] P. Cromwell, Embedding knots and links in an open book i: basic properties, Topology and its Applications, 64 (1995), pp. 37–58.
  * [7] J. Etnyre and K. Honda, Knots and contact geometry i: Torus knots and the figure eight knot, Journal of Symplectic Geometry, 1 (2001).
  * [8] C. Even-Zohar, J. Hass, N. Linial, and T. Nowik, The distribution of knots in the petaluma model, Algebraic & Geometric Topology, 18 (2017).
  * [9] N. H. Kuiper, A new knot invariant., Mathematische Annalen, 278 (1987), pp. 193–210.
  * [10] H. J. Lee and G. T. Jin, Petal number of torus knots of type $(r,r+2)$, arXiv preprint arXiv:2112.13211, (2021).


11institutetext: Seoul National University 11email:
<EMAIL_ADDRESS>
# Harmonizing Visual and Textual Embeddings
for Zero-Shot Text-to-Image Customization

Yeji Song 11 Jimyeong Kim∗ 11 Wonhark Park∗ 11 Wonsik Shin 11
Wonjong Rhee 11 Nojun Kwak† 11

###### Abstract

In a surge of text-to-image (T2I) models and their customization methods that
generate new images of a user-provided subject, current works focus on
alleviating the costs incurred by a lengthy per-subject optimization. These
zero-shot customization methods encode the image of a specified subject into a
visual embedding which is then utilized alongside the textual embedding for
diffusion guidance. The visual embedding incorporates intrinsic information
about the subject, while the textual embedding provides a new, transient
context. However, the existing methods often 1) are significantly affected by
the input images, _e.g_., generating images with the same pose, and 2) exhibit
deterioration in the subject’s identity. We first pin down the problem and
show that redundant pose information in the visual embedding interferes with
the textual embedding containing the desired pose information. To address this
issue, we propose orthogonal visual embedding which effectively harmonizes
with the given textual embedding. We also adopt the visual-only embedding and
inject the subject’s clear features utilizing a self-attention swap. Our
results demonstrate the effectiveness and robustness of our method, which
offers highly flexible zero-shot generation while effectively maintaining the
subject’s identity. The code will be publicly released.

###### Keywords:

T2I model Zero-shot Customization Orthogonal Visual Embedding Self-attention
Swap

††footnotetext:  $\ast$ Equal contribution $\dagger$ Corresponding author

##  1 Introduction

Recent advancements in text-to-image (T2I) generation, especially diffusion
models [29, 4, 31, 28, 24] have opened up a new era of image creation.
Subject-driven generation [30, 8, 35, 26, 38] aims to generate novel images
featuring a specific subject provided by the user. The common approach
represents the subject as a new pseudo-word ($\text{S}^{*}$) in the textual
embedding space of the text encoder. They optimize a pseudo-word by updating
the textual embedding of the pseudo-word [9] or the diffusion model’s
parameters [30, 36]. However, both approaches require per-subject
optimization, posing challenges for real-time applications and multi-subject
customization. In response to this challenge, zero-shot customization methods
[17, 39, 6, 32, 21, 12, 42, 40] have been proposed.

They adopt the mappers (_e.g_., MLP network [39, 42, 40], adapter [6, 32, 21]
or multi-modal encoder [17]) to transform a subject’s image into the visual
embedding, which is subsequently utilized along with the textual embedding.
The visual embedding provides representative information about the subject’s
identity from the input image, eliminating the need for additional training
processes. To improve the user’s convenience, most zero-shot customization
methods [17, 39, 6, 21, 12, 42, 40] encode a single image as the input. In
line with this practice, we focus on single-image-based zero-shot
customization in this paper.

Figure 1: Our method deals with the challenges of pose variation in zero-shot
customization methods _i.e_., (i) strong adherence to the pose in the input
image and (ii) loss in the subject’s identity. Our method paves the way for a
more diverse, efficient, and lively subject-driven generation.

When dealing with a single image, the existing methods focus on separating the
subject’s identity from other irrelevant details within an image, as the input
image typically encompasses more than just the subject’s identity. To address
this challenge, they employ the subject’s segmentation mask [17, 39, 21, 12,
40] or the visual embedding from the deepest layers of an encoder [39].
However, while those techniques effectively separate the subject from the
background or other objects within the image, disentangling the subject’s pose
from its identity is almost impossible as these aspects are tightly
intertwined within the same pixels. We term this inevitable phenomenon as the
pose-identity entanglement within visual embedding, which becomes apparent
when attempting to change the pose of the subject while preserving the
subject’s identity. Regarding this, we have identified two significant
problems when modifying the subject’s pose with a text prompt, namely the (i)
pose bias and (ii) identity loss:

  1. (i)

The generated images tend to either maintain the original pose of the subject
or exhibit some unnaturalness that falls between the pose derived from the
text prompt and the one presented in the input image.

  2. (ii)

The subject in the generated images partially loses its identity, appearing
with a different color or body shape.

In this paper, we focus on resolving the pose bias and identity loss described
above. They are imperative tasks for advancing towards a more diverse and
desirable customization, while highly challenging due to the pose-identity
entanglement. We start from unveiling the more direct and addressable cause of
the two problems. We found that using only the textual embedding generates the
subject in various poses faithfully complying with the text prompt (without
pose bias) while using only the visual embedding results in the images where
the subject retains its whole identity (without identity loss). This leads us
to assume that a conflict exists between the visual and textual embeddings,
from which the two problems mainly originate. We provide the result images in
the supplementary materials. Regarding pose-identity entanglement, the input
image carries the visual features of the subject in a specific pose.
Therefore, when generating a novel image of the subject in a different pose,
the input image and text prompt provide two concurrent but conflicting
embeddings to the model with different pose information. As a result, the
visual embedding readily interferes with the textual embedding, causing the
pose bias. Conversely, textual embedding can also interfere with visual
embedding, affecting the subject’s identity and resulting in identity loss.

To alleviate this conflict, we propose contextual embedding orchestration and
self-attention swap that effectively resolve each problem. The former involves
adjusting the visual embedding to align better with the textual embedding.
This adjustment is achieved by orthogonalizing the visual embedding vector to
the subspace of the textual embedding vectors, essentially reducing the
interference of the visual embedding. The latter adopts another denoising
process guided by the visual-only embedding that fundamentally evades the
collision between the visual and textual embedding and aggregates clean
information about the subject. By swapping the self-attention between the
original and the visual-only-guided diffusion models, we could successfully
maintain the subject’s identity in newly generated images. Our method is
generic and easily applicable to any zero-shot customization method that
utilizes visual and textual embeddings, as it does not require an additional
tuning process. We demonstrate that our method significantly improves the pose
variation of the baseline without compromising its performance in pose-
irrelevant scenarios, _e.g_., placement of novel objects or changing the
subject’s texture. Fig. 1 illustrates that our method effectively resolves the
pose bias. We also provide the results regarding identity loss in Figs. 4 and
5, and supplementary materials.

Our contributions are summarized as:

  * $\bullet$

We unveil the problem of pose bias and identity loss in zero-shot
customization and shed light on the discord among the visual and textual
embeddings.

  * $\bullet$

Our proposed method effectively resolves the pose bias and identity loss,
offering highly diverse and pose-variant subject generation.

  * $\bullet$

Our method is readily applicable to any zero-shot customization methods that
incorporate visual and textual embeddings.

##  2 Related Works

Text-to-Image Generation. Amidst a proliferation of image synthesis models,
diffusion models [34, 11] have demonstrated their strength in producing images
with remarkable fidelity and comprehensive mode coverage. Their capacity to
generate a spectrum of diverse images has facilitated their integration with
large pre-trained language models [27]. This synergy has given rise to
diffusion-based T2I models [29, 4, 31, 28, 24], which can generate high-
quality images with strong controllability by the guidance of natural language
instructions. Recently, to increase the flexibility using this strong prior,
many have combined conditioning embeddings from different modalities, _e.g_.,
concatenating text-aligned visual embedding extracted from an image with
textual embedding. However, simply combining various embeddings may cause
conflict in dealing with different information they have. Hence, our primary
goal is to devise an appropriate methodology for integrating diverse
embeddings from different modalities, considering potential discrepancies in
their inherent information.

Subject-driven Generation. Given a few images of a user-provided subject,
subject-driven generation [30, 8, 35, 26, 38] aims to generate images
containing the subject in various contexts instructed by text guidance.
However, per-subject optimization suffers from computation and memory burden,
leading to an introduction of zero-shot customization methods [17, 39, 6, 32,
21, 12, 42]. They involve pre-training the mapper enabling the transformation
of the input image into text-aligned visual embeddings without per-subject
optimization. The visual embedding is then concatenated with textual embedding
and utilized as guidance to the T2I models. However, we found that they often
struggle with the interference between the visual and textual embeddings,
often failing to properly deliver text prompts. Our proposed method first pins
down the interference between the two embeddings and provides a solution for
it, enabling more diverse and more flexible image generations.

Meanwhile, to edit the context while maintaining the subject and the
background, consistent image editing methods [5, 14] have been proposed. While
the general image editing methods [10, 24, 25, 41, 3, 7, 22, 37, 43, 23]
change an image style, surrounding objects, a subject’s appearance, or a
subject per se, consistent methods target to change the subject’s pose or
viewpoint while preserving the subject’s identity. Compared to customization
methods that deal with noise-to-image generations, those methods focus on
image-to-image translation. They edit the subject without changing the other
context, _e.g_., maintaining the input image’s structure, background, and
other surrounding objects. On the other hand, customization methods aim to
synthesize the subject in diverse context according to the provided text
prompt. Our method concentrates on customization while incorporating the image
editing capability to address issues stemming from zero-shot customization.

Compositional Generation. Due to the limited size of the textual embeddings,
large pre-trained T2I diffusion models suffer from fully compositing complex
text descriptions [20]. Precedent research tackles this problem and offers
various solutions; _e.g_., giving an additional segment mask layout for each
related prompt as a condition [15], composing separate diffusion models where
each is encoded with a divided prompt [20], and using the perpendicular
gradient as a negative prompt guidance [1]. Likewise, we deal with complex
prompts consisting of visual and textual embeddings and suggest how to convey
information without interference between them.

##  3 Preliminaries

In this work, we employ text-to-image latent diffusion model (LDM) [29]. The
denoising process is implemented in the latent space using the autoencoder
structure with the encoder $\mathcal{E}(\cdot)$ and the decoder
$\mathcal{D}(\cdot)$. Specifically, an image $x$ is projected to a latent
representation $z=\mathcal{E}(x)$, and decoded back to the image space giving
$\tilde{x}=\mathcal{D}(z)=\mathcal{D}(\mathcal{E}(x))$, reconstructing $x$,
_i.e_., $\tilde{x}\approx x$. Given the pre-trained autoencoder, the latent
diffusion model $\epsilon_{\theta}(x_{\tau},\tau);\tau=1\cdots T$ is trained
with the following objective where c represents the contextual embedding of
textual/visual condition generally obtained from the pre-trained CLIP encoder
[27]:

$L_{LDM}=\mathbb{E}_{x\sim
p(x),\epsilon\sim\mathcal{N}(0,I),\textbf{c},\tau\sim\text{uniform}(1,T)}[\|\epsilon-\epsilon_{\theta}(z_{\tau}(x),\tau,\textbf{c})\|^{2}_{2}].$
(1)

During inference, $z_{T}\sim\mathcal{N}(0,I)$ is iteratively denoised to the
initial representation $z_{0}(x)=\mathcal{E}(x)$.

Text prompts act as a condition on LDM through the cross-attention mechanism.
The latent spatial feature $f\in\mathbb{R}^{l\times h}$ is projected to
produce the query $Q=f\cdot W_{Q}\in\mathbb{R}^{l\times d}$ while the text
prompts are first encoded into the text embedding
$\textbf{c}\in\mathbb{R}^{l_{c}\times h_{c}}$ and projected to yield the key,
$K=\textbf{c}\cdot W_{K}\in\mathbb{R}^{l_{c}\times d}$, and value,
$V=\textbf{c}\cdot W_{V}\in\mathbb{R}^{l_{c}\times d}$. Then the cross-
attention is calculated as follows:

$\text{Attention}(Q,K,V)=\text{softmax}(QK^{T}/\sqrt{d})\cdot V$ (2)

where $l$, $l_{c}$, $d$ and $h$ are spatial sequence length, context sequence
length, dimension of key/query/value, and dimension of spatial feature/context
respectively. Self-attetntion mechanism uses the same Eq. 2 where the key and
value are attained from latent spatial features $f$ instead of the contextual
embedding c.

##  4 Methods

###  4.1 Discord among Contextual embeddings

Figure 2: Visualization of discord among contextual embeddings. (a) The
contextual embedding of zero-shot customization methods is generally composed
of visual and textual embedding, each of which provides the subject’s identity
and a novel context, respectively. (b) However, pose information in the visual
embedding interferes with the textual embedding, resulting in biased images
that are closely attached to the input image. Meanwhile, our orchestration
resolves the conflict, generating images that are both identity-conservative
and faithful to the text prompt.

Single-image-based zero-shot customization methods [17, 39, 6, 21, 12, 42, 40]
compose the contextual embedding by concatenating two heterogeneous
embeddings; visual embedding $\textbf{v}\in\mathbb{R}^{M\times h_{c}}$ and
textual embedding $\textbf{t}\in\mathbb{R}^{N\times h_{c}}$
($\textbf{c}=[\textbf{v};\textbf{t}]=[v_{1}\cdots v_{M};t_{1}\cdots t_{N}]$).
Image features of a given subject’s image are transformed into the visual
embedding using the pre-trained mapper. The visual embedding is then combined
with the textual embedding, engaging in the spatial features $z$ via cross-
attention layers as in Eq. 2 or additional adapters. We depict the overall
zero-shot pipeline in Fig. 2(a).

Meanwhile, when modifying the subject’s pose with a text prompt, the visual
embedding, which includes pose information from the input image, conflicts
with the textual embedding, rendering the images confined to the specific pose
or partially losing the subject’s identity. identity of the subject’s concept.
To verify the interference between the visual and textual embedding, we
conduct an experiment utilizing three types of contextual embedding
respectively: 1) contextual embedding composed solely of the visual embedding
($\textbf{c}_{\textbf{v},\varnothing}=[v_{1}\cdots v_{M};\varnothing]$), 2)
solely of the textual embedding
($\textbf{c}_{\varnothing,\textbf{t}}=[\varnothing;t_{1}\cdots t_{N}]$), and
3) the original combined contextual embedding ($\textbf{c}=[v_{1}\cdots
v_{M};t_{1}\cdots t_{N}]$). We generate 500 images conditioned on the
respective contextual embedding and obtain their representations from the
highest layer of VGG-16 [33] as visualized in Fig. 2(b). Ideally, the
generated images conditioned on the original contextual embedding should
include the subject’s identity and, at the same time, faithfully follow the
text prompts. However, their image features exhibit a strong bias toward
outputs from the visual-only embedding. The pre-existing pose information in
visual embedding interferes with the textual embedding, restricting the
generated images to be closely tied to the visual embedding, _i.e_.,
maintaining the pose of the subject in the input images instead of
constructing a novel pose as the textual embedding instructs. The result above
clarifies that the visual embedding interferes with the textual embedding,
confining the generated images away from text guidance. Conversely, the
textual embedding also affects the visual embedding, harming the subject’s
identity. We include the analysis regarding the identity loss in supplementary
materials. These results underline the necessity of adjusting the visual
embedding to resolve this discord.

###  4.2 Contextual embedding Orchestration

To resolve the interference between the visual embedding v and the textual
embedding t as presented in Sec. 4.1, we propose orthogonal visual embedding
$\textbf{v}^{\bot}$ as follows:

$v^{\bot}=v-v^{||}=v-\\!\sum_{j=1,j\neq\text{sbj}}^{N}\langle\bar{t}_{j},v\rangle\bar{t}_{j},\quad\bar{t}_{j}=\text{normal}\left(t_{j}-\\!\sum_{i=1,i\neq\text{sbj}}^{j-1}\langle\bar{t}_{i},t_{j}\rangle\bar{t}_{i}\right)$
(3)

where $t_{\text{sbj}}$ represents the textual embedding corresponding to the
pseudo-word (_e.g_., ‘S∗’ in Fig. 1) and $\text{normal}(\cdot)$ means $l_{2}$
normalization††We incorporate all text tokens except for $\text{S}^{*}$ and
its class name since they are closely related to pose _e.g_., when we generate
an image of “a $\text{S}^{*}$ eating a hamburger”, the token “hamburger” also
affects the eating pose and should be considered, too..
$\\{\bar{t}_{j}\\}_{j\neq\text{sbj}}$ are the basis vectors of the textual
subspace, obtained by the Gram-Schmidt orthogonalization process. Eq. 3 can be
viewed as breaking down a vector $v$ into two components, $v^{\bot}$ and
$v^{||}$, of which $v^{||}$ resides in the textual subspace which is spanned
by $\\{\bar{t}_{j}\\}_{j\neq\text{sbj}}$.

We argue that $v^{||}$ causes the interference with t when presented
concurrently. The other component, $v^{\bot}$, is perpendicular to the textual
subspace, embodying the essential information about the subject’s identity
alleviating the interference with t. Using $v^{\bot}$ instead of $v$, we could
establish the new axes in the visual embedding that interplays more
effectively with t, helping resolve the interference. The new contextual
embedding $\textbf{c}^{\bot}=[v^{\bot}_{1}\cdots v^{\bot}_{M};t_{1}\cdots
t_{N}]$ is then incorporated with latent spatial features $f$ via cross-
attention. Note that we also exclude the textual embedding of articles or the
subject’s class name from Eq. 3 when such tokens are used. Fig. 3(a)
illustrates the contextual embedding orchestration where the textual embedding
t effectively provides guidance with alleviated interference from the visual
embedding $v^{\bot}$.

###  4.3 Self-attention Swap

We found that the conflict between visual and textual embeddings results in
the loss of the subject’s identity, when the textual embedding affects the
visual embeddings. This phenomenon is still visible even after orchestration,
due to the inevitable pose-identity entanglement; changing the visual
embedding inevitably results in a change in identity. To alleviate the loss in
the subject’s identity, we propose a self-attention swap to retain essential
information about the subject. Our intuition is based on the observation that
when the visual-only embedding is used as the contextual embedding
($\textbf{c}_{\textbf{v},\varnothing}=[v_{1}\cdots v_{M};\varnothing]$), the
output images are clean, effectively preserving the subject’s identity, as it
is free of the interference of the textual embedding.

Figure 3: The overall pipeline of our method. (a) To alleviate the pose-bias
due to the visual embedding, we conduct Orchestration of the contextual
embedding _i.e_., adjust the visual embedding to be orthogonal to the textual
embedding. Orchestration accurately guides the denoising process toward a
direction that follows the pose directed by the text prompt. (b) Self-
attention Swap obtains self-attention key and value from another denoising
process guided by visual-only embedding, which offers the subject’s clean
identity. After swapping, our method retains the subject’s identity while
changing its pose faithfully following the text prompt.

To effectively attain the subject’s identity using the visual-only embedding
while simultaneously maintaining a novel pose of the subject, we delve into
the image editing methods. The image editing methods generally employ two
denoising processes, one for a source image reconstruction and another for
generating a new, edited image. The properties of the source image obtained
during its reconstruction are transferred to the process of generating the
edited image by swapping self-attention [5, 37] or cross-attention [10, 25,
7].

Inspired by them, we adopt the second denoising process
$\\{z^{\prime}_{\tau}\\}_{\tau=1\cdots T}$ where the visual-only embedding is
provided as the contextual embedding
($\textbf{c}_{\textbf{v},\varnothing}=[v_{1}\cdots v_{M};\varnothing]$). The
original denoising process is denoted as $\\{z_{\tau}\\}_{\tau=1\cdots T}$
that utilizes orthogonal visual embedding in Eq. 3 along with the textual
embedding ($\textbf{c}^{\bot}=[v^{\bot}_{1}\cdots v^{\bot}_{M};t_{1}\cdots
t_{N}]$). Then, we modify the self-attention layers of
$\\{z_{\tau}\\}_{\tau=1\cdots T}$, to swap key and value with those from
$\\{z^{\prime}_{\tau}\\}_{\tau=1\cdots T}$. Our proposed self-attention swap
can be formulated as follows:

$\text{AttnSwap}(z,z^{\prime})=M^{\prime}_{S}V^{\prime},\hskip
3.99994pt\text{where}\hskip 3.99994ptM^{\prime}_{S}=\text{softmax}(QK^{\prime
T}/\sqrt{d}).$ (4)

Here, $Q\in\mathbb{R}^{l\times d}$ is the query from
$\\{z_{\tau}\\}_{\tau=1\cdots T}$ and
$K^{\prime},V^{\prime}\in\mathbb{R}^{l\times d}$ are the key and the value
from $\\{z^{\prime}_{\tau}\\}_{\tau=1\cdots T}$ where $l$ and $d$ are spatial
sequence length and features dimension, respectively.
$M^{\prime}_{S}\in\mathbb{R}^{l\times l}$ evaluates how closely connected
latent spatial features from $z_{t}$ and $z^{\prime}_{t}$ are, assigning
higher scores to latent pixels when they have a strong relationship. Using Eq.
4, we inject $V^{\prime}$ in each latent pixel from
$\\{z^{\prime}_{\tau}\\}_{\tau=1\cdots T}$ based on $M^{\prime}_{S}$. In other
words, while generating a novel pose of the subject, we incorporate values
from the clear identity of the subject into the location where the
corresponding latent pixels exist.

As we aim to preserve the subject’s identity while allowing flexibility in the
remaining aspects, we restrict self-attention swap within latent pixels
assigned to the subject. Inspired by [10, 5], we utilize the cross-attention
maps $M_{C}$ from the original denoising process to create a mask
distinguishing the subject from the background. $M_{C}\in\mathbb{R}^{l\times
l_{c}}$ indicates how much information of text tokens is associated with each
latent pixel where $l_{c}$ is the context sequence length. We could obtain a
binary mask $m\in\\{0,1\\}^{l}$ for the subject token with fixed thresholds.
We ultimately adopt self-attention swap as follows:

$\text{MaskedAttnSwap}(z,z^{\prime})=\text{AttnSwap}(z,z^{\prime})*m+M_{S}V*(1-m),$
(5)

where $M_{S}=\text{softmax}(QK^{T}/\sqrt{d})$ and $K,V$ are key and value
features from the original process $\\{z_{\tau}\\}_{\tau=1\cdots T}$. Fig.
3(b) shows the overall process of self-attention swap. We also provide more
details in the supplementary materials.

##  5 Experiments

#### Datasets.

Since prevailing benchmark datasets [30, 16] primarily utilize generating
prompts related to changing backgrounds or introducing new objects, they often
fall short in effectively evaluating the crucial aspect of modifying subject
poses. To address this limitation, we have constructed a new dataset,
Deformable Subject Set (DS set), to effectively assess the model’s capability
to modify a subject’s pose. The DS set comprises 19 live animals from the
DreamBooth [30] and CustomDiffusion [16], along with 11 prompts specifically
designed to focus on the deformation of the subjects’ poses. Furthermore, we
also utilized the DreamBooth dataset (DB set) [30] to evaluate the model’s
capacity in typical scenarios.

#### Metrics.

Following DreamBooth [30], we measured the subject fidelity using CLIP-I and
DINO-I, and measured text alignment using CLIP-T. For the DS set, we
additionally measured the masked scores for CLIP-I and DINO-I, incorporating
the subject’s segmentation mask for both the input images and generated
images, as performed in [2]. We integrated segmentation masks into our image
alignment measures to mitigate the impact of newly introduced objects in
generated images, allowing for a more focused comparison of the subject’s
identity. This approach is particularly valuable when prompts necessitate the
addition of new objects alongside the subject in the image.

#### Implementation Details.

In our main experiment, we employed open-sourced zero-shot customization
models, BLIP-Diffusion [17] and ELITE [39] as the base models. We utilized the
PNDM scheduler [19] for Blip-Diffusion and the LMS scheduler [13] for ELITE,
each with 100 denoising steps, according to their default settings. Self-
attention swap was applied after the 20th denoising step.

Figure 4:  Comparisons with BLIP-Diffusion.

###  5.1 Qualitative Results

We present a comparative analysis of our method with BLIP-Diffusion across
three scenarios in Fig. 4. In the 1st to 4th rows of Fig. 4, our method
demonstrates the superior text alignment compared to the baseline when pose
variation prompts are given. While the BLIP-Diffusion tends to replicate the
subject’s pose from the input image without adjustment, our method effectively
modifies the pose in accordance with the provided prompt. Furthermore, we also
found a similar tendency when using object-related prompts depicted in the 5th
and 6th rows of Fig. 4. It is noted that while the BLIP-Diffusion simply
generates objects without executing the specified actions, our method
demonstrates the capability to perform the appropriate action in response to
the given prompt. In the perspective of subject’s identity preservation, as
shown in the last row of Fig. 4, ours preserve the identity better than the
baseline. We also demonstrate the improvements of our method compared to
ELITE, particularly in terms of object-related action and subject identity
preservation in Fig. 5. We provide more qualitative results in supplementary
materials.

Figure 5: Comparisons with ELITE.

###  5.2 Quantitative Results

Tab. 1 shows quantitative analyses using both the DS set and the DB set [30].
In case of the DS set, which includes object-related action prompts as shown
in 5–6th row of Fig. 4, we found that the newly generated object affects the
image-alignment score, for example, by occluding the subject. Therefore, we
additionally report metrics using the subject’s segmentation masks (M-CLIP-I
and M-DINO-I) to more properly evaluate the image alignment in a pose
modification setting. For the DS set that consists of the prompts
necessitating the deformation of the subject’s pose, our method demonstrates
significant improvement in the text alignment metric without compromising the
image alignment metric compared to the baseline. This suggests that our method
effectively puts the subject in a pose according to the provided text prompt
while preserving the subject’s identity. Furthermore, quantitative analysis on
the DB set confirms that our methodology remains comparable to the existing
baseline, even in typical scenarios where the subject is non-deformable object
and the subject’s pose is unchanged.

Table 1: Quantitative Comparison on Deformable Subject Set and DreamBooth
dataset [30]. ‘M-’ indicates the metrics using segmentation masks.

| Deformable Subject Set | DreamBooth Set [30]
---|---|---
Method |  CLIP-T($\uparrow$) |  M-CLIP-I($\uparrow$) |  M-DINO-I($\uparrow$) |  CLIP-I($\uparrow$) |  DINO-I($\uparrow$) |  CLIP-T($\uparrow$) |  CLIP-I($\uparrow$) |  DINO-I($\uparrow$)
BLIP-D [17] | 0.262 | 0.846 | 0.617 | 0.853 | 0.684 | 0.295 | 0.812 | 0.660
w/Orchestration | 0.276 | 0.843 | 0.610 | 0.819 | 0.662 | 0.298 | 0.805 | 0.647
w/SA Swap | 0.259 | 0.848 | 0.631 | 0.834 | 0.693 | 0.293 | 0.815 | 0.675
Ours | 0.272 | 0.848 | 0.630 | 0.819 | 0.681 | 0.296 | 0.809 | 0.665
ELITE [39] | 0.285 | 0.835 | 0.574 | 0.751 | 0.486 | 0.294 | 0.792 | 0.664
w/Orchestration | 0.292 | 0.834 | 0.570 | 0.754 | 0.493 | 0.295 | 0.789 | 0.658
w/SA Swap | 0.288 | 0.837 | 0.581 | 0.754 | 0.489 | 0.292 | 0.796 | 0.673
Ours | 0.300 | 0.837 | 0.581 | 0.755 | 0.502 | 0.294 | 0.792 | 0.670

###  5.3 User Study

We further evaluate our method through the user study conducted with Amazon
Mechanical Turk. Human raters were given a subject’s input image, a prompt,
and two synthesized images (ours and the baseline) of the subject. They were
then asked to select the preferable one for each of the following questions:
(1) Image Alignment: “Which of the images best reproduces the identity
(_e.g_., item type and details) of the reference item?” (2) Text Alignment:
“Which of the images is best described by the reference text?”. We used 19
animal subjects and 11 pose-related prompts per subject, assigning 5 raters
for each example.

Figure 6: User Study Results.

As shown in Fig. 6, our method is preferred over the baseline in both text
alignment (Ours 62.0% vs. BLIP-Diffusion 38.0%, Ours 56.3% vs. ELITE 43.7%)
and image alignment (Ours 51.5% vs. BLIP-Diffusion 48.5%, Ours 60.0% vs. ELITE
40.0%). It is noteworthy that our method was selected for the better alignment
of both text and image, underscoring its ability to faithfully follow the text
prompt and preserve the subject’s identity from a human perspective.

###  5.4 Ablations

Our orchestration method eliminates the conflicting elements in the visual
embedding that interferes with the textual embedding. Then, a subsequent
question arises: why don’t we directly orthogonalize the visual embedding with
respect to the text description about the input image? In other words, why not
eliminate such elements by only accessing the subject’s image without
considering the accompanying text prompt embedding? To investigate the effect
of orthogonalization with respect to the embedding vectors obtained from (i)
the text prompt for generation and (ii) the caption describing the input image
(Fig. 7(a)), we conduct the following experiment: we adopt the existing image
captioning model [18] and transform the visual embedding to be orthogonal to
its caption’s embedding using Eq. 3. We also use human annotations that
describe the input image in both short and long versions. Fig. 7(a) shows that
without considering the text prompt embedding, the generated images are
strongly biased toward the input image. More concretely, when we orthogonalize
the visual embedding with respect to the embedding vectors derived from the
captions describing the input image, like BLIP captioning (“A dog standing on
a rock”) or human annotations, (“A dog standing” and “A dog standing upright
on a rock, facing forward, with its mouth slightly open and holding the right
front paw a bit elevated”) the discord among the contextual embeddings remains
unalleviated. On the other hand, our method of orthogonalization with respect
to the text embedding effectively resolves the discord, rendering the images
faithful to the text prompt without requiring any additional language model or
human endeavor.

Figure 7: Ablation Results. (a) We compare different orthogonalizations with
respect to various textual embeddings, derived from the input image’s caption
and text prompt (ours). Explicitly considering a relationship between the
input image and the text prompt effectively aligns the visual embedding. (b)
We verify the effectiveness of each component in our method. Orchestration
encourages generated images to faithfully follow the text prompt while self-
attention swap successfully retains the subject’s identity.

We also conducted an ablation study on the roles of each component,
orchestration and self-attention swap, by progressively applying them. As
shown in Fig. 7(b) and Tab. 1, orchestration alone makes huge progress on
faithfully following the text prompt, while loss in the subject’s identity
remains unsolved. After applying self-attention swap which restores the
subject’s identity, our proposed method successfully generates images that
accurately adhere to the text prompt while maintaining the subject’s identity.

###  5.5 Analysis

We conduct the same experiment as in Sec. 4.1 with our method and report the
result. Red dots in Fig. 2(b) shows that with the orchestration of the
contextual embedding, the image features of the generated images are properly
aligned between two image features _i.e_., the outputs when solely using
visual embedding $\text{c}_{\textbf{v},\varnothing}$ and solely using textual
embedding $\text{c}_{\varnothing,\textbf{c}}$, respectively. We also computed
LPIPS [44] distance of our generated images with the outputs using the visual-
only embedding (LPIPS=6.78) and with the outputs using the textual-only
embedding (LPIPS=6.64), respectively. Compared to the baseline (LPIPS=5.69,
7.25) remarkably closer to the visual-only outputs, the contextual embedding
orchestration put the generated images in appropriate points between identity
preservation and text alignment. Note that an increase in LPIPS distance with
the outputs from the visual-only embedding is an expected consequence of a
change in the subject’s context (_e.g_., pose).

Figure 8: Visualization of each token’s cross-attention map (red: high value,
blue: low value). In baseline, the visual and textual embeddings interfere
with each other, attending to irrelevant areas (_e.g_., beneath the dog).
Meanwhile, our method effectively resolves the discord among the contextual
embedding, injecting each visual and text information in the proper areas.

To verify that the visual embedding is properly adjusted after orchestration,
we also visualized cross-attention maps for tokens corresponding to the visual
and textual embeddings. Cross-attention maps indicate the amount of
information conveyed from the tokens to the latent spatial features $f$,
therefore, we can obtain insight about the information flows of each token. We
interpolated cross-attention maps from each layer in both the encoder and
decoder, averaging them across layers and tokens. As shown in Fig. 8, in the
baseline, the discord among the contextual embedding leads the textual
embedding (“<standing>”) to highlight the irrelevant areas (_e.g_., beneath
the dog), resulting in the image noncompliant with the text prompt. Meanwhile,
our orchestration effectively resolves the discord, integrating the textual
embedding in the proper areas.

##  6 Conclusion

Limitations. Our method inherits the weakness from T2I diffusion models and
falls short of complying several text prompts at once. When a set of prompts
in different instructions is given, our method cannot accurately orthogonalize
the visual embedding to each textual embedding, resulting in the images
omitting some instructions. We provide more details in supplementary
materials.

In this paper, we explore notable problems in pose variation tasks when using
the zero-shot customization methods. We focus on a conflict between the visual
and textual embeddings, which affects both embeddings and leads to the pose
bias and identity loss, respectively. We alleviate the conflict by contextual
embedding orchestratation, that is, orthogonalize the visual embedding with
respect to the textual embedding subspace. While orchestration effectively put
the subject in various poses complying with the text prompts, the subject’s
identity remains partially damaged. To resolve the issue, we propose self-
attention swap, that adopts the subject’s clean identity from the visual-only
embedding to the appropriate latent pixels. Amidst the proliferation of zero-
shot customization methods and the utilization of the visual embedding, our
key insight unveils a crucial aspect for achieving a more diverse and pose-
variant generation.

## References

  * [1] Armandpour, M., Sadeghian, A., Zheng, H., Sadeghian, A., Zhou, M.: Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond (2023)
  * [2] Avrahami, O., Aberman, K., Fried, O., Cohen-Or, D., Lischinski, D.: Break-a-scene: Extracting multiple concepts from a single image. arXiv preprint arXiv:2305.16311 (2023)
  * [3] Avrahami, O., Lischinski, D., Fried, O.: Blended diffusion for text-driven editing of natural images. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (Jun 2022). https://doi.org/10.1109/cvpr52688.2022.01767, http://dx.doi.org/10.1109/CVPR52688.2022.01767
  * [4] Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Zhang, Q., Kreis, K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., Karras, T., Liu, M.Y.: ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers (2023)
  * [5] Cao, M., Wang, X., Qi, Z., Shan, Y., Qie, X., Zheng, Y.: Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 22560–22570 (October 2023)
  * [6] Chen, L., Zhao, M., Liu, Y., Ding, M., Song, Y., Wang, S., Wang, X., Yang, H., Liu, J., Du, K., Zheng, M.: Photoverse: Tuning-free image customization with text-to-image diffusion models (2023)
  * [7] Couairon, G., Verbeek, J., Schwenk, H., Cord, M.: Diffedit: Diffusion-based semantic image editing with mask guidance (2022)
  * [8] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-Or, D.: An image is worth one word: Personalizing text-to-image generation using textual inversion (2022)
  * [9] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-Or, D.: An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 (2022)
  * [10] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 (2022)
  * [11] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models (2020)
  * [12] Jia, X., Zhao, Y., Chan, K.C.K., Li, Y., Zhang, H., Gong, B., Hou, T., Wang, H., Su, Y.C.: Taming encoder for zero fine-tuning image customization with text-to-image diffusion models (2023)
  * [13] Karras, T., Aittala, M., Aila, T., Laine, S.: Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems 35, 26565–26577 (2022)
  * [14] Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., Irani, M.: Imagic: Text-based real image editing with diffusion models (2023)
  * [15] Kim, Y., Lee, J., Kim, J.H., Ha, J.W., Zhu, J.Y.: Dense text-to-image generation with attention modulation (2023)
  * [16] Kumari, N., Zhang, B., Zhang, R., Shechtman, E., Zhu, J.Y.: Multi-concept customization of text-to-image diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1931–1941 (2023)
  * [17] Li, D., Li, J., Hoi, S.: Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Processing Systems 36 (2024)
  * [18] Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning. pp. 12888–12900. PMLR (2022)
  * [19] Liu, L., Ren, Y., Lin, Z., Zhao, Z.: Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778 (2022)
  * [20] Liu, N., Li, S., Du, Y., Torralba, A., Tenenbaum, J.B.: Compositional visual generation with composable diffusion models (2023)
  * [21] Ma, J., Liang, J., Chen, C., Lu, H.: Subject-diffusion:open domain personalized text-to-image generation without test-time fine-tuning (2023)
  * [22] Mokady, R., Hertz, A., Aberman, K., Pritch, Y., Cohen-Or, D.: Null-text inversion for editing real images using guided diffusion models (2022)
  * [23] Mou, C., Wang, X., Xie, L., Wu, Y., Zhang, J., Qi, Z., Shan, Y., Qie, X.: T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models (2023)
  * [24] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models (2022)
  * [25] Parmar, G., Singh, K.K., Zhang, R., Li, Y., Lu, J., Zhu, J.Y.: Zero-shot image-to-image translation (2023)
  * [26] Qiu, Z., Liu, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., Zhang, D., Weller, A., Schölkopf, B.: Controlling text-to-image diffusion by orthogonal finetuning (2023)
  * [27] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision (2021)
  * [28] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents (2022)
  * [29] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models (2022)
  * [30] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22500–22510 (2023)
  * [31] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S.K.S., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., Salimans, T., Ho, J., Fleet, D.J., Norouzi, M.: Photorealistic text-to-image diffusion models with deep language understanding (2022)
  * [32] Shi, J., Xiong, W., Lin, Z., Jung, H.J.: Instantbooth: Personalized text-to-image generation without test-time finetuning (2023)
  * [33] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [34] Sohl-Dickstein, J., Weiss, E.A., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics (2015)
  * [35] Tewel, Y., Gal, R., Chechik, G., Atzmon, Y.: Key-locked rank one editing for text-to-image personalization (2023)
  * [36] Tewel, Y., Gal, R., Chechik, G., Atzmon, Y.: Key-locked rank one editing for text-to-image personalization. In: ACM SIGGRAPH 2023 Conference Proceedings. pp. 1–11 (2023)
  * [37] Tumanyan, N., Geyer, M., Bagon, S., Dekel, T.: Plug-and-play diffusion features for text-driven image-to-image translation (2022)
  * [38] Voynov, A., Chu, Q., Cohen-Or, D., Aberman, K.: P+: Extended textual conditioning in text-to-image generation (2023)
  * [39] Wei, Y., Zhang, Y., Ji, Z., Bai, J., Zhang, L., Zuo, W.: Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302.13848 (2023)
  * [40] Xiao, G., Yin, T., Freeman, W.T., Durand, F., Han, S.: Fastcomposer: Tuning-free multi-subject image generation with localized attention (2023)
  * [41] Yang, B., Gu, S., Zhang, B., Zhang, T., Chen, X., Sun, X., Chen, D., Wen, F.: Paint by example: Exemplar-based image editing with diffusion models (2022)
  * [42] Yuan, Z., Cao, M., Wang, X., Qi, Z., Yuan, C., Shan, Y.: Customnet: Zero-shot object customization with variable-viewpoints in text-to-image diffusion models (2023)
  * [43] Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models (2023)
  * [44] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 586–595 (2018)


11institutetext:  Stockholm, Sweden
11email<EMAIL_ADDRESS>22institutetext: Center for Smart Computing
Continuum, Forschung Burgenland, Eisenstadt, Austria
22email<EMAIL_ADDRESS>
# Data-driven Real-time Short-term Prediction of Air Quality: Comparison of
ES, ARIMA, and LSTM ††thanks: Preprint

Iryna Talamanova 11 Sabri Pllana 22

###### Abstract

Air pollution is a worldwide issue that affects the lives of many people in
urban areas. It is considered that the air pollution may lead to heart and
lung diseases. A careful and timely forecast of the air quality could help to
reduce the exposure risk for affected people. In this paper, we use a data-
driven approach to predict air quality based on historical data. We compare
three popular methods for time series prediction: Exponential Smoothing (ES),
Auto-Regressive Integrated Moving Average (ARIMA) and Long short-term memory
(LSTM). Considering prediction accuracy and time complexity, our experiments
reveal that for short-term air pollution prediction ES performs better than
ARIMA and LSTM.

ir pollution, Exponential Smoothing, ARIMA, LSTM

###### Keywords:

a

##  1 Introduction

The high-density of urban population may often lead to high-levels of various
kinds of pollution, such as, air pollution [25], light pollution [11], or
noise pollution[2, 3]. Air pollution is an important indicator of the life
quality of a city. It may affect human health by causing heart and lung
diseases [7]. Furthermore, air pollution is a significant contributor to
climate change [19]. Table 1 describes the health impact of some of the most
harmful air pollutants.

Table 1: Impact of air pollution on human health. Pollutant |  Health Impact
---|---
CO |  Mostly affects the cardiovascular system and leads to a lack of oxygen in the human body. The outcome is bad concentration and slow reflexes. It also may cause lung inflammation.
O3 |  Affects lungs and respiratory system and may lead to lung inflammation. It also reduces lung functions and may cause asthma or even lung cancer.
NO2, SO2 |  Affects the respiratory system by reducing its resistance to respiratory infections.
PM |  May cause inflammatory lung changes. It is dangerous due to its small size, which allows it to reach the heart and the brain and may cause inflammation there.

Sensors for personal use enable to measure the air pollution at a specific
location. Based on the retrieved data, it is possible to predict future air
pollution using time series forecasting methods. Existing solutions, are
already capable to quantify the air quality state. However, these solutions do
not provide short-term forecasts in real time.

Prediction of time series in near real time implies that the model should be
periodically updated to account for recently generated data. Amjad et al. [5]
propose to monitor the trend of the financial time series and update the model
if the trend changes direction. Qin et al. [18] suggest to create a schedule
for updating the data within fixed time frame. For an adaptive model
selection, Le et al. [10] suggest to update the model each time new data
arrives. A trend in the series of air pollution data may change unpredictably
because of environmental changes. Therefore, updating the model each time new
data arrives is a promising approach for air pollution data.

In this paper, we compare methods that may be applied to real-time prediction
of air quality. We conduct a performance comparison with respect to the
prediction accuracy and time complexity for three methods: Exponential
Smoothing, ARIMA, and LSTM. For experimental evaluation we use a data set [16]
that comprises air pollution data collected in Skopje. Our experimental
results indicate that Exponential Smoothing outperforms ARIMA and LSTM for
real time prediction of air quality.

The rest of the paper is structured as follows. We describe the data set, code
implementation, and metrics in Section 2. Section 3 describes the experimental
evaluation. Related work is discussed in Section 4. Section 5 concludes the
paper.

##  2 Methodology

In this section, we describe data preparation, code implementation of
prediction methods, and metrics.

###  2.1 Data preparation

We use for experimental evaluation the data collected in Skopje [16]. This
data set contains air pollution measurements from seven stations, and includes
CO, NO2, O3, PM10, and PM2.5. The pollutants behave quite similar over time,
and our focus in this study is on PM2.5.

Figure 1 visualizes the data set that originally had missing values and
outliers.

Figure 1: Initial data set.

We use linear interpolation for filling missing values. It is assumed that
missing values lie on the line which could be drawn using a set of known
points. The data set after applying interpolation is depicted in Figure 2.

Figure 2: Data set after filling missing values.

Because we are interested in short-term predictions, we decided to predict the
air pollution 24 hours ahead; that is, the test interval is 24 hours. In
accordance with [21] the training and test interval are split as 70% and 30%
respectively. We should consider that the prediction accuracy of a Neural
Network usually improves with the increase of the amount of training data.
Therefore, for the LSTM model the training set is much larger than for
statistical models.

###  2.2 Code implementation

We have implemented all algorithms in this study in Python using open source
libraries and frameworks (including, statsmodels [12], pmdarima [22], keras
[8], tensorflow [1]).

###  2.3 Measurement metrics

For the real-time environment emulation, we use the rolling window technique.
After building the model and making the forecast, the data is moved one hour
ahead and model building and forecasting is repeated.

Figure 3 depicts the _rolling forecast_. The forecast is done multiple times
using different data and the error is calculated by averaging all errors that
we have got. This technique is independent of the data and provides correct
results.

Figure 3: Rolling forecast enables to emulate the real-time environment.

We use the Root Mean Squared Error (RMSE), because it is considered to be less
sensitive with respect to outliers. RMSE is the square root of the average of
squared differences between the predicted value and actual value of the
series.

$RMSE=\sqrt{\frac{\sum_{t=1}^{n}(y_{i}-x_{i})^{2}}{n}}$ (1)

All experiments were conducted multiple times on different time intervals and
the mean of the results has been taken. Also for each method, the multi-step
forecast has been produced for comparison.

##  3 Evaluation

In this section, we present the training intervals and the Root Mean Squared
Error (RMSE) for Exponential Smoothing, ARIMA, and LSTM. Additionally, a
performance comparison of the studied methods is provided.

###  3.1 Exponential smoothing (ES)

To determine the best training intervals we conducted the following
experiment. We have trained the model for various numbers of hours and
selected the one that resulted with the lowest RMSE. Because we are aiming for
short-term predictions, it is not necessary to consider larger training
intervals.

Table 2 presents the average RMSE for different training intervals. We may
observe that the best training interval for Exponential Smoothing is 96 hours,
because RMSE for this period is the lowest.

Table 2: RMSE of ES model for various training intervals. Time [h] | 48 | 72 | 96 | 120 | 144 | 168 | 196
---|---|---|---|---|---|---|---
RMSE | 10.54 | 9.76 | 6.39 | 8.26 | 10.47 | 12.6 | 10.12

###  3.2 ARIMA

To make the prediction process faster and reduce the model selection time an
experiment for determining hyperparameter intervals has been conducted. We
have evaluated various time intervals to determine what the minimum and
maximum values of the hyperparameters could be. The results of this experiment
are presented in Table 3.

Table 3: ARIMA hyperparameter intervals. Hyperparameter | Minimum | Maximum
---|---|---
p | 0 | 6
d | 0 | 2
q | 0 | 5
P | 0 | 3
D | 0 | 1
Q | 0 | 2

After the hyperparameters boundaries were determined, we observed a
significant improvement in the speed of building an ARIMA model.

Table 4 indicates that the best training interval for the ARIMA model is 120
hours, because it results with the lowest RMSE across the considered
intervals.

Table 4: RMSE of ARIMA model for various training intervals Time [h] | 72 | 96 | 120 | 144 | 168 | 196 | 220
---|---|---|---|---|---|---|---
RMSE | 11.24 | 8.89 | 8.59 | 12.1 | 14.32 | 12.83 | 11.56

The time for building the ARIMA model and performing prediction is 19.8
seconds in our experiment. While the time required for building ARIMA model
and performing prediction is shorter than for Exponential Smoothing (20.5
seconds), the prediction error of ARIMA model is larger.

###  3.3 LSTM

We have implemented LSTM using Keras framework [8]. In the first stage of the
experiment, we evaluated different LSTM configurations: Simple, Stacked,
Bidirectional, and Encoder-decoder.

Table 5: Accuracy and execution time of various LSTM network configurations. Network Configuration | RMSE | Time [s]
---|---|---
Simple | 3.26 | 1196.32
Stacked | 3.32 | 1253.21
Bidirectional | 4.19 | 1372.65
Encoder-decoder | 3.75 | 1476.87

Evaluation results of LSTM network configurations are presented in Table 5. We
may observe that the best accuracy is achieved using a Simple LSTM
configuration. In what follows in this section, we describe experimental
results for the Simple LSTM configuration. Table 6 shows the selected values
of hyperparameters for the Simple LSTM configuration.

Table 6: LSTM hyperparameter values. Hyper-parameter | Value
---|---
Epoch | 800
Patience coefficient | 0.1
Validation size | 72 hours
Dropout | 0.1
Recurrent dropout | 0.3
Batch size | 12
Type | Statefull
Coefficient for counting units | 3
Training size | 8000

Table 7 shows RMSE and execution time of the Simple LSTM configuration for
various values of prediction horizon.

We may observe that the execution time of LSTM is significantly higher than
for the other methods considered in this study (that is, ES and ARIMA).
Furthermore, the prediction accuracy worsens with the increase of the
prediction horizon.

Table 7: RMSE and execution time of the Simple LSTM configuration for various values of Horizon. Horizon [h] | RMSE | Time [s]
---|---|---
1 | 3.26 | 1282.46
2 | 4.47 | 953.40
3 | 5.35 | 1140.42
4 | 6.17 | 1185.93
5 | 7.07 | 1101.65
6 | 7.31 | 967.06
7 | 8.24 | 1303.88
8 | 6.21 | 900.61
9 | 9.06 | 1111.17
10 | 9.14 | 1055.00
11 | 9.67 | 1354.31
12 | 10.41 | 1081.18
13 | 10.44 | 1258.37
14 | 11.06 | 1195.12
15 | 10.71 | 936.59
16 | 11.29 | 1107.04
17 | 11.88 | 1213.67
18 | 13.40 | 1155.57
19 | 12.02 | 977.91
20 | 12.55 | 966.46
21 | 12.71 | 1199.05
22 | 12.93 | 1078.39
23 | 11.74 | 1003.03
24 | 14.69 | 1117.89

###  3.4 Performance comparison of ES, ARIMA, and LSTM

Figure 4 depicts the relationship of RMSE and prediction horizon for ES,
ARIMA, and LSTM. RMSE increases for all three considered methods with the
increase of prediction horizon. For all considered values of prediction
horizon, ES has the lowest RMSE (that is, the highest prediction accuracy)
compared to ARIMA and LSTM.

Figure 4: RMSE and prediction Horizon [h] for ES, ARIMA, and LSTM.

With respect to the time for building a model and making a forecast, ARIMA and
ES require about 20 seconds in our experiments. LSTM is significantly slower
and requires about 1000 seconds for model building and forecasting.

Considering prediction accuracy (that is, RMSE) and model building and
forecasting time, we may conclude that ES is the most suitable among the
studied methods for short-term prediction of air pollution.

###  3.5 Future research directions

The future work may investigate the performance of additional prediction
methods in the context of air pollution. Techniques for parallel processing
([4, 17]), acceleration ([24, 13, 6]), and intelligent parameter selection
([14]) could be studied to further improve the efficiency.

##  4 Related Work

In this section, we provide an overview of the related work.

Shaban et al. [20] investigate three machine learning algorithms that could be
used for alarming applications in the context of air pollution: Support Vector
Machines, M5P Model Trees, and Artificial Neural Networks. In future they plan
to study real-time prediction of air pollution.

Subramanian [23] studies application of Multiple Linear Regression and Neural
Networks to forecasting of the pollution concentration.

Le et al. [9] describe an air-pollution prediction model that is based on
spatio-temporal data that is collected from air quality sensors installed on
taxis running across the city Daegu, Korea. The prediction model is based on
the Convolutional Neural Network for an image like spatial distribution of air
pollution. The temporal information in the data is handled using a combination
of a Long Short-Term Memory unit for time series data and a Neural Network
model for other air pollution impact factors (such as, weather conditions.)

Ochando et al. [15] use traffic and weather data for prediction of the air
pollution. The aim is to provide general information about the air quality of
the city, and they do not focus on a particular spot within the city. In this
study, the Random Forest model performs best.

Related work mostly focuses on studying air pollution data that has been
already generated and stored in the past. In contrast to related work, we
focus on using continuously updated data (that may be generated by sensors)
for short-term prediction of the air pollution in near real-time.

##  5 Conclusions

We have presented various methods that may be applied to short-term air
quality prediction in real time. Using a real-world data set we conducted a
performance comparison of three methods: Exponential Smoothing, ARIMA, and
LSTM.

We have observed that Exponential Smoothing performs more efficiently for the
short-term air pollution prediction compared to ARIMA and LSTM. LSTM has
larger prediction error and takes more time to make the prediction. For the
multi-step ahead forecast the gap between prediction errors of these methods
grows with the increase of the number of steps.

## References

  * [1] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.: Tensorflow: A system for large-scale machine learning. In: 12th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 16). pp. 265–283 (2016)
  * [2] Alsouda, Y., Pllana, S., Kurti, A.: A Machine Learning Driven IoT Solution for Noise Classification in Smart Cities. In: Machine Learning Driven Technologies and Architectures for Intelligent Internet of Things (ML-IoT). pp. 1–6. Euromicro (2018). https://doi.org/10.48550/arXiv.1809.00238
  * [3] Alsouda, Y., Pllana, S., Kurti, A.: IoT-based Urban Noise Identification Using Machine Learning: Performance of SVM, KNN, Bagging, and Random Forest. In: Proceedings of the International Conference on Omni-Layer Intelligent Systems. pp. 62–67. COINS ’19, ACM, New York, NY, USA (2019). https://doi.org/10.1145/3312614.3312631
  * [4] Amaral, V., Norberto, B., Goulão, M., Aldinucci, M., Benkner, S., Bracciali, A., Carreira, P., Celms, E., Correia, L., Grelck, C., Karatza, H., Kessler, C., Kilpatrick, P., Martiniano, H., Mavridis, I., Pllana, S., Respício, A., Simão, J., Veiga, L., Visa, A.: Programming languages for data-intensive HPC applications: A systematic mapping study. Parallel Computing 91, 102584 (2020). https://doi.org/https://doi.org/10.1016/j.parco.2019.102584
  * [5] Amjad, M., Shah, D.: Trading bitcoin and online time series prediction. In: NIPS 2016 Time Series Workshop. pp. 1–15 (2017)
  * [6] Benkner, S., Pllana, S., Traff, J., Tsigas, P., Dolinsky, U., Augonnet, C., Bachmayer, B., Kessler, C., Moloney, D., Osipov, V.: PEPPHER: Efficient and Productive Usage of Hybrid Computing Systems. IEEE Micro 31(5), 28–41 (2011). https://doi.org/10.1109/MM.2011.67
  * [7] Das, P.K., A, D.V., Meher, S., Panda, R., Abraham, A.: A systematic review on recent advancements in deep and machine learning based detection and classification of acute lymphoblastic leukemia. IEEE Access 10, 81741–81763 (2022). https://doi.org/10.1109/ACCESS.2022.3196037
  * [8] Gulli, A., Pal, S.: Deep Learning with Keras. Packt Publishing Ltd (2017)
  * [9] Le, D.: Real-time air pollution prediction model based on spatiotemporal big data. arXiv preprint arXiv:1805.00432 (2018)
  * [10] Le Borgne, Y.A., Santini, S., Bontempi, G.: Adaptive model selection for time series prediction in wireless sensor networks. Signal Processing 87(12), 3010–3020 (2007)
  * [11] Longcore, T., Rich, C.: Ecological light pollution. Frontiers in Ecology and the Environment 2(4), 191–198 (2004)
  * [12] Massaron, L., Boschetti, A.: Regression Analysis with Python. Packt Publishing Ltd (2016)
  * [13] Memeti, S., Pllana, S.: Accelerating DNA Sequence Analysis Using Intel(R) Xeon Phi(TM). In: 2015 IEEE Trustcom/BigDataSE/ISPA. vol. 3, pp. 222–227 (2015). https://doi.org/10.1109/Trustcom.2015.636
  * [14] Memeti, S., Pllana, S.: Optimization of heterogeneous systems with AI planning heuristics and machine learning: a performance and energy aware approach. Computing 103(12), 2943–2966 (Dec 2021). https://doi.org/10.1007/s00607-021-01017-6
  * [15] Ochando, L.C., Julián, C.I.F., Ochando, F.C., Ferri, C.: Airvlc: An application for real-time forecasting urban air pollution. In: Proceedings of the 2nd International Conference on Mining Urban Data - Volume 1392. p. 72–79. MUD’15, CEUR-WS.org, Aachen, DE (2015)
  * [16] Petrushevski, S.: Air pollution in Skopje from 2008 to 2018 (2018), https://www.kaggle.com/cokastefan/pm10-pollution-data-in-skopje-from-2008-to-2018
  * [17] Pllana, S., Xhafa, F.: Programming Multicore and Many-core Computing Systems. John Wiley & Sons, Inc., Hoboken, New Jersey, USA, 1 edn. (2017). https://doi.org/10.1002/9781119332015
  * [18] Qin, X., Mahmassani, H.S.: Adaptive calibration of dynamic speed-density relations for online network traffic estimation and prediction applications. Transportation research record 1876(1), 82–89 (2004)
  * [19] Seinfeld, J.H., Pandis, S.N.: Atmospheric chemistry and physics: from air pollution to climate change. John Wiley & Sons (2016)
  * [20] Shaban, K.B., Kadri, A., Rezk, E.: Urban air pollution monitoring system with forecasting models. IEEE Sensors Journal 16(8), 2598–2606 (April 2016). https://doi.org/10.1109/JSEN.2016.2514378
  * [21] Siami-Namini, S., Namin, A.S.: Forecasting economics and financial time series: Arima vs. lstm. arXiv preprint arXiv:1803.06386 (2018)
  * [22] Smith, T.G.: pmdarima: Arima estimators for Python (2017)
  * [23] Subramanian, V.N.: Data analysis for predicting air pollutant concentration in smart city uppsala (2016)
  * [24] Viebke, A., Memeti, S., Pllana, S., Abraham, A.: CHAOS: a parallelization scheme for training convolutional neural networks on Intel Xeon Phi. The Journal of Supercomputing 75(1), 197–227 (Jan 2019). https://doi.org/10.1007/s11227-017-1994-x
  * [25] Yang, D., Wang, J., Yan, X., Liu, H.: Subway air quality modeling using improved deep learning framework. Process Safety and Environmental Protection 163, 487–497 (2022). https://doi.org/https://doi.org/10.1016/j.psep.2022.05.055