# On countability of Teichmüller modular groups for analytically infinite
Riemann surfaces defined by generalized Cantor sets

Erina Kinjo  Department of Mechanical Engineering, Ehime University, 3 Bunkyo-
cho, Matsuyama, Ehime 790-8577, Japan<EMAIL_ADDRESS>
(Date: July 6, 2024)

###### Abstract.

For any analytically finite Riemann surface, the Teichmüller modular group is
countable, but it is not easy to find an analytically infinite Riemann surface
for which the Teichmüller modular group is countable. In this paper, we show
that the Teichmüller modular group is countable or uncountable for some
analytically infinite Riemann surfaces defined by generalized Cantor sets.

###### Key words and phrases:

Riemann surface of infinite type, Teichmüller modular group, Generalized
Cantor set

###### 2020 Mathematics Subject Classification:

Primary 30F60; Secondary 32G15

##  1\. Introduction

###  1.1. Terminology of Riemann surfaces

We call a Riemann surface $X$ hyperbolic if $X$ is represented by a quotient
space $\mathbb{D}/\Gamma$ of the unit disk $\mathbb{D}$ by a torsion-free
Fuchsian group $\Gamma$. In this paper, any Riemann surface is supposed to be
hyperbolic. A Riemann surface $X$ is of analytically finite type if $X$ is
obtained from a compact surface by removing at most finitely many points, and
$X$ is of analytically infinite type if $X$ is not of analytically finite
type. On the other hand, a Riemann surface $X$ is of topologically finite type
if the fundamental group $\pi_{1}(X)\cong\Gamma$ is finitely generated, and
$X$ is of topologically infinite type if $X$ is not of topologically finite
type. Also, a Fuchsian group $\Gamma$ is of the first kind if the limit set of
$\Gamma$ coincides with the unit circle: $\Lambda(\Gamma)=\partial\mathbb{D}$,
and $\Gamma$ is of the second kind if
$\Lambda(\Gamma)\subsetneq\partial\mathbb{D}$. Now, if $X$ has the boundary,
then we write it as $\partial X$. Also, a Fuchsian group $\Gamma$ acts
properly discontinuously on $\overline{\mathbb{D}}\setminus\Lambda(\Gamma)$,
so if $\Gamma$ is of the second kind, then we obtain a bordered Riemann
surface $(\overline{\mathbb{D}}\setminus\Lambda(\Gamma))/\Gamma$ containing
$X$ as its interior. We refer to
$(\partial\mathbb{D}\setminus\Lambda(\Gamma))/\Gamma$ as the boundary at
infinity of $X$ and write it as $\partial X$, too.

###  1.2. Teichmüller space and its Teichmüller modular group

For a Riemann surface $X$, the Teichmüller space $T(X)$ is the set of
Teichmüller equivalence classes of quasiconformal mappings $f$ of $X$ onto
another Riemann surface, where two quasiconformal mappings $f_{1}$ and $f_{2}$
are Teichmüller equivalent if there exists a conformal mapping $h:f_{1}(X)\to
f_{2}(X)$ such that $f_{2}^{-1}\circ h\circ f_{1}:X\to X$ is homotopic to the
identity. If $\partial X\neq\emptyset$, the homotopy is considered to be
relative to $\partial X$ ($:=rel.$ $\partial X$), that is, the homotopy fixes
points of $\partial X$. We write the Teichmüller equivalence class of $f$ as
$[f]$. It is known that $T(X)$ has a complex Banach manifold structure, and if
$X$ is of analytically finite type, then $\dim T(X)<\infty$; othewise $\dim
T(X)=\infty$. On $T(X)$, a distance between two points $[f_{1}]$ and $[f_{2}]$
is defined by $d_{T}([f_{1}],[f_{2}])=\inf_{f}\log K(f)$, where the infimum is
taken over all quasiconformal mappings from $f_{1}(X)$ to $f_{2}(X)$ homotopic
to $f_{2}\circ f_{1}^{-1}$ ($rel.$ $\partial X$ if $\partial X\neq\emptyset$),
and $K(f)$ is the maximal dilatation of $f$. This is a complete distance on
$T(X)$ and is called Teichmüller distance.

For a Riemann surface $X$, the quasiconformal mapping class group MCG$(X)$ is
defined as the group of all homotopy classes $[g]$ of quasiconformal
automorphisms $g$ of $X$ ($rel.$ $\partial X$ if $\partial X\neq\emptyset$).
For each $[g]\in$ MCG$(X)$, define the transformation $[g]_{*}$ of $T(X)$ as
$[f]\mapsto[f\circ g^{-1}]$, then MCG$(X)$ acts on $T(X)$ isometrically with
respect to $d_{T}$. Now, let Aut$(T(X))$ be the group of all isometric
biholomorphic automorphisms of $T(X)$. We consider the homomorphism $\iota:$
MCG$(X)\to$ Aut$(T(X))$ defined by $[g]\mapsto[g]_{*}$ and define the
Teichmüller modular group for $X$, which is denoted by Mod$(X)$, as the image
Im $\iota\subset$ Aut$(T(X))$ of $\iota$. Except for a few low-dimensional
Teichmüller spaces, the homomorphism $\iota$ is injective (cf. [2], [8]) and
surjective (cf. [5]). Therefore, in this paper, we identify the quasiconformal
mapping class group with the Teichmüller modular group.

In section 3, we think a bit about the reduced Teichmüller modular group
Mod${}^{\sharp}(X)$ for a Riemann surface $X$. This is the quotient group of
Mod$(X)$ by free homotopy equivalence, that is, the homotopy does not
necessarily fix points of $\partial X$ if $\partial X\neq\emptyset$.

###  1.3. Some Riemann surfaces of topologically infinite type and Teichmüller
modular groups for them

In 2003, Shiga ([9]) considered two distances on the Teichmüller space $T(X)$;
the Teichmüller distance $d_{T}$ and the length spectrum distance $d_{L}$. By
the definition, the Teichmüller distance $d_{T}([f_{1}],[f_{2}])$ means how
different the complex structures of two Riemann surfaces $f_{1}(X)$ and
$f_{2}(X)$ are. On the other hand, though we do not describe the definition in
this paper, the length spectrum distance $d_{L}([f_{1}],[f_{2}])$ means how
different the hyperbolic structures of two Riemann surfaces $f_{1}(X)$ and
$f_{2}(X)$ are. If $X$ is analytically finite Riemann surface, then the two
distances $d_{T}$ and $d_{L}$ define the same topology on $T(X)$, but
otherwise it is not always true. Shiga constructed a topologically infinite
Riemann surface $S$ such that the two distances define different topologies on
$T(S)$. His Riemann surface is essentially the same as the Riemann surface $S$
constructed as follows: let $\\{a_{n}\\}_{n=1}^{\infty}$ be a monotonic
divergent sequence of positive numbers such that $a_{n+1}>na_{n}$, and let
$\\{P_{n}\\}_{n=1}^{\infty}$ be a sequence of pairs of pants such that the
hyperbolic lengths of three boundary geodesics of $P_{n}$ are
$a_{n},a_{n+1},a_{n+1}$ $(n=1,2,...)$. Firstly, make $2$ copies of $P_{1}$ and
glue them together along the boundaries of length $a_{1}$, then we obtain a
Riemann surface $S_{1}$ of type $(0,4)$. Secondly, make $4$ copies of $P_{2}$
and glue them to $S_{1}$ along the boundaries of length $a_{2}$, then we
obtain a Riemann surface $S_{2}$ of type $(0,8)$. Inductively, for each $n$,
make $2^{n}$ copies of $P_{n}$ and glue them to $S_{n-1}$ along the boundaries
of length $a_{n}$, then we obtain a Riemann surface $S_{n}$ of type
$(0,2^{n+1})$. We define the Riemann surface $S$ as the exhaustion of
$\\{S_{n}\\}_{n=1}^{\infty}$, i.e., $S=\bigcup_{n=1}^{\infty}S_{n}$. (By the
way, he also showed that if a topologically infinite Riemann surface $X$
satisfies some condition, the two distances define the same topology on $T(X)$
in the same paper [9]. And in 2018, we generalized his theorem, more
precisely, we showed that if $X$ is a Riemann surface with bounded geometry,
then the two distances define the same topology on $T(X)$ ([3]).)

In 2005, Matsuzaki ([7]) considered Shiga’s Riemann surface $S$, a
reconstructed Riemann surface $R$ from $S$ and the Teichmüller modular group
Mod$(R)$ for $R$. Before mentioning it, we introduce a proposition for
countability of the Teichmüller modular group.

######  Proposition 1.1 ((Proposition 1 of [7]).

Suppose $X$ is a hyperbolic Riemann surface. If Mod$(X)$ is countable, then
$X=\mathbb{D}/\Gamma$ satisfies the following conditions.

  1. (1)

The number of simple closed geodesics on $X$ of which lengths are smaller than
$M$ for arbitrary $M>0$ is finite.

  2. (2)

The Fuchsian group $\Gamma$ is of the first kind.

In §3 of [7], Matsuzaki showed that if a Riemann surface $S$ is constructed by
gluing above-mentioned pants $\\{P_{n}\\}_{n=1}^{\infty}$ in the usual way,
then $S$ is not geodesically complete, that is, there exists a geodesic
connecting $\partial P_{1}$ and $\partial P_{n}$ such that its length
converges as $n\to\infty$. This means that the geodesic completion of $S$ does
not coincide with $S$, hence the Fuchsian group corresponding to $S$ is of the
second kind. (cf. Proposition 3.7 of [1].) In particular, Mod$(S)$ is
uncountable by Proposition 1.1 (2). However, if a Riemann surface $R$ is
constructed by gluing above-mentioned pants $\\{P_{n}\\}_{n=1}^{\infty}$ in a
special way, then $R$ is geodesically complete, so the geodesic completion of
$R$ coincides with $R$. Here, a special way is to give each boundary geodesic
of each pair of pants some amount of twist when we glue pants together. Then,
the corresponding Fuchsian group is of the first kind, and also he could show
that Mod$(R)$ is countable.

###  1.4. Generalized Cantor sets

Let $\\{q_{n}\\}_{n=1}^{\infty}$ be a sequence of numbers in $(0,1)$. Put
$I:=[0,1]\subset\mathbb{R}$. A generalized Cantor set $E(\omega)$ for
$\omega=\\{q_{n}\\}_{n=1}^{\infty}$ is defined as follows: Firstly, remove an
open interval with the length $q_{1}$ from $I$ so that the remaining intervals
$I_{1}^{1},I_{1}^{2}\subset I$ have the same length. Secondly, remove an open
interval with the length $q_{2}|I_{1}^{1}|$ from each $I_{1}^{i}$ $(i=1,2)$ so
that the remaining intervals $I_{2}^{1},I_{2}^{2},I_{2}^{3},I_{2}^{4}\subset
I$ have the same length, where $|\cdot|$ means the length of the interval.
Inductively, remove an open interval with the length $q_{n}|I_{n-1}^{1}|$ from
each $I_{n-1}^{i}$ $(i=1,...,2^{n-1})$ so that the remaining intervals
$I_{n}^{1},...,I_{n}^{2^{n}}\subset I$ have the same length. For each
$n\in\mathbb{N}$, put $E_{n}=\bigcup_{i=1}^{2^{n}}I_{n}^{i}$. We define a
generalized Cantor set $E(\omega)$ for $\omega$ as
$\bigcap_{n=1}^{\infty}E_{n}$. In our previous paper ([4]), we considered the
Riemann surface $X_{E(\omega)}:=\hat{\mathbb{C}}\setminus E(\omega)$ (obtained
from the Riemann sphere $\hat{\mathbb{C}}$ by removing $E(\omega)$) and the
Teichmüller space $T(X_{E(\omega)})$ of $X_{E(\omega)}$, and proved a theorem
about the Teichmüller distance $d_{T}$ and the length spectrum distance
$d_{l}$ on $T(X_{E(\omega)})$. In this paper, we consider the Teichmüller
modular group for $X_{E(\omega)}$.

###  1.5. Our results

At first, we give a sufficient condition for Mod$(X_{E(\omega)})$ to be
uncountable. It is obtained by Proposition 1.1 above and some lemma of our
previous paper [4].

######  Theorem 1.2.

If there exists a subsequence $\\{q_{n(k)}\\}_{k=1}^{\infty}$ of
$\omega=\\{q_{n}\\}_{n=1}^{\infty}$ such that $q_{n(k)}>c$ for some constant
$c\in(0,1)$, then the Teichmüller modular group for the Riemann surface
$X_{E(\omega)}$ is uncountable. In particular, if $\inf_{n}q_{n}\neq 0$, then
Mod$(X_{E(\omega)})$ is uncountable.

Not only $\omega$ such that $\inf_{n}q_{n}\neq 0$ but also some $\omega$ such
that $\inf_{n}q_{n}=0$ satisfies the condition of Theorem 1.2. For example,
let $\omega=\\{q_{n}\\}_{n=1}^{\infty}$ be a sequence defined by

$q_{n}=\begin{cases}\frac{1}{2}&(n=2m-1;m\in\mathbb{N})\\\
(\frac{1}{2})^{n}&(n=2m;m\in\mathbb{N}).\end{cases}$

Then $\inf_{n}q_{n}=0$ and there exists a subsequence
$\\{q_{2m}\\}_{m=1}^{\infty}$ of $\omega$ such that $q_{n(k)}>1/3$.

Next, we give a sufficient condition for Mod$(X_{E(\omega)})$ to be countable.
In Theorem 1.1 of our previous paper ([4]), we considered two conditions
(I),(II) for $\omega$ such that $\inf_{n}q_{n}=0$, and showed that if $\omega$
satisfies either (I) or (II), then the two distances $d_{T}$ and $d_{L}$
define the different topologies on $T(X_{E(\omega)})$. Now, if $\omega$
satisfies (II), then it satisfies the condition of Theorem 1.2 above, too. On
the other hand, if $\omega$ satisfies (I), then it does not do so. In this
paper, our main theorem below says that if $\omega$ satisfies (I), then
Mod$(X_{E(\omega)})$ is countable:

######  Theorem 1.3.

If the sequence $\omega$ satisfying $q_{n}\cdot\log(\log(1/q_{n+1}))\to\infty$
as $n\to\infty$, then Mod$(X_{E(\omega)})$ is countable.

The sequence $\omega$ satisfying Theorem 1.3 converges to $0$ very rapidly.
The following is an example of such sequences which is a little different from
Example 1.2 of [4].

######  Example 1.4.

Take a sequence $\omega=\\{q_{n}\\}_{n=1}^{\infty}$ so that
$q_{n+1}=1/\exp(n^{1/q_{n}})$. Then

$q_{n}\cdot\log(\log(1/q_{n+1}))=q_{n}\cdot(1/q_{n})\log n=\log n\to\infty$

as $n\to\infty$.

The merit of $X_{E(\omega)}$ is the following:

######  Proposition 1.5.

For any $\omega$, the Fuchsian group $\Gamma$ corresponding to $X_{E(\omega)}$
is of the first kind.

By the property, we can construct the analytically infinite Riemann surface
for which the Teichmüller modular group is countable without caring about
twist of boundary geodesics of pairs of pants. In section 2, we prove Theorem
1.2 and Proposition 1.5. In section 3, we prove Theorem 1.3.

Acknowledgement. The author would like to thank Professor H. Shiga for his
valuable comments at the seminar.

##  2\. Proofs of Theorem 1.2 and Proposition 1.5

We decompose $X_{E(\omega)}$ into pairs of pants as we (§2 of [4]) or Shiga
(§3 of [10]) did. Recall that for a sequence
$\omega=\\{q_{n}\\}_{n=1}^{\infty}$, the generalized Cantor set $E(\omega)$ is
$\bigcap_{n=1}^{\infty}E_{n}$, where $E_{n}$ is the union of closed intervals
$\\{I_{n}^{i}\\}_{i=1}^{2^{n}}$ in $I=[0,1]$ ($n=1,2,...$). Now, for each
$n\in\mathbb{N}$ and each $i\in\\{1,...,2^{n}\\}$, let $\gamma_{n}^{i}$ be a
simple closed curve separating $I_{n}^{i}$ from other intervals
$I_{n}^{i^{\prime}}$, $i^{\prime}\in\\{1,...,2^{n}\\}\setminus\\{i\\}$. Then,
in $E(\omega)$, let $[\gamma_{n}^{i}]$ be the simple closed geodesic which is
freely homotopic to $\gamma_{n}^{i}$, where
$[\gamma_{1}^{1}]=[\gamma_{1}^{2}]$, so we write $[\gamma_{1}]$ for the
geodesic. Let $P_{1}^{i}$ be a pair of pants with boundary geodesics
$[\gamma_{1}]$, $[\gamma_{2}^{2i-1}]$ and $[\gamma_{2}^{2i}]$ $(i=1,2)$. And
for each $n\in\mathbb{N}$ and $i\in\\{1,...,2^{n}\\}$, let $P_{n}^{i}$ be a
pair of pants with boundary geodesics $[\gamma_{n}^{i}]$,
$[\gamma_{n+1}^{2i-1}]$ and $[\gamma_{n+1}^{2i}]$. (See Figure 1.) Then we
obtain a pants decomposition:
$X_{E(\omega)}=\bigcup_{n=1}^{\infty}(\bigcup_{i=1}^{2^{n}}P_{n}^{i})$. We
call this a natural pants decomposition of $X_{E(\omega)}$.

Figure 1. Pairs of pants $\bigcup_{n=1}^{3}(\bigcup_{i=1}^{2^{n}}P_{n}^{i})$

To prove Theorem 1.2, we use the lemma below, which is a part of some lemma in
our previous paper [4]. Here, $\ell_{X}(\gamma)$ means the hyperbolic length
of a curve $\gamma$ on a hyperbolic Riemann surface $X$.

######  Lemma 2.1 (Lemma 2.1 (1) of [4]).

For any $\omega$ and any $n\geq 1$,

$\ell_{X_{E(\omega)}}([\gamma_{n}^{1}])<\frac{\pi^{2}}{\tanh^{-1}q_{n}}$

holds.

######  Remark 1.

For each $n$, there are $2^{n}$ simple closed geodesics
$\\{[\gamma_{n}^{i}]\\}_{i=1}^{2^{n}}$ as the $n$-th geodesic, and in the case
where $2\leq i\leq 2^{n-1}$,
$\ell_{X_{E(\omega)}}([\gamma_{n}^{i}])<\pi^{2}/\tanh^{-1}q_{n}$ does not
hold, in general. (cf. Lemma 2.1 (2) of [4].) However, if $\omega$ is
monotonic decreasing, then
$\ell_{X_{E(\omega)}}([\gamma_{n}^{i}])<\pi^{2}/\tanh^{-1}q_{n}$ holds for any
$n\in\mathbb{N}$ and $i\in\\{1,...,2^{n}\\}$. (cf. Remark 3 of [4].)

From the above, Theorem 1.2 is immediately proved.

###### Proof of Theorem 1.2.

By Lemma 2.1, if there exists a subsequence $\\{q_{n(k)}\\}_{k=1}^{\infty}$ of
$\omega$ such that $q_{n(k)}>c$, then
$\ell_{X_{E(\omega)}}([\gamma_{n(k)}^{1}])<\pi^{2}/\tanh^{-1}c$ for
$k=1,2,...$ Hence, by Proposition 1.1 (1), Mod$(X_{E(\omega)})$ is
uncountable. ∎

Before proving Theorem 1.3, we check Proposition 1.5. The following proof is
based on the idea of Professor H. Shiga (because the original proof the author
wrote turned out to be wrong at the seminar).

###### Proof of Proposition 1.5.

Assume that the Fuchsian group $\Gamma$ is of the second kind for some
$\omega$. Then, for any point $p\in\mathbb{D}$, there exists a geodesic ray
$\hat{r}_{p}$ in $\mathbb{D}$ starting at $p$ such that
$\ell_{\mathbb{D}}(\hat{r}_{p})$ is infinite, but
$\ell_{X_{E(\omega)}}(\pi(\hat{r}_{p})\cap X_{E(\omega)})$ is finite, where
$\pi:\mathbb{D}\to\mathbb{D}/\Gamma$ is the the universal covering. Put
$r_{p}:=\pi(\hat{r}_{p})$. Let $\\{P_{n}^{*}\\}_{n=k_{p}}^{\infty}$ be a
family of the pants (of the natural pants decomposition) containing $r_{p}$
and $\\{\gamma_{n}\\}_{n=k_{p}}^{\infty}$ be a component of each $\partial
P_{n}^{*}$ intersecting $r_{p}$. Here, $E(\omega)$ is totally disconnected,
hence the diameter of $\gamma_{n}$ in $\mathbb{C}$ converges to $0$ as
$n\to\infty$, and there exists a point $p_{\infty}\in E(\omega)$ such that
$\gamma_{n}\to p_{\infty}$. Now take points $p_{1},p_{2}\in E(\omega)$ which
are contained in
$\hat{\mathbb{C}}\setminus\bigcup_{n=k_{p}}^{\infty}P_{n}^{*}$ and put
$W:=\hat{\mathbb{C}}\setminus\\{p_{\infty},p_{1},p_{2}\\}$. Since $r_{p}$ goes
to $p_{\infty}$, we have $\ell_{W}(r_{p})=\infty$. However,
$X_{E(\omega)}\subset W$, so $\ell_{X_{E(\omega)}}(r_{p})\geq\ell_{W}(r_{p})$
and this is a contradiction. ∎

##  3\. Proof of Theorem 1.3

Let $\omega=\\{q_{n}\\}_{n}^{\infty}$ be a sequence of numbers in $(0,1)$, and
let $[\gamma_{n}^{i}]$ and $P_{n}^{i}$ ($n\in\mathbb{N}$,
$i\in\\{1,...,2^{n}\\}$) be a closed geodesic and a pair of pants of
$X_{E(\omega)}$ taken in Section 2, respectively. For each $n\in\mathbb{N}$,
we take the subsurface
$X_{n}:=\bigcup_{k=1}^{n}(\bigcup_{i=1}^{2^{k}}P_{k}^{i})$ of $X_{E(\omega)}$.
Firstly, we show the following lemma.

######  Lemma 3.1.

Let $\omega$ be the sequence satisfying the condition of Theorem 1.3. Then,
for any $K$-quasiconformal automorphism $g:X_{E(\omega)}\to X_{E(\omega)}$,
there exists $n_{1}\in\mathbb{N}$ such that if $n\geq n_{1}$, then the image
$g(X_{n})$ of $X_{n}$ is freely homotopic to $X_{n}$ in $X_{E(\omega)}$, that
is, each component of $\partial g(X_{n})$ is homotopic to some component of
$\partial X_{n}$.

To prove this, we use Lemma 3.2, ([4]), Lemma 3.3 ([4]) and Lemma 3.4
(Wolpert’s Lemma).

######  Lemma 3.2 (Lemma 2.3 of [4]).

Let $\omega$ be the sequence satisfying the condition of Theorem 1.3. Then,
for any $i\in\\{1,...,2^{n}\\}$ and $j\in\\{1,...,2^{n+1}\\}$,

$\frac{\ell_{X_{E(\omega)}}([\gamma_{n+1}^{j}])}{\ell_{X_{E(\omega)}}([\gamma_{n}^{i}])}\to\infty$

as $n\to\infty$.

Below, $s_{n}^{i}(\subset\mathbb{R}\cup\\{\infty\\})$ is the shortest geodesic
segment connecting $[\gamma_{n+1}^{2i-1}]$ and $[\gamma_{n+1}^{2i}]$ in each
pair of pants $P_{n}^{i}$ with boundary geodesics
$\\{[\gamma_{n}^{i}],[\gamma_{n+1}^{2i-1}],[\gamma_{n+1}^{2i}]\\}$, and
$d(\cdot,\cdot)$ is the hyperbolic distance on $X_{E(\omega)}$.

######  Lemma 3.3 (Lemma 2.4 of [4]).

Let $\omega$ be the sequence satisfying the condition of Theorem 1.3. Then,
for any $i\in\\{1,...,2^{n}\\}$,

$\frac{d([\gamma_{n}^{i}],s_{n}^{i})}{\ell_{X_{E(\omega)}}([\gamma_{n}^{i}])}\to\infty$

as $n\to\infty$.

######  Lemma 3.4 ([11]).

Let $f:X\to X^{\prime}$ be a $K$-quasiconformal homeomorphism from a
hyperbolic Riemann surface $X$ onto another hyperbolic Riemann surface
$X^{\prime}$. And let $\gamma$ be a closed geodesic on $X$ and $[f(\gamma)]$
be the geodesic of the free homotopy class of $f(\gamma)$. Then

$\frac{1}{K}\leq\frac{\ell_{X^{\prime}}([f(\gamma)])}{\ell_{X}([\gamma])}\leq
K$

###### Proof of Lemma 3.1.

Note that for $K\geq 1$, there exists $n_{1}\in\mathbb{N}$ such that if $n\geq
n_{1}$, then

$\frac{d([\gamma_{n}^{i}],s_{n}^{i})}{\ell_{X_{E(\omega)}}([\gamma_{n}^{i}])}>K$

for any $i\in\\{1,...,2^{n}\\}$ by Lemma 3.3. Now, assume that for any
$N\in\mathbb{N}$, $g(X_{n})$ is not freely homotopic to $X_{n}$ in
$X_{E(\omega)}$ for some $n\geq N$. Then, for some $n\geq n_{1}$, $g(X_{n})$
is not freely homotopic to $X_{n}$, so there exists a component $\gamma_{n}$
of $\partial X_{n}$ such that $[g(\gamma_{n})]$ crosses $\partial X_{n}$,
where $[g(\gamma_{n})]$ is the closed geodesic freely homotopic to
$g(\gamma_{n})$. Then, the length of $[g(\gamma_{n})]$ is larger than
$d([\gamma_{n+m}^{j}],s_{n+m}^{j})$ for some $m\geq 0$ and
$j\in\\{1,...,2^{n+m}\\}$ since $[g([\gamma_{n}])$ crosses
$[\gamma_{n+m}^{j}]$ and $s_{n+m}^{j}$. Therefore,
$\ell_{X_{E(\omega)}}([g(\gamma_{n})])>d([\gamma_{n+m}^{j}],s_{n+m}^{j})>K\ell_{X_{E(\omega)}}([\gamma_{n+m}^{j}])\geq
K\ell_{X_{E(\omega)}}([\gamma_{n}^{i}])$ holds by Lemmas 3.3 and 3.2. It
contradicts Lemma 3.4. ∎

Next, we consider a Dehn twist about each component of $\partial X_{n}$ for a
sufficiently large number $n$.

######  Lemma 3.5.

Let $\omega$ be the sequence satisfying the condition of Theorem 1.3. For for
any $K$-quasiconformal automorphism $g:X_{E(\omega)}\to X_{E(\omega)}$, there
exists $n_{2}\in\mathbb{N}$ such that if $n\geq n_{2}$, on any component of
$\partial X_{n}$, $g$ does not cause nor a half Dehn twist, a Dehn twist nor
multiple twists.

We use a lemma of our previous paper [4] and a theorem of Matsuzaki ([6]). In
the following, $\eta$ is the collar function:
$\eta(x)=\sinh^{-1}(1/\sinh(x/2))$.

######  Lemma 3.6 (Lemma 2.2 of [4]).

Let $\omega=\\{q_{n}\\}_{n=1}^{\infty}$ be an arbitrary sequence of numbers in
$(0,1)$. For any $n\in\mathbb{N}$ and $i\in\\{1,...,2^{n}\\}$,

$\ell_{X_{E(\omega)}}([\gamma_{n}^{i}])>2\eta\left(\frac{\pi^{2}}{\log((1+q_{n})/(2q_{n}))}\right)$

holds.

######  Theorem 3.7 (Part of Theorem 1 of [6]).

Let $\gamma$ be a simple closed geodesic on a Riemann surface $X$ and $f$ be
$n$-times Dehn twist about $\gamma$. Then the maximal dilatation of an
extremal quasiconformal automorphism of $f$ satisfies

$K(f)\geq\sqrt{\\{(2|n|-1)\ell_{X}(\gamma)/\pi\\}^{2}+1}.$

###### Proof of Lemma 3.5.

Since $\omega$ satisfies the condition of Theorem 1.3, $q_{n}\to 0$ as
$n\to\infty$, therefore $\ell_{X_{E(\omega)}}([\gamma_{n}^{i}])\to\infty$ for
any $i$ by Lemma 3.6. Hence, for $K\geq 1$, there exists $n_{2}\in\mathbb{N}$
such that $\ell_{X_{E(\omega)}}([\gamma_{n}^{i}])>\pi K^{2}$ if $n\geq n_{2}$.
Assume that $g$ cause a half Dehn twist $f_{n}$ on some component $\gamma_{n}$
of $\partial X_{n}$ for some $n\geq n_{2}$. Then $f_{n}^{2}:=f_{n}\circ f_{n}$
is a Dehn twist about $\gamma_{n}$, so the maximal dilatation $K(f_{n}^{2})$
of $f_{n}^{2}$ is larger than $\sqrt{K^{4}+1}>K^{2}$ by Theorem 3.7. Since
$K(f_{n}^{2})\leq K(f_{n})^{2}$, we have $K<K(f_{n})$, and this is a
contradiction. From above, $g$ does not cause nor a Dehn twist nor multiple
twists. ∎

Finally we prove the main theorem.

###### Proof of Theorem 1.3.

For $K$-quasiconformal automorphism $g:X_{E(\omega)}\to X_{E(\omega)}$, put
$N:=\max\\{n_{1},n_{2}\\}$, where $n_{1},n_{2}$ are numbers of Lemma 3.1 and
Lemma 3.5, respectively. Then, for any $n\geq N$, $g(X_{n}\setminus X_{n-1})$
is homotopic to $X_{n}\setminus X_{n-1}=\bigsqcup_{i=1}^{2^{n}}P_{n}^{i}$ in
$X_{E(\omega)}$, and on any component of $\partial X_{n}$, $g$ does not cause
nor a half Dehn twist, a Dehn twist nor multiple twists.

Now, for an arbitrary $K\in\mathbb{N}$, let Mod$(X_{E(\omega)})_{K}$ be a
subset of the Teichmüller modular group Mod$(X_{E(\omega)})$ such that each
element has $K$-quasiconformal automorphism $g$ as a representative. From
above, Mod$(X_{E(\omega)})_{K}$ is embedded in the reduced Teichmüller modular
group $\text{Mod}^{\sharp}(X_{N})$ for the bordered Riemann surface $X_{N}$.
Indeed, $g$ is determined by $g|_{X_{N}}$, that is, for quasiconformal
automorphisms $g_{1},g_{2}$ of $X_{E(\omega)}$, if
$g_{1}|_{X_{N}}=g_{2}|_{X_{N}}$, then $[g_{1}]=[g_{2}]$.

Since $X_{N}$ is topologically finite, $\text{Mod}^{\sharp}(X_{N})$ is
finitely generated, thus countable. Hence Mod$(X_{E(\omega)})_{K}$ is
countable for any $K\in\mathbb{N}$, and Mod$(X_{E(\omega)})$ is countable,
too. ∎

## References

  * [1] A. Basmajian and D. Šarić, Geodesically Complete Hyperbolic Structures; Mathematical Proceedings of the Cambridge Philosophical Society. 166(2) (2019), 219-242.
  * [2] A. L. Epstein, Effectiveness of Teichmüller modular groups; In the tradition of Ahlfors and Bers, Contemporary Math. 256, AMS, (2000), 69-74.
  * [3] E. Kinjo, On the length spectrum Teichmüller spaces of Riemann surfaces of infinite type; Conform. Geom. Dyn. 22 (2018), 1-14.
  * [4] E. Kinjo, On the length spectrums of Riemann surfaces given by generalized Cantor sets; Kodai Math. J. 47 (2024), 34–51.
  * [5] V. Markovic, Biholomorphic maps between Teichmüller spaces; Duke Math.J. 120 (2003), 405–431.
  * [6] K. Matsuzaki, The infinite direct product of Dehn twists acting on infinite dimensional Teichmüller spaces; Kodai Math. J. 26 (2003), 279-287.
  * [7] K. Matsuzaki, A countable Teichmüller modular group; Trans. Amer. Math. Soc. 357 (2005), 3119–3131.
  * [8] K. Matsuzaki, Infinite-dimensional Teichmüller spaces and modular groups; Handbook of Teichmüller theory (A. Papadopoulos, ed.), Volume IV, EMS Publishing House, Zürich (2014), 681–716.
  * [9] H. Shiga, On a distance by the length spectrum on Teichmüller space; Ann. Acad. Sci. Fenn. Series A, I Math. 28 (2003), 315–326.
  * [10] H. Shiga, On the quasiconformal equivalence of dynamical Cantor sets; J. d’Analyse Math. 147 (2022), 1-28.
  * [11] S. Wolpert, The length spectrum as moduli for compact Riemann surfaces; Ann. of Math. 109 (1979), 323-351.


# Task-Aware Encoder Control for Deep Video Compression

Xingtong Ge1,2 Jixiang Luo2 Xinjie Zhang3 Tongda Xu4 Guo Lu5
Dailan He6 Jing Geng122footnotemark: 2 Yan Wang4 Jun Zhang3 Hongwei Qin2
1Beijing Institute of Technology 2SenseTime Research
3The Hong Kong University of Science and Technology
4Institute for AI Industry Research (AIR), Tsinghua University
5Shanghai Jiaotong University 6The Chinese University of Hong Kong
Work was done when Xingtong Ge interned at SenseTime
<EMAIL_ADDRESS>author<EMAIL_ADDRESS>
###### Abstract

Prior research on deep video compression (DVC) for machine tasks typically
necessitates training a unique codec for each specific task, mandating a
dedicated decoder per task. In contrast, traditional video codecs employ a
flexible encoder controller, enabling the adaptation of a single codec to
different tasks through mechanisms like mode prediction. Drawing inspiration
from this, we introduce an innovative encoder controller for deep video
compression for machines. This controller features a mode prediction and a
Group of Pictures (GoP) selection module. Our approach centralizes control at
the encoding stage, allowing for adaptable encoder adjustments across
different tasks, such as detection and tracking, while maintaining
compatibility with a standard pre-trained DVC decoder. Empirical evidence
demonstrates that our method is applicable across multiple tasks with various
existing pre-trained DVCs. Moreover, extensive experiments demonstrate that
our method outperforms previous DVC by about 25% bitrate for different tasks,
with only one pre-trained decoder.

##  1 Introduction

Over the past decades, video analysis techniques have proliferated across a
variety of fields, including smart cities, autonomous vehicles, and traffic
surveillance. For these applications, videos are often compressed before being
transmitted to cloud-based systems for further machine vision analyses, such
as object detection and tracking. However, current video compression
techniques, which range from traditional codecs [43, 34, 2] to recent
learning-based codecs [26, 25, 15, 19, 30], primarily cater to the human
visual system, as shown in Fig. 1(a). This specificity leads to
inefficiencies, as machine vision tasks typically focus on selective semantic
details and regions within the video frames, rather than the whole frames.

To tackle this problem, recent studies [39, 1, 13, 21, 41, 45, 11, 4, 48, 12]
have explored scalable compression for multiple tasks through different layers
in image and video coding. They propose a base layer dedicated to machine
vision, with an enhancement layer containing additional information for human
vision. Additionally, the latest work [37] has tried to compress semantic
features at the encoder side, which serve as side information to supplement
decoded images with more semantic details for machine vision at lower
bitrates. Despite these advances, these methods require individually
customized codecs when applied to different downstream tasks, as the “One-to-
one Codecs” shown in Fig. 1(c). In contrast, traditional codecs can control
the encoding process for different objectives, like PSNR and SSIM
optimization, using a single decoder. Inspired by this feature, in this paper,
we focus on how to use one pre-trained DVC-decoder to support both human and
multiple machine vision tasks.

Figure 1: (a) Mainstream video codec that serves the human viewing. (b) Our
controlled video codec for machine vision with fixed decoder. (c) Other video
codecs for machine vison with one-to-one encoders and decoders.

In both residual [25, 15, 16] and conditional [19, 30, 20] DVC series, the
encoded bitstreams predominantly comprise two parts: motion and
residual/contextual information. It is observed that a substantial portion of
the bitrate, often exceeding 80% in high bitrate models, is allocated to the
residual/contextual information. This allocation is primarily for high-quality
frame reconstruction. However, this approach is inappropriate for downstream
vision tasks such as tracking and detection, which primarily concentrate on
the objects and their movements rather than the full frames.

To mitigate this issue, we introduce a dynamic vision mode prediction (DVMP)
module specifically tailored for machine vision tasks. This module optimizes
the entropy coding process by dynamically predicting the skip/no-skip coding
mode for each feature element, based on its relevance to machine vision.
Specifically, it takes hyperprior data from motion or residual/contextual
features within the DVC framework as the input and assesses the utility of
each feature element for machine vision. By replacing non-essential elements
with their predicted mean value derived from the hyperprior network, our
method effectively reduces bit consumption and circumvent the entropy decoding
step for these elements to expedite the decoding process. In this case, we
generate a novel frame type that significantly reduces bitrate usage compared
to traditional P frames in the Group of Pictures (GoP) structure, while
maintaining the information valuable for machine vision tasks. This new frame
is designated as the $P_{m}$ frame.

At the same time, it is important for DVC to maintain reconstruction quality
because there exists significant reference relationship between frames, which
is a crucial difference between image and video compression. Since our $P_{m}$
frame degrades the fidelity of reconstruction compared to the $P$ frame, we
explore the reorganization of the Group of Pictures (GoP) structure. To be
specific, we firstly explore to use a hand-crafted GoP structure, which
already achieves significant rate-precision improvements. In the revised
structure, both $P$ frames and $P_{m}$ frames derive their prediction from the
preceding $I/P$ frame. Further, we introduce a GoP selection network, which
can dynamically determine the GoP structure during the encoding process and
achieves a better rate-precision trade-off.

We choose three representative residual and conditional DVC methods as
touchstones in our proposed video coding for machine vision framework. It’s
worth mentioning that our method utilizes pre-trained DVCs without altering
the weights of the original encoders and decoders. Our control happens at
encoding stage, leaving the decoders’ architecture intact. Moreover, when
human viewing is required, we can switch back to the original encoding
procedure to restore reconstruction performance. In essence, using the
proposed method, we can control the encoder of a DVC to adapt for both machine
and human vision requirements.

The main contributions of our work are summarised as follows: (1) Built upon
the mainstream DVC codecs, we propose a novel video coding for machine vision
framework that controls the encoder to adapt for different downstream vision
tasks, such as video object detection and multi-object tracking. (2) We employ
a dynamic vision mode prediction approach to refine the original P frame,
effectively reducing the bitrate while preserving critical information
pertinent to vision tasks. Furthermore, we utilize a GoP selection strategy to
dynamically forecast the coding GoP structure during the encoding stage, which
controls the bitstream for better rate-precision trade-off in downstream
vision tasks. (3) Experiments show that our controlling method is novel and
flexible for different DVC codecs, achieving up to more than 25% bitrate
savings in various downstream tasks.

##  2 Related Works

Figure 2: (a) Overview of our “Controlling DVC for Machine” framwork. Given an
input GoP, we firstly use GoP Selection network to predict the GoP sructure,
then the predicted structure controls the encoding procedure to encode input
frames for machine vision tasks. (b) The “0” element controls encoder to use
DVMP. (c) The GoP Selection network, including the pre-analysis stage and GoP
prediction stage.

###  2.1 Video Compression

Previous video codecs are designed to remove spatial-temporal redundancies
effectively. Traditional video coding standards, including H.264/AVC [43],
H.265/HEVC [34], and H.266/VVC [2], significantly increase the compression
efficiency of images and videos. Recently, a number of learning-based video
codecs [26, 25, 15, 19, 30, 36] have been proposed, designed based on residual
coding or conditional coding, achieved better and better pixel-wise signal
quality metrics, e.g., PSNR and MS-SSIM [42], which mainly serve the human
visual experience. Recently, there are also some generative video coding
methods [46, 28] that mainly consider visual comfort and perceptual quality.

###  2.2 Compression for Machine Vision

To facilitate the efficiency of machine vision tasks, early research works
were devoted to extracting visual features from signals. Early standards, such
as CDVA [8] and CDVS [7], suggest the pre-extraction and transportation of
image keypoints to facilitate image indexing or retrieval tasks. With the
development of deep learning, some studies began to explore connections
between compression and downstream tasks, some studies [3, 39, 1, 13, 21, 41,
45, 11, 4, 48] also focus on the joint optimization of image compression and
downstream machine vision tasks by introducing a rate-distortion optimization
strategy guided by downstream tasks or by adding a task-specific feature
encoding stream. For instance, Torfason et al. [39] introduced a method for
executing image understanding tasks, such as classification and segmentation,
on compressed outputs from learning-based image compression techniques.
Furthermore, Lu et al. [27] enhanced traditional codecs with a preprocessing
step, thereby improving codec performance for downstream vision tasks. The
field has also witnessed the emergence of self-supervised representation
learning methods aimed at deriving compact semantic representations. Dubois et
al. [9] presented a theoretical framework suggesting that the distortion term
in the lossy rate-distortion trade-off for image classification could be
approximated by a contrastive learning objective. Feng et al. [10] proposed a
method to learn a unified feature representation for AI tasks from unlabeled
data. These methodologies necessitate fine-tuning the downstream models to
adapt to the learned features.

Despite advancements in image coding for machine applications, reconstructing
high-fidelity video from extracted features remains a formidable challenge due
to the critical inter-frame reference relationships. Addressing the
requirements of both machine and human vision, Tian et al. [37] introduced a
self-supervised edge representation as a semantic intermediary, which
preserves the video’s semantic structure. Furthermore, Lin et al. [22]
developed a scalable video coding framework tailored for machines, segregating
semantic features for machine analysis and human viewing into separate
bitstreams.

Most existing approaches [38, 22, 37]necessitate the use of distinct encoders
and decoders for various tasks, which complicates and burdens the coding
pipelines. Our methodology advances the “Coding for Machine” paradigm by
focusing on the encoder side. We present a versatile framework capable of
accommodating both human and machine vision demands without the need to modify
the pre-trained decoders of existing video codecs.

###  2.3 Compressed Video Analysis/Understanding

There are also amounts of works performing video analysis tasks [24, 40, 18],
such as image recognition, action recognition [44, 32] and multiple object
tracking (MOT) [17, 6], in the compressed video domain. However, these methods
focus on developing video analysis models that better leverage the partially
decoded video stream, such as the motion vector. In contrast, our work focuses
on the coding procedure, specifically the encoding procedure.

##  3 Method

###  3.1 Overview

Our coding framework is illustrated in Fig. 2 (a). Let
$\mathcal{X}=\\{\boldsymbol{x}_{1},\cdots,\boldsymbol{x}_{T}\\}$ represent the
video sequence. The framework categorizes the $P$ frames into two types:
original $P$ frames and machine vision-specific $P_{m}$ frames. The
categorization of frame $\boldsymbol{x}_{t}$ is determined by the GoP
Selection Network, where a value of $1$ signifies a $P$ frame and $0$ a
$P_{m}$ frame. The $P_{m}$ frames, designed to reduce the bitrate while
preserving critical information for future vision tasks, are generated from
the preceding $P$ frames using Dynamic Vision Mode Prediction (DVMP). To
maintain the quality of the decoded video sequence, each $P$ frame is computed
from the previous $P$ frame with superior visual quality, rather than from a
$P_{m}$ frame. In scenarios requiring coding for downstream vision tasks, a
machine-centric control applies the GoP selection and DVMP, altering the
encoding structure to ”$I,P_{m},P,\cdots$”. Conversely, for human viewing, a
human-centric control restores the encoding structure to its original form,
”$I,P,P,P,\cdots$”.

###  3.2 Dynamic Vision Mode Prediction

In our coding framework, we aim to minimize the video sequence bitrate.
Initial analyses of residual video codecs reveal that they allocate a
substantial portion of the bitrate—exceeding 80% in high bitrate
configurations—to encode and transmit residual information. This approach
ensures high-quality visuals for human perception, as measured by metrics such
as PSNR or MS-SSIM [42]. However, this method introduces significant
redundancy, especially since downstream tasks like video object detection or
tracking predominantly concentrate on specific regions of interest rather than
the entirety of the frame. Consequently, it is crucial for our study to
develop an approach that reduces this redundancy in the coding bitstream.

Figure 3: Hyper-prior guided Dynamic Vision Mode Prediction network.

Inspired by Hu et al. [14], who developed a mode prediction technique
specifically designed for the human visual system within a slimmable encoder
and decoder, we introduce the Dynamic Vision Mode Prediction (DVMP) module.
This innovative module autonomously selects the optimal coding mode and
decides whether each feature element should be encoded and transmitted, as
illustrated in Fig. 3.

To elucidate the functionality of our proposed Dynamic Vision Mode Prediction
(DVMP) module, we utilize the encoded residual feature $m_{t}$ as an
illustrative example. Assume $m_{t}$ possesses dimensions $c\times h\times w$
, indicating $c$ channels each with a spatial dimension of $h\times w$. The
hyperprior network then predicts the mean and variance for each element,
resulting in dimensions of $2c\times h\times w$. The mode prediction network’s
architecture, detailed in the upper branch, consists of two convolution layers
and three ResBlocks. To render the “mode prediction” process differentiable,
we employ the Gumbel Softmax technique during training to ascertain the skip
mode for each encoded residual feature element. This module generates a binary
mask for the feature map $m_{t}$, where a “1” denotes elements imperative for
machine vision that require entropy coding, and a ”0” signifies elements
either irrelevant for machine vision or accurately predictable by the
hyperprior network, thus streamlining the entropy coding process and bitrate
reduction. Note that the showed architecture is not suitble for prior models
with autoregressive components, like which in DCVC [19]. However, we can
extend the module into an autoregressive way to enable the support for DCVC.
The detailed structure is shown in the supplementary material.

To optimize the DVMP module, the loss function can be given by:

$\displaystyle\mathcal{L}_{m}$ $\displaystyle=R+\lambda_{m}\mathcal{L}_{t}$
(1)

where $R$ denotes the coding bitrate and $\mathcal{L}_{t}$ denotes the loss of
downstream vision task, respectively. $\lambda_{m}$ is a hyper-parameter used
to control the trade-off.

Figure 4: Different coding GoP structures. The original structure consists of
I and P frames. In the middle, the hand-crafted structure for machine vision
consists of I, P and $P_{m}$ frames which are arranged alternately. In the
predicted (GoP selected) structure, the type of frames are predicted by the
GoP selection network, targeting on better bitrate and machine vision
performance trade-off.

###  3.3 DivGoP → GoP Selection

Obtaining the $P_{m}$ frames, we firstly use a hand-crafted structure in a
GoP, where the $P_{m}$ frames and $P$ frames are arranged alternately
(referred as “DivGoP” and shown in Fig. 4). It is observed that this
arrangement can bring a certan degree of bitrate saving in detection, since
the $P_{m}$ frames cost much less bitrate than $P$ frames while maintaining
the motion and machine vision information. Experimental findings, as depicted
in the “FVC DivGoP” curve within Fig. 5, demonstrate that the
”$I,P_{m},P,\cdots$” structure markedly achieve about 18.6% bitrate saving.

The “DivGoP” configuration, while methodically structured, may not be
universally optimal due to the variability in motion and object information
across different videos, which could necessitate distinct GoP structures. For
instance, videos characterized by rapid target movement might demand a higher
frequency of $P$ frames to preserve the quality of reconstruction, as an
excess of $P_{m}$ frames may lead to degradation that negatively affects
subsequent frames. Conversely, in scenarios where the camera remains static or
the target movement is minimal, the introduction of more $P_{m}$ frames could
be feasible without significantly impacting the reconstruction quality of
following frames. Stimulated by this notion, our research endeavors to tailor
the GoP structure dynamically, enhancing codec performance for machine vision
tasks. To approach this issue, we adopt FVC [15] as the foundational codec,
select a GoP size of 10, and address the detection task utilizing the MOT
Dataset. The optimization problem is thus defined:

$\displaystyle\underset{\theta}{\text{arg min}}\quad
R(\theta)+\lambda_{p}L_{\text{det}}(\theta)$ (2)

where $R$ denotes the bitrate, $L_{det}$ denotes the detection loss,
$\lambda_{p}$ is a weighting factor that balances the effects of bitrate and
detection loss, and $\theta$ denotes the setting of GoP structure here.

Obtaining a GoP, each of the 9 predicted frames can be either $P$ frame or
$P_{m}$ frame, which can be dynamically decided. To explore the optimal GoP
structure for target function 2, and evaluate the performance gap between
DivGoP and optimal GoP structure, we use a deep-first-searching (DFS)
algorithm, where we use different $\lambda$ for different bitrate models. The
DFS algorithm searches for the structure that makes the target function
smallest for each GoP. Results are shown in the “FVC DFS” curve of Fig. 5. The
application of the DFS algorithm led to a substantial increase in Bpp-mAP
performance,, achieving about 32.3% bitrate saving in terms of the BD-RATE
metric, while the “DivGoP” method achieves about 18.6% bitrate saving. Similar
conclusions can also be observed on DCVC [19] and TCM [30], reinforcing our
hypothesis that distinct videos necessitate unique GoP structures tailored for
machine vision tasks. However, the DFS algorithm is extreamly time consuming,
and the labels do not exist during real inference process, making the DFS not
avaliable to be applied in real time encoding. How to further find an
“adaptive GoP structure” in an available time complexity is worth exploring.

From this perspective, we introduce our GoP selection method to adaptively
predict the GoP structure at encoding stage, which has a low time complexity
and is available in application. Illustrated in Fig. 2 (c), our GoP selection
network operates in two phases: the pre-analysis and the GoP prediction
stages. Initially, input frames undergo pre-analysis, where a detector
identifies objects and generates an overlapping mask for each frame, while the
RAFT [35] optical flow estimation method calculates optical flow features for
each predicted frame using its adjacent predecessor. These features are then
refined by masking with the object bounding box mask to produce masked optical
flow features, alongside motion feature priors (mean and variance, averaged by
channel) derived from the codec’s motion encoder. Subsequently, these
processed features are input into the GoP prediction network, which employs a
feature extractor with convolution blocks for downsampling and aggregation,
followed by adaptive average pooling and linear layers that output a
normalized $s_{logit}$ for this GoP, which comprises two elements that sum to
one. The averaged detection confidence information from the detector is also
injected into the linear layers as meta data.

Upon acquiring the $s_{\text{logit}}$, we employ the Gumbel softmax sampling
technique throughout the training phase to iteratively generate a binary value
for each frame. This process culminates in the formation of a GoP structure
vector, where the binary value—$0$ for a $P_{m}$ frame and $1$ for a $P$
frame—specifies the type of each frame within the GoP. During inference stage,
we apply a softmax operation on $s_{\text{logit}}$ to derive probabilities,
which are then used to evenly distribute $1$s and $0$s, thereby determining
the actual encoded GoP structure. Similar to our mode prediction network, the
loss function for optimizing GoP structure selection network is given by:

$\displaystyle\mathcal{L}_{g}$
$\displaystyle=\bar{R}+\lambda_{g}\bar{\mathcal{L}_{t}}$ (3)

where the $\bar{R}$ and $\bar{\mathcal{L}_{t}}$ denote the GoP-wise average
coding bitrate and average loss of downstream vision task.

(a)

(b)

Figure 5: Left: Using DFS to search for the near-optimal GoP structure for
Bpp-mAP. Right: Results of simply fine-tuning FVC

##  4 Experiments

(a)

(b)

(c)

(d)

(e)

(f)

Figure 6: (a) Bpp-MOTA curves on MOT Dataset. (b) Bpp-mAP curves on MOT
Dataset. (c) Bpp-MOTP curves on MOT Dataset. (d) Bpp-FN curves on MOT Dataset.
(e) Bpp-mAP50 curves on ImageNet VID Dataset using YOLOV [31]. (f) Bpp-Top1
Accuracy curves on UCF101 Dataset.

In this section, we describe the dataset settings, downstream task models, and
evaluation metrics. We then provide empirical evidence underscoring the
effectiveness of our proposed method across multiple tasks and codecs. Lastly,
we validate the contributions of the mode prediction and GoP selection modules
to performance improvements through ablation studies.

Datasets. For multi-object tracking (MOT) task, we choose MOT17 dataset. For
video object detection (VOD) task, we adopt the Imagenet VID dataset [29] for
training and evaluation. For video action recognition (VAR) task, we adopt
UCF101[33] dataset. For the data processing procedure, we downsample the short
side of the evaluation videos to $512$ for MOT17 dataset and keep the original
frame ratios for both training and evaluation. For the evaluation of VOD, we
follow the most VOD works to resize the frames to $512\times 512$ for training
and $576\times 576$ for evaluation.

Downstream Task Models. For MOT task, we adpot the popular ByteTrack [47]
method as the backbone model. The model weights are provided by their official
framework. For VOD task, we adpot the detector YOLOV [31], and the model
weights are also provided by its official framework. For the VAR task, we
adpot the TSM [23] model as the recognition network, the weights are provided
by mmaction2 [5]. During training and evaluation of our proposed framework,
the weights of the downstream task models are fixed, without any fine-tuning.

Evaluation Metrics. For compression, We use bpp (bits per pixel) to measure
the average number of bits for one pixel. For the MOT task, we adopt the MOTA
(multiple object tracking accuracy), MOTP (multiple object tracking
precision), FN (false negative detection number) and mAP (mean average
precision) metrics to evaluate the tracking and detection performance. For the
VOD task, we adopt the mAP50 metric to evaluate the detection performance,
following most VOD methods. For the VAR task, we adopt the Top1 Accuracy as
the performance indicator.

Implementation Details. Our whole framework is implemented on PyTorch with
CUDA support and trained on eight V100 or A100 GPUs. We use the Adam optimizer
by setting the learning rate as 0.0001, $\beta_{1}$ as 0.9 and $\beta_{2}$ as
0.999, respectively. The whole training process has the follow stages:
Firstly, we use the loss function 1 to train the mode prediction network and
obtain the $P_{m}$ frames. This stage we use single frame training. Note that
for TCM, DVMP is applied to both motion and contextual information. For DCVC
and FVC, DVMP is exclusively utilized for motion information, and all
residual/contextual information is omitted. After that, we introduce the GoP
selection network in a GOP length unit. Using the GoP size of 32 on MOT and
VAR tasks, we devide the GoP into 2 mini GoPs with size of 16, and the
predicted vector of each mini GoP has a length of 15. For VOD task, we
directly set the GoP size as 16. Using multi-frame training and loss function
3, we fix the weights of mode prediction network and train GoP selection
network for about 100 epochs on MOT half-train dataset and 20 epochs on
ImageNet VID training dataset, respectively for multi-object tracking and
video object detection tasks. The whole training procedure takes about 2 days
for each task.

###  4.1 Multi-Object Tracking

In Fig. 6, we present a comparison between our method and various codecs,
specifically DCVC, FVC, and TCM. It is evident that our approach yields a
markedly improved trade-off in terms of Bpp-MOTA, Bpp-mAP, Bpp-MOTP and Bpp-FN
metrics. When compared to the baseline codecs FVC, DCVC and TCM, our control
method results in bitrate savings of approximately 27.54%, 41.82% and 12.64%,
respectively, for equivalent MOTA metrics. With DCVC established as the anchor
baseline, Table 1 displays the BD-RATE values, where our controlled TCM,
controlled DCVC and controlled FVC codecs demonstrate the best, second-best
and third-best performance, respectively. Additionally, we contrast our method
with the task-oriented “One-to-one Codecs” of VCM, with detailed results
provided in the supplementary materials.

###  4.2 Video Object Detection

For the evaluation of the video object detection task, we juxtapose our method
with established codecs such as FVC, and TCM, with the comparative results
illustrated in Fig. 6(e). Our method is noted to achieve a superior trade-off,
particularly in the Bpp-mAP50 metric, when applied to the downstream model
YOLOV. For instance, in comparison to the FVC and TCM baseline codecs, our
method facilitates bitrate reductions of approximately 21.76% and 20.42%,
respectively, while maintaining the same mAP50 metric. The uncompressed video
represents the upper bound for the mAP50 metric, recorded at 76.4, and is
delineated by the grey line within the figure.

(a)

(b)

Figure 7: Video Reconstruction Quality in terms of Bpp-PSNR on HEVC Class B dataset (Left) and Class C dataset (Right) Table 1: BD-RATE results on multi-object tracking. Method | MOTA | mAP | mAP50 | MOTP | FN | Average
---|---|---|---|---|---|---
DCVC [19] | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
Ours+DCVC | -41.82 | -39.43 | -40.60 | -37.73 | -41.74 | -40.26
FVC [15] | -5.32 | -9.75 | -14.53 | -1.39 | -6.20 | -7.44
Ours+FVC | -34.28 | -31.27 | -32.09 | -35.02 | -32.89 | -33.11
TCM [30] | -25.19 | -32.34 | -31.02 | -39.85 | -26.44 | -30.97
Ours+TCM | -40.82 | -45.10 | -46.15 | -51.66 | -38.98 | -44.54
HEVC [34] | -33.88 | -31.35 | -40.43 | 15.80 | -34.02 | -24.78

###  4.3 Video Action Recognition

We conducted an evaluation of our method on the UCF101 dataset, benchmarking
against TCM and FVC codecs. The outcomes of this comparison are depicted in
Fig. 6(f). For the task of video action recognition, rather than initiating a
new set of network trainings, we employed pre-trained mode prediction networks
previously utilized for the MOT task and use the “DivGoP” structure for
validation. The original videos were compressed using the various codecs, and
the resulting decoded videos served as inputs to the TSM model as cited in Lin
et al. [23]. The experimental results indicate that our method maintains
efficacy in this context, securing approximately 21.81% and 26.03% in bitrate
savings for TCM and FVC, respectively.

###  4.4 Video Reconstruction Quality

When it comes to the performance of reconstructing the video for human
viewing, the situation is straightforward for conventional video codecs, no
matter learned or traditional ones: just encode the frames, transmit and
decode the compressed bitstream, and measure their quality using appropriate
metrics (PSNR, MS-SSIM, etc). In our method, there’re two options: Control the
encoding procedure for human viewing, which is the ideal way for best frame
reconstruction quality, or just continue the encoding procedure used for
machine vision tasks.

Note that our method does not change the weights of the original encoder and
decoder in pre-trained DVC, so using this option can directly recover the
human viewing quality to original best performance of the codecs. On the other
hand, we also evaluate the human viewing quality of the encoded bitstream for
machine, using “DivGop” encoding structure on HEVC Class B and Class C test
datasets. These are referred to as “FVC DivGoP” and “TCM DivGoP” in Fig. 7.
Results show that our control for machine does decrease the video
reconstruction quality of the original TCM codec by about 1dB in PSNR.
Besides, it is also observed that the PSNR of residual codec FVC drops more
sharply than that of conditional codecs like TCM. However, these are
acceptable since we can always adjust the encoder to compress the video for
human viewing when it’s needed.

###  4.5 Ablation Study

(a)

(b)

Figure 8: Left: Ablation study for Mode Prediction network. Right: Ablation
study for GoP Selection network (a) Figure 9: Visualization results of the
mode prediction networks.

Mode Prediction. In our method, we propose mode prediction network to decide
the coding modes of motion and residual/contextual feature elements for
machine vision, mainly targeting on reducing the bitrate and keep the crucial
information for important objects in the frame. Moreover, mainstream DVC,
including FVC, DCVC, TCM and others, use motion estimation and warping
operation to generate a rough predicted frame, which we call $P_{r}$ frame
here. The $P_{r}$ frame also meets our requirement of “consuming lower code
streams but retaining object and motion information” to a certain extent. So
comparing it with the $P_{m}$ frame used in our method is also a point worthy
of attention. In “DivGoP” structure, we use the $P_{r}$ frame to replace the
original $P_{m}$ frames, and the GoP structure will be
$[I,P_{r},P,P_{r},...]$. As shown in the left figure in Fig. 8, based on DCVC
and FVC, we compare the structure using $P_{r}$ frames with the original
“DivGoP” structure using $P_{m}$ frames, corresponding to the “DCVC/FVC DivGoP
w $P_{r}$” and “DCVC/FVC DivGoP” curves respectively. It is observed that the
“DivGoP w $P_{r}$” curve also achieves about 28.68% and 16.24% bitrate savings
for DCVC and FVC, respectively. But our “DivGoP” curve shows better trade-off,
by achieving 32.67% and 19.43% bitrate savings.

GoP Selection. We prove the effectiveness of Gop Selection module in terms of
rate-precision performance on MOT dataset. As shown in the right figure in
Fig. 8, we compare the Bpp-mAP performance of “DCVC/FVC Gop Selection” and
“DCVC/FVC DivGoP” structure on MOT Datast. Using DCVC as an example,
experiment results show that our Gop Selection achieves about 39.43% bitrate
saving, which does show better trade-off than that 32.67% bitrate saving of
hand-crafted “DivGoP” structure.

Simply fine-tuning DVC for machine vision tasks. We employed the loss function
$L_{f}=R+\lambda_{1}MSE+\lambda_{2}L_{det}$ to fine-tune the FVC encoder and
the entire FVC codec for the MOT task. As shown in Fig. 5 right, while simple
fine-tuning improves accuracy, it also notably increases the bitrate. This
phenomenon could be attributed to the degradation in the fidelity of decoded
frames when optimized for downstream tasks. Due to the inter-frame reference
nature of video codecs, this degradation is propagated across frames,
resulting in a high bitrate consumption. This observation aligns with findings
reported in DeepSVC [22], indicating that such task-specific performance can
not be achieved with simply finetuning a DVC. This also reflects the
superiority of our method.

Visualization Results. We present the visualization results of our mode
prediction network, as shown in Fig. 9. The first column shows the input
images, the second column shows the skip mode masks for contextual features
(averaged by channel), and the third and fourth columns are masked motion and
contextual features, respectively, where the red color indicates large value
and green color indicates small value. It is observed that the predicted
coding mode mask for contextual features kept the regions of the small objects
in the frames, and masked the conspicuous objects which are easy to recognize
and the background regions. This shows the network’s preference for objects in
the frame. As for the motion information, it is observed that the predicted
masks kept most of the elments of moving objects, which are crucial for
detection and tracking.

Complexity and Encoding Latency Analysis. The parameter numbers of our dynamic
vision mode prediction network and GoP selection network are 1.48M and 5.36M.
Since our method controls the video encoder during the encoding phase, such an
operation brings encoding latency. To measure the impact on the encoding
phase, we include a comparison of average encoding latency versus performance
on MOT Dataset ($960\times 512$) in Tab. 2. While our method bring an increase
in encoding latency (1.738s vs. 1.675s), the improvement in R-P performance is
substantial. Besides, the decoding procedure and decoder side models are not
changed, which brings convenience to the deployment of the decoder and support
for multiple tasks.

Table 2: Encoding Latency vs. Performance. Method | Encoding Latency(s) | BD-BR Performance
---|---|---
DCVC | 1.675 | 0.0
Ours+DCVC | 1.738 | -40.26
FVC | 0.116 | -7.44
Ours+FVC | 0.159 | -33.11

##  5 Conclusion

In this paper, we propose a flexible framework for deep video compression
models, where the pre-trained encoders can be controlled to change the
encoding pipeline for machine vision tasks, achieving significant better bpp-
precision trade-off without changing the decoders or decoding procedure. The
controlling methods are attributed to two main developments, including dynamic
vision mode prediction network and GoP structure selection network. Experiment
results show that our framework is general for residual and conditional deep
video codecs and different downsteam vision tasks like detection, tracking and
action recognition. Our comprehensive evaluation, conducted against existing
deep video codecs, confirms the superior performance of our models,
establishing a new benchmark for future deep video coding for machine. The
effectiveness of each component is clearly evidenced through extensive
ablation studies.

Acknowledgments. This work was supported by the National Natural Science Fund
of China (Project No. 42201461) and the General Research Fund (Project No.
16209622) from the Hong Kong Research Grants Council.

## References

  * Akbari et al. [2019] Mohammad Akbari, Jie Liang, and Jingning Han.  Dsslic: Deep semantic segmentation-based layered image compression.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 2042–2046. IEEE, 2019.
  * Bross et al. [2021] Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J. Sullivan, and Jens-Rainer Ohm.  Overview of the versatile video coding (vvc) standard and its applications.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 31(10):3736–3764, 2021.
  * Chen et al. [2020] Zhuo Chen, Kui Fan, Shiqi Wang, Lingyu Duan, Weisi Lin, and Alex Chichung Kot.  Toward intelligent sensing: Intermediate deep feature compression.  _IEEE Transactions on Image Processing_ , 29:2230–2243, 2020.
  * Choi and Bajić [2022] Hyomin Choi and Ivan V Bajić.  Scalable image coding for humans and machines.  _IEEE Transactions on Image Processing_ , 31:2739–2754, 2022.
  * Contributors [2020] MMAction2 Contributors.  Openmmlab’s next generation video understanding toolbox and benchmark.  https://github.com/open-mmlab/mmaction2, 2020.
  * Dai et al. [2022] Yan Dai, Ziyu Hu, Shuqi Zhang, and Lianjun Liu.  A survey of detection-based video multi-object tracking.  _Displays_ , page 102317, 2022.
  * Duan et al. [2015] Ling-Yu Duan, Tiejun Huang, and Wen Gao.  Overview of the mpeg cdvs standard.  In _2015 Data Compression Conference_ , pages 323–332, 2015.
  * Duan et al. [2019] Ling-Yu Duan, Yihang Lou, Yan Bai, Tiejun Huang, Wen Gao, Vijay Chandrasekhar, Jie Lin, Shiqi Wang, and Alex Chichung Kot.  Compact descriptors for video analysis: The emerging mpeg standard.  _IEEE MultiMedia_ , 26(2):44–54, 2019.
  * Dubois et al. [2021] Yann Dubois, Benjamin Bloem-Reddy, Karen Ullrich, and Chris J Maddison.  Lossy compression for lossless prediction.  _Advances in Neural Information Processing Systems_ , 34:14014–14028, 2021.
  * Feng et al. [2022] Ruoyu Feng, Xin Jin, Zongyu Guo, Runsen Feng, Yixin Gao, Tianyu He, Zhizheng Zhang, Simeng Sun, and Zhibo Chen.  Image coding for machines with omnipotent feature learning.  In _European Conference on Computer Vision_ , pages 510–528. Springer, 2022.
  * Fischer et al. [2021] Kristian Fischer, Fabian Brand, and André Kaup.  Boosting neural image compression for machines using latent space masking.  _arXiv preprint arXiv:2112.08168_ , 2021.
  * Hadizadeh and Bajić [2023] Hadi Hadizadeh and Ivan V Bajić.  Learned scalable video coding for humans and machines.  _arXiv preprint arXiv:2307.08978_ , 2023.
  * Hu et al. [2020] Yueyu Hu, Shuai Yang, Wenhan Yang, Ling-Yu Duan, and Jiaying Liu.  Towards coding for human and machine vision: A scalable image coding approach.  In _2020 IEEE International Conference on Multimedia and Expo (ICME)_ , pages 1–6. IEEE, 2020.
  * Hu and Xu [2023] Zhihao Hu and Dong Xu.  Complexity-guided slimmable decoder for efficient deep video compression.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14358–14367, 2023.
  * Hu et al. [2021] Zhihao Hu, Guo Lu, and Dong Xu.  Fvc: A new framework towards deep video compression in feature space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1502–1511, 2021.
  * Hu et al. [2022] Zhihao Hu, Guo Lu, Jinyang Guo, Shan Liu, Wei Jiang, and Dong Xu.  Coarse-to-fine deep video coding with hyperprior-guided mode prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5921–5930, 2022.
  * Khatoonabadi and Bajic [2012] Sayed Hossein Khatoonabadi and Ivan V Bajic.  Video object tracking in the compressed domain using spatio-temporal markov random fields.  _IEEE transactions on image processing_ , 22(1):300–313, 2012.
  * Li et al. [2022] Congcong Li, Xinyao Wang, Longyin Wen, Dexiang Hong, Tiejian Luo, and Libo Zhang.  End-to-end compressed video representation learning for generic event boundary detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13967–13976, 2022.
  * Li et al. [2021a] Jiahao Li, Bin Li, and Yan Lu.  Deep contextual video compression.  _Advances in Neural Information Processing Systems_ , 34:18114–18125, 2021a.
  * Li et al. [2023] Jiahao Li, Bin Li, and Yan Lu.  Neural video compression with diverse contexts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22616–22626, 2023.
  * Li et al. [2021b] Xin Li, Jun Shi, and Zhibo Chen.  Task-driven semantic coding via reinforcement learning.  _IEEE Transactions on Image Processing_ , 30:6307–6320, 2021b.
  * Lin et al. [2023] Hongbin Lin, Bolin Chen, Zhichen Zhang, Jielian Lin, Xu Wang, and Tiesong Zhao.  Deepsvc: Deep scalable video coding for both machine and human vision.  In _Proceedings of the 31st ACM International Conference on Multimedia_ , pages 9205–9214, 2023.
  * Lin et al. [2019] Ji Lin, Chuang Gan, and Song Han.  Tsm: Temporal shift module for efficient video understanding.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 7083–7093, 2019.
  * Liu et al. [2022] Qiankun Liu, Bin Liu, Yue Wu, Weihai Li, and Nenghai Yu.  Real-time online multi-object tracking in compressed domain.  _arXiv preprint arXiv:2204.02081_ , 2022.
  * Lu et al. [2019] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao.  Dvc: An end-to-end deep video compression framework.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11006–11015, 2019.
  * Lu et al. [2021] Guo Lu, Xiaoyun Zhang, Wanli Ouyang, Li Chen, Zhiyong Gao, and Dong Xu.  An end-to-end learning framework for video compression.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 43(10):3292–3308, 2021.
  * Lu et al. [2022] Guo Lu, Xingtong Ge, Tianxiong Zhong, Jing Geng, and Qiang Hu.  Preprocessing enhanced image compression for machine vision.  _arXiv preprint arXiv:2206.05650_ , 2022.
  * Mentzer et al. [2022] Fabian Mentzer, Eirikur Agustsson, Johannes Ballé, David Minnen, Nick Johnston, and George Toderici.  Neural video compression using gans for detail synthesis and propagation.  In _European Conference on Computer Vision_ , pages 562–578. Springer, 2022.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115:211–252, 2015.
  * Sheng et al. [2022] Xihua Sheng, Jiahao Li, Bin Li, Li Li, Dong Liu, and Yan Lu.  Temporal context mining for learned video compression.  _IEEE Transactions on Multimedia_ , 2022.
  * Shi et al. [2023] Yuheng Shi, Naiyan Wang, and Xiaojie Guo.  Yolov: making still image object detectors great at video object detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 2254–2262, 2023.
  * Shou et al. [2019] Zheng Shou, Xudong Lin, Yannis Kalantidis, Laura Sevilla-Lara, Marcus Rohrbach, Shih-Fu Chang, and Zhicheng Yan.  Dmc-net: Generating discriminative motion cues for fast compressed video action recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1268–1277, 2019.
  * Soomro et al. [2012] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  _arXiv preprint arXiv:1212.0402_ , 2012.
  * Sullivan et al. [2012] Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand.  Overview of the high efficiency video coding (hevc) standard.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 22(12):1649–1668, 2012.
  * Teed and Deng [2020] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_ , pages 402–419. Springer, 2020.
  * Tian et al. [2021] Yuan Tian, Guo Lu, Xiongkuo Min, Zhaohui Che, Guangtao Zhai, Guodong Guo, and Zhiyong Gao.  Self-conditioned probabilistic learning of video rescaling.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 4490–4499, 2021.
  * Tian et al. [2023] Yuan Tian, Guo Lu, Guangtao Zhai, and Zhiyong Gao.  Non-semantics suppressed mask learning for unsupervised video semantic compression.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 13610–13622, 2023.
  * Tian et al. [2024] Yuan Tian, Guo Lu, Yichao Yan, Guangtao Zhai, Li Chen, and Zhiyong Gao.  A coding framework and benchmark towards low-bitrate video understanding.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2024.
  * Torfason et al. [2018] Robert Torfason, Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool.  Towards image understanding from deep compression without decoding.  _arXiv preprint arXiv:1803.06131_ , 2018.
  * Wang et al. [2019] Shiyao Wang, Hongchao Lu, and Zhidong Deng.  Fast object detection in compressed video.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 7104–7113, 2019.
  * Wang et al. [2021] Shurun Wang, Shiqi Wang, Wenhan Yang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao.  Towards analysis-friendly face representation with scalable feature and texture compression.  _IEEE Transactions on Multimedia_ , 2021.
  * Wang et al. [2003] Zhou Wang, Eero P Simoncelli, and Alan C Bovik.  Multiscale structural similarity for image quality assessment.  In _The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003_, pages 1398–1402. Ieee, 2003.
  * Wiegand et al. [2003] T. Wiegand, G.J. Sullivan, G. Bjontegaard, and A. Luthra.  Overview of the h.264/avc video coding standard.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 13(7):560–576, 2003.
  * Wu et al. [2018] Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, R Manmatha, Alexander J Smola, and Philipp Krähenbühl.  Compressed video action recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6026–6035, 2018.
  * Yan et al. [2021] Ning Yan, Changsheng Gao, Dong Liu, Houqiang Li, Li Li, and Feng Wu.  Sssic: semantics-to-signal scalable image coding with learned structural representations.  _IEEE Transactions on Image Processing_ , 30:8939–8954, 2021.
  * Yang et al. [2021] Ren Yang, Luc Van Gool, and Radu Timofte.  Perceptual learned video compression with recurrent conditional gan.  _arXiv preprint arXiv:2109.03082_ , 1, 2021.
  * Zhang et al. [2022] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang.  Bytetrack: Multi-object tracking by associating every detection box.  In _European Conference on Computer Vision_ , pages 1–21. Springer, 2022.
  * Özyılkan et al. [2023] Ezgi Özyılkan, Mateen Ulhaq, Hyomin Choi, and Fabien Racapé.  Learned disentangled latent representations for scalable image coding for humans and machines.  In _2023 Data Compression Conference (DCC)_ , pages 42–51, 2023.