$\displaystyle\left<\pi_{i}(x)\pi_{0}(0)\right>=\left<\pi_{0}(x)\pi_{i}(0)\right>=\frac{i}{2}\left(1+\frac{\xi_{2}^{2}}{\xi_{1}}\right)\mbox{Sign}(t)\partial_{i}\delta^{(3)}(\vec{x})+\frac{i\xi_{2}}{4}t^{2}\mbox{Sign}(t)\partial_{i}\vec{\partial}^{2}\delta^{(3)}(\vec{x}),$
$\displaystyle\left<\pi_{0}(x)\pi_{0}(0)\right>=i\left(1+\frac{\xi_{2}^{2}}{\xi_{1}}\right)\delta(t)\delta^{(3)}(\vec{x})+\frac{i\xi_{2}}{2}t^{2}\absolutevalue{t}\vec{\partial}^{2}\delta^{(3)}(\vec{x}).$

It can be checked directly that for every choice of $\xi_{1},\xi_{2}$, the
Ward Identities are all satisfied, which reveals the Carrollian conformal
invariance of the theory. Though these expressions look complicated, we can
select the Landau-type gauge $\xi_{2}=0$ to simply them and obtain the
nonvanishing correlators listed in (4.16). Here we list them again for
completeness.

$\displaystyle\left<A_{v}(x)A_{v}(0)\right>=-2i\absolutevalue{t}\delta^{(3)}(\vec{x}),\quad\left<A_{v}(x)\pi_{i}(0)\right>=-\left<\pi_{i}(x)A_{v}(0)\right>=\frac{3i}{2}\absolutevalue{t}\partial_{i}\delta^{(3)}(\vec{x}),$
(B.19)
$\displaystyle\left<A_{v}(x)\pi_{0}(0)\right>=-\left<\pi_{0}(x)A_{v}(0)\right>=i\mbox{Sign}(t)\delta^{(3)}(\vec{x}),$
$\displaystyle\left<A_{i}(x)\pi_{j}(0)\right>=-\left<\pi_{j}(x)A_{i}(0)\right>=-\frac{i}{2}\delta_{ij}\mbox{Sign}(t)\delta^{(3)}(\vec{x}),$
$\displaystyle\left<\pi_{i}(x)\pi_{j}(0)\right>=\left<\pi_{j}(x)\pi_{i}(0)\right>=\frac{i}{2}\absolutevalue{t}\left(\partial_{i}\partial_{j}\delta^{(3)}(\vec{x})+\delta_{ij}\vec{\partial}^{2}\delta^{(3)}(\vec{x})\right),$
$\displaystyle\left<\pi_{i}(x)\pi_{0}(0)\right>=\left<\pi_{0}(x)\pi_{i}(0)\right>=\frac{i}{2}\mbox{Sign}(t)\partial_{i}\delta^{(3)}(\vec{x}),\quad\left<\pi_{0}(x)\pi_{0}(0)\right>=i\delta(t)\delta^{(3)}(\vec{x}).$

##  Appendix C C Ward identities and 2-point correlation functions

In this Appendix, we review the constraints on the 2-point correlation
functions of the primary operators from the Ward identities of Carrollian
conformal symmetries. There could be four classes of the correlators with
different structures, which will be labeled by Case 1.1, Case 1.2, Case 2.1,
and Case 2.2. It turns out that the correlators discussed in the main text
belong to Case 2.1.

Similar to the case in CFT, the structure of 2-point correlation functions in
CCFT is very much constrained by the Ward identities of the symmetries. For
the Carrollian conformal symmetries, the corresponding Ward identities are
listed in (C.1),

$\displaystyle P_{\mu}:$
$\displaystyle\quad(\partial^{\mu}_{1}+\partial^{\mu}_{2})\left<\mathcal{O}_{1}\mathcal{O}_{2}\right>=0,$
(C.1) $\displaystyle D:$ $\displaystyle\quad
x^{\mu}\partial^{\mu}\left<\mathcal{O}_{1}\mathcal{O}_{2}\right>+\Delta_{1}\left<\mathcal{O}_{1}\mathcal{O}_{2}\right>+\Delta_{2}\left<\mathcal{O}_{1}\mathcal{O}_{2}\right>=0,$
$\displaystyle J_{ij}:$
$\displaystyle\quad(x^{i}\partial^{j}-x^{j}\partial^{i})\left<\mathcal{O}_{1}\mathcal{O}_{2}\right>+\left<(J^{ij}\mathcal{O}_{1})\mathcal{O}_{2}\right>+\left<\mathcal{O}_{1}(J^{ij}\mathcal{O}_{2})\right>=0,$
$\displaystyle B_{i}:$ $\displaystyle\quad
x^{i}\partial_{t}\left<\mathcal{O}_{1}\mathcal{O}_{2}\right>+\left<[B_{i},\mathcal{O}_{1}]\mathcal{O}_{2}\right>+\left<\mathcal{O}_{1}[B_{i},\mathcal{O}_{2}]\right>=0,$
$\displaystyle K_{0}:$
$\displaystyle\quad\left(\left<[K_{0},\mathcal{O}_{1}]\mathcal{O}_{2}\right>+\left<\mathcal{O}_{1}[K_{0},\mathcal{O}_{2}]\right>\right)-x^{i}\left(\left<[B_{i},\mathcal{O}_{1}]\mathcal{O}_{2}\right>-\left<\mathcal{O}_{1}[B_{i},\mathcal{O}_{2}]\right>\right)=0,$
$\displaystyle K_{i}:$
$\displaystyle\quad\left(\left<[K_{i},\mathcal{O}_{1}]\mathcal{O}_{2}\right>+\left<\mathcal{O}_{1}[K_{i},\mathcal{O}_{2}]\right>\right)+x^{i}(\Delta_{1}-\Delta_{2})\left<\mathcal{O}_{1}\mathcal{O}_{2}\right>$
$\displaystyle\qquad\quad+x^{j}\left(\left<[J^{i}_{~{}j},\mathcal{O}_{1}]\mathcal{O}_{2}\right>-\left<\mathcal{O}_{1}[J^{i}_{~{}j},\mathcal{O}_{2}]\right>\right)+t\left(\left<[B_{i},\mathcal{O}_{1}]\mathcal{O}_{2}\right>-\left<\mathcal{O}_{1}[B_{i},\mathcal{O}_{2}]\right>\right)=0.$

It should be mentioned that we have used the techniques explained in the
appendix of [43] to simplify the expression for Carrollian special conformal
transformation generators $K_{0},K_{i}$. These identities hold for all of the
correlators appearing in this article. The one from the translational
generator $P_{\mu}$ requires that

$\left<\mathcal{O}_{1}\mathcal{O}_{2}\right>=f(x^{\mu}),$ (C.2)

where $x^{\mu}=x^{\mu}_{1}-x^{\mu}_{2}$.

As shown in [43], by solving the Ward identities, the 2-point correlators of
the operators in a CCFT is generically composed of two independent types, one
being of the power-law form, the other being proportional to the Dirac
$\delta$-function. In [43], the authors have discussed the one of the power-
law form in detail. In this appendix, we mainly focus on the 2-point
correlators for the primary operators in chain representations, and pay more
attention to the correlators which appear as the generalized functions999A
nice introduction to the generalized functions can be found in [58]. in
general $d$ dimensions, including the Dirac $\delta$-functions. The techniques
used here is similar to the ones in [43], and we strongly recommend the reader
to find more details there.

It should also be stressed that here we only consider the correlators of the
primary operators. Some operators in the staggered modules, like $\pi$ in the
magnetic scalar theory, are special in the sense that they are neither primary
($KO\neq 0$) nor descendent, and their correlators can not be constrained by
the discussions here. Even though these operators do obey some Ward identities
from their transformation laws, which help us to determine their correlators,
there is short of general rules on the correlators of these operators.

For the primary operators $\mathcal{O}_{1},\mathcal{O}_{2}$, their 2-point
correlation function $f=\left<\mathcal{O}_{1}\mathcal{O}_{2}\right>$ is a
homogeneous function by using the Ward identity of $D$,

$D:\quad(t\partial_{t}+x^{i}\partial_{i})f(t,\vec{x})+(\Delta_{1}+\Delta_{2})f(t,\vec{x})=0.$
(C.3)

The solution to this equation is a combination of two independent solutions,
the power-law functions and the generalized functions like the (derivatives
of) Dirac $\delta$-distribution. For example, the one-dimensional version of
this differential equation is

$x\partial f(x)+\lambda f(x)=0,$ (C.4)

with the solution being

$f(x)=c_{1}x^{-\lambda}+c_{2}\partial^{(\lambda-1)}\delta(x),$ (C.5)

where $c_{i}$ are constants, and $c_{2}\neq 0$ for $\lambda=1,2,...,$. In the
Carrollian case, $t$ direction and $x_{i}$ directions could be considered
separately, and thus the solution to (C.3) is simply

$f(t,\vec{x})=g(t)g(\vec{x}),$ (C.6)

where $g(t)$ and $g(\vec{x})$ are the homogeneous generalized functions of the
form (C.5).

Another important constraint is from the Ward identity of $B_{i}$ on the
lowest-level correlators $f=\left<\mathcal{O}_{1}\mathcal{O}_{2}\right>$:

$B_{i}:\quad x^{i}\partial_{t}f(t,\vec{x})=0.$ (C.7)

By the “lowest-level”, we mean
$[B_{i},\mathcal{O}_{1}]=[B_{i},\mathcal{O}_{2}]=0$. Considering the fact
$x\delta(x)=0$, we find four independent solutions,

$\displaystyle\partial_{t}f=0:$ $\displaystyle\quad\left\\{\begin{aligned}
&f(t,\vec{x})\propto P(\vec{x}),&&&&\textbf{(Case 1.1)}\\\
&f(t,\vec{x})\propto\prod_{i}\partial_{i}^{n_{i}}(\vec{\partial}^{2})^{n}\delta^{(d-1)}(\vec{x}),&&\Delta_{1}+\Delta_{2}=d-1+\sum_{i}n_{i}+2n,&&\textbf{(Case
1.2)}\end{aligned}\right.$ $\displaystyle x^{i}f=0:$
$\displaystyle\quad\left\\{\begin{aligned} &f(t,\vec{x})\propto
P(t)\delta^{(d-1)}(\vec{x}),&&&&\qquad\quad\textbf{(Case 2.1)}\\\
&f(t,\vec{x})\propto\partial_{t}^{n_{t}}\delta(t)\delta^{(d-1)}(\vec{x}),&&\Delta_{1}+\Delta_{2}=d+n_{t},&&\qquad\quad\textbf{(Case
2.2)}\end{aligned}\right.$

where both $P(t)$ and $P(\vec{x})$ are the power-law functions, and Case 1.2
appears for $\Delta_{1}+\Delta_{2}=d-1,d,d+1,...$ and Case 2.2 appears for
$\Delta_{1}+\Delta_{2}=d,d+1,d+2,...$. In fact, the correlators of the primary
operators in this paper belong to Case 2.1.

The Case 1.1 with $f(t,\vec{x})\propto P(\vec{x})$ being the power-law
function has been discussed in [43]. In the rest of this section, we first
repeat the constraints in Case 1.1 and then discuss the other situations.

###  C.1 Case 1.1 and Case 1.2

As shown in [43], the chain representations can have the following forms

$\displaystyle(j)$ (C.8) $\displaystyle(j)\rightarrow$
$\displaystyle(j),\qquad j\neq 0$ $\displaystyle(0)\rightarrow(1)$
$\displaystyle\rightarrow(0),$
$\displaystyle\cdots\rightarrow(j)\rightarrow(j+1)$
$\displaystyle\rightarrow(j+2)\rightarrow\cdots,$
$\displaystyle\cdots\rightarrow(j)\rightarrow(j-1)$
$\displaystyle\rightarrow(j-2)\rightarrow\cdots.$

For Case 1.1, the correlators could be the power-law functions of $x^{\mu}$,
and the non-vanishing 2-point correlators only appear in the case that
$\mathcal{O}_{1},\mathcal{O}_{2}$ have (partially) inverse structure, and the
selection rule is $\Delta_{1}=\Delta_{2}$. The correlator takes the form

$\left<\mathcal{O}_{1}\mathcal{O}_{2}\right>=\frac{C~{}(t/|\vec{x}|)^{r}~{}I^{m_{1},m_{2}}_{j_{1},j_{2}}}{|\vec{x}|^{(\Delta_{1}+\Delta_{2})}}\delta_{\Delta_{1},\Delta_{2}},$
(C.9)

where $I$ is a rank-$0$ homogeneous function of $x_{i}$ representing the
tensor structure of $O_{i}$.

For Case 1.2, $\Delta_{1}+\Delta_{2}\geq d-1\in\mathbb{Z}$, there exists
another solution for the lowest-level 2-point correlators,

$f(t,\vec{x})\propto\prod_{i}\partial_{i}^{n_{i}}(\vec{\partial}^{2})^{n}\delta^{(d-1)}(\vec{x}),\hskip
12.91663pt\sum_{i}n_{i}=\Delta_{1}+\Delta_{2}-(d-1)-2n,\hskip
8.61108ptn_{i}\in\mathbf{N}^{+}.$ (C.10)

For the higher-level correlators, the solutions are of the form
$f^{\prime}(t,\vec{x})\propto
t^{r}\prod_{i}\partial_{i}^{n^{\prime}_{i}}(\vec{\partial}^{2})^{n}\delta^{(d-1)}(\vec{x})$
with
$\sum_{i}n^{\prime}_{i}-2n-r=\Delta_{1}+\Delta_{2}-(d-1),n^{\prime}_{i}\in\mathbf{N}^{+}$.
The full restriction on the 2-point correlators in Case 1.2 is similar to Case
1.1, except the case that one of the operators is a scalar, which will be
discussed separately later. The reason that the selection rule is (almost) the
same is that the power laws are proportional to (derivatives of) Dirac
$\delta$-functions under canonical regularization [58]:

$\displaystyle\frac{2}{\Omega_{(d-1)}}\left.\frac{r^{\lambda}}{\Gamma\left(\frac{\lambda+d-1}{2}\right)}\right|_{\lambda=-(d-1)-2k}=\frac{(-1)^{k}(d-2)!}{2^{k}k!(d-1+2k-2)!}(\vec{\partial}^{2})^{k}\delta^{(d-1)}(\vec{x})$
(C.11)

for $k=0,1,2,...$, with $r^{2}=\sum_{i}x_{i}^{2}$. As a result, most of the
constraints from the Ward identities are the same as the ones in Case 1.1.
Thus if $\Delta_{1}+\Delta_{2}\geq d-1\in\mathbb{Z}$ and
$\Delta_{1}=\Delta_{2}$, the correlators are non-vanishing for
$\mathcal{O}_{1}$ and $\mathcal{O}_{2}$ in partially inverse representations,
and the structures of the correlators are of the form

$\left<\mathcal{O}_{1,l_{1}}^{\\{s_{1}\\}}\mathcal{O}_{2,l_{2}}^{\\{s_{2}\\}}\right>=C~{}t^{r}~{}(D_{s_{1}}D_{s_{2}}(\vec{\partial}^{2})^{n}\delta^{(d-1)}(\vec{x})-\text{traces}),\quad\text{with
}D_{s_{i}}=\partial_{s_{i,1}}\cdots\partial_{s_{i,l_{i}}}$ (C.12)

The explicit selection rule is rather tedious, and we do not repeat them here.
The interested readers may refer [43] for detailed discussions.

The exceptional situation in Case 1.2 is when one of the primary operators is
in scalar representation $(0)$. In this case, there is one additional set of
the selection rules, due to the special property of Dirac $\delta$-function.
In the following, we explain how this additional selection rule emerges and
show the structure of the correlators in this situation. Firstly, for the
simplest case that both $\mathcal{O}_{1}$ and $\mathcal{O}_{2}$ are scalars
with $\Delta_{1}+\Delta_{2}=d-1$, the correlator is
$f=\left<\mathcal{O}_{1}\mathcal{O}_{2}\right>\propto\delta^{(d-1)}(\vec{x})$
in Case 1.2. It is known that

Case 1.1: $\displaystyle x_{i}f\propto\frac{x_{i}}{r^{(d-1)}}\neq 0,$ (C.13)
Case 1.2: $\displaystyle x_{i}f\propto x_{i}\delta^{(d-1)}(\vec{x})=0,$

which makes the constraints from the Ward identities of $K_{i}$ on $f$ for
Case 1.1 and 1.2 different,

Case 1.1: $\displaystyle\text{solution: }f=\frac{C_{1}}{r^{(d-1)}},$
$\displaystyle\text{constraint: }\Delta_{1}=\Delta_{2}=\frac{d-1}{2},$ (C.14)
Case 1.2: $\displaystyle\text{solution: }f=C_{2}\delta^{(d-1)}(\vec{x}),$
$\displaystyle\text{constraint: }\Delta_{1}+\Delta_{2}=d-1.$

Thus for Case 1.2, we have the selection rule

$\left<\mathcal{O}_{1}\mathcal{O}_{2}\right>=C~{}\delta^{(d-1)}(\vec{x}),\hskip
12.91663pt\mathcal{O}_{1},\mathcal{O}_{2}\in(0),\qquad\Delta_{1}+\Delta_{2}=d-1.$
(C.15)

Next, we consider the case that $\mathcal{O}_{1}$ is in more complicated chain
representation. In the case that $\mathcal{O}_{1}\in(j)$ is a symmetric
traceless tensor (STT) with spin $j$, $\mathcal{O}_{2}\in(0)$ is a scalar.
Using the fact
$x_{i}\partial_{i}\delta^{(d-1)}(\vec{x})=-\delta^{(d-1)}(\vec{x})$, we find
that the restrictions from the Ward identities of $K_{i}$ are

$\left<\mathcal{O}_{1}^{\\{s_{1},...,s_{j}\\}}\mathcal{O}_{2}\right>=C~{}(\partial_{s1}\cdots\partial_{s_{j}}\delta^{(d-1)}(\vec{x})-\text{traces}),\qquad\Delta_{1}=1,\quad\Delta_{2}=d-2+j.$
(C.16)

The “traces” term is the trace of
$\partial_{s1}\cdots\partial_{s_{j}}\delta^{(d-1)}(\vec{x})$, and subtracting
this term makes the correlators respect the traceless condition of
$\mathcal{O}_{1}$. Moreover for $\mathcal{O}_{1}\in(j)_{2}\to(j)_{1}$ and
$\mathcal{O}_{2}\in(0)$, we have101010Here we use subscripts to distinguish
different sectors of $(j)_{2}\to(j)_{1}$ with the same spin $j$. Similar
notation for $(0)_{3}\to(1)_{2}\to(0)_{1}$ will appear below.

$\displaystyle\left<\mathcal{O}_{1,(j)_{2}}^{\\{s_{1},...,s_{j}\\}}\mathcal{O}_{2}\right>=C~{}(\partial_{s_{1}}\cdots\partial_{s_{j}}\delta^{(d-1)}(\vec{x})-\text{traces}),\qquad\left<\mathcal{O}_{1,\text{others}}\mathcal{O}_{2}\right>=0,$
(C.17) $\displaystyle\qquad\Delta_{1}=1,\quad\Delta_{2}=d-2+j,$

For $\mathcal{O}_{1}$ being a decreasing chain,
$O_{1}\in(j+n)\to(j+n-1)\cdots\to(j+1)\to(j)$ and $O_{2}\in(0)$, we have

$\left<\mathcal{O}_{1,l_{1}=j+r}^{\\{s_{1},...,s_{l_{1}}\\}}\mathcal{O}_{2}\right>=\frac{C~{}t^{r}}{r!}~{}(\partial_{s_{1}}\cdots\partial_{s_{l_{1}}}\delta^{(d-1)}(\vec{x})-\text{traces}),\qquad\Delta_{1}=1,\Delta_{2}=d-2+j,$
(C.18)

where $\mathcal{O}_{1,l_{1}}$ is the spin-$l_{1}$ part of $\mathcal{O}_{1}$.
For $\mathcal{O}_{1}$ in an increasing chain representation,
$\mathcal{O}_{1}\in(j)\to(j+1)\cdots\to(j+n-1)\to(j+n)$, and
$\mathcal{O}_{2}\in(0)$, the correlators vanish except for the highest-rank
sector in $\mathcal{O}_{1}$. Namely, we have

$\left<\mathcal{O}_{1,(j)}^{\\{s_{1},...,s_{j}\\}}\mathcal{O}_{2}\right>=C~{}(\partial_{s_{1}}\cdots\partial_{s_{j}}\delta^{(d-1)}(\vec{x})-\text{traces}),\quad\left<\mathcal{O}_{1,\text{others}}\mathcal{O}_{2}\right>=0,\quad\Delta_{1}+\Delta_{2}=d-1+j.$
(C.19)

And finally, for $\mathcal{O}_{1}\in(0)_{3}\to(1)_{2}\to(0)_{1}$ and
$\mathcal{O}_{2}\in(0)$, we have

$\left<\mathcal{O}_{1,(0)_{3}}\mathcal{O}_{2}\right>=C~{}\delta^{(d-1)}(\vec{x}),\qquad\left<\mathcal{O}_{1,\text{others}}\mathcal{O}_{2}\right>=0,\qquad\Delta_{1}+\Delta_{2}=d-1.$
(C.20)

We have presented all the exceptional cases involving a scalar primary
operator. Here we only discuss the case that the other operator belong to a
chain representation, and we do not discuss the case that the other operator
is in a net-like representation.

###  C.2 Case 2.1 and 2.2

Case 2.1 and Case 2.2 come from the fact that $x^{i}\delta^{(d-1)}(\vec{x})=0$
solves the equation of the Ward identities of $B_{i}$. The selection rules for
these two cases are very different from the ones in Case 1.1 and Case 1.2.

First we consider Case 2.1 with the operators being the symmetric traceless
tensors (STTs) and in the singlet representations $(j)$. Since the spacial
dependence in the correlators is always $\delta^{(d-1)}(\vec{x})$, the only
possible non-vanishing lowest-level correlator is from the case that
$\mathcal{O}_{1}$ and $\mathcal{O}_{2}$ have the same spin, $l_{1}=l_{2}$. It
can be checked that the Ward identities of $K_{i}$ are manifestly satisfied
using the fact that $x^{i}\delta^{(d-1)}(\vec{x})=0$, and there is no
selection rule on $\Delta_{1}$ and $\Delta_{2}$. Therefore we have

$\displaystyle\left<\mathcal{O}_{1}\mathcal{O}_{2}\right>=C~{}t^{(d-1-\Delta_{1}-\Delta_{2})}\delta^{(d-1)}(\vec{x}),$
$\displaystyle l_{1}=l_{2}=0,$ (C.21)
$\displaystyle\left<\mathcal{O}_{1}^{i_{1}}\mathcal{O}_{2}^{j_{1}}\right>=C~{}\delta^{i_{1}}_{j_{1}}t^{(d-1-\Delta_{1}-\Delta_{2})}\delta^{(d-1)}(\vec{x}),$
$\displaystyle l_{1}=l_{2}=1,$
$\displaystyle\left<\mathcal{O}_{1}^{i_{1}i_{2}}\mathcal{O}_{2}^{j_{1}j_{2}}\right>=C\left(\delta^{i_{1}}_{j_{1}}\delta^{i_{2}}_{j_{2}}+\delta^{i_{1}}_{j_{2}}\delta^{i_{2}}_{j_{1}}-\frac{2}{d-1}\delta^{i_{1}i_{2}}\delta_{j_{1}j_{2}}\right)t^{(d-1-\Delta_{1}-\Delta_{2})}\delta^{(d-1)}(\vec{x}),$
$\displaystyle l_{1}=l_{2}=2,$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\vdots$
$\displaystyle\left<\mathcal{O}_{1}^{i_{1}\cdots
i_{s}}\mathcal{O}_{2}^{j_{1}\cdots
j_{s}}\right>=C\left(\delta^{i_{1}}_{(j_{1}}\cdots\delta^{i_{s}}_{j_{s})}-\text{trace}\right)t^{(d-1-\Delta_{1}-\Delta_{2})}\delta^{(d-1)}(\vec{x}),$
$\displaystyle l_{1}=l_{2}=s.$

The “trace” term is to cancel the trace of $O_{1}$ indices and the trace of
$O_{2}$ indices, as both $\mathcal{O}_{1}$ and $\mathcal{O}_{2}$ are STTs. The
coefficient $C$’s are undetermined constants.

For the chain representations, there are very limited restrictions for the
correlators being non-vanishing. The calculations show that if two chain
representations have the same sub-sector, the correlators of the operators in
these subs-sectors and in the higher levels are non-vanishing. In other words,
if

$\displaystyle\mathcal{O}_{1}$
$\displaystyle\in\cdots\to(j_{n+1})\to(j_{n})\to(j_{n-1})\to\cdots,$ (C.22)
$\displaystyle\mathcal{O}_{2}$
$\displaystyle\in\cdots\to(j_{m+1})\to(j_{m})\to(j_{m-1})\to\cdots,\qquad\text{with
}j_{n}=j_{m}$

then

$\left<\mathcal{O}_{1,l_{1}=j_{\geq n}}\mathcal{O}_{2,l_{2}=j_{\geq
m}}\right>\neq 0.$ (C.23)

For the chains, there are the selection rules on $\Delta_{1}$ and
$\Delta_{2}$, but the specific selection rule must be discussed case by case.
For examples, we have

$\displaystyle\left<\mathcal{O}_{1,l_{1}}^{\\{s_{1},...,s_{l_{1}}\\}}\mathcal{O}_{2,l_{2}}^{\\{r_{1},...,r_{l_{2}}\\}}\right>$
(C.24)
$\displaystyle=C\frac{(d-1-\Delta_{1}-\Delta_{2})!~{}t^{(d-1-\Delta_{1}-\Delta_{2}+l_{1}+l_{2})}}{(d-1-\Delta_{1}-\Delta_{2}+l_{1}+l_{2})!}(\partial_{s1}\cdots\partial_{s_{l_{1}}}\partial_{r1}\cdots\partial_{r_{l_{2}}}\delta^{(d-1)}(\vec{x})-\text{traces})$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\text{for
}\mathcal{O}_{1},\mathcal{O}_{2}\in\cdots\to(2)\to(1)\to(0),\quad\text{with
}\Delta_{1}=\Delta_{2}=1.$
$\displaystyle\left<\mathcal{O}_{1,l_{1}}^{\\{s_{1},...,s_{l_{1}}\\}}\mathcal{O}_{2,l_{2}}^{\\{r_{1},...,r_{l_{2}}\\}}\right>$
(C.25)
$\displaystyle=C\frac{(d-1-\Delta_{1}-\Delta_{2})!~{}t^{(d-1-\Delta_{1}-\Delta_{2}+l_{1}+l_{2}-2)}}{(d-1-\Delta_{1}-\Delta_{2}+l_{1}+l_{2})!}\left(\delta_{(s_{1}}^{(r_{1}}\partial_{s2}\cdots\partial_{s_{l_{1}})}\partial^{r2}\cdots\partial^{r_{l_{2}})}\delta^{(d-1)}(\vec{x})-\text{traces}\right)$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\text{for
}\mathcal{O}_{1},\mathcal{O}_{2}\in\cdots\to(3)\to(2)\to(1),\quad\text{with
}\Delta_{1}=\Delta_{2}=0.$

Especially, we have

$\displaystyle\begin{aligned}
&\left<\mathcal{O}_{1,(0)_{3}}\mathcal{O}_{2,(0)_{3}}\right>=C~{}\frac{t^{(d-1-2\Delta+2)}}{(5-2\Delta)(\Delta-3)}\partial^{2}\delta^{(d-1)}(\vec{x})\\\
\end{aligned}$ (C.26) $\displaystyle\begin{aligned}
&\left<\mathcal{O}_{1,(0)_{3}}\mathcal{O}_{2,(1)_{2}}^{r}\right>=C~{}\frac{t^{(d-1-2\Delta+1)}}{(\Delta-3)}\partial_{r}\delta^{(d-1)}(\vec{x})\\\
&\left<\mathcal{O}_{1,(1)_{2}}^{s}\mathcal{O}_{2,(0)_{3}}\right>=C~{}\frac{t^{(d-1-2\Delta+1)}}{(\Delta-3)}\partial_{s}\delta^{(d-1)}(\vec{x})\\\
\end{aligned}$ $\displaystyle\begin{aligned}
&\left<\mathcal{O}_{1,(0)_{3}}\mathcal{O}_{2,(0)_{1}}\right>=C~{}t^{(d-1-2\Delta)}\delta^{(d-1)}(\vec{x})\\\
&\left<\mathcal{O}_{1,(1)_{2}}^{s}\mathcal{O}_{2,(1)_{2}}^{r}\right>=C~{}\frac{1-\Delta}{\Delta-3}t^{(d-1-2\Delta)}\delta_{sr}\delta^{(d-1)}(\vec{x})\\\
&\left<\mathcal{O}_{1,(0)_{1}}\mathcal{O}_{2,(0)_{3}}\right>=C~{}t^{(d-1-2\Delta)}\delta^{(d-1)}(\vec{x})\qquad\qquad\qquad\qquad\left<\text{others}\right>=0\\\
\end{aligned}$ $\displaystyle\qquad\qquad\qquad\qquad\qquad\text{for
}\mathcal{O}_{1},\mathcal{O}_{2}\in(0)_{3}\to(1)_{2}\to(0)_{1},\text{ with
}\Delta_{1}=\Delta_{2}=\Delta$

The selection rule for Case 2.2 is the same as the ones for Case 2.1.
Different from the relation between Case 1.1 and 1.2, there is no exceptional
situation. The analog of the exceptional case in Case 1.2 is when
$\Delta_{1}+\Delta_{2}=d$ with the correlator
$f\propto\delta(t)\delta^{(d-1)}(\vec{x})$, but the constraint from the Ward
identities of $K_{i}$ gives similar selection rules for Case 2.1 and 2.2.

The correlators appeared in the main text are all of Case 2.1. The primary
operator in the electric scalar theory is the field $\phi$, and the correlator
is
$\left<\phi(x)\phi(0)\right>=\frac{i}{2}\absolutevalue{t}\delta^{(d-1)}(\vec{x}).$
It can be checked that the correlator satisfies the Ward identities, no matter
if the temporal part is in power of $t$ or $\absolutevalue{t}$, and this
correlator matches the form of (C.21). Similar to the electric scalar theory,
the magnetic scalar theory have the primary operator $\phi$ with
$\left<\phi(x)\phi(0)\right>=0$, which obviously matches the form of (C.21).
The primary operators in the electric sector of electromagnetic theory are
$A_{\mu}=(A_{0},A_{i})$. They are in $(1)\to(0)$ representation and the
corresponding correlators are (B.10). These correlators have the same form
with (C.24). Finally, the fundamental operators
$A_{\alpha}=(A_{v},A_{i},A_{0})$ in the magnetic sector of electromagnetic
theory are primary operators , which are in $(0)\to(1)\to(0)$ representation.
Their correlators are in (B.18) which match the ones in (C.26) with
$\Delta_{1}=\Delta_{2}=1$.

## References

  * [1] J. Levy-Leblond, _Une nouvelle limite non-relativiste du groupe de poincaré_ , _Annales de l’IHP Physique théorique_ 3 (1965) 1.
  * [2] V. Sen Gupta, _On an Analogue of the Galileo Group_ , _Nuovo Cim._ 54 (1966) 512.
  * [3] H. Bacry and J. Levy-Leblond, _Possible kinematics_ , _J. Math. Phys._ 9 (1968) 1605.
  * [4] C. Duval, G.W. Gibbons, P.A. Horvathy and P.M. Zhang, _Carroll versus Newton and Galilei: two dual non-Einsteinian concepts of time_ , _Class. Quant. Grav._ 31 (2014) 085016 [1402.0657].
  * [5] E. Bergshoeff, J. Gomis and G. Longhi, _Dynamics of Carroll Particles_ , _Class. Quant. Grav._ 31 (2014) 205009 [1405.2264].
  * [6] J. Souriau, _Ondes et radiations gravitationnelles_ , _Colloques Internationaux du CNRS_ 220 (1973) 243.
  * [7] C. Duval, G.W. Gibbons, P.A. Horvathy and P.M. Zhang, _Carroll symmetry of plane gravitational waves_ , _Class. Quant. Grav._ 34 (2017) 175003 [1702.08284].
  * [8] M. Henneaux, _Geometry of Zero Signature Space-times_ , _Bull. Soc. Math. Belg._ 31 (1979) 47.
  * [9] V. Belinski and M. Henneaux, _The Cosmological Singularity_ , Cambridge Monogr.Math.Phys., Cambridge Univ. Pr., Cambridge (2017), 10.1017/9781107239333.
  * [10] R.F. Penna, _Near-horizon Carroll symmetry and black hole Love numbers_ , 1812.05643.
  * [11] L. Donnay and C. Marteau, _Carrollian Physics at the Black Hole Horizon_ , _Class. Quant. Grav._ 36 (2019) 165002 [1903.09654].
  * [12] L. Freidel and P. Jai-akson, _Carrollian hydrodynamics and symplectic structure on stretched horizons_ , 2211.06415.
  * [13] J. Redondo-Yuste and L. Lehner, _Non-linear black hole dynamics and Carrollian fluids_ , 2212.06175.
  * [14] J. de Boer, J. Hartong, N.A. Obers, W. Sybesma and S. Vandoren, _Carroll symmetry, dark energy and inflation_ , 2110.02319.
  * [15] R. Casalbuoni, J. Gomis and D. Hidalgo, _World-Line Description of Fractons_ , 2107.09010.
  * [16] F. Peña Benitez, _Fractons, Symmetric Gauge Fields and Geometry_ , 2107.13884.
  * [17] C. Duval, G.W. Gibbons and P.A. Horvathy, _Conformal Carroll groups and BMS symmetry_ , _Class. Quant. Grav._ 31 (2014) 092001 [1402.5894].
  * [18] H. Bondi, M.G.J. van der Burg and A.W.K. Metzner, _Gravitational waves in general relativity. 7. Waves from axisymmetric isolated systems_ , _Proc. Roy. Soc. Lond. A_ 269 (1962) 21.
  * [19] R.K. Sachs, _Gravitational waves in general relativity. 8. Waves in asymptotically flat space-times_ , _Proc. Roy. Soc. Lond. A_ 270 (1962) 103.
  * [20] R. Sachs, _Asymptotic symmetries in gravitational theory_ , _Phys. Rev._ 128 (1962) 2851.
  * [21] G. Barnich and C. Troessaert, _Symmetries of asymptotically flat 4 dimensional spacetimes at null infinity revisited_ , _Phys. Rev. Lett._ 105 (2010) 111103 [0909.2617].
  * [22] A. Bagchi, S. Detournay, R. Fareghbal and J. Simón, _Holography of 3D Flat Cosmological Horizons_ , _Phys. Rev. Lett._ 110 (2013) 141302 [1208.4372].
  * [23] J. Hartong, _Holographic Reconstruction of 3D Flat Space-Time_ , _JHEP_ 10 (2016) 104 [1511.01387].
  * [24] L. Ciambelli, R.G. Leigh, C. Marteau and P.M. Petropoulos, _Carroll Structures, Null Geometry and Conformal Isometries_ , _Phys. Rev. D_ 100 (2019) 046010 [1905.02221].
  * [25] L. Donnay, A. Fiorucci, Y. Herfray and R. Ruzziconi, _Carrollian Perspective on Celestial Holography_ , _Phys. Rev. Lett._ 129 (2022) 071602 [2202.04702].
  * [26] A. Bagchi, S. Banerjee, R. Basu and S. Dutta, _Scattering Amplitudes: Celestial and Carrollian_ , _Phys. Rev. Lett._ 128 (2022) 241601 [2202.08438].
  * [27] L. Donnay, A. Fiorucci, Y. Herfray and R. Ruzziconi, _Bridging Carrollian and Celestial Holography_ , 2212.12553.
  * [28] M. Henneaux and P. Salgado-Rebolledo, _Carroll contractions of Lorentz-invariant theories_ , _JHEP_ 11 (2021) 180 [2109.06708].
  * [29] A. Bagchi, R. Basu, A. Kakkar and A. Mehra, _Flat Holography: Aspects of the dual field theory_ , _JHEP_ 12 (2016) 147 [1609.06203].
  * [30] A. Bagchi, R. Basu, A. Mehra and P. Nandi, _Field Theories on Null Manifolds_ , _JHEP_ 02 (2020) 141 [1912.09388].
  * [31] A. Bagchi, A. Mehra and P. Nandi, _Field Theories with Conformal Carrollian Symmetry_ , _JHEP_ 05 (2019) 108 [1901.10147].
  * [32] E.A. Bergshoeff, J. Gomis and A. Kleinschmidt, _Non-Lorentzian theories with and without constraints_ , _JHEP_ 01 (2023) 167 [2210.14848].
  * [33] E. Bergshoeff, J. Figueroa-O’Farrill and J. Gomis, _A non-lorentzian primer_ , _SciPost Phys. Lect. Notes_ 69 (2023) 1 [2206.12177].
  * [34] D. Rivera-Betancour and M. Vilatte, _Revisiting the Carrollian scalar field_ , _Phys. Rev. D_ 106 (2022) 085004 [2207.01647].
  * [35] D. Hansen, N.A. Obers, G. Oling and B.T. Søgaard, _Carroll Expansion of General Relativity_ , 2112.12684.
  * [36] B. Julia and H. Nicolai, _Null-killing vector dimensional reduction and galilean geometrodynamics_ , _Nuclear Physics B_ 439 (1995) 291.
  * [37] A. Bagchi, A. Banerjee, S. Dutta, K.S. Kolekar and P. Sharma, _Carroll covariant scalar fields in two dimensions_ , 2203.13197.
  * [38] A. Bagchi, A. Banerjee and H. Muraki, _Boosting to BMS_ , _JHEP_ 09 (2022) 251 [2205.05094].
  * [39] S. Baiguera, G. Oling, W. Sybesma and B.T. Søgaard, _Conformal Carroll Scalars with Boosts_ , 2207.03468.
  * [40] A. Saha, _Intrinsic Approach to $1+1$D Carrollian Conformal Field Theory_, 2207.11684.
  * [41] W.-B. Liu and J. Long, _Symmetry group at future null infinity I: scalar theory_ , 2210.00516.
  * [42] S. Dutta, _Stress tensors of 3d Carroll CFTs_ , 2212.11002.
  * [43] B. Chen, R. Liu and Y.-f. Zheng, _On Higher-dimensional Carrollian and Galilean Conformal Field Theories_ , _SciPost Phys._ 14 (2023) 088 [2112.10514].
  * [44] C. Duval, G. Burdet, H.P. Kunzle and M. Perrin, _Bargmann Structures and Newton-cartan Theory_ , _Phys. Rev. D_ 31 (1985) 1841.
  * [45] F. Rohsiepe, _On reducible but indecomposable representations of the Virasoro algebra_ , hep-th/9611160.
  * [46] M.R. Gaberdiel, _An Algebraic approach to logarithmic conformal field theory_ , _Int. J. Mod. Phys. A_ 18 (2003) 4593 [hep-th/0111260].
  * [47] K. Kytola and D. Ridout, _On Staggered Indecomposable Virasoro Modules_ , _J. Math. Phys._ 50 (2009) 123503 [0905.0108].
  * [48] T. Creutzig and D. Ridout, _Logarithmic Conformal Field Theory: Beyond an Introduction_ , _J. Phys. A_ 46 (2013) 4006 [1303.0847].
  * [49] P.-x. Hao, W. Song, X. Xie and Y. Zhong, _A BMS-invariant free scalar model_ , 2111.04701.
  * [50] Z.-f. Yu and B. Chen, _Free field realization of the BMS Ising model_ , 2211.06926.
  * [51] P.-X. Hao, W. Song, Z. Xiao and X. Xie, _A BMS-invariant free fermion model_ , 2211.06927.
  * [52] H.P. Jakobsen, _Indecomposable finite-dimensional representations of a Lie algebras and Lie superalgebras_ , _Lect. Notes Math._ 2027 (2011) 125.
  * [53] M. Hogervorst, M. Paulos and A. Vichi, _The ABC (in any D) of Logarithmic CFT_ , _JHEP_ 10 (2017) 201 [1605.03959].
  * [54] B. Chen, P.-X. Hao, R. Liu and Z.-F. Yu, _On Galilean conformal bootstrap_ , _JHEP_ 06 (2021) 112 [2011.11092].
  * [55] B. Chen and R. Liu, _The Shadow Formalism of Galilean CFT 2_, 2203.10490.
  * [56] B. Chen, P.-x. Hao, R. Liu and Z.-f. Yu, _On Galilean Conformal Bootstrap II: $\xi=0$ sector_, 2207.01474.
  * [57] A. Banerjee, S. Dutta and S. Mondal, _Carroll fermions in two dimensions_ , 2211.11639.
  * [58] I. Gel’fand and G. Shilov, _Generalized Functions: Properties and Operations_ (1964).
  * [59] B. Chen, H. Sun and Y.-f. Zheng, _work in progress_ , .
  * [60] M. Islam, _Carrollian Yang-Mills Theory_ , 2301.00953.
  * [61] A. Bagchi, A. Banerjee, R. Basu, M. Islam and S. Mondal, _Magic Fermions: Carroll and Flat Bands_ , 2211.11640.
  * [62] J. Hartong, _Gauging the Carroll Algebra and Ultra-Relativistic Gravity_ , _JHEP_ 08 (2015) 069 [1505.05011].
  * [63] E. Bergshoeff, J. Gomis, B. Rollier, J. Rosseel and T. ter Veldhuis, _Carroll versus Galilei Gravity_ , _JHEP_ 03 (2017) 165 [1701.06156].
  * [64] J. Figueroa-O’Farrill, E. Have, S. Prohazka and J. Salzer, _The gauging procedure and carrollian gravity_ , _JHEP_ 09 (2022) 243 [2206.14178].
  * [65] A. Campoleoni, M. Henneaux, S. Pekar, A. Pérez and P. Salgado-Rebolledo, _Magnetic Carrollian gravity from the Carroll algebra_ , _JHEP_ 09 (2022) 127 [2207.14167].
  * [66] J.E. Humphreys, _Representations of Semisimple Lie Algebras in the BGG Category $\mathcal{O}$_, vol. 94, American Mathematical Soc. (2008).
  * [67] C.A. Weibel, _An introduction to homological algebra_ , no. 38, Cambridge university press (1995).
  * [68] S. Golkar and D.T. Son, _Operator Product Expansion and Conservation Laws in Non-Relativistic Conformal Field Theories_ , _JHEP_ 12 (2014) 063 [1408.3629].
  * [69] G. Hochschild and J.-P. Serre, _Cohomology of group extensions_ , _Transactions of the American Mathematical Society_ (1953) 110.
  * [70] G. Hochschild and J.-P. Serre, _Cohomology of lie algebras_ , _Annals of Mathematics_ (1953) 591.
  * [71] M.D. Schwartz, _Quantum Field Theory and the Standard Model_ , Cambridge University Press (3, 2014).


# Integrating IP broadcasting with audio tags: workflow and Challenges

###### Abstract

The broadcasting industry is increasingly adopting IP techniques,
revolutionising both live and pre-recorded content production, from news
gathering to live music events. IP broadcasting allows for the transport of
audio and video signals in an easily configurable way, aligning with modern
networking techniques. This shift towards an IP workflow allows for much
greater flexibility, not only in routing signals but with the integration of
tools using standard web development techniques. One possible tool could
include the use of live audio tagging, which has a number of uses in the
production of content. These include from automated closed captioning to
identifying unwanted sound events within a scene. In this paper, we describe
the process of containerising an audio tagging model into a microservice, a
small segregated code module that can be integrated into a multitude of
different network setups. The goal is to develop a modular, accessible, and
flexible tool capable of seamless deployment into broadcasting workflows of
all sizes, from small productions to large corporations. Challenges
surrounding latency of the selected audio tagging model and its effect on the
usefulness of the end product are discussed.

Index Terms—  IP broadcasting, challenges, workflow, AI, Audio tagging.

##  1 Introduction

Internet Protocol (IP) broadcasting describes the process of transmitting
audio and video signals from one location to another using IP networking. One
technique traditionally used for transmitting audio/video is the Serial
Digital Interface (SDI), with fixed connections between dedicated hardware
devices. In comparison, IP broadcasting allows software to replace some of
these hardware devices, enabling greater scalability and easy re-
configuration. Cloud technology and containerisation methods such as Docker
[1] can be utilised to take advantage of such scalability.

There are a few challenges while creating software for use in an IP
broadcasting environment. First, as with most modern web applications, is
scalability and containerisation of software which allows the infrastructure
to adapt depending on the demand on the system by starting up new containers
when required. Containerisation also allows for the same task to be conducted
independently on different streams or sources by having a container per
stream. Another advantage of a containerisation approach means that if a fault
occurs on one container, it does not damage the entire system as a whole and
can be fixed independently. The second challenge is handling of the inelastic
audio and video traffic without introducing delay and jitter to the
transmission.

The audio track contains a wide array of descriptive information about the
sound events in the scene. Detecting sound events in real-time could have a
number of uses from aiding operators in within the broadcasting industry
programme creation to enhancing end user accessibility. For example, BBC
Research and Development [2] employs sound events detection framework to
identify sounds that may disrupt the ambience of a program. This work was
targeted at the BBC Autumnwatch programme which uses wildlife cameras
capturing the movement of animals. To avoid undesirable noises interrupting
the live stream such as cars passing by or people talking, an icon is
overlayed onto the operator’s interface, indicating the undesirable sound
event so the operator does not to switch to that source. Another use of sound
event detection is closed captioning. While majority of the existing work in
broadcasting audio has been focused on analysing the speech events [3, 4] and
only a few attempts of identifying sound events in real time in a live
transmission has been conducted [2]. To achieve full captioning (closed
captions of both sound events and speech) within IP broadcasting, a general
sound event detection models that detects sound events including speech are
required.

Figure 1: Basic flow of audio, video and metadata frames travelling through
their respective data streams using the Network Device Interface (NDI) system.

To overcome above challenges and integrate sound event data into IP
broadcasting, this paper contributes; (1) by containerising applications to
isolate each component from the other elements of the system, i.e. other
processing units, transmission and reception code. Code isolation means faults
are limited to that specific container as well as allowing the component to
run on any machine. Containerisation makes it easy to create multiple
instances of a component if needed allowing for scalability and the use of
cloud platforms. (2) We leverage an Artificial Intelligence (AI) model to
generate audio tags to transmit meta-information alongside audio. This added
information about the contents of the audio track has a number of uses in the
area of automation from better production tools to improved accessibility via
more descriptive automated captioning. An overall proposed framework is shown
in Figure 1 and more details including challenges around our approach can be
found in Section 5. Our code is made available at the
GitHub111https://github.com/Rhysbv/panns_ndi.

The rest of this paper is organised as follows. In Section 2, a background on
IP broadcasting technology and audio recognition in broadcasting is explained.
Section 3 presents system design to select appropriate broadcasting
technology, AI model and integration of broadcasting framework. Section 4
describes an example workflow and experimental setup. The challenges within IP
broadcasting to integrate AI model are explained in Section 5. Finally,
Section 6 presents the discussion and concludes the paper.

##  2 Related work

###  2.1  A Brief Overview on IP Broadcasting Technology

There are a few technologies currently used for IP broadcasting. The first of
these are described in standards from the Society of Motion Picture and
Television Engineers (SMPTE) as the ST 2110 suite of standards [5]. SMPTE
standards are used by the industry with examples including the Serial Digital
Interface (SDI) standards for transmission between equipment over a direct
connection, i.e. coaxial or fibre optic cable. The Networked Media Open
Specifications (NMOS) from the Advanced Media Workflow Association (AMWA) uses
ST 2110 along with other standards to define APIs allowing for the connection
of multiple receivers and senders on a network in a vendor agnostic way. NMOS
is not software, but specifications aiding development in the software.
Network Device Interface (NDI) by NewTek [6] on the other hand is an open
standard with fully developed software and Software Development Kit (SDK),
designed to allows for easy integration of IP broadcasting into existing
software by utilising the NDI SDK.

###  2.2 Audio Recognition in Broadcasting

There has been some work conducted in the broadcasting related to recognition
of audio events [2]. However mostly related to speech recognition and
transcription that is commonly used for tagging content for archiving
purposes. Recognising speech allows for easy searching without having to
manually tag content. For example, Raimond et al. [3] describe a system to
automate the tagging of content within the BBC’s radio archive based on speech
audio. Levin et al. [4] describes a system using automatic speech recognition
for captioning of news programming. This system runs against a re-speaker
which in this context is a person repeating speech in a more readable and
understandable way for the system, avoiding the issues surrounding the
acoustic environment and overlapping speakers. However, this systems only
supports the processing of speech and does not consider sound events in
general. More modern solutions proprietary do exists [7, 8] which remove the
requirement for a re-speaker but are still incapable of including sound
events. Additionally, BBC Research and Development [2] have designed an
application program (a software) to identify sound events for the purpose of
audio monitoring.

In contrast to previous work, we separate the audio tagging software from any
other application programs. In our work, a modular approach is opted and a
container specifically for general audio tagging is built to allow multiple
applications on the network to take advantage of the technology without
repeating work. This is helpful considering the computational overhead
associated with AI models. Our system can include the monitoring system as
described by BBC Research and Development [2] in addition to other systems
e.g. for captioning.

##  3 System Design

###  3.1 Selecting IP Broadcasting Technology

For our work, we need an IP technology that is both well used by the industry
allowing for wide adoption of the audio tagging technology and is simple to
implement. Standards have been created to support new IP based workflows. One
example from the Society of Motion Picture and Television Engineers (SMPTE) is
the ST 2110 suite of standards, which describes the transport of compressed
and uncompressed audio, video and metadata via individual Real-time Transport
Protocol (RTP) streams. However, the complexity of understanding these
standards means that it is only practical for large corporations to implement.
Alternative standards such as the Network Device Interface (NDI), which is an
open standard created by NewTek [6], has an easy to use Software Development
Kit (SDK). Due to easy integration, NDI is available in a wide variety of
software and hardware applications, enabling its wide spread adoption in both
large and small operations. This has lead to NDI being the selected technology
for our work. NDI transports data in the form of frames that contain the
relevant data as well as supporting information to support its use. There are
three types of frames used by NDI: audio, video and metadata. NDI also handles
the detection of sources allowing for routing of NDI frames.

###  3.2 AI Model used for Audio Tagging

To identify audio tags, we leverage AI models particularly convolutional
neural networks (CNN) that has shown remarkable performance in many audio
classification tasks [9, 10]. For example, pre-trained audio neural networks
(PANNs) [10] have been widely used to recognize a variety of audio events. A
description of AI models used in this paper for predicting the audio tags is
given below,

Pre-trained Audio Neural Networks (PANNs):  CNN14 [10] is a pre-trained audio
neural network that is trained on Google Audioset dataset [11]. CNN14 is
trained by extracting the log-mel spectrograms from the audio clips. CNN14 has
81M parameters and it takes 21G multiply-accumulate operations to predict tags
of the audio of length 10 seconds. The trained CNN14 can predict wide range of
sound events such as car passing by, speech, siren, animal etc. This helps
identifying sounds in the wide array of possible scenarios the system could be
exposed to, i.e. different types of broadcast programming such as news
gathering in various locations or a panel show within a studio.

Efficient PANNs:  E-PANNs [12] is an efficient version of original PANNs with
reduced memory requirement (24M parameters) and a reduced computational
complexity (13G MACs per 10 seconds audio). The efficient AI models are
beneficial in an IP networking environment, especially one involving inelastic
traffic (network traffic that is sensitive to variations in delay, e.g. audio
and video streams). This will be explored in Section 5.

###  3.3 NDI Integration

We use the NDI SDK [13] to create a software module including the PANNs
algorithm. Due to the reliance on Python based packages within PANNs module
such as “PANNs inference” [14], we use Python for implementation.
Specifically, Python binding made by the community [15] to interface between
python and the C++ SDK are used to enable NDI support. An additional Python
package is created to simplify the process of integrating NDI into both the
PANNs module and the suggested proof of concept applications described in
Section 4.

Our python package contains three classes: A receiver, transmitter and finder.
This allows an application to receive frames using the receiver class from a
given NDI source, which is detected using the finder class. These can then be
processed and transmitted by creating its own NDI source using the transmitter
class. An example of how this is used here can be seen in Figure 1. It is
important to note here that the flow of audio, video and metadata frames is
uninterrupted between the receiver and transmitter. Each audio frame is
intercepted and a copy is taken for analysis while the original copy is sent
straight to the transmitter, minimising the delay and jitter. One issue
surrounding the community supplied Python bindings were the associated bugs,
especially surrounding memory management. This led to having to convert each
frame to a Python dataclass so that it could be effectively freed and delt
with by the Python garbage collector, an issue that would not have been
encountered using the original C++ SDK.

Figure 2: Metadata generation pipeline.

###  3.4 Integrating Sound Events Metadata

In order to produce sound event predictions from PANNs model and make it
compatible with other NDI applications, we follow the pipeline as shown in
Figure 2 that takes the incoming audio frames from NDI and creates metadata
frames containing the audio tag to be sent across the network.

We use two ring buffers, first ring buffer stores incoming audio frames. From
each audio frame, we extract the individual audio samples and store them in a
second ring buffer. Once a sufficient number of samples has been collected in
the second ring buffer the entire contents of the ring buffer is fed into the
PANNs model. The size of the second ring buffer is crucial as it determines
the duration of the audio window that PANNs analyses. The impact on the size
of the window on the models latency is discussed in Section 5. To distribute
the predicted sound event across the NDI network, we use metadata frames.
These frames transport XML data, which can include third-party metadata as
used here. The output string from PANNs is inserted into an XML template for
transmission. Other NDI applications can then receive this XML via the
metadata frames to access the sound event prediction.

A summary of various steps is explained below,

  1. 1.

Store received audio frames in ring buffer one.

  2. 2.

Extract the floating point Pulse Code Modulated (PCM) audio samples from each
frame and store these in ring buffer two.

  3. 3.

Wait until a given number of samples have been collected.

  4. 4.

Feed the entire contents of ring buffer two into AI model.

  5. 5.

Generate a metadata frame containing the prediction from AI model.

Figure 3: Proposed integrated pipeline: Audiowatch [2] framework with a
separated audio tagging unit.

##  4 Example Workflow

The proposed containerised component allows for the integration of audio
tagging capabilities into a multitude of different systems and use cases.
Below, we provide two examples of integration of audio tagging system into
existing IP broadcasting framework,

###  4.1 Audiowatch Example

Figure 3 demonstrates our system inspired by the BBC audiowatch project, where
we integrate a separate audio tagging software from other application
programs. We use Docker [1] for containerisation, creating multiple instances
of the audio tagging software to analyse several NDI sources simultaneously. A
sound event detection front end is a dashboard user interface as shown in
Figure 2 and it generates metadata corresponding to input audio. Metadata
containing sound event information is then sent to the icon selector module
for processing. Next, various icon selector containers extract the sound
events from the audio track supplied within the metadata frames. After
identifying the unwanted sound events, an appropriate icon overlay is
transmitted as an NDI video frame. Next, a video mixing software such as Open
Broadcaster Software (OBS) [16] is used to superimpose the icon onto the
original video source for displaying on the operators multiview, which is used
to monitor all video sources.

###  4.2 Online Closed Captioning

Another example integration could be the use of audio tagging to enhance
closed captioning. As discussed in Section 2.2 while work has been conducted
to automated closed captioning in real time using automatic speech
recognition, these do not include descriptions of sound events. By combining
the two technologies, full closed captioning could be achieved. This would
involve first parsing the audio through the audio tagging model using our
container. When the result is returned as human speech, the audio would then
be passed through a second speech recognition model to generate subtitles. One
major concern would be the accuracy of the audio tagging model. If the speech
was not always detected, we would miss large portions of speech text.
Additionally the difference in latency between a sound event being inserted
and speech going through two models would have to be accounted for.

##  5 AI model Integration Challenges

There are a number of integration challenges to consider while designing AI
based software fit for broadcasting. These challenges include the accuracy of
the prediction and the latency of the model delaying the signal. Generally,
PANNs and E-PANNs give similar prediction results.

Model latency: The latency of the model here describes the amount of time it
takes given a number of audio samples to produce an accurate sound event
prediction. Consideration of the model latency is significant given that we
are dealing with inelastic audio and video network traffic. This means that
any delay in processing contributes to a delay in the resulting transmission
depending on the infrastructure. Delay can be mitigated using a design similar
to shown in Figure 1, however there is the issue of predictions being desynced
to the audio track. Although we have minimal control over the IP network using
the audio tagging module, and thus cannot manage the network’s latency, we can
still select an optimal model that minimises latency while maintaining
accuracy.

Buffer size versus model latency: To analyse buffer size and latency of model,
we perform experimentation using a set of audio recordings with known sound
events. The first audio recording is taken directly from the PANNs repository
that involves a telephone ringing followed by human speech and is of seven
seconds. The second audio recording of a car driving into the distance. The
third audio recording is created by mixing a car driving and a running river
sound events.

Given the audio recordings, we analyse latency of the AI model at different
number of audio samples. We generate different length audio segments. The
audio samples taken are of multiples of 1024 (assuming frames containing 1024
samples are used) and represents the size of the buffer. Given the audio
samples of different length, we use the PANNs or E-PANNs model to produce
predictions while measuring the time taken for the model to produce a
prediction. Figure 4 shows latency by PANNs and E-PANNs model at different
buffer size. Both PANNs and E-PANNs follow a similar trajectory with E-PANNs
showing a considerable improvement in latency. This suggests that choosing an
appropriate model contribute to improve latency and hence making integration
of audio events more real-time while using less resources.

It is found that a buffer size of 48128 samples (47 * 1024 sample frames) is a
sensible choice in having a low latency while producing an accurate result in
detecting the sound events correctly. This equates to an audio window with a
duration of 1.002s sampled at 48KHz, that gives correct results with the
minimal latency. Prediction results and model latency computed on a AMD Ryzen
5 2500U and Intel Core i9-13900HX hardware can be found here.

Figure 4: PANNs/E-PANNs model latency vs buffer size when inputting audio
sampled at 48KHz. Experiments are performed a on AMD Ryzen 5 2500U system at
2GHz.

##  6 Discussion and Conclusion

The integration of IP broadcasting with audio tagging offers significant
potential for enhancing broadcast workflows, but it also presents several
challenges. The transition to IP broadcasting enables a more flexible,
scalable, and reconfigurable infrastructure compared to traditional methods
based on Serial Digital Interface (SDI). This flexibility is further enhanced
by containerisation technologies making the system more resilient and
adaptable. However, implementing an audio tagging system introduces challenges
primarily related to latency and the accuracy of audio tagging models.

One of the primary challenges discussed is the latency associated with the
audio tagging model. Given the real-time nature of broadcasting, any delays
introduced by processing can impact the overall operation. This makes the
choice of buffer size crucial. A smaller buffer reduces latency but might
compromise the accuracy of sound event detection. Conversely, a larger buffer
improves accuracy but increases latency. The experiments conducted show that
an acceptable balance can be achieved with a buffer size of 48128 samples,
which provides an acceptable latency while maintaining accuracy. The use of
Efficient PANNs (E-PANNs) further helps in reducing the computational
complexity and memory requirements, making it a suitable choice for real-time
applications.

Containerisation offers a robust solution to scalability issues. By isolating
the audio tagging functionality into a microservice, it becomes possible to
scale the system by simply adding more containers as needed. This isolation
also ensures that a fault in one container does not affect the entire system,
enhancing overall reliability. The use of Docker to containerise these
services allows for easy deployment and management across different network
setups. Additionally, the integration with NDI technology, which is widely
adopted in the industry, ensures broad applicability.

Despite these advantages, real-world deployment of such a system is not
without hurdles. The reliance on Python bindings to interface with the NDI
SDK, while practical, introduces potential issues with memory management that
need careful handling.

###  6.1 Conclusion

Integrating IP broadcasting with audio tagging presents a promising
advancement for the broadcasting industry. The use of containerisation and
audio tagging for real-time sound event detection can significantly enhance
content production and accessibility. However, addressing the challenges of
latency, accuracy and real-world deployment is crucial for the successful
implementation of this technology. Future work includes re-writing the
codebase to use the NDI C++ SDK directly, avoiding the issues surrounding the
Python bindings. Additionally, we would like to analyse more complex models
such as transformers [17, 18] within our broadcasting framework. Finally, the
creation of the discussed proof of concept applications would allow for full
demonstration of the usefulness of this technology.

##  7 Acknowledgements

This work was supported by Engineering and Physical Sciences Research Council
(EPSRC) Grant EP/T019751/1 “AI for Sound (AI4S)”. For the purpose of open
access, the authors have applied a Creative Commons Attribution (CC BY)
licence to any Author Accepted Manuscript version arising.

## References

  * [1] Docker: Accelerated container application development. [Online]. Available: https://www.docker.com/
  * [2] S. Ward and R. Dawes. AudioWatch - live audio monitoring for autumnwatch 2021 - BBC r&d. [Online]. Available: https://www.bbc.co.uk/rd/blog/2021-11-live-audio-monitoring-autumnwatch-ai
  * [3] Y. Raimond, C. Lowis, R. Hodgson, and J. Tweed, “Automated semantic tagging of speech audio,” in _Proceedings of the 21st International Conference on World Wide Web_ , 2012, pp. 405–408.
  * [4] K. Levin, I. Ponomareva, A. Bulusheva, G. Chernykh, I. Medennikov, N. Merkin, A. Prudnikov, and N. Tomashenko, “Automated closed captioning for russian live broadcasting,” in _Fifteenth Annual Conference of the International Speech Communication Association_ , 2014.
  * [5] “SMPTE OV 2110-0:2018 - SMPTE overview document - professional media over managed IP networks roadmap for the 2110 document suite.” [Online]. Available: https://ieeexplore.ieee.org/document/8626804
  * [6] NewTek, “NDI 5.6 white paper.” [Online]. Available: https://ndi.video/wp-content/uploads/2023/09/NDI-5.6-White-Paper-2023.pdf
  * [7] Closed captioning software - IBM watson media. [Online]. Available: https://www.ibm.com/products/video-streaming/closed-captioning
  * [8] enCaption: Automated closed captioning system | ENCO systems. [Online]. Available: https://www.enco.com/products/encaption
  * [9] H. Purwins, B. Li, T. Virtanen, J. Schlüter, S.-Y. Chang, and T. Sainath, “Deep learning for audio signal processing,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 13, no. 2, pp. 206–219, 2019.
  * [10] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2880–2894, 2020.
  * [11] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in _IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , 2017, pp. 776–780.
  * [12] A. Singh, H. Liu, and M. D. Plumbley, “E-panns: Sound recognition using efficient pre-trained audio neural networks,” in _INTER-NOISE and NOISE-CON Congress and Conference Proceedings_ , vol. 268, no. 1. Institute of Noise Control Engineering, 2023, pp. 7220–7228.
  * [13] N. INC, “Ndi sdk v6.0.1.” [Online]. Available: https://ndi.video/for-developers/ndi-sdk
  * [14] Q. kong, “qiuqiangkong/panns_inference,” original-date: 2020-03-08T06:22:30Z. [Online]. Available: https://github.com/qiuqiangkong/panns˙inference
  * [15] N. Kondo, “buresu/ndi-python,” original-date: 2019-04-16T10:59:04Z. [Online]. Available: https://github.com/buresu/ndi-python
  * [16] “obsproject/obs-studio,” original-date: 2013-10-01T02:40:31Z. [Online]. Available: https://github.com/obsproject/obs-studio
  * [17] Y. Gong, Y.-A. Chung, and J. Glass, “Ast: Audio spectrogram transformer,” _Interspeech_ , 2021.
  * [18] F. Schmid, K. Koutini, and G. Widmer, “Efficient large-scale audio tagging via transformer-to-cnn knowledge distillation,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.


We set $R = \sqrt{\dx} + \sqrt{2(1+\beta)\log{T}}$ where $\beta \geq 2$ is a free parameter.
Define the event $\calE$ as:

\begin{align}
\calE := \left\{ \max_{0 \leq t \leq T-1} \norm{V_t}_2 \leq R \right\}. \label{eq:GLM_good_truncation_event}
\end{align}

Note that by the setting of $R$, we have $\Pr(\calE^c) \leq 1/T^{\beta}$ using standard Gaussian concentration results plus a union bound. Furthermore on $\calE$, the original GLM process
driven by Gaussian noise (<ref>)
coincides with the truncated process (<ref>).
Let $\widehat{f}$ denote the LSE on the original process
(<ref>), and
let $\bar{f}$ denote the LSE on the truncated process (<ref>).

\begin{align}
\E\norm{\widehat{f} - f_\star}_{L^2}^2 &= \E \norm{\widehat{f} - f_\star}_{L^2}^2 \ind\{\calE\} + \E \norm{\widehat{f} - f_\star}_{L^2}^2 \ind\{\calE^c\} \nonumber \\
&\leq \E\norm{\bar{f} - f_\star}_{L^2}^2 + \E \norm{\widehat{f} - f_\star}_{L^2}^2 \ind\{\calE^c\} \label{eq:glm_truncate_decomp}.
\end{align}

Let us now control the error term
$\E \norm{\widehat{f} - f_\star}_{L^2}^2 \ind\{\calE^c\}$.
Write $\widehat{f}(x) = \sigma(\widehat{A} x)$, and put $\widehat{\Delta} = \widehat{A}-A_\star$.
We have:

\begin{align}
\E \norm{\widehat{f} - f_\star}_{L^2}^2 \ind\{\calE^c\} &= \frac{1}{T} \sum_{t=0}^{T-1} \E\norm{\sigma(\widehat{A} X_t) - \sigma(A_\star X_t) }_2^2 \ind\{\calE^c\} \stackrel{(a)}{\leq} \frac{1}{T}\sum_{t=0}^{T-1} \E\norm{\widehat{\Delta} X_t}_2^2\ind\{\calE^c\} \nonumber \\
&\stackrel{(b)}{\leq} \frac{4B^2}{T} \sum_{t=0}^{T-1} \E\norm{X_t}_2^2 \ind\{\calE^c\}
\stackrel{(c)}{\leq} \frac{4B^2}{T^{1 + \beta/2}}\sum_{t=0}^{T-1} \sqrt{\E\norm{X_t}_2^4}
\stackrel{(d)}{\leq} \frac{4B^2 B_X^2}{T^{\beta/2}}. \label{eq:glm_fhat_lse_error_term}
\end{align}

Here, (a) follows since $\sigma$ is $1$-Lipschitz,
(b) uses the definition of $\scrF$ in (<ref>), (c) follows by
Cauchy-Schwarz, and (d) uses <Ref>.

The remainder of the proof is to bound the
LSE error $\E\norm{\bar{f} - f_\star}_{L^2}^2$.
First, we establish an almost sure bound on
$\{\bar{X}_t\}_{t \geq 0}$.

Consider the truncated GLM process (<ref>).
Under <Ref>, the process $\{\bar{X}_t\}_{t \geq 0}$ satisfies:

\begin{align}
\sup_{t \in \N} \norm{\bar{X}_t}_{P_\star} \leq \frac{2\opnorm{P_\star}^{1/2} \opnorm{H} (\sqrt{\dx} + \sqrt{2(1+\beta)\log{T}})}{1-\rho} \triangleq B_{\bar{X}}. \label{eq:glm_B_Xbar}
\end{align}

By triangle inequality and (<ref>):

\begin{align*}
\norm{\bar{X}_{t+1}}_{P_\star} &= \norm{ \sigma(A_\star \bar{X}_t) + H \bar{V}_t }_{P_\star} \leq \norm{\sigma(A_\star \bar{X}_t)}_{P_\star} + \norm{H \bar{V}_t}_{P_\star} \\
&\leq \rho^{1/2} \norm{\bar{X}_t}_{P_\star} + \norm{H \bar{V}_t}_{P_\star} \leq \rho^{1/2} \norm{\bar{X}_t}_{P_\star} + \opnorm{P_\star^{1/2} H} R.
\end{align*}

Unrolling this recursion,
and using the fact that $\inf_{x \in [0, 1]} \frac{1-\sqrt{x}}{1-x} = 1/2$ yields the result.

We next establish uniform bounds for the covariance matrices
of the truncated process.

Suppose $T \geq 4$.
Consider the truncated GLM process (<ref>),
and let the covariance matrices for the process $\{\bar{X}_t\}_{t \geq 0}$ be denoted as $\bar{\Gamma}_t \triangleq \E[ \bar{X}_t \bar{X}_t^\T ]$.
Under <Ref>:

\begin{align*}
\frac{1}{2} HH^\T \preccurlyeq \bar{\Gamma}_t \preccurlyeq B_{\bar{X}}^2 \cdot I.
\end{align*}

The upper bound is immediate from <Ref>,
since $\E[\bar{X}_t\bar{X}_t^\T] \preccurlyeq \E[\norm{\bar{X}_t}_2^2] I \preccurlyeq B_{\bar{X}}^2 I$.
For the lower bound, it is immediate when $t=0$
using <Ref>.
On the other hand, for $t \geq 1$, since $\bar{V}_t$ is zero-mean:

\begin{align*}
\E[ \bar{X}_t \bar{X}_t^\T] &= \E[(\sigma(A_\star \bar{X}_{t-1}) + H\bar{V}_{t-1})(\sigma(A_\star \bar{X}_{t-1}) + H\bar{V}_{t-1})^\T] \\
&= \E[\sigma(A_\star \bar{X}_{t-1}) \sigma(A_\star \bar{X}_{t-1})^\T] + \E[ H \bar{V}_{t-1} \bar{V}_{t-1}^\T H^\T]
\succcurlyeq \E[ H \bar{V}_{t-1} \bar{V}_{t-1}^\T H^\T] \succcurlyeq \frac{1}{2}HH^\T.
\end{align*}

The last inequality again holds from <Ref>.

§.§.§ Trajectory hypercontractivity for truncated GLM

For our purposes, the link function assumption in
<Ref> ensures the following
approximate isometry inequality which holds for all $x \in \R^{\dx}$ and all
matrices $A,A' \in \R^{\dx \times \dx}$:

\begin{align}
\zeta^2 \norm{Ax-A'x}_2^2 \leq \norm{\sigma(Ax)-\sigma(A'x)}_2^2 \leq \norm{Ax-A'x}_2^2. \label{eq:glm_approx_isometry}
\end{align}

This inequality is needed to establish trajectory hypercontractivity for $\scrF_\star$.

Suppose that $T \geq 4$.
Fix any matrix $A \in \R^{\dx \times \dx}$.
Under <Ref>, the truncated process (<ref>) satisfies:

\begin{align}
\frac{1}{T} \sum_{t=0}^{T-1} \E\norm{\sigma(A \bar{X}_t) - \sigma(A_\star \bar{X}_t)}_2^4 \leq \frac{4B_{\bar{X}}^4}{\sigma_{\min}(H)^4 \zeta^4}  \left(\frac{1}{T}\sum_{t=0}^{T-1}\E\norm{\sigma(A \bar{X}_t) - \sigma(A_\star \bar{X}_t)}_2^2\right)^2.
\end{align}

Hence, the function class $\scrF_\star$ with $\scrF$ defined in
(<ref>) satisfies the $(C_{\mathsf{GLM}}, 2)$-trajectory
hypercontractivity condition
with $C_{\mathsf{GLM}} = \frac{4B_{\bar{X}}^4}{\sigma_{\min}(H)^4 \zeta^4}$.

Put $\Delta \triangleq A - A_\star$ and $M \triangleq \Delta^\T \Delta$.
We have:

\begin{align*}
\E \norm{\Delta \bar{X}_t}_2^4 &= \E[ \bar{X}_t^\T M \bar{X}_t \bar{X}_t^\T M \bar{X}_t ] \\
&\leq B_{\bar{X}}^2 \tr(M^2 \bar{\Gamma}_t) &&\text{using \Cref{prop:glm_truncated_state_bounds}} \\
&\leq B_{\bar{X}}^2 \opnorm{M} \tr(M \bar{\Gamma}_t) &&\text{H{\"{o}}lder's inequality}\\
&\leq B_{\bar{X}}^2 \tr(M) \tr(M \bar{\Gamma}_t) &&\text{since $M$ is positive semidefinite} \\
&\leq B_{\bar{X}}^4 \tr(M)^2 &&\text{using \Cref{prop:glm_trunc_cov_bounds}} \\
&\leq \frac{B_{\bar{X}}^4}{\lambda_{\min}(HH^\T)^2} \tr(M HH^\T)^2.
\end{align*}

On the other hand, by <Ref>:

\begin{align*}
\E\norm{\Delta \bar{X}_t}_2^2 = \tr(M \bar{\Gamma}_t) \geq \frac{1}{2}\tr(M HH^\T).
\end{align*}

Combining these bounds yields:

\begin{align*}
\frac{1}{T}\sum_{t=0}^{T-1}\E\norm{\Delta \bar{X}_t}_2^4 \leq \frac{B_{\bar{X}}^4}{\lambda_{\min}(HH^\T)^2} \tr(M HH^\T)^2 \leq \frac{4B_{\bar{X}}^4}{\lambda_{\min}(HH^\T)^2}   \left( \frac{1}{T}\sum_{t=0}^{T-1} \E\norm{\Delta \bar{X}_t}_2^2 \right)^2.
\end{align*}

The claim now follows via the approximate isometry inequality (<ref>).

§.§.§ Bounding the dependency matrix for truncated GLM

We will use the result in <Ref>
to bound the total-variation distance by the $1$-Wasserstein distance.
This is where the non-degenerate noise assumption in <Ref>
is necessary.

The starting point is the observation that the diagonal Lyapunov function
in <Ref> actually yields
incremental stability <cit.> in addition to Lyapunov stability. In particular, let $\{a_i\}$ denote the rows of $A_\star$.
For any $x,x' \in \R^{\dx}$:

\begin{align}
\norm{\sigma(A_\star x) - \sigma(A_\star x')}^2_{P_\star} &= \sum_{i=1}^{\dx} (P_\star)_{ii} (\sigma(\ip{a_i}{x}) - \sigma(\ip{a_i}{x'}))^2 \nonumber \\
&\leq \sum_{i=1}^{\dx} (P_\star)_{ii} (\ip{a_i}{x} - \ip{a_i}{x'})^2 \nonumber \\
&= (x-x')^\T A_\star^\T P_\star A_\star (x-x') \nonumber \\
&\leq \rho \norm{x-x'}_{P_\star}^2. \label{eq:glm_incremental_stability}
\end{align}

This incremental stability property allows us to control the dependency matrix as follows.

Consider the truncated GLM process $\{\bar{X}_t\}_{t \geq 0}$
from (<ref>).
Let $\Pxbar$ denote the joint distribution of $\{\bar{X}_t\}_{t=0}^{T-1}$.
Under <Ref> and when $B \geq 1$,
we have that:

\begin{align*}
\opnorm{\Gammadep(\Pxbar)} \leq \frac{22}{1-\rho} \log\left( \frac{B \sqrt{\dx}(B_{\bar{X}} + B_X)}{2\sigma_{\min}(H)}\right).
\end{align*}

Let $\{X_t\}_{t \geq 0}$ denote the original GLM dynamics from (<ref>).
Fix indices $t \geq 0$ and $k \geq 1$.
We construct a coupling of $(\sfP_{X_{t+k}}(\cdot \mid X_t=x), \sfP_{X_{t+k}})$ as follows.
Let $\{V_t\}_{t \geq 0}$ be  $N(0, I)$.
Let $\{Z_s\}_{s \geq t}$ be the process such that
$Z_{t} = x$, and follows the GLM dynamics (<ref>) using the noise $\{V_t\}_{t \geq 0}$
(we do not bother defining $Z_{t'}$ for $t' < t$ since we do not need it).
Similarly, let $\{Z'_s\}_{s \geq 0}$ be the process following the GLM dynamics (<ref>)
using the same noise $\{V_t\}_{t \geq 0}$.
Now we have:

\begin{align*}
\E\norm{Z_{t+k} - Z'_{t+k}}_{P_\star} &= \E\norm{\sigma(A_\star Z_{t+k-1}) - \sigma(A_\star Z'_{t+k-1})}_{P_\star} \\
&\leq \rho^{1/2} \E\norm{Z_{t+k-1} - Z'_{t+k-1}}_{P_\star} &&\text{using \Cref{eq:glm_incremental_stability}}.
\end{align*}

We now unroll this recursion down to $t$:

\begin{align*}
\E\norm{Z_{t+k} - Z'_{t+k}}_{P_\star} \leq \rho^{k/2} \E\norm{Z_t - Z'_t}_{P_\star} = \rho^{k/2} \E\norm{x - Z'_t}_{P_\star}.
\end{align*}

Since $P_\star \succcurlyeq I$, this shows that:

\begin{align*}
W_1(\sfP_{X_{t+k}}(\cdot \mid X_t = x), \sfP_{X_{t+k}}) \leq \rho^{k/2} (\norm{x}_{P_\star} + \E\norm{X_{t}}_{P_\star}) \leq \rho^{k/2}(\norm{x}_{P_\star} + B_X),
\end{align*}

where the last inequality follows from <Ref> and Jensen's inequality.

Next, it is easy to see the map $x \mapsto \sigma(A_\star x)$
is $\opnorm{A}$-Lipschitz.
Furthermore, since $H$ is full rank
by <Ref>, then for any $t$ and $k \geq 1$ both
$\sfP_t$ and
$\sfP_{X_{t+k}}(\cdot \mid X_t=x)$
are absolutely continuous
w.r.t. the Lebesgue measure in $\R^{\dx}$.
Using <Ref>, we have for any
$k \geq 2$:

\begin{align*}
\tvnorm{\sfP_{X_{t+k}}(\cdot \mid X_t = x) - \sfP_{X_{t+k}}} &\leq \frac{\opnorm{A_\star} \sqrt{\tr((HH^\T)^{-1})}}{2}  W_1(\sfP_{X_{t+k-1}}(\cdot \mid X_t = x), \sfP_{X_{t+k-1}}) \\
&\leq \frac{\opnorm{A_\star} \sqrt{\tr((HH^\T)^{-1})}}{2} \rho^{(k-1)/2} (\norm{x}_{P_\star} + B_X).
\end{align*}

Using <Ref> to bound $\opnorm{\Gammadep(\Pxbar)}$ (which is valid because we constrained $\beta \geq 2$),
and <Ref> to bound $x \in \bar{\sfX}_t$,
for any $\ell \geq 1$:

\begin{align*}
\opnorm{\Gammadep(\sfP_{\bar{X}})} &\leq 3 + \sqrt{2} \sum_{k=1}^{T-1} \max_{t=0,\dots,T-1-k} \esssup_{x \in \bar{\sfX}_t} \sqrt{\tvnorm{\sfP_{X_{t+k}}(\cdot \mid X_t=x) - \sfP_{X_{t+k}}}} \\
&\leq 3 + \sqrt{2}\ell + \left[ \frac{\opnorm{A_\star} \sqrt{\tr((HH^\T)^{-1})} (B_{\bar{X}} + B_X)}{2} \right]^{1/2} \sum_{k=\ell+1}^{T-1} \rho^{(k-1)/4} \\
&\stackrel{(a)}{\leq} 5\ell + \left[ \frac{B \sqrt{\dx} (B_{\bar{X}} + B_X)}{2\sigma_{\min}(H)} \right]^{1/2} \frac{\rho^{\ell/4}}{1-\rho^{1/4}}.
\end{align*}

Above, (a) uses the bounds $\opnorm{A_\star} \leq B$ and
$\tr(HH^{-1}) \leq \dx/\sigma_{\min}(H)^2$.
Now put $\psi \triangleq \frac{B \sqrt{\dx} (B_{\bar{X}} + B_X)}{2\sigma_{\min}(H)}$.
We choose $\ell = \bigceil{\frac{\log(\sqrt{\psi})}{1-\rho^{1/4}}}$ so that
$\rho^{\ell/4} \leq 1/\sqrt{\psi}$.
This yields:

\begin{align*}
\opnorm{\Gammadep(\sfP_{\bar{X}})} &\leq \frac{11\log{\psi}}{2(1-\rho^{1/4})} \stackrel{(a)}{\leq} \frac{22 \log{\psi}}{1-\rho} = \frac{22}{1-\rho} \log\left( \frac{B \sqrt{\dx}(B_{\bar{X}} + B_X)}{2\sigma_{\min}(H)}\right).
\end{align*}

Above, (a) follows from
$\inf_{x \in [0,1]} \frac{1-x^{1/4}}{1-x} = 1/4$.

§.§.§ Finishing the proof of <Ref>

Below, we let $c_i$ be universal positive constants
that we do not track precisely.

For any $\e>0$ we now construct an $\e$-covering of $\scrF_\star \setminus B(r)$, with $\scrF_\star$ the offset class of $\scrF$
from (<ref>).
Note that we are not covering $\partial B(r)$ since the class
$\scrF_\star$ is not star-shaped. However, an inspection
of the proof of <Ref> shows that
one can remove the star-shaped assumption by instead
covering the set $\scrF_\star \setminus B(r)$.
To this end, we let $\{A_1,\dots,A_N\}$ be a $\delta$-cover of $\scrA \triangleq \{A \in \mathbb{R}^{\dx \times \dx} \mid \|A\|_F \leq B \}$,
for a $\delta$ to be specified.
By a volumetric argument we may choose $\{A_1,\dots,A_N\}$ such that $N \leq \left(1 +\frac{2B}{\delta} \right)^{\dx^2}$.
Now, any realization of $\{\bar{X}_t\}$ will have norm less than
$B_{\bar{X}}$ from (<ref>),
where $B_{\bar{X}}$ is bounded by:

\begin{align*}
B_{\bar{X}} \leq \frac{c_0\opnorm{P_\star}^{1/2} \opnorm{H} (\sqrt{\dx} + \sqrt{(1+\beta)\log{T}})}{1-\rho}.
\end{align*}

Now fix any $A \in \scrA$,
and let $A_i$ be an element in the $\delta$-cover satisfying $\norm{A - A_i}_F \leq \delta$.
We observe that for any $x$ satisfying $\norm{x}_2 \leq B_{\bar{X}}$:

\begin{align*}
\norm{(\sigma(A_ix) - \sigma(A_\star x)) - (\sigma(A x) - \sigma(A_\star x))}_2 &= \norm{\sigma(A_i x) - \sigma(A x)}_2 \leq \norm{(A_i - A) x}_2 \\
&\leq \norm{A_i-A}_F\norm{x}_2 \leq \delta B_{\bar{X}}.
\end{align*}

Thus, it suffices to take $\delta = \e /B_{\bar{X}}$
to construct the $\e$ cover of $\scrF_\star$,
i.e., $\calN_\infty(\scrF_\star, \e) \leq \left(1 +\frac{2BB_{\bar{X}}}{\e}\right)^{\dx^2}$. This then implies <cit.>:

\begin{align*}
\calN_\infty(\scrF_\star \setminus B(r), \e) \leq \calN_\infty(\scrF_\star, \e/2) \leq \left(1 +\frac{4BB_{\bar{X}}}{\e}\right)^{\dx^2}.
\end{align*}

Next, by <Ref>, $(\scrF_\star, \Pxbar)$ is $(C_{\mathsf{GLM}},2)$-hypercontractive for all $T \geq 4$,

\begin{align*}
C_{\mathsf{GLM}} \leq \frac{4B_{\bar{X}}^4}{\sigma_{\min}(H)^4 \zeta^4} \leq \frac{c_1 \opnorm{P_\star}^2 \mathrm{cond}(H)^4 (\dx^2 + ((1+\beta)\log{T})^2) }{  \zeta^4 (1-\rho)^4 }.
\end{align*}

Furthermore, by <Ref>:

\begin{align*}
\opnorm{\Gammadep(\Pxbar)}^2 \leq \frac{c_2}{(1-\rho)^2} \log^2\left( \frac{B \sqrt{\dx}(B_{\bar{X}} + B_X)}{2\sigma_{\min}(H)}\right) \triangleq \gamma^2.
\end{align*}

The class $\scrF_\star$ is $2BB_{\bar{X}}$-bounded on (<ref>).
Invoking <Ref>, for every $r>0$:

\begin{align}
\E \|\bar{f}- f_\star\|_{L^2}^2 \leq 8 \E \bar{\sfM}_T(\mathscr{F}_\star) + r^2 + 4B^2B_{\bar{X}}^2 \left(1 +\frac{4\sqrt{8} B B_{\bar{X}}}{r} \right)^{\dx^2} \exp \left( \frac{-T  }{8C_{\mathsf{GLM}} \gamma^2 } \right). \label{eq:glm_truncated_LSE}
\end{align}

Here, the notation $\E \bar{\sfM}_T(\scrF_\star)$ is meant to emphasize
that the offset complexity is with respect to the truncated process
$\Pxbar$ and not the original process $\Px$.
We now set $r^2 = \opnorm{H}^2 \dx^2/T$,
and compute a $T_0$ such that the third term in (<ref>) is also bounded by $\opnorm{H}^2 \dx^2/T$.
To do this, it suffices to compute $T_0$ such that for all
$T \geq T_0$:

\begin{align*}
T \geq c_3 C_{\mathsf{GLM}} \gamma^2 \dx^2 \log\left( \frac{T B B_{\bar{X}}}{\opnorm{H} \sqrt{\dx}} \right).
\end{align*}

It suffices to take (assuming $\beta$ is at most polylogarithmic in any problem constants):

\begin{align}
T_0 \geq c_4
\frac{\opnorm{P_\star}^{2} \mathrm{cond}(H)^4  \dx^4}{\zeta^4 (1-\rho)^6} \mathrm{polylog}\left( B, \dx, \opnorm{P_\star}, \mathrm{cond}(H), \frac{1}{\zeta}, \frac{1}{1-\rho} \right).
\label{eq:glm_T0_bound}
\end{align}

Again, we do not attempt to compute the exact power of the
polylog term, but note it can in principle be done via

Next, from (<ref>) we have that
the error term
$\E \norm{\widehat{f} - f_\star}_{L^2}^2 \ind\{\calE^c\} \leq \frac{4B^2 B_X^2}{T^{\beta/2}}$.
Thus if we further constrain $\beta > 2$ and require $T_0 \geq c_5 \left[ \frac{B^2 \opnorm{P_\star}}{(1-\rho)^2} \right]^{\frac{1}{\beta/2-1}}$,
$\E \norm{\widehat{f} - f_\star}_{L^2}^2 \ind\{\calE^c\} \leq \frac{\opnorm{H}^2 \dx^2}{T}$.
Note that setting $\beta = \max\{3, c_6 \log{B}\}$ suffices.

To finish the proof, it remains to bound $\E \bar{\sfM}_T(\scrF_\star)$.
Now, unlike the linear dynamical systems case,
there is no closed-form expression for $\E \bar{M}_T(\scrF_\star)$.
Hence, we will bound it via the chaining bound (<ref>). This computation is done in <cit.>.
Before we can use the result, however, we need to verify that
the truncated noise process $\{H \bar{V}_t\}_{t \geq 0}$ is
a sub-Gaussian MDS. The MDS part is clear since $\bar{V}_{t} \perp \bar{V}_{t'}$ for $t \neq t'$, and $\bar{V}_t$ is zero-mean.
Furthermore, <Ref>
yields that $H \bar{V}_t$ is a $4\opnorm{H}^2$-sub-Gaussian random vector. Hence, we have:

\begin{align*}
\E \bar{\sfM}_T(\scrF_\star) \leq c_7 \frac{\opnorm{H}^2 \dx^2}{T} \log(1 + \opnorm{H} \sqrt{\dx} B B_{\bar{X}} T^2).
\end{align*}

The claim now follows.