###### Proof.

We prove the inequality for $A$ and $r<\infty$; the others are identically
proved. We write $x\lesssim y$ to mean $x<ky$ for some constant $k$ not
depending on $M$. Suppose $p>0$. A variant of [DMS21, Lemma A.5] gives, with
$j^{*}$ the value of $t\in[j,j+1]$ maximizing $X_{t}$, we have

$\mathbb{E}[(\sum_{j=0}^{\lfloor r\rfloor}\gamma X_{j^{*}})^{p}\mid M]\lesssim
e^{p\gamma M}.$

Then the argument of [DMS21, (A.11)] gives the desired result.

Now suppose $p\leq 0$. Conditon on $M$ and on the time $t_{*}\in(0,r-1)$ at
which $X_{t_{*}}$ is maximal. The conditional law of $(X_{t})_{[t_{*},r-1]}$
is variance 2 Brownian motion with drift $-a$ conditioned to stay below $M$,
and $(X_{t})_{[r-1,r]}$ then evolves as unconditioned variance 2 Brownian
motion with drift $-a$. Thus $Y:=M-\inf_{[t_{*},t_{*}+1]}X_{t}$ is
stochastically dominated by $\hat{Y}:=-\inf_{0<s<1}(B_{2s}-as)$ where $B$ is
standard Brownian motion. Let $h_{2}$ be the projection of $h$ to
$\mathcal{H}_{2}({\mathcal{S}})$, then $h_{2}$ is independent of $(X_{t})$ and
hence $(M_{r},t_{*})$, so writing $S=(t_{*},t_{*}+1)\times(0,\pi)$,

$\displaystyle\mathbb{E}[A^{p}\mid M,t_{*}]$
$\displaystyle\leq\mathbb{E}[{\mathcal{A}}_{\widehat{h}}(S)^{p}\mid
M,t_{*}]\leq\mathbb{E}[{\mathcal{A}}_{h_{2}+M-Y}(S)^{p}\mid M_{r},t_{*}]$
$\displaystyle\leq e^{p\gamma
M}\mathbb{E}[e^{p\gamma\hat{Y}}]\mathbb{E}[{\mathcal{A}}_{h_{2}}(0,1)^{p}]\lesssim
e^{p\gamma M}.$

∎

###### Proof of Proposition 2.17.

Morera’s theorem gives analyticity in $\mu$. Next, [AHS23b, Proposition 2.18]
relates the two-pointed quantum disk weighted by the quantum length of a side
to the three-point Liouville field: writing
$A={\mathcal{A}}_{\phi}({\mathbb{H}}),L_{1}={\mathcal{L}}_{\phi}(-\infty,0),L_{2}={\mathcal{L}}_{\phi}(0,\infty)$
and $s=\beta-Q$, we have

$R_{\mu_{1},\mu_{2}}(\beta)=\frac{1}{Q-\beta}\int(\frac{\gamma}{s}A+\frac{\gamma}{2s}\frac{\gamma}{s+\frac{\gamma}{2}}A(\sum_{i}\mu_{i}L_{i})+\frac{\gamma}{2s}\frac{\frac{\gamma}{2}}{s+\frac{\gamma}{2}}(\sum_{i}\mu_{i}L_{i})^{2})\frac{e^{-A-\sum_{i}\mu_{i}L_{i}}}{L_{2}}\mathrm{LF}_{{\mathbb{H}}}^{(\beta,0),(\gamma,1),(\beta,\infty)}(d\phi).$

Therefore Proposition B.1 gives the analyticity of
$R_{\mu_{1},\mu_{2}}(\beta)$ in $\beta$. ∎

Finally, we give a holomorphicity result for an observable of
${{\mathcal{M}}^{\textrm{disk}}_{2}}(\beta)$.

######  Lemma B.5.

Let $n>\frac{4}{\gamma^{2}}$ be an integer and let $\mu_{1},\mu_{2}>0$, then
the function

$\beta\mapsto{{\mathcal{M}}^{\textrm{disk}}_{2}}(\beta)[L_{1}^{n}e^{-A-\mu_{1}L_{1}-\mu_{2}L_{2}}]\quad\textrm{for
}\beta\in(\frac{\gamma}{2},Q)$

extends holomorphically to a neighborhood of $(\frac{\gamma}{2},Q)$ in
${\mathbb{C}}$. Here, $A$ and $(L_{1},L_{2})$ are the quantum area and
boundary arc lengths respectively of the quantum disk.

###### Proof.

The proof is similar to that of Proposition B.1 but is substantially easier.
We only point out the differences. As before, we assume $\mu_{1}=\mu_{2}=1$,
but the general case follows the same argument. [AHS23b, Proposition 2.18]
explains that the law of a sample from
${{\mathcal{M}}^{\textrm{disk}}_{2}}(\beta)$ weighted by $L_{1}$ can be
described by a three-point Liouville field with insertions
$\beta,\beta,\gamma$; we choose the three-pointed domain
$({\mathbb{H}},p_{1},p_{2},p_{3})$ where $(p_{1},p_{2},p_{3})=(-2,0,2)$. It
thus suffices to show that

$f(\beta)=\int{\mathcal{L}}_{\phi}((p_{1},p_{3}))^{n-1}e^{-{\mathcal{A}}_{\phi}({\mathbb{H}})-\mu{\mathcal{L}}_{\phi}({\mathbb{R}})}\mathrm{LF}_{{\mathbb{H}}}^{(\beta,p_{1}),(\gamma,p_{2}),(\beta,p_{3})}(d\phi)$

extends analytically to a neighborhood of $(\frac{\gamma}{2},Q)$ in
${\mathbb{C}}$.

We use the setup and notation of the proof of Proposition B.1. Instead of $g$,
we use $\hat{g}(a,k,\ell)=\ell^{n-1}e^{-a-k}$ for $a>0$ and $k>\ell>0$, for
which we have the bound

$|\hat{g}(a_{r+1},k_{r+1},\ell_{r+1})-\hat{g}(a_{r},k_{r},\ell_{r})|<C_{1}(\ell_{r+1}^{n-2}(\ell_{r+1}-\ell_{r})e^{-\ell_{r+1}}+\ell_{r}^{n-1}(e^{-k_{r}}-e^{-k_{r+1}})+\ell_{r}^{n-1}(e^{-a_{r}}-e^{a_{r+1}}))$

for $(a_{r+1},k_{r+1},\ell_{r+1})$ and $(a_{r},k_{r},\ell_{r})$ satisfying
$a_{r+1}>a_{r},k_{r+1}>k_{r},\ell_{r+1}>\ell_{r}$.

Our assumption $n>\frac{4}{\gamma^{2}}$ implies $(n-1)+\frac{2s}{\gamma}>0$.
So we can obtain

$\int
e^{(s+(n-1)\frac{\gamma}{2})c}\mathbb{E}[\widetilde{L}_{r+1}^{n-2}(\widetilde{L}_{r+1}-\widetilde{L}_{r})e^{-e^{\frac{\gamma}{2}c}\widetilde{L}_{r+1}}]\,dc=\frac{2}{\gamma}\Gamma((n-1)+\frac{2s}{\gamma})\mathbb{E}[\widetilde{L}_{r+1}^{-\frac{2s}{\gamma}-1}(\widetilde{L}_{r+1}-\widetilde{L}_{r})]$

and two similar identities. These identities and the bound for $\hat{g}$
reduce the problem to proving

$\mathbb{E}[\widetilde{L}_{r+1}^{-\frac{2s}{\gamma}-1}(\widetilde{L}_{r+1}-\widetilde{L}_{r})],\mathbb{E}[\widetilde{L}_{r}^{n-1}(\widetilde{K}_{r}^{-(n-1)-\frac{2s}{\gamma}}-\widetilde{K}_{r+1}^{-(n-1)-\frac{2s}{\gamma}})],\mathbb{E}[\widetilde{L}_{r}^{n-1}(\widetilde{A}_{r}^{-\frac{n-1}{2}-\frac{s}{\gamma}}-\widetilde{A}_{r+1}^{-\frac{n-1}{2}-\frac{s}{\gamma}})]<Ce^{-r/C}$

for sufficiently large $C$ for all $r$. Using Lemma B.3, this can be done
exactly as in Lemma 2.23. ∎

##  Appendix C Operator product expansion

In this appendix we take a look at the Operator Product Expansion (OPE) lemmas
that are used in Section 3. These proofs were first done for LCFT on the
sphere in [KRV20] and then in the boundary case in [RZ22]. The main novelty in
our case is that we must handle a Liouville functional containing both area
and boundary GMC measures and therefore it cannot be reduced to a simple
moment of GMC as in [KRV20, RZ22]. Nonetheless we will be quite brief in
several places of the proofs below when the arguments are similar to those of
[RZ22].

Recall the functions $H_{\chi}$ and $\tilde{H}_{\chi}$ defined in and below
(3.13). We first give the OPE result without reflection which is valid only
for $\chi=\frac{\gamma}{2}$.

######  Lemma C.1.

(OPE without reflection) Let $\chi=\frac{\gamma}{2}$. Assume that the
constraints of (3.7) hold, meaning that
$\sigma_{1},\sigma_{2},\sigma_{3},\sigma_{1}-\frac{\gamma}{4},\sigma_{2}-\frac{\gamma}{4}\in[-\frac{1}{2\gamma}+\frac{Q}{2},\frac{1}{2\gamma}+\frac{Q}{2}]\times\mathbb{R}$,
$\beta_{1},\beta_{2},\beta_{3}<Q$,
$\sum_{i=1}^{3}\beta_{i}>2Q+\frac{\gamma}{2}$. Assume also that
$\beta_{1}\in(\frac{\gamma}{2},\frac{2}{\gamma})$. Then as $t\rightarrow 0$,
one has

(C.1) $\displaystyle
H_{\frac{\gamma}{2}}(t)-H_{\frac{\gamma}{2}}(0)\underset{t>0}{=}C_{2}^{+}t^{1-C}+o(t^{1-C}),\quad
H_{\frac{\gamma}{2}}(t)-H_{\frac{\gamma}{2}}(0)\underset{t<0}{=}C_{2}^{-}t^{1-C}+o(t^{1-C}),$

where:

$\displaystyle C_{2}^{+}$
$\displaystyle=-\frac{\Gamma(-1+\frac{\gamma\beta_{1}}{2}-\frac{\gamma^{2}}{4})\Gamma(1-\frac{\gamma\beta_{1}}{2})}{\Gamma(-\frac{\gamma^{2}}{4})}\left(g_{\frac{\gamma}{2}}(\sigma_{1}-\frac{\gamma}{4})-g_{\frac{\gamma}{2}}(\sigma_{2}+\frac{\beta_{1}}{2}-\frac{\gamma}{4})\right)H\begin{pmatrix}\beta_{1}+\frac{\gamma}{2},\beta_{2},\beta_{3}\\\
\sigma_{1}-\frac{\gamma}{4},\sigma_{2},\sigma_{3}\end{pmatrix},$
$\displaystyle C_{2}^{-}$
$\displaystyle=-\frac{\Gamma(-1+\frac{\gamma\beta_{1}}{2}-\frac{\gamma^{2}}{4})\Gamma(1-\frac{\gamma\beta_{1}}{2})}{\Gamma(-\frac{\gamma^{2}}{4})}\left(g_{\frac{\gamma}{2}}(\sigma_{1})-g_{\frac{\gamma}{2}}(\sigma_{2}+\frac{\beta_{1}}{2})\right)H\begin{pmatrix}\beta_{1}+\frac{\gamma}{2},\beta_{2},\beta_{3}\\\
\sigma_{1}-\frac{\gamma}{4},\sigma_{2},\sigma_{3}\end{pmatrix}.$

A similar statement holds for $\tilde{H}_{\frac{\gamma}{2}}(t)$. Assume this
time that (3.8) hold, and also
$\beta_{1}\in(\frac{\gamma}{2},\frac{2}{\gamma})$. Then as $t\rightarrow 0$,
one has

(C.2)
$\displaystyle\tilde{H}_{\frac{\gamma}{2}}(t)-\tilde{H}_{\frac{\gamma}{2}}(0)\underset{t>0}{=}\tilde{C}_{2}^{+}t^{1-C}+o(t^{1-C}),\quad\tilde{H}_{\frac{\gamma}{2}}(t)-\tilde{H}_{\frac{\gamma}{2}}(0)\underset{t<0}{=}\tilde{C}_{2}^{-}t^{1-C}+o(t^{1-C}),$

where:

(C.3) $\displaystyle\tilde{C}_{2}^{+}$
$\displaystyle=-\frac{\Gamma(-1+\frac{\gamma\beta_{1}}{2}-\frac{\gamma^{2}}{4})\Gamma(1-\frac{\gamma\beta_{1}}{2})}{\Gamma(-\frac{\gamma^{2}}{4})}\left(g_{\frac{\gamma}{2}}(\sigma_{1}+\frac{\gamma}{4})-g_{\frac{\gamma}{2}}(\sigma_{2}-\frac{\beta_{1}}{2}+\frac{\gamma}{4})\right)H\begin{pmatrix}\beta_{1}+\frac{\gamma}{2},\beta_{2},\beta_{3}\\\
\sigma_{1}+\frac{\gamma}{4},\sigma_{2},\sigma_{3}\end{pmatrix},$ (C.4)
$\displaystyle\tilde{C}_{2}^{-}$
$\displaystyle=-\frac{\Gamma(-1+\frac{\gamma\beta_{1}}{2}-\frac{\gamma^{2}}{4})\Gamma(1-\frac{\gamma\beta_{1}}{2})}{\Gamma(-\frac{\gamma^{2}}{4})}\left(g_{\frac{\gamma}{2}}(\sigma_{1})-g_{\frac{\gamma}{2}}(\sigma_{2}-\frac{\beta_{1}}{2})\right)H\begin{pmatrix}\beta_{1}+\frac{\gamma}{2},\beta_{2},\beta_{3}\\\
\sigma_{1}+\frac{\gamma}{4},\sigma_{2},\sigma_{3}\end{pmatrix}.$

In a similar fashion one can perform expansions as $t\rightarrow 1$, we give
the following one, under (3.8) and
$\beta_{2}\in(\frac{\gamma}{2},\frac{2}{\gamma})$, one has
$\tilde{H}_{\frac{\gamma}{2}}(t)-\tilde{H}_{\frac{\gamma}{2}}(1)\underset{t<1}{=}\tilde{B}_{2}^{-}(1-t)^{C-A-B}+o((1-t)^{C-A-B})$
where:

(C.5)
$\tilde{B}_{2}^{-}=-\frac{\Gamma(-1+\frac{\gamma\beta_{2}}{2}-\frac{\gamma^{2}}{4})\Gamma(1-\frac{\gamma\beta_{2}}{2})}{\Gamma(-\frac{\gamma^{2}}{4})}\left(g_{\frac{\gamma}{2}}(\sigma_{3})-g_{\frac{\gamma}{2}}(\sigma_{2}+\frac{\beta_{2}}{2})\right)H\begin{pmatrix}\beta_{1},\beta_{2}+\chi,\beta_{3}\\\
\sigma_{1}+\frac{\gamma}{4},\sigma_{2}+\frac{\gamma}{4},\sigma_{3}\end{pmatrix}.$

###### Proof.

This proof follows exactly the steps of [RZ22, Equation (3.15)]. When
computing the asymptotic of the difference
$H_{\frac{\gamma}{2}}(t)-H_{\frac{\gamma}{2}}(0)$, even if in our case there
is an area GMC term in the functional, there will be no contribution at order
$t^{1-C}$ from this area GMC term. We will only write out the proof for the
first asymptotic as the others follow from a similar argument. Now starting
from the expression (3.13) of $H_{\frac{\gamma}{2}}(t)$ one can write:

$\displaystyle H_{\frac{\gamma}{2}}(t)-H_{\frac{\gamma}{2}}(0)$
$\displaystyle=-\int_{\mathbb{R}}dc\,e^{(\frac{\gamma
p}{2}-\frac{\gamma}{4})c}\mathbb{E}\Bigg{[}e^{\frac{\gamma
c}{2}}\int_{\mathbb{R}}\frac{(|r_{1}-t|^{\frac{\gamma^{2}}{4}}-|r_{1}|^{\frac{\gamma^{2}}{4}})\hat{g}(r_{1})^{\frac{\gamma^{2}q}{8}}}{|r_{1}|^{\frac{\gamma\beta_{1}}{2}}|r_{1}-1|^{\frac{\gamma\beta_{2}}{2}}}e^{\frac{\gamma}{2}h(r_{1})}d\mu_{\partial}^{t}(r_{1})$
$\displaystyle\times\exp\Bigg{(}-e^{\gamma
c}\int_{\mathbb{H}}\frac{|x|^{\frac{\gamma^{2}}{2}}\hat{g}(x)^{\frac{\gamma^{2}}{4}(q+1)}}{|x|^{\gamma\beta_{1}}|x-1|^{\gamma\beta_{2}}}e^{\gamma
h(x)}d^{2}x-e^{\frac{\gamma
c}{2}}\int_{\mathbb{R}}\frac{|r|^{\frac{\gamma^{2}}{4}}\hat{g}(r)^{\frac{\gamma^{2}q}{8}}}{|r|^{\frac{\gamma\beta_{1}}{2}}|r-1|^{\frac{\gamma\beta_{2}}{2}}}e^{\frac{\gamma}{2}h(r)}d\mu_{\partial}^{t}(r)\Bigg{)}\Bigg{]}+o(t^{1-C})$
$\displaystyle=-t^{1-C}\sqrt{\frac{1}{\sin(\pi\frac{\gamma^{2}}{4})}}\Bigg{(}\cos(\pi\gamma(\sigma_{1}-\frac{\gamma}{4}-\frac{Q}{2}))\int_{\mathbb{R}_{+}}du\frac{(1+u)^{\frac{\gamma^{2}}{4}}-u^{\frac{\gamma^{2}}{4}}}{u^{\frac{\gamma\beta_{1}}{2}}}$
$\displaystyle\quad+\cos(\sigma_{2}-\frac{Q}{2}-\frac{\gamma}{4})\int_{0}^{1}du\frac{|u-1|^{\frac{\gamma^{2}}{4}}-|u|^{\frac{\gamma^{2}}{4}}}{|u|^{\frac{\gamma\beta_{1}}{2}}}+\cos(\sigma_{2}-\frac{Q}{2})\int_{1}^{+\infty}du\frac{|u-1|^{\frac{\gamma^{2}}{4}}-|u|^{\frac{\gamma^{2}}{4}}}{|u|^{\frac{\gamma\beta_{1}}{2}}}\Bigg{)}$
$\displaystyle\quad\times
H\begin{pmatrix}\beta_{1}+\frac{\gamma}{2},\beta_{2},\beta_{3}\\\
\sigma_{1}-\frac{\gamma}{4},\sigma_{2},\sigma_{3}\end{pmatrix}+o(t^{1-C}).$

As in [RZ22], in the last equality above we have used the change of variable
$r_{1}=ut$ and applied the Girsanov theorem to the boundary GMC measure
$e^{\frac{\gamma}{2}h(r_{1})}d\mu_{\partial}^{t}(r_{1})$. To get the final
answer we can simply use the following integral identity,

$\displaystyle\cos(\sigma_{2}-\frac{Q}{2}-\frac{\gamma}{4})\int_{0}^{1}du\frac{|u-1|^{\frac{\gamma^{2}}{4}}-|u|^{\frac{\gamma^{2}}{4}}}{|u|^{\frac{\gamma\beta_{1}}{2}}}+\cos(\sigma_{2}-\frac{Q}{2})\int_{1}^{+\infty}du\frac{|u-1|^{\frac{\gamma^{2}}{4}}-|u|^{\frac{\gamma^{2}}{4}}}{|u|^{\frac{\gamma\beta_{1}}{2}}}$
$\displaystyle=-\frac{1}{2}e^{\mathbf{i}\pi\gamma(\sigma_{2}-\frac{Q}{2}-\frac{\gamma}{4}+\frac{\beta_{1}}{2})}\int_{\mathbb{R}_{+}e^{\mathbf{i}\pi}}du\frac{(1+u)^{\frac{\gamma^{2}}{4}}-u^{\frac{\gamma^{2}}{4}}}{u^{\frac{\gamma\beta_{1}}{2}}}-\frac{1}{2}e^{-\mathbf{i}\pi\gamma(\sigma_{2}-\frac{Q}{2}-\frac{\gamma}{4}+\frac{\beta_{1}}{2})}\int_{\mathbb{R}_{+}e^{-\mathbf{i}\pi}}du\frac{(1+u)^{\frac{\gamma^{2}}{4}}-u^{\frac{\gamma^{2}}{4}}}{u^{\frac{\gamma\beta_{1}}{2}}}$
$\displaystyle=-\frac{\Gamma(-1+\frac{\gamma\beta_{1}}{2}-\frac{\gamma^{2}}{4})\Gamma(1-\frac{\gamma\beta_{1}}{2})}{\Gamma(-\frac{\gamma^{2}}{4})}\cos(\sigma_{2}-\frac{Q}{2}-\frac{\gamma}{4}+\frac{\beta_{1}}{2}),$

where in the second line $\mathbb{R}_{+}e^{\mathbf{i}\pi}$ (resp.
$\mathbb{R}_{+}e^{-\mathbf{i}\pi}$) should be understood as a $(-\infty,0)$
contour passing above (resp. below) $-1$. The computations for other cases are
similar. ∎

Let us now turn to the OPE with reflection. The following result holds for
both values of $\chi$.

######  Lemma C.2.

(OPE with reflection) For $\chi=\frac{\gamma}{2}$ or $\frac{2}{\gamma}$,
assume again the constraints of (3.7) are satisfied. Then there exists a
parameter $\beta_{0}>0$ small enough so that under the assumption that
$\beta_{1}\in(Q-\beta_{0},Q)$, as $t\rightarrow 0$ the following asymptotic
holds

(C.6) $\displaystyle
H_{\chi}(t)-H_{\chi}(0)\underset{t>0}{=}C_{2}^{+}t^{1-\chi\beta_{1}+\chi^{2}}+o(|t|^{1-\chi\beta_{1}+\chi^{2}}),\>\>H_{\chi}(t)-H_{\chi}(0)\underset{t<0}{=}C_{2}^{-}t^{1-\chi\beta_{1}+\chi^{2}}+o(|t|^{1-\chi\beta_{1}+\chi^{2}}),$

where:

(C.7) $\displaystyle
C_{2}^{+}=R(\beta_{1},\sigma_{1}-\frac{\chi}{2},\sigma_{2}-\frac{\chi}{2})H\begin{pmatrix}2Q-\beta_{1}-\chi,\beta_{2},\beta_{3}\\\
\sigma_{1}-\frac{\chi}{2},\sigma_{2},\sigma_{3}\end{pmatrix},$ (C.8)
$\displaystyle
C_{2}^{-}=e^{-\mathbf{i}\pi(1-\chi\beta_{1}+\chi^{2})}R(\beta_{1},\sigma_{1},\sigma_{2})H\begin{pmatrix}2Q-\beta_{1}-\chi,\beta_{2},\beta_{3}\\\
\sigma_{1}-\frac{\chi}{2},\sigma_{2},\sigma_{3}\end{pmatrix}.$

Similarly for $\tilde{H}_{\chi}$, still for $\beta_{1}\in(Q-\beta_{0},Q)$ and
assuming this time (3.8) holds, the following asymptotic holds

(C.9)
$\displaystyle\tilde{H}_{\chi}(t)-\tilde{H}_{\chi}(0)\underset{t>0}{=}\tilde{C}_{2}^{+}t^{1-\chi\beta_{1}+\chi^{2}}+o(|t|^{1-\chi\beta_{1}+\chi^{2}}),\>\>\tilde{H}_{\chi}(t)-\tilde{H}_{\chi}(0)\underset{t<0}{=}\tilde{C}_{2}^{-}t^{1-\chi\beta_{1}+\chi^{2}}+o(|t|^{1-\chi\beta_{1}+\chi^{2}}),$

with this time:

(C.10)
$\displaystyle\tilde{C}_{2}^{+}=R(\beta_{1},\sigma_{1}+\frac{\chi}{2},\sigma_{2}+\frac{\chi}{2})H\begin{pmatrix}2Q-\beta_{1}+\chi,\beta_{2},\beta_{3}\\\
\sigma_{1}+\frac{\chi}{2},\sigma_{2},\sigma_{3}\end{pmatrix},$ (C.11)
$\displaystyle\tilde{C}_{2}^{-}=e^{-\mathbf{i}\pi(1-\chi\beta_{1}+\chi^{2})}R(\beta_{1},\sigma_{1},\sigma_{2})H\begin{pmatrix}2Q-\beta_{1}+\chi,\beta_{2},\beta_{3}\\\
\sigma_{1}+\frac{\chi}{2},\sigma_{2},\sigma_{3}\end{pmatrix}.$

###### Proof.

We will only give the proof for $\chi=\frac{2}{\gamma}$, as the case of
$\chi=\frac{\gamma}{2}$ can also be deduced similarly by adapting the proof
strategy of [RZ22]. Throughout this proof we will use the notation
$q=\frac{1}{\gamma}(2Q-\beta_{1}-\beta_{2}-\beta_{3}+\frac{2}{\gamma})$. For a
Borel set $I\subseteq[0,\infty)$, consider the symmetric interval
$\hat{I}=I\cup-I$ and the domain of the upper-half plane
$D_{I}=\\{e^{\mathbf{i}\theta}t\>|\>t\in I,\>\theta\in[0,\pi]\\}$. We
introduce the notation:

(C.12) $K_{I}(t):=e^{\gamma
c}\int_{D_{I}}\frac{|x-t|^{2}\hat{g}(x)^{\frac{\gamma^{2}}{4}(q+1)}}{|x|^{\gamma\beta_{1}}|x-1|^{\gamma\beta_{2}}}e^{\gamma
h(x)}d^{2}x+e^{\frac{\gamma
c}{2}}\int_{\hat{I}}\frac{|r-t|\hat{g}(r)^{\frac{\gamma^{2}q}{8}}}{|r|^{\frac{\gamma\beta_{1}}{2}}|r-1|^{\frac{\gamma\beta_{2}}{2}}}e^{\frac{\gamma}{2}h(r)}d\mu^{t}_{\partial}(r).$

Now we want to study the asymptotic of

(C.13)
$\int_{\mathbb{R}}e^{-\frac{\gamma}{2}qc}\mathbb{E}[\exp(-K_{\mathbb{R}_{+}}(t))]dc-\int_{\mathbb{R}}e^{-\frac{\gamma}{2}qc}\mathbb{E}[\exp(-K_{\mathbb{R}_{+}}(0))]dc=:T_{1}+T_{2},$

where we defined:

$\displaystyle
T_{1}:=\int_{\mathbb{R}}e^{-\frac{\gamma}{2}qc}\mathbb{E}\left[\exp(-K_{(|t|,+\infty)}(t))-\exp(-K_{\mathbb{R}_{+}}(0))\right]dc,$
$\displaystyle
T_{2}:=\int_{\mathbb{R}}e^{-\frac{\gamma}{2}qc}\mathbb{E}\left[\exp(-K_{\mathbb{R}_{+}}(t))-\exp(-K_{(|t|,+\infty)}(t))\right]dc.$

Following the same inequalities as [RZ22, Equations (5.21) - (5.24)] one
obtains
$T_{1}=o(|t|^{1-\frac{2\beta_{1}}{\gamma}+\frac{4}{\gamma^{2}}})=o(|t|^{\frac{2}{\gamma}(Q-\beta_{1})})$.
Now we focus on $T_{2}$. The goal is to restrict $K_{I}$ to the choice
$I=(0,|t|^{1+\eta})\cup(|t|,\infty)$, with $\eta>0$ a small positive constant
to be fixed, and then the area and boundary GMCs on the three disjoint parts
will be weakly correlated. By using the inequalities of [RZ22, Equation
(5.26)] and taking $\eta$ satisfying the condition

(C.14)
$\eta<\frac{1+(\frac{2}{\gamma}-\frac{\gamma}{2})\beta_{1}-\frac{4}{\gamma^{2}}}{\frac{\gamma\beta_{1}}{2}-1},$

we have
$\int_{\mathbb{R}}e^{-\frac{\gamma}{2}qc}\;\mathbb{E}\left[\exp(-K_{\mathbb{R}_{+}}(t))-\exp(-K_{(0,|t|^{1+\eta})\cup(|t|,\infty)}(t))\right]dc=o(|t|^{\frac{2}{\gamma}(Q-\beta_{1})}).$
It remains to evaluate
$\int_{\mathbb{R}}e^{-\frac{\gamma}{2}qc}\mathbb{E}\left[\exp(-K_{(0,|t|^{1+\eta})\cup(|t|,\infty)}(t))-\exp(-K_{(|t|,+\infty)}(t))\right]dc$.
We now introduce the radial decomposition of the field $h$,
$h(x)=B_{-2\log|x|}+Y(x)$, where $B$, $Y$ are independent Gaussian processes
with $(B_{s})_{s\in\mathbb{R}}$ a Brownian motion starting from $0$ for $s\geq
0$, $B_{s}=0$ when $s<0$, and $Y$ is a centered Gaussian process with
covariance,

(C.15)
$\mathbb{E}[Y(x)Y(y)]=\begin{cases}2\log\frac{|x|\lor|y|}{|x-y|},\quad&|x|,|y|\leq
1,\\\
2\log\frac{1}{|x-y|}-\frac{1}{2}\log\hat{g}(x)-\frac{1}{2}\log\hat{g}(y),\quad&\text{else.}\end{cases}$

Now with this decomposition one can write:

(C.16) $\displaystyle K_{I}(t)$ $\displaystyle=e^{\gamma
c}\int_{D_{I}}\frac{|x-t|^{2}\hat{g}(x)^{\frac{\gamma^{2}}{4}(q+1)}e^{\gamma
B_{-2\log|x|}}}{|x|^{\gamma\beta_{1}}|x-1|^{\gamma\beta_{2}}}e^{\gamma
Y(x)}d^{2}x+e^{\frac{\gamma
c}{2}}\int_{\hat{I}}\frac{|r-t|\hat{g}(r)^{\frac{\gamma^{2}q}{8}}e^{\frac{\gamma}{2}B_{-2\log|r|}}}{|r|^{\frac{\gamma\beta_{1}}{2}}|r-1|^{\frac{\gamma\beta_{2}}{2}}}e^{\frac{\gamma}{2}Y(r)}d\mu_{\partial}^{t}(r).$

Define the processes $P(x)=Y(x)\mathbf{1}_{|x|\leq
t^{1+\eta}}+Y(x)\mathbf{1}_{|x|\geq t}$,
$\tilde{P}(x)=\tilde{Y}(x)\mathbf{1}_{|x|\leq
t^{1+\eta}}+Y(x)\mathbf{1}_{|x|\geq t}$ where $\tilde{Y}$ is an independent
copy of $Y$. By using the inequalities as in [RZ22, Equation (5.39)]

(C.17)
$\displaystyle\left|\int_{\mathbb{R}}e^{-\frac{\gamma}{2}qc}\mathbb{E}\left[\exp(-K_{(0,|t|^{1+\eta})\cup(|t|,\infty)}(t))-\exp(-\tilde{K}_{(0,|t|^{1+\eta})}(t)-K_{(|t|,+\infty)}(t))\right]dc\right|\leq
c_{3}\,t^{\eta},$

for some constant $c_{3}>0$, and where in $\tilde{K}_{(0,|t|^{1+\eta})}(t)$ we
simply use the field $\tilde{Y}$ instead of $Y$. When
$\eta>\frac{2}{\gamma}(Q-\beta_{1})$, we can bound the previous term by
$o(|t|^{\frac{2}{\gamma}(Q-\beta_{1})})$. Consider now the change of variable
$x=t^{1+\eta}e^{-s/2}$. By the Markov property of the Brownian motion and
stationarity of

$N_{\tilde{Y}}(dsd\theta):=e^{\gamma\tilde{Y}(e^{-\frac{s}{2}}e^{\mathbf{i}\theta})-\frac{\gamma^{2}}{2}\mathbb{E}[\tilde{Y}(e^{-\frac{s}{2}}e^{\mathbf{i}\theta})^{2}]}dsd\theta,\quad
d\mu_{\tilde{Y}}(s):=\mu_{1}e^{\frac{\gamma}{2}\tilde{Y}(-e^{-s/2})}ds+\mu_{2}e^{\frac{\gamma}{2}\tilde{Y}(e^{-s/2})}ds,$

we can express $\tilde{K}_{(0,|t|^{1+\eta})}(t)$ by the formula

$\displaystyle\tilde{K}_{(0,|t|^{1+\eta})}(t)$ $\displaystyle=e^{\frac{\gamma
c}{2}}\frac{1}{2}t^{1+(1+\eta)(1-\frac{\gamma\beta_{1}}{2}+\frac{\gamma^{2}}{4})}e^{\frac{\gamma}{2}B_{2(1+h)\log(1/t)}}\int_{0}^{\infty}\frac{|1-t^{\eta}e^{-s/2}|}{|t^{1+\eta}e^{-s/2}-1|^{\frac{\gamma\beta_{2}}{2}}}e^{\frac{\gamma}{2}(\tilde{B}_{s}-\frac{s}{2}(Q-\beta_{1}))}d\mu_{\tilde{Y}}(s)$
$\displaystyle+e^{\gamma
c}\frac{1}{2}\sigma_{t}^{2}\int_{0}^{\infty}e^{\gamma(\tilde{B}_{s}-\frac{s}{2}(Q-\beta_{1}))}\int_{0}^{\pi}\frac{|e^{\mathbf{i}\theta}t^{\eta}e^{-\frac{s}{2}}-1|^{2}}{|e^{\mathbf{i}\theta}-e^{-\mathbf{i}\theta}|^{\frac{\gamma^{2}}{2}}|e^{\mathbf{i}\theta}t^{1+\eta}e^{-\frac{s}{2}}-1|^{\gamma\beta_{2}}}N_{\tilde{Y}}(dsd\theta),$

with $\tilde{B}$ an independent Brownian motion and
$\sigma_{t}:=t^{1+(1+\eta)(1-\frac{\gamma\beta_{1}}{2}+\frac{\gamma^{2}}{4})}e^{\frac{\gamma}{2}B_{2(1+\eta)\log(1/t)}}$.
We denote:

$V_{1}:=\frac{1}{2}e^{\frac{\gamma
c}{2}}\int_{0}^{\infty}e^{\frac{\gamma}{2}(\tilde{B}_{s}-\frac{s}{2}(Q-\beta_{1}))}d\mu_{\tilde{Y}}(s),\>\>V_{2}:=\frac{1}{2}e^{\gamma
c}\int_{0}^{\infty}e^{\gamma(\tilde{B}_{s}-\frac{s}{2}(Q-\beta_{1}))}\int_{0}^{\pi}\frac{1}{|e^{\mathbf{i}\theta}-e^{-\mathbf{i}\theta}|^{\frac{\gamma^{2}}{2}}}N_{\tilde{Y}}(dsd\theta).$

One can then prove that

$\displaystyle\left|\int_{\mathbb{R}}e^{-\frac{\gamma}{2}qc}\mathbb{E}\left[\exp(-\tilde{K}_{(0,|t|^{1+\eta})}(t)-K_{(|t|,+\infty)}(t))-\exp(-\sigma_{t}V_{1}-\sigma_{t}^{2}V_{2}-K_{(|t|,+\infty)}(t))\right]dc\right|\leq
c_{4}\,|t|^{(1+\eta)(2-\frac{\gamma\beta_{1}}{2})},$

for some $c_{4}>0$. This upper bound is also a
$o(|t|^{\frac{2}{\gamma}(Q-\beta_{1})})$. Now let $M=\sup_{s\geqslant
0}(\tilde{B}_{s}-\frac{Q-\beta_{1}}{2}s)$ and let $L_{M}$ be the last time
$\left(\mathcal{B}^{\frac{Q-\beta_{1}}{2}}_{-s}\right)_{s\geq 0}$ hits $-M$.
Recall that the law of $M$ is known, for $v\geq 1$,
$\mathbb{P}(e^{\frac{\gamma}{2}M}>v)=v^{-\frac{2}{\gamma}(Q-\beta_{1})}$. For
simplicity, we introduce the notation:

$\displaystyle\rho_{1}(\beta_{1}):=\frac{1}{2}\int_{-\infty}^{\infty}e^{\gamma\mathcal{B}^{\frac{Q-\beta_{1}}{2}}_{s}}\int_{0}^{\pi}\frac{1}{|e^{\mathbf{i}\theta}-e^{-\mathbf{i}\theta}|^{\frac{\gamma^{2}}{2}}}N_{\tilde{Y}}(dsd\theta),\quad\rho_{2}(\beta_{1}):=\frac{1}{2}\int_{-\infty}^{\infty}e^{\frac{\gamma}{2}\mathcal{B}^{\frac{Q-\beta_{1}}{2}}_{s}}\mu_{\tilde{Y}}(ds).$

We can finally show that:

$\displaystyle T_{2}$ $\displaystyle=\int_{\mathbb{R}}dce^{\frac{q\gamma
c}{2}}\mathbb{E}\left[\exp(-K_{(|t|,+\infty)}(t)-\sigma^{2}_{t}e^{\gamma
M+\gamma
c}\rho_{1}(\beta_{1})-\sigma_{t}e^{\frac{\gamma}{2}M+\frac{\gamma}{2}c}\rho_{2}(\beta_{1}))-\exp(-K_{(|t|,+\infty)}(t))\right]$
$\displaystyle+o(|t|^{\frac{2}{\gamma}(Q-\beta_{1})}).$

We now evaluate the above difference at first order explicitly using the fact
that density of $e^{\frac{\gamma}{2}M}$ is known:

$\displaystyle\int_{\mathbb{R}}dce^{\frac{q\gamma
c}{2}}\left(\mathbb{E}[\exp(-K_{(|t|,+\infty)}(t)-\sigma^{2}_{t}e^{\gamma
M+\gamma
c}\rho_{1}(\beta_{1})-\sigma_{t}e^{\frac{\gamma}{2}M+\frac{\gamma}{2}c}\rho_{2}(\beta_{1}))]-\mathbb{E}[\exp(-K_{(|t|,+\infty)}(t))]\right)$
$\displaystyle=\int_{\mathbb{R}}dce^{\frac{q\gamma
c}{2}}\mathbb{E}\left[\int_{1}^{\infty}\frac{dv}{v^{\frac{2}{\gamma}(Q-\beta_{1})+1}}\left(\exp\left(-K_{(|t|,+\infty)}(t)-\sigma^{2}_{t}v^{2}e^{\gamma
c}\rho_{1}(\beta_{1})-\sigma_{t}ve^{\frac{\gamma}{2}c}\rho_{2}(\beta_{1})\right)-\exp(-K_{(|t|,+\infty)}(t))\right)\right]$
$\displaystyle=\frac{\gamma}{2}\int_{\mathbb{R}}dce^{\frac{q\gamma
c}{2}}\mathbb{E}\Bigg{[}\int_{c+\frac{2}{\gamma}\log\sigma_{t}}^{\infty}dw\sigma_{t}^{\frac{2}{\gamma}(Q-\beta_{1})}e^{(c-w)(Q-\beta_{1})}$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\times\Big{(}\exp\left(-K_{(|t|,+\infty)}(t)-e^{\gamma
w}\rho_{1}(\beta_{1})-e^{\frac{\gamma}{2}w}\rho_{2}(\beta_{1})\right)-\exp(-K_{(|t|,+\infty)}(t))\Big{)}\Bigg{]}$
$\displaystyle=\frac{\gamma}{2}\int_{\mathbb{R}}dce^{\frac{q\gamma
c}{2}}e^{c(Q-\beta_{1})}\mathbb{E}\left[\sigma_{t}^{\frac{2}{\gamma}(Q-\beta_{1})}\exp\left(-K_{(|t|,+\infty)}(t)\right)\right]$
$\displaystyle\times\mathbb{E}\left[\int_{\mathbb{R}}dwe^{-w(Q-\beta_{1})}\left(\exp\left(-e^{\gamma
w}\rho_{1}(\beta_{1})-e^{\frac{\gamma}{2}w}\rho_{2}(\beta_{1})\right)-1\right)\right]+o(|t|^{\frac{2}{\gamma}(Q-\beta_{1})}).$

By apply the Girsanov theorem to the power of $\sigma_{t}$ in the first
expectation and taking the limit $t\rightarrow 0$, the first expectation will
become $t^{\frac{2}{\gamma}(Q-\beta_{1})}$ times an $H$ function. The second
expectation is the reflection coefficient $R$. Therefore we arrive at the
final claim for the $t>0$ limit:

(C.18) $\displaystyle\int_{\mathbb{R}}dce^{q\gamma
c/2}\left(\mathbb{E}[\exp(-K_{(|t|,+\infty)}(t)-\sigma^{2}_{t}e^{\gamma
M+\gamma
c}\rho_{1}(\beta_{1})-\sigma_{t}e^{\frac{\gamma}{2}M+\frac{\gamma}{2}c}\rho_{2}(\beta_{1}))]-\mathbb{E}[\exp(-K_{(|t|,+\infty)}(t))]\right)$
$\displaystyle=t^{\frac{2}{\gamma}(Q-\beta_{1})}R(\beta_{1},\sigma_{1}-\frac{1}{\gamma},\sigma_{2}-\frac{1}{\gamma})H\begin{pmatrix}2Q-\beta_{1}-\frac{2}{\gamma},\beta_{2},\beta_{3}\\\
\sigma_{1}-\frac{1}{\gamma},\sigma_{2},\sigma_{3}\end{pmatrix}+o(|t|^{\frac{2}{\gamma}(Q-\beta_{1})}).$

The $t<0$ case is obtained in a similar fashion. ∎

## References

  * [AG21] Morris Ang and Ewain Gwynne.  Liouville quantum gravity surfaces with boundary as matings of trees.  Annales de l’Institut Henri Poincaré, Probabilités et Statistiques, 57(1):1 – 53, 2021.
  * [AHS23a] Morris Ang, Nina Holden, and Xin Sun.  Conformal welding of quantum disks.  Electron. J. Probab., 28:Paper No. 52, 50, 2023.
  * [AHS23b] Morris Ang, Nina Holden, and Xin Sun.  Integrability of SLE via conformal welding of random surfaces.  Communications on Pure and Applied Mathematics, pages 1–57, 2023\.
  * [Ang23] Morris Ang.  Liouville conformal field theory and the quantum zipper.  arXiv e-prints, page arXiv:2301.13200, January 2023.
  * [ARS23] Morris Ang, Guillaume Remy, and Xin Sun.  FZZ formula of boundary Liouville CFT via conformal welding.  Journal of the European Mathematical Society, pages 1–58, 2023\.
  * [ASY22] Morris Ang, Xin Sun, and Pu Yu.  Quantum triangles and imaginary geometry flow lines.  arXiv e-prints, page arXiv:2211.04580, November 2022.
  * [Ber17] Nathanael Berestycki.  An elementary approach to Gaussian multiplicative chaos.  Electronic Communications in Probability, 22(none):1 – 12, 2017\.
  * [BHSY23] Jacopo Borga, Nina Holden, Xin Sun, and Pu Yu.  Baxter permuton and liouville quantum gravity.  Probability Theory and Related Fields, pages 1–49, 2023.
  * [BK01] Bojko Bakalov and Alexander Kirillov, Jr.  Lectures on tensor categories and modular functors, volume 21 of University Lecture Series.  American Mathematical Society, Providence, RI, 2001.
  * [Bor23] Jacopo Borga.  The skew brownian permuton: a new universality class for random constrained permutations.  Proceedings of the London Mathematical Society, 2023.
  * [BP] Nathanaël Berestycki and Ellen Powell.  Introduction to the Gaussian Free Field and Liouville Quantum Gravity.  Available at https://homepage.univie.ac.at/nathanael.berestycki/articles.html.
  * [BPZ84] A.A. Belavin, A.M. Polyakov, and A.B. Zamolodchikov.  Infinite conformal symmetry in two-dimensional quantum field theory.  Nuclear Physics B, 241(2):333–380, 1984.
  * [BW23] Guillaume Baverez and Baojun Wu.  Higher equations of motion at level 2 in Liouville CFT.  arXiv preprint arXiv:2312.13900, 2023.
  * [Cer22] Baptiste Cerclé.  Three-point correlation functions in the $\mathfrak{sl}_{3}$ Toda theory I: Reflection coefficients.  arXiv e-prints, page arXiv:2206.06902, June 2022.
  * [Cer24] Baptiste Cerclé.  Higher equations of motion for boundary Liouville Conformal Field Theory from the Ward identities.  arXiv e-prints, page arXiv:2401.13271, January 2024.
  * [CEZ23] Scott Collier, Lorenz Eberhardt, and Mengyang Zhang.  Solving 3d Gravity with Virasoro TQFT.  arXiv e-prints, page arXiv:2304.13650, April 2023.
  * [DKRV16] François David, Antti Kupiainen, Rémi Rhodes, and Vincent Vargas.  Liouville quantum gravity on the Riemann sphere.  Comm. Math. Phys., 342(3):869–907, 2016.
  * [DLMF] NIST Digital Library of Mathematical Functions.  http://dlmf.nist.gov/, Release 1.1.0 of 2020-12-15.  F. W. J. Olver, A. B. Olde Daalhuis, D. W. Lozier, B. I. Schneider, R. F. Boisvert, C. W. Clark, B. R. Miller, B. V. Saunders, H. S. Cohl, and M. A. McClain, eds.
  * [DMS21] Bertrand Duplantier, Jason Miller, and Scott Sheffield.  Liouville quantum gravity as a mating of trees.  Astérisque, (427):viii+257, 2021.
  * [DO94] Harald Dorn and H. J. Otto.  Two and three point functions in Liouville theory.  Nucl. Phys. B, 429:375–388, 1994.
  * [DRV16] François David, Rémi Rhodes, and Vincent Vargas.  Liouville quantum gravity on complex tori.  Journal of Mathematical Physics, 57(2):022302, 2016.
  * [Dub09] Julien Dubédat.  SLE and the free field: partition functions and couplings.  J. Amer. Math. Soc., 22(4):995–1054, 2009.
  * [Ebe23] Lorenz Eberhardt.  Notes on crossing transformations of virasoro conformal blocks.  2023\.
  * [FZZ00] V. Fateev, A. Zamolodchikov, and Al. Zamolodchikov.  Boundary Liouville Field Theory I. Boundary State and Boundary Two-point Function.  arXiv e-prints, pages hep–th/0001012, January 2000.
  * [GKRV20] Colin Guillarmou, Antti Kupiainen, Rémi Rhodes, and Vincent Vargas.  Conformal bootstrap in Liouville Theory.  arXiv e-prints, page arXiv:2005.11530, May 2020.
  * [GKRV21] Colin Guillarmou, Antti Kupiainen, Rémi Rhodes, and Vincent Vargas.  Segal’s axioms and bootstrap for Liouville Theory.  arXiv e-prints, page arXiv:2112.14859, December 2021.
  * [GRV19] Colin Guillarmou, Rémi Rhodes, and Vincent Vargas.  Polyakov’s formulation of 2d bosonic string theory.  Publications mathématiques de l’IHÉS, 130(1):111–185, 2019\.
  * [Hos01] Kazuo Hosomichi.  Bulk boundary propagator in Liouville theory on a disc.  JHEP, 11:044, 2001.
  * [HRV18] Yichao Huang, Rémi Rhodes, and Vincent Vargas.  Liouville quantum gravity on the unit disk.  Ann. Inst. Henri Poincaré Probab. Stat., 54(3):1694–1730, 2018\.
  * [KRV20] Antti Kupiainen, Rémi Rhodes, and Vincent Vargas.  Integrability of Liouville theory: proof of the DOZZ formula.  Annals of Mathematics, 191(1):81–166, 2020.
  * [MS89] Gregory W. Moore and Nathan Seiberg.  Classical and Quantum Conformal Field Theory.  Commun. Math. Phys., 123:177, 1989.
  * [MS17] Jason Miller and Scott Sheffield.  Imaginary geometry IV: interior rays, whole-plane reversibility, and space-filling trees.  Probab. Theory Related Fields, 169(3-4):729–869, 2017.
  * [NQSZ23] Pierre Nolin, Wei Qian, Xin Sun, and Zijie Zhuang.  Backbone exponent for two-dimensional percolation.  arXiv e-prints, page arXiv:2309.05050, September 2023.
  * [Pol81] A.M. Polyakov.  Quantum geometry of bosonic strings.  Physics Letters B, 103(3):207–210, 1981.
  * [PT99] B. Ponsot and J. Teschner.  Liouville bootstrap via harmonic analysis on a noncompact quantum group.  arXiv e-prints, pages hep–th/9911110, November 1999.
  * [PT01] B. Ponsot and J. Teschner.  Clebsch-Gordan and Racah-Wigner coefficients for a continuous series of representations of $U_{q}(\mathrm{sl}(2,\mathbb{R}))$.  Comm. Math. Phys., 224(3):613–655, 2001.
  * [PT02] B. Ponsot and J. Teschner.  Boundary Liouville field theory: boundary three-point function.  Nuclear Physics B, 622(1):309–327, 2002.
  * [Rem18] Guillaume Remy.  Liouville quantum gravity on the annulus.  J. Math. Phys., 59(8):082303, 2018.
  * [Rem20] Guillaume Remy.  The Fyodorov-Bouchaud formula and Liouville conformal field theory.  Duke Math. J., 169(1):177–211, 2020.
  * [Run99] Ingo Runkel.  Boundary structure constants for the A series Virasoro minimal models.  Nucl. Phys. B, 549:563–578, 1999.
  * [RV14] Rémi Rhodes and Vincent Vargas.  Gaussian multiplicative chaos and applications: A review.  Probab. Surv., 11:315–392, 2014.
  * [RV19] Rémi Rhodes and Vincent Vargas.  The tail expansion of Gaussian multiplicative chaos and the Liouville reflection coefficient.  The Annals of Probability, 47(5):3082 – 3107, 2019.
  * [RZ20] Guillaume Remy and Tunan Zhu.  The distribution of Gaussian multiplicative chaos on the unit interval.  The Annals of Probability, 48(2):872 – 915, 2020.
  * [RZ22] Guillaume Remy and Tunan Zhu.  Integrability of boundary liouville conformal field theory.  Communications in Mathematical Physics, 395(1):179–268, 2022.
  * [She07] Scott Sheffield.  Gaussian free fields for mathematicians.  Probab. Theory Related Fields, 139(3-4):521–541, 2007.
  * [Tes01] J. Teschner.  Liouville theory revisited.  Class. Quant. Grav., 18:R153–R222, 2001.
  * [Tes03] J. Teschner.  From Liouville theory to the quantum geometry of Riemann surfaces.  In 14th International Congress on Mathematical Physics, 8 2003\.
  * [Ver88] Erik P. Verlinde.  Fusion Rules and Modular Transformations in 2D Conformal Field Theory.  Nucl. Phys. B, 300:360–376, 1988.
  * [VT13] G. Vartanov and J. Teschner.  Supersymmetric gauge theories, quantization of moduli spaces of flat connections, and conformal field theory, 2013.
  * [Wu22] Baojun Wu.  Conformal Bootstrap on the Annulus in Liouville CFT.  arXiv e-prints, page arXiv:2203.11830, March 2022.
  * [Wu23] Da Wu.  The SLE Bubble Measures via Conformal Welding of Quantum Surfaces.  arXiv e-prints, page arXiv:2302.13992, February 2023.
  * [Zha22] Dapeng Zhan.  SLE${}_{\kappa}(\rho)$ bubble measures.  arXiv e-prints, page arXiv:2206.04481, June 2022.
  * [ZZ96] Alexander B. Zamolodchikov and Alexei B. Zamolodchikov.  Structure constants and conformal bootstrap in Liouville field theory.  Nucl. Phys. B, 477:577–605, 1996.


# Emotion Recognition from the perspective of Activity Recognition

Savinay Nagendra Prapti Panigrahi
Department of Computer Science
The Pennsylvania State University
University Park
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Automatic emotion recognition from using visual signals is one of the
burgeoning research areas in the domain of Computer Vision. Applications of an
efficient emotion recognition system can be found in several domains such as
medicine, driver fatigue surveillance, social robotics and human-computer
interaction. Appraising human emotional states, behaviors and reactions
displayed in real world settings can be accomplished using latent continuous
dimensions. Continuous dimensional models of human affect, such as those based
on valence and arousal have been shown to be more accurate in describing a
broad range of spontaneous everyday emotions than more traditional models of
discrete stereotypical emotion categories (e.g. happiness, surprise). Most of
the prior work on estimating valence and arousal consider laboratory settings
and acted data. But, for emotion recognition systems to be deployed and
integrated into real-world mobile and computing devices, we need to consider
data collected in the wold. Action recognition is a domain of Computer Vision
that involves capturing complementary information on appearance from still
frames and motion between frames. In this paper, we treat emotion recognition
from the perspective of action recognition by exploring the application of
deep learning architectures specifically designed for action recognition, for
continuous affect recognition. We propose a novel three stream end-to-end deep
learning regression pipeline with attention mechanism, which is an ensemble
design based on sub-modules of multiple state-of-the-art action recognition
systems. The pipeline constitutes a novel data pre-processing approach with
spatial self-attention mechanism to extract key frames. Optical flow of high
attention regions of the face are extracted to capture temporal context. AFEW-
VA in-the-wild dataset has been used to conduct comparative experiments.
Quantitative analysis shows that the proposed model outperforms multiple
standard baselines of both emotion recognition and action recognition models.

##  1 Introduction

The field of Automatic Emotion Recognition analysis has grown rapidly in
recent years, with applications spread across a variety of fields, such as
medicine [59], [60], health monitoring [27], entertainment, lie-detection
[75], [28]. Current research in automatic analysis of facial affect aims at
developing systems, such as robots and virtual humans, that will interact with
humans in a naturalistic way under real-world settings. Such systems should
automatically sense and interpret visual signals relevant to emotions,
appraisals and intentions. There are two major emotion computing models
according to theories in psychology research [34]: discrete and dimensional
theory. When designing emotion recognition systems for real-world settings,
where subjects operate in a diversity of contexts and environments, systems
that perform automatic analysis of human behavior should be robust to the
diversity of contexts, the timing of display and video recording conditions.

There are two major emotion computing models according to theories in
psychology research [34]: discrete and dimensional theory. Research in the
past [43, 41, 13, 31, 50, 42, 39, 44, 40, 45, 76] has revolved around the
recognition of the so-called six universal expressions - Happiness, Sadness,
Fear, Disgust, Surprise and Anger. [11], [25], [29], [10], [26]. These are
intuitive and simple, but cannot express complex affective states. For many
years, research in automatic analysis of facial behavior was mainly limited to
posed behavior which was captured in highly controlled recording conditions
[32, 59, 61, 64]. Some representative datasets, which are still used in many
recent works [19], are Cohn-Kanade database [32, 61], MMI database [59, 64],
Multi-PIE database [28] and the BU-3D and BU-4D databases [74, 70].

However, the fact that the facial expressions of naturalistic behaviors can be
radically different from the posed ones has been widely accepted in the
research community [9, 54, 71]. Hence, efforts have been made in order to
collect subjects displaying naturalistic behavior. Examples include the
recently collected EmoPain [3] and UNBC-McMaster [33] databases for analysis
of pain, the RU-FACS database of subjects participating in a false opinion
scenario [4] and the SEMAINE corpus [35] which contains recordings of subjects
interacting with a Sensitive Artificial Listener (SAL) in controlled
conditions. All the above databases have been captured in well-controlled
recording conditions and mainly under a strictly defined scenario eliciting
pain.

Figure 1: The 2D Emotion wheel or the 2D Valence-Arousal Space

Valence defines how negative or positive the experience is, and intensity of
arousal defines how calming or exciting the experience is. The circumplex
model of emotion shown in Figure 1 represents the 2D Emotion Wheel or the 2D
Valence-Arousal Space. The dimensions represent arousal, indicating the level
of affective activation, and valence, measuring the level of pleasure. Hence,
dimensional theory can model subtle, complicated, and continuous affective
behaviors.

Action recognition task involves the identification of different actions from
video clips (a sequence of 2D frames) where the action may or may not be
performed throughout the entire duration of the video. This seems like a
natural extension of image classification tasks to multiple frames and then
aggregating the predictions from each frame. Despite the stratospheric success
of deep learning architectures in image classification (ImageNet), progress in
architectures for video classification and representation learning has been
slower. Both Emotion and Action recognition algorithms are similar in terms
of:(1) Input: (a)Input for both tasks is a sequence of 2D video frames. (b)
The action/emotion may or may not be performed throughout the entire video.
(c) Multiple actions/emotions possible in the same video. (2) Capturing Long
Context: (a) Have to capture spatio-temporal context across frames. (b)
Spatial information captured has to be compensated for camera movement. (c)
Both local and global context with respect to motion information needs to be
captured for robust predictions. (3) Architectures and Training: (a)One
network for capturing spatio-temporal information. (b) Two separate networks,
for each spatial and temporal. (c) Fusing predictions across multiple frames.
(d) End-to-end training vs feature extraction and classifying separately.

The motivation to explore this problem comes from the fact that the field of
emotion recognition is nascent and not as advanced as the field of action
recognition in the domain of Computer Vision. With this research we seek the
answers to the following questions: (1) With the above mentioned similarities
between the two tasks, is it feasible to explore emotion recognition from the
perspective of Action Recognition? (2) Can the complex deep learning
mechanisms of Action recognition be adopted for the task of emotion
recognition? (3) Can we come up with a general model which can work for both
tasks without invoking specifics of a particular task?

Figure 2: Pipeline of our Emotion Recognition System

In this work, we aim to explore the problem of emotion recognition from the
perspective of action recognition. We use a continuous affect recognition
dataset collected in-the-wild, called AFEW-VA which consists of highly
accurate per-frame annotations of valence and arousal for 600 challenging,
real-world video clips extracted from feature films Added to these are per-
frame annotations of 68 facial landmarks. The dataset has been made publicly
available.

We conduct a quantitative study of multiple standard and state-of-the-art
emotion and action recognition algorithms applied to our dataset. We propose a
novel three stream ensemble deep learning pipeline shown in Figure 2. The
pipeline is designed as a regression system to detect continuous
affect.Illumination equalization is performed across all video clips using
Contrast Limited Adaptive Histogram Equalization (CLAHE). This helps a uniform
illumination over all subjects, hence increasing the performance of the
system. Action recognition research [68] shows that not all video frames are
necessary for video classification. Hence, a spatial self-attention mechanism
is applied to compute the key frames which capture the same spatio-temporal
information as that of the original video frames. The frame-wise Gaussian
heatmaps provided by the attention network is used to identify the most
discriminating regions of the face for efficient emotion recognition. Optical
flow from eye and mouth regions, which get the highest score from attention
maps, are extracted to capture temporal context in the dataset. Three streams
of input, thus consists of RGB video frames, optical flows from eye and mouth.
A Convolutional feature extractor with ResNet-50 backbone is used to extract
features from all the three streams. Further, variable length temporal
Gaussian attention filters are used for capturing temporal attention. The
output from the temporal attention filters are given to Fully Connected layers
to get the final outputs.

The contributions of our work are as follows:

  1. 1.

A novel end-to-end three stream ensemble deep learning regression pipeline for
continuous affect recognition.

  2. 2.

Key frame extraction and identification of the most discriminative features of
the face using spatial self-attention.

  3. 3.

Using optical flow frames of eye and mouth regions to capture temporal
context.

  4. 4.

Incorporation of Temporal Gaussian Attention filters into the three stream
pipeline.

  5. 5.

A quantitative comparative study of multiple standard and state-of-the-art
action and emotion recognition algorithms applied to our chosen dataset .

The rest of the paper is organized as follows: Section 2 mentions a brief
literature survey regarding the evolution of emotion and action recognition
algorithms and the work that has been implemented for this paper. Section 3
provides information about the chosen dataset. Section 4 provides information
about our pipeline. Section 5 provides information about the baseline
algorithms that have been compared with our method and quantitative results of
the comparative study. Section 6 provides a summary and discussion of future
work. Finally, Section 7 provides acknowledgments for our work.

##  2 Related Work

There has been significant work on audio-based valence and arousal estimation
[14]. Our paper focuses on video-based valence and arousal estimation. Both
valence and arousal are defined as continuous emotional dimensions. Therefore,
it seems suitable to study them directly in the continuous domain. Even though
much of the early work considered coarse levels of valence and arousal (e.g.
positive vs negative), and posed the problem as one of classification [9],
more recent work casts the problem in the continuous domain [14, 55, 38].

With the onset of AVEC challenges in 2011, there has been tremendous progress
in the field. Pioneering work was done with a subset of the Semaine dataset
[56], which originally formulated the problem as one of classification, using
binarized values (±1), before moving to continuous annotations in 2012. The
best results obtained that year were an average Pearson Correlation
Coefficient (PCC) of 0.456 [47]. The 2013 and 2014 editions of the challenge
used the audio-visual depression language corpus [65, 63]. The best results
obtained on that corpus were lower than in the first year with a PCC of 0.1409
in 2013 [36] but improved to 0.5946 for the best performer in 2014 [20].
RECOLA dataset was used from 2015 in AVEC challenges. The results were good
even though RECOLA [52] is a challenging dataset. For instance, the best
performer in the 2015 edition of the challenge obtained an average PCC of
0.685 [16], while the best performer in 2016 obtained an average PCC of 0.731
[6].

The most obvious approaches, which can also be treated as non-deep learning
baselines to valence and arousal estimation are static regression, such as
linear regression [15, 62], partial least squares [36], and Support Vector
Machine for Regression (SVR) [46, 53, 30]. In [53], SVR is combined with
Canonical Correlation Analysis (CCA) to iteratively fuse predictions. In [30],
SVR is used as a baseline to test the performance of multiple facial
appearance and geometric features. SVR is also employed in the work of [20]
but its use differs in that template trajectories for each emotional dimension
are first built and then matched to a new testing sample using metadata as
features. Meta-data is also used in [58] where audio, video and contextual
(meta-data) features are combined in a multimodal fuzzy inference system. More
powerful kernel based methods can be used such as Nadaraya-Watson kernel
regression [47] or Doubly Sparse Relevance Vector Machine [22] that can impose
sparsity on both the kernels and the training samples.

In [30], a comparative study has been done on multiple appearance and
geometric features. In appearance features, the paper experiments with Hybrid-
SIFT [1], Block LBP [73], Hybrid LBP, Holistic-DCT [37], Block-DCT and Hybrid-
DCT. Normalized features [24] are explored as geometric features. When dealing
with several features, e.g. geometric and appearance features, or multiple
modalities, e.g. audio and video, the question of how to fuse them arises.
There are four main types of fusions: feature level (early fusion), decision
level (late fusion), model-level and output-associative fusion [72, 48]. In
the early-fusion case, the features are combined most commonly by simply
concatenating them and using the output for estimation(e.g.as is done in [8]
for predicting valence). However, in general, this approach tends to create
very high dimensional feature vectors and lead to overfitting. Late fusion is
the process of first generating separate estimations from each modality before
combining – fusing – them into one final estimation. This fusion can be
achieved in numerous ways, from simple mapping, e.g. averaging, as in [21], to
more complex methods such as linear and multi-linear regression
[15,25,26,28,30,37,38], SVR [30], random forests [7] or Kalman filters based
[6]. This type of late fusion is akin to stacking, a classical ensemble
learning technique.

Figure 3: Data Statistics of our dataset

Deep learning approaches to solve the problem of continuous affect use blocks
of convolutional neural networks as feature extractors with SVM/SVR as
classifiers, or have an end-to-end strategy of training. A combination of
convolutional and recurrent neural networks is also used to capture spatio-
temporal context. The most commonly used metric is CCC (Concordance
Correlation Coefficient) which is a metric to penalize the shift in time
series predictions. Mean Squared Error is also a commonly used metric to
evaluate the performance of continuous affect models. [5] uses a network
called AffWildNet whihc consists of convolutional and pooling layers of either
VGG-Face or ResNet-50 structures, followed by a fully connected layer and two
RNN layers with GRU units. This is used as the deep learning baseline for our
paper, as this paper was one of the most recent studies of continuous affect
using deep learning [43, 41, 13, 31, 50, 42, 39, 44, 40, 45, 76], which was
published in 2019.

Before deep learning came along, most of the traditional Computer Vision
algorithm variants for action recognition can be broken down into the
following 3 broad steps: (1) Local high-dimensional visual features that
describe a region of the video are extracted either densely or at a sparse set
of interest points. (2) The extracted features get combined into a fixed-sized
video level description. One popular variant to the step is to bag of visual
words (derived using hierarchical or k-means clustering) for encoding features
at video-level. (3) A classifier, like SVM or RF, is trained on bag of visual
words for final prediction.

Of these algorithms that use shallow hand-crafted features in (1), improved
Dense Trajectories [67] (iDT) which uses densely sampled trajectory features
was the state-of-the-art. Simultaneously, 3D convolutions were used as is for
action recognition without much help in 2013 [18]. Soon after this in 2014,
two breakthrough research papers were released which form the backbone for all
the papers we are going to discuss in this post. The major differences between
them was the design choice around combining spatio-temporal information.

This [23] is on of the foremost attempts of video classification using deep
learning. The paper explores multiple ways to fuse temporal information from
consecutive frames using 2D pre-trained convolutions. the consecutive frames
of the video are presented as input in all setups. Single frame uses single
architecture that fuses information from all frames at the last stage. Late
fusion uses two nets with shared parameters, spaced 15 frames apart, and also
combines predictions at the end. Early fusion combines in the first layer by
convolving over 10 frames. Slow fusion involves fusing at multiple stages, a
balance between early and late fusion. For final predictions, multiple clips
were sampled from entire video and prediction scores from them were averaged
for final prediction. The slow fusion model, which performed the best
according to the authors of this paper, is used as the first action
recognition baseline in our work. In this work [57], the authors build on the
failures of the previous work by Karpathy et al. Given the toughness of deep
architectures to learn motion features, authors explicitly modeled motion
features in the form of stacked optical flow vectors. So instead of single
network for spatial context, this architecture has two separate networks - one
for spatial context (pre-trained), one for motion context. The input to the
spatial net is a single frame of the video. Authors experimented with the
input to the temporal net and found bi-directional optical flow stacked across
10 successive frames was performing best. The two streams were trained
separately and combined using SVM. The final prediction was the same as the
previous paper, i.e. averaging across sampled frames. This work is used as the
second action recognition baseline in our paper. In this work [12], authors
use the base two-stream architecture with two novel approaches and demonstrate
performance increment without any significant increase in the size of
parameters. The authors explore the efficacy of two major ideas: (1) Fusion of
spatial and temporal streams (how and when) - For a task discriminating
between brushing hair and brushing teeth - a spatial net can capture the
spatial dependency in a video (if it’s hair or teeth) while temporal net can
capture the presence of periodic motion for each spatial location in the
video. Hence it’s important to map spatial feature maps about say a particular
facial region to a temporal feature map for the corresponding region. To
achieve the same, the nets need to be fused at an early level such that
responses at the same pixel position are put in correspondence rather than
fusing at the end (like in base two-stream architecture). (2) Combining
temporal net output across time frames so that long-term dependency is also
modeled. Our architecture is designed using this framework. Finally, [69]
provides strategies for good practices in action recognition: (1) They suggest
sampling clips sparsely across the video to better model long-range temporal
signal instead of random sampling across the entire video. (2) For the final
prediction at the video-level authors explored multiple strategies. The best
strategy was: (a) Combining scores of temporal and spatial streams (and other
streams if other input modalities are involved) separately by averaging across
snippets (b) Fusing scores of final spatial and temporal scores using weighted
average and applying softmax over all classes. These points have been
incorporated while designing our pipeline.

##  3 AFEW-VA: emotion recognition dataset in-the-wild

This section provides information about the dataset that has been used for all
the experiments in this paper.

###  3.1 Data

Figure 4: Screenshot of the annotation tool developed and used to annotate the
AFEW-VA dataset. Figure 5: Example of annotated valence and arousal levels for
a sample video from our dataset along with some representative frames. Figure
6: Examples of tracked landmarks in the dataset

AFEW-VA consists of 600 videos extracted from features films. The videos are
processed at 30fps and range from short (around 10 frames) to longer clips
(more than 120 frames), and display various facial expressions. They are
captured under challenging indoor and outdoor conditions such as complex
cluttered backgrounds, poor illumination, large out-of-plane head rotations,
variations in scale, and occlusions. In total, there are $30000$ frames with
per-frame levels of valence and arousal intensities in the range of $-10$ to
$10$. Figure 3 shows the distribution of the values of valence and arousal
present in the dataset. It matches the expected distribution and, as can be
seen, there is a wide range of values for both valence and arousal. In some
videos, we observe a significant signal change in valence and arousal across
the frames. In some other videos, the temporal change of valence and arousal
is negligible. Figure 3 also shows the distribution of the annotations in the
valence and arousal circle. As can be observed, our data show large variations
in valence and arousal values and complement well existing databases.

###  3.2 Annotations

Per-frame annotations of valence and arousal levels are provided for all
frames within all video clips of our dataset. Annotations are created by two
expert annotators, a male and a female, FACS AU coding certified. Both
annotators have annotated all videos together, therefore discussing all
disagreements and coming up with a unique solution. Hence, AFEW-VA annotations
are detailed and highly accurate. The range of annotation levels for both
valence and arousal is from -10 to 10, resulting in a total of 21 levels.
Figure 4 shows a screenshot of the annotation tool used by the experts. Figure
5 shows the temporal evolution of the valence and arousal signals for an
example video from our dataset, along with some representative frames. The
dataset also provides accurate location for 68 landmarks, as shown in Figure
6, including both interior and boundary points of the face. The facial
annotations were generated in a semi-automatic way. First, the face was
detected in the first frame of each video sequence using the tree-based
deformable part model (DPM).

##  4 Our Approach

We propose a novel three stream ensemble deep learning pipeline for emotion
recognition using concepts of action recognition mechanisms. In this section,
the proposed pipeline shown in Figure 2 will be discussed.

###  4.1 Key Frame Sub-sampling

Figure 7: The CNN takes the video frame as its input and produces local
features. Using the local features, the multi-head attention network computes
the weight importance of each local image feature. The aggregated
representation is computed by multiplying multi-head attention output and the
local image features. This representation is then propagated through temporal
softmax pooling to extract global features over the entire video. Figure 8:
Steps of Key Frame subsampling. Row 1: Raw face extracted images. Row2: Facial
Landmarks for faces. Row3: Illumination Equalization using CLAHE Row4: Sub-
sampled Key-Frames and their 2D Gaussian attention heatmaps showing most
important facial regions for efficient emotion recognition.

Each video clip has variable number of video frames, which makes it difficult
to design a deep learning architecture. As described in Section 2, [69] tells
us that we do not need all the frames of every video clip to get a high
performing architecture. Instead of randomly sub-sampling video frames, we
incorporate a spatial self-attention mechanism to sample equal number of key
discriminative frames from each video clip. Every video clip, as shown in
Figure 6 shows varying illumination and distance from camera perspective of
subjects in every video clip. This poses a problem to capture spatio-temporal
context for facial expression recognition efficiently. The reasons for not
exploring body language are: (1) There was no publicly available in-the-wild
continuous affect dataset to perform experiments. (2) There were not enough
deep learning baselines for a comparative study. (3) Emotions are highly
correlated with the face.

Using the 68 facial landmarks that were provided, faces are extracted from
every video clip by drawing a bounding box covering all the landmarks.
Illumination of every video frame is equalized by using Contrast Limited
Adaptive Histogram Equalization (CLAHE) [77]. Adaptive histogram equalization
(AHE) is a computer image processing technique used to improve contrast in
images. It differs from ordinary histogram equalization in the respect that
the adaptive method computes several histograms, each corresponding to a
distinct section of the image, and uses them to redistribute the lightness
values of the image. It is therefore suitable for improving the local contrast
and enhancing the definitions of edges in each region of an image. However,
AHE has a tendency to over-amplify noise in relatively homogeneous regions of
an image. A variant of adaptive histogram equalization called contrast limited
adaptive histogram equalization (CLAHE) prevents this by limiting the
amplification.

####  4.1.1 Local Feature Extraction

The idea of key frame sub-sampling is taken from the paper [2]. We have frame-
by-frame annotation of our dataset. This enabled us to apply this approach to
capture the most important frames for regression. Consider a video sample
$\mathbf{S_{i}}$ and its associated emotion $y_{i}\in R^{E}$, where we
represent the video as a sequence of $F$ frames $[X_{0,i},...X_{F,i}]$, each
of size $W\times H\times 3$. VGG-16, pre-trained with VGG-Face model [49] was
used for extracting an independent description of a face on each frame in the
video. For a given frame $X$ of a video, we consider the feature map produced
by the last convolutional layer of the network as representation. This feature
map has spatial resolution of $L=H/16\times W/16$ and $D$ channels.
channels.We discard the spatial resolution and reshape the feature map as a
matrix $R$ composed of $L$ $D$-dimensional local descriptors (row vectors).

$\mathbf{R}=VGG_{16}(X)$ (1)

These descriptors will be associated to a corresponding weight and used for
the attention mechanism.

####  4.1.2 Spatial Attention

For the spatial attention we rely on the self-attention mechanism [66], which
aggregates a set of local frame descriptors $R$ into a single weighted sum $v$
that summarizes the most important regions of a given video frame:

$v=a\mathbf{R}$ (2)

where $a$ is a row vector of dimension $L$, which defines the importance of
each frame region. The weights $a$ are generated by two layers fully connected
network that associates each local feature (row of $R$) to a corresponding
weight. This vector representation usually focuses on a specific region in the
facial feature, like the mouth. However, it is possible that multiple regions
of the face contain different type of information that can be combined to
obtain a better idea of the person emotional state. Hence, we need multiple
attention units that focus on different parts of the image. For doing that, we
transform $w_{s2}$ into a matrix $W_{s2}$ of size $R\times L$, in which every
row represents a different attention.

####  4.1.3 Temporal Pooling

After extracting the local features and aggregating them using the attention
mechanism for each frame, we have to take into account frame features over the
whole video. As the length of a video can be different for each example, we
need an approach that supports different input lengths. The most commonly used
approaches are average and max pooling; however, these techniques assume that
every frame of the video has the same importance in the final decision
(average pooling) or that only a single frame is considered as a general
representation of the video (max pooling). In order to use the best of both
techniques, we use an aggregation based on softmax, which can be considered a
generalization of the average and max pooling.

Given a video sample S, after feature extraction and spatial attention, we
obtain a matrix $V$ in which each row represents the features of a frame. The
21 labels of valence and arousal are discretized into 7 classes based on the
2D continuous emotion wheel for this process. These features are converted
into class scores through a final fully connected layer
$\mathbf{O}=\mathbf{W_{sm}V}$. Here, $\mathbf{O}$ is a matrix in which an
element $o_{c,f}$ is the score for class $c$ of the frame $f$. We then
transform the scores over frames and classes in probabilities with a softmax.

$p(c,f|S)=\frac{exp(o_{c,f})}{\sum_{j,k}exp(o_{j,k})}$ (3)

In this way, we obtain a joint probability on class $c$ and frame $f$. This
representation can be marginalized over classes $p(f|S)=\sum_{c}p(c,f|S)$.
This will give us information about the most important frames of a given
video. The activations of each frame are converted to 2D Gaussian heatmaps,
which will give us a qualitative evaluation of the most important regions of
the face as shown in Figure 8. The figure shows all the steps of key frame
sub-sampling. Using this method, we sample 10 key frames for each video clip.
10 key frames with highest classification scores are chosen.

###  4.2 Optical Flow

Figure 9: Train-Test Distributions of Arousal and Valence

Key Frame sub-sampling shows us that eyes and mouth are the most important
regions of the face, from which discriminative features can be extracted.
Hence, as shown in Figure 2, we use the extracted key frames and facial
landmarks to automatically draw bounding boxes over the mouth and eyes of
every frame. This way, the mouth and eyes are extracted from every key frame.
Now, we have RGB images video frames and RGB frames of mouth and eyes.

Optical flow [17] is the pattern of apparent motion of image objects between
two consecutive frames caused by the movemement of object or camera. It is 2D
vector field where each vector is a displacement vector showing the movement
of points from first frame to second. Optical flow is explicitly used in
Action Recognition algorithms to extract motion information. To incorporate
action recognition mechanisms and to effectively capture temporal context, we
extract optical frame information between every two consecutive frames. After
this step, we have our input data for any action recognition mechanism that is
rich in spatial and temporal context.

###  4.3 Temporal Gaussian Attention Filters

Figure 10: An illustration of our temporal attention filter. The filter is
differentiable and represented with three parameters.

Many high-level activities are often composed of multiple temporal parts
(e.g., sub-events) with different duration/speed, and our objective is to make
the model explicitly learn such temporal structure using multiple attention
filters and benefit from them. The idea of temporal filters is taken from
[51]. The filters are designed to be fully differentiable, allowing end-to-end
training. We can learn a set of optimal static temporal attention filters to
be shared across different videos. Hence, in our work, we incorporate this
idea to capture global temporal attention between video clips.

The temporal attention filters are applied after we do convolutional feature
extraction from segments of each video clip, as shown in Figure 2. Each
temporal filter learns three parameters: a center $g$, a stride $\delta$ and a
width $\sigma$. These parameters determine where the filter is placed and the
size of the interval focused on. A filter consists of $N$ Gaussian filters
separated by a astride of $\delta$ frames. The goal of this model is to learn
where in the video the most useful features appear.

Figure 11: Table of quantitative results

###  4.4 Network Architecture and Training

We propose a three stream convolutional feed-forward architecture for feature
extraction. The inputs to the network are key facial RGB frames, optical flow
frames of eyes and optical flow frames of mouth. All the inputs are of shape
96 X 96 X 3. Every block of the architecture consists of Conv - Batch
Normalization - Max Pooling layers. The network has one spatial stream which
takes RGB images as input and two temporal streams which take optical flow as
inputs. The architecture is an encoder with decreasing resolution and
increasing number of channels with depth. Dropout regularization is used to
avoid over-fitting. Out of a total of 600 video clips, 500 are used for
training and 100 are for test. The train and test video clips are chosen such
that they have identical independent frame-wise valence-arousal distributions
as shown in Figure 9. The features from the three streams are fused using
average pooling after after 5 blocks. The fused features are given to temporal
attention filters, after which there are two fully connected blocks. 21 levels
of Arousal and Valence [-10, 10] are converted to values between 0,1 by using
min-max normalization. All layers use ReLU activation, while the last layer
uses sigmoid activation. 1 - (Average of Concordance Correlation Coefficient
(CCC) or Arousal and Valence) is used as the Loss function. The network is
trained using Nvidia RTX 2080 Ti GPU for 200 epochs with a batch size of 32.
We use a Learning Rate Scheduler with an initial learning rate of 5e-5.

##  5 Results

In this section we are going to discuss quantitative results and the baselines
used. Table 11 shows the quantitative scores of CCC and MSE. It can be
observed that Our Approach performs the best on afew-va dataset over all other
emotion and action recognition models. This is because our approach is an
ensemble approach that includes sub-modules of multiple action recognition
algorithms, such as optical flow streams, parallel feature extraction and
temporal attention filters. On top of this, key frame extraction using spatial
self-attention ensures the accuracy with discriminative frames with a low
processing overhead.

###  5.1 Baselines

We use Support Vector Machine for Regression (SVR) as our baseline. Hybrid LBP
features are extracted from Facial video frames. LBP is a visual descriptor
used for texture classification. We use a simple linear transformation model
to scale and translate all the images onto a common coordinate frame. We
extract 27X27 patches around each of the 68 landmarks. Each patch is
represented by a 59-D histogram. All histograms are concatenated to form a
global descriptor of size 4012. SVR is trained on these features to establish
a baseline. The CNN-RNN model called AffWildNet, mentioned in Section 2 is
used as our deep learning baseline for the task of emotion recognition.
Further, all the action recognition papers mentioned in Section 2 are
implemented on our dataset for comparative analysis.

###  5.2 Evaluation Metrics

Two evaluation metrics are used in this paper for quantitative comparative
analysis:

####  5.2.1 Concordance Correlation Coefficient (CCC)

$\rho_{c}=\frac{2s_{xy}}{s_{x}^{2}+s_{y}^{2}+(\mu_{x}-\mu_{y})^{2}}$ (4)

, where $s_{x}$ and $s_{y}$ are standard deviations and $\mu_{x}$ and
$\mu_{y}$ are corresponding means. This is a widely used metric to measure the
performance of dimensional emotion recognition methods. CCC evaluates the
agreement between two time-series data, in our case, video annotations and
predictions. CCC is calculated by scaling the mean square difference between
ground truth and predictions. Predictions that are well correlated with
annotations, but shifted in value are penalized in proportion to the
deviation. CCC takes the values in the range [-1,1]. +1 indicates perfect
concordance, while -1 indicates perfect discordance. The loss function is
given by

$Loss=1-\frac{\rho_{a}+\rho_{v}}{2}$ (5)

####  5.2.2 Mean Squared Error

$MSE=\frac{1}{N}\sum_{i=1}^{N}(x_{i}-y_{i})^{2}$ (6)

This is a common evaluation metric used for regression problems.

##  6 Conclusion and Future Work

This paper presents a novel three-stream action recognition-based emotion
recognition pipeline with spatial key frame extraction mechanism and temporal
Gaussian filters. This is an ensemble approach that combines multiple sub-
modules of standard and state-of-the-art action recognition algorithms to get
state-of-the-art accuracy on the continuous, in-the-wild affect dataset AFEW-
VA. Further, our work is a proof of concept that the action recognition
mechanisms can be extrapolated to the task of continuous emotion recognition.
To the best of our knowledge, this is the first multi-stream pipeline with
spatial self-attention, optical flow inputs of facial regions and temporal
Gaussian Attention filters to be designed to solve the problem of continuous
affect recognition. Further, the paper provides a detailed comparative study
to compare multiple standard and state-of-the-art emotion and action
recognition algorithms.

For future work, the whole body could be considered for expression recognition
by extracting bodily features. This would be a more challenging problem. The
model could be fine-tuned further for better accuracy. Adversarial training
and Semi-supervised training can be used for further improvement in accuracy.

##  7 Acknowledgements

We would like to thank Dr. James Wang for all his support. The course was a
great learning opportunity and a stepping stone to the interesting domain of
emotion recognition. We would like to thank I-Bug for the dataset.

## References

  * [1] Mundher Al-Shabi, Wooi Ping Cheah, and Tee Connie.  Facial expression recognition using a hybrid cnn-sift aggregator.  arXiv preprint arXiv:1608.02833, 2016.
  * [2] Masih Aminbeidokhti, Marco Pedersoli, Patrick Cardinal, and Eric Granger.  Emotion recognition with spatial attention and temporal softmax pooling.  Image Analysis and Recognition, page 323–331, 2019.
  * [3] Min SH Aung, Sebastian Kaltwang, Bernardino Romera-Paredes, Brais Martinez, Aneesha Singh, Matteo Cella, Michel Valstar, Hongying Meng, Andrew Kemp, Moshen Shafizadeh, et al.  The automatic detection of chronic pain-related expression: requirements, challenges and the multimodal emopain dataset.  IEEE transactions on affective computing, 7(4):435–451, 2015.
  * [4] Marian Stewart Bartlett, Gwen Littlewort, Mark Frank, Claudia Lainscsek, Ian Fasel, and Javier Movellan.  Fully automatic facial action recognition in spontaneous behavior.  In 7th International Conference on Automatic Face and Gesture Recognition (FGR06), pages 223–230. IEEE, 2006.
  * [5] Alvertos Benroumpi and Dimitrios Kollias.  Affwild net and aff-wild database, 2019.
  * [6] Kevin Brady, Youngjune Gwon, Pooya Khorrami, Elizabeth Godoy, William Campbell, Charlie Dagli, and Thomas S Huang.  Multi-modal audio, video and physiological sensor learning for continuous emotion prediction.  In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge, pages 97–104, 2016.
  * [7] Patrick Cardinal, Najim Dehak, Alessandro Lameiras Koerich, Jahangir Alam, and Patrice Boucher.  Ets system for av+ ec 2015 challenge.  In Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge, pages 17–23, 2015.
  * [8] Shizhe Chen and Qin Jin.  Multi-modal dimensional emotion recognition using recurrent neural networks.  In Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge, pages 49–56, 2015.
  * [9] Ciprian Adrian Corneanu, Marc Oliu Simón, Jeffrey F Cohn, and Sergio Escalera Guerrero.  Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications.  IEEE transactions on pattern analysis and machine intelligence, 38(8):1548–1568, 2016.
  * [10] Roddy Cowie and Randolph R Cornelius.  Describing the emotional states that are expressed in speech.  Speech communication, 40(1-2):5–32, 2003.
  * [11] Tim Dalgleish and Mick Power.  Handbook of cognition and emotion.  John Wiley & Sons, 2000.
  * [12] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.  Convolutional two-stream network fusion for video action recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016.
  * [13] Christopher Funk, Savinay Nagendra, Jesse Scott, Bharadwaj Ravichandran, John H Challis, Robert T Collins, and Yanxi Liu.  Learning dynamics from kinematics: Estimating 2d foot pressure maps from video frames.  arXiv preprint arXiv:1811.12607, 2018.
  * [14] Hatice Gunes and Björn Schuller.  Categorical and dimensional affect analysis in continuous input: Current trends and future directions.  Image and Vision Computing, 31(2):120–136, 2013.
  * [15] Rahul Gupta, Nikolaos Malandrakis, Bo Xiao, Tanaya Guha, Maarten Van Segbroeck, Matthew Black, Alexandros Potamianos, and Shrikanth Narayanan.  Multimodal prediction of affective dimensions and depression in human-computer interactions.  In Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, pages 33–40, 2014.
  * [16] Lang He, Dongmei Jiang, Le Yang, Ercheng Pei, Peng Wu, and Hichem Sahli.  Multimodal affective dimension prediction using deep bidirectional long short-term memory recurrent neural networks.  In Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge, pages 73–80, 2015.
  * [17] Berthold KP Horn and Brian G Schunck.  Determining optical flow.  In Techniques and Applications of Image Understanding, volume 281, pages 319–331. International Society for Optics and Photonics, 1981.
  * [18] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu.  3d convolutional neural networks for human action recognition.  IEEE transactions on pattern analysis and machine intelligence, 35(1):221–231, 2012.
  * [19] Heechul Jung, Sihaeng Lee, Junho Yim, Sunjeong Park, and Junmo Kim.  Joint fine-tuning in deep neural networks for facial expression recognition.  In Proceedings of the IEEE international conference on computer vision, pages 2983–2991, 2015.
  * [20] Markus Kächele, Martin Schels, and Friedhelm Schwenker.  Inferring depression and affect from application dependent meta knowledge.  In Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, pages 41–48, 2014.
  * [21] Markus Kächele, Patrick Thiam, Günther Palm, Friedhelm Schwenker, and Martin Schels.  Ensemble methods for continuous affect recognition: Multi-modality, temporality, and challenges.  In Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge, pages 9–16, 2015.
  * [22] Sebastian Kaltwang, Sinisa Todorovic, and Maja Pantic.  Doubly sparse relevance vector machine for continuous facial behavior estimation.  IEEE transactions on pattern analysis and machine intelligence, 38(9):1748–1761, 2015.
  * [23] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei.  Large-scale video classification with convolutional neural networks.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725–1732, 2014.
  * [24] Tae Kyun Kim, Seok Cheol Kee, and Sang Ryong Kim.  Real-time normalization and feature extraction of 3d face data using curvature characteristics.  In Proceedings 10th IEEE International Workshop on Robot and Human Interactive Communication. ROMAN 2001 (Cat. No. 01TH8591), pages 74–79. IEEE, 2001.
  * [25] Dimitris Kollias, George Marandianos, Amaryllis Raouzaiou, and Andreas-Georgios Stafylopatis.  Interweaving deep learning and semantic techniques for emotion analysis in human-machine interaction.  In 2015 10th International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP), pages 1–6. IEEE, 2015.
  * [26] Dimitrios Kollias, Athanasios Tagaris, and Andreas Stafylopatis.  On line emotion detection using retrainable deep neural networks.  In 2016 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1–8. IEEE, 2016.
  * [27] Dimitrios Kollias, Athanasios Tagaris, Andreas Stafylopatis, Stefanos Kollias, and Georgios Tagaris.  Deep neural architectures for prediction in healthcare.  Complex & Intelligent Systems, 4(2):119–131, 2018.
  * [28] Dimitrios Kollias, Miao Yu, Athanasios Tagaris, Georgios Leontidis, Andreas Stafylopatis, and Stefanos Kollias.  Adaptation and contextualization of deep neural network models.  In 2017 IEEE symposium series on computational intelligence (SSCI), pages 1–8. IEEE.
  * [29] Dimitrios Kollias and Stefanos Zafeiriou.  Training deep neural networks with different datasets in-the-wild: The emotion recognition paradigm.  In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2018.
  * [30] Jean Kossaifi, Georgios Tzimiropoulos, Sinisa Todorovic, and Maja Pantic.  Afew-va database for valence and arousal estimation in-the-wild.  Image and Vision Computing, 65:23–36, 2017.
  * [31] Jiangtao Liu, Chaopeng Shen, Te Pei, Kathryn Lawson, Daniel Kifer, Savinay Nagendra, and Srikanth Banagere Manjunatha.  A new rainfall-induced deep learning strategy for landslide susceptibility prediction.  In AGU Fall Meeting Abstracts, volume 2021, pages NH35E–0504, 2021.
  * [32] Patrick Lucey, Jeffrey F Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews.  The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression.  In 2010 ieee computer society conference on computer vision and pattern recognition-workshops, pages 94–101. IEEE, 2010.
  * [33] Patrick Lucey, Jeffrey F Cohn, Kenneth M Prkachin, Patricia E Solomon, and Iain Matthews.  Painful data: The unbc-mcmaster shoulder pain expression archive database.  In Face and Gesture 2011, pages 57–64. IEEE, 2011.
  * [34] Stacy Marsella and Jonathan Gratch.  Computationally modeling human emotion.  Communications of the ACM, 57(12):56–67, 2014.
  * [35] Gary McKeown, Michel Valstar, Roddy Cowie, Maja Pantic, and Marc Schroder.  The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent.  IEEE transactions on affective computing, 3(1):5–17, 2011.
  * [36] Hongying Meng, Di Huang, Heng Wang, Hongyu Yang, Mohammed Ai-Shuraifi, and Yunhong Wang.  Depression recognition based on dynamic facial and vocal expression features using partial least square regression.  In Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge, pages 21–30, 2013.
  * [37] Vimalkumar Mohanraj, M Vimalkumar, M Mithila, and V Vaidehi.  Robust face recognition system in video using hybrid scale invariant feature transform.  Procedia Computer Science, 93:503–512, 2016.
  * [38] Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor.  Affectnet: A database for facial expression, valence, and arousal computing in the wild.  IEEE Transactions on Affective Computing, 10(1):18–31, 2017.
  * [39] Savinay Nagendra, S Banagere Manjunatha, Chaopeng Shen, Daniel Kifer, and Te Pei.  An efficient deep learning mechanism for cross-region generalization of landslide events.  In AGU Fall Meeting Abstracts, volume 2020, pages NH030–0010, 2020.
  * [40] Savinay Nagendra and Daniel Kifer.  Patchrefinenet: Improving binary segmentation by incorporating signals from optimal patch-wise binarization.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1361–1372, 2024.
  * [41] Savinay Nagendra, Daniel Kifer, Benjamin Mirus, Te Pei, Kathryn Lawson, Srikanth Banagere Manjunatha, Weixin Li, Hien Nguyen, Tong Qiu, Sarah Tran, et al.  Constructing a large-scale landslide database across heterogeneous environments using task-specific model updates.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15:4349–4370, 2022.
  * [42] S Nagendra, T Pei, S Banagere Manjunatha, G He, T Qiu, D Kifer, and C Shen.  Cloud-based interactive database management suite integrated with deep learning-based annotation tool for landslide mapping.  In AGU Fall Meeting Abstracts, volume 2020, pages NH030–0011, 2020.
  * [43] Savinay Nagendra, Nikhil Podila, Rashmi Ugarakhod, and Koshy George.  Comparison of reinforcement learning algorithms applied to the cart-pole problem.  In 2017 international conference on advances in computing, communications and informatics (ICACCI), pages 26–32. IEEE, 2017.
  * [44] Savinay Nagendra, Chaopeng Shen, and Daniel Kifer.  Threshnet: Segmentation refinement inspired by region-specific thresholding.  arXiv preprint arXiv:2211.06560, 2, 2022.
  * [45] Savinay Nagendra, Chaopeng Shen, and Daniel Kifer.  Estimating uncertainty in landslide segmentation models.  arXiv preprint arXiv:2311.11138, 2023.
  * [46] Mihalis A Nicolaou, Hatice Gunes, and Maja Pantic.  Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space.  IEEE Transactions on Affective Computing, 2(2):92–105, 2011.
  * [47] Jérémie Nicolle, Vincent Rapp, Kévin Bailly, Lionel Prevost, and Mohamed Chetouani.  Robust continuous prediction of human emotions using multiscale dynamic cues.  In Proceedings of the 14th ACM international conference on Multimodal interaction, pages 501–508, 2012.
  * [48] Maja Pantic and Leon JM Rothkrantz.  Toward an affect-sensitive multimodal human-computer interaction.  Proceedings of the IEEE, 91(9):1370–1390, 2003.
  * [49] Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman.  Deep face recognition.  2015\.
  * [50] Te Pei, Savinay Nagendra, Srikanth Banagere Manjunatha, Guanlin He, Daniel Kifer, Tong Qiu, and Chaopeng Shen.  Utilizing an interactive ai-empowered web portal for landslide labeling for establishing a landslide database in washington state, usa.  In EGU General Assembly Conference Abstracts, pages EGU21–13974, 2021.
  * [51] AJ Piergiovanni, Chenyou Fan, and Michael S Ryoo.  Learning latent subevents in activity videos using temporal attention filters.  In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
  * [52] Fabien Ringeval, Björn Schuller, Michel Valstar, Shashank Jaiswal, Erik Marchi, Denis Lalanne, Roddy Cowie, and Maja Pantic.  Av+ ec 2015: The first affect recognition challenge bridging across audio, video, and physiological data.  In Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge, pages 3–8, 2015.
  * [53] Enrique Sánchez-Lozano, Paula Lopez-Otero, Laura Docio-Fernandez, Enrique Argones-Rúa, and José Luis Alba-Castro.  Audiovisual three-level fusion for continuous estimation of russell’s emotion circumplex.  In Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge, pages 31–40, 2013.
  * [54] Evangelos Sariyanidi, Hatice Gunes, and Andrea Cavallaro.  Automatic analysis of facial affect: A survey of registration, representation, and recognition.  IEEE transactions on pattern analysis and machine intelligence, 37(6):1113–1133, 2014.
  * [55] Evangelos Sariyanidi, Hatice Gunes, and Andrea Cavallaro.  Automatic analysis of facial affect: A survey of registration, representation, and recognition.  IEEE transactions on pattern analysis and machine intelligence, 37(6):1113–1133, 2014.
  * [56] Björn Schuller, Michel Valstar, Florian Eyben, Gary McKeown, Roddy Cowie, and Maja Pantic.  Avec 2011–the first international audio/visual emotion challenge.  In International Conference on Affective Computing and Intelligent Interaction, pages 415–424. Springer, 2011.
  * [57] Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  In Advances in neural information processing systems, pages 568–576, 2014.
  * [58] Catherine Soladié, Hanan Salam, Catherine Pelachaud, Nicolas Stoiber, and Renaud Séguier.  A multimodal fuzzy inference system using a continuous facial expression representation for emotion detection.  In Proceedings of the 14th ACM international conference on Multimodal interaction, pages 493–500, 2012.
  * [59] Athanasios Tagaris, Dimitrios Kollias, and Andreas Stafylopatis.  Assessment of parkinson’s disease based on deep neural networks.  In International Conference on Engineering Applications of Neural Networks, pages 391–403. Springer, 2017.
  * [60] Athanasios Tagaris, Dimitrios Kollias, Andreas Stafylopatis, Georgios Tagaris, and Stefanos Kollias.  Machine learning for neurodegenerative disorder diagnosis—survey of practices and launch of benchmark dataset.  International Journal on Artificial Intelligence Tools, 27(03):1850011, 2018.
  * [61] Y-I Tian, Takeo Kanade, and Jeffrey F Cohn.  Recognizing action units for facial expression analysis.  IEEE Transactions on pattern analysis and machine intelligence, 23(2):97–115, 2001.
  * [62] Panagiotis Tzirakis, George Trigeorgis, Mihalis A Nicolaou, Björn W Schuller, and Stefanos Zafeiriou.  End-to-end multimodal emotion recognition using deep neural networks.  IEEE Journal of Selected Topics in Signal Processing, 11(8):1301–1309, 2017.
  * [63] Michel Valstar, Jonathan Gratch, Björn Schuller, Fabien Ringeval, Denis Lalanne, Mercedes Torres Torres, Stefan Scherer, Giota Stratou, Roddy Cowie, and Maja Pantic.  Avec 2016: Depression, mood, and emotion recognition workshop and challenge.  In Proceedings of the 6th international workshop on audio/visual emotion challenge, pages 3–10, 2016.
  * [64] Michel Valstar and Maja Pantic.  Induced disgust, happiness and surprise: an addition to the mmi facial expression database.
  * [65] Michel Valstar, Björn Schuller, Kirsty Smith, Florian Eyben, Bihan Jiang, Sanjay Bilakhia, Sebastian Schnieder, Roddy Cowie, and Maja Pantic.  Avec 2013: the continuous audio/visual emotion and depression recognition challenge.  In Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge, pages 3–10, 2013.
  * [66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [67] Heng Wang and Cordelia Schmid.  Action recognition with improved trajectories.  In Proceedings of the IEEE international conference on computer vision, pages 3551–3558, 2013.
  * [68] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In European conference on computer vision, pages 20–36. Springer, 2016.
  * [69] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  Lecture Notes in Computer Science, page 20–36, 2016.
  * [70] Lijun Yin, Xiaozhou Wei, Yi Sun, Jun Wang, and Matthew J Rosato.  A 3d facial expression database for facial behavior research.  In 7th international conference on automatic face and gesture recognition (FGR06), pages 211–216. IEEE, 2006.
  * [71] Zhihong Zeng, Maja Pantic, Glenn I Roisman, and Thomas S Huang.  A survey of affect recognition methods: Audio, visual, and spontaneous expressions.  IEEE transactions on pattern analysis and machine intelligence, 31(1):39–58, 2008.
  * [72] Zhihong Zeng, Maja Pantic, Glenn I Roisman, and Thomas S Huang.  A survey of affect recognition methods: Audio, visual, and spontaneous expressions.  IEEE transactions on pattern analysis and machine intelligence, 31(1):39–58, 2008.
  * [73] Lun Zhang, Rufeng Chu, Shiming Xiang, Shengcai Liao, and Stan Z Li.  Face detection based on multi-block lbp representation.  In International conference on biometrics, pages 11–18. Springer, 2007.
  * [74] Xing Zhang, Lijun Yin, Jeffrey F Cohn, Shaun Canavan, Michael Reale, Andy Horowitz, and Peng Liu.  A high-resolution spontaneous 3d dynamic facial expression database.  In 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pages 1–6. IEEE, 2013.
  * [75] Yan Zhou, Heming Zhao, and Xinyu Pan.  Lie detection from speech analysis based on k–svd deep belief network model.  In International Conference on Intelligent Computing, pages 189–196. Springer, 2015.
  * [76] L Zhu, P Tilke, S Nagendra, M Etchebes, and M LeFranc.  A rapid and realistic 3d stratigraphic model generator conditioned on reference well log data.  In Second EAGE Digitalization Conference and Exhibition, volume 2022, pages 1–5. European Association of Geoscientists & Engineers, 2022.
  * [77] Karel Zuiderveld.  Contrast limited adaptive histogram equalization.  In Graphics gems IV, pages 474–485. Academic Press Professional, Inc., 1994.