# Decorrelation in Local Statistics for random operators

M. Krishna
Ashoka University
Plot 2, Rajiv Gandhi Education City
Rai 131029, Haryana, India
email<EMAIL_ADDRESS>
###### Abstract

In this paper we study the local spectral statistics in the localised region
of various random operator models, including the $d$-dimensional the Anderson
model and random Schrödinger operators. It is already established, in the
above models, that at an energy $E$, in the localised energy region of the
spectrum, where the density of states $n(E)>0$, the local eigenvalue
statistics $X_{E}$ is a Poisson processes with intensity $n(E)\mathcal{L}$,
$\mathcal{L}$ being the Lebesgue measure on $\mathbb{R}$. The question of
independence of $X_{E},X_{E^{\prime}}$ for distinct energies was partially
solved in the literature. We solve it completely for all the models for which
the Minami technique works.

##  1 Introduction

The Anderson model is well studied and is shown to exhibit localised states
for large disorder and at extreme energies on the lattice and at the bottom of
the spectrum in the continuous version of the model.

In the localised region of the spectrum the local statistics of eigenvalues at
a fixed energy $E$ is found, by Molchanov [29] and Minami [27] to be a Poisson
point process. Klopp [25] proved that the Poisson processes at distinct
energies $E\neq E^{\prime}$ are independent in $d=1$ and for $d\geq 2$ under
the condition $|E-E^{\prime}|>2d$. The conjecture that it should be true for
$E\neq E^{\prime}$ formally appears in Klopp [25], while we came to know of
this conjecture from Minami. Minami [28] assumed a decorrelation estimate of
Klopp and showed that it leads to the independence for the Anderson model.
Shirley [31, 32] proved decorrelation estimates in $d=1$ for some random
Schrödinger operators and some Anderson models. Hislop-Krishna [23] proved a
similar result, for the limiting compound Poisson processes under the
condition $|E-E^{\prime}|>2d$, in the case of higher rank Anderson models.

In this paper we prove the independence conjecture not only for the Anderson
model on $\mathbb{Z}^{d}$, but also for random Schrödinger operators. Our
results essentially depend upon localisation, Wegner and Minami estimates for
these operators. They are new for random Schrödinger operators for $d\geq 2$.
While we do not explicitly write the results, our proof goes through for the
point interaction model [22], the higher rank Anderson model studied by
Herschenfeld-Hislop [21] and the Canopy tree (the local statistics of
eigenvalues on the Bethe lattice reduces to that of the Canopy tree as pointed
out by Aizenman-Warzel [4] ) in the form given in Narayanan [30].

The localisation results come in two varieties, one uses the multi-scale
analysis initiated by Fröhlich-Spencer [15] and followed by Simon-Wolff [34],
Kunz-Souillard [26], Fröhlich et.al. [16]. The other uses the fractional
moment method, it was initiated by Aizenman-Molchanov [2]. These results were
for the discrete Anderson model. In the continuous version, which are random
Schrödinger operators, localisation was proved by Combes-Hislop [8], Germinet-
Klein [20, 18, 19, 17], which used the multi-scale analysis while the work of
Aizenman et. al. [3] replicated the fractional moment method for a large class
of random Schrödinger operators. Hislop et.al. [22] worked this out for point
interactions.

The books of Carmona-Lacroix [6], Figotin-Pastur [14], Cycon et.al. [10],
Stollman [35], Simon [33], Demuth-Krishna [12] and Aizenman-Warzel [4], the
lecture notes of Kirsch [24], Stolz [35], give various details on
localisation. They also give the Wegner estimate, originally proved by Wegner
[36] and the Minami estimate, originally proved by Minami [27] used in this
paper. Combes et.al. [7] gave a simple proof of the Minami estimate. Dietlein-
Elgart [11] proved the first Minami estimate for Schrödinger operators.
Herschenfeld-Hislop [21] used a similar method to prove Minami estimate and
Poisson statistics for higher rank Anderson models.

The original proof of Poisson statistics is by Molchanov [29] for one
dimension and Minami [27] gave a general technique for proving Poisson
statistics that is widely applicable across models. The other results on
Poisson statistics for the Bethe lattice is proved by Aizenman-Warzel [4] and
by Narayanan [30], whose result follows Minami technique. We note that our
results also extend almost verbatim to Canopy tree case, using the formulation
of Narayanan [30].

Our main theorems are Theorems 2.2 and 2.4.

##  2 The Main Theorems

###  2.1 Anderson Model

The Anderson Model is extensively studied for its spectral properties and the
local spectral statistics valid in the localised regime of the spectrum.

The model we consider is given on $\ell^{2}(\mathbb{Z}^{d})$ by the self-
adjoint operator

$h^{\omega}=\Delta+\sum_{n\in\mathbb{Z}^{d}}\omega_{n}|\delta_{n}\rangle\langle\delta_{n}|,~{}~{}\displaystyle{(\Delta
u)(n)=\sum_{|n-j|=1}u(j)},~{}~{}u\in\ell^{2}(\mathbb{Z}^{d}),$ (1)

with $\\{\omega_{n}\\}$ i.i.d. random variables distributed according to
probability measure $\mu$. Here $\\{\delta_{n}\\}$ is the standard basis for
$\ell^{2}(\mathbb{Z}^{d})$. In the following we refer to the closure of the
set of energies $E\in\sigma(h^{\omega})$ where the fractional moment bound

$\sup_{\epsilon>0}\mathbb{E}(|\langle\delta_{x},(h^{\omega}-E-i\epsilon)^{-1}\delta_{y}\rangle|^{s}\leq
C_{s}e^{-\gamma_{s}|x-y|},~{}~{}\text{for ~{} some }~{}~{}0<s<1.$ (2)

holds as the localised region.

The fractional moment bound was proved, by Aizenman-Molchanov [2], for the
operators $h^{\omega}$ when the measure $\mu$ is absolutely continuous and
Hölder continuous with exponent $0<s<1$ and the variance of $\mu$ is large
compared to $2d$.

For defining the local eigenvalue statistics processes, we need to restrict
the operators $h^{\omega}$ to finite dimensional cubes, hence we define a cube
$\Lambda_{L}$ in $\mathbb{Z}^{d}$ to be

$\displaystyle\Lambda_{L}(0)$
$\displaystyle=\\{x\in\mathbb{Z}^{d}:x_{j}\in\\{-L+1,\dots,L\\}\\}.$ (3)

We write $\Lambda_{L}(n)$ to be similar cubes centred at $n$ in
$\mathbb{Z}^{d}$. We denote the projection valued spectral measure of a self-
adjoint operator $H$ by $E_{H}(\cdot)$. Then we define a few random measures.
Let $\Lambda_{L}(0)$ be covered by a disjoint collection of translates of
cubes $\Lambda_{L^{\beta}}(0)$, $0<\beta<1$, with $\beta$ to be chosen later.
Such a cover will have $n_{L}=(\frac{L}{L^{\beta}})^{d}$ number of elements in
it, so we label these cubes by the set $\\{1,2,\dots,n_{L}\\}$ and refer to
the cubes by $\Lambda_{p,L}$. Thus
$\Lambda_{L}(0)=\displaystyle{\sqcup_{p=1}^{n_{L}}}\Lambda_{p,L}$ (where the
symbol denotes disjoint union).

We set

$\displaystyle h_{L}^{\omega}$
$\displaystyle=\chi_{\Lambda_{L}(0)}h^{\omega}\chi_{\Lambda_{L}(0)}.$ (4)

We define the self-adjoint operators

$\displaystyle h^{\omega}_{p,L}$
$\displaystyle=\chi_{\Lambda_{p,L}}h^{\omega}\chi_{\Lambda_{p,L}}.$ (5)

For $B\in{\mathcal{B}}_{\mathbb{R}}$, let $B_{L,E}=E+BL^{-d}$. We set

$\displaystyle\eta_{L,E}^{\omega}(B)$
$\displaystyle=Tr(\chi_{\Lambda_{L}(0)}E_{h^{\omega}_{L}}(B_{L,E}))$ (6)

and

$\displaystyle\eta_{p,E,L}^{\omega}(B)$
$\displaystyle=Tr(\chi_{\Lambda_{p,L}}E_{h^{\omega}_{p,L}}(B_{L,E})),~{}p=1,2,\dots,n_{L}.$
(7)

We note that the self-adjoint operators defined in the equations (4), (5) have
discrete spectrum and the processes defined above count the number of their
eigenvalues in $B_{L,E}$. Thus they are point processes. We also see from the
definitions of the operators $h_{p,L}^{\omega}$ that for $p\neq q$,
$h_{p,L}^{\omega}$ and $h_{q,L}^{\omega}$ depend on mutually independent sets
of random variables
$\\{\omega_{n}:n\in\Lambda_{p,L}\\},\\{\omega_{n}:n\in\Lambda_{q,L}\\}$
respectively, so $\eta^{\omega}_{p,E,L},\eta^{\omega}_{q,E^{\prime},L}$ are
mutually independent as long as $p\neq q$ irrespective of the whether
$E=E^{\prime}$ or not.

The integrated density of states associated with the random operators
$h^{\omega}$ is the average spectral measure

$\mathcal{N}(\cdot)=\mathbb{E}(\langle\delta_{0},E_{h^{\omega}}(\cdot)\delta_{0}\rangle)$

and it is known to be absolutely continuous. The density of $\mathcal{N}$ is
denoted by $n(\cdot)$.

We state a theorem, proved by Minami [27], on the existence and nature of
local spectral statistics in the Anderson model. In the following
$\mathcal{L}$ denotes the Lebesgue measure on $\mathbb{R}.$

######  Theorem 2.1 (Minami).

Consider $\ell^{2}(\mathbb{Z}^{d}),~{}d\geq 1$ and the random self-adjoint
operators $h^{\omega}$, given in equation (2) with an absolutely continuous,
Hölder continuous measure $\mu$, with exponent $0<s<1$ and has large variance.
Let $E$ be in the localised region of the spectrum of $h^{\omega}$ such that
$n(E)\neq 0$. Then the sequences of random point processes
$\eta^{\omega}_{L,E}$ and $\sum_{p=1}^{n_{L}}\eta^{\omega}_{p,E,L}$ have the
same limit $X_{E}^{\omega}$ in distribution, which is a Poisson point process
with intensity $n(E)\mathcal{L}$. Further

$\lim_{L\rightarrow\infty}\mathbb{E}(\eta^{\omega}_{L,E}(B))=\lim_{L\rightarrow\infty}\mathbb{E}(\sum_{p=1}^{n_{L}}\eta^{\omega}_{p,E,L}(B))=n(E)|B|,$

for all bounded borel sets $B$.

Minami [27], used the fractional moment bounds (2), the Wegner estimate, valid
for $p\in\\{1,2,\dots,n_{L}\\}.$

$\displaystyle\mathbb{E}(\eta_{p,E,L}^{\omega}(B))\leq Cn_{L}^{-1}|B|,$ (8)

and the Minami estimate

$\displaystyle\mathbb{P}(\eta_{p,E,L}^{\omega}(B))\geq 2)\leq
Cn_{L}^{-2}|B|^{2},$ (9)

to prove the above theorem.

Then our theorem for the Anderson model is the following.

######  Theorem 2.2.

Consider $\ell^{2}(\mathbb{Z}^{d}),~{}d\geq 1$ and the random operators self-
adjoint operators $h^{\omega}$, given in equation (1) with an absolutely
continuous, Hölder continuous measure $\mu$, with exponent $0<s<1$ and has
large variance. Let $E\neq E^{\prime}$ in the localised region of the spectrum
of $h^{\omega}$ such that $n(E),n(E^{\prime})>0$. Then the Poisson point
processes $X_{E}^{\omega}$ and $X_{E^{\prime}}^{\omega}$ associated with
$h^{\omega}$ are independent processes.

###  2.2 Random Schrödinger Operators

The continuous version of the model is a random Schrödinger operator on
$L^{2}(\mathbb{R}^{d})$ given by

$H^{\omega}=-\Delta+\sum_{n\in\mathbb{Z}^{d}}\omega_{n}u_{n},$ (10)

where $u_{n}$ is an operator of multiplication by a function $u_{n}(x)$. The
collection $\\{u_{n}(x)\\}$ satisfy the conditions $V1,V2$ of Dietlein-Elgart
[11] (our $u_{n}=V_{n}$ of [11])and the i.i.d random variables
$\\{\omega_{n}\\}$ have a common distribution with an absolutely continuous
Lipschitz continuous density $\rho$ which is bounded below, thus the operators
$H^{\omega}$ satisfy the conditions $V1-V4$ of [11].

For defining the local eigenvalue statistics processes, we need to restrict
the operators to finite dimensional cubes, hence we define a cube
$\Lambda_{L}$ in $\mathbb{R}^{d}$ to be

$\displaystyle\Lambda_{L}(0)$
$\displaystyle=\\{x\in\mathbb{R}^{d}:x_{j}\in(-\frac{L}{2},\frac{L}{2}]\\}.$
(11)

We write $\Lambda_{L}(n)$ to be similar cubes centred at $n$ in
$\mathbb{R}^{d}.$

We set

$\displaystyle H_{L}^{\omega}$
$\displaystyle=-\Delta_{\Lambda_{L}(0)}+\displaystyle{\sum_{n\in\Lambda_{L}(0)\cap\mathbb{Z}^{d}}\omega_{n}u_{n},}$
(12)

where $\Delta_{S}$ denotes the Laplacian with some fixed boundary condition
(periodic, Dirichlet or Neumann) on the, if necessary slightly smoothened,
boundary of $S$.

We define the cubes $\Lambda_{p,\ell},~{}p=1,\dots,n_{L}$,
$n_{L}=(\frac{L}{\ell})^{d},~{}0<\beta<1$, in $\mathbb{R^{d}}$, ‘similar’ to
that done in the previous section. We choose these so that
$\sqcup_{p=1}^{n_{L}}\Lambda_{p,\ell}=\Lambda_{L}(0)$. We define the self-
adjoint operators, similar to the definitions in the previous section,

$\displaystyle H^{\omega}_{p,L}$
$\displaystyle=-\Delta_{\Lambda_{p,L}}+\displaystyle{\sum_{n\in\Lambda_{p,\ell}\cap\mathbb{Z}^{d}}\omega_{n}u_{n}.}$
(13)

For $B\in{\mathcal{B}}_{\mathbb{R}}$, let $B_{L,E}=E+BL^{-1}$. We set

$\displaystyle\eta_{L,E}^{\omega}(B)$
$\displaystyle=Tr(\chi_{\Lambda_{L}(0)}E_{H^{\omega}_{L}}(B_{L,E})).$ (14)

and

$\displaystyle\eta_{p,E,L}^{\omega}(B)$
$\displaystyle=Tr(\chi_{\Lambda_{p,L}}E_{H^{\omega}_{p,L}}(B_{L,E})),~{}p=1,2,\dots,n_{L}.$
(15)

As in the case of the Anderson model, localisation is proved by Combes-Hislop
[9], Germinet-Klein [20, 18, 19, 17],Elgart-Klein [13], Aizenman et.al. [3],
Aizenman et.al. [1]. We state the localisation result in the form proved in
[1].

######  Theorem 2.3.

Consider the random operators $H^{\omega},H^{\omega}_{L},H^{\omega}_{p,L}$ and
denote by $A^{\omega}$ any one of them. Then there is a number $E_{M}>0$ such
that for all $E<E_{M}$ we have

$\sup_{\epsilon>0}\mathbb{E}\big{(}\|u_{x}(A^{\omega}-E-i\epsilon)^{-1}u_{y}\|^{s}\big{)}<C_{s}e^{-\gamma_{s}|x-y|},$
(16)

where $x,y$ are in $\mathbb{Z}^{d}$, $\Lambda(0)$ and $\Lambda_{p,L}$
according as $A^{\omega}$ is $H^{\omega}$, $H_{L}^{\omega}$ and
$H_{p,L}^{\omega}$ respectively.

The Wegner estimate for the random measures given in equations (14, 15) are
identical to those of the Anderson model, namely

$\mathbb{E}(\eta_{p,L,E}(A))\leq
C(\frac{\ell}{L})^{d}|A|,~{}~{}\mathbb{E}(\eta_{L,E}(A))\leq CL^{d}|A|.$ (17)

The Minami estimate was hard in this case and it was proved by Dietlein-
Elgart [11, Theorem 2.2] (see also [HerHis]), in the form

$\displaystyle\mathbb{P}\big{(}Tr(E_{H_{p,L}^{\omega}}([E-\delta,E+\delta])\geq
2\big{)}\leq C\ell^{4d}\delta|\ln\delta|^{-K},$ (18)

if $E<E_{M}$, $E_{M}$ as in Theorem 2.3 and $\delta<e^{-\ln(|\ell|)^{5d}}$. If
we chose $\delta=L^{-d}$, the condition on $\delta$ translates to
$\ell<C_{d}e^{(\ln(L)^{\frac{1}{5d}}}$, a choice which is possible consistent
with the equation (19).

Therefore For an appropriate choice of $0<\beta<1$ and $0<s<1$ and
$\ell,\delta=L^{-d},$ (in Theorem 2.3), we will have

$\lim_{L\rightarrow\infty}\sum_{p=1}^{n_{L}}\mathbb{P}(\\{\eta_{p,L,E}(A)\geq
2\\})=0.$ (19)

for large enough $L$. Therefore the existence of the limit in distribution of

$\eta_{L,E}^{\omega}(A),\sum_{p=1}^{n_{L}}\eta_{p,L,E}^{\omega}(A)$

both converging to the same Poisson point process $X_{E}^{\omega}$ with
intensity $n(E)\mathcal{L}$, at points $E<E_{M},~{}~{}n(E)>0$ follows almost
verbatim as in the proof of the Anderson model of the previous section. We
call this process as $X_{E}^{\omega}$.

Then we have the theorem on independence.

######  Theorem 2.4.

Consider the random operators $H^{\omega}$ on $L^{2}(\mathbb{R}^{d})$ as in
equation (10) satisfying the conditions mentioned there. Then the Poisson
point processes $X_{E}^{\omega},X_{E^{\prime}}^{\omega}$ corresponding to two
energies $E\neq E^{\prime},E,E^{\prime}<E_{M}$. Then the processes are
independent.

##  3 Proofs

There is a rather simple but critical decorrelation result that is at the
heart of the proof of independence which applies to all the models in this
paper. So we state the result as a lemma. In the following
$\mathcal{B}_{\mathbb{R}}$ denotes the Borel $\sigma$-algebra of subsets of
$\mathbb{R}$.

######  Lemma 3.1 (Decorrelation Lemma).

Let $(\Omega,\mathcal{B}_{\Omega},\mathbb{P})$ be a probability space and
suppose $\\{\xi^{\omega}_{p,L},~{}L\in\mathbb{N},1\leq p\leq n_{L}\\}$,
$n_{L}\rightarrow\infty,~{}~{}as~{}~{}L\rightarrow\infty$, are a collection of
random counting measures on $\mathbb{R}$, such that for any bounded
$S\in\mathcal{B}_{\mathbb{R}}$,

$\sum_{p=1}^{n_{L}}\mathbb{P}(\\{\xi^{\omega}_{p,L}(S)\geq 2\\})\rightarrow
0.$ (20)

Then for any bounded pair $A,B\in\mathcal{B}_{\mathbb{R}},~{}~{}A\cap
B=\emptyset$,

$\displaystyle\sum_{p=1}^{n_{L}}\mathbb{P}(\\{\xi^{\omega}_{p,L}(A)\geq
1,\xi^{\omega}_{p,L}(B)\geq 1\\})\rightarrow 0.$ (21)

Proof: Since for each $p\in\\{1,\dots,n_{L}\\}$, $\xi^{\omega}_{p,L}$ is a
measure on $\mathcal{B}_{\mathbb{R}}$, we have, for disjoint bounded pairs
$A,B\in\mathcal{B}_{\mathbb{R}}$, $\sqcup$ denoting disjoint union,

$\xi^{\omega}_{p,L}(A\sqcup B)=\xi^{\omega}_{p,L}(A)+\xi^{\omega}_{p,L}(B).$

Hence,

$\\{\omega:\xi^{\omega}_{p,L}(A)\geq 1,\xi^{\omega}_{p,L}(B)\geq
1\\}\subset\\{\omega:\xi^{\omega}_{p,L}(A\sqcup B)\geq 2\\}.$

Therefore,

$\mathbb{P}(\\{\omega:\xi^{\omega}_{p,L}(A)\geq 1,\xi^{\omega}_{p,L}(B)\geq
1\\})\leq\mathbb{P}(\\{\omega:\xi^{\omega}_{p,L}(A\sqcup B)\geq 2\\}).$

Then summing both sides of the above equation over $p$ and using inequality
(20), we get the result, since $S=A\sqcup B$ is also a bounded borel set. ∎

######  Remark 3.2.

It is precisely the content of this Lemma that Minami [28] assumes to prove
independence and which Klopp [25] proves with restrictions on the pair of
energies $E,E^{\prime}$.

Proof of Theorem 2.2 We consider two points $E\neq E^{\prime}$ in the
localised region of the Anderson operators $h^{\omega}$ and choose and fix
arbitrary bounded intervals $A,B\in\mathcal{B}_{\mathbb{R}}$. Let
$d(A)=sup\\{|x-y|:x,y\in A\\}$, denote the diameter of $A$.

We set $\alpha=|E-E^{\prime}|$. By assumption $\alpha>0$. Since $A,B$ are
bounded sets there is a $L_{0}\in\mathbb{N}$ such that

$d(AL^{-d})+d(BL^{-d})<\frac{\alpha}{2},~{}~{}\text{for ~{} all}~{}~{}L\geq
L_{0}.$ (22)

Therefore, for all $L\geq L_{0}$

$dist(A_{L,E},B_{L,E^{\prime}})\geq||E-E^{\prime}|-(d(AL^{-d})+d(BL^{-d}))|\geq\frac{\alpha}{2}.$
(23)

Therefore $A_{L,E},B_{L,E^{\prime}}$ are disjoint for all $L\geq L_{0}$.

Since by Minami’s theorem (Theorem 2.1), the limits of
$\eta_{L,E}^{\omega}(S)$ and that of
$\sum_{p=1}^{n_{L}}\eta_{p,E,L}^{\omega}(S)$ are the same for each bounded
borel set $S$, we consider the random variables

$\zeta_{L,E}^{\omega}(A)=\sum_{p=1}^{n_{L}}\eta_{p,E,L}^{\omega}(A)$ (24)

and show that their limits in distribution, which are
$X_{E}^{\omega}(A),X_{E^{\prime}}^{\omega}(B)$, are independent.

Since convergence in distribution of random variables is equivalent to the
convergence of their characteristic functions, (see [5, Theorem 7.6]), it is
sufficient to prove that for any $t\in\mathbb{R}^{2}$,

$\mathbb{E}\big{(}e^{i(t_{1}\zeta_{L,E}^{\omega}(A)+t_{2}\zeta_{L,E}^{\omega}(B))}\big{)}\rightarrow\mathbb{E}\big{(}e^{it_{1}X_{E}^{\omega}(A)}\big{)}\mathbb{E}\big{(}e^{it_{2}X_{E^{\prime}}^{\omega}(B)}\big{)},$
(25)

as $L\rightarrow\infty$. Since the random variables
$X_{E}^{\omega}(A),X_{E^{\prime}}^{\omega}(B)$ are Poisson distributed with
intensity $n(E)|A|$ and $n(E^{\prime})|B|$ respectively, their characteristic
functions are, by Minami’s theorem (Theorem 2.1),

$\mathbb{E}\big{(}e^{it_{1}X_{E}^{\omega}(A)}\big{)}=e^{(e^{t_{1}}-1)n(E)|A|},~{}~{}\mathbb{E}\big{(}e^{it_{2}X_{E^{\prime}}^{\omega}(B)}\big{)}=e^{(e^{t_{2}}-1)n(E^{\prime})|B|}.$

Therefore to show independence, it is enough to show

$\mathbb{E}\big{(}e^{i(t_{1}\zeta_{L,E}^{\omega}(A)+t_{2}\zeta_{L,E^{\prime}}^{\omega}(B))}\big{)}\rightarrow
e^{(e^{t_{1}}-1)(n(E)|A|)+(e^{t_{2}}-1)(n(E^{\prime})|B|)},$ (26)

to prove the theorem. The above left hand side of the expression equals, in
view of the discussion after equation (7),

$\prod_{p=1}^{n_{L}}\mathbb{E}\big{(}e^{i(t_{1}\eta_{p,E,L}^{\omega}(A)+t_{2}\eta^{\omega}_{p,E^{\prime},L}(B))}\big{)}$
(27)

Writing the terms in the product as exponential of a logarithm (for which we
take the principal branch), approximate the logarithm $\ln(x+1)\approx
x+O(|x|^{2})$ for small $x$ and compute using the Wegner estimate equation
(8), (as in the proof of [23, Proposition 4.3]) to get

$\displaystyle\prod_{p=1}^{n_{L}}\mathbb{E}\big{(}e^{i(t_{1}\eta_{p,E,L}^{\omega}(A)+t_{2}\eta^{\omega}_{p,E^{\prime},L}(B))}\big{)}$
(28)
$\displaystyle=\prod_{p=1}^{n_{L}}e^{\sum_{k,\ell=0}^{\infty}(e^{i(t_{1}k+t_{2}\ell}-1)\mathbb{P}(\\{\eta^{\omega}_{p,E,L}(A)=k,\eta^{\omega}_{p,E^{\prime},L}(B)=\ell\\})+o(n_{L}^{-1})}.$

The inequality (23), implies that for all $L\geq L_{0}$, we have the sets
$A_{L,E},B_{L,E^{\prime}}$ are disjoint, so by Lemma 3.1, that

$\displaystyle\prod_{p=1}^{n_{L}}\mathbb{E}\big{(}e^{i(t_{1}\eta_{p,E,L}^{\omega}(A)+t_{2}\eta^{\omega}_{p,E^{\prime},L}(B))}\big{)}$
(29)
$\displaystyle=\big{(}e^{\sum_{p=1}^{n_{L}}\sum_{k,\ell=0}^{1}(e^{i(t_{1}k+t_{2}\ell}-1)\mathbb{P}(\\{\eta^{\omega}_{p,E,L}(A)=k,\eta^{\omega}_{p,E^{\prime},L}(B)=\ell\\})}\big{)}O(1).$

We write the exponent on the right hand side and compute

$\displaystyle\sum_{p=1}^{n_{L}}\sum_{k,\ell=0}^{1}(e^{i(t_{1}k+t_{2}\ell}-1)\mathbb{P}(\\{\eta^{\omega}_{p,E,L}(A)=k,\eta^{\omega}_{p,E^{\prime},L}(B)=\ell\\})$
(30)
$\displaystyle=\sum_{p=1}^{n_{L}}(e^{it_{1}}-1)\mathbb{P}(\\{\eta^{\omega}_{p,E,L}(A)=1,\eta^{\omega}_{p,E^{\prime},L}(B)=0\\})$
$\displaystyle~{}~{}~{}+\sum_{p=1}^{n_{L}}(e^{it_{2}}-1)\mathbb{P}(\\{\eta^{\omega}_{p,E,L}(A)=0,\eta^{\omega}_{p,E^{\prime},L}(B)=1\\})$
$\displaystyle=\sum_{p=1}^{n_{L}}(e^{it_{1}}-1)\mathbb{P}(\\{\eta^{\omega}_{p,E,L}(A)=1\\})$
$\displaystyle~{}~{}~{}+\sum_{p=1}^{n_{L}}(e^{it_{2}}-1)\mathbb{P}(\\{\eta^{\omega}_{p,E^{\prime},L}(B)=1\\})+o(n_{L}^{-1}).$

To get the last step we used the fact that, with
$\mathbb{N}=\\{0,1,2,3,\dots\\}$,

$\displaystyle\mathbb{P}(\\{\eta^{\omega}_{p,E,L}(A)=0,\eta^{\omega}_{p,E^{\prime},L}(B)=1\\})$
(31)
$\displaystyle=\mathbb{P}(\\{\eta^{\omega}_{p,E^{\prime},L}(B)=1\\})-\mathbb{P}(\\{\eta^{\omega}_{p,E,L}(A)\geq
1,\eta^{\omega}_{p,E^{\prime},L}(B)=1\\})$
$\displaystyle=\mathbb{P}(\\{\eta^{\omega}_{p,E^{\prime},L}(B)=1\\})-o(n_{L}^{-1})$

using Lemma 3.1. Similarly

$\mathbb{P}(\\{\eta^{\omega}_{p,E,L}(A)=1,\eta^{\omega}_{p,E^{\prime},L}(B)=0\\})=\mathbb{P}(\\{\eta^{\omega}_{p,E,L}(A)=1\\})-o(n_{L}^{-1}).$
(32)

Combining equations (30, 31, 32) together with Minami’s theorem, Theorem 2.1,
(it is not hard to see from these theorems that
$\sum_{p=1}^{n_{L}}\mathbb{P}(\\{\eta^{\omega}_{p,L,E}(A)=1\\})=n(E)|A|$) we
get that

$\lim_{L\rightarrow\infty}\prod_{p=1}^{n_{L}}\mathbb{E}\big{(}e^{i(t_{1}\eta_{p,E,L}^{\omega}(A)+t_{2}\eta^{\omega}_{p,E^{\prime},L}(B))}\big{)}=e^{(e^{t_{1}}-1)(n(E)|A|)+(e^{t_{2}}-1)(n(E^{\prime})|B|)}.$

This completes the proof. ∎

Proof of Theorem 2.4: The proof of this theorem is almost identical to the
proof of Theorem 2.2 given above, hence we omit it.∎

## Acknowledgement

We thank Chennai Mathematical Institute for the hospitality provided while
writing this paper.

## Data Availability

Data sharing not applicable to this article as no datasets were generated or
analysed during the current study.

## Declarations

Conflict of interest There is no conflict of interest.

## References

  * [1] Michael Aizenman, Alexander Elgart, Serguei Naboko, H. Jeffrey Schenker, and Gunter Stolz.  Moment analysis for localization in random schrödinger operators.  Inventiones mathematicae, 163(2):343–413, 2005.
  * [2] Michael Aizenman and Stanislav Molchanov.  Localization at large disorder and at extreme energies: An elementary derivation.  Communications in Mathematical Physics, 157(2):245–278, 1993\.
  * [3] Michael Aizenman, Jeffrey H Schenker, Roland M Friedrich, and Dirk Hundertmark.  Finite-volume fractional-moment criteria for Anderson localization.  Communications in Mathematical Physics, 224(1):219–253, 2001\.
  * [4] Michael Aizenman and Simone Warzel.  The canopy graph and level statistics for random operators on trees.  Mathematical Physics, Analysis and Geometry, 9(4):291–333, 2006\.
  * [5] P. Billingsley.  Probability and Measure.  Wiley Series in Probability and Statistics. Wiley, 1995.
  * [6] René Carmona and Jean Lacroix.  Spectral theory of random Schrödinger operators.  Springer Science & Business Media, 2012.
  * [7] Jean-Michel Combes, François Germinet, and Abel Klein.  Generalized eigenvalue-counting estimates for the Anderson model.  J. Stat. Phys., 135(2):201–216, 2009.
  * [8] J.M. Combes and P.D. Hislop.  Localization for some continuous, random hamiltonians in d-dimensions.  Journal of Functional Analysis, 124(1):149 – 180, 1994.
  * [9] J.M. Combes and P.D. Hislop.  Localization for some continuous, random Hamiltonians in d-dimensions.  Journal of Functional Analysis, 124(1):149 – 180, 1994.
  * [10] H. L. Cycon, R. G. Froese, W. Kirsch, and B. Simon.  Schrödinger operators with application to quantum mechanics and global geometry.  Texts and Monographs in Physics. Springer-Verlag, Berlin, study edition, 1987.
  * [11] Adrian Dietlein and Alexander Elgart.  Level spacing for continuum schrödniger operators with applications.  J. Euro. Math. Soc., 23(4):1257–1293, 2021.
  * [12] Dhriti Ranjan Dolai and M Krishna.  Poisson statitiscs for anderson model with singular randomness.  J. Ramanujan Math. Soc., 30(3):251–266, 2015.
  * [13] Alexander Elgart and Abel Klein.  Ground state energy of trimmed discrete Schrödinger operators and localization for trimmed Anderson models.  J. Spectr. Theory, 4(2):391–413, 2014.
  * [14] A Figotin and L Pastur.  Spectra of Random and Almost-Periodic Operators.  Springer Verlag, Berlin, 1992.
  * [15] J. Fröhlich and T. Spencer.  Absence of diffusion in the Anderson tight binding model for large disorder or low energy.  Comm. Math. Phys, 88:151–184, 1983.
  * [16] Jürg Fröhlich, Fabio Martinelli, Elisabetta Scoppola, and Thomas Spencer.  Constructive proof of localization in the Anderson tight binding model.  Communications in Mathematical Physics, 101(1):21–46, 1985\.
  * [17] F Germinet and A Klein.  Explicit finite volume criteria for localization in random media and applications.  Geom. Funct. Anal. To appear, 2002.
  * [18] François Germinet and Abel Klein.  Bootstrap multiscale analysis and localization in random media.  Communications in Mathematical Physics, 222(2):415–448, 2001.
  * [19] François Germinet and Abel Klein.  Bootstrap multiscale analysis and localizationśin random media.  Communications in Mathematical Physics, 222(2):415–448, 2001.
  * [20] Francois Germinet and Abel Klein.  New characterizations of the region of complete localization for random schrödinger operators.  Journal of Statistical Physics, 122(1):73–94, 2006.
  * [21] Samuel Herschenfeld and Peter D. Hislop.  Local eigenvalue statistics for higher-rank anderson models after dietlein–elgart.  Reviews in Mathematical Physics, 35(08):2350017, 2023.
  * [22] Peter D Hislop, Werner Kirsch, and M Krishna.  Eigenvalue statistics for schrodinger operators with random point interactions on rd, d=1,2,3.  Jour. Math. Phys., 61:92–103, 2020.
  * [23] Peter D. Hislop and M. Krishna.  Eigenvalue statistics for random Schrödinger operators with non rank one perturbations.  Communications in Mathematical Physics, 340(1):125–143, 2015\.
  * [24] Werner Kirsch.  An invitation to random Schrödinger operators.  In Random Schrödinger operators, volume 25 of Panor. Synthèses, pages 1–119. Soc. Math. France, Paris, 2008.  With an appendix by Frédéric Klopp.
  * [25] Frédéric Klopp.  Decorrelation estimates for the eigenlevels of the discrete Anderson model in the localized regime.  Comm. Math. Phys., 303(1):233–260, 2011.
  * [26] H. Kunz and B. Souillard.  Sur le spectre des opèrateurs aux diffèrences finies alèatoires.  Comm. Math. Phys., 78:201–246, 1980/81.
  * [27] Nariyuki Minami.  Local fluctuation of the spectrum of a multidimensional Anderson tight binding model.  Comm. Math. Phys., 177(3):709–725, 1996.
  * [28] Nariyuki Minami.  Energy level statistics: a formulation and some examples.  In Spectra of random operators and related topics, volume B27 of RIMS Kôkyûroku Bessatsu, pages 79–97. Res. Inst. Math. Sci. (RIMS), Kyoto, 2011.
  * [29] S.A. Molchanov.  The local structure of the spectrum of one-dimensional schrödinger operator.  Commun. math. Phys., 78:429–446, 1981.
  * [30] P.A Narayanan.  Eigenvalue Statistics of Higher Rank Anderson Tight Binding Model Over The Canopy Tree.  Phd thesis, The Institute for Mathematical Sciences, Dec 2021.  Available at url http://www.hbni.ac.in/phdthesis/math/MATH10201304004.pdf.
  * [31] Christopher Shirley.  Decorrelation estimates for random discrete Schrödinger operators in dimension one and applications to spectral statistics.  J. Stat. Phys., 158(6):1298–1340, 2015.
  * [32] Christopher Shirley.  Decorrelation estimates for some continuous and discrete random Schrödinger operators in dimension one and applications to spectral statistics.  J. Differential Equations, 263(9):5987–6024, 2017.
  * [33] Barry Simon.  Trace ideals and their applications, volume 120 of Mathematical Surveys and Monographs.  American Mathematical Society, Providence, RI, second edition, 2005.
  * [34] T. Simon, B. Wolf.  Singular continuum spectrum under rank one perturbations and localisation for random Hamiltonians.  Comm. Pure. App. Math., 39:75–90, 1986.
  * [35] Günter Stolz.  An introduction to the mathematics of Anderson localization.  In Entropy and the quantum II, volume 552 of Contemp. Math., pages 71–108. Amer. Math. Soc., Providence, RI, 2011.
  * [36] F. Wegner.  Bounds on the density of states in disordered systems.  Zeit. fur Phy, B44:9–15, 1981.


# Leveraging Synthetic Data for Generalizable and Fair Facial Action Unit
Detection

Liupei Lu, Yufeng Yin, Yuming Gu, Yizhen Wu, Pratusha Prasad, Yajie Zhao, and
Mohammad Soleymani All authors are with the Institute for Creative
Technologies, University of Southern California, Playa Vista, CA, 90094, USA
The work was done when Lu was with USC Institute for Creative Technologies in
2021.

###### Abstract

Facial action unit (AU) detection is a fundamental block for objective facial
expression analysis. Supervised learning approaches require a large amount of
manual labeling which is costly. The limited labeled data are also not diverse
in terms of gender which can affect model fairness. In this paper, we propose
to use synthetically generated data and multi-source domain adaptation (MSDA)
to address the problems of the scarcity of labeled data and the diversity of
subjects. Specifically, we propose to generate a diverse dataset through
synthetic facial expression re-targeting by transferring the expressions from
real faces to synthetic avatars. Then, we use MSDA to transfer the AU
detection knowledge from a real dataset and the synthetic dataset to a target
dataset. Instead of aligning the overall distributions of different domains,
we propose Paired Moment Matching (PM2) to align the features of the paired
real and synthetic data with the same facial expression. To further improve
gender fairness, PM2 matches the features of the real data with a female and a
male synthetic image. Our results indicate that synthetic data and the
proposed model improve both AU detection performance and fairness across
genders, demonstrating its potential to solve AU detection in-the-wild.

###### Index Terms:

Facial Expression Analysis, Action Unit Detection, Synthetic Data Generation,
Domain Adaptation, Fairness.

##  1 Introduction

Facial expressions play an important role in interpersonal social
communication by conveying human emotions and intentions [1]. The Facial
Action Coding System (FACS) is a taxonomy of facial activities that can
describe expressions through anatomical action units, _e.g._ , inner brow
raiser (AU1), lip corner puller (AU 12) [2]. Unlike facial expressions of
emotions, _e.g._ , happiness and sadness, for which there is no absolute
agreement among raters [3, 4], FACS provides an objective measure tied to the
exact movements of facial muscles to describe specific facial behaviors [5].

Most of the existing work on AU detection performs within-domain evaluation,
with training and testing data from the same dataset [6, 7, 8] while cross-
domain evaluation, with training and testing data from different datasets, has
not been fully investigated. Such supervised learning approaches require a
large amount of manually labeled data by trained coders [9]. In addition to
the cost of data labeling, current AU detection datasets are not
representatives of the general population. Three of the most commonly used
datasets, BP4D [10], DISFA [11], and GFT [12] suffer from imbalanced gender
distributions. In this paper, we use ”gender” to specify the apparent gender
identity, as judged by observers and not the real gender of the subjects. The
gender bias in the data can be detrimental to the fairness of the resulting
machine learning models. To alleviate the problem of data scarcity and lack of
diversity, synthetic AU datasets have been used in recent facial expression
recognition and AU detection studies [13, 14, 15].

Domain adaptation (DA) aims to minimize the impact of domain shift between the
source and target domains. Multi-source domain adaptation (MSDA) is a powerful
extension that can transfer knowledge from different source domains to a
target domain [16]. MSDA can fully utilize data from both real and synthetic
domains to learn a generalizable AU detection model.

In this paper, we propose to use synthetically generated data and multi-source
domain adaptation to address the problems of the scarcity of labeled data and
lack of diversity in the subjects (see Figure LABEL:fig:teaser). We propose a
pipeline to generate a diverse and balanced dataset through synthetic facial
expression re-targeting. Specifically, we extract the blendshape parameters
from a real dataset, _i.e._ , BP4D, to re-target the expressions from the real
videos to the synthetic avatars. The synthetic dataset consists of 60 avatars
and 214,146 frames in total. We then use multi-source domain adaptation
techniques to transfer the AU detection knowledge from a real dataset and a
synthetic dataset to a target dataset.  Existing methods such as Moment
Matching for Multi-Source Domain Adaptation ($\rm M^{3}SDA$) [17] align the
feature distributions of multiple domains. These methods can reduce the domain
gap to some extent, however, there are three problems: (i) It is difficult to
completely align the feature distributions of different domains because of
their different characteristics. (ii) Matching the overall distributions does
not consider any class-specific boundaries which results in reduced AU
detection performance. (iii) More importantly, matching the overall
distributions does not align different protected groups explicitly which may
result in inferior fairness. To address these problems, we propose Paired
Moment Matching (PM2) to align the features of the paired real and synthetic
data with the same facial display. Instead of matching the overall feature
distributions of different domains, the proposed model aims to align the data
with the same class label and match the features of the real data across
protected groups _i.e._ , female and male synthetic avatars. Therefore, PM2
can keep the well-learned AU features during the domain alignment and obtain a
fair representation, _i.e._ , across genders.

Extensive experiments are conducted and the results indicate that the
synthetically generated data and the proposed PM2 model increase both AU
detection performance and fairness across corpora. Specifically, PM2
outperforms all the single-source and multi-source domain adaptation baselines
in two directions of domain adaption (BP4D to DISFA and BP4D to GFT). In
addition, PM2 results in increased fairness according to the equal opportunity
(EO) and statistical parity difference (SPD) between different genders
compared with the baselines.

The major contributions of this work are as follows: (i) We propose a pipeline
to generate a diverse and balanced dataset for fairer machine learning models
through synthetic expression re-targeting. We release the synthetic dataset
(upon acceptance) and hope that this paper motivates others to consider
fairness evaluation and further innovate to address this issue. (ii) We
propose Paired Moment Matching (PM2) which aligns the features of the paired
real and synthetic images with the same facial expression. The proposed model
aligns the data with the same class label and matches the features of the real
data with both female and male synthetic avatars. Therefore, PM2 can maintain
discriminative features for AU detection and obtain a fair representation in
terms of gender. (iii) The experimental results indicate that our generated
data and proposed model improve both cross-domain AU detection performance and
fairness, demonstrating its potential to solve AU detection in a real life
scenario.

##  2 Related Work

In this section, we introduce the background and the previous work of domain
adaptation and facial action unit detection.

###  2.1 Domain Adaptation (DA)

Variations in data distributions between the training and test data often
result in reduced recognition performance. Domain adaptation methods [18, 19,
17] have been proposed to alleviate such problems. In particular, deep domain
adaptation methods explore learning a representation that is both effective
for the main task and invariant to domains [20].

Deep Adaptation Networks (DAN) is proposed to reduce the discrepancies across
source and target distributions by embedding the representations in a
reproducing kernel Hilbert space where the mean embedding of different domains
can be made similar [18]. Domain-Adversarial Neural Network (DANN) [21]
utilizes a gradient reversal layer to ensure that the feature distributions
over the two domains are made similar.  Joint Adaptation Networks (JAN) align
the joint distributions of the input features and output labels in multiple
domain-specific layers based on a joint maximum mean discrepancy [22].
Conditional Domain Adaptation Networks (CDAN) condition the domain adaptation
on discriminative information conveyed in the classifier predictions to enable
alignment of the multimodal distributions [23]. Zhang _et al._[24] propose
Margin Disparity Discrepancy (MDD), which is tailored to the distribution
comparison and the min-max optimization for easier training.  Saito _et
al._[19] introduce the Maximum Classifier Discrepancy (MCD) where two
classifiers are treated as discriminators and are trained to maximize the
discrepancy on target samples, while a feature extractor is trained to
minimize the discrepancy on target samples.

Unlike single-source domain adaptation, multi-source domain adaptation assumes
that training data from multiple sources are available [17]. Ben-David _et
al._[25] introduce an $\mathcal{H}\Delta\mathcal{H}$-divergence between the
weighted combination of source domains and target domain. Deep Cocktail
Network (DCTN) [26] proposes a $k$-way domain discriminator and category
classifier for digit classification and real-world object recognition. Hoffman
_et al._[27] present new normalized solutions with strong theoretical
guarantees for the cross-entropy loss and provide a full solution for the
multi-source domain adaptation problem with practical benefits. Peng _et
al._[17] propose Moment Matching for Multi-Source Domain Adaptation ($\rm
M^{3}SDA$), which directly matches all the distributions by matching their
statistical moments.

Previous methods for MSDA align the overall feature distributions by domain
discriminator or minimizing the moment distance does not consider any class-
specific boundaries which may result in inferior performance. While our method
aligns the paired real and synthetic data with the same expression and thus
maintains the discriminative features for AU detection.

###  2.2 Action Unit (AU) Detection

A facial action unit (AU) is a facial behavior indicative of activation of an
individual or a group of muscles, _e.g._ , cheek raiser (AU 6). AUs were
formalized by Ekman in Facial Action Coding System (FACS) [2].

Zhao _et al._[28] propose Deep Region and Multi-label Learning (DRML). The
model is trained with region learning (RL) and multi-label learning (ML) and
can identify more specific regions for different AUs than conventional patch-
based methods. Shao _et al._[6, 7] propose a deep learning framework named
J$\hat{\text{A}}$A-Net for joint AU detection and face alignment.
J$\hat{\text{A}}$A-Net uses an adaptive attention learning module to refine
the attention map for each AU.

The temporal dependencies across video frames are helpful for AU detection.
Yang _et al._[29] propose to learn a proxy of temporal information from the
input and anchor images by estimating the hypothetical optical flow. FAb-Net
[30] uses facial movements from videos as the supervisory signal to learn a
representation for recognizing facial attributes, through self-supervised
learning. Lu _et al._[31] propose a self-supervised model through learning
temporal consistencies in videos. They use a triplet-based ranking approach
that ranks the frames based on temporal distance from an anchor frame.

Recent work on AU detection use graph neural networks [32, 33, 34]. Zhang _et
al._[32] utilize a heatmap regression-based approach for AU detection. The
ground-truth heatmaps are defined based on the region of interest (ROI) and
graph convolution is used for feature refinement. Song _et al._[33] propose a
hybrid message-passing neural network with performance-driven structures (HMP-
PS). They propose a performance-driven Monte Carlo Markov Chain sampling
method to generate the graph structures. Besides, hybrid message passing is
proposed to combine different types of messages. Luo _et al._[34] propose an
AU relationship modeling approach that learns a unique graph to explicitly
describe the relationship between each pair of AUs of the target facial
display.

Previous studies on AU detection achieve promising within-domain performance.
However, the generalization ability, _i.e._ , cross-domain performance, for AU
detection has not been widely investigated.  In this work, we refer to within-
domain AU detection where the AU detector is trained and tested on the same
datasets; and cross-domain AU detection where the model is trained and tested
on different datasets. Specifically, the training set is the source domain
while the test set is the target domain.

####  2.2.1 Cross-domain AU Detection

Chu _et al._[35] are among the first to study cross-domain AU detection. They
find that individual differences dramatically influence the classifier
generalized to the unseen subjects and propose Selective Transfer Machine
(STM) to personalize a general classifier in an unsupervised manner. Ghosh _et
al._[36] propose a multi-label CNN method that learns a shared representation
between multiple AUs. The experiments show that the method generalizes well to
the other datasets through multi-task learning. Later, Ertugrul _et al._[37,
38] demonstrate that the deep-learning-based AU detectors achieve poor cross-
domain performance due to the variations in the cameras, environments, and
subjects. Tu _et al._[39] propose Identity-Aware Facial Action Unit Detection
(IdenNet). IdenNet is jointly trained by AU detection and face clustering
datasets that contain numerous subjects to improve the model’s generalization
ability. Yin _et al._[40] propose to use domain adaptation and self-supervised
patch localization to improve the cross-corpora performance for AU detection.

Previous studies either utilize unlabeled target data [35, 40] or multi-task
learning with large-scale data [39] to improve cross-domain AU detection. In
this work, we propose to generate and utilize a diverse and balanced synthetic
dataset to improve the generalization ability.

####  2.2.2 Fairness in Face Understanding

The bias in deep-learning-based models can be traced back to the imbalanced
data distributions in the training set, in terms of gender, race, and age.
Previous methods mitigate the bias through either data augmentation [41] or
model training [42, 43, 44].

Iosifidis and Ntoutsi [41] generate instances from the under-represented group
by over-sampling and SMOTE [45]. Wang _et al._[42] propose a domain-
independent classifier for bias mitigation. Hernandez _et al._[43] conduct an
in-depth analysis of performance differences across subjects, genders, skin
types, and databases. They propose deep face normalization (DeepFN) that
transfers the facial expressions of different people onto a common facial
template. Churamani _et al._[44] propose to use Domain-Incremental Learning
(Domain-IL) to enhance the fairness of facial expression recognition and AU
detection.

Our work combines two types of aforementioned methods. We synthetically
generate additional data for data augmentation and explicitly align the
different protected groups during model training.

####  2.2.3 Synthetic Data in Facial Expression Analysis

Synthetic data is widely used in modern computer vision including
segmentation, object detection, and simulation tasks [46, 47, 48, 49]. In
general, many machine learning applications suffer from the data insufficiency
problem, either due to small datasets or costly manual labeling [46]. AU
detection in particular is susceptible to this problem since coding AUs is
highly labor-intensive. It takes over a hundred hours of training to become a
certified FACS coder [9]. Thus, some studies utilize synthetic facial data in
AU detection and face expression analysis [13, 14, 15].

Zhu _et al._[14] propose a data augmentation method using CycleGAN to find
better boundaries between different unbalanced classes for expression
classification. Abbasnejad _et al._[13] train a deep convolutional network on
a generated synthetic dataset and directly fine-tune the network on the real
dataset for facial expressions analysis. Recently, Wood _et al._[50] propose
to render training images with unprecedented realism and diversity by
combining a procedurally-generated parametric 3D face model with a
comprehensive library of hand-crafted assets. They demonstrate that it is
possible to use synthetic data alone to perform face-related computer vision
in-the-wild. Niinuma _et al._[15] train a GAN-based constructor to synthesize
novel images with AU labeling to solve the imbalance in AU labels and
generates a large balanced dataset. The major difference between these
previous studies and ours is as follows. (i) We use facial geometry re-
targeting instead of GAN for synthetic data generation to avoid unstable
training and mode collapse problems. (ii) We take advantage of the expression
re-targeting to formulate paired real-syn data for model training.

Figure 1: Overview of the proposed Paired Moment Matching (PM2). In this work,
the encoder is a Swin Transformer and the classifier is an MLP. PM2 has two
modules: (1) Maximum Classifier Discrepancy (MCD) aligns the source and target
by minimizing the discrepancy between two classifiers on target samples. (2)
PM2 aligns the real and synthetic domains by minimizing the moment distances
between the real-syn pairs. Each pair contains one real image and two
synthetic images generated with a male and a female avatar, sharing the same
facial expression.

##  3 Method

In this section, we first formulate the problem and notations regarding the
facial action unit detection in Section 3.1 and provide an overview of our
proposed method in Section 3.2. Then, we describe the model in Section 3.3. At
last, we explain the training procedure in Section 3.4 and the inference in
Section 3.5.

###  3.1 Problem Formulation

Facial Action Unit Detection. Given a video set $S$, for each frame $x\in S$,
the goal is to detect the occurrence for each AU $y_{i}$ $(i=1,2,...,n)$ using
function $\text{F}(\cdot)$.

$y_{1},y_{2},...,y_{n}=\text{F}(x),$ (1)

where $n$ is the number of AUs to be detected. $y_{i}=1$ if the AU is active
otherwise $y_{i}=0$.

###  3.2 Overview

We propose Paired Moment Matching (PM2) for generalizable and fair facial
action unit detection. We show the model overview in Figure 1. The model
comprises of an encoder E (Swin Transformer [51]) and two classifiers
$\text{C}_{1},\text{C}_{2}$ (multilayer perceptron, MLP). Two classifiers are
leveraged to align the distributions of the source and target domains [19]
(see Section 3.3.2). In the meantime, we apply moment matching to align the
representations of the paired real and synthetic data to further improve the
model’s generalization ability and fairness (see Section 3.3.3).

###  3.3 Model

####  3.3.1 Base Model

We choose Swin Transformer base [51] pre-trained on ImageNet-1K [52] as our
base model. Swin Transformer is a hierarchical Transformer in which the
representation is computed with shifted windows and can be served as a
general-purpose backbone for computer vision [51].

The base model outputs the logits for the AU occurrences through multi-task
learning. The AU loss is calculated by the sum of the Binary Cross-Entropy
(BCE) for every AU, as follows

$\mathcal{L}_{\text{AU}}=\sum_{i=1}^{n}\mathcal{L}_{\text{BCE}}(y_{i},\hat{y}_{i})$
(2)

where $n$ is the number of AU labels to be detected, $y_{i}$ and $\hat{y}_{i}$
are the ground-truth and the prediction for the $i$-th AU.

####  3.3.2 Maximum Classifier Discrepancy (MCD)

To align the source and target domains, MCD aims to detect the target samples
far from the source domain and then push them closer to the source. These
samples are likely to be classified wrongly by the classifier trained on the
source data since they are far from the source [19]. Moreover, Saito _et
al._[19] argue that these target samples outside the source are likely to be
classified differently by the two distinct source classifiers. In practice,
the disagreement between the two classifiers, discrepancy, is measured by the
absolute difference of their output logits. The goal of MCD is to obtain an
encoder that minimizes the discrepancy in the target samples.

$\mathcal{L}_{\text{Dis}}=\sum_{x\in\mathcal{D}_{T}}|p_{1}(x)-p_{2}(x)|$ (3)

where $\mathcal{D}_{T}$ is the target domain, $p_{i}$ is the output logits of
the classifier $\text{C}_{i}$ ($i=1,2$).

Figure 2: (Best viewed in color) Comparison of previous and proposed methods
for domain alignment. Middle: Previous methods align the overall distributions
and corrupt the class boundary. Right: Our method aligns the paired data with
the same facial expression and maintains the class boundary.

####  3.3.3 Paired Moment Matching (PM2)

To align the features of the source $\mathcal{D}_{S}$ and target
$\mathcal{D}_{T}$, Peng _et al._[17] propose $\rm M^{3}SDA$ to align the
moments of their feature distributions. The objective is to minimize the
moment distance

$\displaystyle
d(\mathcal{D}_{S},\mathcal{D}_{T})=\sum_{k=1}^{2}||\mathbb{E}_{x\sim\mathcal{D}_{S}}(\text{E}(x)^{k})-\mathbb{E}_{x\sim\mathcal{D}_{T}}(\text{E}(x)^{k})||_{2}$
(4)

where $k$ represents the $k$-th order moment of the feature (to the power of
$k$), $\text{E}(x)$ is the features of image $x$ extracted from the encoder E.

Though $\rm M^{3}SDA$ aims to match the distributions of different domains and
can reduce the domain gap to some extent, it has three problems. (i)
Completely matching the features of different domains is difficult due to
their different characteristics. In our case, the domain gap between the real
and synthetic data is huge due to the lighting condition, head pose, and
texture. (ii) Matching the overall distributions does not consider any class-
specific boundaries. For example, the model may align the features of real and
synthetic faces with different AUs thus resulting in inferior AU detection
performance (see Figure 2). (iii) More importantly, matching the overall
distributions does not align different protected groups explicitly which may
result in inferior fairness (see Figure 3). In this work, we focus on fairness
across genders.

To address the above problems, instead of completely aligning the overall
distributions of different domains, we propose to align the paired real and
synthetic faces with the same facial expression. Our synthetic data is
generated through facial expression re-targeting, thus, we can easily obtain
the paired data. In this way, the model will avoid aligning the images with
different AU labels which may corrupt the well-learned AU representations. To
further improve gender fairness, for each real face, we generate two synthetic
images with the same facial expression, one with a female avatar and another
one with a male avatar. By aligning the real face to these two synthetic
images, the model can obtain a fair representation in terms of gender.

Figure 3: (Best viewed in color) Comparison of previous and proposed methods
for bias mitigation. Left: The real domain has gender bias thus female and
male distributions are not aligned. While the synthetic (syn.) data is
balanced and the gender distributions are matched. Middle: Though previous
methods match the real and synthetic domains well, they fail to align the
gender distributions. Right: Our method aligns each real image with a male and
a female avatar. The method explicitly aligns the real and synthetic, female
and male domains.

Formally, given an input image $x$, through facial expression re-targeting, we
generate two synthetic images with the same facial expression, $x_{f}$ (with a
female avatar) and $x_{m}$ (with a male avatar). We minimize the moment
distance between the paired real and synthetic data to align the domains and
thus improve the model generalization ability and fairness.

$\displaystyle
d(x_{1},x_{2})=\sum_{k=1}^{2}||(\text{E}(x_{1})^{k})-(\text{E}(x_{2})^{k})||_{2}$
(5)
$\displaystyle\mathcal{L}_{\text{PM2}}=\frac{1}{3}\sum_{x\in\mathcal{D}_{S}}(d(x,x_{f})+d(x,x_{m})+d(x_{f},x_{m}))$
(6)

###  3.4 Training

The overall training procedure for the proposed PM2 includes three steps which
are performed alternately until convergence.

Step 1 We train the encoder E and the classifiers $\text{C}_{1},\text{C}_{2}$
to classify the real source samples correctly and align the real and synthetic
domains in the meantime.

$\mathcal{L}_{1}=\frac{1}{2}(\mathcal{L}_{\text{AU}_{1}}+\mathcal{L}_{\text{AU}_{2}})+\beta\cdot\mathcal{L}_{\text{PM2}}$
(7)

where $\mathcal{L}_{\text{AU}_{i}}$ is the AU loss (Equation 2) for the
classifier $\text{C}_{i}$ on the real source domain ($i=1,2$),
$\mathcal{L}_{\text{PM2}}$ is the moment distance loss (Equation 6) between
the paired real and synthetic data, and $\beta$ is a hyper-parameter.

Step 2 The encoder E is fixed and the classifiers $\text{C}_{1},\text{C}_{2}$
are trained to maximize the discrepancy on the target samples.

$\mathcal{L}_{2}=\frac{1}{2}(\mathcal{L}_{\text{AU}_{1}}+\mathcal{L}_{\text{AU}_{2}})+\beta\cdot\mathcal{L}_{\text{PM2}}-\alpha\cdot\mathcal{L}_{\text{Dis}}$
(8)

where $\mathcal{L}_{\text{Dis}}$ is the discrepancy loss (Equation 3) on the
target domain and $\alpha$ is a hyper-parameter.

Step 3 The classifiers $\text{C}_{1},\text{C}_{2}$ are fixed, and the encoder
E is trained to minimize the discrepancy on the target domain.

$\mathcal{L}_{3}=\alpha\cdot\mathcal{L}_{\text{Dis}}$ (9)

Figure 4: Examples of synthetically generated data through expression re-
targeting. Real images come from BP4D. For each real subject, we generate the
synthetic data with two avatars, one male and one female. The synthetic data
has similar facial expressions as the real images.

###  3.5 Inference

During the testing phase, we calculate the average output logits of the two
classifiers as the final prediction.

$p(x)=\frac{1}{2}(p_{1}(x)+p_{2}(x))$ (10)

where $p_{i}$ is the output logits of the classifier $\text{C}_{i}$ ($i=1,2$).

##  4 Experiments

(a) BP4D.

(b) DISFA.

(c) GFT.

(d) Syn.

Figure 5: Occurrence rates of AU labels for different datasets. BP4D has the
highest occurrence rate on average while DISFA and GFT are severely
imbalanced. Figure 6: Gender distributions for different datasets. All the
real datasets suffer from imbalanced gender distributions while our
synthetically generated data (Syn.) is completely balanced.

In this section, we conduct extensive experiments to show the effectiveness of
our proposed method compared to the existing work. We first introduce the
datasets in Section 4.1. The implementation and training details are given in
Section 4.2. Finally, in Section 4.3, we show the experimental results
including within- and cross-domain evaluations, fairness evaluation, ablation
study, and case study.

###  4.1 Data

In this work, we have two source domains, namely real and synthetic data, and
one target domain for the multi-source domain adaptation. The synthetic data
is generated by facial geometry re-targeting to increase the subject diversity
and fairness.

####  4.1.1 Real Data

Three widely used benchmarks for AU detection are selected, namely,
Binghamton–Pittsburgh 4D Spontaneous Expression Database (BP4D) [10], Denver
Intensity of Spontaneous Facial Actions (DISFA) [11], and Sayette Group
Formation Task Spontaneous Facial Expression Database (GFT) [12].

BP4D consists of videos from 41 subjects with around 146,000 frames. Each
frame comes with AU occurrence labels and 49 landmarks detected by a CLM
method [53]. Each frame has labels for 12 AUs (1, 2, 4, 6, 7, 10, 12, 14, 15,
17, 23, and 24).

DISFA includes videos from 27 subjects, with a total of 130,000 frames. Each
frame is annotated with AU intensities from 0 to 5 and 66 landmarks detected
by an AAM [54]. Each frame has labels for eight AU intensities (1, 2, 4, 6, 9,
12, 25, and 26). Following [28, 55, 6], we map AU intensity greater than 1 to
the positive class.

GFT contains 96 subjects from 32 three-person groups with unscripted social
interaction. Each subject has about 1,800 annotated frames. Video frames are
annotated for ten AUs (1, 2, 4, 6, 10, 12, 14, 15, 23, and 24) and 49
landmarks detected by ZFace [56].

BP4D is recorded under strictly controlled lighting conditions and thus an
“easier” dataset compared to DISFA and GFT, with a higher average reported
performance [6, 57, 7]. Thus, we choose BP4D as the source domain and DISFA
and GFT as the target domain.

####  4.1.2 Synthetic Data

With the facial AU configurations from the real data, _i.e._ , BP4D, we can
generate multiple synthetic avatars with the expression transferred to the
avatars. Specifically, we first detect the facial landmarks and extract the
blendshape parameters by the Faceware
Studio111https://facewaretech.com/software/studio/. Then, we use the
parameters to map the tracked expressions to the synthetic avatars to generate
the same AU configurations. The face attributes and the generated textures are
based on Li _et al._[58]. The rendering is performed using the Unreal
Engine222https://www.unrealengine.com/. We utilize a customized 3D morphable
model (3DMM) to have a controllable subject appearance. The 3D morphable model
has two adjustable components: Control shape and Shader Parameter. Control
shape decomposes 3D face shape attributes like gender, cheek, and jaw by
principal component analysis (PCA), and Shader Parameter allows us to
synthesize various skin colors through different shading on texture. In the
end, we use the Unreal Engine to render the morphed avatars to generate 2D
images with the corresponding facial expressions.

The extracted blendshape is the linear weighted sum of facial muscle actions,
which ensures the facial AU is kept. Moreover, such transfer is aligned at the
frame level. Thus, the correctness of AU transfer is very likely via such
transformations, and we annotate the synthetic data with the corresponding
real labels.

TABLE I: Implementation details. Values of the hyper-parameters for within- and cross-domain evaluations. Hyper-parameter | Within-domain | Cross-domain
---|---|---
# epochs | 30 | 10
learning rate | 1e-4 | 1e-6
batch size | 128 | 32
$\alpha$ for $\mathcal{L}_{\text{Dis}}$ | - | 0.3
$\beta$ for $\mathcal{L}_{\text{PM2}}$ | - | 0.5

We randomly select 30 subjects from the real dataset, _i.e._ , BP4D. For each
real subject, we perform expression re-targeting for all their frames with two
synthetic avatars, one male and one female. Overall, the synthetic dataset
consists of 60 avatars with a total of 214,146 frames.

Figure 4 provides more visual examples for expression re-targeting from the
real data to the synthetic avatars. Specifically, the synthetic data can
transfer the eyelid movement well, but the eyebrow changes are less obvious.
This is due to the limitation of our 3D morphable model capacity. As for the
AUs in the lower face, the lips movements (raiser/puller) of synthetic images
are also consistent with the source for obvious muscle movements. Overall, the
synthetically generated data has similar facial expressions as the real
images.

TABLE II: Within-domain evaluation for the base model (Swin Transformer [51]) on different datasets. F1-score (% $\uparrow$) is the evaluation metric. GFT has a lower average F1-score than BP4D and DISFA. AU | 1 | 2 | 4 | 6 | 7 | 9 | 10 | 12 | 14 | 15 | 17 | 23 | 24 | 25 | 26 | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
BP4D [10] | 50.6 | 50.1 | 63.2 | 77.9 | 80.3 | - | 83.1 | 88.3 | 57.9 | 49.6 | 61.4 | 44.5 | 44.2 | - | - | 62.6
DISFA [11] | 58.6 | 56.5 | 72.5 | 47.6 | - | 51.3 | - | 76.4 | - | - | - | - | - | 95.2 | 65.5 | 65.5
GFT [12] | 41.9 | 57.5 | 3.9 | 73.1 | - | - | 65.0 | 78.8 | 24.1 | 46.0 | - | 44.1 | 53.6 | - | - | 48.8
TABLE III: Cross-domain evaluation with different models in different
directions of domain adaptation. Swin stands for Swin Transformer [51].
$*^{\text{src}}$ means the model is trained with the source data and directly
transferred to the target domain. $*^{\text{tgt}}$ means the model is both
trained and tested with the target data. F1-score (% $\uparrow$) is the
evaluation metric. The best results are shown in orange. The proposed PM2 has
the highest average F1-score, indicating the best generalization ability.

Direction | BP4D $\rightarrow$ DISFA | BP4D $\rightarrow$ GFT | Avg.
---|---|---|---
AU | 1 | 2 | 4 | 6 | 12 | Avg. | 1 | 2 | 4 | 6 | 10 | 12 | 14 | 15 | 23 | 24 | Avg. |
DRML${}^{\text{src}}$ [28] | 9.4 | 4.2 | 23.7 | 18.2 | 27.2 | 16.5 | 7.5 | 18.3 | 5.6 | 55.1 | 48.8 | 56.2 | 5.9 | 1.8 | 14.6 | 14.6 | 22.8 | 19.7
J$\hat{\text{A}}$A-Net${}^{\text{src}}$ [7] | 11.0 | 15.1 | 25.2 | 19.9 | 38.7 | 22.0 | 10.4 | 19.9 | 6.6 | 54.4 | 47.6 | 63.5 | 5.9 | 8.6 | 18.0 | 17.7 | 26.9 | 24.4
ResNet${}^{\text{src}}$ [59] | 16.7 | 13.7 | 30.4 | 20.5 | 27.5 | 21.8 | 8.5 | 3.4 | 10.2 | 48.1 | 36.3 | 45.6 | 6.5 | 7.9 | 0.1 | 23.4 | 19.0 | 20.4
Swin${}^{\text{src}}$ [51] | 21.6 | 18.8 | 46.8 | 32.9 | 50.7 | 34.2 | 15.9 | 35.5 | 12.4 | 54.2 | 52.0 | 68.9 | 7.9 | 2.5 | 13.9 | 26.8 | 29.0 | 31.6
DAN [18] | 26.0 | 23.3 | 47.5 | 33.8 | 46.0 | 35.3 | 19.6 | 39.6 | 8.7 | 59.5 | 62.8 | 69.2 | 10.1 | 26.6 | 35.8 | 38.1 | 37.0 | 36.2
DANN [21] | 27.4 | 28.9 | 42.2 | 39.8 | 52.8 | 38.2 | 24.6 | 28.8 | 7.9 | 67.6 | 68.5 | 74.2 | 11.5 | 25.8 | 32.4 | 38.3 | 38.0 | 38.1
JAN [22] | 26.7 | 26.1 | 51.9 | 32.1 | 42.7 | 35.9 | 27.3 | 41.3 | 8.6 | 65.7 | 64.5 | 69.7 | 11.9 | 23.1 | 34.7 | 40.4 | 38.7 | 37.3
CDAN [23] | 28.0 | 23.0 | 39.7 | 37.0 | 51.2 | 35.8 | 30.1 | 25.6 | 8.2 | 68.9 | 68.8 | 75.1 | 11.5 | 25.3 | 30.1 | 39.6 | 38.3 | 37.1
MCD [19] | 31.4 | 28.7 | 44.8 | 39.1 | 51.3 | 39.1 | 28.4 | 34.3 | 10.1 | 67.0 | 70.3 | 75.2 | 12.8 | 14.0 | 16.2 | 40.8 | 36.9 | 38.0
$\rm M^{3}SDA$ [17] | 43.3 | 26.3 | 49.8 | 40.9 | 50.5 | 42.1 | 27.8 | 38.4 | 8.8 | 68.9 | 68.7 | 73.9 | 11.8 | 17.9 | 30.3 | 40.7 | 38.7 | 40.4
PM2 (Ours) | 46.0 | 27.6 | 49.8 | 35.6 | 54.6 | 42.7 | 29.0 | 32.0 | 9.0 | 67.9 | 70.2 | 74.2 | 11.4 | 18.4 | 37.1 | 44.1 | 39.3 | 41.0
Swin${}^{\text{tgt}}$ [51] | 58.6 | 56.5 | 72.5 | 47.6 | 76.4 | 62.3 | 41.9 | 57.5 | 3.9 | 73.1 | 65.0 | 78.8 | 24.1 | 46.0 | 44.1 | 53.6 | 48.8 | 55.6

####  4.1.3 Data Observation

To statistically evaluate the fairness and diversity of subjects for the
aforementioned datasets, we report the distributions of AU labels and genders.

Label Distribution. The occurrence rates of AU labels for different datasets
are given in Figure 5. We observe that many of the AUs occur infrequently.
Specifically, BP4D has the highest occurrence rate on average. DISFA and GFT
are severely imbalanced as some AUs occur in less than 15% of frames (AU 1, 2,
6, 9, 12, 26 in DISFA, and AU 1, 12, 14, 24 in GFT). The AU occurrence rate
for the synthetic data is similar to BP4D as it originates from the dataset.

Gender Distribution. Figure 6 shows the gender distributions. All three real
datasets are imbalanced while our synthetic data is generated to have an equal
number of apparent female/male subjects.

###  4.2 Implementation Details

We use a machine with two Intel(R) Xeon(R) Gold 5218 CPUs @ 2.30GHz with eight
NVIDIA Quadro RTX8000 GPUs for all the experiments.

All methods are implemented in Python 3.9.13 and PyTorch [60] 1.10.1. The
related packages and versions are: torchvision 0.11.2, numpy 1.23.1, scikit-
learn 1.1.2, scipy 1.9.1, pillow 9.2.0, pandas 1.4.4, and opencv-python
4.6.0.66. Code and model weights are available, for the sake of
reproducibility333htttp://-url-to-appear-upon-acceptance.

All frames are cropped and aligned using the facial landmarks detected by dlib
[61]. Each image is first resized into $256\times 256$. We apply random
horizontal flip and random cropping during training for data augmentation.

All models are trained with an AdamW optimizer [62]. Table I provides the
values per (hyper-)parameter. Thresholds for AU detection for all AUs are set
to 0.5, for all experiments. This means that if the output logit is greater
than $0.5$, then the prediction is active otherwise inactive.

###  4.3 Experimental Results

We perform both within- and cross-domain experiments. Our main focus is to
improve AU detection performance across corpora indicative of the
generalization ability of the model and its fairness (across genders).

TABLE IV: Fairness evaluation between different gender groups according to the
equal opportunity (EO % $\uparrow$). Swin stands for Swin Transformer [51].
The best results are shown in orange. The proposed PM2 has the highest equal
opportunity, indicating the best fairness.

Direction | BP4D $\rightarrow$ DISFA | BP4D $\rightarrow$ GFT | Avg.
---|---|---|---
AU | 1 | 2 | 4 | 6 | 12 | Avg. | 1 | 2 | 4 | 6 | 10 | 12 | 14 | 15 | 23 | 24 | Avg. |
Swin${}^{\text{src}}$ [51] | 69.3 | 32.3 | 74.5 | 97.2 | 89.4 | 72.6 | 80.3 | 96.0 | 48.9 | 99.2 | 72.1 | 88.3 | 48.1 | 26.6 | 90.8 | 92.3 | 74.2 | 73.4
DAN [18] | 62.6 | 48.3 | 90.9 | 74.6 | 95.1 | 74.3 | 64.7 | 99.1 | 56.4 | 88.7 | 84.5 | 88.8 | 65.9 | 46.3 | 87.6 | 97.9 | 78.0 | 76.2
DANN [21] | 65.9 | 44.3 | 88.2 | 72.4 | 98.4 | 73.8 | 64.1 | 97.8 | 98.8 | 85.7 | 89.8 | 94.8 | 63.6 | 52.6 | 63.3 | 93.1 | 80.4 | 77.1
JAN [22] | 58.4 | 39.3 | 99.8 | 77.6 | 84.4 | 71.9 | 77.6 | 91.4 | 46.3 | 82.3 | 82.0 | 84.1 | 61.4 | 45.4 | 79.5 | 92.7 | 74.3 | 73.1
CDAN [23] | 66.7 | 50.7 | 81.4 | 76.3 | 99.3 | 74.9 | 74.4 | 90.8 | 85.3 | 84.6 | 90.6 | 94.2 | 65.0 | 53.9 | 67.1 | 94.5 | 80.0 | 77.5
MCD [19] | 73.9 | 45.3 | 93.0 | 72.1 | 96.7 | 76.2 | 58.5 | 87.4 | 72.1 | 90.2 | 92.2 | 96.3 | 49.0 | 77.5 | 77.6 | 91.2 | 79.2 | 77.7
$\rm M^{3}SDA$ [17] | 98.3 | 39.1 | 70.7 | 76.9 | 97.2 | 76.4 | 67.2 | 85.9 | 41.5 | 86.9 | 90.1 | 95.3 | 55.2 | 52.0 | 97.9 | 90.1 | 76.2 | 76.3
PM2 (Ours) | 61.0 | 78.7 | 83.9 | 73.7 | 99.9 | 79.5 | 77.1 | 71.3 | 87.9 | 86.1 | 94.2 | 93.3 | 55.2 | 52.4 | 96.3 | 93.6 | 80.7 | 80.1

TABLE V: Fairness evaluation between different gender groups in terms of the
statistical parity difference (SPD % $\downarrow$). Swin stands for Swin
Transformer [51]. The best results are shown in orange. The proposed PM2 has
the lowest SPD, indicating the best fairness.

Direction | BP4D $\rightarrow$ DISFA | BP4D $\rightarrow$ GFT | Avg.
---|---|---|---
AU | 1 | 2 | 4 | 6 | 12 | Avg. | 1 | 2 | 4 | 6 | 10 | 12 | 14 | 15 | 23 | 24 | Avg. |
Swin${}^{\text{src}}$ [51] | 7.5 | 18.5 | 8.0 | 22.1 | 1.0 | 11.4 | 8.7 | 9.1 | 11.5 | 1.8 | 4.7 | 13.1 | 0.2 | 0.2 | 2.8 | 7.0 | 5.9 | 8.7
DAN [18] | 1.1 | 3.7 | 5.0 | 9.3 | 10.1 | 5.8 | 8.1 | 7.2 | 3.1 | 2.5 | 1.2 | 0.8 | 2.4 | 0.9 | 3.1 | 4.9 | 3.4 | 4.6
DANN [21] | 0.6 | 0.4 | 4.7 | 4.6 | 6.3 | 3.3 | 4.9 | 3.8 | 5.5 | 0.4 | 2.1 | 4.3 | 0.3 | 0.6 | 2.4 | 5.0 | 2.9 | 3.1
JAN [22] | 1.2 | 6.5 | 5.5 | 13.0 | 14.7 | 8.2 | 4.0 | 5.1 | 3.7 | 0.1 | 0.0 | 2.1 | 0.1 | 0.9 | 0.7 | 3.0 | 2.0 | 5.1
CDAN [23] | 0.5 | 0.2 | 5.4 | 8.4 | 6.4 | 4.2 | 4.1 | 2.9 | 5.1 | 1.1 | 1.8 | 5.1 | 1.0 | 0.0 | 0.4 | 5.1 | 2.7 | 3.5
MCD [19] | 3.0 | 0.2 | 4.9 | 8.4 | 6.6 | 4.6 | 2.9 | 3.1 | 6.4 | 0.2 | 3.3 | 5.4 | 3.8 | 1.2 | 1.9 | 3.1 | 3.1 | 3.9
$\rm M^{3}SDA$ [17] | 7.0 | 2.7 | 5.2 | 3.1 | 9.4 | 5.5 | 4.6 | 2.9 | 5.6 | 3.1 | 3.9 | 3.7 | 0.4 | 1.3 | 1.3 | 2.4 | 2.9 | 4.2
PM2 (Ours) | 1.0 | 0.2 | 1.7 | 2.8 | 3.3 | 1.8 | 0.7 | 0.6 | 5.0 | 1.0 | 0.4 | 1.6 | 0.2 | 0.1 | 2.1 | 1.7 | 1.3 | 1.6

####  4.3.1 Within-domain Evaluation

In terms of BP4D, DISFA, and the synthetic dataset, the base model is
evaluated using subject-independent three-fold cross-validation, with two
folds for training and one fold for validation, similar to [59, 7]. For GFT,
following the original training/validation/test partitions [12], we use 60,
18, and 18 subjects for training, validation, and test respectively. F1-score
is the evaluation metric.

Table II shows the within-domain results for the base model on the three real
datasets. From the table, we observe that GFT has a lower average F1-score
than BP4D and DISFA. We suspect it is because BP4D and DISFA have better video
quality than GFT.

####  4.3.2 Cross-domain Evaluation

We perform two directions of unsupervised domain adaptation. We set BP4D as
the real source domain and DISFA/GFT as the target domain. Specifically, we
employ all the labeled source data and all the unlabeled target data for
training, and all the target data for testing. For the multi-source domain
adaptation baseline and our own method, we also train the models with the
synthetic data. When DISFA is the target domain, we evaluate the five common
AU labels (1, 2, 4, 6, 12), and we evaluate the ten common AUs (1, 2, 4, 6,
10, 12, 14, 15, 23, 24) when GFT is the target domain. F1-score ($\uparrow$)
is the evaluation metric. The results of the cross-domain evaluations are
shown in Table III.

We implement four base models, _i.e._ , DRML [28], J$\hat{\text{A}}$A-Net [7],
ResNet-18 [59], and Swin Transformer base [51] and report their direct
transfer performance. In Table III, we observe that Swin Transformer has the
highest cross-domain performance in both directions of domain adaptation. We
suspect it is due to the following reasons. (i) Compared with DRML and
J$\hat{\text{A}}$A-Net which are trained from scratch, Swin Transformer is
first pre-trained on ImageNet-1K [52] and thus has better generalization
ability. (ii) Compared with ResNet, Swin Transformer has more training
parameters (121M vs 11M), thus Swin Transformer has a higher capacity.

We also report the supervised performance of the Swin Transformer on the
target domain. Compared to the performance of direct transfer, we observe a
large performance drop in both directions of domain adaptation (24.0% F1-score
decrease on average). This demonstrates the challenging nature of cross-domain
AU detection and the importance of developing generalizable AU detection.

For the rest of the experiments, we employ Swin Transformer as the base model
for domain adaptation. We implement five single-source domain adaptation
methods and one multi-source domain adaptation model as the baselines, _i.e._
, DAN [18], DANN [21], JAN [22], CDAN [23], MCD [19], and $\rm M^{3}SDA$ [17].

We observe that domain adaptation can mitigate the performance drop to some
extent. All the domain adaptation baselines improve the cross-domain
performance in both directions. In particular, MCD [19] and DANN [21] are the
best two single-source domain adaption models while DAN [18] achieves the
worst performance in terms of the average F1-score. $\rm M^{3}SDA$ [17] and
our own method are built upon MCD but utilize the synthetic data for multi-
source domain adaptation. Compared with MCD, these two models can further
improve the cross-domain performance in both directions, demonstrating the
effectiveness of our synthetically generated data. The proposed PM2 achieves
the highest average F1-score, indicating the best generalization ability. The
major improvement of PM2 over other baselines comes from AU 12 (lip corner
puller) from BP4D to DISFA and AU 23 (lip tightener), AU 24 (lip pressor) from
BP4D to GFT. We suspect it is because our synthetic data are more consistent
with the original source in terms of lip movements than the movements in the
upper face. Future work should introduce more rigs around eyebrows to better
mimic the animation. Though the improvement of PM2 over $\rm M^{3}SDA$ is not
significant, PM2 has better fairness across genders (see the next section).

TABLE VI: Ablation study for PM2 with different loss weights. F1-score (%
$\uparrow$) is the evaluation metric. The best results are shown in orange. DT
stands for direct transfer. PM2 achieves the best performance with joint
training of $\mathcal{L}_{\text{Dis}}$ and $\mathcal{L}_{\text{PM2}}$.

| Direction | BP4D $\rightarrow$ DISFA | BP4D $\rightarrow$ GFT | Avg.
---|---|---|---|---
Method | $\alpha$ | $\beta$ | 1 | 2 | 4 | 6 | 12 | Avg. | 1 | 2 | 4 | 6 | 10 | 12 | 14 | 15 | 23 | 24 | Avg. |
DT | 0 | 0 | 21.6 | 18.8 | 46.8 | 32.9 | 50.7 | 34.2 | 15.9 | 35.5 | 12.4 | 54.2 | 52.0 | 68.9 | 7.9 | 2.5 | 13.9 | 26.8 | 29.0 | 31.6
$\mathcal{L}_{\text{Dis}}$ only | 1 | 0 | 31.4 | 28.7 | 44.8 | 39.1 | 51.3 | 39.1 | 28.7 | 30.3 | 10.6 | 63.2 | 69.4 | 72.2 | 12.6 | 9.4 | 11.5 | 30.1 | 33.8 | 36.5
0.3 | 0 | 31.6 | 25.6 | 45.3 | 35.0 | 49.0 | 37.3 | 28.4 | 34.3 | 10.1 | 67.0 | 70.3 | 75.2 | 12.8 | 14.0 | 16.2 | 40.8 | 36.9 | 37.1
$\mathcal{L}_{\text{PM2}}$ only | 0 | 1 | 31.6 | 29.7 | 42.0 | 31.7 | 50.5 | 37.1 | 21.4 | 10.6 | 7.9 | 67.9 | 70.9 | 75.3 | 10.4 | 16.7 | 30.0 | 39.8 | 35.1 | 36.1
0 | 0.5 | 32.8 | 28.1 | 47.1 | 35.0 | 53.1 | 39.2 | 22.3 | 13.0 | 8.0 | 67.1 | 70.3 | 74.6 | 10.3 | 17.0 | 29.2 | 38.5 | 35.0 | 37.1
PM2 | 1 | 1 | 29.6 | 23.2 | 50.0 | 42.7 | 57.8 | 40.7 | 26.1 | 15.0 | 8.3 | 67.4 | 69.6 | 74.7 | 10.8 | 12.9 | 26.0 | 37.9 | 34.8 | 37.8
1 | 0.5 | 27.5 | 29.4 | 45.3 | 38.8 | 58.4 | 39.9 | 25.4 | 23.4 | 9.3 | 61.2 | 65.4 | 71.0 | 11.5 | 16.1 | 31.8 | 37.1 | 35.2 | 37.6
0.3 | 1 | 37.1 | 29.2 | 46.9 | 37.3 | 55.1 | 41.1 | 26.0 | 18.7 | 8.1 | 68.1 | 70.5 | 75.5 | 10.4 | 19.2 | 32.7 | 40.9 | 37.0 | 39.1
0.3 | 0.5 | 46.0 | 27.6 | 49.8 | 35.6 | 54.6 | 42.7 | 29.0 | 32.0 | 9.0 | 67.9 | 70.2 | 74.2 | 11.4 | 18.4 | 37.1 | 44.1 | 39.3 | 41.0

Image |  |  |  |  |  |
---|---|---|---|---|---|---

Active AU |  Inner Brow Raiser |  Inner Brow Raiser |  |  Lip Pressor |  Lip Pressor |
---|---|---|---|---|---|---
Gender |  Female |  Male |  |  Female |  Male |
Output logit |  |  |  Diff. |  |  |  Diff.
$\rm M^{3}SDA$ [17] |  0.49 () |  0.44 () |  0.05 |  0.59 () |  0.47 () |  0.12
PM2 (Ours) |  0.59 () |  0.60 () |  0.01 |  0.73 () |  0.75 () |  0.02

Figure 7: Case analysis on $\rm M^{3}SDA$ [17] and our method. The output
logits for the AU occurrences are reported. Logit greater than $0.5$ means the
predicted AU is active. Lower logit difference (Diff.) indicate better
fairness. Our method achieves better AU detection performance and gender
fairness.

####  4.3.3 Fairness Evaluation

Two metrics are applied to evaluate the fairness of different models, _i.e._ ,
equal opportunity (EO $\uparrow$) [63, 44] and statistical parity difference
(SPD $\downarrow$) [64] across genders (female and male). We report and
compare the performances of the base model and all the domain adaptation
baselines with our own method. The results are shown in Table IV for EO and
Table V for SPD.

Equal opportunity suggests that given a sensitive attribute, the classifier’s
performance should be equal for all groups of that attribute [63]. In
practice, Churamani _et al._[44] calculate the metric through the ratio of the
lowest performance for a sensitive attribute with respect to the highest
performance for that sensitive attribute.

$\text{EO}_{i}=\frac{\min(\text{f}(\hat{y}_{i},y_{i}|a=0),\text{f}(\hat{y}_{i},y_{i}|a=1))}{\max(\text{f}(\hat{y}_{i},y_{i}|a=0),\text{f}(\hat{y}_{i},y_{i}|a=1))}$
(11)

where $\text{f}(\hat{y}_{i},y_{i}|a)$ is the F1-score for the $i$-th AU with
the sensitive attribute $a$, $a=1$ means the subject belongs to the minority
group otherwise the majority group.

From Table IV, we observe that the direct transfer (Swin${}^{\text{src}}$)
achieves the lowest EO, indicating the worst fairness. Compared with direct
transfer, the domain adaptation baselines can improve fairness except for JAN.
MCD obtains the highest average EO.

Statistical parity recognizes a predictor to be unbiased if the prediction
$\hat{y}$ is independent of the protected attribute $a$ so that
$\text{P}(\hat{y}|a)=\text{P}(\hat{y})$ [64]. Here, the same proportion of
each population is classified as positive. Deviations from statistical parity
are measured by the statistical parity difference (SPD)

$\text{SPD}_{i}=|\text{P}(\hat{y}_{i}=1|a=1)-\text{P}(\hat{y}_{i}=1|a=0)|$
(12)

where $\text{P}(\hat{y}_{i}=1|a)$ means the predicted occurrence rate for the
$i$-th AU with the sensitive attribute $a$, $a=1$ means the subject belongs to
the minority group otherwise the majority group.

Table V shows the SPD results. Similar to EO, we observe that the direct
transfer (Swin${}^{\text{src}}$) achieves the highest SPD, indicating the
worst fairness while the domain adaptation baselines outperform the direct
transfer.

Compared with MCD, $\rm M^{3}SDA$ utilizes the balanced synthetic data for
training. $\rm M^{3}SDA$ improves the AU detection performance, however, it
becomes more biased in terms of gender fairness for both EO and SPD. We
speculate that the model matches the overall distributions and does not
explicitly align the female and male domains.

Compared with $\rm M^{3}SDA$, our method aims to match the features of the
real face with both female and male avatars. Such explicit domain alignment
helps PM2 to obtain a fair representation in terms of gender. For both EO and
SPD evaluations, the proposed PM2 achieves superior fairness than the other
baselines, demonstrating the effectiveness of training with diverse synthetic
data and paired moment matching.

####  4.3.4 Ablation Study

We investigate the individual contributions of different components of PM2. We
report these results in Table VI. Specifically, we evaluate PM2 with different
loss weights, _i.e._ , $\alpha$ for $\mathcal{L}_{\text{Dis}}$ and $\beta$ for
$\mathcal{L}_{\text{PM2}}$. In particular, $\mathcal{L}_{\text{Dis}}$ helps
PM2 to align the source and target domains while $\mathcal{L}_{\text{PM2}}$
matches the distributions of the real and synthetic data.

When both $\alpha$ and $\beta$ are $0$, PM2 is reduced to the direct transfer,
achieving the lowest performance in both directions of domain adaptation.
Compared with the direct transfer, adding $\mathcal{L}_{\text{Dis}}$ or
$\mathcal{L}_{\text{PM2}}$ to the objective improves the overall performance.
In addition, from BP4D to GFT, compared with $\mathcal{L}_{\text{Dis}}$ only,
$\mathcal{L}_{\text{PM2}}$ only achieves superior performance for AU 23 (lip
tightener) but inferior results for AU 1 and AU 2 (inner brow raiser and outer
brow raiser). We suspect it is because the eyebrow behaviors are not well
transferred from the real faces onto avatars.

Finally, combining both loss terms obtains the best performance, demonstrating
the effectiveness of joint training with source, target, and synthetic data.
In particular, we have consistent findings that the model achieves higher
performance when $\alpha=0.3$ and $\beta=0.5$. This could be due to the trade-
off between AU detection and domain alignment. The discriminative features for
AU detection may be corrupted if the model puts too much emphasis on domain
alignment.

####  4.3.5 Case Study

Figure 7 shows the comparison between the output logits of $\rm M^{3}SDA$ and
our model on two pairs of samples. Each pair has two images. The first pair
comes from DISFA while the second one is from GFT. The first pair has active
AU 1 (inner brow raiser) while the second one has active AU 24 (lip pressor).
When the output logit is greater than $0.5$, the model prediction is active
otherwise inactive.

We observe that $\rm M^{3}SDA$ fails in three of the cases while our method
accurately predicts the AUs, indicating that our method has a strong
generalization ability. Moreover, our method achieves a lower logit difference
than $\rm M^{3}SDA$. The results show that our method is fairer across
genders.

##  5 Discussions

Synthetic facial expression re-targeting allows us to generate a diverse and
balanced dataset using avatars. However, the current data generation pipeline
is a multi-stage process. First, the facial landmarks and blendshape are
extracted by the Faceware Studio from real images using a graphical interface.
Then, we apply extracted blenshape to the avatars to transfer the expression.
Finally, we utilize a 3D morphabale model to change the physical identities of
the avatar. Due to the lack of the software API interface, all three steps are
not scripted in an end-to-end fashion and are operated manually, resulting in
cases of frame drop and requiring extensive manual work. In addition, as
mentioned in Section 4.1.2, eyebrow behaviors are not well transferred from
real human images to avatars, this is due to the limitation of our 3D
morphable model capacity. For future work, the entire process could be
automated for larger scope usage, and more rigs should be introduced around
eyebrows to better mimic the animation.

In this work, we only focus on gender fairness. We would like to generate more
paired synthetic data for other attributes, _e.g._ , race, and age in future
work.

On the model level, due to the difficulty of cross-corpus facial AU detection,
there is still a huge gap between the source and target domain performance.
Though the proposed PM2 improves fairness, the improvement for the cross-
domain evaluation is not significant. Future work should consider addressing
these problems in the method design, optimization, and choice of the loss
function.

##  6 Conclusions

In this paper, we leverage synthetic data to build a more generalizable facial
AU detection model. We find that gender biases exist in widely used AU
detection datasets resulting in reduced fairness of AU detection. To address
these problems, we propose to generate a diverse and balanced dataset through
synthetic facial expression re-targeting. We propose Paired Moment Matching
(PM2) to align the features of the paired real and synthetic data with the
same facial expression. Instead of completely matching the overall
distributions of different domains, the proposed model aims to align the data
with the same class label and match the features of the real face with both
female and male synthetic avatars. Therefore, PM2 can keep the well-learned AU
features during the domain alignment and obtain a fair representation (across
genders). Extensive experiments are conducted whose results indicate that the
synthetically generated data and the proposed PM2 model increase both AU
detection performance and fairness across corpora, demonstrating its potential
to solve AU detection in a real-life scenario.

For our future work, we would like to generate a larger-scale synthetic
dataset across different subject attributes and appearance, _e.g._ , race and
age.

## Acknowledgments

Research was sponsored by the Army Research Office and was accomplished under
Cooperative Agreement Number W911NF-20-2-0053. The views and conclusions
contained in this document are those of the authors and should not be
interpreted as representing the official policies, either expressed or
implied, of the Army Research Office or the U.S. Government. The U.S.
Government is authorized to reproduce and distribute reprints for Government
purposes notwithstanding any copyright notation herein.

## References

  * [1] M. Rychlowska, R. E. Jack, O. G. Garrod, P. G. Schyns, J. D. Martin, and P. M. Niedenthal, “Functional smiles: Tools for love, sympathy, and war,” Psychological science, vol. 28, no. 9, pp. 1259–1270, 2017.
  * [2] P. Ekman, “Facial action coding system,” 1977.
  * [3] L. F. Barrett, R. Adolphs, S. Marsella, A. M. Martinez, and S. D. Pollak, “Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements,” Psychological Science in the Public Interest, vol. 20, no. 1, pp. 1–68, 2019.  PMID: 31313636.
  * [4] A. M. Yoder, S. C. Widen, and J. A. Russell, “The word disgust may refer to more than one emotion,” Emotion, vol. 16, pp. 301–308, 2016.  Place: US Publisher: American Psychological Association.
  * [5] B. Martinez, M. F. Valstar, B. Jiang, and M. Pantic, “Automatic analysis of facial actions: A survey,” IEEE transactions on affective computing, 2017\.
  * [6] Z. Shao, Z. Liu, J. Cai, and L. Ma, “Deep adaptive attention for joint facial action unit detection and face alignment,” in European Conference on Computer Vision (ECCV), pp. 705–720, 2018.
  * [7] Z. Shao, Z. Liu, J. Cai, and L. Ma, “Jâa-net: Joint facial action unit detection and face alignment via adaptive attention,” International Journal of Computer Vision, 2020.
  * [8] I. Ö. Ertugrul, L. A. Jeni, and J. F. Cohn, “Pattnet: Patch-attentive deep network for action unit detection.,” in BMVC, p. 114, 2019.
  * [9] M. S. Bartlett, J. C. Hager, P. Ekman, and T. J. Sejnowski, “Measuring facial expressions by computer image analysis,” Psychophysiology, vol. 36, no. 2, pp. 253–263, 1999.
  * [10] X. Zhang, L. Yin, J. F. Cohn, S. Canavan, M. Reale, A. Horowitz, P. Liu, and J. M. Girard, “Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database,” Image and Vision Computing, vol. 32, no. 10, pp. 692–706, 2014.
  * [11] S. M. Mavadati, M. H. Mahoor, K. Bartlett, P. Trinh, and J. F. Cohn, “Disfa: A spontaneous facial action intensity database,” IEEE Transactions on Affective Computing, vol. 4, no. 2, pp. 151–160, 2013.
  * [12] J. M. Girard, W.-S. Chu, L. A. Jeni, and J. F. Cohn, “Sayette group formation task (gft) spontaneous facial expression database,” in IEEE Int’l Conf on Automatic Face & Gesture Recognition (FG 2017), pp. 581–588, IEEE, 2017\.
  * [13] I. Abbasnejad, S. Sridharan, D. Nguyen, S. Denman, C. Fookes, and S. Lucey, “Using synthetic data to improve facial expression analysis with 3d convolutional networks,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1609–1618, 2017.
  * [14] X. Zhu, Y. Liu, J. Li, T. Wan, and Z. Qin, “Emotion classification with data augmentation using generative adversarial networks,” in Pacific-Asia conference on knowledge discovery and data mining, pp. 349–360, Springer, 2018\.
  * [15] K. Niinuma, L. A. Jeni, I. Ö. Ertugrul, and J. F. Cohn, “Unmasking the devil in the details: What works for deep facial action coding?,” in BMVC: proceedings of the British Machine Vision Conference. British Machine Vision Conference, vol. 2019, NIH Public Access, 2019.
  * [16] S. Zhao, B. Li, P. Xu, and K. Keutzer, “Multi-source domain adaptation in the deep learning era: A systematic survey,” arXiv preprint arXiv:2002.12169, 2020.
  * [17] X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, “Moment matching for multi-source domain adaptation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1406–1415, 2019.
  * [18] M. Long, Y. Cao, J. Wang, and M. I. Jordan, “Learning transferable features with deep adaptation networks,” in Proceedings of the 32nd International Conference on International Conference on Machine Learning-Volume 37, pp. 97–105, JMLR. org, 2015.
  * [19] K. Saito, K. Watanabe, Y. Ushiku, and T. Harada, “Maximum classifier discrepancy for unsupervised domain adaptation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3723–3732, 2018\.
  * [20] M. Wang and W. Deng, “Deep Visual Domain Adaptation: A Survey,” Neurocomputing, vol. 312, pp. 135–153, 2018.
  * [21] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in Proceedings of the 32nd International Conference on Machine Learning (F. Bach and D. Blei, eds.), vol. 37 of Proceedings of Machine Learning Research, (Lille, France), pp. 1180–1189, PMLR, 07–09 Jul 2015.
  * [22] M. Long, H. Zhu, J. Wang, and M. I. Jordan, “Deep transfer learning with joint adaptation networks,” in International conference on machine learning, pp. 2208–2217, PMLR, 2017.
  * [23] M. Long, Z. Cao, J. Wang, and M. I. Jordan, “Conditional adversarial domain adaptation,” in Advances in Neural Information Processing Systems, pp. 1640–1650, 2018.
  * [24] Y. Zhang, T. Liu, M. Long, and M. Jordan, “Bridging theory and algorithm for domain adaptation,” in International Conference on Machine Learning, pp. 7404–7413, PMLR, 2019.
  * [25] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan, “A theory of learning from different domains,” Machine learning, vol. 79, no. 1, pp. 151–175, 2010.
  * [26] R. Xu, Z. Chen, W. Zuo, J. Yan, and L. Lin, “Deep cocktail network: Multi-source unsupervised domain adaptation with category shift,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3964–3973, 2018.
  * [27] J. Hoffman, M. Mohri, and N. Zhang, “Algorithms and theory for multiple-source adaptation,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 8256–8266, 2018.
  * [28] K. Zhao, W.-S. Chu, and H. Zhang, “Deep region and multi-label learning for facial action unit detection,” in CVPR, pp. 3391–3399, 2016.
  * [29] H. Yang and L. Yin, “Learning temporal information from a single image for au detection,” in 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–8, IEEE, 2019.
  * [30] O. Wiles, A. S. Koepke, and A. Zisserman, “Self-supervised learning of a facial attribute embedding from video,” in BMVC, 2018.
  * [31] L. T. Liupei Lu and M. Soleymani, “Self-supervised learning for facial action unit recognition through temporal consistency,” in British Machine Vision Conference (BMVC), 2020.
  * [32] Z. Zhang, T. Wang, and L. Yin, “Region of interest based graph convolution: A heatmap regression approach for action unit detection,” in Proceedings of the 28th ACM International Conference on Multimedia, pp. 2890–2898, 2020\.
  * [33] T. Song, Z. Cui, W. Zheng, and Q. Ji, “Hybrid message passing with performance-driven structures for facial action unit detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6267–6276, 2021.
  * [34] C. Luo, S. Song, W. Xie, L. Shen, and H. Gunes, “Learning multi-dimensional edge feature-based au relation graph for facial action unit recognition,” arXiv preprint arXiv:2205.01782, 2022.
  * [35] W.-S. Chu, F. De la Torre, and J. F. Cohn, “Selective transfer machine for personalized facial action unit detection,” in CVPR, pp. 3515–3522, 2013\.
  * [36] S. Ghosh, E. Laksana, S. Scherer, and L.-P. Morency, “A multi-label convolutional neural network approach to cross-domain action unit detection,” in 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 609–615, IEEE, 2015.
  * [37] I. Ö. Ertugrul, J. F. Cohn, L. A. Jeni, Z. Zhang, L. Yin, and Q. Ji, “Cross-domain au detection: Domains, learning approaches, and measures,” in IEEE Int’l Conference on Automatic Face & Gesture Recognition (FG), pp. 1–8, IEEE, 2019.
  * [38] I. Ö. Ertugrul, J. F. Cohn, L. A. Jeni, Z. Zhang, L. Yin, and Q. Ji, “Crossing domains for au coding: Perspectives, approaches, and measures,” IEEE Trans. Biometrics, Behavior, and Identity Science, vol. 2, no. 2, pp. 158–171, 2020.
  * [39] C.-H. Tu, C.-Y. Yang, and J. Y.-j. Hsu, “Idennet: Identity-aware facial action unit detection,” in 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–8, IEEE, 2019.
  * [40] Y. Yin, L. Lu, Y. Wu, and M. Soleymani, “Self-supervised patch localization for cross-domain facial action unit detection,” in 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pp. 1–8, IEEE, 2021.
  * [41] V. Iosifidis and E. Ntoutsi, “Dealing with bias via data augmentation in supervised learning scenarios,” Jo Bates Paul D. Clough Robert Jäschke, vol. 24, p. 11, 2018.
  * [42] Z. Wang, K. Qinami, I. C. Karakozis, K. Genova, P. Nair, K. Hata, and O. Russakovsky, “Towards fairness in visual recognition: Effective strategies for bias mitigation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8919–8928, 2020.
  * [43] J. Hernandez, D. McDuff, A. Fung, M. Czerwinski, et al., “Deepfn: towards generalizable facial action unit recognition with deep face normalization,” arXiv preprint arXiv:2103.02484, 2021.
  * [44] N. Churamani, O. Kara, and H. Gunes, “Domain-incremental continual learning for mitigating bias in facial expression and action unit recognition,” IEEE Transactions on Affective Computing, 2022.
  * [45] A. Fernández, S. Garcia, F. Herrera, and N. V. Chawla, “Smote for learning from imbalanced data: progress and challenges, marking the 15-year anniversary,” Journal of artificial intelligence research, vol. 61, pp. 863–905, 2018.
  * [46] S. Nikolenko, “Synthetic data in deep learning,” in School-conference “Approximation and Data Analysis 2019”, p. 21, 2019.
  * [47] S. Zhao, B. Li, X. Yue, Y. Gu, P. Xu, R. Hu, H. Chai, and K. Keutzer, “Multi-source domain adaptation for semantic segmentation”, advances in neural information processing systems,” Advances in neural information processing systems, 2019.
  * [48] X. Peng, B. Sun, K. Ali, and K. Saenko, “Learning deep object detectors from 3d models,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 1278–1286, 2015.
  * [49] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23–30, IEEE, 2017.
  * [50] E. Wood, T. Baltrušaitis, C. Hewitt, S. Dziadzio, T. J. Cashman, and J. Shotton, “Fake it till you make it: Face analysis in the wild using synthetic data alone,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3681–3691, 2021.
  * [51] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022, 2021.
  * [52] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee, 2009.
  * [53] J. M. Saragih, S. Lucey, and J. F. Cohn, “Deformable model fitting by regularized landmark mean-shift,” International journal of computer vision, vol. 91, no. 2, pp. 200–215, 2011.
  * [54] T. F. Cootes, G. J. Edwards, and C. J. Taylor, “Active appearance models,” in European conference on computer vision, pp. 484–498, Springer, 1998.
  * [55] Y. Li, J. Zeng, S. Shan, and X. Chen, “Self-supervised representation learning from videos for facial action unit detection,” in IEEE Conf. on Computer Vision and Pattern Recognition, pp. 10924–10933, 2019.
  * [56] L. A. Jeni, J. F. Cohn, and T. Kanade, “Dense 3d face alignment from 2d videos in real-time,” in 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 2015.
  * [57] Z. Shao, Z. Liu, J. Cai, Y. Wu, and L. Ma, “Facial action unit detection using attention and relation learning,” IEEE Transactions on Affective Computing, 2019.
  * [58] R. Li, K. Bladin, Y. Zhao, C. Chinara, O. Ingraham, P. Xiang, X. Ren, P. Prasad, B. Kishore, J. Xing, et al., “Learning formation of physically-based face attributes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3410–3419, 2020.
  * [59] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [60] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in PyTorch,” in NeurIPS Autodiff Workshop, 2017.
  * [61] D. E. King, “Dlib-ml: A machine learning toolkit,” Journal of Machine Learning Research, vol. 10, pp. 1755–1758, 2009.
  * [62] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017.
  * [63] M. Hardt, E. Price, and N. Srebro, “Equality of opportunity in supervised learning,” Advances in neural information processing systems, vol. 29, 2016\.
  * [64] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel, “Fairness through awareness,” in Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214–226, 2012.