Given a $[p]$-adic linear program
$\min\\{w^{\top}x:Ax=b,x\geq\mathbf{0},x\text{ $[p]$-adic}\\}$ that has an
optimal solution, where $A$ has $m$ rows, we provided upper bound guarantees
on the support size of an optimal solution, where the bound depended
polynomially on $m,p$ and the encoding size of $A$. A helpful twist in this
case was extending the notion of $[p]$-adic numbers to include the case of
$p=1$, by declaring the $[1]$-adic numbers as the integers. As such, we
obtained a spectrum of guarantees ranging from ILPs ($p=1$) on the one end,
passing through dyadic linear programs ($p=2$), and reaching LPs ($p=\infty$)
on the other and matching Carathéodory’s bound. Along the way, we also
provided tight upper bounds on the support size of a solution to a feasible
$[p]$-adic linear system of the form $Ax=b,x\text{ $[p]$-adic}$.

While our upper bound guarantees for $[p]$-adic linear programs are tight for
the two ends of the spectrum, $p=1$ (5.3) and $p=\infty$, there remains a gap
between our best lower bound (5.2) and our upper bound for $2\leq p<\infty$,
as discussed in Section 5.5. We believe that due to the density of the
feasible region for $p\geq 2$, the upper bounds in this case should look more
like the upper bounds for $[p]$-adic linear systems.

In the special case of $\|A\|_{\infty}=1$, the best lower bound on the support
size of an optimal solution to a $[p]$-adic linear program that we can show is
at most $O(m)$, while our upper bound is $O(m\ln m)$. Closing the gap in this
case remains an intriguing open question. An important special case comes in
the dyadic ($p=2$) case from the fractional $T$-join packing problem mentioned
above. By developing a column generation technique for solving dyadic linear
programs, and by leveraging tools from matching theory, we achieve a matching
upper bound of $O(m)$ (note $m=|E|$ in this case) [1].

We have used the size of the input, in particular $\ln\|A\|_{\infty}$ to state
our results (for bounds on computational complexity as well as support size
bounds etc.). However, for specially structured instances, there are better
complexity measures, capturing more intrinsic properties of the instance. This
typically yields tighter and more insightful bounds. Thus, it would be
fruitful to pursue this direction in future research.

Let $a^{1},\ldots,a^{n}\in\mathbb{Z}^{m}$. The set $\\{a^{1},\ldots,a^{n}\\}$
is a _dyadic generating set for a cone (DGSC)_ if every integral vector in the
conic hull of the vectors can be expressed as a dyadic conic combination of
the vectors. This notion was coined and studied in our first work on dyadic
linear programming [4]. Given a DGSC $\\{a^{1},\ldots,a^{n}\\}$ and an
integral vector $b$ in the conic hull, we know that $b$ can be expressed as a
dyadic conic combination of the vectors. What is the fewest number $k$ of
nonzero coefficients in such a representation? While 5.12 gives an upper bound
of $O(m\ln(m\|A\|^{2}_{\infty}))$ on $k$, we conjecture that there is a $O(m)$
upper bound on $k$. The rationale behind this comes from the observation that
a DGSC may be viewed as the dyadic analogue of _Hilbert bases_ for integer
linear programming [16] for which the analogous upper bound guarantee is
$2m-2$ [29].

Finally, we propose a weakening of Seymour’s dyadic conjecture.

######  Conjecture 6.1.

Let $A\in\\{0,1\\}^{m\times n}$ be an ideal matrix, and for some
$c\in\mathbb{Z}^{n}_{\geq 0}$, let
$\tau_{c}:=\min\\{c^{\top}x:Ax\geq\mathbf{1},x\geq\mathbf{0}\\}$. Let $p$ be
the largest prime in $[\tau_{c}]$. Then
$\max\\{\mathbf{1}^{\top}y:A^{\top}y\leq c,y\geq\mathbf{0}\\}$ has a
$[p]$-adic optimal solution.

This conjecture has been verified for the clutter of dijoins of a digraph
([20], Theorem 2.13).

## Acknowledgements

We would like to acknowledge pleasurable discussions with Yatharth Dubey and
Siyue Liu about sparsity aspects of this work; progress towards 5.6 was most
fruitful. This work was supported in part by ONR grant N00014-22-1-2528, EPSRC
grant EP/X030989/1 and Discovery Grants from NSERC.

## References

  * [1] A. Abdi, G. Cornuéjols, B. Guenin, and L. Tunçel.  Dyadic linear programming: implicit models and applications.  In preparation.
  * [2] A. Abdi, G. Cornuéjols, B. Guenin, and L. Tunçel.  Clean clutters and dyadic fractional packings.  SIAM Journal on Discrete Mathematics, 36:1012–1037, 2022.
  * [3] A. Abdi, G. Cornuéjols, B. Guenin, and L. Tunçel.  Total dual dyadicness and dyadic generating sets.  In K. Aardal and L. Sanità, editors, Integer Programming and Combinatorial Optimization, pages 1–14, Cham, 2022. Springer International Publishing.
  * [4] A. Abdi, G. Cornuéjols, B. Guenin, and L. Tunçel.  Total dual dyadicness and dyadic generating sets.  Mathematical Programming, 2023.
  * [5] A. Abdi, G. Cornuéjols, and Z. Palion.  On dyadic fractional packings of $T$-joins.  SIAM Journal on Discrete Mathematics, 36:2445–2451, 2022.
  * [6] I. Aliev, J. A. De Loera, T. Oertel, and C. O’Neill.  Sparse solutions of linear Diophantine equations.  SIAM J. Appl. Algebra Geometry, 1(1):239–253, 2017.
  * [7] S. Arora and B. Barak.  Complexity Theory: A Modern Approach.  Cambridge University Press, 2009.
  * [8] E. F. Beckenbach and R. Bellman.  Inequalities.  Ergebnisse der Mathematik und ihrer Grenzgebiete. 2. Folge. Springer Berlin, Heidelberg, 1961.
  * [9] E. Bombieri and J. Vaaler.  On Siegel’s lemma.  Inventiones mathematicae, 73(1):11–32, 1983.
  * [10] M. Conforti, G. Cornuéjols, and G. Zambelli.  Integer Programming.  Springer International Publishing, 2014.
  * [11] W. Cook, J. Fonlupt, and A. Schrijver.  An integer analogue of Carathéodory’s theorem.  Journal of Combinatorial Theory, Series B, 40(1):63–70, 1986.
  * [12] G. Cornuéjols.  Combinatorial Optimization: Packing and Covering.  SIAM, 2001.
  * [13] Y. Dubey and S. Liu.  A short proof of tight bounds on the smallest support size of integer solutions to linear equations.  arXiv:2307.08826, July 2023.
  * [14] F. Eisenbrand and G. Shmonin.  Carathéodory bounds for integer cones.  Operations Research Letters, 34(5):564–568, 2006.
  * [15] D. Espinoza.  On Linear Programming, Integer Programming and Cutting Planes.  PhD thesis, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA, 2006.
  * [16] F. Giles and W. Pulleyblank.  Total dual integrality and integer polyhedra.  Linear Algebra and its Applications, 25:191 – 196, 1979.
  * [17] F. Gouvêa.  p-adic Numbers - An Introduction.  Springer, 1997.
  * [18] J. Hadamard.  Resolution d’une question relative aux determinants.  Bull. Sci. Math., 2:240–248, 1893.
  * [19] J. C. K. Ho and L. Tunçel.  Reconciliation of various complexity and condition measures for linear programming problems and a generalization of Tardos’ theorem.  In Foundations of computational mathematics (Hong Kong, 2000), pages 93–147. World Sci. Publ., River Edge, NJ, 2002.
  * [20] S. Hwang.  The Edmonds-Giles conjecture and its relaxations.  Master’s thesis, University of Waterloo, 2022.
  * [21] R. Kannan and A. Bachem.  Polynomial algorithms for computing the Smith and Hermite normal forms of an integer matrix.  SIAM J. Comput., 8(4):499–507, 1979.
  * [22] L. Lovász.  On two minimax theorems in graph theory.  Journal of Combinatorial Theory, Series B, 21:96 – 103, 1976.
  * [23] J. Martinet.  Perfect Lattices in Euclidean Spaces.  Springer-Verlag Berlin Heidelberg, 2003.
  * [24] N. Megiddo.  On finding primal- and dual-optimal bases.  ORSA J. Comput., 3(1):63–65, 1991.
  * [25] S. Mehrotra and Y. Ye.  Finding an interior point in the optimal face of linear programs.  Math. Programming, 62(3):497–515, 1993.
  * [26] V. V. Prasolov.  Problems and Theorems in Linear Algebra, volume 134 of Translations of Mathematical Monographs.  American Mathematical Society, 1994.
  * [27] A. Schrijver.  Theory of Linear and Integer Programming.  Wiley, April 1998.
  * [28] A. Schrijver.  Combinatorial Optimization. Polyhedra and Efficiency, volume 24 of Algorithms and Combinatorics.  Springer-Verlag, Berlin, 2003.
  * [29] A. Sebő.  Hilbert bases, Carathéodory’s theorem, and combinatorial optimization.  In R. Kannan and W. Pulleyblank, editors, Integer Programming and Combinatorial Optimization (IPCO 1990), Lecture Notes in Computer Science, pages 431–456, 1990.
  * [30] A. Storjohann.  Algorithms for matrix canonical forms.  PhD thesis, ETH Zurich, 2000.
  * [31] L. Tunçel.  Polyhedral and semidefinite programming methods in combinatorial optimization, volume 27 of Fields Institute Monographs.  American Mathematical Society, Providence, RI; Fields Institute for Research in Mathematical Sciences, Toronto, ON, 2010.
  * [32] J. D. Vaaler.  A geometric inequality with applications to linear forms.  Pacific Journal of Mathematics, 83(2):543 – 553, 1979.
  * [33] Y. Ye.  Interior point algorithms.  Wiley-Interscience Series in Discrete Mathematics and Optimization. John Wiley & Sons, Inc., New York, 1997.  Theory and analysis, A Wiley-Interscience Publication.
  * [34] Y. Ye, M. J. Todd, and S. Mizuno.  An $O(\sqrt{n}L)$-iteration homogeneous and self-dual linear programming algorithm.  Math. Oper. Res., 19(1):53–67, 1994.

##  Appendix A Primer on lattices and determinants

Generally we follow [23, 8, 26] as reference texts on lattices, inequalities,
and equalities in linear algebra, respectively. More specifically, we use the
following specific results in the paper.

##### Hermite normal form.

Let $A\in\mathbb{Z}^{m\times n}$ be a matrix of full row rank. Then $A$ can be
brought into _Hermite normal form_ by means of _elementary unimodular column
operations_. In particular, there exists an $n$-by-$n$ unimodular matrix $U$
such that $AU=(B~{}\mathbf{0})$, where $B$ is a non-singular $m$-by-$m$
matrix, and $\mathbf{0}$ is an $m$-by-$(n-m)$ matrix with zero entries. See
([10], Section 1.5.2) or ([27], Chapter 4) for more details.

##### Matrix GCD.

Denote by $\gcd(A)$ the greatest common divisor (GCD) of all order-$m$ minors
of $A$. This quantity can be found as follows. First, bring $A$ into Hermite
normal form after applying elementary unimodular column operations, say we
obtain $(B~{}\mathbf{0})$ where $B$ is a square matrix. Then
$\gcd(A)=\gcd(B)=|\det(B)|$, where the first equality can be readily checked
by the reader, and the second one follows immediately from definition.

##### Lattice “closure” and index.

Let $L$ be an $m$-dimensional lattice contained in $\mathbb{Z}^{n}$, and let
$\overline{L}:=\operatorname{aff}(L)\cap\mathbb{Z}^{n}$. Clearly,
$\overline{L}$ is a lattice containing $L$. Observe that within the affine
hull of $L$, and confined to the integers, $\overline{L}$ is the maximal
lattice containing $L$. It is known that $\overline{L}$ can be partitioned
into a number of integral linear translations of $L$; this number is denoted
as the _index_ $[\overline{L}:L]\in\mathbb{Z}_{\geq 1}$. Thus,
$L=\overline{L}$ if and only if $[\overline{L}:L]=1$.

##### Lattice determinant and orthogonal complement.

The _determinant of $L$_, denoted $\det(L)$, is the $m$-dimensional volume of
its fundamental parallelepiped. The inverse of $\det(L)$ may be used to
measure the “density” of $L$: the smaller the determinant, the larger the
density. It is known that $\det(L)=[\overline{L}:L]\det(\overline{L})$ (see
[23], Proposition 1.1.5 (4)). The _orthogonal complement_ of $L$ is the
lattice $L^{\perp}:=\left\\{y\in\mathbb{Z}^{n}:y^{\top}x=0~{}~{}\forall~{}x\in
L\right\\}$. It can be readily checked that $L^{\perp}$ has index $1$, that
is, $L^{\perp}=\overline{L^{\perp}}$. It is known that
$\det(L^{\perp})=\det(\overline{L})$ (see [23], Proposition 1.9.8).

##### Lattice formulas.

Let $C\in\mathbb{Z}^{n\times m}$ be a matrix with $m$ linearly independent
columns, and consider the lattice
$L:=\left\\{Cy:y\in\mathbb{Z}^{m}\right\\}\subseteq\mathbb{Z}^{n}$. Let us
find $\det(L),\overline{L},[\overline{L}:L]$ and $L^{\perp}$ in terms of $C$.
First, the fundamental parallelepiped of $L$ is
$\left\\{C\lambda:\mathbf{0}\leq\lambda<\mathbf{1}\right\\}$, whose
$m$-dimensional volume is $\det(L)$. Subsequently, it can be readily checked
that $\det(L)^{2}$ is the Gram determinant of the columns of $C$, that is,
$\det(L)=\sqrt{\det(C^{\top}C)}$. It can be readily seen that
$\overline{L}=\\{Cy:y\in\mathbb{R}^{m}\\}\cap\mathbb{Z}^{n}$. By turning
$C^{\top}$ into Hermite normal form after applying elementary unimodular
column operations, it can be shown that $[\overline{L}:L]=\gcd(C)$ (see [23],
Proposition 1.1.5 (4)). Finally, observe that
$L^{\perp}=\left\\{x\in\mathbb{Z}^{n}:C^{\top}x=\mathbf{0}\right\\}$. By
combining the equalities provided thus far, we see that
$\det(L^{\perp})=\sqrt{\det(C^{\top}C)}/\gcd(C)$.

##### Lattice-free sets.

Observe that $P:=\\{C\lambda:-\mathbf{1}<\lambda<\mathbf{1}\\}$ is _lattice-
free_ , that is, $P$ contains no nonzero vector from the lattice $L$. The set
$P$ is the union of $2^{m}$ reflections of the fundamental parallelepiped of
$L$, whose relative interiors are pairwise disjoint, so
$\operatorname{\mathrm{vol}}_{m}(P)=2^{m}\det(L)$. Observe further that $P$ is
a convex set that is symmetric about the origin. An important result of
Minkowski from the Geometry of Numbers states that among all such sets which
are lattice-free, $P$ reaches the maximum $m$-dimensional volume.

######  Minkowski’s First Theorem (see [23], Theorem 2.7.1).

Let $L\subseteq\mathbb{Z}^{n}$ be an $m$-dimensional lattice. Let $C$ be a
convex set that is symmetric about the origin. If $C\cap L=\\{\mathbf{0}\\}$,
then $\operatorname{\mathrm{vol}}_{m}(C)\leq 2^{m}\det(L)$.

##### Hadamard’s inequality.

Let $B\in\mathbb{Z}^{m\times m}$ be a matrix with columns
$b^{1},\ldots,b^{m}$. Observe that $|\det(B)|$ is the volume of the
parallelepiped with sides $b^{1},\ldots,b^{m}$, implying in turn the well-
known _Hadamard inequality_ , stating that
$|\det(B)|\leq\prod_{i=1}^{m}\|b^{i}\|_{2}$ ([18], see [8], Chapter 2, §11).
Subsequently, since $\|b^{i}\|_{2}\leq\sqrt{m}\|b^{i}\|_{\infty}$ for each
$i\in[m]$, it follows that $|\det(B)|^{1/m}\leq\sqrt{m}\|B\|_{\infty}.$

##### Minors of the adjugate matrix.

Let $U$ be an $n$-by-$n$ matrix. The _adjugate of $U$_, denoted
$\mathrm{adj}(U)$, is the $n$-by-$n$ matrix such that
$U\mathrm{adj}(U)=\det(U)I$. It is well-known that for all $i,j\in[n]$,
$\mathrm{adj}(U)_{ji}$ is a cofactor of $U$, namely, it is $(-1)^{i+j}$ times
the determinant of the submatrix of $U$ obtained after removing row $i$ and
column $j$. What is less known is an extension of this known attributed to
Jacobi, which expresses every minor of $\mathrm{adj}(U)$ as a cofactor of the
original matrix $U$.

######  Jacobi’s Theorem (see [26], Theorem 2.5.2).

Let $U$ be an $n$-by-$n$ matrix, and $m\in[n-1]$. For row labels
$I\subseteq[n]$ and column labels $J\subseteq[n]$ of $U$ such that
$|I|=|J|=m$, we have

$\det(\mathrm{adj}(U)_{J\times
I})=\mathrm{sgn}(\sigma)\det(U_{\overline{I}\times\overline{J}})\det(U)^{m-1}.$

where $\sigma:[n]\to[n]$ is the permutation defined as $\sigma(i_{k})=j_{k}$
for all $k\in[n]$, for the following labelling of the rows and columns of $U$:
the rows$/$columns of $U_{I\times J}$ are labelled
$\\{i_{1},\ldots,i_{m}\\}/\\{j_{1},\ldots,j_{m}\\}$ from top to bottom$/$left
to right, and the rows$/$columns of $U_{\overline{I}\times\overline{J}}$ are
labelled $\\{i_{m+1},\ldots,i_{n}\\}/\\{j_{m+1},\ldots,j_{n}\\}$ from top to
bottom$/$left to right.


11institutetext: State University of New York at Binghamton

# Knowledge-Spreader: Learning Facial Action Unit Dynamics with Extremely
Limited Labels

Xiaotian Li  Xiang Zhang  Taoyue Wang  Lijun Yin

###### Abstract

Recent studies on the automatic detection of facial action unit (AU) have
extensively relied on large-sized annotations. However, manually AU labeling
is difficult, time-consuming, and costly. Most existing semi-supervised works
ignore the informative cues from the temporal domain, and are highly dependent
on densely annotated videos, making the learning process less efficient. To
alleviate these problems, we propose a deep semi-supervised framework
Knowledge-Spreader (KS), which differs from conventional methods in two
aspects. First, rather than only encoding human knowledge as constraints, KS
also learns the Spatial-Temporal AU correlation knowledge in order to
strengthen its out-of-distribution generalization ability. Second, we approach
KS by applying consistency regularization and pseudo-labeling in multiple
student networks alternately and dynamically. It spreads the spatial knowledge
from labeled frames to unlabeled data, and completes the temporal information
of partially labeled video clips. Thus, the design allows KS to learn AU
dynamics from video clips with only one label allocated, which significantly
reduce the requirements of using annotations. Extensive experiments
demonstrate that the proposed KS achieves competitive performance as compared
to the state of the arts under the circumstances of using only 2% labels on
BP4D and 5% labels on DISFA. In addition, we test it on our newly developed
large-scale comprehensive emotion database, which contains considerable
samples across well-synchronized and aligned sensor modalities for easing the
scarcity issue of annotations and identities in human affective computing. The
code and new database will be released to the research community.

###### Keywords:

Semi-supervised learning, facial action unit detection, knowledge
distillation, sparsely labeled data, Spatial-Temporal, pseudo-labeling

##  1 Introduction

Understanding human facial action dynamics is crucial for human-computer
interaction (HCI), which reflects the individuals’ affective states to enhance
the affinity of communication. Facial action unit (AU) detection plays a vital
role in automatic facial action analysis. Over the past few years, a
substantial amount of works [11] [47] [1] based on deep learning have shown
the superior power of constructing informative features against the
traditional methods. The majority of the recent advances [22] [43] [27] [6]
[39] [48] [12] in this area are heavily relied on large-scale labeled datasets
for achieving remarkable learning performance. However, a lab-controlled AU
video typically contains thousands of frames that need to be densely labeled
by human annotators. As a result, datasets for dynamic facial action detection
suffer high redundancy both content-wise and annotation-wise.

There are only limited works attempting to mitigate the demand for dense AU
occurrence annotations. For example, the researchers [41] [46] [32] [31]
summarize the distribution from existing ground-truth AU labels as the prior
for detecting AUs with partially labeled data. However, the approaches of
applying the distribution prior may fall into sub-optimal due to lacking
adaptation mechanism. Deep semi-supervised learning has become a new choice to
overcome these issues for their powerful representation and generalization
ability. The existing deep semi-supervised learning methods can be roughly
categorised into three methods (i.e. generative method, consistency
regularization method, and pseudo-labeling methods). Recently, some hybrid
methods attract the attention from many researchers. MixMatch [23] combines
entropy minimization and consistency regularization in an unified loss
function. FixMatch [37] combines consistency regularization and pseudo-
labeling considering both labeled and unlabeled data should be trained
simultaneously. Whereas, applying these image-level models to dynamic datasets
is challenging due to some nuisance factors such as motion blur, video defuse,
and frequent pose occlusions. Besides, human facial action characteristics
present strong dependencies and mutual exclusive relation among spatially
local regions [21]. For instance, the AU1 and AU2 are usually co-existence due
to the constraint of facial muscles. Some recent works [24] [3] prove that the
relationship of different AUs in temporal domain is also an important factor
for robust AU detection. Both of the two factors have been studied for fully
supervised methods, but there lacks a framework to explore if learning both
spatial and temporal AU correlation knowledge can boost the semi-supervised
process.

Figure 1: Overall pipeline of Knowledge Spreading. (1) Train a Spatial Teacher
for learning supervised image-level spatial knowledge in Branch $A$
(_qualitative training_), and train a group of Spatial Students for semi-
supervised learning in Branch $B$ (_quantitative training_). (2) Process
spatial knowledge knowledge distillation [10] when one of the Spatial Students
gets labeled key frames (red square) as inputs while other Spatial Students
infer pseudo labels using unlabeled frames. With the position of key frame
shifting from 1 to 3 repeatedly, the spatial AU knowledge (blue) will be
gradually spread to every Spatial Student and boost the _quantitative
training_ of pseudo-labeling. (3) By completing and encoding the outputs of
all Spatial Students, Temporal Teacher is deployed to learn the AU temporal
cues (orange). (4) Distill it back to Temporal Student in A for getting an
optimal fusion of Spatial-Temporal information.

In this paper, we propose an end-to-end trainable framework Knowledge-Spreader
(KS) for learning semi-supervised facial action dynamics. Knowledge-Spreader
improves conventional semi-supervised learning in three ways. First, different
from previous video-level works that need fully supervisory information or at
least a few labeled frames as input segments, KS just needs limited sparsely
labeled video clips, with only one label allocated. This alleviates the demand
on densely labeled data effectively. Second, incorporating the weak
supervisory information from both human prior and AU correlation knowledge
makes KS more efficient and robust for inferring out-of-distribution
information. KS learns both spatial and temporal AU correlation knowledge by
transformers. The powerful generalization capability of deep learning can
reduce some side effects caused by incorrect knowledge summarized by humans.
For instance, some AU intensity estimation works [50] assume that there are
only one peak or valley in a video sequence, while ignore the special
situation that multiple peaks and valleys exist in a video. Third, our work
improves the hybrid semi-supervised method (consistency regularization and
pseudo-labeling) to a video level model by introducing an operation “Knowledge
Spreading”. The overall pipeline of Knowledge Spreading is shown in Figure 1.
The _qualitative training_ in Figure 1 refers that the network is trained with
a small amount of labeled images, while the _quantitative training_ indicates
the training with a large-sized unlabeled data. Our contribution lies in
three-fold: (1) We propose a deep semi-supervised architecture for AU
detection by jointly utilizing the learned Spatial-Temporal AU correlation
knowledge and human knowledge. (2) This is the first work to incorporate
knowledge spreading by dynamic knowledge distillation and temporal confirmed
pseudo-label in a self-supervised manner, to learn spatio-temporal information
using extremely limited single-frame labeled video clips. (3) We have built a
new spontaneous emotion database by capturing 3D geometric facial sequences,
2D facial videos, thermal videos, and physiological data sequences from 233
participants across two years period. The new database will be released to the
research community along with the paper being published.

##  2 Related Work

###  2.1 Automatic Facial AU Detection

Different from general computer vision tasks, facial AUs are defined to be
associated with atomic local facial muscles, which in turn correspond to the
appearance features of different regions in the face. Thus, previous research
[53] [42] [35] [43] [52] has extensively studied how to use manually-defined
regions, local patches, facial landmarks, heatmap, attention mechanism, and
other methods to localize detailed facial parts. However, considering the
structural information and dependencies among different AUs, conventional CNN
is criticized to be incapable of fully characterizing their correlations.
Recent advances [6] [40] [48] [12] [17] start to explore the AU relations by
using data distribution, conditional random field, Bayesian Network, graph
neural networks, attention mechanism, and other approaches. [3] [19] [26]
attempt to integrate the inter-relationship of AUs on both spatial and
temporal dimensions for learning more robust AU dependencies.

Some of the existing semi-supervised or self-supervised methods [22] [43] [27]
[6] still rely on large-sized annotations and extra unlabeled data. Only a
small amount of research has attempted to ease the intensive demand for AU
annotations. [41] and [46] summarize the distribution from ground-truth AU
labels as the prior for detecting AUs with limited labeled data with semi-
supervised learning. [32] [31] leverage the joint distribution of features,
AUs, and facial expression to explore the global dependencies among them.
Different from all aforementioned works, KS can learn the informative Spatial-
Temporal cues from sparse annotations by incorporating dynamic knowledge
distillation and pseudo label.

###  2.2 Semi-supervised learning

Semi-supervised learning is an approach to combine a few labeled data with a
large amount of unlabeled data during training. Deep semi-supervised learning
is a fast-growing filed with a wide broad of practical applications. FixMatch
[37] has been proved to be a simple but efficient SSL method by combining both
consistency regularization and pseudo-labeling. Noisy Student improves the
idea of self-training and distillation with the use of noise added to the
student networks. However, these works are incapable to capture the facial
action dynamics as an image-level model. Recent works [36] [44] [30] propose
to recognize actions from only a handful of sparsely labeled videos with the
rich supervisory information in a semi-supervised way. Unlike these approaches
that still need continuous and dense annotation, we aim to allocate only
single-frame label for each video clip to handle the condition that the labels
are extremely sparse, missing, or scarce.

##  3 Methodology

###  3.1 Overview

Our goal is to detect facial action units with video clips with single-frame
labels. Specifically, we assume the facial action labels in training data are
available every $k$ frames. The $i$th video clip consisted of $n$ frames is a
set
$V^{i}=\left\\{F_{u,1}^{i},F_{u,2}^{i},\cdot\cdot\cdot,F_{l,m}^{i},\cdot\cdot\cdot,F_{u,n}^{i}\right\\}$
where $F_{u}^{i}$ means unlabeled frame and $F_{l}^{i}$ means labeled frame.
Knowledge-Spreader is consisted of three key modules: (1) Spatial-Temporal
information learning module (SIL), (2) knowledge spreading module (KSM), and
(3) temporal confirmed pseudo-label module (TPL). The network structure of KS
is depicted in Figure 2.

Figure 2: Illustration of the detailed architecture for Knowledge-Spreader.
The framework is consisted of two branches without sharing weights for
learning image-level and video-level features respectively. Model $S_{a}$,
$S_{b}$, and $T_{b}$ (purple) are networks used for learning Spatial-Temporal
AU correlation knowledge. Two level distillation (grey) and pseudo-labeling
(green) consist the proposed “knowledge spreading”. A self-supervised module
“temporal confirmed pseudo-label” (yellow) is designed for sensing the
temporal perturbation and selecting pseudo labels with high confidence.
Details are described in Section 3.1.3.

####  3.1.1 Spatial-Temporal Information Learning (SIL)

SIL is an self-attention-based module for learning semi-supervised AU
dependencies in both spatial and temporal domain. Transformers have been
proved to be an effective method of learning informative temporal context in
natural language processing. In order to address the limitation of applying
conventional transformers on computer vision, ViT [7] splits an image into
fixed-sized patches to linearly embed them with position embedding. We extend
the original ViT to learn the semantic relationship of AUs in both spatial-
wise and temporal-wise. As shown in Figure 2, the features from branch $A$ and
branch $B$ are extracted by base models which are ResNet-18 [9] pre-trained on
ImageNet [33]. Model $S_{a}$ is designed for learning the spatial AU
correlation. We firstly decouple the extracted global features $p_{s}$ into
multiple branches by applying global-average-pooling. The collection of
decoupled features is represented as
$D_{s}=\left\\{f_{s}^{1},f_{s}^{2},\cdot\cdot\cdot,f_{s}^{u}\right\\}$ where
$u$ means the number of facial action units. Afterward, we assign $u$ 1D
learnable positional embedding to be added with these features as AU-specific
embedding. Then, the AU-specific embedding sequence is fed to a standard
Transformer encoder for learning the spatial-wise AU correlation knowledge.
Note that, the module does not need any extra classification token, for no
obvious performance gains are observed by applying this design. Likewise, the
temporal model $T_{b}$ in branch B also adopts the same model design. The
collection of frame-specific features is depicted as
$D_{b}=\left\\{p_{b}^{1},p_{b}^{2},\cdot\cdot\cdot,p_{b}^{n}\right\\}$ where
$n$ is the number of frames in each input sequence. The frame-specific
features are extracted by the sub-networks in $S^{n}_{b}$ and added with the
corresponding temporal embedding.

####  3.1.2 Knowledge Spreading Module (KSM)

KSM is the core module of the proposed Knowledge-Spreader. Inspired by
FixMatch, our model also adopts a hybrid semi-supervised learning method by
combining consistency regularization and pseudo-labeling. In this paper, we
improve it to a video level method in the purpose of learning efficient AU
dynamics. We approach KSM by several main steps: (1) train a group of Spatial
Students $S_{b}$ which accommodate all the frames of input sequence, (2) let
the length of input clip equals to the number of Spatial Students, (3) couple
the input image in branch $A$ with the key frame of input video clip in branch
$B$, (4) shift the location of key frame during training different batch of
training samples, and (5) process distillation when Spatial Student gets the
key frames, otherwise process pseudo-labeling. Furthermore, in terms of the
distillation, we adopt two-level and bidirectional knowledge distillation
design. The first level is from Spatial Teacher $S_{a}$ to Spatial Students
$S_{b}$. For the MLP-based Spatial Students $S_{b}$, we apply data random
augmentation and model dropout as the noise. We assume the knowledge learned
by Spatial Teacher without noise is consistent with that learned by noisy
Spatial Students. The second level is from Temporal Teacher $T_{b}$ to
Temporal Student $S_{a}$. Instead of applying any artificial perturbation for
temporal distillation, we utilize the original difference of input data for
$T_{b}$ and $S_{a}$. We assume the spatial knowledge learned by $S_{a}$ should
be consistent with the temporal knowledge learned by $T_{b}$. Thus, our model
forces the Temporal Student $S_{a}$ to learn the temporal AU knowledge harder,
even if it only gets the image-level inputs. Here, Model $S_{a}$ plays
different roles in the two level distillation. Inspired by recent work [8], we
adopt an online distillation method using KL divergence. The KL divergence
loss in this work is used to minimize the probability distribution of student
networks and their corresponding ensembles. The KL loss function of spatial
knowledge distillation is defined as

$L_{skd}=\frac{1}{b}\left(\sum_{i=1}^{n}T^{2}KL\left(p_{i},q_{i}\right)+\sum_{i=1}^{n}T^{2}KL\left(w^{m}_{i},q_{i}\right)\right)$
(1)

where $b$ is the batch size, $T$ is the temperature parameter. $p$ and $w$
denote the soften probability distribution calculated by the Spatial Teacher
$S_{a}$ and one Spatial Student selected from
$\left\\{S^{1}_{b},S^{2}_{b},...,S^{n}_{b}\right\\}$. The soft target $q$ is
expressed as $q=softmax(z_{s}/T)$ where $z_{s}$ is performed by the mean
pooling of the outputs from both $S_{a}$ and one selected $S_{b}$. $m$ is the
key frame position used for controlling which Spatial Student is selected as
the target of knowledge spreading. $m$ equals $B$ mod $n$, where $B$ is the
$B$th batch of training samples, $n$ is the clip length. In this paper,
temperature parameter is set as 1. The KL loss function of temporal knowledge
distillation is defined as

$L_{tkd}=\frac{1}{b}\left(\sum_{i=1}^{n}T^{2}KL\left(p_{i},q_{i}\right)+\sum_{i=1}^{n}T^{2}KL\left(w_{i},q_{i}\right)\right)$
(2)

$p$ and $w$ denote the soften probability distribution calculated by the
Temporal Teacher $T_{b}$ and the Temporal Student $S_{a}$ respectively.
Considering the data imbalance issues from skewing the training process and
affect the performance of the model. We choose weighted BCE with logits as the
multi-label classification loss of student networks and the function can be
described as:

$L_{bce}=-w[ylog\sigma\left(x\right)+\left(1-y\right)log\left(1-\sigma\left(x\right)\right)]$
(3)

where $x$ refers to the output $O_{a}$ and $O_{b}$ for the loss function
$L_{s}$ and $L_{t}$ in Equation 4. $x$ also refers to the final ensemble
output $O_{output}$ for $L_{bce}$ in Equation 7. $\sigma(x)$ is the
corresponding predicted probability. $y$ is the AU occurrence ground truth.
The loss functions of spatial knowledge distillation and temporal knowledge
distillation are expressed as:

$L_{s}=\sum_{i=1}^{z}L_{bce}^{i}+\alpha
L_{skd},L_{t}=\sum_{i=1}^{z}L_{bce}^{i}+\alpha L_{tkd}$ (4)

where $\alpha$ is the trade-off weight, $z$ is the number of student networks.
Although branch $B$ contains multiple student networks
$\left\\{S^{1}_{b},S^{2}_{b},...,S^{n}_{b}\right\\}$, only one of them is
selected for calculating the loss function at the same time. Thus, $z$ equals
2. In this paper, $\alpha$ is set as 0.5.

Pseudo-label [15] is a simple but efficient formulation of training models in
a semi-supervised way. We deploy it to train the Spatial Students in
$\left\\{S^{1}_{b},S^{2}_{b},...,S^{n}_{b}\right\\}$ when unlabeled frames are
assigned as the input. Note that, the labeled data and unlabeled data are
jointly trained. The loss function is defined as

$L_{pdbce}=-w[\hat{y}log\sigma\left(x\right)+\left(1-\hat{y}\right)log\left(1-\sigma\left(x\right)\right)]$
(5)

where $\hat{y}$ denotes the pseudo labels of unlabeled frames by picking up
the class which has the maximum predicted probability. The total loss function
of semi-supervised learning can be denoted as

$L_{semi}=\sum_{i=1}^{n-1}L_{pdbce}^{i}$ (6)

Figure 3: Pipeline of the temporal confirmed pseudo-label. The module is based
on self-supervised learning for a binary classification task. At training
stage, it feeds $D^{i}_{b}$ (without shuffling) and $D^{i}_{pb}$ (shuffling
$D^{i}_{b}$ along the timeline of the features) into the model. At the
prediction stage, we let the feature $D^{m}_{b}$ (without shuffling) as the
input. If it is classified as false, the corresponding pseudo labels will be
discarded and vice versa. The training stage starts from the first epoch. By
observation, we find the simple binary classifier takes only one or two epochs
to convergence. Thus, we set the third epoch as the flag to start the
prediction stage.

####  3.1.3 Temporal Confirmed Pseudo-label (TPL)

We exploit a simple self-supervised method for confirming confident pseudo
labels by predicting if the features have any temporal perturbation. The
smooth evolution of facial muscles leads that the facial appearance also moves
gradually and smoothly over time. Thus, the pseudo labels generated by
incorrect features contains anomalies in the temporal domain. Researchers [44]
utilize the self-supervised sequential perturbation to improve their model’s
robustness and generalization. Inspired by this work, we use the temporal
feature shuffling to simulate the sequential perturbation and generate
negative feature samples. Although a few video clips contain fully still
frames which may diminish the contribution of temporal feature shuffling, they
only occupy a small portion in the whole dataset. In addition, these samples
do not contain any temporal information. Therefore, labeling them incorrectly
does not affect the learning ability of the spatial model. The auxiliary task
is defined as a binary classification task that is jointly trained with the AU
classifier on Model $T_{b}$. $L_{ssl}$ denotes the loss of the binary
classification task. Specifically, We first label the $i$th collection of
sequential features $D^{i}_{b}$ as true while the shuffled feature collection
$D^{i}_{pb}$ as false. Afterward, $D^{i}_{b}$ and $D^{i}_{pb}$ are
sequentially fed into Model $T_{b}$. The loss $L_{ssl}$ encourages the model
to classify them correctly. Figure 3 illustrates the pipeline of the proposed
module. As shown in Figure 6, the negative samples show the irregular pattern
that AU occurs and disappears repeatedly in a short-term period. By detecting
the temporal perturbation of the features and filtering out untrustable pseudo
labels, TPL reliefs the confirmation bias issue existing in conventional
pseudo-labeling methods.

####  3.1.4 Overall loss function and algorithm

The total loss is composed of the losses from previous sections, as follows:

$L_{total}=\lambda_{1}L_{s}+w_{ramp}\left(L_{bce}+\lambda_{2}L_{t}+\lambda_{3}L_{ssl}+\lambda_{4}L_{semi}\right)$
(7)

where $w_{ramp}$ is a ramp-up function to make sure the semi-supervised
learning and self-supervised learning converge relatively slowly compared with
the fully-supervised task. It is inspired by [14]. The function $\mu$, as a
simple Gaussian curve function, is defined as:

$w_{ramp}=exp\left(-\omega(1-\frac{(x-\mu)^{2}}{\sigma^{2}})\right)$ (8)

where $x$ is the epoch number. In this paper, we set $\omega=2$, $\mu=0$, and
$\sigma=5$. Here the $w_{ramp}$ is set as 1 after 5 epochs’ warming-up for
getting the optimal effect. The algorithm of KS is shown in Algorithm 1.

Algorithm 1 Pseudocode of Knowledge-Spreader

1:The input frame $F_{l}^{i}$, the input clip $V^{i}$ and its frame number
$N$, the position of key frame $m$. $B$ means the $B$th batch of training
sample. Functions of the models in branch A: base model $b_{\theta}(x)$, Model
$S_{a}$ $f_{\theta}(x)$. Functions of the models in branch B: base model
$b_{\sigma}(x)$, sub-networks in $S^{n}_{b}$ with supervision $l_{\sigma}(x)$
and pseudo-labeling $k_{\sigma}(x)$, The AU detection classifier of Model
$T_{b}$ $f_{\sigma}(x)$ and the binary classifier for self-supervised learning
$g_{\sigma}(x)$.

2:for each epoch E do

3: for each mini-batch b do

4: $O_{abase}^{i}\leftarrow b_{\theta}(F_{l}^{i});O_{bbase}^{(i,n)}\leftarrow
b_{\sigma}(V^{i})$

5: $O_{a}^{i}\leftarrow f_{\theta}(O_{abase}^{i})$

6: for $q$ $=$ 1,...,$N$ do

7: if $q==B$ mod $N$ then

8: $O_{k}^{i}\leftarrow l_{\sigma}(O_{bbase}^{(i,q)})$

9: $\text{process spatial KD with }O_{a}^{i}\text{ and }O_{k}^{i}$

10: else

11: $\text{process pseudo-labeling with }O_{ps}^{(i,n-1)}\leftarrow
k_{\sigma}(O_{bbase}^{(i,q)})$

12: end if

13: end for

14: $O_{bti}^{i}\leftarrow\text{Concatenate(}O_{ps}^{(i,n-1)},O_{k}^{i})$

15: $O_{b}^{i}\leftarrow f_{\sigma}(O_{bti}^{i})$

16: $O_{pb}^{i}\leftarrow f_{\sigma}(\text{Shuffle}(O_{bti}^{i}))$

17: $O_{ssl}^{i}\leftarrow g_{\sigma}(O_{b}^{i}=0,O_{pb}^{i}=1)$

18: if $E<=2$ or $O_{ssl}^{i}==1$ then

19: $\text{Let the weight of }L_{semi}\text{, }\lambda_{4}=0$

20: end if

21: $\text{process temporal KD with }O_{b}^{i}\text{ and }O_{a}^{i}$

22: $O^{i}_{output}\leftarrow\text{MeanValue}(O_{b}^{i},O_{a}^{i})$

23: $\text{Update }\theta\text{ and }\sigma\text{ via SGD of
\lx@cref{creftypecap~refnum}{eq:equation1} }$

24: end for

25:end for

##  4 Experiments

###  4.1 Datasets

BP4D and DISFA: BP4D [49] and DISFA [25] are are widely used benchmark
databases for dynamic AU detection. We followed the experimental setting of
the previous work [17] to evaluate our approach for a fair comparison.

New MME: The existing facial action datasets are limited in terms of subjects
number, diversity, and metadata. Thanks to the existing available multi-modal
datasets [49] [51], we extend to develop a new larger-scale multi-modal
emotion (MME) database, which consists of 233 participants (132 females and
101 males). The data is significantly expanded in terms of participants number
as compared to the existing databases: DISFA (27 subjects) [25], MMI (44
subjects) [29], BP4D (41 subjects) [49], BP4D+ (140 subjects) [51]. Following
ethical principles, our data collection was approved by the institutional
review board (IRB). Each subject signed an informed consent form. A
professional performer/interviewer applied a procedure containing 10
seamlessly-integrated tasks as [49] [51] that resulted in effective
elicitation of spontaneous emotions. The dataset was well-synchronized and
aligned with multi-modalities including 3D geometric facial model, 2D facial
videos, thermal videos, and physiology data sequences (e.g. heart rate, blood
pressure, skin conductance (EDA), and respiration rate). Around 94,000 frames
were well-annotated by three expert FACS coders for AU coding. More details
are described in the supplemental material. The new database is ready for
public and will be released to the research community by the time of the paper
being published.

###  4.2 Implementation Details

We process the image by cropping off redundant area which is not relevant to
face recognition. Then the images are resized to $224\times 224\times 3$
($H\times W\times C$) to fit the model. Each of the training images is
randomly rotated, flipped horizontally, and with color jitters (saturation,
contrast, and brightness) for data augmentation. We choose SGD as the
optimizer with a learning rate of 0.01 for 50 epochs. The model was
implemented with Pytorch framework. The hyper-parameters in Equation 7 are set
as $\lambda_{1}=0.5$, $\lambda_{2}=0.5$, $\lambda_{3}=0.2$, and
$\lambda_{4}=0.25$. We use video clips with 5 frames for learning the temporal
information.

###  4.3 Model analysis

Figure 4: Comparison with other advanced semi-supervised algorithms using
different percentages of labels on BP4D, DISFA, and MME.

####  4.3.1 Comparison with semi-supervised methods

Figure 4 shows the performance comparison with semi-supervised methods from
two areas (AU detection and general action recognition). We carefully
investigated the existing works that adopt limited labels for AU detection.
BGCS [46] and DAUR [41] are selected for comparison. Figure 4 (d), (e), and
(f) shows the proposed model achieves significant performance improvement.
Considering our foundation model may have advantages in generalization
ability, we can compare the performance trend. With the available labels
decreasing (from 90% to 50%), Knowledge-Spreader shows no obvious performance
attenuation. Note that some other semi-supervised models from [32] [43] [27]
[5] are not selected for comparison considering they use full annotation
pools, extra data, or other jointly trained tasks. We further report the
comparison results with some semi-supervised methods from general action
recognition for a comprehensive evaluation, including Pseudo-label [15],
FixMatch [37], and a video-level TCL [36]. Compared with the conventional
setting of previous AU works [46] [41] [31] [32], the percentages of available
AU annotations are significantly reduced (1%, 2%, 5%, 10%, 20%, 50%, 60%, 70%,
80%, 90%, and 100%) to explore where the limit of KS is. Figure 4 (a), (b),
and (c) shows prominent improvement of KS, especially when extremely limited
annotations are available (1%, 2%, 5% , 10%, 15% and 20% on BP4D; 1%, 2%, and
5% on DISFA; 1%, 2%, 5% , 10%, 15% and 20% on MME). More quantitative reports
are shown in the supplementary.

Table 1: Comparison with state-of-the-art methods using F1 score. The left
table indicates other methods using 100% labeled data. The right table
indicates a baseline model (left) and the proposed KS (right) using different
percentages of labels. Underlines indicate the best result of other methods.
Bold numbers indicate KS surpasses others’ best results. B and D indicates
BP4D and DISFA.

Model |  Reference |  BP4D |  DISFA
---|---|---|---
JAA [35] |  ECCV’18 |  60.0 |  56.0
DSIN [4] |  ECCV’18 |  58.9 |  53.6
LP [28] |  CVPR’19 |  61.0 |  56.9
ARL [34] |  AC’19 |  61.1 |  58.7
SRERL [18] |  AAAI’19 |  62.1 |  55.9
UGN [38] |  AAAI’21 |  63.3 |  60.0
SEV [48] |  CVPR’21 |  63.9 |  58.8
HMP-PS [40] |  CVPR’21 |  63.4 |  61.0
FAUDT [12] |  CVPR’21 |  64.2 |  61.5

Model |  B |  D |  Model |  B |  D
---|---|---|---|---|---
EAC (1%) [20] |  43.8 |  31.8 |  KS (1%) |  59.9 |  49.4
EAC (2%) |  48.7 |  33.3 |  KS (2%) |  62.5 |  52.8
EAC (5%) |  52.2 |  39.4 |  KS (5%) |  63.9 |  56.9
EAC (10%) |  54.8 |  43.9 |  KS (10%) |  64.4 |  58
EAC (15%) |  55.0 |  45.1 |  KS (15%) |  64.5 |  58.8
EAC (50%) |  55.6 |  48.0 |  KS (50%) |  64.3 |  61.4
EAC (60%) |  55.9 |  48.7 |  KS (60%) |  64.4 |  62.6
EAC (100%) |  56.3 |  51.2 |  KS (100%) |  64.5 |  62.8

Table 2: Comparison with state-of-the-art methods using F1 score in terms of
individual AUs. The upper part is the F1 score on BP4D; The bottom part is the
F1 score on DISFA. Bold numbers indicate the best performance.

Model | Used labels | AU1 | AU2 | AU4 | AU6 | AU7 | AU10 | AU12 | AU14 | AU15 | AU17 | AU23 | AU24 | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ARL | 100% | 45.8 | 39.8 | 55.1 | 75.7 | 77.2 | 82.3 | 86.6 | 58.8 | 47.6 | 62.1 | 47.4 | 55.4 | 55.4
SRERL | 100% | 46.9 | 45.3 | 55.6 | 77.1 | 78.4 | 83.5 | 87.6 | 63.9 | 52.2 | 63.9 | 47.1 | 53.3 | 62.9
UGN | 100% | 54.2 | 46.4 | 56.8 | 76.2 | 76.7 | 82.4 | 86.1 | 64.7 | 51.2 | 63.1 | 48.5 | 53.6 | 63.3
HMP-PS | 100% | 53.1 | 46.1 | 56.0 | 76.5 | 76.9 | 82.1 | 86.4 | 64.8 | 51.5 | 63.0 | 49.9 | 54.5 | 63.4
FAUDT | 100% | 51.7 | 49.3 | 61.0 | 77.8 | 79.5 | 82.9 | 86.3 | 67.6 | 51.9 | 63.0 | 43.7 | 56.3 | 64.2
Our KS | 15% | 58.7 | 50.3 | 62.0 | 79.5 | 75.4 | 84.9 | 87.1 | 65.9 | 45.5 | 62.9 | 48.3 | 53.3 | 64.5
Our KS | 100% | 55.1 | 48.9 | 56.2 | 77.3 | 81.8 | 83.3 | 86.4 | 62.6 | 51.9 | 61.3 | 51.0 | 58.3 | 64.5

Model | Used labels |  AU1 |  AU2 |  AU4 |  AU6 |  AU9 |  AU12 |  AU25 |  AU26 | Avg.
---|---|---|---|---|---|---|---|---|---|---
ARL | 100% |  43.9 |  42.1 |  63.6 |  41.8 |  40.0 |  76.2 |  95.2 |  66.8 | 58.7
SRERL | 100% |  45.7 |  47.8 |  59.6 |  47.1 |  45.6 |  73.5 |  84.3 |  43.6 | 55.9
UGN | 100% |  43.3 |  48.1 |  63.4 |  49.5 |  48.2 |  72.9 |  90.8 |  59.0 | 60.0
HMP-PS | 100% |  38.0 |  45.9 |  65.2 |  50.9 |  50.8 |  76.0 |  93.3 |  67.6 | 61.0
FAUDT | 100% |  46.1 |  48.6 |  72.8 |  56.7 |  50.0 |  72.1 |  90.8 |  55.4 | 61.5
Our KS | 15% |  41.7 |  53.5 |  69.7 |  41.3 |  46.2 |  72.0 |  92.3 |  54.0 | 58.8
Our KS | 100% |  53.8 |  59.9 |  69.2 |  54.2 |  50.8 |  75.8 |  92.2 |  46.8 | 62.8

####  4.3.2 Comparison with supervised methods

We report the results under two training setups as [45]: (1) Compare KS
against the fully-supervised state-of-the-art methods with 100% labeled data.
(2) Compare KS against a supervised counterpart under different training label
ratios. As shown in Table 4, a collection of recent and strong benchmark
algorithms are selected for better evaluation. Knowledge-Spreader outperforms
all other advances using only 10% labels on BP4D and 60% labels on DISFA. KS
still performs competitively using only 2% labels on BP4D and 5% labels on
DISFA. In addition, the experiment conducted with 100% labels shows the
effeteness of Knowledge-Spreader in a supervised manner, where the main
contribution comes from Spatial-Temporal information learning module.
Especially, it shows that KS surpasses the best benchmark (FAUDT) by 1.3
f1-score on DISFA. Table 5 further shows the comparison results in terms of
individual AUs. The proposed method performs best on 8 out of 12 AUs on BP4D
and 3 out of 8 AUs on DISFA.

###  4.4 Data Structure Analysis

Figure 5: Evaluation of the label sampling methods. The left top (a) indicates
our strategy, and (b) indicates the conventional method. The left bottom,
right top, and right bottom show the quantitative statistics of non-
overlapping labels and the performance comparison of KS using different
methods on BP4D, DISFA, and MME. X-axis refers to the percentage of used
labels.

Through the statistical experiments, we find the non-overlapping AU
annotations (different AU combinations) account for only a small proportion of
the overall frames number (1693 out of 140,000 frames in BP4D, 102 out of
130,000 frames in DISFA, 748 out of 94,000 frames in MME). A large number of
similar labels and data densely exist across adjacent frames. It reveals that
why using only a few sparsely sampled clips and annotations can achieve
competitive or even better performance, which is consistent with the “less is
better” principle from [16]. Different from existing video-level semi-
supervised works [2] [44] [36] [13] that adopt continuous annotations, we
sparsely sample the annotations and allocate only one annotation for every $k$
frames. Figure 5 demonstrates that applying our method can reserve more non-
overlapping AU annotations than conventional approaches with using the same
number of annotations. Consequently, by utilizing abundant non-overlapping AU
annotations, semi-supervised models testing on BP4D and MME is relatively more
robust to resist the interference caused by missing annotations.

###  4.5 Ablation study

Table 3:  Ablation study on BP4D, DISFA, and MME using F1 score. Bold numbers
indicate the best performance.

Modules | BP4D (5%) | DISFA (5%) | MME (5%) | BP4D (50%) | DISFA (50%) | MME (50%)
---|---|---|---|---|---|---
Knowledge-Spreader | 63.9 | 56.9 | 57.6 | 64.3 | 61.4 | 59.5
KSM+TPL | 61.6 | 54.8 | 55.5 | 62.8 | 60.2 | 58.1
SIL+TPL | 61.7 | 54.6 | 55.8 | 62.5 | 60.1 | 57.7
SIL+KSM | 62.6 | 55.1 | 55.2 | 63.7 | 60.6 | 58.4

Figure 6: Pseudo labels discarded by TPL. The dotted and solid circles
indicate the occurrence of AU6 and AU23. “GT” and “PL” indicate the ground-
truth and the pseudo labels. Red number means incorrect pseudo labels. The
ground-truth labels usually present constant pattern with no change (GT of AU6
in this figure) or simple changes (GT of AU 23 in this figure), while the
incorrect pseudo labels (PL of AU6 and AU23) with temporal perturbation show
the pattern that AU occurs and disappears repeatedly in a short-term period.

Effect of the Spatial-Temporal information learning module (SIL): We replace
the Transformer-based SIL design with a vanilla MPL module to learn the
spatial information and integrate them for learning temporal cues for a fair
comparison. As shown in Table 3, the simple design leads to obvious
performance degradation of KS due to being incapable of learning the AUs
correlation knowledge.

Effect of the knowledge spreading module (KSM): We keep the design of key
frame shifting operation, while remove the spatial knowledge and temporal
knowledge distillation by setting the corresponding loss $L_{s}$ and $L_{t}$
to be unavailable. As a core module of Knowledge-Spreader, it plays an
important role in fully spreading the AU semantic knowledge. There is an
obvious performance gap between the model with and without KSM. For instance,
without this module, the F1 score decreases 2.2, 2.3, and 1.8 on three
databases with 5% labeled data. Similarly, it decreases 1.8, 1.3, and 1.8 with
50% labels. It demonstrates that the performance degradation of the model
without KSM becomes more serious as the affordable labels decrease.

Effect of the temporal confirmed pseudo-label (TPL): As shown in Table 3, with
the assistant of TPL, the performance of KS is improved under different
circumstances. In addition, we compare the accuracy of pseudo labels generated
by TPL and naïve pseudo-label design on BP4D using 10% labels. We get 76.35%
accuracy with TPL and 73.36% with the original pseudo-label, which shows the
TPL achieves prominent improvement by discarding the incorrect pseudo labels
with temporal perturbation. A sample is given in Figure 6 to illustrate how
TPL senses and processes the incorrect pseudo labels. All the above ablation
studies prove that our complete model performs the best by integrating the
three key components.

##  5 Conclusion

In this paper, we propose a new unified Knowledge-Spreader architecture to
learn the interactive Spatial-Temporal correlation knowledge with sparsely
labeled videos by incorporating dynamic knowledge distillation, pseudo-
labeling, and AU semantic encoding modules. Results show that the proposed
model using extremely limited annotations achieves superior performance than
existing methods. Comprehensive studies have demonstrated the key factor that
leads to the success of Knowledge-Spreader in terms of model design and data
structure, in hope of inspiring future works. In addition, a large-scale
dataset for spontaneous and dynamic facial action analysis is introduced to
alleviate the scarcity issue of AU annotation and subject samples. In the
future, we plan to explore the application of Knowledge-Spreader on multi-
modal recognition tasks. This material is based on the work supported by the
US National Science Foundation.

##  6 Supplementary of Model Design

###  6.1 Details of the Spatial-Temporal Information Learning Module

Figure 7: Illustration of transformer-based model $S_{a}$ and model $T_{b}$
for learning spatial and temporal AU correlation.

The Spatial Teacher (or Temporal Student) $S_{a}$ and the Temporal Teacher
$T_{b}$, in Figure 7, are designed for modeling the relationships between AUs
in both the intra-frame and inter-frame levels. In Spatial-Temporal
Information Learning (SIL) module, we first assign $U$ 1D learnable AU
positional embedding to be added with input feature set $D_{s}$ as AU-specific
embedding, where $U$ means the number of facial action units. For $u$th AU-
specific features, a ViT transformer encoder $S_{a}$ generate a set of query,
key, and value tensors ($Q_{u},K_{u},V_{u}$), each with dimension
$\mathbb{R}^{C\times W}$. Afterwards, a $C\times U\times U$ learnable
attention matrix is obtained as follows:

$A^{u}=softmax\left(\frac{Q_{u}K_{u}^{U}}{\sqrt{H}}\right)$ (9)

Similarly, we assign $N$ 1D learnable frame positional embedding to be added
with input feature set $D_{b}$ as frame-specific embedding, where $N$ means
the length of the input sequence. For $n$th frame in a sequence, another set
of query, key, and value tensors ($Q_{n},K_{n},V_{n}$) are generated with the
dimension $\mathbb{R}^{C\times W}$. A $C\times N\times N$ attention matrix is
obtained as follows:

$A^{n}=softmax\left(\frac{Q_{n}K_{n}^{N}}{\sqrt{H}}\right)$ (10)

Through a series of dimensional changes, model $S_{a}$ and $T_{b}$ outputs $U$
dimensional tensors $O_{a}$ and $O_{b}$ respectively. Finally, we take the
mean ensemble output $O_{output}$ of $O_{a}$ and $O_{b}$ for multi-label
prediction.

Besides, we design $N$ small-sized MLP-based models for the Spatial Students
$S_{b}$. The dimension of each output feature $p_{b}$ for Spatial Student is
$C\times 1\times W$.

###  6.2 Q&A for Model Design and Experiments

In this section, we explain some doubts of the model design and experiments.

Why the knowledge spreading operation matters for semi-supervised learning?
How to maximize the use of a small but credible learned knowledge to spread
into a large amount of unknown data is the core problem solved by KS. The
dynamic knowledge spread with two-level distillation, which can also be seen
as a two-level lever, can allow KS to transform a few spatial knowledge into a
large amount of spatial knowledge, and then form more temporal knowledge.
Compared with static knowledge distillation, dynamic knowledge spread contains
multiple level of leverage to infer more out-of-distribution knowledge with
the fewest labels.

What’s the difference between general Spatial-Temporal information and our
Spatial-Temporal AU correlation knowledge? Two attention matrices, in Figure
7, contains the relevance of each atomic AU class and frame-level class. By
modeling the spatial and temporal AU dependency in attention matrices, the
video-level and frame-level AU co-occurrence and mutual exclusive relation is
refined and learned to improve the the general Spatial-Temporal information.

Why do we choose transformer? First, self-attention based methods (i.e.,
JAAnet [35]) have been proved to be very effective in learning AUs semantic
relation. Second, the residual connection, multi-head attention, and
positional embedding designs make it an efficient tool to learn long-range
temporal cues and more discriminative representations. Third, our experiments
show that transformer performs better than other popular models such as
regular GCNs.

Why do we need both consistency regularization and pseudo-labeling? The
consistency regularization is used for building two level knowledge
distillation. It distills the fully modeled spatial AU relationship knowledge
from branch A to branch B, and distill the temporal AU correlation knowledge
from branch B to branch A. However, without pseudo-labeling, some of the
Spatial Students stay idle and the unlabeled data is not fully utilized,
making the training process less efficient. By combining consistency
regularization and pseudo-labeling, KS can accelerate the speed and effect of
knowledge distillation and spread.

Why are Spatial Student models based on MLP instead of transformer? In most
cases, KS need feed Spatial Student models with unlabeled data, which means
the results derived by Spatial Students is less credible. Thus, a weak design
of Spatial Student is necessary. In additional, different design of Spatial
Student amd Spatial Teacher can be recognized as a noise or a model-wise
perturbation for better applying the consistency regularization.

Why do we need the temporal constrain for pseudo-labeling? First, automatic AU
detection, as a multi-label task, faces some difficulties in selecting and
retaining the pseudo labels when only partial labels are with high confidence
score. Compared with multi-class tasks, the one-hot format of multi-labels
makes it hard to decide if the pseudo labels are confident in a holistic way
or a local way. Setting the temporal constrain makes it easier to filter the
cases which are not consist with the regular pattern of facial action
movements. Second, the temporal label smoothness is also a soft constrains
from human knowledge for better generalizing out-of-distribution AU data.

How to generate pseudo labels? We pick up the class which has maximum
predicted probability for each binary class of unlabeled samples.

Can KS be trained with video clips without any labels? If yes, how? Yes, it
can. When feeding KS with the unlabeled video clips, we only update our model
with the loss of pseudo-labeling and Temporal Teacher. It worth noting that
the unlabeled video clips are not allowed to use before the model training is
stable (10 epochs in this project). Otherwise learning unreliable knowledge
first will lead the model to the error-prone issue.

Why using 20% labels achieves nearly the same performance as using 100% on
BP4D? This is due to the large portion of overlapped annotations, using 20%
labels with sparsely sampled annotations makes the performance reach the
“saturation” quickly, hence there is no significant performance gain after 20%
towards the use of 100% labels. Our finding complies with the “less is better”
principle confirmed by the other existing works [16].

How about applying different sequential perturbation for the pseudo label
confirmation module? The fact is that using or mixing certain perturbations
(e.g., temporal feature shift, random mask, and flip) does not bring obvious
performance improvement. We speculate that other perturbations can not model
the temporal fluctuations caused by incorrect pseudo labels well. Applying
inappropriate or excessive perturbation operation can even degrade the
performance.

###  6.3 Inference Strategy

We adopt the inductive learning manner at testing stage. A random key frame
number is assigned to each batch of input video clips. Given a video clip,
both branch A and branch B are used to extract the feature $O_{output}$ for
prediction.

###  6.4 Additional Implementation Details

All training images in the same video clips are randomly rotated (-45 to 45
degrees), flipped horizontally (50% possibility), and with color jitters
(saturation, contrast, and brightness) simultaneously. The detailed
specification of Knowledge-Spreader is shown in the original code (model
designing part). The complete code will be released to the research community
by the time of the paper being published. We choose 5 as the frame length of
each input video clip for the optimal time-and-accuracy trading-off. The
analysis of the hyper-parameters can be seen in Figure 8. We implement our
Knowledge-Spreader (KS) with the Pytorch framework and perform training and
testing on the NVIDIA GeForce 2080Ti GPU.

###  6.5 Additional Quantitative Evaluation

The quantitative results with different label ratios are shown in Table 4 for
reference. It corresponds to Figure 4 in the original paper. In addition, due
to the page limitation, only partial comparison results with supervised
methods in terms of individual AU are shown in Table 2 of the original paper.
Table 5 shows the complete comparison results.

Table 4: Quantitative comparison with semi-supervised methods using F1 score.
Underlines indicate the best results of individual models.

Model |  BP4D |  DISFA |  MME
---|---|---|---
Pseudo-label (1%) |  54.3 |  40.4 |  45.8
Pseudo-label (2%) |  57.8 |  50.8 |  47.5
Pseudo-label (5%) |  59.7 |  51.5 |  52.1
Pseudo-label (10%) |  60.7 |  56.8 |  54.2
Pseudo-label (15%) |  61.2 |  57.1 |  54.9
Pseudo-label (20%) |  62 |  58.5 |  55.2
Pseudo-label (50%) |  63.6 |  57.9 |  55.3
Pseudo-label (60%) |  62.7 |  56.7 |  55.3
Pseudo-label (70%) |  63.3 |  57.9 |  55.3
Pseudo-label (80%) |  62.4 |  58.3 |  56.6
Pseudo-label (90%) |  62.3 |  57.5 |  55.5
Pseudo-label (100%) |  62.7 |  58.8 |  56.9

Model |  BP4D |  DISFA |  MME
---|---|---|---
FixMatch (1%) |  49.9 |  35.6 |  41.6
FixMatch (2%) |  55.1 |  46.2 |  46.5
FixMatch (5%) |  59.2 |  52.7 |  52.6
FixMatch (10%) |  60.5 |  55 |  55.4
FixMatch (15%) |  62.1 |  57.7 |  55.6
FixMatch (20%) |  62 |  58.4 |  56.4
FixMatch (50%) |  62 |  57.9 |  58.3
FixMatch (60%) |  62.1 |  56 |  56.4
FixMatch (70%) |  61.9 |  57.8 |  57.2
FixMatch (80%) |  62.2 |  56.9 |  55.5
FixMatch (90%) |  61.9 |  57.5 |  55.3
FixMatch (100%) |  62.7 |  58.8 |  56.9

Model |  BP4D |  DISFA |  MME
---|---|---|---
TCL (1%) |  55.6 |  42.3 |  43.3
TCL (2%) |  58.9 |  51.2 |  48.2
TCL (5%) |  60.5 |  53.6 |  53.4
TCL (10%) |  61.7 |  55.8 |  55.7
TCL (15%) |  62.3 |  56.7 |  56.2
TCL (20%) |  62.7 |  57.9 |  55.6
TCL (50%) |  63.2 |  59.2 |  57.6
TCL (60%) |  62.8 |  60.1 |  57.9
TCL (70%) |  63.0 |  59.6 |  57.9
TCL (80%) |  62.9 |  60.4 |  58.3
TCL (90%) |  62.7 |  58.3 |  57.8
TCL (100%) |  63.1 |  59.7 |  58.1

Model |  BP4D |  DISFA |  MME
---|---|---|---
Our KS (1%) |  59.9 |  64.9 |  51.2
Our KS (2%) |  62.5 |  52.8 |  54.8
Our KS (5%) |  63.9 |  56.9 |  57.6
Our KS (10%) |  64.4 |  58 |  58.4
Our KS (15%) |  64.5 |  58.8 |  58.7
Our KS (20%) |  64.6 |  59.5 |  58.9
Our KS (50%) |  64.3 |  61.4 |  59.5
Our KS (60%) |  64.4 |  62.6 |  59.2
Our KS (70%) |  64.5 |  61.9 |  58.8
Our KS (80%) |  64.4 |  62 |  59.3
Our KS (90%) |  64.4 |  61.9 |  59.4
Our KS (100%) |  64.5 |  62.8 |  59.7

Table 5: Comparison with state-of-the-art methods using F1 score in terms of
individual AUs. The upper part is the F1 score on BP4D; The bottom part is the
F1 score on DISFA. Bold numbers indicate the best performance.

Model | Used labels | AU1 | AU2 | AU4 | AU6 | AU7 | AU10 | AU12 | AU14 | AU15 | AU17 | AU23 | AU24 | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
DSIN | 100% | 51.7 | 40.4 | 56.0 | 76.1 | 73.5 | 79.9 | 85.4 | 62.7 | 37.3 | 62.9 | 38.8 | 41.6 | 58.9
JAA | 100% | 47.2 | 44.0 | 54.9 | 77.5 | 74.6 | 84.0 | 86.9 | 61.9 | 43.6 | 60.3 | 42.7 | 41.9 | 60.0
LP | 100% | 43.4 | 38.0 | 54.2 | 77.1 | 76.7 | 83.8 | 87.2 | 63.3 | 45.3 | 60.5 | 48.1 | 54.2 | 61.0
ARL | 100% | 45.8 | 39.8 | 55.1 | 75.7 | 77.2 | 82.3 | 86.6 | 58.8 | 47.6 | 62.1 | 47.4 | 55.4 | 55.4
SRERL | 100% | 46.9 | 45.3 | 55.6 | 77.1 | 78.4 | 83.5 | 87.6 | 63.9 | 52.2 | 63.9 | 47.1 | 53.3 | 62.9
UGN | 100% | 54.2 | 46.4 | 56.8 | 76.2 | 76.7 | 82.4 | 86.1 | 64.7 | 51.2 | 63.1 | 48.5 | 53.6 | 63.3
HMP-PS | 100% | 53.1 | 46.1 | 56.0 | 76.5 | 76.9 | 82.1 | 86.4 | 64.8 | 51.5 | 63.0 | 49.9 | 54.5 | 63.4
FAUDT | 100% | 51.7 | 49.3 | 61.0 | 77.8 | 79.5 | 82.9 | 86.3 | 67.6 | 51.9 | 63.0 | 43.7 | 56.3 | 64.2
Our KS | 15% | 58.7 | 50.3 | 62.0 | 79.5 | 75.4 | 84.9 | 87.1 | 65.9 | 45.5 | 62.9 | 48.3 | 53.3 | 64.5
Our KS | 100% | 55.1 | 48.9 | 56.2 | 77.3 | 81.8 | 83.3 | 86.4 | 62.6 | 51.9 | 61.3 | 51.0 | 58.3 | 64.5

Model | Used labels |  AU1 |  AU2 |  AU4 |  AU6 |  AU9 |  AU12 |  AU25 |  AU26 | Avg.
---|---|---|---|---|---|---|---|---|---|---
DSIN | 100% |  42.4 |  39.0 |  68.4 |  28.6 |  46.8 |  70.8 |  90.4 |  42.2 | 53.6
JAA | 100% |  43.7 |  46.2 |  56.0 |  41.4 |  44.7 |  69.6 |  88.3 |  58.4 | 56.0
LP | 100% |  29.9 |  24.7 |  72.7 |  46.8 |  49.6 |  72.9 |  93.8 |  65.0 | 56.9
ARL | 100% |  43.9 |  42.1 |  63.6 |  41.8 |  40.0 |  76.2 |  95.2 |  66.8 | 58.7
SRERL | 100% |  45.7 |  47.8 |  59.6 |  47.1 |  45.6 |  73.5 |  84.3 |  43.6 | 55.9
UGN | 100% |  43.3 |  48.1 |  63.4 |  49.5 |  48.2 |  72.9 |  90.8 |  59.0 | 60.0
HMP-PS | 100% |  38.0 |  45.9 |  65.2 |  50.9 |  50.8 |  76.0 |  93.3 |  67.6 | 61.0
FAUDT | 100% |  46.1 |  48.6 |  72.8 |  56.7 |  50.0 |  72.1 |  90.8 |  55.4 | 61.5
Our KS | 15% |  41.7 |  53.5 |  69.7 |  41.3 |  46.2 |  72.0 |  92.3 |  54.0 | 58.8
Our KS | 100% |  53.8 |  59.9 |  69.2 |  54.2 |  50.8 |  75.8 |  92.2 |  46.8 | 62.8

###  6.6 Effect of the Video Clip Length $n$

Figure 8: Effect of the video clip length $n$. (a) and (b) indicates the F1
score with different video clip lengths on BP4D and DISFA.

To investigate the influence of the input clip length, we perform experiments
by the proposed model with 10% sparsely sampled annotations on BP4D and DISFA.
Figure 8 shows the F1 score curve with $n$ changes. Overall, the performance
improve with the $n$ increases from 2 to a certain threshold. A long video
clip, on the other hand, results in high computational and memory costs. For
optimal trading-off, 5 to 7 is a proper setting for the video clip length $n$.

###  6.7 Parameter Scale Analysis

The trainable parameter size of the proposed model is around 25 million, which
makes KS a very light-weighted model. Compared with the baseline algorithm
EACnet [20], which contains 138 million parameters, Knowledge-Spreader, as a
video-level model, reduces considerable parameter (80%) but achieves excellent
performance improvement.

##  7 Supplementary of New MME

###  7.1 Participants

233 participants were recruited from our University. There are 132 females and
101 males, with ages ranging from 18 to 70 years old. Ethnic/Racial Ancestries
include Asian, Black, Hispanic/Latino, White, and others (e.g., Native
American).

###  7.2 Recording System and Synchronization

Our data collection system consists of a 3D dynamic imaging camera system, a
thermal sensor, a physiological signal sensor system, and a studio-quality
audio recorder. The system setup and synchronization method are basically
consistent with BP4D+ [51].

###  7.3 Emotion Stimulus

Table 6:  The stimulus tasks designed for the data collection.

Task ID | Activity | Target Emotion
---|---|---
1 | Have a pleasant chat with the interviewer | Happiness
2 | Watch a 3D face model of the participant | Surprise
3 | Watch an audio recording of 911 emergency call | Sadness
4 | Experience a sudden sound from a horn | Startle or Surprise
5 | React to a fake news | Skeptical
6 | Asked to sing an impromptu song | Embarrassment
7 | Experience physical fear of the threat in a dart game | Fear or Nervous
8 | Experience the cold feeling by submerging |
| hands into a bucket with ice water | Pain
9 | React to the blame from the interviewer | Offended or Unpleasant
10 | Experience a bad smell from decaying food | Disgust

Ten tasks were performed to elicit a wide range of spontaneous emotion
expression (from positive, to neutral, and to negative) and inter-personal
facial action behavior by a professional interviewer. Table 6 illustrates the
detailed description for the designed tasks.

###  7.4 Data Organization

Each subject is associated with 10 different emotions and multi-modal data
including the 3D sequence, 2D RGB sequence, thermal sequence, and the
sequences of physiological data (i.e., blood pressure, EDA, heart rate, and
respiration rate). The sample sequences of different modalities from two
subjects are shown in Figure 9. Besides, the metadata including manually
labeled action units occurrence and intensity, 3D/2D/IR facial landmarks, and
3D head poses are also generated for better analysis of automatic human facial
action.

Figure 9: A sample sequence from our MME. 2D texture image, 3D mesh model, 3D
shaded model, 3D texture model, thermal image, and physiological signal
(respiration rate, blood pressure, EDA, heart rate) and corresponding AU
occurrence are shown from top to bottom.

## References

  * [1] Brock, A., De, S., Smith, S.L., Simonyan, K.: High-performance large-scale image recognition without normalization. arXiv preprint:2102.06171 (2021)
  * [2] Choi, J., Sharma, G., Chandraker, M., Huang, J.B.: Unsupervised and semi-supervised domain adaptation for action recognition from drones. In: WACV (2020)
  * [3] Chu, W.S., De la Torre, F., Cohn, J.F.: Learning spatial and temporal cues for multi-label facial action unit detection. In: 2017 12th IEEE International Conference on Automatic Face Gesture Recognition (FG) (2017)
  * [4] Corneanu, C., et al.: Deep structure inference network for facial action unit recognition. In: Proceedings of ECCV (2018)
  * [5] Cui, Z., Song, T., Wang, Y., Ji, Q.: Knowledge augmented deep neural networks for joint facial expression and action unit recognition. In: NIPS (2020)
  * [6] Cui, Z., Zhang, Y., Ji, Q.: Label error correction and generation through label relationships. Proceedings of the AAAI Conference on Artificial Intelligence pp. 3693–3700 (2020)
  * [7] Dosovitskiy, A., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint:2010.11929 (2020)
  * [8] Guo, Q., Wang, X., Wu, Y., Yu, Z., Liang, D., Hu, X., Luo, P.: Online knowledge distillation via collaborative learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [9] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [10] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015)
  * [11] Huang, Y., et al.: Gpipe: Efficient training of giant neural networks using pipeline parallelism. arXiv preprint:1811.06965 (2019)
  * [12] Jacob, G.M., Stenger, B.: Facial action unit detection with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [13] Jing, L., Parag, T., Wu, Z., Tian, Y., Wang, H.: Videossl: Semi-supervised learning for video classification. In: WACV (2021)
  * [14] Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. In: 5th International Conference on Learning Representations, ICLR (2017)
  * [15] Lee, D.H.: Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop : Challenges in Representation Learning (WREPL) (2013)
  * [16] Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T.L., Bansal, M., Liu, J.: Less is more: Clipbert for video-and-language learning via sparse sampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [17] Li, G., Zhu, X., Zeng, Y., Wang, Q., Lin, L.: Semantic relationships guided representation learning for facial action unit recognition. Proceedings of the AAAI Conference on Artificial Intelligence (2019)
  * [18] Li, G., et al.: Semantic relationships guided representation learning for facial action unit recognition. In: AAAI (2019)
  * [19] Li, W., Abtahi, F., Zhu, Z.: Action unit detection with region adaptation, multi-labeling learning and optimal temporal fusing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
  * [20] Li, W., Abtahi, F., Zhu, Z., Yin, L.: Eac-net: Deep nets with enhancing and cropping for facial action unit detection. IEEE Transactions on Pattern Analysis and Machine Intelligence (2018)
  * [21] Li, X., Li, Z., Yang, H., Zhao, G., Yin, L.: Your “attention” deserves attention: A self-diversified multi-channel attention for facial action analysis. In: 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (2021)
  * [22] Li, Y., Zeng, J., Shan, S., Chen, X.: Self-supervised representation learning from videos for facial action unit detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
  * [23] Li, Z., Deng, X., Li, X., Yin, L.: Integrating Semantic and Temporal Relationships in Facial Action Unit Detection (2021)
  * [24] Li, Z., Deng, X., Li, X., Yin, L.: Integrating Semantic and Temporal Relationships in Facial Action Unit Detection (2021)
  * [25] Mavadati, S.M., et al.: Disfa: A spontaneous facial action intensity database. IEEE Transactions on Affective Computing 4(2), 151–160 (2013)
  * [26] Mei, C., Jiang, F., Shen, R., Hu, Q.: Region and temporal dependency fusion for multi-label action unit detection. In: 2018 24th International Conference on Pattern Recognition (ICPR) (2018)
  * [27] Niu, X., Han, H., Shan, S., Chen, X.: Multi-label co-regularization for semi-supervised facial action unit recognition. In: NeurIPS (2019)
  * [28] Niu, X., et al.: Local relationship learning with person-specific shape regularization for facial action unit detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
  * [29] Pantic, M., Valstar, M., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: 2005 IEEE International Conference on Multimedia and Expo. pp. 5 pp.– (2005)
  * [30] Park, H., Yoo, J., Jeong, S., Venkatesh, G., Kwak, N.: Learning dynamic network using a reuse gate function in semi-supervised video object segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [31] Peng, G., Wang, S.: Weakly supervised facial action unit recognition through adversarial training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 2188–2196 (2018)
  * [32] Peng, G., Wang, S.: Dual semi-supervised learning for facial action unit recognition. Proceedings of the AAAI Conference on Artificial Intelligence (2019)
  * [33] Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. International Journal of Computer Vision (2014)
  * [34] Shao, Z., et al.: Facial action unit detection using attention and relation learning. IEEE Transactions on Affective Computing p. 1–1 (2019)
  * [35] Shao, Z., et al.: Deep adaptive attention for joint facial action unit detection and face alignment. In: Proceedings of ECCV (2018)
  * [36] Singh, A., Chakraborty, O., Varshney, A., Panda, R., Feris, R., Saenko, K., Das, A.: Semi-supervised action recognition with temporal contrastive learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [37] Sohn, K., Berthelot, D., Li, C.L., Zhang, Z., Carlini, N., Cubuk, E.D., Kurakin, A., Zhang, H., Raffel, C.: Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In: NIPS (2020)
  * [38] Song, T., Chen, L., Zheng, W., Ji, Q.: Uncertain graph neural networks for facial action unit detection. Proceedings of the AAAI Conference on Artificial Intelligence pp. 5993–6001 (2021)
  * [39] Song, T., Cui, Z., Zheng, W., Ji, Q.: Hybrid message passing with performance-driven structures for facial action unit detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [40] Song, T., Cui, Z., Zheng, W., Ji, Q.: Hybrid message passing with performance-driven structures for facial action unit detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [41] Song, Y., McDuff, D., Vasisht, D., Kapoor, A.: Exploiting sparsity and co-occurrence structure for action unit recognition. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) (2015)
  * [42] Sánchez-Lozano, E., Tzimiropoulos, G., Valstar, M.: Joint action unit localisation and intensity estimation through heatmap regression. In: BMVC (2018)
  * [43] Tang, Y., Zeng, W., Zhao, D., Zhang, H.: Piap-df: Pixel-interested and anti person-specific facial action unit detection net with discrete feedback learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2021)
  * [44] Wang, X., Zhang, S., Qing, Z., Shao, Y., Gao, C., Sang, N.: Self-supervised learning for semi-supervised temporal action proposal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [45] Wang, X., Zhang, S., Qing, Z., Shao, Y., Gao, C., Sang, N.: Self-supervised learning for semi-supervised temporal action proposal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1905–1914 (June 2021)
  * [46] Wu, S., Wang, S., Pan, B., Ji, Q.: Deep facial action unit recognition from partially labeled data. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)
  * [47] Xie, S., et al.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE Conference on CVPR (2017)
  * [48] Yang, H., Yin, L., Zhou, Y., Gu, J.: Exploiting semantic embedding and visual feature for facial action unit detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [49] Zhang, X., Yin, L., Cohn, J.F., Canavan, S., Reale, M., Horowitz, A., Liu, P., Girard, J.M.: Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database. Image and Vision Computing 32(10), 692–706 (2014)
  * [50] Zhang, Y., Wu, B., Dong, W., Li, Z., Liu, W., Hu, B.G., Ji, Q.: Joint representation and estimator learning for facial action unit intensity estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019)
  * [51] Zhang, Z., Girard, J.M., Wu, Y., Zhang, X., Liu, P., Ciftci, U., Canavan, S., Reale, M., Horowitz, A., Yang, H., Cohn, J.F., Ji, Q., Yin, L.: Multimodal spontaneous emotion corpus for human behavior analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [52] Zhang, Z., Wang, T., Yin, L.: Region of interest based graph convolution: A heatmap regression approach for action unit detection. In: ACM MM (2020)
  * [53] Zhao, K., Chu, W.S., De la Torre, F., Cohn, J.F., Zhang, H.: Joint patch and multi-label learning for facial action unit detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)