# Assur graphs, marginally jammed packings, and reconfigurable metamaterials

Jose Ortiz  Ethan Stanifer  Xiaoming Mao  Department of Physics, University of
Michigan, Ann Arbor, MI 48109-1040, USA

###### Abstract

Isostatic frames are mechanical networks that are simultaneously rigid and
free of self-stress states, and is a powerful concept in understanding phase
transitions in soft matter and designing of mechanical metamaterials. Here we
analyze substructures of isostatic frames, by generalizing “Assur graphs” to
the torus and examine them in physical systems. We show that the contact
network of marginally jammed packings approach torus Assur graphs in the
thermodynamic limit, and demostrate how Assur graphs offer a new design
principle for mechanical metamaterials in which motion and stress can
propagate in reconfigurable pathways, while rigidity of the entire structure
is maintained.

_Introduction.—_ The concept of rigidity is central to both soft matter
physics, from gelation Zhang2019 ; colombo2014stress , jamming OHern2003  to
mechanical transitions in biological tissues bi2015density , and the design of
mechanical metamaterials where deformation and stress responses are programmed
via geometry and topology Lubensky2015 ; Mao2018 . Given the tensorial nature
of rigidity (as opposed to connectivity or conductivity which are scalars),
rigidity transitions can take many forms characterized by distinct
universality classes. Among them, the special point of “isostaticity”, where
the numbers of nontrivial zero modes (ZMs, normal modes of zero energy) and
states of self stress (SSSs, force-balanced stress eigenstates) both vanish,
characterizes a type of mechanical critical point where the whole system is
coordinated in a unique way where all degrees of freedom are marginally
constrained laman1970graphs ; asimow1978rigidity ; alexander1998amorphous . It
is known that the jamming transition of athermal frictionless repulsive disks
(or spheres in 3D) occurs at the isostatic point OHern2003 ; van2009jamming ;
liu2010jamming ; goodrich2016scaling , whereas many other rigidity transitions
don’t jacobs1995generic ; donev2004linear ; Mailman2009 ;
broedersz2011criticality ; Ellenbroek_2011 ; bi2011jamming ; Yan2014 ;
rigidity_loss_three ; Henkes2016 ; behringer2018physics ; Zhang2019 .

It is known to the mechanics and mathematics literature that pinned isostatic
frames can have substructures, called “Assur graphs”, which define minimal
components that determine the propagation of motion and stress (Fig. 1)
assur1952issledovanie ; DigraphsDecomp ; pinned_pebble_game ; uniqueenginprops
; PENNE199437 . Interestingly, recently it was found that marginally jammed
packings (MJPs) of frictionless repulsive disks are not only isostatic but
this isostaticity is “global” rigidity_loss_three , whereas computational
modular representations have been proposed to characterize mechanical networks
with floppy motion chen2022modular .

Here we generalize Assur decomposition from the case of pinned graphs to
graphs on the torus, and apply it to MJPs under periodic boundary conditions
(PBC) to reveal that their contact networks not only approach isostaticity,
but also _minimal isostaticity_ , in the thermodynamic limit, therefore
partially explaining the observation in Ref. rigidity_loss_three . We also
prove that this is a necessary consequence of purely repulsive interactions in
the special case of jamming in circular containers. Furthermore, we propose a
new design principle based on Assur decomposition for mechanical metamaterials
in which the switch of one connection can sharply and remotely control the
range of motion and stress propagation.

_Assur decomposition and minimal isostatic graphs (MIGs).—_ We start by
reviewing the notion of generic (i.e., combinatorial) rigidity. The mechanical
properties of a frame of vertices (point masses) and edges (springs) denoted
as $(G,\mathbf{X})$, come from both its underlying graph $G$ and its geometry
in terms of positions $\mathbf{X}$ of the vertices. Generic rigidity only
depends on $G$. A graph $G$ is said to be rigid if it can be realized into a
frame $(G,\mathbf{X})$ that is infinitesimally rigid (i.e., no nontrivial ZMs
in the linear theory). Geometries $\mathbf{X}$ that make a generically rigid
graph $G$ nonrigid are called geometric singularities. An isostatic graph is a
rigid graph in which the removal of any edge leads to a generic ZM (i.e., a ZM
in generic realizations).

Figure 1:  Assur decomposition for pinned and torus isostatic graphs. (a) A
pinned isostatic graph with its 6 Assur components identified. (b) Partial
order of the Assur components of the graph in (a), with two isostatic
subgraphs identified with boxes and the directions for the propagation of
motion and stress marked. In the following panels the pruned edges are marked
by dashed magenta lines, external forces represented as the addition of a
redundant edge to ground (magenta cross). The resulting ZM is plotted in
orange arrows (with the displacement amplitude proportional to the length of
the line excluding the arrowheads) and the resulting stress in red (tension)
and blue (compression). (c) Pruning an edge in component 4 results in a ZM
that only moves component 4 and its descendants. (d) Pruning an edge in
component 1 results in a full ZM that moves all vertices. e) External force on
a vertex of component 4 stresses component 4 and its ancestors. f) External
force on component 1 stresses only component 1. (g,h) Assur decompositions of
a torus isostatic graph, where different choices of the ground (black “0”
vertex with 2 pebbles) led to different decompositions. PBC boxes are shown as
black dashed squares. This ambiguity is avoided in the torus PGA algorithm we
use (see SM).

An MIG is an isostatic graph with no proper rigid subgraphs
assur1952issledovanie ; DigraphsDecomp ; pinned_pebble_game ; uniqueenginprops
; PENNE199437 . The removal of any portion of a MIG results in a global loss
of rigidity (otherwise the part that remained rigid would be a proper rigid
subgraph). So far all studies of MIGs are either on pinned or unpinned graphs
on the plane. Fig. 1a shows an example of a pinned isostatic graph, and we
orient the graph, assign direction to each edge, by using the pebble game
algorithm (PGA) for pinned graphs pinned_pebble_game  based on Laman’s theorem
laman1970graphs . Here all internal vertices have in-degree of 2, representing
the 2 degrees of freedom (DOFs, i.e., the 2 “pebbles”) of that vertex being
constrained by the corresponding oriented edges. The ground has in-degree 0
(no inherent DOFs). Such a result of the PGA where all edges are oriented and
all internal vertices have in-degree 2 indicates that all the DOFs are paired
with corresponding constraints and there are no redundant edges. The existence
of this orientation is sufficient and necessary for pinned isostaticity and
the orientation is unique up to the reversal of cycles DigraphsDecomp .
Important information is carried by this 2-in orientation: upon the removal of
one edge the motion of each vertex is determined by its two immediate
ancestors (vertices upstream), and this information is then passed to its
descendants (vertices downstream), as shown in Fig. 1c,d. Conversely, stress
(i.e., violation of length constraints from the edges) travels upstream as
shown in Fig. 1e,f.

The orientation of a pinned isostatic graph gives a decomposition of this
graph into strongly connected components (defined as clusters in which every
vertex can be reached from every other vertex along directed edges): the so-
called Assur components pinned_pebble_game ; DigraphsDecomp . This
decomposition identifies _clusters on the graph where their motion or stress
must emerge or disappear in a synchronous way_. Crucially, this decomposition
tells us all possible pinned isostatic subgraphs. This is because Assur
components of an isostatic graph have a partial order according to the
orientation such as depicted in Fig. 1b. Isostatic subgraphs correspond to
subsets of these Assur components such that for every component all of its
ancestors are also in the subset.

This decomposition provides a lower block triangular form for the
compatibility matrix $C$, which is defined as the linear map from vertex
displacements $u$ to edge extensions $e$, such that $e=C\cdot u$. When the
rows and columns of the compatibility matrix are arranged according to this
partial order, it becomes lower block triangular,

$C=\left(\begin{matrix}C^{(1)}&0&\cdots&0\\\ C^{(2,1)}&C^{(2)}&\cdots&0\\\
\vdots&\vdots&\ddots&\vdots\\\ C^{(m,1)}&C^{(m,2)}&\cdots&C^{(m)}\\\
\end{matrix}\right),$ (1)

where each block in the diagonal is square and full rank and identified with
one Assur component. Mechanically, this tells us that when an edge in
component $i$ is pruned, the resulting ZM will involve all vertices in
components $j\geq i$, and the addition of any new edge in component $i$
(either between nodes in $i$ or connecting them to the ground) will introduce
an SSS that involves all edges in components $j\leq i$ (because the
equilibrium matrix $Q=C^{T}$, is upper block triangular).

Therefore, for a pinned isostatic graph, (i) it contains no proper isostatic
subgraphs, (ii) it is indecomposable (i.e. strongly connected for any 2-in
directed orientation), (iii) the compatibility matrix has no proper block
triangular decomposition, (iv) removal of any edge results in a ZM that moves
all vertices, and (v) forces exerted on any vertex stresses all edges, are all
equivalent statements, and graphs satisfying these conditions are called Assur
graphs or MIGs DigraphsDecomp ; uniqueenginprops .

We next generalize these concepts to frames on the torus, as PBC is adopted in
most studies of jamming. In this case, the 2 trivial translation ZMs always
remain, and Laman’s theorem for isostaticity has been generalized in Ref.
Rossthesis  such that for a graph of $N$ vertices to be generically isostatic
when embedded on the torus (“torus isotatic” for short), it must be both
$(2,2)$-tight [i.e., it has $2N-2$ edges and $(2,2)$-sparse (no subgraph with
$n$ vertices has more than $2n-2$ edges)], and all $(2,2)$-tight subgraphs
wrap around the torus (i.e., are embedded constructively, following the notion
of Ref. Rossthesis ). We use the torus PGA algorithm devised in Ref.
Rossthesis  to obtain orientations of our graphs and from them Assur
decompositions (Fig. 1gh).

We thereby generalize the definition of MIGs to torus isostatic graphs, along
with their unique properties in the following theorem [see the Supplementary
Material (SM) for the proof].

######  Theorem 1.

Given a graph $G$ that is isostatic on the torus, the following are equivalent

  1. a

$G$ has no proper subgraphs that are isostatic on the torus.

  2. b

Any orientation of $G$ where the in degree of all but one vertex is 2 is
strongly connected on all but that one vertex.

  3. c

Removal of any edge results in a generic ZM that moves all vertices relative
to each other.

  4. d

A generic torque on any edge stresses all bonds

  5. e

Adding one edge introduces an SSS that either stresses all edges or only a
non-constructive subgraph.

Figure 2: Torus Assur decomposition of jammed packings. (a) An MJP contains a
large MIG with almost all vertices. There are two redundant edges in this
contact network (black dashed line). (b) A torus isostic graph obtained by
pruning redundant edges (black dashed lines) from a packing above the jamming
transition contains a large number of Assur components. (Same convention for
PBC and ground as in Fig. 1gh.) (c) The fraction $\eta$ of vertices belonging
to the MIG, as a function of total number of vertices in the graph. The
removal of rattlers led to horizontal error bars which are too small to be
visible. Vertical error bars indicate 2nd and 98th percentile, from the
statistics of testing $100$ packings at each system size, and different
choices of the 2 uncovered edges in each packing. The inset shows the
statistics of the number of vertices not in the MIG, which appears independent
of system size. (d) A jammed packing (close to marginal) in a circular
container, with a full SSS shown.

_Assur decomposition of marginally jammed packings.—_ We first prepare jammed
packings of soft frictionless disks of one-sided Hertzian repulsion by
starting from a random configuration with high volume fraction ($\phi=0.90$),
and decompress until we have only 2 excess contacts. At each step the energy
is minimized using the FIRE algorithm. For a system of $N$ disks, isostaticity
on a torus is reached when the system has $2N-2$ contacts (leaving the 2
trivial translations). This exact state is very hard to reach in large
systems, so we choose to terminate decompression at 2 SSSs ($2N$ contacts) and
obtain MJPs at various system sizes. We then run the torus PGA Rossthesis  on
the contact network, which leaves 2 uncovered edges (i.e., the redundant
contacts) and 2 free pebbles (i.e., the 2 trivial ZMs) on the torus. The
covered portion of the contact network, which is now directed, is spanning and
torus isostatic.

We then find the Assur components of this spanning isostatic graph, using the
strongly connected decomposition adapted for graphs on torus as detailed in
the SM. This generally results in a first Assur component, an MIG by
definition from Th. 1, covering the entire graph $G$ except a few “diads” (an
Assur component of 1 vertex and 2 edges), as shown in Fig. 2a. We perform this
analysis at different system sizes, and collect the fraction $\eta$ of
vertices in this MIG (Fig. 2c). We find that $\eta=1-\mathcal{O}(N^{-1})$ as
$N\to\infty$, meaning that the number of vertices in $G$ not belonging to the
MIG does not grow with system size. The choice of the ground does not change
the Assur decomposition in the case of the MJPs, as we show in the SM.

The fact that almost all nodes belong to the one MIG is a unique property of
MJPs. This can be shown by taking other isostatic graphs, e.g., dense packings
above the jamming transitition and randomly pruning redundent edges in the
contact network until isostaticity is reached. This results in a large number
of small Assur components (Fig. 2b), in direct contrast with the case of the
MJPs.

_Relation between minimal isostaticity and purely repulsive interactions.—_ It
is natural to ask whether the minimal isostaticity of MJPs come from the fact
that the disks assemble under purely repulsive interactions. Interestingly we
have an example where this is indeed the case. Consider a set of repulsive
disks in a hard frictionles circular container (Fig. 2d). Contacts of disks
with the wall can be represented as edges to the ground, making the contact
network a pinned frame. A system of fully repulsive disks is only stable if
there is an SSS that involves all disks and is compressive at all contacts. At
the same time, any packing in a circular container has a ZM that involves all
disks: a global rotation. Therefore any MJP in a circular container has a full
SSS and a full ZM. The existence of a realization that satisfies this
condition is sufficient and necessary for the graph to be a pinned MIG
GeometryAssur . Thus, minimal isostaticity emerges as a consequence of the
repulsive nature of the interaction in this case. One may conjecture that in
the thermodynamic limit, the difference between circular container and torus
becomes unimportant, thereby extending the conclusion to the case of PBC.
However we don’t have a rigorous proof of this argument.

_Relation to observations of global response in Ref.rigidity_loss_three .—_
Our findings here partially explain the observation that the removal of any
contact makes all vertices in the contact network hinges, and the introduction
of any new edge stresses all contacts. This closely relates to Th. 1$c$ and
$e$. The key difference is the existence of non-constructive plane isostatic
subgraphs [also known as Laman or (2,3)-tight] with number of vertices
$N_{ISG}>3$ (the cases of $N_{ISG}=2$ or $3$ are simply edges and triangles,
which don’t affect vertices becoming hinges and can’t be stressed by the
addition of a new edge). If there were no such subgraphs and the network was a
MIG on the torus, observations in Ref. rigidity_loss_three  follow. However if
an MIG on the torus had such subgraphs, removal of any edge that is not in
such a non-constructive plane isostatic subgraph would make this subgraph,
along with all other vertices, mobile, but internal vertices in this subgraph
would not be hinges, and adding one edge to this subgraph would only stress
this subgraph. This is a generalization of the minimal isostaticity discussed
by Penne PENNE199437  where the graph in question is isostatic on the plane
and the only allowed plane isostatic graphs are single bonds. Thus results in
Ref. rigidity_loss_three  indicates that it is very rare to find any large
plane isostatic subgraphs in MJPs, making these networks akin to Penne’s MIGs.

Figure 3: Assur-decomposition based design principle for reconfigurable
mechanical metamaterials. (a) Given an isostatic graph, the flip of one edge
(black dashed lines) leads to global changes in the Assur Decomposition. (b)
The ZMs produced by the pruning of the same edge (magenta dashed line) before
and after the flip. (c) The difference in stress response when an external
force (marked by the magenta “X”) is added on the same vertex before and after
the flip.

_Reconfigurable mechanical metamaterials based on Assur graphs.—_ The notion
of Assur decomposition depicts the remarkable nonlocality of graph rigidity,
where a small change in connectivity can affect rigidity _arbitrarily far
away_ PG_Original . This unique property can be powerful in the design of
“Maxwell” mechanical metamaterials where the proximity to isostaticity gives
rise to rich phenomena in terms of modes, stress, and reconfiguration
paulose2015topological ; rocklin2017transformable ; rocklin2018folding ;
Sun2020 ; Xiu2022 ; Zhang2022.

Here we propose to utilize Assur graphs to design mechanical metamaterials
which reconfigure the spatial distribution of motion and stress, and thus
precisely direct actuations. One such example is shown in Fig. 3, where the
flip of just one edge changes the Assur decomposition of the whole graph,
because we loop a far-downstream vertex back to the MIG. This causes the whole
interior region of the graph to change from downstream Assur components to
part of the MIG. Thus, before the flip the interior region can be mobile (upon
pruning any edges in the region) without moving the MIG and will not be
stressed when the MIG is stressed, whereas in after the flip their motion and
stress are locked in the MIG. A unique advantage of this principle is that the
graph remain isostatic before the after the reconfiguration, maintaining good
stability.

_Conclusions and discussions.—_ Here we examine substructures of isostatic
graphs in physical system, by extending the concept of Assur decomposition to
graphs with PBC and show that MJPs are not only isostatic but minimally
isostatic, as well as proposing a new design rule for reconfigurable
mechanical metamaterials reconfigurable via remote mechanical control.

Many intriguing new questions follow from these findings: What is the
substructure of rigidity in packings of more complex particles such as
frictional bi2011jamming , nonspherical particles Mailman2009 , or packings in
3D OHern2003 ? How to optimize this type of mechanical remote control to
obtain significant change in motion and stress propagation in experimental
systems with imperfections? How to obtain networks with mechanical properties
that resemble jammed packings without the packing process Lopez2013 ;
hagh2018jamming ? And more interestingly, can we control isostatic
substructures by programming particle properties and protocols, thereby
obtaining self-assembled reconfigurable metamaterials? These would be
interesting questions for future research.

_Acknowledgements.—_ This work was supported in part by the National Science
Foundation under Grant No. NSF-DMR-2026825 and the Office of Naval Research
under Grant No. MURI N00014-20-1-2479.

##  Appendix A Theorems about rigidity on torus

Here we first list two theorems from the literature on rigidity, (2,2)-tight
graphs, and orientation.

######  Theorem 2.

A graph $G$ with its embedding on the torus is isostatic on the torus iff it
is (2,2)-tight and all of its (2,2)-tight sub-graphs are embedded
constructively.

This theorem is the generalization of Laman’s theorem to graphs on the torus,
and was proven in Ref. Rossthesis  (Theorem 6.3.1). Note that the language we
use is slightly different but equivalent to the version in Ref. Rossthesis .
We have written it in this way since it explicitly states the condition
checked by the pebble game on the torus. The main difference between the two
statements of the theorem is that Ref. Rossthesis  encodes the information of
the embedding in a gain assignment. When we say “all of its (2,2)-tight sub-
graphs are embedded constructively” it is equivalent to saying that the “gain
assignment is constructive” in the language of Ref. Rossthesis .

######  Theorem 3.

A graph $G$ is (2,2)-tight iff there exist an orientation such that one vertex
has in-degree 0 and all others have in degree 2.

This theorem follows from Theorem 8 in Ref. pga_sparse_graphs , where it is
shown that a graph is (2,2)-tight iff it can be fully oriented by the
(2,2)-pebble game with only two free pebbles remaining. We can place the two
free pebbles on any vertex, then it would have in-degree 0 and the rest of the
vertices would have in-degree 2. These pebble game orientations are all the
possible orientations such that the in-degree of one vertex is 0 and the rest
have in-degree 2. This follows from the fact the once the in-degree of each
vertex has been specified the orientation is unique up to reversal of cycles
DigraphsDecomp .

##  Appendix B Proof of Theorem 1 in the main text

###  B.1 Th.1.a $\Longleftrightarrow$Th.1.b

We will show that statement Th.1.a: “$G$ has no proper subgraphs that are
isostatic on the torus” is equivalent to statement Th.1.b “Any orientation of
$G$ where the in-degree of all but one vertex is 2 is strongly connected on
all but that one vertex.”, given that $G$ is isostatic on the torus.

Let us define “minimally (2,2)-tight graphs” as graphs that are (2,2)-tight
and has no proper (2,2)-tight subgraphs. Then from Th. 2 it follows
immediately:

######  Lemma 4.

A graph $G$ is minimally isostatic on the torus iff it is minimally
(2,2)-tight and is embedded constructively.

Next we prove a lemma that relates “minimally (2,2)-tight” with graph
orientations.

######  Lemma 5.

A graph $G$ is minimally (2,2)-tight iff it is (2,2)-tight and all
orientations in which one vertex has in-degree 0 and all others have in-degree
2 are strongly connected on the in-degree 2 vertices.

_Proof._ Consider a (2,2)-tight graph $G$. We prove this lemma by
contradiction. Suppose there exist a proper subgraph $G^{\prime}$ that is
(2,2)-tight, the number of edges in $G^{\prime}$ is
$N^{\prime}_{B}=2N^{\prime}-2$ where $N^{\prime}$ is the number of vertices in
$G^{\prime}$. Because the 0 in-degree vertex can be chosen to be any
vertex,(See pga_sparse_graphs  specifically Lemma 13) we can place it in
$G^{\prime}$. Thus, the in-degree of the subgraph $G^{\prime}$ is given by
$InDegree(G^{\prime})=2(N^{\prime}-1)-N_{B}^{\prime}=0$. This means that there
must be no edges pointing from the rest of $G$ to $G^{\prime}$, and thus
$G^{\prime}$ is a “source” for the rest of $G$. Then $G$ is not strongly
connected on all in-degree 2 vertices. Thus, if all orientations such that one
vertex has in-degree 0 and all others have in degree 2 are strongly connected
on the in degree 2 vertices, $G$ is minimally (2,2)-tight.

Conversely, if $G$ is (2,2)-tight but there exist an orientation that is not
strongly connected on all in-degree 2 vertices, there must be a “source” in
which one vertex has in-degree 0 and all others have in-degree 2. As a result,
by Th. 3, this “source” is a proper (2,2)-tight subgraph, and thus $G$ is not
minimally (2,2)-tight. Therefore, $G$ being minimally (2,2)-tight indicates
that all all orientations in which one vertex has in-degree 0 and all others
have in degree 2 are strongly connected on the in degree 2 vertices. These two
arguments together prove Lemma 5

The combination of Lemma 4 and Lemma 5 imply Th.1.a
$\Longleftrightarrow$Th.1.b.

###  B.2 Th.1.a $\Longleftrightarrow$Th.1.c

This is rather straightforward, given $G$ being minimally isostatic on the
torus the loss of an edge results in a floppy mode. The part of the graph that
does not become generically floppy as a result of this edge being pruned will
be a proper rigid subgraph. No such subgraphs exist (since $G$ is minimally
isostatic) therefore all the vertices become floppy. To prove the converse
note that if a graph become fully floppy when one edge is cut it means that
any rigid subgraph must have contained that edge. If the graph becomes fully
floppy upon the cutting of any edge then all any rigid subgraph must contain
all edges, therefore there are no proper rigid subgraphs.

###  B.3 Th.1.a $\Longleftrightarrow$Th.1.d,e

Now let us move on to the equivalence of Th.1.a $\Longleftrightarrow$Th.1.d
and Th.1.a $\Longleftrightarrow$Th.1.e, the stress conditions.

Rigidity in general requires that given any external force $\mathbf{f}$
orthogonal to the trivial degrees of freedom there can be found a set of bond
tensions $\mathbf{t}$ that balances this external force
$\mathbf{f}=Q\mathbf{t}$. We will call such a set of tensions the response. It
is clear that for isostatic networks the response must be unique otherwise if
we had distinct responses $\mathbf{t}_{1},\mathbf{t}_{2}$ we could find a
state of self stress $0=Q(\mathbf{t}_{1}-\mathbf{t}_{2})$. Given the
uniqueness of the response it is then clear that for isostatic networks the
response to an external force $\mathbf{f}$ will be provided only by the
smallest subgraph capable of providing a response. The equivalence of 1.a to
1.d and 1.e follow immediately from this discussion. A subgraph can provide a
response iff the force is orthogonal to its space of zero modes. Note that
while only a torus isostatic subgraph can respond to a torque, if we exert a
force that is orthogonal to translations and rotation (e.g., a force dipole
induced by the addition of an extra edge, as stated in Th.1.e) then a
nonconstructive plane isostatic (i.e Laman) subgraph can respond.

###  B.4 Generalization to Minimally (k,k)-tight graphs

We can generalize Lemma 5 to minimally $(k,k)$ tight graphs since the proof
for arbitrary $k$ is completely analogous to the $k=2$ case which we have
done.

######  Theorem 6.

A graph $G$ is minimally $(k,k)$-tight iff it is $(k,k)$-tight and all
orientations such that one vertex has in-degree 0 and all others have in-
degree $k$ are strongly connected on the in-degree $k$ vertices.

The effects of pruning an edge or adding an external edge are also
generalized. It is an interesting question to find the physical setup that
allows us to realize a minimally $(k,k)$-tight graph as a mechanical frame.
The (3,3)-tight case is achieved by embedding the graph on the plane, without
pins, and having the vertices be rigid bodies in 2D which therefore would have
3 inherent degrees of freedom. Another interesting physical example are point-
like particles in 3D under periodic boundary conditions, which leads to
isostatic graphs on 3D torus. However it is worth noting that (3,3)-tightness
plus constructive embedding will most likely be a necessary but not sufficient
condition for isostaticity on the 3D torus.3DPGA

##  Appendix C Assur Decomposition on the Torus and Its Uniqueness

Any graph that is isostatic on the torus has at least one subgraph that is
minimally isostatic on the torus, although it may have multiple as in Fig. 4.
If there exist multiple disjoint minimally isostatic subgraphs our
generalization of the Assur decomposition will not be unique as in the pinned
case.

Fortunately, this is not an issue for isostatic graphs obtained from MJPs.
This networks always had a unique minimally isostatic subgraph. In what
follows we explain the Assur decomposition on the torus and why, given that
there are not multiple minimally isostatic subgraphs the decomposition we
propose is unique.

What would make the Assur decomposition non-unique when there are multiple
disjoint minimally isostatic subgraphs is choice of ground $s_{0}$, which is
the vertex chosen to have in-degree 0. For any (2,2)-tight graph given the
choice of ground the decomposition into strongly connected components is
unique. Note, $s_{0}$ will always be its own strongly connected component.
Consider a vertex $s_{1}$ which is a neighbor of $s_{0}$, which is to say the
edge $(s_{0},s_{1})$ belongs to the graph. The strongly connected component to
which $s_{1}$ belongs to, along with the ground $s_{0}$ constitute what is
called the critical subgraph of edge $(s_{0},s_{1})$. The critical subgraph is
the smallest (2,2)-tight subgraph that contains $(s_{0},s_{1})$ (see Lemma
2.6.4d in Ref. Rossthesis ). It is clear that the critical subgraph is always
strongly connected on all vertices except the ground, but this does not imply
that this subgraph is minimally (2,2)-tight. Recall Lemma 5 requires all
possible orientations to be strongly connected on the non-ground sites for the
graph to be minimally (2,2)-tight, and the existence of a particular
orientation that is strongly connected is not sufficient. If we want to
identify the critical subgraph $(s_{0},s_{1})$ as a first Assur component it
must be indecomposable for any choice of ground, which is to say minimally
(2,2)-tight. In order to do this we must choose a ground that belongs to a
minimally (2,2)-tight subgraph. Luckily the PGA on the torus (Ref. Rossthesis
) naturally does this as we discuss below.

The first step of the PGA on the torus is to orient according to the
(2,3)-PGA. Then in the second step the bonds that remain uncovered are
checked. Three pebbles are called to such bonds, if 3 pebbles are found and
the critical subgraph of the bond is constructive then the bond is covered.
Note, just after the first step the subgraph composed of all covered bonds is
(2,3)-tight and therefore contains no (2,2)-tight subgraphs. When the first
bond $(s_{0},s_{1})$ is covered during the second step the covered subgraph
obtains a (2,2)-tight subgraph. This implies that all (2,2)-tight subgraphs of
the covered subgraph contain the edge $(s_{0},s_{1})$. Therefore the first
bond covered during this second step belongs to a minimally-(2,2) tight graph
which will be identical to its critical subgraph. For a graph rigid on the
torus with no disjoint minimally isotatic subgraphs only one bond will be
covered during the second step. This is what we found to always happen for MJP
contact networks. Then we see that in such cases choosing the ground to be one
of the vertices of the last covered bond guarantees that the critical subgraph
is minimally (2,2)-tight and therefore we can identify is as the first Assur
component. Specifically, the steps of our decomposition are as follows

Given $G$ isostatic on the torus:

  1. 1.

Orient $G$ according to the PGA on the torus

  2. 2.

Place the remaining two pebbles on one of the ends of the last covered edge,
making it the ground $s_{0}$. (This happens automatically when the PGA
finishes.)

  3. 3.

Partition the vertices into strongly connected components according the PGA
orientation

  4. 4.

Define an assur component for each strongly connected component except the
ground as follows: All the vertices in the strongly connected component along
with edges pointing to these vertices.

  5. 5.

Choose a neighbor $s_{1}$ of the ground vertex $s_{0}$, assign the ground
$s_{0}$ to the same Assur component as $s_{1}$ is in. This is an induced
subgraph of $G$ and also minimally isostatic on the torus.

Figure 4: An example of a graph, isostatic on the torus with two distinct
minimally isostatic subgraphs and therefore a non unique Assur decomposition.
Figure 5: A minimally (2,2)-tight graph oriented according to the pebble game
on the plane. Note that all vertices except $s_{0}$ are strongly connected.

It is perhaps worth noting that every minimally (2,2)-tight graph can be
realized as a MIG on the torus, a pinned MIG (i.e., an Assur graph), and as a
free frame on the plane with only one SSS (which must be a global SSS),
sometimes called a Laman circuit. And so we see the equivalence between these
three structures. To see that frames on the plane with only one SSS, which is
global, are always minimally (2,2)-tight, note that when oriented according to
the (2,3)-PGA as in Fig. 5 the critical subgraph always identifies a subgraph
with one global SSS. This is a result that goes back to Jacobs and Thorpe
PG_Original . It is interesting that this full stress condition, satisfied by
MJPs, shows up here. This is related to the special geometric singularity of
pinned Assur graphs which we mention in the main text and leads to MJPs on
circular containers to necessarily have Assur graphs as their contact
networks. Also, note that a 2D MJP on the surface of a sphere will necessarily
be a Laman circuit and therefore minimally (2,2)-tight. Such simple arguments
don’t seem to extend straightforwardly to the torus and so the question of how
the fully repulsive interaction leads in general to minimal isostaticity
remains an open problem.

## References

  * (1) S. Zhang, L. Zhang, M. Bouzid, D. Z. Rocklin, E. Del Gado, and X. Mao, “Correlated rigidity percolation and colloidal gels,” Phys. Rev. Lett., vol. 123, p. 058001, Jul 2019.
  * (2) J. Colombo and E. Del Gado, “Stress localization, stiffening, and yielding in a model colloidal gel,” Journal of rheology, vol. 58, no. 5, pp. 1089–1116, 2014.
  * (3) C. S. O’Hern, L. E. Silbert, A. J. Liu, and S. R. Nagel, “Jamming at zero temperature and zero applied stress: The epitome of disorder,” Phys. Rev. E, vol. 68, p. 011306, Jul 2003.
  * (4) D. Bi, J. Lopez, J. M. Schwarz, and M. L. Manning, “A density-independent rigidity transition in biological tissues,” Nature Physics, vol. 11, no. 12, pp. 1074–1079, 2015.
  * (5) T. C. Lubensky, C. L. Kane, X. Mao, A. Souslov, and K. Sun, “Phonons and elasticity in critically coordinated lattices,” Reports on Progress in Physics, vol. 78, p. 073901, jun 2015.
  * (6) X. Mao and T. C. Lubensky, “Maxwell lattices and topological mechanics,” Annual Review of Condensed Matter Physics, vol. 9, no. 1, pp. 413–433, 2018\.
  * (7) G. Laman, “On graphs and rigidity of plane skeletal structures,” Journal of Engineering mathematics, vol. 4, no. 4, pp. 331–340, 1970.
  * (8) L. Asimow and B. Roth, “The rigidity of graphs,” Transactions of the American Mathematical Society, vol. 245, pp. 279–289, 1978.
  * (9) S. Alexander, “Amorphous solids: their structure, lattice dynamics and elasticity,” Physics reports, vol. 296, no. 2-4, pp. 65–236, 1998.
  * (10) M. van Hecke, “Jamming of soft particles: geometry, mechanics, scaling and isostaticity,” Journal of Physics: Condensed Matter, vol. 22, no. 3, p. 033101, 2009.
  * (11) A. J. Liu and S. R. Nagel, “The jamming transition and the marginally jammed solid,” Annu. Rev. Condens. Matter Phys., vol. 1, no. 1, pp. 347–369, 2010\.
  * (12) C. P. Goodrich, A. J. Liu, and J. P. Sethna, “Scaling ansatz for the jamming transition,” Proceedings of the National Academy of Sciences, vol. 113, no. 35, pp. 9745–9750, 2016.
  * (13) D. J. Jacobs and M. F. Thorpe, “Generic rigidity percolation: the pebble game,” Physical review letters, vol. 75, no. 22, p. 4051, 1995.
  * (14) A. Donev, S. Torquato, F. H. Stillinger, and R. Connelly, “A linear programming algorithm to test for jamming in hard-sphere packings,” Journal of Computational Physics, vol. 197, no. 1, pp. 139–166, 2004.
  * (15) M. Mailman, C. F. Schreck, C. S. O’Hern, and B. Chakraborty, “Jamming in systems composed of frictionless ellipse-shaped particles,” Phys. Rev. Lett., vol. 102, p. 255501, Jun 2009.
  * (16) C. P. Broedersz, X. Mao, T. C. Lubensky, and F. C. MacKintosh, “Criticality and isostaticity in fibre networks,” Nature Physics, vol. 7, no. 12, pp. 983–988, 2011.
  * (17) W. G. Ellenbroek and X. Mao, “Rigidity percolation on the square lattice,” Europhysics Letters, vol. 96, p. 54002, nov 2011.
  * (18) D. Bi, J. Zhang, B. Chakraborty, and R. P. Behringer, “Jamming by shear,” Nature, vol. 480, no. 7377, pp. 355–358, 2011.
  * (19) L. Yan and M. Wyart, “Evolution of covalent networks under cooling: Contrasting the rigidity window and jamming scenarios,” Phys. Rev. Lett., vol. 113, p. 215504, Nov 2014.
  * (20) W. Ellenbroek, V. Hagh, A. Kumar, M. Thorpe, and M. Hecke, “Rigidity loss in disordered systems: Three scenarios,” Physical review letters, vol. 114, 11 2014.
  * (21) S. Henkes, D. A. Quint, Y. Fily, and J. M. Schwarz, “Rigid cluster decomposition reveals criticality in frictional jamming,” Phys. Rev. Lett., vol. 116, p. 028301, Jan 2016.
  * (22) R. P. Behringer and B. Chakraborty, “The physics of jamming for granular materials: a review,” Reports on Progress in Physics, vol. 82, no. 1, p. 012601, 2018.
  * (23) L. V. Assur and I. I. Artobolevskij, Issledovanie ploskih steržnevyh mehanizmov s nizšimi parami s točki zreniâ ih struktury i klassifikacii.  Izdatel’stvo Akademii Nauk SSSR, 1952.
  * (24) O. Shai, A. Sljoka, and W. Whiteley, “Directed graphs, decompositions, and spatial linkages,” Discrete Applied Mathematics, vol. 161, no. 18, pp. 3028 – 3047, 2013.
  * (25) A. Sljoka, “Checking mobility and decomposition of linkages via pebble game algorithm,” 08 2011.
  * (26) E. Hahn and O. Shai, “The unique engineering properties of assur groups/graphs, assur kinematic chains, baranov trusses and parallel robots,” in International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, vol. 50169, p. V05BT07A074, American Society of Mechanical Engineers, 2016.
  * (27) R. Penne, “Isostatic bar and joint frameworks in the plane with irreducible pure conditions,” Discrete Applied Mathematics, vol. 55, no. 1, pp. 37–57, 1994.
  * (28) S. Chen, F. Giardina, G. P. Choi, and L. Mahadevan, “Modular representation and control of floppy networks,” Proceedings of the Royal Society A, vol. 478, no. 2264, p. 20220082, 2022.
  * (29) E. Ross, Geometric and Combinatorial Rigidity of Periodic Frameworks as Graphs on The Torus.  PhD thesis, 05 2011.
  * (30) B. Servatius, O. Shai, and W. Whiteley, “Geometric properties of assur graphs,” European Journal of Combinatorics, vol. 31, no. 4, pp. 1105–1120, 2010.  Rigidity and Related Topics in Geometry.
  * (31) D. J. Jacobs and M. F. Thorpe, “Generic rigidity percolation: The pebble game,” Phys. Rev. Lett., vol. 75, pp. 4051–4054, Nov 1995.
  * (32) J. Paulose, B. G.-g. Chen, and V. Vitelli, “Topological modes bound to dislocations in mechanical metamaterials,” Nature Physics, vol. 11, no. 2, pp. 153–156, 2015.
  * (33) D. Rocklin, S. Zhou, K. Sun, and X. Mao, “Transformable topological mechanical metamaterials,” Nature communications, vol. 8, no. 1, pp. 1–9, 2017.
  * (34) D. Rocklin, V. Vitelli, and X. Mao, “Folding mechanisms at finite temperature,” arXiv preprint arXiv:1802.02704, 2018.
  * (35) K. Sun and X. Mao, “Continuum theory for topological edge soft modes,” Phys. Rev. Lett., vol. 124, p. 207601, May 2020.
  * (36) H. Xiu, H. Liu, A. Poli, G. Wan, K. Sun, E. M. Arruda, X. Mao, and Z. Chen, “Topological transformability and reprogrammability of multistable mechanical metamaterials,” Proceedings of the National Academy of Sciences, vol. 119, no. 52, p. e2211725119, 2022.
  * (37) J. H. Lopez, L. Cao, and J. M. Schwarz, “Jamming graphs: A local approach to global mechanical rigidity,” Phys. Rev. E, vol. 88, p. 062130, Dec 2013\.
  * (38) V. F. Hagh, E. I. Corwin, K. Stephenson, and M. Thorpe, “Jamming in perspective,” arXiv preprint arXiv:1803.03869, 2018.
  * (39) A. Lee and I. Streinu, “Pebble game algorithms and sparse graphs,” Discrete Mathematics, vol. 308, no. 8, pp. 1425–1437, 2008.  Third European Conference on Combinatorics.
  * (40) M. V. Chubynsky and M. F. Thorpe, “Algorithms for three-dimensional rigidity analysis and a first-order percolation transition,” Phys. Rev. E, vol. 76, p. 041135, Oct 2007.


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Stony Brook University, Stony Brook, NY 11794, USA 11email:
<EMAIL_ADDRESS>
# Diffusion-Refined VQA Annotations for Semi-Supervised Gaze Following

Qiaomu Miao  Alexandros Graikos  Jingwei Zhang  Sounak Mondal  Minh Hoai
Dimitris Samaras

###### Abstract

Training gaze following models requires a large number of images with gaze
target coordinates annotated by human annotators, which is a laborious and
inherently ambiguous process. We propose the first semi-supervised method for
gaze following by introducing two novel priors to the task. We obtain the
first prior using a large pretrained Visual Question Answering (VQA) model,
where we compute Grad-CAM heatmaps by ‘prompting’ the VQA model with a gaze
following question. These heatmaps can be noisy and not suited for use in
training. The need to refine these noisy annotations leads us to incorporate a
second prior. We utilize a diffusion model trained on limited human
annotations and modify the reverse sampling process to refine the Grad-CAM
heatmaps. By tuning the diffusion process we achieve a trade-off between the
human annotation prior and the VQA heatmap prior, which retains the useful VQA
prior information while exhibiting similar properties to the training data
distribution. Our method outperforms simple pseudo-annotation generation
baselines on the GazeFollow image dataset. More importantly, our pseudo-
annotation strategy, applied to a widely used supervised gaze following model
(VAT), reduces the annotation need by 50%. Our method also performs the best
on the VideoAttentionTarget dataset.

###### Keywords:

Gaze following Semi-supervised learning Diffusion Model

##  1 Introduction

Human gaze behavior is a vital cue for applications such as human-machine
interaction [39, 11, 1], social interaction analysis [35, 5], and human
intention interpretation [48, 52]. Although eye-tracking glasses can be used
to track gaze behavior, they are intrusive, require personalized calibration,
and are mostly restricted to laboratory environments. In contrast, the gaze
following task [45, 9, 12] predicts a person’s gaze target in an image in-the-
wild, by predicting a target heatmap from an input scene image with the
person’s head bounding box.

The effectiveness of gaze following methods depends on the quantity and
quality of training data with annotated gaze targets. Annotating gaze targets
is a laborious and ambiguous process. For an image and a subject in the image,
an annotator must consider the subject’s head orientation, gaze angle, and
follow the gaze direction to determine where the point of gaze falls on an
object or surface within the image. This can be very challenging, as the
subject may be looking away from the camera, their eyes may be occluded, the
image may be cluttered, and there may be multiple plausible targets, as shown
in Fig. 1(a).

Figure 1: (a) Gaze following annotation challenges. Annotating gaze is a
laborious task with inherent ambiguities. (b) Pseudo annotations for gaze
following. We generate pseudo annotations by first computing Grad-CAM heatmaps
from a pre-trained VQA model, and then refining the noisy heatmaps with a
diffusion model.

It is much more efficient to utilize large quantities of unlabeled data, along
with a limited amount of labeled data, and leverage semi-supervised learning
to improve model performance. Semi-supervised learning has been successful in
tasks such as image recognition and semantic segmentation [64, 59, 26, 2, 23].
In the context of gaze following, semi-supervised learning can be particularly
useful when developing models for a specific scene, such as a psychology lab
conducting social interaction studies or a supermarket studying customer gaze
behaviors, by requiring only a small fraction of the collected data to be
annotated.

This is the first semi-supervised gaze following method, to the best of our
knowledge. Semi-supervised recognition and segmentation methods are typically
not directly applicable to gaze following due to task differences. As gaze
following models predict a target heatmap instead of classification logits and
require an intact scene image as input, methods that operate on the predicted
multi-class distribution [57, 26], learn disentangled representations [42,
33], or perform strong data augmentation (e.g.,CutOut[54, 13]) cannot easily
be adapted to gaze following. Additionally, for semi-supervised methods that
use a teacher model to pseudo-annotate the unlabeled data [30, 59, 29], it is
unclear how to design a gaze following teacher model to generate good quality
pseudo-annotations.

To generate high-quality pseudo-gaze labels, our semi-supervised method
combines the power of pre-trained large vision-language (VL) models with an
annotation refinement method. Due to their extensive training sets, VL models
have naturally acquired a wealth of knowledge, including the knowledge for
inferring human gaze targets. Specifically, we “prompt” a pre-trained Visual
Question Answering (VQA) model [37, 44, 67] with an appropriate question to
obtain the text description of the gaze target of a subject in the image. We
then use Grad-CAM [49, 6], to ground the VQA model’s description to spatial
locations, referred to as Grad-CAM heatmaps. We want to turn these heatmaps
into pseudo-annotations for training gaze following models.

There are technical challenges in utilizing Grad-CAM heatmaps. First, Grad-CAM
heatmaps do not precisely overlap with the gaze targets because they are
dispersed and noisy, as shown in Fig. 1(b). Second, some Grad-CAM heatmaps do
not provide useful prior knowledge and may highlight incorrect gaze target
locations. Hence, the noisy Grad-CAM heatmaps should be refined into clean
pseudo-annotations, with a trade-off between retaining the Grad-CAM heatmap
information and the pre-trained prior on the annotations available.

We choose a diffusion model as the annotation prior of our refinement method.
We are inspired by the recent use of diffusion models as inverse problem
solvers and image editors that modify the reverse sampling process with a
conditional input [36, 56, 41]. The goal is to generate an output image that
retains the semantics of the degraded input (here the Grad-CAM heatmap), while
having similar properties (i.e., geometry and gaze context) to the training
data distribution.

We begin by training a diffusion model to capture the distribution of human-
labeled annotations. The trained diffusion model then generates refined
annotations by running the reverse sampling process, initialized from Grad-CAM
heatmaps with an appropriate level of Gaussian noise. The added noise serves
to smooth artifacts and noisy activations in the Grad-CAM heatmaps; the
magnitude of the noise dictates how much of the information is preserved.

In summary, we propose the first semi-supervised gaze following method, in
which we introduce two novel priors to the task: (1) Prior knowledge from pre-
trained vision-language models for initial annotation generation. (2) A novel
diffusion-based annotation prior to refine the noisy VL annotations into
reliable pseudo-labels.

Our method outperforms pseudo-annotation generation baselines on the
GazeFollow image dataset [45]. More importantly, our pseudo-annotation
strategy, applied to the widely used VAT model [9], outperforms the fully
supervised model trained with double the amount of annotations when only 5%
and 10% labels are available. Our method also performs the best on the
VideoAttentionTarget dataset [9], where we adapt a pre-trained gaze following
model to new videos using only around 100 (2–23%) annotated frames.

##  2 Prior Work

Gaze Following was first introduced in [45] together with the GazeFollow
dataset and a model composed of two separate pathways for encoding gaze
orientation and scene saliency information, also adopted by later work [34, 8,
9, 12]. Chong _et al_. [8] considered out-of-frame targets and also extended
the task to videos [9], proposing the VideoAttentionTarget dataset and the VAT
model. Later work has improved gaze following models by leveraging monocular
depth estimations [12, 25, 60, 21] and human poses [16, 4]. Tu _et al_. [62]
proposed a transformer model for gaze following, while [24, 38] added training
losses for numerical coordinate regression and patch-level gaze distribution
prediction. Recent works investigate 3D gaze following [22, 20], gaze
following for children [58], and object-aware gaze target prediction [61]. All
of these methods are fully-supervised.

Semi-supervised Learning. The majority of semi-supervised learning methods
adopt a teacher-student framework, which can be categorized into two groups:
self-training [30, 7, 15, 54] and consistency regularization [29, 59, 13, 18].
Self-training methods generate pseudo annotations from teacher models trained
with labeled data, while consistency regularization applies a consistency loss
between the student and teacher model outputs, updating the teacher model
gradually during training. As most semi-supervised learning methods are built
for recognition or segmentation tasks, they are not directly applicable to
gaze following [54, 42, 13, 26] due to task differences. However, some general
methods that do not perform task-specific operations [29, 59] can be applied
to gaze following with appropriate modifications (details in the
Supplementary). In our method, we adopt a self-training pipeline using the VQA
prior and diffusion-based annotation prior, and show that we can get further
improvements by using applicable consistency regularization methods (e.g. Mean
Teacher [59]) to enhance the training of the diffusion model.

Vision-Language Models. Vision-Language (VL) models [63, 27, 51, 31] have
succeeded in tasks such as visual grounding, visual question answering (VQA),
and image captioning. They also do well in low-shot generalization tasks by
“prompting” [43, 37, 31], where appropriate task instructions (usually in the
form of text) are given to a pretrained model to generate outputs useful for
other tasks. Grad-CAM [50] visualizations on the cross-attention maps localize
the linguistic input effect on the visual input [32, 66] thus achieving a
degree of interpretability.

Diffusion Models. Denoising diffusion probabilistic models (DDPM) [53, 19]
give state-of-the-art results on image synthesis, by capturing intricate data
properties [10, 40, 46, 28]. They have also been used in inverse problems
[56], image editing [36], and adversarial purification [41]. These works treat
images with added noise as intermediate steps of the reverse sampling process,
showing that the noise magnitude affects the information retained from input
images. In our task, we add appropriate magnitude noise to the Grad-CAM
heatmaps to achieve the best trade-off between the VQA prior and annotation
prior. Recent work adopts diffusion models in semi-supervised learning. [18]
uses a diffusion model as the student model for 3D object detection, whereas
[65] trains a diffusion model for image generation with images pseudo-labeled
by existing semi-supervised models. In contrast, we use diffusion model as
annotation prior, to refine the initial noisy annotations from VQA into high-
quality pseudo annotations.

##  3 Proposed Method

###  3.1 Overview

Our approach is illustrated in Fig. 2a. The goal is to leverage the knowledge
priors from a VQA model to improve gaze following with minimal manual
annotation. To achieve this, we utilize Grad-CAM heatmaps of a VQA model that
is prompted with gaze-related questions.

However, the Grad-CAM heatmaps need refinement to be useful for training
supervision. For this, we train a diffusion model on the groundtruth heatmaps
from labeled data to learn to generate heatmaps from the human-label
distribution. We use the model to refine the “noisy” Grad-CAM into high-
quality pseudo annotations. We train existing gaze following models on both
groundtruth heatmaps from labeled data and refined heatmaps from unlabeled
data.

Suppose we only have a small number of images that are labeled with gaze
target locations ${\mathcal{L}=\\{(\bm{X}_{i},\bm{p}_{i})\\}_{i=1}^{N_{l}}}$
and a larger number of unlabeled images
${\mathcal{U}=\\{\bm{X}_{j}\\}_{j=1}^{N_{u}}}$. $\bm{X}_{i}$ indicates an
input triplet to the gaze following models, which includes a scene image
$\bm{I}_{i}\in R^{3\times H\times W}$, the cropped head image
$\bm{I}_{h}^{i}\in R^{3\times H\times W}$, and the head location binary mask
$\bm{M}_{h}^{i}\in R^{H\times W}$. We represent the annotated gaze coordinate
of the "gazing" person as $\bm{p}_{i}=[p_{i}^{x},p_{i}^{y}]$. For each
$\bm{p}_{i}$, a ground truth heatmap $\mathbf{h}^{i}\in R^{H^{\prime}\times
W^{\prime}}$ is generated by placing a Gaussian of fixed variance centered at
${\bm{p}_{i}}$, as is done in previous gaze following frameworks [9, 12, 34].

Figure 2: (a) Overall pipeline for semi-supervised training. We compute Grad-
CAM heatmaps for a large set of unlabeled images and train the diffusion model
with a small human-labeled set. We use the diffusion model to refine the Grad-
CAM heatmaps into pseudo-annotations. Both the pseudo-annotations and the
human-labeled training set are used to train a gaze following model. (b) Grad-
CAM heatmap generation procedure. Given an input image with an overlaid
bounding box for the person, we “prompt" a large pretrained VQA model with a
gaze question and compute the Grad-CAM heatmap from the answer. (c) Grad-CAM
refinement process. We perturb the raw Grad-CAM heatmaps with Gaussian noise
and pass through the reverse diffusion process to generate the refined pseudo-
annotations.

###  3.2 Grad-CAM heatmap extraction

We use OFA [63], an open-source large pretrained VL model as the VQA model.
Fig. 2b illustrates the procedure for Grad-CAM heatmap generation. Given an
image $\bm{I}$ and the head bounding box of a person $\bm{l}_{h}$, we use Mask
R-CNN [17] for person detection and find the bounding box that maximally
overlaps with $\bm{l}_{h}$. We overlay this bounding box on top of the input
image and give it as input to the VQA model along with a “prompt” in the form
of a question: “What is the person in the green bounding box looking at?".

OFA has a transformer-based encoder-decoder structure. The model generates a
sequence of words as an answer. We compute a Grad-CAM heatmap
$\mathcal{G}=\\{\bm{g}_{j}\\}_{j=1}^{N_{u}}$ on the decoder cross-attention
weights between the input query token, and the image patch tokens based on [6]
after selecting the noun from the answer. Details are in the supplementary.

In our experiments, we found that Grad-CAM heatmaps are beneficial for gaze
following. Naively using these heatmaps as additional input to the current
gaze following model results in a significant performance boost. However, in
real applications, computing Grad-CAM heatmaps at test-time can be impractical
due to the memory and time costs of accessing large VL models. Instead we use
Grad-CAM heatmaps offline as pseudo-annotations for training a gaze following
model. As discussed, using noisy Grad-CAMs directly as pseudo labels can hurt
final model performance. Thus, we propose to utilize a diffusion model to
refine the initial Grad-CAM heatmaps into more suitable pseudo labels for
training.

###  3.3 Diffusion Model Training

As there is no available diffusion model for the gaze following task, we first
train a diffusion model on the labeled data to generate gaze heatmaps
$\mathbf{h}\sim p(\mathbf{h|\bm{X}})$, following the general training
procedure of DDPM [19]. For ease of understanding, we provide a synopsis of
Gaussian diffusion models.

Figure 3: Diffusion model training and refinement. (a) The diffusion model is
trained on supervised data with noise added to the ground truth heatmap at
random time steps. (b) During refinement, we add noise at a specific time step
to the Grad-CAM heatmap. We treat this heatmap as an intermediate step input
during the reverse process. Heatmaps are overlayed on the original images for
illustration purposes. The conditional feature extraction for the diffusion
model is omitted for simplicity.

The diffusion model consists of a forward process that gradually corrupts the
input by adding Gaussian noise, and a reverse process that iteratively
denoises the noisy input. The forward process is defined by a noise schedule
$\alpha_{t}$ as:

$\displaystyle q(\mathbf{h}_{t}|\mathbf{h}_{t-1})$
$\displaystyle:=\mathcal{N}(\mathbf{h}_{t};\sqrt{\alpha_{t}}\mathbf{h}_{t-1},(1-\alpha_{t})\mathbf{I})$
(1) $\displaystyle q(\mathbf{h}_{1:T}|\mathbf{h}_{0})$
$\displaystyle:=\prod_{t=1}^{T}q(\mathbf{h}_{t-1}|\mathbf{h}_{t}).$ (2)

where $\mathbf{h}_{0}$ is the ground truth heatmap $\mathbf{h}$,
$\mathbf{h}_{1},\dots,\mathbf{h}_{T-1}$ are intermediate latent variables that
represent noisy versions of $\mathbf{h}_{0}$, and $\mathbf{h}_{T}$ is the
terminal state which corresponds to a unit Gaussian distribution:
$\mathbf{h}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$.

The reverse process uses Gaussian transitions with fixed covariance
$\sigma_{t}$ and gradually denoises the data, starting from $\mathbf{h}_{T}$.
Since $q(\mathbf{h}_{t-1}|\mathbf{h}_{t})$ is intractable, it is approximated
with a neural network:

$p_{\theta}(\mathbf{h}_{t-1}\mid\mathbf{h}_{t}):=\mathcal{N}(\mathbf{h}_{t-1};\bm{\mu}_{\theta}(\mathbf{h}_{t},t),\sigma_{t}\mathbf{I}).$
(3)

A useful property of the forward process is that it allows sampling of any
intermediate $\mathbf{h}_{t}$ given $\mathbf{h}_{0}$, and
$\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$:

$q(\mathbf{h}_{t}|\mathbf{h}_{0})=\mathcal{N}(\mathbf{h}_{t};\sqrt{\bar{\alpha}_{t}}\mathbf{h}_{0},(1-\bar{\alpha}_{t})\mathbf{I})$
(4)

which is utilized in learning the reverse process. For that, we adopt the
$\mathbf{x}_{0}$-parameterization as in [3, 28] and directly predict the final
$\mathbf{h}_{0}$, leading to the following minimization objective, where
$\mathbf{h}_{0}^{\theta}$ is parameterized by a U-Net [47] :

$\mathbb{E}_{\mathbf{h}_{0},t}\left[\lVert\mathbf{h}_{0}-\mathbf{h}_{0}^{\theta}(\mathbf{h}_{t},t)\rVert^{2}\right],$
(5)

Fig. 3(a) shows the training procedure. At each iteration, we sample a random
time step $t$ from $U[1,T]$ to generate a noisy input $\mathbf{h}_{t}$ using
Eq. 4. We feed $\mathbf{h}_{t}$ to the diffusion model for single-step
denoising, and optimize with the loss of Eq. 5. We condition the diffusion
model with the gaze feature $\mathbf{c}$ extracted from the input triplet
$\bm{X}$ (input image, head crop, and head location mask) to output a gaze
heatmap grounded on the gazing person; the parametrization of the U-Net
denoiser becomes $\mathbf{h}_{0}^{\theta}(\mathbf{h}_{t},t,\mathbf{c})$. The
feature extractor follows the structure of the VAT model [9], which takes the
input triplet $\bm{X}$ and outputs the extracted features concatenated from
two pathways for encoding the scene and gaze features. Details of the VAT
structure are shown in Supplementary. Following [14], we add the extracted
features as conditional features to each layer in the U-Net denoiser with the
necessary transposed convolution layers to match the feature size of U-Net.
During the reverse sampling process, we used DDIM [55] to sample the next step
input using the predicted $\mathbf{h}_{0}$ from the U-Net.

###  3.4 Heatmap Refinement using Diffusion Models

After training on labeled images $\mathcal{L}$, the learned diffusion process
models an annotation prior over ground truth annotations
$p(\mathbf{h|\bm{X}})$. During the semi-supervised training process, the
trained diffusion model is applied to the unsupervised data $\mathcal{U}$ to
produce the refined pseudo annotations.

The refinement process is shown in Fig. 3(b). Given a noisy Grad-CAM heatmap
$\bm{g}$ of an unsupervised input sample $\bm{X}$, we first add an appropriate
amount of noise using the forward diffusion process
$\hat{\mathbf{h}}_{t}=\sqrt{\bar{\alpha}_{t}}\bm{g}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}$
according to Eq. 4. The added noise smoothes out the noise artifacts of the
Grad-CAM heatmaps, while preserving the information of the highlighted
regions. Then, by iteratively applying the steps of the denoising procedure in
Eq. 3, we get samples of the final heatmap $\hat{\mathbf{h}}_{0}$ that are
geometrically and contextually similar to the human-labeled heatmap
$\mathbf{h}_{0}$ while retaining gaze target information from the noisy Grad-
CAM outputs.

The amount of information retained depends on the choice of $t$ (_i.e_., the
magnitude of added noise), as proven in [36, 41]. A larger magnitude of noise
corrupts the Grad-CAM heatmaps more, decreasing the similarity between the
diffusion output and the Grad-CAM priors. Similar to [36] which achieves a
good trade-off between realism and faithfulness in image editing with an
intermediate level of noise, we also found that initializing from an
intermediate timestep in the diffusion process achieved the best trade-off
between the Grad-CAM information and the model’s learned distribution
$p(\mathbf{h|\bm{X}})$. With this formulation, the diffusion model can
incorporate the high-quality Grad-CAM heatmaps, while ignoring the Grad-CAM
priors when they are highly noisy or highlight unlikely gaze target locations.
We provide analyses of the added noise magnitude in Sec. 4.3.2.

###  3.5 Semi-supervised Training

In semi-supervised training, we use the refined heatmaps $\mathcal{R}$ as
pseudo labels for the unsupervised data $\mathcal{U}$. We train the gaze
following model with the set of ground-truth heatmaps $\mathcal{H}$ from the
labeled data $\mathcal{L}$ and the refined heatmaps $\mathcal{R}$ from the
unlabeled data $\mathcal{U}$. We use the Mean Squared Error (MSE) loss between
the predicted heatmap $\bm{h^{\prime}}$ and the target label ${\bf h}$. ${\bf
h}$ is either a human annotation (labeled data) or a pseudo annotation
(unlabeled data). We minimize:

$\mathcal{L}=\frac{1}{|\mathcal{H}\cup\mathcal{R}|}\sum_{{\bf
h}\in\mathcal{H}\cup\mathcal{R}}\mathcal{L}_{mse}(\bm{h^{\prime}},\bm{h}).$
(6)

##  4 Experiments

###  4.1 Setting and Implementation Details

Datasets. GazeFollow [45], the largest real image dataset, contains the
annotated gaze targets of 130,339 people in 122,143 images. 4782 people are
used for testing, and the rest are used in training. For the test set, there
are 10 annotations per person-image pair to account for the ambiguity of the
gaze target, whereas the training set contains a single annotation for each
pair. VideoAttentionTarget [9] is a video dataset of 50 different shows
collected from YouTube. Each person-image pair in both training and test set
has only one annotation.

Evaluation metrics. We employ evaluation metrics suggested by previous work
[45, 9, 34]. The distance metric (Dist.) refers to the normalized $L_{2}$
distance between the predicted gaze point (point with the highest heatmap
response) and the ground truth location. In GazeFollow reports both average
(AvgDist) and minimum distances (MinDist). Area Under Curve (AUC) evaluates
the concordance of predicted heatmaps with ground truth [9]. For GazeFollow,
ground truth is the 10 annotations for each test set image, whereas in
VideoAttentionTarget it is a thresholded Gaussian centered at the single given
annotation.

Implementation Details. We used a diffusion model with a linear noise schedule
and $T{=}500$ steps. The heatmap size was $64{\times}64$. We used a batch size
of 80 and a learning rate of $2.5\times 10^{-4}$ in the semi-supervised
learning experiments. For training the diffusion model on the pool of labeled
data, we used a learning rate of $5\times 10^{-5}$. When refining Grad-CAM
heatmaps, we used DDIM [55] for inference, with two inference steps.

###  4.2 Semi-supervised Training Results

We build our gaze following experiments around the popular, publicly available
VAT model [9]. To showcase the impact of our pseudo annotation generation
method, we did not use additional modalities, processing, or supervisions,
such as depth input, pose estimations, and object-level annotations [12, 4,
16, 61] in all experiments for the baseline methods and the diffusion model.

Following the standard experimental settings in semi-supervised learning [13,
30, 59], we considered different amounts of annotations from the GazeFollow
training set, namely 5%, 10%, and 20%, and treated the remaining data as
unlabeled. The trained student model is evaluated on the test set in all
cases.

We used VAT as the student model and compared our method of generating pseudo
annotations with several alternatives111Implementation details of the
baselines are provided in the supplementary material.: 1) Semi-VAT: the
pseudo-annotations for the unlabeled data were generated by the VAT model
trained with labeled images. 2) Semi-VAT-GC: We created VAT-GC by adding Grad-
CAM heatmaps as conditional inputs to the VAT model and modifying the VAT
model accordingly. VAT-GC was trained on the labeled data to generate pseudo
annotations for the unlabeled data. 3) VAT-MT: We use the Mean Teacher (MT)
method [59] as a representative consistency regularization method to train the
VAT model, which updates the teacher model using both labeled and unlabeled
data through an exponential moving average (EMA) of student model weights.
Here we applied the consistency loss in MT between the heatmap output from the
teacher and student models. The first two baselines are self-training methods,
while the 3rd belongs to the consistency regularization category.

On the other hand, we also build two versions of our method: 1) GradCAM-
Diffusion-Refinement (GCDR): our method of using a diffusion model trained
with labeled data to refine the ‘noisy’ Grad-CAM heatmaps and use them as
pseudo-annotations. 2) GCDR-MT: we used the Mean-Teacher method to train the
diffusion model with both labeled and unlabeled data, and this enhanced
diffusion model was used to refine the Grad-CAM heatmaps.

Table 1: Results of semi-supervised training methods with different ratios of labeled data on GazeFollow dataset. In the top and bottom rows, we show the performances of VAT trained with supervised data only or with full training data, to show the potential lower and upper limits. Best numbers are marked as bold. Our methods outperform supervised VAT trained with double the amount of annotations. Method | 5% labels | 10% labels | 20% labels
---|---|---|---
|  Dist. $\downarrow$ | AUC $\uparrow$ |  Dist $\downarrow$ | AUC $\uparrow$ |  Dist. $\downarrow$ | AUC $\uparrow$
| Avg. | Min. | Avg. | Min. | Avg. | Min.
VAT (Supervised) | 0.230 | 0.161 | 0.835 | 0.202 | 0.133 | 0.869 | 0.182 | 0.116 | 0.892
Semi-VAT | 0.222 | 0.152 | 0.846 | 0.195 | 0.128 | 0.875 | 0.176 | 0.110 | 0.896
Semi-VAT-GC | 0.217 | 0.149 | 0.846 | 0.195 | 0.127 | 0.879 | 0.178 | 0.112 | 0.895
VAT-MT | 0.219 | 0.150 | 0.850 | 0.189 | 0.122 | 0.882 | 0.174 | 0.108 | 0.898
GCDR (Ours) | 0.201 | 0.135 | 0.863 | 0.179 | 0.115 | 0.886 | 0.166 | 0.103 | 0.902
GCDR-MT (Ours) | 0.194 | 0.128 | 0.870 | 0.172 | 0.108 | 0.892 | 0.162 | 0.098 | 0.904
VAT(100% labels) [9] |  |  |  | 0.137 | 0.077 | 0.921 |  |  |
|  |  |  |  |  |  |  |  |

Semi-supervised training results are in Tab. 1. Our two proposed methods
consistently outperform the three semi-supervised baselines in all three
annotation scenarios. When training with 5% and 10% annotations, the
performance of GCDR-MT surpasses the supervised VAT trained with double the
amount of annotations. Results of training with 50% are in the supplementary.

Our methods show more prominent improvements in Dist. compared to AUC. The AUC
on GazeFollow evaluates the concordance of the predicted heatmap with the 10
annotations on each test image, while Dist. evaluates the $L_{2}$ distance
between the predicted gaze point and the annotations. Therefore, the results
mean that we did better in predicting the probable target location than
predicting the exact shape of the heatmap. The refined heatmaps from the
diffusion model closely resemble the spatial distribution of a human-labeled
annotation, which is a single Gaussian (Fig. 4). In contrast, the baseline
methods predict the target location less accurately, while outputting less
certain and larger heatmaps which tend to overlap more with the group-level
annotations, thus favoring the AUC (see supplementary material).

VAT | VAT-GC | Raw Grad-CAM | GCDR | Groundtruth
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Figure 4: Visualizations of pseudo heatmaps generated by different teachers.
Our method generates the cleanest pseudo annotations while retaining the Grad-
CAM heatmaps priors (Rows 1–3). When the initial Grad-CAM heatmap responds
strongly to unlikely locations or is completely noisy, our method can also
ignore it (Row 4).

Fig. 4 visualizes the pseudo annotations predicted by the teacher models. In
the top three rows, our model retains positional priors from the Grad-CAM
heatmaps with almost identical structure to ground truth heatmaps. On the
other hand, when the raw Grad-CAM heatmaps are inaccurate or very noisy (Row
4), our method can ignore the Grad-CAM heatmaps.

###  4.3 Ablation Studies

In the ablation studies shown in Tab. 3, we tested the contribution of the
various components of our method. We also tested different parameters of the
diffusion sampling process in Tab. 3. All ablation studies were performed
using 10% of the GazeFollow labels and the teacher models were trained only on
supervised data (without MT) to simplify training. Additionally, in Tab. 4, we
analyze the effect of the VQA priors by training the baselines in fully-
supervised settings with/without Grad-CAM heatmaps.

Table 2: Ablations of ways for generating the pseudo annotations. Method | Dist. $\downarrow$ | AUC $\uparrow$
---|---|---
Avg. | Min.
No Grad-CAM | 0.191 | 0.125 | 0.867
No Refine | 0.237 | 0.166 | 0.833
Argmax Refine | 0.207 | 0.139 | 0.869
Direct Mapping | 0.190 | 0.122 | 0.878
Proposed GCDR | 0.179 | 0.115 | 0.886
|  |  |

Table 3: Ablations of parameters of the diffusion model refining process. Noise.$t$ | Inf. Steps | Dist. $\downarrow$ | AUC $\uparrow$
---|---|---|---
Avg. | Min.
300 | 2 | 0.186 | 0.120 | 0.877
200 | 2 | 0.184 | 0.117 | 0.883
250 | 5 | 0.180 | 0.115 | 0.883
250 | 2 | 0.179 | 0.115 | 0.886
|  |  |  |

####  4.3.1 Refinement methods

In No Grad-CAM, the diffusion model samples pseudo annotations from Gaussian
noise without using Grad-CAM heatmaps. Without the VQA model priors, there is
a significant performance decrease. No Refinement directly uses the raw Grad-
CAM heatmaps as pseudo annotations without refinement, which also leads to a
significant performance decrease due to the noisy nature of the Grad-CAM
heatmaps. Argmax Refinement generates a Gaussian heatmap from the maximum
point of the Grad-CAM heatmap as pseudo annotation. Despite the improvement
over No refinement due to the “cleaner" annotation pattern, performance is
still far behind our method, because some of the Grad-CAM heatmaps are low-
quality (Fig. 4), so Gaussian ends up on outliers or incorrect locations. On
the contrary, our method can ignore the Grad-CAM heatmaps in these cases. In
Direct Mapping, we trained a U-Net to learn a “direct mapping” from Grad-CAM
heatmaps to groundtruth heatmaps which we then applied to the unlabeled data.
This shows a large performance decrease, because it cannot capture the full
distribution that a diffusion model learns.

All the above ablations validate the importance of both the Grad-CAM heatmaps
priors and the priors that the diffusion model learns.

####  4.3.2 Added Noise Magnitude Effects

In this section, we analyze the effect of the added noise magnitude on the
diffusion output. As shown in Tab. 3, adding noise at the 250th step achieved
the best trade-off between the VQA prior and the pretrained annotation prior.
As a reminder, the later the timestep we add the noise, the larger its
magnitude (the more the model ignores the Grad-CAM).

Grad-CAM | $t=100$ | $\bm{t=250}$ | $t=400$ | Pure noise
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Figure 5:  Diffusion model output for noise added at different timesteps. Red
dots represent the ground truth annotation. Adding noise at earlier steps
generates outputs on high Grad-CAM response regions. Adding noise at later
steps generates outputs similar to sampling from pure noise. Noise at step 250
is the best trade-off.

We further visualized the outputs from the same diffusion model when adding
noise at different time steps in Fig. 5. In the top 2 rows, the Grad-CAM
heatmaps highlight correct locations, whereas the Grad-CAM heatmaps of the
bottom 2 rows are noisy or highlight incorrect locations. In all these cases,
when adding noise at the 100th step, the outputs are located on the
highlighted region of the Grad-CAM heatmap. In contrast, when adding noise at
the 400th step, the large magnitude of added noise leads to outputs similar to
sampling directly from Gaussian noise without using the Grad-CAM heatmaps.
Adding noise at step 250 appears to be the best trade-off.

Table 4: Results of using Grad-CAM heatmaps as a direct input. When Grad-CAM heatmaps are directly input into the gaze inference process, both the VAT and the diffusion model show significant improvement in performance. Method |  Auxiliary Input | 5% labels | 10% labels | 20% labels
---|---|---|---|---
Dist. $\downarrow$ | AUC $\uparrow$ | Dist $\downarrow$ | AUC $\uparrow$ | Dist. $\downarrow$ | AUC $\uparrow$
Avg. | Min. | Avg. | Min. | Avg. | Min.
VAT | None | 0.230 | 0.161 | 0.835 | 0.202 | 0.133 | 0.869 | 0.182 | 0.116 | 0.892
VAT-GC | Grad-CAM | 0.210 | 0.144 | 0.848 | 0.188 | 0.121 | 0.884 | 0.173 | 0.107 | 0.898
Diffusion | None | 0.230 | 0.160 | 0.768 | 0.203 | 0.135 | 0.847 | 0.189 | 0.124 | 0.849
Diffusion-GC | Grad-CAM | 0.199 | 0.131 | 0.803 | 0.177 | 0.112 | 0.846 | 0.167 | 0.103 | 0.870
|  |  |  |  |  |  |  |  |  |

####  4.3.3 The Effect of the Grad-CAM Heatmaps

In this section, we test how much guidance signal is provided by the VQA
priors. This cannot be directly tested in the semi-supervised scenario, so we
tested by using the Grad-CAM heatmaps as a direct input to a fully supervised
gaze following model. We train the teacher models of Sec. 4.2 using different
amounts of labeled data only, and introduce a new baseline: Diffusion-GC,
where the diffusion model trained on labeled images was used to refine the
Grad-CAM heatmaps computed directly on the test set. Note that VAT-GC and
Diffusion-GC require the Grad-CAM heatmaps of the test images during
inference, which may not be feasible in an online setting as it necessitates
access to a large vision-language model.

Tab. 4 presents the results. VAT-GC outperforms the baseline VAT model in all
cases, and Diffusion-GC performs best in the distance metrics. Both the VAT
and diffusion model show significant performance boosts when Grad-CAM heatmaps
are directly input into the gaze following models, which offers evidence that
the VQA Grad-CAM heatmaps bring strong prior knowledge. As illustrated in Sec.
4.2, the lower AUC for Diffusion and Diffusion-GC can be attributed to the AUC
on GazeFollow being evaluated with group-level annotations and favoring
heatmaps spanning larger areas. We analyzed this in more detail in the
supplementary material.

###  4.4 Semi-supervised Gaze Following for Video

In this experiment, we finetune a pretrained image gaze following model to
specific video scenes in a semi-supervised manner. We experiment on 10 TV
shows from the VideoAttentionTarget dataset [9], comprising a total of 31,978
gaze annotations. For each video, we randomly select a clip that contains
about 100 annotated frames (2–23% of the entire video, mostly below 10%) as
groundtruth data. The remainder of the videos are regarded as unlabeled.
Following [9], we started with the image version of the teacher model pre-
trained on the GazeFollow dataset and extended it temporally through an
additional ConvLSTM network. The model was then fine-tuned on the given
annotations and the generated pseudo-annotations for each video. The
experiment simulates a real-world scenario where only a few frames of a
specific scene are annotated.

Fig. 6 presents the $L_{2}$-distance and AUC in video adaptation experiments
on each of the 10 videos. Our method outperforms other pseudo-annotation
generation methods in both metrics on almost all videos. When the baseline
models do not perform well, the improvement of our method is more pronounced
(2%–5% improvement in both metrics, more details in supplementary). These
findings demonstrate that our method can be used effectively in video gaze
following, needing to initially label only a few frames.

Figure 6: Applying gaze following to videos in a semi-supervised manner. GCDR
achieves the best performance on almost all videos (Dist and AUC averaged
across videos: GCDR: 0.142, 0.843; Semi-VAT-GC: 0.155, 0.818; Semi-VAT: 0.164,
0.797 ; VAT: 0.166, 0.773). More details in the Supplementary.

##  5 Conclusion

We have proposed the first approach for generating high-quality pseudo-labels
for the semi-supervised gaze following task. We leverage the priors from large
VL models by computing Grad-CAM heatmaps from a pretrained VQA model that is
prompted with a gaze following question. The Grad-CAM heatmaps offer strong
guidance to the gaze target, but can be noisy. This led to a novel diffusion-
based refinement method that refines these initial pseudo annotations with an
annotation prior. Our approach works well on both image and video gaze
following tasks with significant savings in the annotation effort. We hope our
method will lead to the collection of larger gaze following datasets with
annotation effort similar to current datasets. We plan to apply diffusion-
based refinement to “noisy” annotations generated by VL models, in new semi-
supervised tasks.

## References

  * [1] Admoni, H., Scassellati, B.: Social eye gaze in human-robot interaction: a review. Journal of Human-Robot Interaction 6(1), 25–63 (2017)
  * [2] Athiwaratkun, B., Finzi, M., Izmailov, P., Wilson, A.G.: There are many consistent explanations of unlabeled data: Why you should average. ICLR (2019)
  * [3] Austin, J., Johnson, D.D., Ho, J., Tarlow, D., Van Den Berg, R.: Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems 34, 17981–17993 (2021)
  * [4] Bao, J., Liu, B., Yu, J.: Escnet: Gaze target detection with the understanding of 3d scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 14126–14135 (June 2022)
  * [5] Cañigueral, R., Hamilton, A.F.d.C.: The role of eye gaze during natural social interactions in typical and autistic people. Frontiers in psychology 10, 560 (2019)
  * [6] Chefer, H., Gur, S., Wolf, L.: Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 397–406 (2021)
  * [7] Chen, B., Jiang, J., Wang, X., Wan, P., Wang, J., Long, M.: Debiased self-training for semi-supervised learning. Advances in Neural Information Processing Systems 35, 32424–32437 (2022)
  * [8] Chong, E., Ruiz, N., Wang, Y., Zhang, Y., Rozga, A., Rehg, J.M.: Connecting gaze, scene, and attention: Generalized attention estimation via joint modeling of gaze and scene saliency. In: Proceedings of the European conference on computer vision (ECCV). pp. 383–398 (2018)
  * [9] Chong, E., Wang, Y., Ruiz, N., Rehg, J.M.: Detecting attended visual targets in video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5396–5406 (2020)
  * [10] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems 34, 8780–8794 (2021)
  * [11] Drewes, H.: Eye gaze tracking for human computer interaction. Ph.D. thesis, lmu (2010)
  * [12] Fang, Y., Tang, J., Shen, W., Shen, W., Gu, X., Song, L., Zhai, G.: Dual attention guided gaze target detection in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11390–11399 (2021)
  * [13] French, G., Laine, S., Aila, T., Mackiewicz, M., Finlayson, G.: Semi-supervised semantic segmentation needs strong, varied perturbations. In: British Machine Vision Conference. No. 31 (2020)
  * [14] Giannone, G., Nielsen, D., Winther, O.: Few-shot diffusion models. In: NeurIPS 2022 Workshop on Score-Based Methods (2022)
  * [15] Grandvalet, Y., Bengio, Y.: Semi-supervised learning by entropy minimization. Advances in neural information processing systems 17 (2004)
  * [16] Gupta, A., Tafasca, S., Odobez, J.M.: A modular multimodal architecture for gaze target prediction: Application to privacy-sensitive settings. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 5041–5050 (2022)
  * [17] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 2961–2969 (2017)
  * [18] Ho, C.J., Tai, C.H., Lin, Y.Y., Yang, M.H., Tsai, Y.H.: Diffusion-ss3d: Diffusion model for semi-supervised 3d object detection. Advances in Neural Information Processing Systems 36 (2024)
  * [19] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33, 6840–6851 (2020)
  * [20] Horanyi, N., Zheng, L., Chong, E., Leonardis, A., Chang, H.J.: Where are they looking in the 3d space? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2677–2686 (2023)
  * [21] Hu, Z., Yang, D., Cheng, S., Zhou, L., Wu, S., Liu, J.: We know where they are looking at from the rgb-d camera: Gaze following in 3d. IEEE Transactions on Instrumentation and Measurement 71, 1–14 (2022)
  * [22] Hu, Z., Yang, Y., Zhai, X., Yang, D., Zhou, B., Liu, J.: Gfie: A dataset and baseline for gaze-following from 2d to 3d in indoor environments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8907–8916 (2023)
  * [23] Hung, W.C., Tsai, Y.H., Liou, Y.T., Lin, Y.Y., Yang, M.H.: Adversarial learning for semi-supervised semantic segmentation. In: 29th British Machine Vision Conference, BMVC 2018 (2019)
  * [24] Jin, T., Lin, Z., Zhu, S., Wang, W., Hu, S.: Multi-person gaze-following with numerical coordinate regression. In: 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021). pp. 01–08. IEEE (2021)
  * [25] Jin, T., Yu, Q., Zhu, S., Lin, Z., Ren, J., Zhou, Y., Song, W.: Depth-aware gaze-following via auxiliary networks for robotics. Engineering Applications of Artificial Intelligence 113, 104924 (2022)
  * [26] Kalluri, T., Varma, G., Chandraker, M., Jawahar, C.: Universal semi-supervised semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5259–5270 (2019)
  * [27] Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., Carion, N.: Mdetr-modulated detection for end-to-end multi-modal understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1780–1790 (2021)
  * [28] Lai, Z., Duan, Y., Dai, J., Li, Z., Fu, Y., Li, H., Qiao, Y., Wang, W.: Denoising diffusion semantic segmentation with mask prior modeling. arXiv preprint arXiv:2306.01721 (2023)
  * [29] Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. In: International Conference on Learning Representations (2017), https://openreview.net/forum?id=BJ6oOfqge
  * [30] Lee, D.H., et al.: Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on challenges in representation learning, ICML. vol. 3, p. 896. Atlanta (2013)
  * [31] Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning. pp. 12888–12900. PMLR (2022)
  * [32] Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., Hoi, S.C.H.: Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems 34, 9694–9705 (2021)
  * [33] Li, Y., Pan, Q., Wang, S., Peng, H., Yang, T., Cambria, E.: Disentangled variational auto-encoder for semi-supervised learning. Information Sciences 482, 73–85 (2019)
  * [34] Lian, D., Yu, Z., Gao, S.: Believe it or not, we know what you are looking at! In: Asian Conference on Computer Vision. pp. 35–50. Springer (2018)
  * [35] Massé, B., Ba, S., Horaud, R.: Tracking gaze and visual focus of attention of people involved in social interaction. IEEE transactions on pattern analysis and machine intelligence 40(11), 2711–2724 (2017)
  * [36] Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.Y., Ermon, S.: Sdedit: Guided image synthesis and editing with stochastic differential equations. In: International Conference on Learning Representations (2021)
  * [37] Menon, S., Vondrick, C.: Visual classification via description from large language models. In: The Eleventh International Conference on Learning Representations (2022)
  * [38] Miao, Q., Hoai, M., Samaras, D.: Patch-level gaze distribution prediction for gaze following. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 880–889 (2023)
  * [39] Morimoto, C.H., Mimica, M.R.: Eye gaze tracking techniques for interactive applications. Computer vision and image understanding 98(1), 4–24 (2005)
  * [40] Nichol, A.Q., Dhariwal, P.: Improved denoising diffusion probabilistic models. In: ICML. Proceedings of Machine Learning Research, vol. 139, pp. 8162–8171. PMLR (2021)
  * [41] Nie, W., Guo, B., Huang, Y., Xiao, C., Vahdat, A., Anandkumar, A.: Diffusion models for adversarial purification. In: ICML. Proceedings of Machine Learning Research, vol. 162, pp. 16805–16827. PMLR (2022)
  * [42] Paige, B., van de Meent, J.W., Desmaison, A., Goodman, N., Kohli, P., Wood, F., Torr, P., et al.: Learning disentangled representations with semi-supervised deep generative models. Advances in neural information processing systems 30 (2017)
  * [43] Pratt, S., Covert, I., Liu, R., Farhadi, A.: What does a platypus look like? generating customized prompts for zero-shot image classification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15691–15701 (2023)
  * [44] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [45] Recasens, A., Khosla, A., Vondrick, C., Torralba, A.: Where are they looking? In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 28. Curran Associates, Inc. (2015), https://proceedings.neurips.cc/paper/2015/file/ec8956637a99787bd197eacd77acce5e-Paper.pdf
  * [46] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR. pp. 10674–10685. IEEE (2022)
  * [47] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI (3). Lecture Notes in Computer Science, vol. 9351, pp. 234–241. Springer (2015)
  * [48] Sakita, K., Ogawara, K., Murakami, S., Kawamura, K., Ikeuchi, K.: Flexible cooperation between human and robot by interpreting human intention from gaze information. In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)(IEEE Cat. No. 04CH37566). vol. 1, pp. 846–851. IEEE (2004)
  * [49] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
  * [50] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
  * [51] Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., Kiela, D.: Flava: A foundational language and vision alignment model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15638–15650 (2022)
  * [52] Singh, R., Miller, T., Newn, J., Velloso, E., Vetere, F., Sonenberg, L.: Combining gaze and ai planning for online human intention recognition. Artificial Intelligence 284, 103275 (2020)
  * [53] Sohl-Dickstein, J., Weiss, E.A., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: Bach, F.R., Blei, D.M. (eds.) Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015. JMLR Workshop and Conference Proceedings, vol. 37, pp. 2256–2265. JMLR.org (2015), http://proceedings.mlr.press/v37/sohl-dickstein15.html
  * [54] Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C.A., Cubuk, E.D., Kurakin, A., Li, C.L.: Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems 33, 596–608 (2020)
  * [55] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. In: International Conference on Learning Representations (2020)
  * [56] Song, Y., Shen, L., Xing, L., Ermon, S.: Solving inverse problems in medical imaging with score-based generative models. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=vaRCHVj0uGI
  * [57] Springenberg, J.T.: Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv preprint arXiv:1511.06390 (2015)
  * [58] Tafasca, S., Gupta, A., Odobez, J.M.: Childplay: A new benchmark for understanding children’s gaze behaviour. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20935–20946 (2023)
  * [59] Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems 30 (2017)
  * [60] Tonini, F., Beyan, C., Ricci, E.: Multimodal across domains gaze target detection. In: Proceedings of the 2022 International Conference on Multimodal Interaction. pp. 420–431 (2022)
  * [61] Tonini, F., Dall’Asen, N., Beyan, C., Ricci, E.: Object-aware gaze target detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 21860–21869 (2023)
  * [62] Tu, D., Min, X., Duan, H., Guo, G., Zhai, G., Shen, W.: End-to-end human-gaze-target detection with transformers. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2192–2200. IEEE (2022)
  * [63] Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., Yang, H.: Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In: International Conference on Machine Learning. pp. 23318–23340. PMLR (2022)
  * [64] Yang, X., Song, Z., King, I., Xu, Z.: A survey on deep semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering (2022)
  * [65] You, Z., Zhong, Y., Bao, F., Sun, J., Li, C., Zhu, J.: Diffusion models and semi-supervised learners benefit mutually with few labels. Advances in Neural Information Processing Systems 36 (2024)
  * [66] Zeng, Y., Zhang, X., Li, H.: Multi-grained vision language pre-training: Aligning texts with visual concepts. In: International Conference on Machine Learning. pp. 25994–26009. PMLR (2022)
  * [67] Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. International Journal of Computer Vision 130(9), 2337–2348 (2022)