Bear

# Unifying (Machine) Vision via Counterfactual World Modeling

Daniel M. Bear  Department of Psychology, Stanford University, Stanford, CA
94305  Kevin Feigelis  Department of Physics, Stanford University, Stanford,
CA 94305  Honglin Chen  Department of Computer Science, Stanford University,
Stanford, CA 94305  Wanhee Lee  Department of Applied Physics, Stanford
University, Stanford, CA 94305  Rahul Venkatesh  Department of Computer
Science, Stanford University, Stanford, CA 94305  Klemen Kotar  Department of
Computer Science, Stanford University, Stanford, CA 94305  Alex Durango
Neurosciences Graduate Program, Stanford University, Stanford, CA 94305
Daniel L. K. Yamins  Department of Psychology, Stanford University, Stanford,
CA 94305  Department of Computer Science, Stanford University, Stanford, CA
94305  Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA
94305

(May 2023)

###### Abstract

Leading approaches in machine vision employ different architectures for
different tasks, trained on costly task-specific labeled datasets. This
complexity has held back progress in areas, such as robotics, where robust
task-general perception remains a bottleneck. In contrast, "foundation models"
of natural language have shown how large pre-trained neural networks can
provide zero-shot solutions to a broad spectrum of apparently distinct tasks.
Here we introduce Counterfactual World Modeling (CWM), a framework for
constructing a visual foundation model: a unified, unsupervised network that
can be prompted to perform a wide variety of visual computations. CWM has two
key components, which resolve the core issues that have hindered application
of the foundation model concept to vision. The first is structured masking, a
generalization of masked prediction methods that encourages a prediction model
to capture the low-dimensional structure in visual data. The model thereby
factors the key physical components of a scene and exposes an interface to
them via small sets of visual tokens. This in turn enables CWM’s second main
idea – counterfactual prompting – the observation that many apparently
distinct visual representations can be computed, in a zero-shot manner, by
comparing the prediction model’s output on real inputs versus slightly
modified ("counterfactual") inputs. We show that CWM generates high-quality
readouts on real-world images and videos for a diversity of tasks, including
estimation of keypoints, optical flow, occlusions, object segments, and
relative depth. Taken together, our results show that CWM is a promising path
to unifying the manifold strands of machine vision in a conceptually simple
foundation.

###### keywords:

computer vision | vision science | mid-level vision | foundation models | world models | counterfactuals

dyamins and gmail.com

##  1 Introduction

Within their first years of life, humans acquire an unparalleled ability to
understand visual scenes and make predictions about them: what objects are
present, how they are arranged in space, where they will (or could) move, and
how they might interact. These abilities in turn support a panoply of complex
behaviors, such as object manipulation and tool use, long-range physical
prediction and planning, spatial navigation, language grounding, and social
interaction.

Despite great recent progress in modeling high-level cognitive functions, such
as language understanding and generation, even today’s large-scale AI
algorithms fall far short of human-like visual scene-understanding. This has
held back artificial systems, such as robots, that would profitably use such
abilities for model-predictive control and planning. Importantly, because they
arise so early in human development – and are shared in part by other animals,
like non-human primates – visual scene understanding and world-modeling
abilities almost certainly do not require language learning or supervision.
This suggests that bridging the machine-human gap in vision (or pre-linguistic
sensorimotor abilities more broadly) may require more powerful and efficient
new forms of unsupervised learning.

One appealing approach, inspired by vision science and visual systems
neuroscience, is to break the problem of scene understanding down into sub-
tasks that can be solved with special-purpose modules. The human visual system
extracts a wide range of low- and mid-level representations of scenes,
including (among many others) contours and border ownership [1], optical flow
and self-induced motion [2], perceptual groups and figure-ground segmentations
[3, 4], a “2.5D sketch” of depth and surface normal vectors [5], and estimates
of 3D object shape [6]. Individual computer vision algorithms successfully
model some of these “intermediate” computations, but most appear to be well
below human level and fail to capture more complex visual behaviors [7],
raising deep questions about this agenda: which intermediates are necessary
and sufficient for holistic scene understanding? How can better algorithms be
found without hard-to-obtain labeled data?111…which humans and animals do
without. While supervised algorithms for (e.g.) object categorization or
segmentation can succeed in producing partially accurate proxies for adult
visual behaviors [8] and their neural correlates [9], these approaches are
implausible as models of real (i.e. unsupervised) visual learning [10]. Are
improvements in one module useful for improving others? Do the many
intermediates need to be assembled carefully into a unified architecture, or
can a generic procedure discover automatically how to make use of them?

A promising alternative may dispense with these challenges. Recently, the
“foundation model” approach [11] has driven rapid progress in natural language
generation and understanding tasks, without breaking them down into pieces
that require special methods. At a high level, this approach has two key
components:

  * •

A _pretrained model_ , typically a large, relatively unspecialized neural
network, trained to solve a “masked token prediction” task on a large real-
world dataset; and,

  * •

A _generic task interface_ , a simple process for translating any task within
a wide domain into a zero- (or few-)shot input to the pretrained model.

Figure 1:  Counterfactual World Modeling unifies visual computations. A
Counterfactual World Model, $\Psi$, takes an input image (or video) and
constructs an internal representation of the scene. This representation can be
used to generate counterfactual simulations of what would happen if different
events occurred (top right) and to estimate properties of the real scene
(static and dynamic predictions, middle and bottom right). Running
counterfactual simulations requires input to $\Psi$ in the form of
“counterfactual prompts,” which indicate which image patches are moving (green
squares with arrows) and which are static (red squares.) Predictions about a
scene’s dynamical properties (keypoints, forward predictions, optical flow,
and occlusions) require a second frame of the true input video.

In the case of language, simple and powerful versions of both of these ideas
have been discovered. Effective large language models, like GPT-n [12], are
trained on large corpuses of web text to take in a context string and predict
the likely next word. The next-word prediction task is straightforward to
formulate (if not to solve) and remains unsurpassed, despite attempts to
identify more powerful training methods. Moreover, the form of this training
immediately yields a universal interface: a well-trained model prompted with a
natural language description of a task – that is, provided with this
description as its context string – must provide an attempted solution, its
prediction of what text would naturally follow the prompt. Thus language
foundation models have gained widespread success from their remarkable
simplicity, compared to the wiring together of specialized subsystems that
predominates in computer vision.

But applying the foundation model framework to vision has not been as
straightforward. Both key ingredients are harder to formulate and successfully
implement: masked prediction in vision, in the guise of future prediction of
video frames [13, 14] (or meaningful image-derived latents [15]) from previous
context frames (or latents), has remained a largely unsolved problem [7]; and
it is not at all clear how to elicit most of the “intermediate” computations
above through an interface to a single vision model. For instance, what sort
of input – which must be visual in nature, if training were purely on
unlabeled visual data – would cause a vision model to output object segments?
Recent unified vision models emphasize a shared architectural stem from which
many outputs are produced, but they all require some combination of (i)
supervised training on labeled datasets for each supported task, (ii) task-
specific decoder architectures, and (iii) a language prompting interface (and
therefore training on joint visual-textual data at some stage) [16, 17, 18,
19, 20, 21]. They do not explain, therefore, how to derive a task-general
interface from a single unsupervised training process, which we take to be the
kernel of the foundation model concept.

Here, we propose to resolve these issues with an approach we call
_Counterfactual World Modeling (CWM)_. The CWM framework has two basic
concepts, which build on the two fundamental components of foundation models:

  1. 1.

[Pretraining via structured masked prediction.] The recent Masked Autoencoder
models [22] directly translate the masked prediction task from text to visual
data (images and video); however, these models do not appear to learn
representations as general as language models. In stark contrast to the
language domain, we find that simple but nontrivial variations of the masks
used to train vision models produce highly distinct learned representations.
In particular, we show that a _temporally-factored_ masking structure teaches
models to separate _appearance_ information from _dynamics_ information. The
resulting prediction model encodes powerful implicit information about the
physical structure of scenes, objects, and their dynamics.

  2. 2.

[Generic task interface via counterfactual prompting.] Different visual
representations do not, at first blush, appear to have much in common: state-
of-the-art computer vision uses quite distinct methods to extract, e.g.,
keypoints, optical flow, and object segments, with few obvious relationships
between them. Yet we show that many such concepts arise from the above
prediction model via counterfactual prompting: by systematically modifying
visual inputs in simple ways and inspecting the perturbed model responses,
core computer vision concepts can be extracted in a zero-shot manner. For
example, extracting optical flow is equivalent to asking for a prediction of
where a hypothetical visual marker would move from one frame to the next.
Moreover, we show that these counterfactuals can be mathematically formalized
as derivatives of the underlying prediction model itself, which provides a way
both to identify “natural” counterfactual prompts and to implement them
efficiently with autograd methods. Figure 1 illustrates the use of a
Counterfactual World Model, $\Psi$, in both generating counterfactual
simulations and predicting properties of the true visual input.

In this work we show how basic application of Counterfactual World Modeling
can construct a variety of visual concepts, including keypoints, optical flow
and occlusion, object segmentation, and monocular depth. Several of these
constructions involve “bootstrapping,” using one extracted representation to
help extract another. Bootstrapping is simple in the CWM framework because it
is effectively an application of the chain rule; for example, we illustrate
below how object segmentation can be considered the derivative of a model that
estimates optical flow, which in turn is the derivative of a temporally-
factored masked prediction model.

Our results illustrate how a high-performing, “true” foundation model can be
formalized and trained on unlabeled visual data. The CWM approach thus
synthesizes a plethora of classical computer vision concepts within the modern
foundation model framework. Because CWM does not require supervision or
specialized architectures for different sub-tasks, it offers a practical
solution for many visual problems and – for the first time – a unified
computational theory of how these abilities could arise in a biological
system.

##  2 Masked Predictors

Many classical forms of unsupervised learning can be generalized as a simple
training task on unlabeled data, masked token prediction. Each element of a
dataset (e.g. a piece of text, an image, a video, or an audio stream), is
broken into non-overlapping chunks (“tokens”); during training, a portion of
the tokens is randomly masked out and the parameters of a neural network are
optimized to predict this missing information from the remaining “visible”
(unmasked) tokens. Intuitively, because the neural network model must infer
things about the structure of the training data to do this task well, its
learned representation may generalize to other downstream tasks that depend on
that structure.

This intuition has been borne out dramatically in the case of Large Language
Models (LLMs) and text data. Highly capable and general language models can be
trained on an especially simple version of masked token prediction in which
only the final token (roughly, the last word) of a text stream must be
predicted from prior context. This model and training structure creates a
natural, universal task interface: different tasks are turned into queries
simply by feeding the task description, which is a text string, as input to
the prediction model.

The Masked Autoencoder (MAE) framework222The MAE is unfortunately named, as
nothing in an input image is autoencoded: the patches to be filled in are
specifically not part of the input to the model. For this reason, we adopt the
term masked prediction throughout. is a direct analogue of LLMs’ model
architecture and masked token prediction training to visual data [22]. MAEs
have gained widespread use because they are easy to scale and transfer to
downstream tasks (via supervision) about as well as more complex unsupervised
methods. The largest MAEs (and their natural extension to multi-frame videos,
VMAEs) learn to capture some of the higher-level structure in visual data,
such as the shapes, textures, and subpart arrangements of common objects.

However, (V)MAEs appear to lack LLMs’ generality and power as both an
unsupervised representation learning method and a universal task interface.
While (V)MAEs perform only slightly worse than alternative unsupervised
learning frameworks, such as contrastive learning [23, 24], the state-of-the-
art methods for most vision tasks are almost exclusively supervised models
with specialized architectures [25]; this is emphatically different from
language modeling, where foundation models are to a first approximation the
best at all language tasks. Furthermore, as with all other unsupervised vision
approaches, (V)MAEs are applied to various tasks through transfer learning.
This requires further training on separate labeled datasets, in contrast to
LLMs’ direct applicability to any language task via zero-shot prompting.

Why is it hard to fully translate LLMs’ success into a vision foundation
model? We hypothesize that it is fundamentally because single text tokens –
roughly, words – more strongly constrain the statistics of surrounding text
than do single visual patch tokens. Reading the word “rhino” dramatically
narrows the sequence of words that can be expected to follow, whereas seeing a
small patch of an image only very weakly constrains what objects are present
in the image, where they are in 3D space, how they are moving, and so on. This
is why it is possible to use LLMs as a universal task interface: a small
number of text tokens is often enough context to serve as a meaningful task
prompt. (V)MAEs, on the other hand, require many tokens (e.g. 10% of an image)
before they begin to generate plausible completions of visual data, making it
very challenging to steer such models via prompting.

Yet there is substantial, low-dimensional structure in visual data. Our key
insight is that this structure can be captured and concentrated into a small
number of visual tokens. This works by training masked prediction models with
more structured forms of token masking (i.e., not uniformly sampling visual
patches to be masked out). In particular, we note that visual scene dynamics
are highly constrained on very short timescales, because they stem from a few
physical objects moving in a relatively small number of ways. A form of
training we call temporally-factored masked prediction causes a standard
vision transformer network to learn this structure implicitly, which creates a
natural set of “handles” with which visual data can be manipulated explicitly
– the basis for steering the model to do different visual tasks (detailed in
§3). Together the procedures of (i) training a model with structured masking
and (ii) prompting it with simple counterfactual inputs constitute
Counterfactual World Modeling.

Figure 2:  Masked Predictors. A) The Masked Autoencoder (MAE) masking policy
reveals an intermediate fraction ($\sim$ 25 %) of patches in a frame. The
standard Video MAE policy extends the MAE policy uniformly to video frames. B)
The temporally-factored masked predictor policy asymmetrically reveals all the
patches in frame0 and a small fraction ($\sim$ 1%) of patches in frame1. This
masking policy forces the resulting predictor $\Psi_{\textbf{TMP}}$ to factor
appearance and dynamics.  Figure 3:  Factual and Counterfactual Predictions
with the temporally-factored masked predictor $\Psi_{\textbf{TMP}}$. A)
Factual Predictions. Sequence of predictions as more patches from frame 1 are
revealed. Top row: This case consists of two frames drawn 2.4s apart from a
Kinetics 400 movie. Here, the first patch fixes the camera panning by
revealing a little bit of the background. The second patch essentially sets
the location of the foreground object (the horse and rider). Subsequent
patches fill in details of the object motion (e.g. horse leg positioning).
Bottom row: This case consists of two frames drawn 2.2s apart from a home
movie captured on a smartphone. Here, the first patch reveals that the person
has moved forward and to the right in the frame. The subsequent several
patches then fix the angle of the face, keeping the expression constant. The
last several patches reveal that the expression has changed to a smile. B)
Counterfactual Predictions. $\Psi$ can simulate counterfactual motions from
single frames. The input to the predictor is frame0 together with a simulated
second frame in which one or several patches from frame0 have been moved to
desired locations to seed motion (see Fig 6a for illustration.) Choosing
different patches to move and directions in which to move them generates
different counterfactual next-frames.

Factoring appearance from dynamics via structured masking. Masked prediction
models may learn different things depending on how tokens of data are masked
during training. A _masking policy_ is a (random-valued) function $g$ that
assigns to each potential input $\mathbf{x}$ a subset of token indices in
$\mathbf{x}$ that will be _visible_. The _masked prediction model associated
with $g$_ is the function $\Psi_{g}$ that takes as input the visible tokens
(those at indices $i$ for which $i\in g(\mathbf{x})$) and estimates the values
of invisible tokens (those at indices $i$ for which $\mathbf{i}\notin
g(\mathbf{x})$).333See supplementary section A.1 for a more precise
mathematical definition.

The standard masked autoencoder $\Psi_{\textbf{MAE}}$ is defined by the
uniform masking policy $g_{p}$ that, for some fixed fraction $p\in[0,1]$,
masks out uniformly sampled444Typically implemented by randomly permuting the
patch indices and masking out the last $p$. proportion $p$ of patches of the
input (Fig. 2a) [22]. The VMAE is given by extending this policy to all frames
in a video or by uniformly sampling “tubes” of patches at the same location
across multiple frames [26, 27]. For the MAE, optimal values of $p$ for
representation learning tend to occur at $p\sim 0.75$ [22]. This is because
setting $p$ too high provides too little information to train an effective
prediction model, while setting $p$ too low provides _too much_ information
and thus not enough incentive to learn general visual representations.

Here, we introduce an alternative masking scheme, based on the observation
that on very short timescales ($\sim$250 ms) the next frame of a video can be
predicted well simply by copying large chunks of the previous frame to nearby
locations; the chunks correspond to independently moving objects, and object
deformation and out-of-plane motion play a far smaller role than at longer
timescales. In other words, information about scene dynamics has low-
dimensional structure in this regime even if per-frame appearance does not.
This asymmetry between dynamics and appearance can be harnessed with an
asymmetric masking policy.

Specifically, given an RGB frame pair $\mathbf{x}=(x_{t},x_{t+\Delta})$,
consider the policy $g_{q,p}$ in which the fraction $q$ of patches in frame
$x_{t}$ are masked, while fraction $p$ of patches in $x_{t+\Delta}$ are
masked. When $q\sim 0$ and $p\sim 1$ (we set $q=0$ and $p=0.99$), the model
$\Psi_{g}$ must complete the second frame given only a few patches of it, plus
the entire first frame (Fig 2b); the only way it can do this is by detecting
how objects are moving from the small set of second-frame patches, then
applying these transformations to the full objects in the first frame. Thus a
prediction model trained with this temporally-factored masking policy,
$\Psi_{\textbf{TMP}}$, will learn implicitly to factor information about scene
state (e.g. object position, pose, size, and camera motion) from information
about scene dynamics – if it learns a general solution at all. Moreover, the
higher the second-frame masking fraction $p$, the more information about scene
dynamics will have to be concentrated into a small number of visual tokens.

Temporal factoring in “factual” predictions. We trained $\Psi_{\textbf{TMP}}$
on a real-world video dataset (Kinetics400 [28]) and evaluated its predictions
when prompted with small sets of second-frame patches (see Supplementary
section A.3 for training details). This revealed that $\Psi_{\textbf{TMP}}$
does in fact learn to factor appearance from dynamics. Going forward, we will
drop the subscript and refer to this prediction model simply as $\Psi$.

The factoring properties of $\Psi$ can be seen when performing a series of
increasingly accurate “factual predictions” of actual videos (Fig 3a). Given a
frame pair $(x_{t},x_{t+\Delta})$ taken from a real movie, we start with one
patch in $x_{t+\Delta}$ given and observe how the prediction improves and
changes as additional patches are revealed. As the figure illustrates, a
single revealed patch allows $\Psi$ to capture large-scale changes in the
scene, e.g. camera panning causing background motion, or gross object motion.
As one or two more patches are added, additional, independent features of the
true scene dynamics are well-predicted, including foreground object position
and pose changes. As further patches are added, small details come into focus.
Throughout this process, position- and pose-invariant object appearances
change very little (other than becoming less blurry), implying that
“prompting” $\Psi$ with individual patches primarily conveys information about
scene dynamics.

Generating counterfactual predictions. Because its representation of scene
dynamics has been concentrated in a small number of tokens, $\Psi$ supports
the easy generation of _counterfactual predictions_ (Fig. 3b).

By definition, a _counterfactual_ is an input to the predictor that has been
modified from the initial ground-truth input in some systematic way. We use
two basic flavors of counterfactual: “appearance counterfactuals,” which
involve modifications to the first frame ($x_{t}$), and “motion
counterfactuals,” which involve modifications to the second frame ($x_{t+1}$).
Given a counterfactual input, the associated _counterfactual prediction_ is
the output of $\Psi$ on this made-up datum.

Figure 3b shows counterfactual predictions for a series of motion
counterfactuals. These are generated by starting with a single image as the
first-frame input to $\Psi$ and producing counterfactual second-frame inputs
by translating one or two patches in the original image by small amounts. The
results show how it is possible to use a small number of simple counterfactual
modifications to move semantically meaningful sub-components of the image
(e.g. objects) in an independent fashion. For this reason, we will sometimes
refer to $\Psi$ as a _counterfactual world model_ (CWM).

Section §3 shows how many seemingly distinct visual computations can be
formulated as counterfactual comparisons: contrasts between $\Psi$’s
predictions on real inputs versus counterfactual inputs. The CWM’s ability to
be flexibly prompted yields zero-shot solutions to a wide variety of tasks;
this bears much closer resemblance to prompting LLMs with simple context
strings, compared with standard MAEs or VMAEs needing many visible patches to
produce realistic outputs.

Extensions of the prediction model and masking policy. A natural extension of
$\Psi$ offers a way to prompt the model with information about viewer motion,
which can confound estimates of distal scene dynamics. During training the
prediction model can be conditioned both on visible patches of video and on
data correlated with head motion, as several recent large-scale video datasets
come equipped with camera accelerometry (IMU) data [29].555These data are
readily available to biological organisms from in-head ear canal gyroscopes
[30], proprioceptive feedback about relative head-body-angles, and motor
efference copies [31] The joint video-and-head-motion-conditioned masked
predictor is denoted $\Psi^{\textbf{H}}$ (See supplementary section A.5 for
details). We show below that prompting this model with different
counterfactual head motions offers a way to improve object segmentation and
estimate depth from a single image. More generally, a data stream from any
modality (including true actions, like head motions) can be combined with a
visual masked prediction model to discover structure in their joint
distribution.

Furthermore, training with different masking policies can cause masked
predictors to learn about different structure in visual data. These include
policies that (i) blend TMP and MAE properties by partly masking the first
frame, (ii) achieve better backward predictions by combining forward and
reverse asymmetric masking, and (iii) perform extrapolation and interpolation
of masked video frames. See Supplementary section A.6 for further detail.

Keypoints. Keypoints are local features of a scene that capture more
information than other features [32, 33]. In our approach, keypoints are
naturally defined as those tokens that, when made visible to the prediction
model $\Psi$, yield lowest reconstruction error on the rest of the scene.
Keypoints turn out to be good loci for exerting counterfactual control.
Intuitively, this is because they are the few visual tokens most able to
reduce uncertainty and yield a concrete prediction about many other tokens.
Note that, whereas the counterfactual prompting constructions below (see 3)
involve perturbing the visual input to $\Psi$, keypoints arise by varying the
prediction model’s _input mask_ ; we therefore see them as part of the CWM
approach.

Figure 4:  Keypoints. A) For the indicated frame pair, the keypoint map shows
the normalized error reduction achieved by revealing to the predictor $\Psi$
each single patch in frame1 independently. For the same frame pair, B) shows
the curve of prediction error (summed over pixels) as the iterative keypoint
selection algorithm identifies keypoints, while C) shows the keypoints thereby
selected. D) Consolidated keypoint maps as predicted from single frames.

We compute keypoints in two ways: either by considering all potential visual
patches at once (Fig. 4a) or by identifying them sequentially (Fig. 4b-c). The
first algorithm computes the reduction in total reconstruction error upon
revealing the full first frame and each second-frame patch, on its own, to
$\Psi$. This produces a map of each patch’s “importance”, with the most
important patches taken to be keypoints of the scene. A second algorithm
constructs keypoints iteratively, starting with an initial mask and adding
visible tokens one-by-one to greedily reduce the reconstruction error. (See
supplementary section A.4 for formal definitions of these constructions.) The
mask sequences used in Fig.3a are constructed in the iterative fashion from
$\Psi$, showing how semantically different subcomponents of the video
naturally emerge in sequence as large, correlated portions of reconstruction
error are reduced.

Computing keypoints with a masked predictor requires the ground-truth visual
data on which to measure reconstruction error. However, given any masked
prediction model $\Psi_{g}$, it is natural to formulate the problem of
estimating the keypoints of $\Psi_{g}$ from subsets of the patches made
visible under masking policy $g$. Applied to $\Psi_{\textbf{TMP}}$, this boils
down to learning which patches in a static image would likely act as keypoints
if that image were part of a video (Fig. 4d). Qualitatively, this recovers
structures that are locally discriminative and invariant to (small) physical
transformations – the hallmarks of classical keypoints.

##  3 Tasks as Counterfactual Derivatives of $\Psi$

The observations of the previous section show how the CWM, $\Psi$, has
implicitly learned powerful information about the physical structure of scenes
and factors the basic causes of change across frames. The information about
the second frame that comes in the form of a few second-frame patches is
similar to “action” data, describing in an action-space-agnostic way a minimal
amount of visual information necessary for $\Psi$ to identify state changes.
Steerability with a small number of tokens makes the CWM a possible interface
for many visual tasks: different computations can be specified by feeding
$\Psi$ different counterfactual “prompts”. In this section we show how several
complementary visual properties can be conceptually unified: optical flow,
occlusion, movable-object segmentation, and relative depth can all be
extracted from the CWM through variants of the same prompting procedure.

###  3.1 Optical Flow

Since its proposal by J.J.Gibson [34], computation of optical flow has been
considered an essential component of object tracking, spatial navigation,
forward prediction, and other behaviors in both animals and machine vision
systems. Optical flow is formally defined as the visual correspondence between
elements in one frame of a video and elements in a nearby frame. Classical
approaches inferred these spatiotemporal correspondences by solving a
regularized energy minimization problem, essentially trying to find matches
between pixels in adjacent frames [35]. More recent state-of-the-art methods
supervise neural networks on ground truth optical flow from visually and
physically realistic simulated data, using special-purpose architectures that
favor energy minimization-like solutions [36]. Modern optical flow algorithms
are often satisfactory for applications – even on real-world movies outside
their training distribution – but do not unify this representation with other
visual computations and do not explain how it could arise in an unsupervised
way (e.g. in animals.) Furthermore, even the best methods struggle to infer
the motion of low-contrast regions that are intrinsically hard to track, which
is addressed in practice via regularization but points to a missing deeper
principle [37].

In contrast, optical flow can be derived from the temporally-factored masked
predictor $\Psi$ simply by “asking it” where counterfactual visual markers in
frame $x_{t}$ would end up in frame $x_{t+\Delta}$ (Fig 5a). Since $\Psi$ has
learned to predict the dynamics of things in the first frame from small
patches of the second frame, a marker placed on a visible surface should move
to wherever $\Psi$ predicts that piece of surface is moving.

Algorithm 1 Optical Flow via Appearance Counterfactuals

Let $(x_{t},x_{t+1})$ be a pair of frames and $z\subset x_{t+1}$ a subset of
patches in the second frame. To estimate optical flow at location
$\mathbf{i}$:

  1. 1.

Add a small perturbation to $x_{t}$ at location $\mathbf{i}$ to create a
counterfactual first frame

$x_{t}^{\mathbf{i}}=x_{t}+\delta_{\mathbf{i}}.$

  2. 2.

Fill in the rest of the second frame by applying the $\Psi$ to $x_{t}\oplus
z$, which creates a “clean” prediction

$\hat{x}_{t+1}=\Psi(x_{t}\oplus z),$

and to $x_{t}^{\mathbf{i}}\oplus z$, which creates a “perturbed” prediction:

$\hat{x}_{t+1}^{\mathbf{i}}=\Psi(x_{t}^{\mathbf{i}}\oplus z).$

  3. 3.

Compute the _perturbation response_ – the difference between the clean and
perturbed predictions predictions,

$\hat{\delta}_{\mathbf{i}}=|\hat{x}^{\mathbf{i}}_{t+1}-\hat{x}_{t}|.$

  4. 4.

Set the estimated optical flow at $\mathbf{i}$ to be the spatial displacement
between the perturbed location and the peak (argmax over spatial positions) of
the perturbation response $\hat{\delta}_{\mathbf{i}}$.

Figure 5:  Flow extraction. A) Finite perturbation method. Left Panel: Three
small perturbations have been added to the original frame0: a red patch on the
horse’s rear, a green patch on the sky, and a blue patch in front of the
horse’s nose. Middle Panel: The perturbed input frame0 is then carried forward
to a perturbed predicted next-frame by the predictor $\Psi$. The red
perturbation on the horse’s rear has moved with the horse; the green
perturbation in the sky has panned with the camera; and the blue perturbation
has disappeared, having been occluded by the horse. Right Panel: The
perturbation response is the difference between $\Psi$’s prediction of the
perturbed image and $\Psi$’s prediction of the original unperturbed ground
ground-truth image. This can be seen either in the composite RGB difference
image (left) or in each color channel separately. Dotted red line indicates
displacement from original red perturbation in frame0. B) Flows and Occlusions
from Derivatives. The method of derivatives allows for a tensor of
infinitesimal counterfactual perturbations, analogous to those from the finite
perturbation method, to be computed in parallel. The resulting flow and
occlusion maps are shown for four example selections from the DAVIS dataset
[38]. Note that, as non-simulated data, these videos have no ground-truth flow
to compare to.

Below we formalize this counterfactual prompting procedure and explain how it
can be considered, and implemented, as taking a derivative of the learned
function $\Psi$. Note that both in principle and in practice (Fig 5b), this
approach does not require regularization or architectural inductive biases to
compute optical flow on low-contrast regions: to the extent that $\Psi$
generalizes to the counterfactual input, it must properly transform elements
of frame $x_{t}$ regardless of what they look like. Furthermore, the
counterfactual method yields additional information compared to supervising on
ground truth flow, as occlusions are places where a counterfactual marker in
frame $x_{t}$ appears to vanish in frame $x_{t+\Delta}$ (Fig 5a, blue point.)

Optical Flow and Occlusion as Counterfactuals. We can estimate optical flow
using the algorithm shown in Algorithm box 1. Essentially, this is like adding
a small “tracer” to the first frame at $\mathbf{i}$ that makes it
(counterfactually) high-contrast, and therefore easy to localize in the second
frame – assuming $\Psi$ has enough information from $z$ to estimate the
dynamics between $x_{t}$ and $x_{t+1}$.

This algorithm is simple and often effective (see Fig 5a) but can fail in two
ways. First, one of the revealed factual patches in $z$ may cover the place
where the perturbation at $\mathbf{i}$ is expected to move. This is easily
detected because $\hat{\delta}_{\mathbf{i}}$ will have small magnitude
compared to $\delta_{\mathbf{i}}$. It can be remedied by running the above
algorithm for multiple choices of $z$ and taking their average result,
removing samples where $|\hat{\delta}_{p}|/|\delta_{\mathbf{i}}|\ll 1$. In the
scenario where all choices of $z$ lead to small-magnitude or poorly localized
$\hat{\delta}_{\mathbf{i}}$, the content of $x_{t}$ at position $\mathbf{i}$
is likely occluded (or out of frame) in $x_{t+1}$. Thus counterfactual
prompting of the masked predictor $\Psi$ unifies estimation of optical flow
and occlusion in a single algorithm.

Counterfactuals as Derivatives of $\Psi$. A second possible failure mode is
that the perturbed frame $x_{t}^{\mathbf{i}}$ is out of distribution for
$\Psi$. This could happen when the perturbation is either too large, so that
it looks “fake” rather than like a piece of the surface it is placed on, or
too small, so that it cannot be detected and moved accurately. This is
naturally addressed by considering $\Psi$’s infinitessimal responses to
infinitessimal perturbations – which are exactly the derivative of the
prediction model:

Normalized Perturbation Response at $\mathbf{i}$
$\displaystyle=\lim_{\delta\mathbf{i}\rightarrow
0}\frac{|\hat{\delta}_{\mathbf{i}}|}{|\delta\mathbf{i}|}$
$\displaystyle=\lim_{\delta\mathbf{i}\rightarrow
0}\frac{|\hat{x}^{\mathbf{i}}_{t+1}-\hat{x}_{t+1}|}{|x^{\mathbf{i}}_{t}-x_{t}|}$
$\displaystyle=\nabla_{x}\Psi\bigg{|}_{\mathbf{i}}$ (1)

To simultaneously estimate optical flow at all locations in an $H\times W$
frame pair, we can compute the Jacobian of $\Psi$. This is a four-tensor
$\mathcal{J}\Psi$ of shape $H\times W\times H\times W$, which assigns to
element $(i,j,k,l)$ the predictor’s change in output at location $(k,l)$ in
the second frame due to an infinitessimal change at location $(i,j)$ in the
first frame. Applying this derivative implementation to the algorithm above,

$\textbf{flow}(k,l)=\begin{cases}\textbf{undefined
}\text{(disocclusion)}&\quad\text{if
}\textbf{argmax}_{i,j}|\mathcal{J}\Psi(i,j,k,l)|\ll 1\\\
(k,l)-\textbf{argmax}_{i,j}\mathcal{J}\Psi(i,j,k,l)&\quad\text{otherwise}\end{cases}$
(2)

with results averaged over several choices of visible patches $z\subset
x_{t+1}$. (Note that this method detects disocclusion rather than occlusion,
since no perturbation at any location in the first frame will cause a response
at a point that becomes disoccluded in the second frame.) Fig. 5b shows
examples of flow computed this way for several image pairs.

Because it is a tensorial operation, the derivative formulation of the
counterfactual enables a more efficient parallel computation for flow than
serial finite perturbations, implemented practically using Jacobian-vector
products available in autograd packages such as PyTorch or Jax.666Speed can be
further improved by consolidating the derivative-based flow readout as a self-
supervised training signal for an efficient forward-inference predictor.

###  3.2 Movable-Object Segmentation

Perceptual grouping of scenes into discrete objects is a fundamental
computation in humans, where it forms a foundation for higher-level learning
about object properties, dynamics, and relationships [39]. In computer vision
the grouping task is formalized as object instance segmentation.

While it is impossible to pin down general definitions of “object” or
“segmentation,” an early developmental shift in perception suggests the
practically useful notion of a Spelke object. Work by cognitive scientist
Elizabeth Spelke and her colleagues revealed that babies begin life grouping
separated visual elements into objects only when they are moving in concert
[40]; it is not until they are almost a year old that babies expect, for
example, two stationary toys resting on each other to be independently movable
[41]. We therefore define a Spelke object as a collection of physical stuff
that moves together during commonplace physical interactions. This includes
common physical objects like cell phones, wallets, coffee cups, dogs, people,
and cars, as well as complex novel objects from semantically unnamed
categories that nonetheless can be seen to move coherently. The ability to
segment Spelke objects in an unsupervised, category-agnostic fashion is an
open problem in computer vision and a clear desideratum for robotic planning
and control.

In previous work [42], we found that optical flow in real-world videos can
sometimes provide a strong training signal for segmenting static objects; this
is the case when an entire object moves against a static background and is the
only thing moving in the scene (except perhaps a well-known mover object, such
as a hand or robot arm, that can be “explained away.”) Most real-world scene
dynamics are more complex, though, as they include (i) camera motion, (ii)
multiple moving entities, and (iii) deformable objects whose parts may be
moving by different amounts (or not at all.)

Counterfactual World Modeling with $\Psi$ and an optical flow estimator (see
3.1 above) offers a natural way to resolve these issues. Spelke objects can be
extracted by asking not what is moving in a real video (which might be
complicated) but rather what could be moving in a simulated video.
Specifically, motion counterfactuals (see Section 2 for definition and Fig 3b
for examples) applied to a single frame $x$ produce estimates of what the
optical flow would be if some patches were moving and others were static (Fig
6a). A formal algorithm for segmenting a Spelke object procedure is shown in
Algorithm box 2.

Algorithm 2 Movable-Object Segmentation via Motion Counterfactuals

  1. 1.

Pick a patch index $\mathbf{i}$.

  2. 2.

Create the appearance of a counterfactual motion by taking the patch $x[p]$ at
location $\mathbf{i}$ and creating a new frame that is largely blank, but in
which the contents of the patch at location $\mathbf{i}$ has been copied (e.g.
translated) to a new location $\mathbf{i}+\delta$. This creates a mostly-
masked new frame $x^{\prime}_{\delta}$.

  3. 3.

Apply $\Psi$ to the image pair consisting of the full original frame and the
simulated masked motion frame, i.e. by computing

$\hat{x}_{\delta}=\Psi(x\oplus x^{\prime}_{\delta}).$

  4. 4.

Now, compute the counterfactual flow

$f_{\delta}=\textbf{Flow}(x,\hat{x}_{\delta}).$

  5. 5.

Declare the pixels $j$ at which $|f_{\delta}[j]|>0$ to be part of the same
Spelke object as that at location $\mathbf{i}$.

Figure 6:  Movable-object segmentation. A) Segments from counterfactual
motion. By placing a small amount of counterfactual motion at one location on
an object (indicated by the small green square), a counterfactual prediction
is generated in which the whole object moves. The flow between the original
input and the counterfactual prediction isolates the object. B) shows two
examples of this for isolated objects. C) Aggregating multiple motion samples.
In more complex scenes, some motions may involve mutiple objects. In this
case, it is useful to collect multiple counterfactual motions and then compute
motion affinities as the flow correlation over the counterfactuals. D)
Resolving physical ambiguity with multi-point counterfactuals. Some situations
– such as one in which one object is on top of another – will have unavoidable
ambiguities in short-range motion correlations. In this case, placing a single
motion patch (green point) and a single stop-motion patch (red point) can
disambiguate the proper Spelke object segment.  Figure 7:  Some examples of
segmentations. A) Examples where a single moving patch is sufficient to
isolate Spelke objects. The selected patch is indicated by the small green
square in each image. B) Examples where one moving patch (green square) and
one stopped patch (red square) is sufficient. C) Isolating the people in a
picture of (most of) the authorship team of this paper.

The algorithm above works well for many objects in real-world scenes, as
counterfactually moving a single patch causes the whole object to move (Fig
6b). This demonstrates that the notion of a Spelke object is learned
implicitly by $\Psi$, without any segmentation labels: segmentation ability
arises purely from having to predict which parts of a scene will move together
under a wide variety of circumstances. However, even this one conception of
perceptual grouping contains a number of subtleties that must be addressed to
arrive at a general-purpose segmenter; below we explain and show how more
sophisticated Counterfactual World Modeling can deal with them.

Spelke Affinities As Derivatives. As discussed above, many possible scene
dynamics do not correspond to a single, entire objects moving on a static
background. Consider the limbs of an animal moving independently of its trunk,
or one moving object colliding with another, for example. Rather than thinking
of coherent motion as all-or-none segments, it makes sense to estimate motion
affinities [42] between pairs of scene elements, which can be low in one scene
but high in another; it is the set of possible affinities – that is, how often
would two elements move together across a wide variety of dynamic scenes –
that supports a good representation of how objects should be grouped and how
they can behave.

This relaxation of segments to pairwise affinities allows the above algorithm
to be interpreted as another derivative of $\Psi$ (combined with the optical
flow estimator.) We define the Spelke affinity between two points $i$ and $j$
to be the response in the flow estimate at $j$ given a counterfactual change
in the flow estimate at $i$. For any input frame pair $(x,y)$ (including frame
pairs for which $x=y$, i.e. static images), in the infinitesimal limit of the
motion counterfactual size, we can write the Spelke affinity between $i$ and
$j$ as $\partial_{y_{i}}\textbf{Flow}(x,y)[j]$, or in tensorial form:

$\textbf{SpelkeAffinity}[x]=\nabla_{y}\textbf{Flow}[x,y].$ (3)

Even when objects can undergo complex motion, such that pairs of points
sometimes move together and sometimes do not, partial or complete Spelke
affinities can be converted to (binarized) estimated segments via a
differential grouping operation, Kaleidoscopic Propagation, introduced in
[42]. Besides providing an efficient method for segmenting Spelke objects,
this derivative construction conceptually unifies segmentation with optical
flow estimation – illustrating that two apparently distinct computer vision
tasks emerge, zero-shot, from the same unsupervised model.

Accounting for Head Motion. A second complication for Spelke object
segmentation is that apparent motion can be due to either a moving object or a
moving camera. Given a single-patch motion counterfactual, the base model
$\Psi$ may infer apparent motion throughout the scene, as expected from a
moving camera, and fail to isolate the target object. We use the head-motion
conditioned predictor $\Psi^{\textbf{H}}$ to solve this simply by also
conditioning on a (counterfactual) input of “zero head motion.” Optical flows
from these multi-modal, joint counterfactual prompts are nearly always
restricted to objects rather than backgrounds.

Averaging Multiple Motion Samples. As with optical flow estimation, some
choices of counterfactual may fail to yield a realistic prediction or a
“correct” answer to the prompt. For example, motion counterfactuals localized
to low-contrast image patches, or those that would result in physically
impossible motions, tend not to produce meaningful estimated segments (for
good reason). To ameliorate this, we assess multiple counterfactual motions of
different magnitudes and directions across a variety of patch locations, then
combine the results to define an aggregate Spelke affinity for any pair of
locations $i,j$ (Fig. 6c): i.e.

$\textbf{SpelkeAffinity}(i,j)\coloneqq\textbf{corr}(\vec{v}_{i},\vec{v}_{j})$

where $\vec{v}_{i}$ (and similarly $\vec{v}_{j}$) is the vector of
counterfactual flows $(|f_{\delta_{0}}[i]|,|f_{\delta_{1}}[i]|,\ldots)$ at
location $i$, taken across multiple counterfactuals
$\delta_{0},\delta_{1},\ldots$.

Resolving Intrinsic Uncertainty in Physical Dynamics. The remediation methods
above produce more accurate Spelke object estimates, but often a single-patch
motion counterfactual is still consistent with many possible scene dynamics;
narrowing these down to isolate a target object is an intrinsic challenge for
segmentation. For example, inanimate objects do not typically move by
themselves, but are instead moved be some effector (e.g. a hand). Similarly,
if two objects are next to each other, moving one will often move the other,
and thus produce motion in both (Fig. 6d). Finally, the imposition of motion
at just the patch $\mathbf{i}$ might be insufficient information for $\Psi$ to
determine which of many possible motions to predict, yielding a counterfactual
frame $\hat{x}_{\delta}$ that looks like a blurry average of different
outcomes. Such images are out of the natural image distribution, and thus hard
for the flow estimator to interpret.

These uncertainty-related limitations – which amount to failures of
translating the definition of Spelke objects into a simple, single-patch
motion counterfactual – will be exacerbated to the extent that $\Psi$ is a
_correct_ model of scene dynamics. But even though a single patch may be
insufficient for segmenting some things, dynamics on short timescales are
still highly constrained; as a result, only a few such patches are typically
required to generate counterfactuals that isolate any object, even in complex
scenes (Fig. 7). Concretely, we can aim to select one or two additional co-
moving patches, together with a similar number of patches deemed to be static
(i.e. where no flow should be observed).

With this goal in mind, we modify our procedure to sample two sets of patch
indices $\textbf{Move}=\\{\mathbf{i}_{0},\mathbf{i}_{1},\ldots\\}$ and
$\textbf{Stop}=\\{\mathbf{j}_{0},\mathbf{j}_{1},\ldots,\\}$ such that for each
pair of patches $\mathbf{i}_{1},\mathbf{i}_{2}\in\textbf{Move}$ or
$\mathbf{i}_{1},\mathbf{i}_{2}\in\textbf{Stop}$, the counterfactual motions
generated for $\mathbf{i}_{1}$ and $\mathbf{i}_{2}$ have high mutual overlap;
but for $\mathbf{i}\in\textbf{Move}$ and $\mathbf{j}\in\textbf{Stop}$, the
mutual overlap between counterfactual motions is low. The procedure for
determining the points with high affinity to $\mathbf{i}_{0}$ is then the same
as before, but with the counterfactual generated by moving all the patches in
Move and fixing all those in Stop. Results of such a procedure can be see in
Fig. 7.

###  3.3 Relative Depth

Algorithm 3 Relative depth via head-motion counterfactuals.

  1. 1.

Let $x$ be an image and let $\rho$ represent a head motion within the camera
plane.

  2. 2.

Compute $x_{\rho}$ as the prediction of the $\Psi^{\textbf{H}}$ prompted only
with $x$ and $\rho$, but no other information about the second video frame.
That is,

$x_{\rho}=\Psi^{\textbf{H}}(x\oplus\varnothing,\rho)$

where $\varnothing$ represents the empty set of patches.

  3. 3.

Set the estimated relative depth,

$\textbf{RelativeDepth}=\frac{1}{\textbf{Flow}(x,x_{\rho})}.$

Figure 8:  Estimating relative depth. A) A process by which relative depth can
be estimated from the head motion-conditioned predictor $\Psi^{\textbf{H}}$.
Given an input image, a small within-plane camera-motion conditioning is
applied, resulting in a counterfactual prediction. Relative depth is the
reciprocal of the corresponding counterfactual flow; the latter is effectively
a simulated disparity map for a small head motion. B) Some examples.

As a final example of Counterfactual World Modeling, we illustrate how to
estimate a relative depth map of an image by prompting a head-motion
conditioned prediction model, $\Psi^{\textbf{H}}$.

In infants and animals, motion parallax is a powerful cue for inferring the
depth (distance) of elements of a scene [43]: when an observer translates
laterally by a given amount, nearer things move across a greater portion of
the visual field than farther things. For small head motions and things in the
center of the visual field, the relationship between apparent motion and depth
is approximately inverse linear.777As can be seen by Taylor-expanding the
right side of the relationship $\text{depth}=\frac{\rho}{\tan{\alpha}}$, where
$\rho$ is the size of the small lateral head displacement and $\alpha$ the
angular apparent motion of a visual element.

This implies that relative depth could be estimated from a single, static
image by simulating the effect of a small, lateral head translation on
apparent motion in the visual field – that is, optical flow. A simple
counterfactual prompting algorithm is shown in Algorithm box 3.

Examples of depth estimates produced by this procedure are shown in Fig 8.
Note that, like optical flow and Spelke object segmentation, this algorithm
could be implemented using derivatives rather than finite perturbations: in
this case, relative depth would be the change in optical flow due to an
infinitessimal in-camera-plane head motion. In practice we find that optical
flow estimates with the RAFT architecture are better when counterfactually
induced motion is $>1$ pixel.

This exercise shows how a single masked predictor carries implicit knowledge
about tracked motion, segmentation, and 3D scene layout. It is reasonable to
wonder, however, whether counterfactually estimating depth in this way has any
real-world value for computer vision or biological organisms. After all, why
simulate head motion when one can actually move one’s head (or have a second
eye at the location one would move it to?) Though speculative, our view is
that running thought experiments about the contents of perception offers a way
to reduce the effects of real-world observation noise and lean more strongly
on an internal model of the world. Elements of real scenes are not perfectly
still while the head is moved, and one’s eyes do not have the right separation
to estimate depth at every distance range. Thus an optimal system will need to
find the right trade-off, for any given scene, between true action-dependent
computations and estimates from its world model.

##  4 Discussion

In this work we have introduced the Counterfactual World Modeling framework, a
way to produce a candidate pure-vision foundational model. The CWM idea
combines the observations that (i) specific statistical properties in a masked
prediction problem force a model to learn a factoring of dynamical information
from object-centric appearance information, and (ii) a wide variety of visual
computations can be executed by counterfactual prediction with these models;
the latter can be implemented as taking (chained) derivatives of the model
itself. We have shown that high-quality readouts of key visual concepts can be
constructed, without any labeled data, via this form of zero-shot prompting.

The counterfactual physical reasoning abilities of CWM are a natural fit for
applications that involve model-predictive control and planning, such as
robotics [44]. Though we have only shown the construction of “2.5D” visual
properties in this work (keypoints, flow, occlusion, segmentation, and depth),
it is natural to extend the approach here toward lifting multiview information
to 3D structure, as evidenced by a variety of recent works [45, 46]. We have
postponed the question of how higher-level semantic structure (e.g. object
categories) relate to our framework, an especially interesting topic given the
recent success of contrastive unsupervised methods like DinoV2 in those
domains [47]. Thought experiments with single-frame masked prediction hint at
a CWM-based approach to categorization: for instance, an MAE that knows it
should complete an elephant’s body with a trunk, but not a rhino’s, must have
an implicit capacity to distinguish these two leathery, gray mammals.
Exploring these intuitions will be an important line of future work.

Figure 9:  Counterfactual World Models as Computational Graphs. The CWM
predictor and its various readouts form an extensible computational graph, in
which each component can be bootstrapped to construct further downstream task
readouts. We hypothesize that the components illustrated in this paper (in
color) will enable the bootstrapping of a variety of additional visual scene
understanding tasks (in gray), including longer-range prediction.

The model $\Psi$, like any masked predictor, is the approximation of a central
tendency of a conditional generative model for the underlying data on which it
is trained. It would be useful to leverage this observation to explicitly
manage uncertainty in an improved version of CWM, in which not only is the
central tendency of the masked tokens predicted, but also an estimate of their
distribution’s higher-order moments. Doing so would likely resolve prediction
failures that arise from mode collapse under the L2 pixel loss currently used
to train $\Psi$ (see 3.2).

A natural extension of CWM is to modalities other than vision, such as
auditory or somatosensory data. While it is likely that the specific masking
policies and counterfactuals that will be of use for extracting structure from
(e.g.) auditory data are different from those useful in vision, the overall
scheme of pairing a masked prediction model with counterfactual-based readouts
may generalize. The same may be true for multi-modal predictive models
combining vision and language token streams.

Another important next step in the development of the CWM framework will be to
build longer-range prediction models. $\Psi$ operates effectively only at
short timescales (< 1s). As we have shown, this turns out to be enough to
extract a variety of important visual representations. It is possible that
creating higher-level ”object tokens” from the movable-object segments
described in section §3.2 would enable better ability to model complex
physical interactions across multiple collisions [7]. A natural hypothesis is
that these intermediates could in turn be “integrated” as inputs to an
improved model, functioning as useful inductive biases for bootstrapping to
longer-range prediction (Fig. 9).

It would be especially useful if the process of discovering masking policies
and useful derivative counterfactuals – that is, building up the computational
graph of CWM – could be automated as a kind of evolutionary hyperparameter
search [48, 49]. It seems straightforward to parameterize the space of masking
policies in terms of which fractions of which data slices are masked. The
space of counterfactual token choices (or, equivalently, which chains of
derivatives of the predictor are taken) may be similarly parameterized. We
hypothesize that this hyperparameter space spans a wide spectrum of distinct
but useful computational architectures, enabling efficient discovery of
systems with the capability and universality required of true perceptual
foundations.

Acknowledgments. This work was supported by the following awards: To D.L.K.Y.:
Simons Foundation grant 543061, National Science Foundation CAREER grant
1844724, Office of Naval Research grant S5122, ONR MURI 00010802 and ONR MURI
S5847. D.M.B. is a Biogen Fellow of the Life Sciences Research Foundation and
a Wu Tsai Neurosciences Institute Interdisciplinary Scholar. We also thank the
Google TPU Research Cloud team for computing support. _S.D.G._

## References

  * Zhou et al. [2000] Hong Zhou, Howard S Friedman, and Rüdiger Von Der Heydt.  Coding of border ownership in monkey visual cortex.  _Journal of Neuroscience_ , 20(17):6594–6611, 2000.
  * Beauchemin and Barron [1995] Steven S. Beauchemin and John L. Barron.  The computation of optical flow.  _ACM computing surveys (CSUR)_ , 27(3):433–466, 1995.
  * Lamme [1995] V A_f Lamme.  The neurophysiology of figure-ground segregation in primary visual cortex.  _Journal of neuroscience_ , 15(2):1605–1615, 1995\.
  * Treisman [1982] Anne Treisman.  Perceptual grouping and attention in visual search for features and for objects.  _Journal of experimental psychology: human perception and performance_ , 8(2):194, 1982.
  * Marr [2010] David Marr.  _Vision: A computational investigation into the human representation and processing of visual information_.  MIT press, 2010.
  * Todd [2004] James T Todd.  The visual perception of 3d shape.  _Trends in cognitive sciences_ , 8(3):115–121, 2004.
  * Bear et al. [2021] Daniel M Bear, Elias Wang, Damian Mrowca, Felix J Binder, Hsiao-Yu Fish Tung, RT Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, et al.  Physion: Evaluating physical prediction from vision in humans and machines.  _arXiv preprint arXiv:2106.08261_ , 2021.
  * Rajalingham et al. [2018] Rishi Rajalingham, Elias B Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J DiCarlo.  Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks.  _Journal of Neuroscience_ , 38(33):7255–7269, 2018.
  * Yamins et al. [2014] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo.  Performance-optimized hierarchical models predict neural responses in higher visual cortex.  _Proceedings of the national academy of sciences_ , 111(23):8619–8624, 2014.
  * Zhuang et al. [2021] Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C Frank, James J DiCarlo, and Daniel LK Yamins.  Unsupervised neural network models of the ventral visual stream.  _Proceedings of the National Academy of Sciences_ , 118(3):e2014196118, 2021.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Voleti et al. [2022] Vikram Voleti, Alexia Jolicoeur-Martineau, and Christopher Pal.  Masked conditional video diffusion for prediction, generation, and interpolation.  _arXiv preprint arXiv:2205.09853_ , 2022.
  * Yan et al. [2022] Wilson Yan, Danijar Hafner, Stephen James, and Pieter Abbeel.  Temporally consistent video transformer for long-term video prediction.  _arXiv preprint arXiv:2210.02396_ , 2022.
  * Nair et al. [2022] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta.  R3m: A universal visual representation for robot manipulation.  _arXiv preprint arXiv:2203.12601_ , 2022.
  * Jaegle et al. [2021] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al.  Perceiver io: A general architecture for structured inputs & outputs.  _arXiv preprint arXiv:2107.14795_ , 2021.
  * Lu et al. [2022] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi.  Unified-io: A unified model for vision, language, and multi-modal tasks.  _arXiv preprint arXiv:2206.08916_ , 2022.
  * Chen et al. [2022a] Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J Fleet, and Geoffrey E Hinton.  A unified sequence interface for vision tasks.  _Advances in Neural Information Processing Systems_ , 35:31333–31346, 2022a.
  * Wu et al. [2023a] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan.  Visual chatgpt: Talking, drawing and editing with visual foundation models.  _arXiv preprint arXiv:2303.04671_ , 2023a.
  * Li et al. [2023] Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hongsheng Li, Chun Yuan, Xiaohua Wang, Yu Qiao, Xiaogang Wang, Wenhai Wang, et al.  Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2691–2700, 2023.
  * Wang et al. [2023] Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang.  Images speak in images: A generalist painter for in-context visual learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6830–6839, 2023.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009, 2022.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020.
  * Grill et al. [2020] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 33:21271–21284, 2020.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Tong et al. [2022] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  _arXiv preprint arXiv:2203.12602_ , 2022.
  * Feichtenhofer et al. [2022] Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He.  Masked autoencoders as spatiotemporal learners, 2022.
  * Kay et al. [2017] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  _arXiv preprint arXiv:1705.06950_ , 2017.
  * Grauman et al. [2022] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18995–19012, 2022.
  * Day and Fitzpatrick [2005] Brian L Day and Richard C Fitzpatrick.  The vestibular system.  _Current biology_ , 15(15):R583–R586, 2005.
  * Crapse and Sommer [2008] Trinity B Crapse and Marc A Sommer.  Corollary discharge across the animal kingdom.  _Nature Reviews Neuroscience_ , 9(8):587–600, 2008.
  * Minderer et al. [2019] Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P Murphy, and Honglak Lee.  Unsupervised learning of object structure and dynamics from videos.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Mian et al. [2008] Ajmal S Mian, Mohammed Bennamoun, and Robyn Owens.  Keypoint detection and local feature matching for textured 3d face recognition.  _International Journal of Computer Vision_ , 79:1–12, 2008\.
  * Lee [1980] Denis N Lee.  The optic flow field: The foundation of vision.  _Philosophical Transactions of the Royal Society of London. B, Biological Sciences_ , 290(1038):169–179, 1980.
  * Fleet and Weiss [2006] David Fleet and Yair Weiss.  Optical flow estimation.  _Handbook of mathematical models in computer vision_ , pages 237–257, 2006.
  * Teed and Deng [2020] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_ , pages 402–419. Springer, 2020.
  * Stone et al. [2021] Austin Stone, Daniel Maurer, Alper Ayvaci, Anelia Angelova, and Rico Jonschkowski.  Smurf: Self-teaching multi-frame unsupervised raft with full-image warping.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3887–3896, 2021.
  * Pont-Tuset et al. [2017] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  _arXiv preprint arXiv:1704.00675_ , 2017.
  * Todorovic [2008] Dejan Todorovic.  Gestalt principles.  _Scholarpedia_ , 3(12):5345, 2008.
  * Spelke [1990] Elizabeth S Spelke.  Principles of object perception.  _Cognitive science_ , 14(1):29–56, 1990.
  * Carey and Xu [2001] Susan Carey and Fei Xu.  Infants’ knowledge of objects: Beyond object files and object tracking.  _Cognition_ , 80(1-2):179–213, 2001.
  * Chen et al. [2022b] Honglin Chen, Rahul Venkatesh, Yoni Friedman, Jiajun Wu, Joshua B Tenenbaum, Daniel LK Yamins, and Daniel M Bear.  Unsupervised segmentation in real-world images via spelke object inference.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIX_ , pages 719–735. Springer, 2022b.
  * Rogers and Graham [1979] Brian Rogers and Maureen Graham.  Motion parallax as an independent cue for depth perception.  _Perception_ , 8(2):125–134, 1979.
  * Ebert et al. [2018] Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine.  Visual foresight: Model-based deep reinforcement learning for vision-based robotic control.  _arXiv preprint arXiv:1812.00568_ , 2018.
  * Yu et al. [2021] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4578–4587, 2021.
  * Wu et al. [2023b] Chao-Yuan Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, and Georgia Gkioxari.  Multiview compressive coding for 3d reconstruction.  _arXiv preprint arXiv:2301.08247_ , 2023b.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Bergstra et al. [2013] James Bergstra, Daniel Yamins, and David Cox.  Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures.  In _International conference on machine learning_ , pages 115–123. PMLR, 2013.
  * Elsken et al. [2019] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Neural architecture search: A survey.  _The Journal of Machine Learning Research_ , 20(1):1997–2017, 2019.

*format=largeformat

##  A Technical Appendix

###  A.1 Formally Defining Masked Predictors

We consider length-$\Delta$ video snippets, described as sequences of $H\times
W$-pixel rectangular RGB frames
$\mathbf{x}=(x_{0},x_{1},\ldots,x_{\Delta-1})$, and the tokens are rectangular
$k\times l$-pixel patches tiling each frame. Let
$\mathcal{I}_{\Delta}^{H,W,k,l}$ be the set of all indices of $k\times l$
patches in $H\times W\times\Delta$ movies (we will drop the superscript for
ease of notation). For each index $i\in\mathcal{I}_{\Delta}$, let
$\mathbf{x}[i]$ be the patch of $\mathbf{x}$ at index $i$, (e.g. the token
value itself). For a subset of patches $p\subset\mathcal{I}$, let
$\mathbf{x}[p]=\\{(i,\mathbf{x}[i])|i\in p\\}$ – the positionally-indexed set
of token values in $p$ – and let $\mathcal{P}_{\Delta}^{H,W,k,l}$ be set of
all possible such token-value sets (again, we will typically drop the sub- and
superscripts for readability).

A _mask generator_ is a function $g$ that maps each $\mathbf{x}$ to a subset
of $\mathcal{I}$, with the interpretation that a patch index $i$ is visible
iff $i\in g(\mathbf{x})$. For any mask generator $g$, let $\neg g(x)$ denote
the complement of $g(x)$ in $\mathcal{P}(\mathbf{x})$, i.e. the set of masked
tokens. For ease of notation, let $g[\mathbf{x}]=\mathbf{x}[g(\mathbf{x})]$
and $\neg g[\mathbf{x}]=\mathbf{x}[\neg g(\mathbf{x})]$, i.e. the sets of
visible and masked tokens in $\mathbf{x}$ respectively.

The $g$-SMAE is then the predictor that tries to recover the masked tokens
from the unmasked tokens. Formally, given a set of functions
$F_{\theta}:\mathcal{P}\longrightarrow\mathcal{P}$ with parameters $\theta$ in
some parameter space $\Theta$, and a training dataset $\mathcal{X}$ of videos,
let

$\theta^{*}_{g,\mathcal{X},\Theta}=\operatorname*{arg\,min}_{\theta\in\Theta}\
\mathbb{E}_{\mathcal{X}}[\mathcal{L}(F_{\Theta}(g[x]),\neg g[x])],$ (S1)

where $\mathcal{L}(\cdot,\cdot)$ is a suitable loss function. The
$g$-_structured mask autoencoder on_ $\mathcal{X}$ is then the function
$F_{g,\mathcal{X},\Theta}\coloneqq F_{\theta^{*}_{g,\mathcal{X},\Theta}}$. (We
will drop the dependence on $\mathcal{X}$ and $\Theta$ because these are
usually clear from context.) SMAEs are typically implemented by deep neural
networks such as Vision Transformers (ViT) or Convolutional Neural Networks
(CNNs) , and parameter optimization indicated in eq. S1 is performed using
standard gradient descent methods. Training datasets $\mathcal{X}$ are
typically large natural image or video databases such as, e.g. ImageNet or
Kinetics , and the loss function $\mathcal{L}$ is typically taken to be L1 or
L2 pixel distance.

###  A.2 Masked AutoEncoders and Video MAEs

The Masked AutoEncoder (MAE) [22] is a masked predictor in which single RGB
images are reconstructed from only a (randomly-chosen) fraction of each image
given as input (Fig. 2a). In this case, the temporal horizon $\Delta=1$, and
for $p\in[0,1]$, the mask generator is the random-valued function
$g_{p}(\mathbf{x})$ that uniformly randomly selects subsets of
$\mathcal{P}(\mathbf{x})$ of size $(1-p)\cdot k_{H}\cdot k_{W}$.

In the original MAE research, three basic facts were noticed. First,
unsurprisingly, performance on the reconstruction task decreases monotonically
with $p$. For large values of $p$ (e.g. $p>0.9$), performance is low – since,
presumably, having such a small portion of the input image simply doesn’t
provide the predictor with enough information to produce an effective guess of
the output image. However, for any value of $p<\sim 0.8$, the reconstruction
problem is well-solved by a sufficiently large network. In other words, it is
possible to learn a neural network that would complete real-world from a small
number of patches. Second, the MAE turns out to be a good inpainter, with the
ability to create effective background replacements the patches of an object
are masked. Finally – and perhaps most significantly – as long as the value of
$p$ is not too low, the features of the learned network support good transfer
learning to tasks such as object categorization. Essentially, as long as the
MAE’s prediction task is sufficiently non-trivial, the requirement to fill in
images from a subset of the patches produces a good general-purpose image
representation. Optimal results were obtained with $p\sim 0.75$. Many
applications of the basic MAE have been made, across a wide variety of
domains.

Soon after the original MAE work, the idea was applied to video data in a
direct fashion [26]. Various masking strategies are possible for videos, but
the Video MAE (VMAE) work used a basic “random tube” strategy — in which a
random set of patches is chosen to be unmasked in the first frame, and the
patches with the same spatial position (comprising “temporal tubes”) are
unmasked in all subsequent frames. The results of the VMAE were reminiscent of
the original MAE, in that the directly-optimized reconstruction problem
appeared to be reasonably well solved by large networks, and transfer to video
tasks (e.g. action recognition) was possible.

###  A.3 Temporally-Factored Masked Predictors

The TMP is a predictor defined on length-2 video snippets
$\mathbf{x}=(x_{t},x_{t+\Delta})$, separated by a small temporal gap (e.g.
$\Delta\sim$150ms). The key ingredient for constructing the TMP is the mask
generator in which _all_ the patch tokens in the first frame, and only a very
small number of patch tokens in the second frame, are given (Fig: 2b). This is
formalized by the definition

$g_{1p}(x_{t},x_{t+\Delta})=\mathcal{I}_{1}\oplus g_{p}(x_{t+\Delta})$ (S2)

where $a\oplus b$ denotes the time-sequential concatenation of single-frame
token index sets $a,b$, and $\mathcal{I}_{1}$ is short for
$\mathcal{I}_{1}^{H,W,k,l}$. Let $\textbf{TMP}_{p}$ denote the predictor
trained with this mask generator. Unlike the typical MAE, where $p$ is chosen
to be about 0.75, for the TMP, we choose $p$ to be much higher, e.g. $p>0.99$.

###  A.4 Keypoints

Let $F$ be a trained masked predictor. Suppose we are given an input
$\mathbf{x}$, initial patch index set $p_{0}$, and an integer $k$. The _$k$
-subsequent keypoint set for $\mathbf{x}$ from $p_{0}$_ is

$K_{F}(\mathbf{x},p_{0},k)=\operatorname*{arg\,min}_{p\ \subset\
\mathcal{I}-p_{0}\text{ such that }|p|=k}\mathcal{L}(F(\mathbf{x}_{p_{0}\cup
p}),\mathbf{x}).$

Ideally it would be possible to efficiently compute
$K_{F}(\mathbf{x},p_{0},k)$ directly for large $k$. However, this is in
general an intractable optimization problem. Instead, it is substantially more
efficient (though still rather inefficient in absolute terms) to take a greedy
approach by finding one keypoint at a time and iterating, e.g. defining

$k_{i}=k_{i-1}\cup K_{F}(\mathbf{x},p_{0}\cup k_{i-1},1),\quad\quad\text{where
}k_{0}=\varnothing.$

In the case of the TMP with two-frame input $(x_{t},x_{t+\Delta})$, a natural
starting point is to start with all patches in $x_{t}$ and no patches in
$x_{t+\Delta}$, e.g $p_{0}=\mathcal{I}_{1}\oplus\varnothing$. This
construction defines dynamical RGB keypoints. Figure 4 shows several examples
of such keypoints for a variety of inputs, using the $\textbf{TMP}_{0.01}$
model discussed above.

Keypoint consolidation. The keypoint construction is associated with several
useful derived prediction tasks. One of these is the $\mathbf{x}$-conditioned
sequence-to-sequence prediction task defined by

$\textbf{Key}_{F}:(\mathbf{x},p)\mapsto K_{F}(\mathbf{x},p,k).$

A predictor trained to solve this tasks can be thought of as _consolidating_
the keypoint construction into an efficient forward-inference procedure. Also
useful are the error and error difference prediction tasks defined
respectively by:

$\textbf{Err}_{F}:(\mathbf{x},p)\mapsto\mathcal{L}(F(\mathbf{x}_{p}),\mathbf{x})\quad\text{and}\quad\textbf{ErrDiff}_{F}:(\mathbf{x},p,q)\mapsto\mathcal{L}(F(\mathbf{x}_{p}),\mathbf{x})-\mathcal{L}(F(\mathbf{x}_{q}),\mathbf{x}).$

###  A.5 Head-Motion Conditioning

In the Ego4D dataset , approximately 1/10th of the video data has usable IMU
(inertial measurement unit) data obtained from the GoPro cameras used in data
collection. To expand the use of this data, we first trained an IMU predictor
from frame pairs
$\textbf{H}(x_{t},x_{t+\Delta})\mapsto\text{IMU}(t-\Delta^{\prime},t+\Delta)$
where $D^{\prime}\sim 1850ms$, just on the subset of Ego4D data where reliable
IMU is present. We then trained an IMU-conditioned TMP:

$\textbf{TMP}^{\textbf{H}}_{q}:(x_{t}\oplus
g_{p}[x_{t+\Delta}],h_{t})\mapsto\neg g_{p}(x_{t+\Delta})$

where $h_{t}=\textbf{H}(x_{t-D^{\prime}},x_{t+\Delta})$ is the estimated IMU
value, on data from a union of Kinetics600 and Ego4D.

###  A.6 Masking Variants

Figure S1:  Masking Policy Variants. A) The Forward-Backward TMP combines the
vanilla TMP with an equivalent backward prediction (with 1% of patches in
frame0 and all patches in frame1). The TMP-FB predictor is better than time-
reversing the vanilla TMP at backward prediction for dynamics that are not
time-reversal symmetric. B) A similar approach allows the definition of
interpolation and extrapolation predictors.

The basic TMP idea can be generalized and extended in a variety of ways.

#### Fractional Forward Masking

For example, the mask generator

$g_{qp}(x_{t},x_{t+\Delta})=g_{q}(x_{t})\oplus g_{p}(x_{t+\Delta})$

will train a predictor that has roughly the same second-frame reconstruction
performance with $q=0.25$ as with $q=1$. This achieves a substantial
efficiency in memory usage during training and speed at inference time.
Moreover, it endows the fully trained model with both dynamics-factoring
properties of the TMP and the static object inpainting abilities of the
regular MAE.

#### Forward-Backward Masking

Another useful variant of the masking policy is to encourage temporal
factoring both forwards and backwards. The mask generator

$g^{\textbf{FB}}_{1p}(\mathbf{x})=\textbf{Bernoulli}_{0.5}\left(\\{g_{1p}(\mathbf{x}),g_{p1}(\mathbf{x})\\}\right)$

randomly selects between training forward prediction and backward predction.
This mask generator trains a predictor $\textbf{TMP}^{\textbf{FB}}_{q}$ that
can predict forwards from $x_{t}$ to $x_{t+\Delta}$ or backwards from
$x_{t+\Delta}$ to $x_{t}$ (Fig S1a). Backward predictions from this model are
better than “time-reversed” predictions of the original forward TMP,
presumably because some aspects of natural motion are not statistically time-
reversal symmetric (e.g. horses don’t walk backwards by reversing their
forward gait).

#### Extrapolation and Interpolation

More complex schemes of masking are possible. For instance one can train
predictors operating on 3 more frames, either using multiple context frames
for forward extrapolation:

$g^{\textbf{ext}}_{p}(x_{t},x_{t+\Delta},x_{t+2\Delta})=\mathcal{I}_{1}\oplus\mathcal{I}_{1}\oplus
g_{p}(x_{t+2\Delta})$

or performing interpolation:

$g^{\textbf{int}}_{p}(\mathbf{x})=\mathcal{I}_{1}\oplus
g_{p}(x_{t+\Delta})\oplus\mathcal{I}_{1},$

and a TMP with both interpolation and extrapolation properties is enabled
simply by mixing the mask policies:

$g^{\textbf{int-
ext}}_{p}(\mathbf{x})=\textbf{Bernoulli}_{0.5}\left(\\{g^{\textbf{int}}_{p}(\mathbf{x}),g^{\textbf{ext}}_{p}(\mathbf{x})\\}\right).$

More complex masking policies with varying combinations different fractional
masking statistics, temporal patterns, and unions thereof, will generate
predictors with more sophisticated scene and object understanding abilities.