#  SMURF: Self-Teaching Multi-Frame
Unsupervised RAFT with Full-Image Warping

Austin Stone∗1 Daniel Maurer∗2 Alper Ayvaci2 Anelia Angelova1 Rico
Jonschkowski1
1Robotics at Google 2Waymo
{austinstone, anelia<EMAIL_ADDRESS>{maurerd<EMAIL_ADDRESS>
https://github.com/google-research/google-research/tree/master/smurf

###### Abstract

We present SMURF, a method for unsupervised learning of optical flow that
improves state of the art on all benchmarks by $36\%$ to $40\%$ (over the
prior best method UFlow) and even outperforms several supervised approaches
such as PWC-Net and FlowNet2. Our method integrates architecture improvements
from supervised optical flow, i.e. the RAFT model, with new ideas for
unsupervised learning that include a sequence-aware self-supervision loss, a
technique for handling out-of-frame motion, and an approach for learning
effectively from multi-frame video data while still only requiring two frames
for inference.

††∗Authors contributed equally.

##  1 Introduction

Optical flow describes a dense pixel-wise correspondence between two images,
specifying for each pixel in the first image, where that pixel is in the
second image. The resulting vector field of relative pixel locations
represents apparent motion or “flow” between the two images. Estimating this
flow field is a fundamental problem in computer vision and any advances in
flow estimation benefit many downstream tasks such as visual odometry, multi-
view depth estimation, and video object tracking.

Classical methods formulate optical flow estimation as an optimization problem
[2, 8]. They infer, for a given image pair, a flow field that maximizes
smoothness and the similarity of matched pixels. Recent supervised learning
approaches instead train deep neural networks to estimate optical flow from
examples of ground-truth annotated image pairs [9, 28, 29, 37]. Since
obtaining ground truth flow is extremely difficult for real images, supervised
learning is generally limited to synthetic data [5, 20]. While these methods
have produced excellent results in the training domain, generalization is
challenging when the gap between the target domain ant the synthetic training
data is too wide.

Unsupervised learning is a promising direction to address this issue as it
allows training optical flow models from unlabeled videos of any domain. The
unsupervised approach works by combining ideas from classical methods and
supervised-learning – training the same neural networks as in supervised
approaches but optimizing them with objectives such as smoothness and
photometric similarity from classical methods. Unlike those classical methods,
unsupervised approaches perform optimization not per image pair but jointly
for the entire training set.

Since unsupervised optical flow takes inspiration from classical and
supervised learning methods, we can make substantial progress by properly
combining novel ideas with insights from these two directions. In this paper,
we do exactly that and make the following three contributions:

  1. 1.

We integrate the current best supervised model, RAFT [29] with unsupervised
learning and perform key changes to the loss functions and data augmentation
to properly regularize this model for unsupervised learning.

  2. 2.

We perform unsupervised learning on image crops while using the full image to
compute unsupervised losses. This technique, which we refer to as full-image
warping, improves flow quality near image boundaries.

  3. 3.

We leverage a classical method for multi-frame flow refinement [19] to
generate better labels for self-supervision from multi-frame input. This
technique improves performance especially in occluded regions without
requiring more than two frames for inference.

Our method _S elf-Teaching Multi-frame Unsupervised RAFT with Full-Image
Warping_ (SMURF) combines these three contributions and improves the state of
the art (SOTA) in unsupervised optical flow in all major benchmarks, i.e. it
reduces errors by 40 / 36 / 39 % in the Sintel Clean / Sintel Final / KITTI
2015 benchmarks relative to the prior SOTA set by UFlow [12]. These
improvements also reduce the gap to supervised approaches, as SMURF is the
first unsupervised optical flow method that outperforms supervised FlowNet2
[9] and PWC-Net [28] on all benchmarks.

##  2 Related Work

Optical flow was first studied in psychology to describe motion in the visual
field [6]. Later, the rise of computing in the 1980s led to analytical
techniques to estimate optical flow from images [8, 18]. These techniques
introduced photometric consistency and smoothness assumptions. These early
methods do not perform any learning but instead solve a system of equations to
find flow vectors that minimize the objective function for a given image pair.
Follow-up work continued to improve flow accuracy, e.g. through better
optimization techniques and occlusion reasoning [2, 27].

Machine learning has helped to improve results substantially. First approaches
used supervised convolutional neural networks that had relatively little flow-
specific structure [5, 9]. Others introduced additional inductive biases from
classical approaches such as coarse-to-fine search [23, 28, 37]. The current
best network architecture is the Recurrent All-Pairs Field Transforms (RAFT)
model. It follows classical work that breaks with the coarse-to-fine
assumption [4, 26, 36] and computes the cost volume between all pairs of
pixels and uses that information to iteratively refine the flow field [29].
All supervised methods rely heavily on synthetic labeled data for training.
While producing excellent results in the supervised setting, RAFT has not
previously been used in unsupervised learning.

Unsupervised approaches appeared after supervised methods and showed that even
without labels, deep learning can greatly outperform classical flow methods
[10, 11, 12, 14, 16, 17, 21, 24, 25, 33, 35, 38, 39, 42, 43]. Besides being
more accurate than classical methods, learned methods are also faster at
inference because all optimization occurs during training instead of during
inference time.111The reason why learned unsupervised methods outperform
classical methods is not obvious as both approaches optimize similar
objectives. We speculate that the network structure introduces helpful
inductive biases and that optimizing a single network for all training images
provides a better regularization than independently optimizing a flow field
for each image.

A recent study [12] performed an extensive comparison of the many proposed
advances in unsupervised flow estimation and amalgamated these different works
into a state of the art method called UFlow. We take this work as a starting
point and build on the techniques suggested there such as range-map based
occlusion reasoning [35], the Census loss for photometric consistency [21,
40], edge aware smoothness [30], and self supervision [16, 17].

We build on and substantially extend this prior work by enabling the RAFT
model to work in an unsupervised learning setting through changes in the loss
function and data augmentation. We also propose full-image warping to make the
photometric loss useful for pixels that leave the (cropped) image plane. And
we utilize a flow refinement technique [19] to leverage multi-frame input
during training to self-generate improved labels for self-supervision.

##  3 SMURF for Unsupervised Optical Flow

This section describes our method _S elf-Teaching Multi-frame Unsupervised
RAFT with Full-Image Warping_ (SMURF) in two parts. The first part covers
preliminaries, such as the problem definition and components adopted from
prior work. The second part describes the three main improvements in our
method.

###  3.1 Preliminaries on Unsupervised Optical Flow

Given a pair of RGB images, $I_{1},I_{2}\in\mathbb{R}^{H\times W\times 3}$, we
want to estimate the flow field $V_{1}\in\mathbb{R}^{H\times W\times 2}$,
which for each pixel in $I_{1}$ indicates the offset of its corresponding
pixel in $I_{2}$. We address this in a learning-based approach where we want
to learn a function $f_{\theta}$ with parameters $\theta$ that estimates the
flow field for any image pair, such that $V_{1}=f_{\theta}(I_{1},I_{2})$. We
learn the parameters $\theta$ from data of unlabeled image sequences
$D=\\{(I_{i})_{i=1}^{m}\\}$ by minimizing a loss function
$\mathcal{L}_{\mathit{}}$,
$\theta^{*}=\arg\min\mathcal{L}_{\mathit{}}(D,\theta)$.

We build on the loss function $\mathcal{L}_{\mathit{}}$ from UFlow [12],

$\displaystyle\mathcal{L}_{\mathit{}}(D,\theta)$
$\displaystyle=\omega_{\mathit{photo}}\mathcal{L}_{\mathit{photo}}(D,\theta)$
$\displaystyle+\omega_{\mathit{smooth}}\mathcal{L}_{\mathit{smooth}}(D,\theta)+\omega_{\mathit{self}}\mathcal{L}_{\mathit{self}}(D,\theta)\,,$

which is a weighted combination of three terms: occlusion aware photometric
consistency $\mathcal{L}_{\mathit{photo}}$, edge-aware smoothness
$\mathcal{L}_{\mathit{smooth}}$, and self-supervision
$\mathcal{L}_{\mathit{self}}$.

The photometric consistency term is defined as

$\mathcal{L}_{\mathit{photo}}(D,\theta)=\frac{1}{HW}\sum
O_{1}\odot\rho\Big{(}I_{1},w\big{(}I_{2},V_{1}\big{)}\Big{)}\,,$

where $\odot$ represents element-wise multiplication and $\frac{1}{HW}\sum$ is
shorthand notation for the mean over all pixels; we omit indices for better
readability. The function $w(\cdot,\cdot)$ warps an image with a flow field,
here estimated as $V_{1}=f_{\theta}(I_{1},I_{2})$. The function
$\rho(\cdot,\cdot)$ measures the photometric difference between two images
based on a soft Hamming distance on the Census-transformed images and applies
the generalized Charbonnier function [21]. The occlusion mask
$O_{1}\in\mathbb{R}^{H\times W}$ with entries $\in[0,1]$ deactivates
photometric consistency for occluded locations. This is crucial because
occluded pixels, by definition, are not visible in the other image and
therefore need not obey appearance consistency – the correct flow vector for
an occluded pixel may point to a dissimilar-looking pixel in the other image.
Like UFlow, we use a range-map based occlusion estimation [35] with gradient
stopping for all datasets except for KITTI, where forward backward consistency
[2] seems to yield better results [12].

The $k$-th order edge-aware smoothness term is defined as
$\displaystyle\mathcal{L}_{\mathit{smooth(k)}}(D,\theta)$
$\displaystyle=\frac{1}{HW}\sum\Bigg{(}\exp{\left(-\frac{\lambda}{3}\sum_{c}\left\lvert\frac{\partial
I_{1_{c}}}{\partial
x}\right\rvert\right)}\odot\left\lvert\frac{\partial^{k}V_{1}}{\partial
x^{k}}\right\rvert$
$\displaystyle+\exp{\left(-\frac{\lambda}{3}\sum_{c}\left\lvert\frac{\partial
I_{1_{c}}}{\partial
y}\right\rvert\right)}\odot\left\lvert\frac{\partial^{k}V_{1}}{\partial
y^{k}}\right\rvert\Bigg{)}\,.$ The expression $\frac{1}{HW}\sum$ again
computes the mean over all pixels, here for two components that compute the
$k$-th derivative of the flow field $V_{1}$ along the $x$ and $y$ axes. These
derivatives are each weighted by an exponential of the mean image derivative
across color channels $c$. This weighting provides edge-awareness by ignoring
non-smooth transitions in the flow field at visual edges [30]. The sensitivity
to visual edges is controlled by $\lambda$.

The self-supervision term is defined as
$\displaystyle\mathcal{L}_{\mathit{self}}(D,\theta)=\frac{1}{HW}\sum\hat{M}_{1}\odot\Big{(}1-M_{1}\Big{)}\odot
c\Big{(}\hat{V}_{1},V_{1}\Big{)}\,.$ where a self-generated flow label
$\hat{V}_{1}$ [16] is compared to the flow prediction $V_{1}$ using the
generalized Charbonnier function [27]
$c(A,B)=\left((A-B)^{2}+\epsilon^{2}\right)^{\alpha}$ with all operations
being applied elementwise. Our experiments use $\epsilon=0.001$ and
$\alpha=0.5$. Prior work has modulated this comparison by masks $\hat{M}$ and
$M$ for the generated labels and the predicted flow respectively, which are
computed with a forward-backward consistency check [2], so that self-
supervision is only applied to image regions where the label is forward-
backward consistent but the prediction is not. However, as we will describe
below, our method actually improves when we remove this masking.

The main reason to perform self-supervision is to improve flow estimation near
the image boundaries where pixels can easily get occluded by moving outside of
the image frame [16]. To address this problem via self-supervision, we apply
the model on an image pair to produce a flow label and use this self-generated
label to supervise flow predictions from a cropped version of the image pair.
This way the model can transfer its own predictions from an easier to a more
difficult setting. We build on and extend this technique with additional
augmentations as described below.

###  3.2 SMURF’s Improvements

After having covered the foundation that our method builds on, we will now
explain our three major improvements: 1) enabling the RAFT architecture [29]
to work with unsupervised learning, 2) performing full-image warping while
training on image crops, and 3) introducing a new method for multi-frame self-
supervision.

####  3.2.1 Unsupervised RAFT

As model $f_{\theta}$ for optical flow estimation, we use the Recurrent All-
Pairs Field Transforms (RAFT) [29] model, a recurrent architecture that has
achieved top performance when trained with supervision but has not previously
been used with unsupervised learning. RAFT works by first generating
convolutional features for the two input images and then compiling a 4D cost
volume $C\in\mathbb{R}^{H\times W\times H\times W}$ that contains feature-
similarities for all pixel pairs between both images. This cost volume is then
repeatedly queried and fed into a recurrent network that iteratively builds
and refines a flow field prediction. The only architectural modification we
make to RAFT is to replace batch normalization with instance normalization
[31] to enable training with very small batch sizes. Reducing the batch size
was necessary to fit the model and the more involved unsupervised training
steps into memory. But more importantly, we found that leveraging RAFT’s
potential for unsupervised learning _requires key modifications to the
unsupervised learning method_ , which we will discuss next.

##### Sequence Losses

In supervised learning of optical flow, it is common to apply losses not only
to the final output but also to intermediate flow predictions [29, 28, 37]. In
unsupervised learning this is not typically done – presumably because
intermediate outputs of other models are often at lower resolutions which
might not work well with photometric and other unsupervised losses [12, 16].
However, RAFT produces intermediate predictions at full resolution and does
not pass gradients between prediction steps. Thus, we apply the unsupervised
losses at the entire sequence of RAFT’s intermediate predictions, which we
found to be essential to prevent divergence and to achieve good results.
Similarly to supervised methods, we exponentially decay the weight of these
losses at earlier iterations [29].
$\mathcal{L}_{\mathit{sequence}}=\sum_{i=1}^{n}\gamma^{n-i}\mathcal{L}_{\mathit{i}}$,
where $n$ is the number of flow iterations, $\gamma$ is the decay factor, and
$\mathcal{L}_{\mathit{i}}$ is the loss at iteration $i$. Our experiments use
$n=12$ and $\gamma=0.8$.

Figure 1: Self-supervision with sequence loss and augmentation. We use a
single model as both “student” and “teacher”. As the teacher, we apply the
model on full non-augmented images. As student, the model only sees a cropped
and augmented version of the same images. The final output of the teacher is
then cropped and used to supervise the predictions at all iterations of the
student (to which the smoothness and photometric losses are applied as well).
The advantages of this self-supervision method are threefold: (1) the model
learns to ignore photometric augmentations (2) the model learns to make better
predictions at the borders and in occluded areas of the image, and (3) early
iterations of the recurrent model learn from the output at the final
iteration.

##### Improved Self-Supervision

For self-supervision, we implement the sequence loss by applying the model to
a full image, taking its final output, and using that to supervise all
iterations of the model applied to the cropped image (see Figure 1). Applying
self-supervision from full to cropped images has been shown to provide a
learning signal to pixels near the image border that move out of the image and
therefore receive no signal from photometric losses [16]. Performing this
self-supervision in a sequence-aware manner allows the earlier stages of the
model to learn from the model’s more refined final output. Potentially because
of this inherent quality difference between the “student” and the “teacher”
flow, we found that simplifying the self-supervision loss by removing masks
$M$ and $\hat{M}$ improves performance further. The self-supervision loss
without masks is $\mathcal{L}_{\mathit{self}}(D,\theta)=\frac{1}{HW}\sum
c\Big{(}\hat{V}_{1},V_{1}\Big{)}$. Note that this change does not affect the
photometric loss $\mathcal{L}_{\mathit{photo}}$, where we continue to use $O$
to mask out occlusions.

Figure 2: Full-image warping. Images are cropped for flow prediction, but
warping of image 2 with the predicted flow (to compute the photometric loss),
is done with the full size image. The advantage is shown in the lower right:
Compared to warping the cropped image (left), full-image warping reduces
occlusions from out-of-frame motion (shown in black) and is able to better
reconstruct image 1. When used during training, full-image warping provides a
learning signal for pixels that move outside the cropped image boundary. The
remaining occlusions in the reconstruction are due to noisy flow predictions
for the out-of-frame motion.

##### Extensive Data Augmentation

To regularize RAFT, we use the same augmentation as supervised RAFT [29] which
is much stronger than what has typically been used in unsupervised optical
flow, except for the recent ARFlow [14]. We randomly vary hue, brightness,
saturation, stretching, scaling, random cropping, random flipping left/right
and up/down, and we apply a random eraser augmentation that removes random
parts of each image. All augmentations are applied to the model inputs, but
not to the images used to compute the photometric and smoothness losses. The
self-generated labels for self-supervision are computed from un-augmented
images, which has the benefit of training the model to ignore these
augmentations (see Figure 1).

Figure 3: Multi-frame self-supervision. From a sequence of three frames $t-1$,
$t$, and $t+1$, we compute the backward flow ($t\rightarrow t-1$) and the
forward flow ($t\rightarrow t+1$). The backward flow is then inverted via a
tiny model (which is trained for this frame pair) and used to inpaint occluded
regions in the forward flow. This inpainted flow field is then used to retrain
the RAFT model.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
Overlayed Frames | True Flow | Predicted Flow | Flow Error | True Occlusions | Predicted Occlusions

Figure 4:  Qualitative results for SMURF on two random examples from KITTI
2015, Sintel Clean, and Sintel Final (top to bottom) not seen during training.
These results show the model’s ability to estimate fast motions, relatively
fine details, and substantial occlusions.

####  3.2.2 Full-Image Warping

The photometric loss, which is essential for unsupervised optical flow
estimation, is generally limited to flow vectors that stay inside the image
frame because vectors that point outside of the frame have no pixels to
compare their photometric appearance to. We address this limitation by
computing the flow field from a cropped version of the images $I_{1}$ and
$I_{2}$ while referencing the full, uncropped image $I_{2}$ when warping it
with the estimated flow $V_{1}$ before computing the photometric loss (see
Figure 2). As we also no longer mark these flow vectors that move outside the
image frame as occluded, they now provide the model with a learning signal. We
use full-image warping for all datasets except Flying Chairs, where we found
that cropping the already small images hurt performance.

####  3.2.3 Multi-Frame Self-Supervision

Finally, we propose to leverage multi-frame information for self-supervision
to generate better labels in occluded areas, inspired by work that used a
similar technique for inference [19]. For multi-frame self-supervision, we
take a frame $t$ and compute the forward flow to the next frame ($t\rightarrow
t+1$) and the backward flow to the previous frame ($t\rightarrow t-1$). We
then use the backward flow to predict the forward flow through a tiny learned
inversion model and use that prediction to inpaint areas that were occluded in
the original forward flow but were not occluded in the backward flow – which
is why the estimate from the backward flow is more accurate (see Figure 3).
The tiny model for the backward-forward inversion consists of three layers of
3$\times$3 convolutions with [16, 16, 2] channels that are applied on the
backward flow and the image coordinates normalized to $[-1,1]$. The model is
re-initialized and trained per frame using the non-occluded forward flow as
supervision, after which the in-painted flow field is stored and used for
self-supervision. We apply multi-frame self-supervision only at the final
stage of training. Importantly, we use multiple frames only during training
and not for inference.

##  4 Experiments

Our hyperparameters are based on UFlow [12] with slight modifications based on
further hyperparameter search. In all experiments, we use weights
$\omega_{\mathit{photo}}=1$ and $\omega_{\mathit{self}}=0.3$. Regarding
smoothness parameters, we set edge sensitivity $\lambda=150$ and
$\omega_{\mathit{smooth}}=4$ for KITTI and Chairs and
$\omega_{\mathit{smooth}}=2.5$ for Sintel. We use 2nd order smoothness ($k=2$)
for KITTI and 1st order smoothness ($k=1$) for all other datasets. We train
for 75K iterations with batch size 8, except for Sintel where we train for
only 15K iterations to avoid overfitting. The self-supervision weight is set
to 0 for the first 40% of gradient steps, then linearly increased to $0.3$
during the next 10% of steps and then kept constant. All of our training uses
Adam [13] ($\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=10^{-8}$) with
learning rate of $0.0002$, which takes about 1 day to converge on 8 GPUs
running synchronous SGD. On all datasets, the learning rate is exponentially
decayed to $\frac{1}{1000}$ of its original value over the last 20% of steps.
On Sintel and on Flying Chairs we train with random crops of $368\times 496$,
and on KITTI we train with random crops of size $296\times 696$. For all
datasets we evaluated our model on the input resolution that had the lowest
average end point error on the training set; $488\times 1144$ for KITTI and
$480\times 928$ for Sintel. All test images were bilinearly resized to these
resolutions during inference, and the resulting flow field was bilinearly
resized and rescaled back to the native image size to compute evaluation
metrics.

During the second stage of training when multi-frame self-supervision is
applied, we generate labels for all images using the model trained according
to the procedure described above. We then continue training the same model
with only the self-supervision loss for an additional 30K iterations using the
multiframe generated labels. We use the same hyperparameters as in the first
stage but exponentially decay the learning rate for the last 5K iterations.
For our best performing Sintel model, we train with the KITTI self-supervision
labels mixed in at a ratio of 50%.

##### Datasets

|  | Sintel Clean [3] |  | Sintel Final [3] |  | KITTI 2015 [22]
---|---|---|---|---|---|---
|  | EPE |  | EPE |  | EPE | EPE (noc) | ER in %
| Method | train | test |  | train | test |  | train | train | train | test
Supervised in domain | FlowNet2-ft [9] | (1.45) | 4.16 |  | (2.01) | 5.74 |  | (2.30) | – | (8.61) | 11.48
PWC-Net-ft [28] | (1.70) | 3.86 |  | (2.21) | 5.13 |  | (2.16) | – | (9.80) | 9.60
SelFlow-ft [17] ${}^{\text{(MF)}}$ | (1.68) | [3.74] |  | (1.77) | {4.26} |  | (1.18) | – | – | 8.42
VCN-ft [37] | (1.66) | 2.81 |  | (2.24) | 4.40 |  | (1.16) | – | (4.10) | 6.30
RAFT-ft [29] | (0.76) | 1.94 |  | (1.22) | 3.18 |  | (0.63) | – | (1.5) | 5.10
Supervised out of domain | FlowNet2 [9] | 2.02 | 3.96 |  | 3.14 | 6.02 |  | 9.84 | – | 28.20 | –
PWC-Net [28] | 2.55 | – |  | 3.93 | – |  | 10.35 | – | 33.67 | –
VCN [37] | 2.21 | – |  | 3.62 | – |  | 8.36 | – | 25.10 | –
RAFT [29] | 1.43 | – |  | 2.71 | – |  | 5.04 | – | 17.4 | –
Unsupervised | EPIFlow [42] | 3.94 | 7.00 |  | 5.08 | 8.51 |  | 5.56 | 2.56 | – | 16.95
DDFlow [16] | {2.92} | 6.18 |  | {3.98} | 7.40 |  | [5.72] | [2.73] | – | 14.29
SelFlow [17] ${}^{\text{(MF)}}$ | [2.88] | [6.56] |  | {3.87} | {6.57} |  | [4.84] | [2.40] | – | 14.19
UnsupSimFlow [10] | {2.86} | 5.92 |  | {3.57} | 6.92 |  | [5.19] | – | – | [13.38]
ARFlow [14] ${}^{\text{(MF)}}$ | {2.73} | {4.49} |  | {3.69} | {5.67} |  | [2.85] | – | – | [11.79]
UFlow [12] | 3.01 | 5.21 |  | 4.09 | 6.50 |  | 2.84 | 1.96 | 9.39 | 11.13
SMURF-test (ours) | 1.99 | – |  | 2.80 | – |  | 2.01 | 1.42 | 6.72 | –
SMURF-train (ours) | {1.71} | 3.15 |  | {2.58} | 4.18 |  | {2.00} | {1.41} | {6.42} | 6.83

Table 1: Comparison to state of the art. SMURF-train / test is our model
trained on the train / test split of the corresponding dataset. The best
results per category are shown in bold – note that supervision “in domain” is
often not possible in practice as flow labels for real images are difficult to
obtain. Braces indicate results that might have overfit because evaluation
data was used for training: “()” evaluated on the same labeled data as was
used for training, “{}” trained on the unlabeled evaluation set, and “[]”
trained on data highly related to the evaluation set (e.g., the entire Sintel
Movie or $<5$ frames away from an evaluation image in KITTI). Methods that use
multiple frames at inference are denoted with “MF”; our method uses multiple
frames only during training.

We train and evaluate our model according to the conventions in the literature
using the following optical flow datasets: Flying Chairs [5], Sintel [3] and
KITTI 2015 [22]. We pretrain on Flying Chairs before fine tuning on Sintel or
KITTI. Similar to UFlow [12], we did not find a benefit to pretraining on more
out-of-domain data, e.g. Flying Things [20]. None of the training techniques
in our method uses any ground truth labels. We train on the “training” portion
of Flying Chairs, and divide the Sintel dataset according to its standard
train / test split. For KITTI, we train on the multi-view extension following
the split used in prior work [12, 42]: We train two models, one on the multi-
view extension of the training set and one on the extension of the test set,
evaluate these models appropriately. For ablations, we report metrics after
training on the “test” portion of the dataset (which does not include labels)
and evaluating on the training set, and for final benchmark numbers we report
results after training on the training portion only. For our benchmark result
on Sintel, we train on a 50% mixture of the KITTI and Sintel multi-frame self-
supervision labels. For all datasets we report endpoint error (“EPE”), and for
KITTI, we additionally report error rates (“ER”), where a prediction is
considered erroneous if its EPE is $>3$ pixels or $>5\%$ of the length of the
true flow vector. We generally compute these metrics for all pixels, except
for “EPE (noc)” where only non-occluded pixels are considered.

##  5 Results

In this section, we compare SMURF to related methods, ablate the proposed
improvements, and show limitations.

###  5.1 Comparison to State of the Art

Qualitative results for SMURF are shown in Figure 4 and a comparison to other
methods can be found in Table 1. This comparison shows that our model
substantially outperforms all prior published methods on unsupervised optical
flow on all benchmarks. Compared to the the previous state of the art method
UFlow [12], our method reduces benchmark test errors by 40 / 36 / 39 % for
Sintel Clean / Sintel Final / KITTI 2015. When we compare SMURF trained in
domain to supervised methods trained out of domain, SMURF outperforms all
supervised methods except RAFT [29] on all benchmarks222This is a fair
comparison as obtaining labels for a given domain is extremely difficult while
our method trains on readily available video data.. Compared to supervised
RAFT, it performs a bit worse on Sintel (1.99 vs 1.43 on Sintel Clean, 2.80
vs. 2.71 on Sintel Final) but much better on KITTI 2015 (2.01 vs. 5.04 EPE,
6.72% vs. 17.4% ER). SMURF even outperforms some supervised methods when they
are finetuned on the test domain, e.g. FlowNet2-ft [9], PWC-Net-ft [28], and
SelFlow-ft [17]. Only VCN-ft [37] and RAFT-ft [29] achieve better performance
here. The inference time of our model is the same as the supervised RAFT
model, approximately 500ms when using 12 recurrent iterations. A qualitative
comparison to supervised RAFT is shown in Figure 5.

Figure 5: Qualitative comparison of unsupervised SMURF and supervised RAFT.
Maybe unsurprisingly supervised training in domain reduces errors especially
for challenging and ambiguous cases, e.g. reflections and shadows of moving
objects. But interestingly, there are also areas (highlighted) where our
unsupervised method works better than the supervised alternative, apparently
at small objects in places where labels are sparse and potentially imperfect
(left) or non-existent (right).

|  |  | Chairs |  | Sintel train |  | KITTI-15 train
---|---|---|---|---|---|---|---
| Method |  | test |  | Clean | Final |  | EPE | ER%
Train on Chairs | DDFlow [16] |  | 2.97 |  | 4.83 | 4.85 |  | 17.26 | –
UFlow [12] |  | 2.55 |  | 4.36 | 5.12 |  | 15.68 | 32.69
SMURF (ours) |  | 1.72 |  | 2.19 | 3.35 |  | 7.94 | 26.51
Train on Sintel | DDFlow [16] |  | 3.46 |  | {2.92} | {3.98} |  | 12.69 | –
UFlow [12] |  | 3.25 |  | 3.01 | 4.09 |  | 7.67 | 17.41
SMURF (ours) |  | 1.99 |  | 1.99 | 2.80 |  | 4.47 | 12.55
Train on KITTI | DDFlow [16] |  | 6.35 |  | 6.20 | 7.08 |  | [5.72] | –
UFlow [12] |  | 5.05 |  | 6.34 | 7.01 |  | 2.84 | 9.39
SMURF (ours) |  | 3.26 |  | 3.38 | 4.47 |  | 2.01 | 6.72

Table 2: Generalization across datasets. These results compare our method to
state of the art unsupervised methods in a setting where a model is trained on
one dataset and tested on different one.

| Trained on |  | KITTI-15 _train_ |  | KITTI-15 _test_
---|---|---|---|---|---
Method | Flow | Stereo |  | EPE | D1(all)% |  | D1(noc)% | D1(all)%
SGM [7] | – | – |  | – | – |  | 5.62 | 6.38
SsSMNet [41] | – | ✓ |  | – | – |  | {3.06} | {3.40}
UnOS [34] | ✓ | ✓ |  | – | 5.94 |  | – | 6.67
Flow2Stereo [15] | ✓ | ✓ |  | 1.34 | 6.13 |  | 6.29 | 6.61
Reversing-PSMNet [1] | – | ✓ |  | 1.01 | 3.85 |  | 3.86 | 4.06
SMURF-train (ours) | ✓ | – |  | 1.03 | 4.31 |  | 4.51 | 4.77

Table 3: Stereo depth estimation. Without fine-tuning, our flow model
estimates stereo depth “zero-shot” at an accuracy comparable to state of the
art unsupervised methods trained for that task.

In Table 2, we compare generalization across domains to prior unsupervised
methods and find substantial improvements for every combination of training
and test domain. Even after only training on Flying Chairs, our model already
achieves an EPE of 2.19 on Sinel Clean and 3.35 on Sinel Final, which is
superior to all prior unsupervised techniques even if fine tuned on Sintel
data (first and third column in Table 1).

We also tested generalization from optical flow to stereo depth estimation.
Here we evaluated our exact trained KITTI-15 flow model in the KITTI 2015
stereo depth benchmark [22]. Despite neither tailoring the architecture or
losses to stereo depth estimation nor training or fine tuning on any stereo
data, our method achieves results that are competitive with the best
unsupervised stereo depth estimation methods (see Table 3).

|  |  |  | Sintel train |  | KITTI-15 train
---|---|---|---|---|---|---
Method | Model | Training input |  | Clean | Final |  | EPE | ER%
UFlow [12] | PWC | High res. |  | 3.01 | 4.09 |  | 2.84 | 9.39
UFlow | PWC | Low res. |  | 3.43 | 4.33 |  | 3.87 | 13.04
UFlow | RAFT | Low res. |  | 3.36 | 4.32 |  | 4.27+ | 14.18+
SMURF | PWC | Image crop |  | 2.63 | 3.66 |  | 2.73 | 9.33
SMURF | RAFT | Image crop |  | 1.99 | 2.80 |  | 2.01 | 6.72

Table 4: Test of whether replacing PWC-Net with RAFT improves the prior best
unsupervised method UFlow. Due to memory constraints, training RAFT in this
setting requires a lower resolution than UFlow [12] (384$\times$512 for
Sintel, 320$\times$704 for KITTI). We use that resolution for a side-by-side
comparison to PWC (rows 2-3), which shows that RAFT performs poorly without
our proposed modifications. Results marked with + overfit the dataset and
generate worse performance than reported here at the end of training.

|  |  |  |  | Sintel train |  | KITTI-15 train
---|---|---|---|---|---|---|---
SQ | AU | FW | MF |  | Clean | Final |  | EPE | ER%
– | – | – | – |  | 2.92 | 3.92 |  | 8.40 | 18.57
– | ✓ | – | – |  | — diverged — |  | — diverged —
✓ | – | – | – |  | 2.66 | 3.86 |  | 5.01+ | 17.50+
✓ | ✓ | – | – |  | 2.38 | 3.17 |  | 3.64+ | 13.05+
– | ✓ | ✓ | – |  | — diverged — |  | — diverged —
✓ | – | ✓ | – |  | 2.71 | 3.46 |  | 2.78 | 8.47
✓ | ✓ | ✓ | – |  | 2.15 | 2.99 |  | 2.45 | 7.53
✓ | ✓ | ✓ | ✓ |  | 1.99 | 2.80 |  | 2.01 | 6.72

Table 5: Ablation of proposed improvements. SQ: sequence loss, AU: heavy
augmentation, FW: full-image warping, MF: multi-frame self-supervision. All
components significantly improve performance and the sequence loss prevents
divergence.

###  5.2 Ablation Study

To determine which aspects of our model are responsible for its improved
performance over prior work, we perform an extensive ablation study. In these
ablations, we always train one model per domain (on KITTI-2015-test and
Sintel-test after pretraining on Flying Chairs), and evaluate those on the
corresponding validation split of the same domain.

##### RAFT Model

Our first ablation investigates how much improvement can be obtained by taking
the prior state of the art method UFlow and replacing its PWC model with RAFT.
As the results in Table 4 show, replacing the model without additional changes
to the unsupervised learning method surprisingly does not improve but instead
decreases performance. Through extensive experimentation, we identified and
added the techniques presented here that enable superior unsupervised learning
with RAFT. The gains from these techniques are much smaller with the PWC
model, potentially because of the more constrained architecture.

|  | Sintel train |  | KITTI-15 train
---|---|---|---|---
Self-sup. variant |  | Clean | Final |  | EPE | ER%
No self-supervision |  | 3.88 | 4.50 |  | 3.22 | 8.40
From intermediate predictions |  | 2.51 | 3.22 |  | 2.58 | 7.54
W/ FB masking |  | 2.53 | 3.31 |  | 2.94 | 8.14
W/o FB masking (Ours) |  | 2.15 | 2.99 |  | 2.45 | 7.53

Table 6: Ablation of self-supervision improvements. All results are without
multi-frame self-supervision but including all other components. FB masking
refers to forward-backward consistency masking that prior work used in the
self-supervision loss. Occlusions in the photometric loss are always masked.

Figure 6: Qualitative ablation and comparison of multi-frame self-supervision. The highlighted areas indicate clear improvements in occluded areas when using multi-frame self-supervision (second vs. third row). These improvements substantially outperform other unsupervised multi-frame methods (fourth and fifth row) although those use additional frames not only during training but also for inference. |
---|---
Input RGB (first image) | Predicted Flow
Figure 7: Limitations of unsupervised flow. Optimizing photometric consistency
can produce incorrect flow at shadows / reflections.

##### SMURF Components

Next, we test different combinations of the novel components in our method.
The results in Table 5 show that every component has a significant impact on
performance. Sequence losses and heavy augmentations are necessary to achieve
good results, especially on Sintel. And when using heavy augmentations, we
need to apply sequence losses to prevent divergence of the model. Full-Image
warping and multi-frame self-supervision have the strongest effect on KITTI,
which makes sense as larger camera motion in that dataset causes more
occlusions at the image boundaries which these components help to address.

##### Self-Supervision Modifications

We also ablated our proposed changes to self-supervision (subsubsection
3.2.1). The ablations in Table 6 show that even with full-image warping, self-
supervision remains an important component, and that it works best when not
masking the loss as in prior work and when we use the final output (not
intermediate model predictions) to generate the self-supervision labels.

##### Multi-Frame Self-Supervision

Lastly, we provide a qualitative ablation of multi-frame self-supervision
(subsubsection 3.2.3) and a comparison to other unsupervised methods that use
multi-frame information [11, 17]. Figure 6 shows that multi-frame self-
supervision substantially improves flow accuracy in occluded areas and does
this much better than related multi-frame methods while being the only
approach that requires multiple frames only during training and not for
inference.

###  5.3 Limitations

A major limitation of unsupervised optical flow is that it estimates apparent
visual motion rather than motion of physical objects (see Figure 7).
Overcoming this limitation requires some form of supervision, reasoning about
the 3-D space as in scene flow [32], reasoning about semantics, or a
combination of these. Future work could try to transfer the techniques from
our method to such approaches.

##  6 Conclusion

We have presented SMURF, an effective method for unsupervised learning of
optical flow that reduces the gap to supervised approaches and shows excellent
generalization across datasets and even to “zero-shot” depth estimation. SMURF
brings key improvements, most importantly (1) enabling the RAFT architecture
to work in an unsupervised setting via modifications to the unsupervised
losses and data augmentation, (2) full-image warping for learning to predict
out of frame motion, and (3) multi-frame self-supervision for improved flow
estimates in occluded regions. We believe that these contributions are a step
towards making unsupervised optical flow truly practical, so that optical flow
models trained on unlabeled videos can provide high quality pixel-matching in
domains without labeled data.

## References

  * [1] Filippo Aleotti, Fabio Tosi, Li Zhang, Matteo Poggi, and Stefano Mattoccia.  Reversing the cycle: Self-supervised deep stereo through enhanced monocular distillation.  In ECCV, 2020.
  * [2] Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert.  High accuracy optical flow estimation based on a theory for warping.  In ECCV, 2004.
  * [3] Daniel J. Butler, Jonas Wulff, Garrett B. Stanley, and Michael J. Black.  A naturalistic open source movie for optical flow evaluation.  In ECCV, 2012.
  * [4] Qifeng Chen and Vladlen Koltun.  Full flow: Optical flow estimation by global optimization over regular grids.  In CVPR, 2016.
  * [5] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Häusser, Caner Hazırbaş, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  In ICCV, 2015.
  * [6] James J. Gibson.  The Perception of the Visual World.  Houghton Mifflin, 1950.
  * [7] H. Hirschmüller.  Stereo processing by semi-global matching and mutual information.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2):328–341, February 2008.
  * [8] Berthold K. P. Horn and Brian G. Schunck.  Determining optical flow.  AI, 1981.
  * [9] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  In CVPR, 2017.
  * [10] Woobin Im, Tae-Kyun Kim, and Sung-Eui Yoon.  Unsupervised learning of optical flow with deep feature similarity.  In ECCV, 2020.
  * [11] Joel Janai, Fatma Güney, Anurag Ranjan, Michael J. Black, and Andreas Geiger.  Unsupervised learning of multi-frame optical flow with occlusions.  In ECCV, 2018.
  * [12] Rico Jonschkowski, Austin Stone, Jonathan T Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova.  What matters in unsupervised optical flow.  In ECCV, 2020.
  * [13] Diederick P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [14] Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang, Ying Tai, Donghao Luo, Chengjie Wang, Jilin Li, and Feiyue Huang.  Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation.  In CVPR, 2020.
  * [15] Pengpeng Liu, Irwin King, Michael Lyu, and Jia Xu.  Flow2Stereo: Effective self-supervised learning of optical flow and stereo matching.  In CVPR, 2020.
  * [16] Pengpeng Liu, Irwin King, Michael R. Lyu, and Jia Xu.  DDFlow: Learning optical flow with unlabeled data distillation.  In AAAI, 2019.
  * [17] Pengpeng Liu, Michael R. Lyu, Irwin King, and Jia Xu.  Selflow: Self-supervised learning of optical flow.  CVPR, 2019.
  * [18] Bruce D. Lucas and Takeo Kanade.  An iterative image registration technique with an application to stereo vision.  DARPA Image Understanding Workshop, 1981.
  * [19] Daniel Maurer and Andrés Bruhn.  Proflow: Learning to predict optical flow.  In BMVC, 2018.
  * [20] Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox.  A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation.  In CVPR, 2016.
  * [21] Simon Meister, Junhwa Hur, and Stefan Roth.  Unflow: Unsupervised learning of optical flow with a bidirectional census loss.  In AAAI, 2018.
  * [22] Moritz Menze, Christian Heipke, and Andreas Geiger.  Joint 3d estimation of vehicles and scene flow.  ISPRS Workshop on Image Sequence Analysis, 2015.
  * [23] Anurag Ranjan and Michael J. Black.  Optical flow estimation using a spatial pyramid network.  In CVPR, 2017.
  * [24] Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J. Black.  Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation.  In CVPR, 2019.
  * [25] Zhe Ren, Junchi Yan, Bingbing Ni, Bin Liu, Xiaokang Yang, and Hongyuan Zha.  Unsupervised deep learning for optical flow estimation.  In AAAI, 2017.
  * [26] Frank Steinbrücker, Thomas Pock, and Daniel Cremers.  Large displacement optical flow computation without warping.  In ICCV, 2009.
  * [27] Deqing Sun, Stefan Roth, and Michael J. Black.  Secrets of optical flow estimation and their principles.  In CVPR, 2010.
  * [28] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume.  In CVPR, 2018.
  * [29] Zachary Teed and Jia Deng.  Raft: Recurrent all pairs field transforms for optical flow.  In ECCV, 2020.
  * [30] Carlo Tomasi and Roberto Manduchi.  Bilateral filtering for gray and color images.  In ICCV, 1998.
  * [31] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky.  Instance normalization: The missing ingredient for fast stylization.  arXiv:1607.08022, 2016.
  * [32] Sundar Vedula, Simon Baker, Peter Rander, Robert Collins, and Takeo Kanade.  Three-dimensional scene flow.  In CVPR, 1999.
  * [33] Yang Wang, Peng Wang, Zhenheng Yang, Chenxu Luo, Yi Yang, and Wei Xu.  Unos: Unified unsupervised optical-flow and stereo-depth estimation by watching videos.  In CVPR, 2019.
  * [34] Yang Wang, Peng Wang, Zhenheng Yang, Chenxu Luo, Yi Yang, and Wei Xu.  Unos: Unified unsupervised optical-flow and stereo-depth estimation by watching videos.  In CVPR, 2019.
  * [35] Yang Wang, Yi Yang, Zhenheng Yang, Liang Zhao, Peng Wang, and Wei Xu.  Occlusion aware unsupervised learning of optical flow.  In CVPR, 2018.
  * [36] Jia Xu, René Ranftl, and Vladlen Koltun.  Accurate optical flow via direct cost volume processing.  In CVPR, 2017.
  * [37] Gengshan Yang and Deva Ramanan.  Volumetric correspondence networks for optical flow.  In NeurIPS, 2019.
  * [38] Zhichao Yin and Jianping Shi.  Geonet: Unsupervised learning of dense depth, optical flow and camera pose.  In CVPR, 2018.
  * [39] Jason J. Yu, Adam W. Harley, and Konstantinos G. Derpanis.  Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness.  In ECCV Workshop, 2016.
  * [40] Ramin Zabih and John Woodfill.  Non-parametric local transforms for computing visual correspondence.  In ECCV, 1994.
  * [41] Yiran Zhong, Yuchao Dai, and Hongdong Li.  Self-supervised learning for stereo matching with self-improving ability.  arXiv:1709.00930, 2017.
  * [42] Yiran Zhong, Pan Ji, Jianyuan Wang, Yuchao Dai, and Hongdong Li.  Unsupervised deep epipolar flow for stationary or dynamic scenes.  In CVPR, 2019.
  * [43] Yuliang Zou, Zelun Luo, and Jia-Bin Huang.  DF-Net: Unsupervised joint learning of depth and flow using cross-task consistency.  In ECCV, 2018.


# Splitting of the superfluid transition in 3He by nematic aerogels.

I. A. Fomin
P. L. Kapitza Institute for Physical Problems
Russian Academy of Science,
Kosygina 2, 119334 Moscow, Russia

###### Abstract

Role of inhomogeneous perturbations of nematic aerogel on the form of the
order parameter emerging at the transition of liquid 3He in the superfluid
state is considered. It is shown that a region of stability of the polar
distorted ABM phase can begin right from the transition temperature. The
symmetry argument is given, which selects the most favorable aerogel for
stabilization of pure polar phase.

##  1 Introduction

In the recent years new superfluid phases of liquid 3He were successfully
stabilized [1, 2]. Existence of multiple phases is a typical manifestation of
unconventional Cooper pairing. In case of 3He it is pairing in the state with
the orbital moment $l=1$ and spin $s=1$. The corresponding order parameter is
proportional to complex 3$\times$3 matrix $A_{\mu j}$, its first index
enumerates 3 projections of spin, while the second index - three projections
of the orbital moment. According to Landau theory transition occurs when the
coefficient at the second order invariant in the expansion of the gain of free
energy in powers of the order parameter changes sign. In 3He the second order
term is $\delta f\sim a(P,T)A_{\mu j}A_{\mu j}^{\ast}$. The temperature of
transition $T_{c}$ at a given pressure $P$ is determined by the condition
$a(P,T_{c})=0$. This temperature is “degenerate” in a sense that at this
temperature the normal phase develops instability to formation of Cooper pairs
with all three possible projections of spin $s_{z}=0$, $s_{z}=\pm 1$ and three
projections of orbital moment $l_{z}=0$, $l_{z}=\pm 1$. The proper combination
of these basis functions is found by minimization of the contribution of the
fourth order invariants to the gain of the free energy. In 3He there are five
such invariants: $I_{1}=A_{\mu j}A_{\mu j}A_{\nu l}^{*}A_{\nu l}^{*}$,
$I_{2}=A_{\mu j}A_{\mu j}^{*}A_{\nu l}A_{\nu l}^{*}$, $I_{3}=A_{\mu j}A_{\nu
j}A_{\mu l}^{*}A_{\nu l}^{*}$, $I_{4}=A_{\mu j}A_{\nu j}^{*}A_{\nu l}A_{\mu
l}^{*}$, $I_{5}=A_{\mu j}A_{\nu j}^{*}A_{\mu l}A_{\nu l}^{*}$, they enter free
energy as a combination $\sum_{s=1}^{5}\beta_{s}I_{s}$ where
$\beta_{1},..,\beta_{5}$ are phenomenological coefficients. Generally this
combination has many extrema and several minima [3, 4], but at realistic
values of coefficients $\beta_{1},...,\beta_{5}$ only two of the minima are
realized. They correspond to Anderson-Brinkman-Morel (ABM) and Balian and
Werthamer (BW) phases [4].

New phases are of different origin. They are stabilized by external forces,
which lower spherical symmetry of normal phase and split the transition.
Symmetry with respect to rotations of spin is broken by magnetic field. The
resulting new phases A1, A2 were studied before [5], we will not discuss it
here. No bulk field is available for splitting of the transition in liquid 3He
over projections of orbital moment. Mechanical rotation would require too high
angular velocities. Aoyama and Ikeda [6] suggested to use for lowering of
orbital symmetry oriented anisotropic impurities. They considered ensemble of
impurities with the average cross-section for scattering of the Fermi
quasiparticles $\sigma(\hat{k})\sim[1+\delta(\hat{k}\cdot\hat{z})^{2}]$, where
$\hat{z}$ is the assumed direction of anisotropy and $\hat{k}$ \- the momentum
transfer at the scattering. Following argument of the theory of
superconducting alloys [7] they concluded that the transition temperature will
split in two - $T_{c0}$ corresponding to projection $l_{z}=0$, and $T_{c1}$ \-
to projections $l_{z}=\pm 1$. If anisotropy constant $\delta$ is negative
$T_{c0}>T_{c1}$ and for each pressure even for small $\delta$ there exist a
finite interval of temperatures starting from the temperature of transition in
the superfluid state where Cooper pairs have $l=1$ and $l_{z}=0$. This state
corresponds to the polar phase of 3He. This proposal stimulated experiments
aimed at stabilization of the polar phase (for a review cf.[2]). As a globally
anisotropic ensemble of impurities were used different types of the so-called
_nematic_ aerogels. These aerogels are formed by straight and nearly parallel
strands. In the earliest experiments it was “obninsk” aerogel [8]. Its strands
of diameters $d\approx 9nm$ consist of amorphous AlOOH. Anisotropy of this
aerogel, as judged from the ratio of the mean free pathes of the Fermi quasi-
particles along the strands and perpendicular to this direction
$l_{\|}/l_{\bot}\approx 1.4$ is much greater than that assumed in the
calculations of Aoyama and Ikeda. So it came as a surprise that instead of the
polar phase the polar distorted ABM-phase which is a superposition of all
projections of the angular moment was observed in these experiments. When in
the further experiments [1] the anisotropy in 3He was induced by another
nematic aerogel with even greater anisotropy - _nafen_ , the polar phase was
securely identified and confirmed [9]. The goal of this paper is to use the
Landau theory of the second order phase transitions for finding out which
characteristics of aerogel are favoring the particular form of the order
parameter, emerging at the transition.

##  2 Local anisotropy

In theoretical discussions of the phase diagram of superfluid 3He in aerogel
[10, 11, 12] in a framework of the Landau theory aerogel was considered as a
continuous media, anisotropic but not chiral. The term of the second order in
the expansion of the gain of free energy in that case can be written as
$\delta f\sim\Lambda_{jl}(P,T)A_{\mu j}A_{\mu l}^{\ast}$, where
$\Lambda_{jl}(P,T)$ is _real symmetric_ matrix. This matrix can be
diagonalized and has real diagonal elements
$\tau_{x}(P,T),\tau_{y}(P,T),\tau_{z}(P,T)$. Directions of its eigenvectors,
which are real and mutually orthogonal can be taken as orts of the coordinate
system. Formally there may be 3 transition temperatures, when one of the
diagonal elements changes sign. Real transition takes place at the highest of
the three temperatures. In the sited above papers aerogel was assumed on the
average axially symmetric, so that e.g. $\tau_{x}(P,T)=\tau_{y}(P,T)$ and $z$
is the symmetry axis. Such “field” description of aerogel leaves aside all
effects originating from the discrete structure of aerogel, in particular a
possible variation of the order parameter in a space between the strands.

According to Ref.[13] the strands of aerogel perturb the condensate in their
neighborhood on a distance of the order of the coherence length $\xi_{0}$
which in the superfluid 3He depending on a pressure varies in the interval
20$\div$80 nm. Reaction of the condensate on this perturbation extends for the
temperature dependent coherence length $\xi(T)$ which in a vicinity of $T_{c}$
meets the condition $\xi(T)\gg\xi_{0}$. In this vicinity perturbation, induced
by aerogel can be treated as local and described by the following term in the
Landau expansion of free energy in powers of the order parameter
$f_{ag}=N(0)\Lambda_{jl}(\mathbf{r})A_{\mu j}A_{\mu l}^{\ast}$. Here $N(0)$ is
the density of states. Matrix $\Lambda_{jl}(\mathbf{r})$ depends on coordinate
$\mathbf{r}$ and it is _Hermitian_ since the free energy has to be real. It is
assumed that the strands of aerogel do not interact directly with spins of
quasiparticles. In the experiments [2] this interaction was eliminated by a
thin film of 4He covering the strands. It is convenient to represent the
matrix $\Lambda_{jl}(\mathbf{r})$ as the sum of its ensemble average:
$\tau_{jl}=\langle\Lambda_{jl}\rangle$, and fluctuating local anisotropy
$\eta_{jl}(\mathbf{r})=\Lambda_{jl}(\mathbf{r})-\tau_{jl}$ so that
$\langle\eta_{jl}\rangle=0$. In what follows we assume that the considered
aerogels on the average are non-chiral, i.e. they do not distinguish left and
right orientation. Then matrix $\tau_{jl}$ is real symmetric, its principal
directions as was explained above determine orientation of coordinate axes
$(x,y,z)$ and the principal values $\tau_{x},\tau_{y},\tau_{z}$ are real
functions of
temperature:$\tau_{jl}=\tau_{x}\hat{x}_{j}\hat{x}_{l}+\tau_{y}\hat{y}_{j}\hat{y}_{l}+\tau_{z}\hat{z}_{j}\hat{z}_{l}$
With these notations the expansion of the free energy density in a vicinity of
temperature of instability of the normal phase can be written as:

$\frac{f_{S}-f_{N}}{N(0)}=[\tau_{jl}+\eta_{jl}(\mathbf{r})]A_{\mu
j}^{\ast}A_{\mu l}+\xi_{s}^{2}\left(\frac{\partial A^{*}_{\mu l}}{\partial
x_{n}}\frac{\partial A_{\mu l}}{\partial
x_{n}}\right)+\frac{1}{2}\sum_{s=1}^{5}\beta_{s}I_{s}.$ $None$

The contribution of the gradient terms is written here in a model isotropic
form to avoid non-essential complications. The equilibrium order parameter is
found from the equation

$[\tau_{jl}+\eta_{jl}(\mathbf{r})]A_{\mu
l}-\xi_{s}^{2}\left(\frac{\partial^{2}A_{\mu j}}{\partial
x_{n}^{2}}\right)+\frac{1}{2}\sum_{s=1}^{5}\beta_{s}\frac{\partial
I_{s}}{\partial A_{\mu j}^{\ast}}=0.$ $None$

This system of differential equations hardly can be solved for a general form
of matrix $\eta_{jl}(\mathbf{r})$. We limit ourselves to the case when the
effect of random anisotropy $\eta_{jl}(\mathbf{r})$ can be treated as a
perturbation. The perturbed solution of Eqn.(2) can be sought in a form
$A_{\mu j}(\mathbf{r})=\bar{A}_{\mu j}+a_{\mu j}(\mathbf{r})$, where
$\bar{A}_{\mu j}$ is the ensemble averaged order parameter. This average, can
be considered as the order parameter of a given phase of 3He in aerogel, it
describes thermodynamic properties of the liquid. The fluctuating part $a_{\mu
j}(\mathbf{r})$ vanishes at such averaging.The condition of instability of the
normal phase of liquid 3He also has to be expressed directly in terms of
$\bar{A}_{\mu j}$. For this we substitute $A_{\mu j}=\bar{A}_{\mu j}+a_{\mu
j}(\mathbf{r})$ in Eq. (2) and average the obtained expression. Keeping terms
up to the second order over $a_{\mu j}$ and $\eta_{jl}(\mathbf{r})$ and linear
with respect to $\bar{A}_{\mu j}$ we arrive at:

$\tau_{jl}\bar{A}_{\mu l}+\langle\eta_{jl}(\mathbf{r})a_{\mu
l}(\mathbf{r})\rangle=0,$ $None$

The Greek index enters this equation as a parameter. In the experiments [2]
all discussed phases belong to the ESP type, i.e. their order parameter can be
factorized into the spin and orbital parts. The spin part is a real vector
$d_{\mu}$. Orbital anisotropy effect orientation of $d_{\mu}$ only via weak
dipole interaction, which is neglected here. In what follows we consider only
orbital part of $A_{\mu j}$ and suppress the index $\mu$:

$\tau_{jl}\bar{A}_{l}+\langle\eta_{jl}a_{l}\rangle=0.$ $None$

Fluctuating part $a_{l}$ is found from the linear equation

$\tau_{jl}a_{l}-\xi_{s}^{2}\left(\frac{\partial^{2}a_{j}}{\partial
x_{n}^{2}}\right)=-\eta_{jl}\bar{A}_{l}.$ $None$

It is solved by Fourier transformation

$a_{l}(\vec{k})=-(\tau_{ln}+\delta_{ln}\xi^{2}k^{2})^{-1}\eta_{nm}(\vec{k})\bar{A}_{m}$
$None$

Substitution of this solution in

$\langle\eta_{jl}a_{l}\rangle=\int\eta_{jl}(-\vec{k})a_{l}(\vec{k})\frac{d^{3}k}{(2\pi)^{3}}$
$None$

renders the linear equation for a temperature of instability of the normal
phase:

$\left(\tau_{jl}-V_{jl}\right)\bar{A}_{l}=0,$ $None$

where

$V_{jl}=\left\langle\int\frac{d^{3}k}{(2\pi)^{3}}\eta_{mj}^{\ast}(\vec{k})(\tau_{mn}+\delta_{mn}\xi^{2}k^{2})^{-1}\eta_{nl}(\vec{k})\right\rangle$
$None$

The relation $\eta_{jl}(-\vec{k})=\eta_{lj}^{\ast}(\vec{k})$ was used here.
Eq. (8) has solution if

$det\left(\tau_{jl}-V_{jl}\right)=0.$ $None$

Formally this is an eigenvalue problem with respect to the transition
temperature $T=T_{c}$, which enters $\tau_{jl}$. For its solution one can
apply the standard perturbation theory (e.g. [14]). In zero order
approximation $\tau_{jl}\bar{A}_{l}=0$ there are 3 solutions
$\tau_{x}=0,\tau_{y}=0,\tau_{z}=0$ with corresponding eigenvectors $(1,0,0)$,
$(0,1,0)$ and $(0,0,1)$. Assume that $\tau_{x}\geq\tau_{y}>\tau_{z}$ so that
the first transition occurs at $\tau_{z}(T=T_{c})=0$. In a spirit of the
Landau theory $\tau_{z}(T)$ can be expanded in the vicinity of $T_{c}$. With a
proper normalization $\tau_{z}=(T-T_{c})/T_{c}$ The first order correction to
the transition temperature according to the recipe of perturbation theory is
$(T-T_{c})=T_{c}V_{zz}$. The emerging order parameter in this approximation
has components $\bar{A}_{x}=\frac{V_{13}}{\tau_{x}}$,
$\bar{A}_{y}=\frac{V_{23}}{\tau_{y}}$ and $\bar{A}_{z}=1$. It follows from the
definition (9) that matrix $V_{jl}$ is Hermitian. It means that its symmetric
part $(V_{jl}+V_{lj})/2$ is real and antisymmetric $(V_{jl}-V_{lj})/2$ \- pure
imaginary. If both $V_{13}$ and $V_{23}$ are real the transverse components
can occur as a result of a small change of direction of $\bar{A}_{l}$. The
emerging phase is still polar, but with a different orientation. Imaginary
increments to transverse components can not be included in the order parameter
of polar phase. Together with the original $z$-component $\bar{A}_{z}=1$ they
form the order parameter of the polar-distorted ABM-phase, which develops at
further lowering of temperature in the ABM-phase continuously, without the
phase transition. Pure polar phase will form if $V_{13}=V_{31}$ and
$V_{23}=V_{32}$. This condition is met for model forms of $\eta_{jl}(\vec{k})$
discussed in the literature [6, 15]. These models are based on generalizations
of the theory of superconducting alloys, representing aerogel as an ensemble
of individual impurities. In the discussed models $\eta_{jl}(\vec{r})$ is real
symmetric tensor. That secures absence of imaginary part in $V_{jl}$, but this
restriction has no physical justification. It is possible to formulate
symmetry based restriction for idealized nematic aerogels, i.e. for aerogels
with infinitely long straight ideally parallel and smooth strands. Such
aerogels have symmetry with respect to the mirror reflection in a plane normal
to the strands. In this case at the transformation $z\rightarrow(-z)$
components $V_{xz}$ and $V_{yz}$ of tensor $V_{jl}$ have to change their
signs, but with the assumed symmetry it means that they have to be equal to
themselves with the opposite sign, i.e. to vanish. The aerogels nafen and
mullite as is explained in Ref.[2] are more close to ideal than the “obninsk”
samples. That can explain why the polar phase was observed in the first two
and not observed in the last type of aerogel. It has to be mentioned that the
ideal nematic aerogel does not suppress temperature of transition of 3He in
the superfluid state [16]. It means that the degree of suppression of the
$T_{c}$ can be used as an indicator of quality of a given sample.

In conclusion, the average uniaxial anisotropy induced in the liquid 3He by
the immersed aerogel does not provide the sufficient condition for
stabilization of the polar phase. Spatial fluctuations of the local anisotropy
depending on a structure of aerogel can favor presence of imaginary components
of the order parameter and formation at $T_{c}$ of distorted ABM phase. The
most favorable for formation of pure polar phase is the ideal nematic aerogel.

##  3 Acknowledgments

I thank V.V.Dmitriev, A.A. Soldatov , and A.N.Yudin for useful discussions and
constructive criticism.

## References

  * [1] V.V.Dmitriev, A.A. Senin, A.A. Soldatov , A.N.Yudin, Phys. Rev. Lett.115, 165304 (2015)
  * [2] V.V.Dmitriev, A.A. Soldatov , A.N.Yudin, Zh.Exp.Teor.Fiz., 158, 6 (2020).
  * [3] . . , ,93, 141 (1987), JETP, 66, 79 (1987).
  * [4] D.Vollhardt and P. Woelfle, The superfluid Phases of Helium 3, Taylor and Francis (1990).
  * [5] R. Ruel and H. Kojima, Phys. Rev. Lett.54, 2238 (1985)
  * [6] K. Aoyama and R. Ikeda, Phys. Rev. B 73, 060504 (2006)
  * [7] A. A. Abrilosov and L. P. Gor‘kov, ZhETF 35, 1558 (1958). Sov. Phys. JETP 9, 220 (1959).
  * [8] R. Sh. Askhadullin, V. V.Dmitriev, D. A. Krasnikhin, P. N. Martynov, A. A. Osipov, A.A. Senin, A.N.Yudin, JETP Lett, 95, 326 (2012).
  * [9] N. Zhelev, M. Reichl, T. S. Abhilash, E. N. Smith, K. X. Nguen, E. J. Mueller, and J. M. Parpia, Nat. Commun. 7, 12975 (2016).
  * [10] J.A. Sauls, Phys. Rev. B 88, 214503 (2013)
  * [11] I. A. Fomin and E. V. Surovtsev, JETP Lett, 97, 742 (2013). .
  * [12] I. A. Fomin, Zh.Exp.Teor.Fiz., 145, 871 (2014).
  * [13] D. Rainer and M. Vuorio, J. Phys. C: Solid State Phys. 10, 3093 (1977).
  * [14] L.D. Landau and E.M. Lifshitz, Kvantovaja Mekhanika, Nauka, Moscow
  * [15] R. C. Regan, J. J. Wiman, and J. A. Sauls, arXiv: 2105.01257 v1
  * [16] I. A. Fomin, Zh.Exp.Teor.Fiz. 154, 1034 (2018), (JETP, 127, 933 (2018)