# Optimizing Video Prediction via Video Frame Interpolation

Yue Wu Qiang Wen Qifeng Chen
The Hong Kong University of Science and Technology

###### Abstract

Video prediction is an extrapolation task that predicts future frames given
past frames, and video frame interpolation is an interpolation task that
estimates intermediate frames between two frames. We have witnessed the
tremendous advancement of video frame interpolation, but the general video
prediction in the wild is still an open question. Inspired by the photo-
realistic results of video frame interpolation, we present a new optimization
framework for video prediction via video frame interpolation, in which we
solve an extrapolation problem based on an interpolation model. Our video
prediction framework is based on optimization with a pretrained differentiable
video frame interpolation module without the need for a training dataset, and
thus there is no domain gap issue between training and test data. Also, our
approach does not need any additional information such as semantic or instance
maps, which makes our framework applicable to any video. Extensive experiments
on the Cityscapes, KITTI, DAVIS, Middlebury, and Vimeo90K datasets show that
our video prediction results are robust in general scenarios, and our approach
outperforms other video prediction methods that require a large amount of
training data or extra semantic information.

Figure 1: Our method can produce plausible video prediction results in diverse
scenarios without external training, such as driving scenes, human motion,
animation, and natural scenes. We use red arrows to indicate the motions from
input frames. Video results are presented in the supplementary material.

##  1 Introduction

Video prediction is an extrapolation task to predict future video frames given
some past frames. Video prediction has broad applications including robotics
planning, autonomous driving and video manipulations [6, 39, 23, 41]. For
instance, predicted videos can help autonomous robots to better plan future
actions with future visual information. Video prediction is also a fundamental
task for unconditional video synthesis that can be decomposed into image
synthesis and future video prediction.

Figure 2: Overview of our method. We optimize optical flow
$\tilde{f}_{t+1\rightarrow t}$ by a video frame interpolation $G$ [11]. Our
optimization objective is image-level distance $\mathcal{L}_{img}$ and a
consistency constraint between our predicted flow $\tilde{f}_{t+1\rightarrow
t}$ and the flow $f^{G}_{t\rightarrow t+1}$ generated by $G$.

Video prediction is a challenging extrapolation problem. Several studies [22,
42, 20, 18] only take RGB frames as input for video prediction and find that
the video prediction problem is difficult to solve, because of the inherent
high complexity of video prediction and the uncertainty of future states.
Thus, recently, a lot of constraints and assumptions about the modeling scene
have been employed to simplify the problem. However, these assumptions reduce
the generalization ability of these video prediction models. FVS [45] and Lee
_et al_. [15] require semantic maps to decompose the scene. Bei  _et al_. [4]
first predict semantic maps and then synthesize future frames. Qi  _et al_.
[35] require depth maps to reconstruct 3D point clouds. However, such
additional information is often hard to obtain or estimate correctly in
general scenarios. These strong assumptions limit these methods to be only
applicable to data when these assumptions hold. For example, failing to detect
some objects (including unseen objects) will lead to performance degradation.
When this extra information is unavailable or of poor quality, these methods
suffer from performance degradation and cannot be even applied in diverse
videos in the real world.

Moreover, these external methods usually need to train one model for each
specific scenario, making them difficult generalize to other scenarios. For
example, it is hard to apply a video prediction model trained on a driving
scene to a human moving dataset, as the motion difference is huge.

To address these issues, we propose an optimization-based video prediction
method, without the requirement of external training (no external dataset is
needed for training), and can produce state-of-the-art results. Our insight is
that we can cast the video prediction problem as a video frame interpolation
(VFI) based optimization problem. Inspired by the recent success of VFI, we
connect these two problems to solve the video prediction problem in a new way.
Our method does not require any assumption such as semantic segmentation and
can be applied to any video. We evaluate our method on multiple datasets, and
our method can outperform the state of the arts. Our contributions can be
summarized as follows:

  * •

We present the first optimization framework for video prediction. We cast the
extrapolation problem of video prediction as an optimization problem with VFI.

  * •

Our framework is highly flexible as it does not require any semantic or
instance maps, prior knowledge about the scene, or external training. Our
method is applicable to video prediction in any scene at any resolution.

  * •

Our method obtains outstanding performance on various datasets and outperforms
state-of-the-art video prediction approaches that require additional
information. Our method surpasses external learning methods taking only RGB
frames as input by a large margin.

##  2 Related work

###  2.1 Video Prediction

Early works, taking RGB frames as input, employ several mechanisms to improve
video prediction. MCNet [42] decomposes a scene into content and motion
components. DVF [20] proposes deep voxel flow to synthesize future frames.
However, video prediction in the wild is still quite hard due to its inherent
high complexity and uncertainty. Thus, explicit modeling, constraints, and
assumptions about the scene are introduced. Qi  _et al_. [35] utilize depth
maps to reconstruct 3D point clouds. Gao _et al_. [7] use semantic maps to
enforce layout consistency. FVS [45] needs semantic segmentation and instance
segmentation to decompose a scene into background and foreground identities.
Bei  _et al_. [4] and Lee  _et al_. [15] also require semantic maps.

Although the performance of video prediction has been gradually improved, the
generalization ability of these approaches is arguably reduced. It is hard to
apply these methods to data without these extra annotations. Moreover, these
methods may suffer from performance degradation when the test data is from a
different domain than the train data. For example, it is hard to apply a model
trained on robot scenarios to driving scenes because the motion in these two
domains is quite different. Thus, we propose an optimization-based video
prediction method, which can produce state-of-the-art results without external
training (thus no domain gap).

###  2.2 Video Frame Interpolation

Contrary to the challenges of video prediction problems, VFI has gained great
success recently, which motivates our model design. VFI aims to interpolate
intermediate frames between successive input frames. There are three
categories of algorithms: kernel-based [29, 30, 3, 2], phase-based [25, 24],
and motion-based [21, 19, 3, 2, 13, 27, 10, 32, 28, 33]. These motion-based
methods use bi-directional optical flows to warp two consecutive frames
forward and backward to obtain an intermediate frame. Our method adopts a
motion-based framework. Super SloMo [13] linearly combines bi-directional
flows as an initial approximation of the intermediate flows for further
refinement. Park  _et al_. [33] conduct asymmetric bilateral motion
estimation. Sim  _et al_. [37] first handle the VFI for 4K videos with large
motion. RIFE [11] is a real-time interpolation algorithm that estimates the
intermediate flows in a coarse-to-fine fashion. Different from the domain gap
problem in video prediction, there are much fewer domain constraints in VFI.
These methods do not require additional information, such as semantic maps and
depth maps, and can produce outstanding interpolation performance even with
complex motion. Inspired by the success of VFI, we cast video prediction as an
optimization problem based on VFI.

###  2.3 Optimization-based Methods

Since learning-based methods suffer from the domain gap between training data
and test data, optimization-based methods on test data are still competitive
nowadays. Gatys _et al_. [8] propose the first optimization-based method for
neural style transfer. Shaham _et al_. [36] optimize a generative model on a
single image and can generate high quality and diverse samples from the image.
Lei  _et al_. [16, 17] optimize a network to improve the temporal consistency.
Mildenhall _et al_. [26] optimize a fully-connected network on a sparse set of
input views from a scene for novel view synthesis. These optimization-based
methods inspire us to propose a framework that not only tackles the domain gap
problem but also provides an on-the-fly control for users during optimization.

##  3 Method

###  3.1 Problem Formulation

Let $x_{t}$ be the video frame at time step $t$. The input to our framework
includes two recent RGB frames $x_{t-1}$ and $x_{t}$. Our goal is to predict
the future frames $\\{\tilde{x}_{t+1},\tilde{x}_{t+2},\ldots\\}$. We adopt a
pretrained video interpolation network [11] denoted as $G$. We will focus on
predicting the next frame $\tilde{x}_{t+1}$ first as we can predict future
frames one by one sequentially. During the optimization process, the
parameters of $G$ are unchanged. Our primary objective is

$\tilde{x}_{t+1}^{*}=\underset{\tilde{x}_{t+1}}{\mathrm{argmin}}\,E(G(x_{t-1},\tilde{x}_{t+1}),x_{t}),$
(1)

where $E$ is an objective function that measures image similarity. Here we
utilize a VFI network $G$ to constrain the relationship among $x_{t-1}$,
$\tilde{x}_{t+1}$, and $x_{t}$.

To ease the optimization process, we choose to optimize optical flow
$\tilde{f}_{t+1\rightarrow t}$ between the predicted frame $\tilde{x}_{t+1}$
and the last observed frame $x_{t}$ instead of directly optimize
$\tilde{x}_{t+1}$. $\tilde{x}_{t+1}$ is computed using backward warping [12]:

$\tilde{x}_{t+1}=warp(x_{t},\tilde{f}_{t+1\rightarrow t}).$ (2)

Then Eq. 1 can be rewritten as

${\tilde{f}_{t+1\rightarrow t}}^{*}=\underset{\tilde{f}_{t+1\rightarrow
t}}{\mathrm{argmin}}\,E(G(x_{t-1},warp(x_{t},\tilde{f}_{t+1\rightarrow
t})),x_{t}).$ (3)

#### Flow initialization.

To ease the optimization of Eq. 3, it is a good practice to start with a flow
$\tilde{f}_{t+1\rightarrow t}$ that produces an approximate motion. Therefore,
we initialize it utilizing the negative flow of $f_{t\rightarrow t-1}$:

$\tilde{f}_{t+1\rightarrow t}=\delta(-f_{t\rightarrow t-1}).$ (4)

We first compute $-f_{t\rightarrow t-1}$ as a rough approximation of
$f_{t->t+1}$. Then we initialize $\tilde{f}_{t+1\rightarrow t}$ as the
inversion of $-f_{t\rightarrow t-1}$. $\delta$ represents the operation
similar to the flow reversal layer [46] to convert a forward flow to a
backward flow (details in the supplement).

However, directly optimizing Eq. 3 is still difficult, because the constraint
towards $\tilde{f}_{t+1\rightarrow t}$ is indirect, and the optimization
process is difficult to converge.

###  3.2 Video Frame Interpolation Network

Thus, we propose to utilize the intermediate results of network $G$. Given
$x_{t-1}$ and $\tilde{x}_{t+1}$ as input, $G$ generates optical flows of two
directions $f^{G}_{t\rightarrow t-1}$, $f^{G}_{t\rightarrow t+1}$, and a mask
$m^{G}$. The superscript $G$ denotes that it is the output of network $G$. The
video interpolation network warps $x_{t-1}$ and $\tilde{x}_{t+1}$ towards time
step $t$:

$\displaystyle I^{G}_{t-1}$ $\displaystyle=warp(x_{t-1},f^{G}_{t\rightarrow
t-1}),$ (5) $\displaystyle I^{G}_{t+1}$
$\displaystyle=warp(\tilde{x}_{t+1},f^{G}_{t\rightarrow t+1}),$ (6)
$\displaystyle I^{G}_{t}$ $\displaystyle=I^{G}_{t-1}\times
m^{G}+I^{G}_{t+1}\times(1-m^{G}),$ (7)

where $I^{G}_{t-1}$ is the intermediate interpolation frame by warping
$x_{t-1}$ using $f^{G}_{t\rightarrow t-1}$. $I^{G}_{t+1}$ is the intermediate
interpolation frame by warping $\tilde{x}_{t+1}$ using $f^{G}_{t\rightarrow
t+1}$. The final interpolation result is a weighted sum of $I^{G}_{t-1}$ and
$I^{G}_{t-1}$.

We utilize $I^{G}_{t+1}$ instead of $I^{G}_{t}$ since $I^{G}_{t+1}$ has a
closer relation with $\tilde{x}_{t+1}$ and can dismiss the effect of $m^{G}$.
We employ a $L_{1}$ distance between $I^{G}_{t+1}$ and $x_{t}$:

$\mathcal{L}_{img}=\left\|I^{G}_{t+1}-x_{t}\right\|_{1}.$ (8)

We also argue that there is a forward-backward consistency relationship
between $f^{G}_{t\rightarrow t+1}$ and $\tilde{f}_{t+1\rightarrow t}$. This
constraint means that after the forward and backward propagation, the pixels
should go back to the original locations:

$\mathcal{L}_{cons}=\sum_{\mathbf{p}}\left\|\Delta\tilde{f}_{{t+1}\rightarrow
t}(\mathbf{p})\right\|_{1},$ (9)

where $\Delta\tilde{f}_{{t+1}\rightarrow t}(\mathbf{p})$ is the discrepancy
obtained from forward and backward flow check at pixel location $\mathbf{p}$:

$\displaystyle\Delta\tilde{f}_{t+1\rightarrow t}(\mathbf{p})$ $\displaystyle=$
$\displaystyle\mathbf{p}-\big{(}\mathbf{p}^{\prime}+f^{G}_{t\rightarrow
t+1}(\mathbf{p}^{\prime})\big{)},$ (10) $\displaystyle\mathbf{p}^{\prime}$
$\displaystyle=$ $\displaystyle\mathbf{p}+\tilde{f}_{t+1\rightarrow
t}(\mathbf{p}).$ (11)

Our overall objective function is
$\mathcal{L}_{total}=\omega_{1}\mathcal{L}_{img}+\omega_{2}\mathcal{L}_{cons}$,
where $\omega_{1}$ and $\omega_{2}$ are the loss weights.

###  3.3 Flow Inpainting

There always exist occlusion areas in optical flow: some pixels do not have
corresponding pixels in successive frames. The estimated optical flow in
occlusion areas is unreliable. Thus, we set a threshold $\alpha$ to mask out
these areas, and using flow inpainting to fill in the holes:

$\phi(\mathbf{p})=\left\\{\begin{matrix}&1&\quad
if\quad\left\|\Delta\tilde{f}_{t+1\rightarrow
t}(\mathbf{p})\right\|_{1}>\alpha,\\\ &0&\quad otherwise.\end{matrix}\right.$
(12)

If $\phi(\mathbf{p})$ is 1, its optical flow is treated unreliable, and is
inpainted by a linear combination of neighboring valid flow values, whose
weights are inversely proportional to the distance between the invalid pixel
and the valid pixels.

We try to use adaptive weights like Softmax Splatting [28] to resolve the
occlusions and find it unsuitable for our framework. Detailed analysis is
described in the supplement.

External learning methods
---
| External training | Semantic | Instance | Depth
PredNet [22] | ✓ | $\times$ | $\times$ | $\times$
MCNET [42] | ✓ | $\times$ | $\times$ | $\times$
DVF [20] | ✓ | $\times$ | $\times$ | $\times$
Vid2vid [43] | ✓ | ✓ | $\times$ | $\times$
Qi _et al_. [35] | ✓ | ✓ | $\times$ | ✓
Seg2vid [31] | ✓ | ✓ | $\times$ | $\times$
FVS [45] | ✓ | ✓ | ✓ | $\times$
HVP [15] | ✓ | ✓ | $\times$ | $\times$
SADM [4] | ✓ | ✓ | $\times$ | $\times$
Optimization methods
Ours | $\times$ | $\times$ | $\times$ | $\times$

Table 1: Comparison with other video prediction methods in terms of methods’
requirements.

###  3.4 Implementation

#### Multi-frame prediction.

For multi-frame prediction, we choose to optimize the next frame recurrently.
We try to set the optimization target as multiple optical flows and optimize
multiple future frames together. The result turns out recurrently optimizing
the next frame is more stable.

The input frame length for all datasets is set to 2. The hyperparameters
$\omega_{1}$, $\omega_{2}$, $\alpha$ are set to 1.0, 3.0, 1.5 empirically. We
adopt RAFT [40] to estimate optical flow used for optimization target
initialization. The Adam optimizer [14] is used with a learning rate of 0.1
for 3000 iterations for each future frame. We adopt the VFI method RIFE [11],
which is pretrained only on Vimeo90K [47], for all the experiments. During the
process of optimization, the network weight of RIFE [11] is fixed.

|  | Cityscapes | KITTI
---|---|---|---
|  | MS-SSIM ($\times 1\mathrm{e}{-2}$)$\uparrow$ | LPIPS ($\times 1\mathrm{e}{-2}$)$\downarrow$ | MS-SSIM ($\times 1\mathrm{e}{-2}$)$\uparrow$ | LPIPS ($\times 1\mathrm{e}{-2}$)$\downarrow$
| Input | t+1 | t+3 | t+5 | t+1 | t+3 | t+5 | t+1 | t+3 | t+5 | t+1 | t+3 | t+5
External learning methods
PredNet [22] | RGB | 84.03 | 79.25 | 75.21 | 25.99 | 29.99 | 36.03 | 56.26 | 51.47 | 47.56 | 55.35 | 58.66 | 62.95
MCNET [42] | RGB | 89.69 | 78.07 | 70.58 | 18.88 | 31.34 | 37.34 | 75.35 | 63.52 | 55.48 | 24.05 | 31.71 | 37.39
DVF [20] | RGB | 83.85 | 76.23 | 71.11 | 17.37 | 24.05 | 28.79 | 53.93 | 46.99 | 42.62 | 32.47 | 37.43 | 41.59
Vid2vid [43] | RGB+S. | 88.16 | 80.55 | 75.13 | 10.58 | 15.92 | 20.14 | N/A | N/A | N/A | N/A | N/A | N/A
Seg2vid [31] | RGB+S. | 88.32 | N/A | 61.63 | 9.69 | N/A | 25.99 | N/A | N/A | N/A | N/A | N/A | N/A
FVS [45] | RGB+S.+I. | 89.10 | 81.13 | 75.68 | 8.50 | 12.98 | 16.50 | 79.28 | 67.65 | 60.77 | 18.48 | 24.61 | 30.49
Optimization methods
Ours | No external training | 94.54 | 86.89 | 80.40 | 6.46 | 12.50 | 17.83 | 82.71 | 69.50 | 61.09 | 12.34 | 20.29 | 26.35

Table 2: Comparison with state-of-the-art methods on the Cityscapes and KITTI
datasets. S. and I. denote that the method requires semantic maps or instance
maps as input. Our method can outperform previous video prediction methods by
a large margin. Figure 3: Multi-frame prediction comparison on KITTI. As the
pink boxes show, DVF [20] fails to separate the motion of the blue car from
the background and produces a “zooming-in” effect, which is the major motion
exhibited in the dataset, caused by the forward movement of the running car.
FVS [45] wrongly predicts the motion of the blue car at $t+3$ and $t+5$,
resulting in incorrect frame prediction results. Our model can correctly
capture the motion of the blue car without external training.

##  4 Experiments

We compare our method with state-of-the-art methods as shown in Table 1 in
terms of methods’ requirements. Due to the complexity of future video
prediction, many recent methods add some extra assumptions for video
prediction, such as semantic maps [43, 35, 31, 45, 15, 4], instance maps [45],
and depth maps [35]. These assumptions may improve prediction performance but
vastly decrease their generalization capability. Furthermore, these methods
demand a training dataset to train a neural network. However, our method can
avoid these restrictions by casting the video prediction problem as
optimization. We do not require external training and do not have any
assumptions about the data. Our method is very general and outperforms
previous external RGB-based methods and methods using additional assumptions
[43, 31, 45]. Statistical analysis for long-term prediction and more visual
comparisons are provided in the supplement.

###  4.1 Evaluation on Driving Datasets

We first evaluate our approach and relevant baselines on driving datasets,
where semantic information is available, because some baselines require
additional semantic maps.

Datasets. Cityscapes [5] and KITTI datasets [9] contain driving sequences. Our
evaluation setting follows [45].

| DAVIS | Middlebury | Vimeo90K
---|---|---|---
| MS-SSIM ($\times 1\mathrm{e}{-2}$)$\uparrow$ | LPIPS ($\times 1\mathrm{e}{-2}$)$\downarrow$ | MS-SSIM ($\times 1\mathrm{e}{-2}$)$\uparrow$ | LPIPS ($\times 1\mathrm{e}{-2}$)$\downarrow$ | MS-SSIM ($\times 1\mathrm{e}{-2}$)$\uparrow$ | LPIPS ($\times 1\mathrm{e}{-2}$)$\downarrow$
| t+1 | t+3 | t+1 | t+3 | t+1 | t+3 | t+1 | t+3 | t+1 | t+1
External learning methods
DVF [20] | 68.61 | 55.47 | 23.23 | 34.22 | 83.98 | 65.54 | 13.57 | 25.70 | 92.11 | 7.73
DYAN [18] | 78.96 | 70.41 | 13.09 | 21.43 | 92.96 | 83.91 | 7.98 | 15.03 | N/A | N/A
Optimization methods
Ours | 83.26 | 73.85 | 11.40 | 18.21 | 94.49 | 87.96 | 6.07 | 10.82 | 96.75 | 3.59

Table 3: Evaluation on diverse datasets. Comparison with state-of-the-art methods on DAVIS, Middlebury, and Vimeo90K. |  |  |
---|---|---|---
|  |  |
|  |  |
GT | DVF [20] | DYAN [18] | Ours
Figure 4: Visual comparison on DAVIS and Middlebury datasets.

Baselines. All baselines are trained on the corresponding training set in
Cityscapes and KITTI. We categorize baselines into two types. One type is the
methods that take only RGB frames as input, such as PredNet [22], MCNet [42],
and DVF [20]. Another type is the methods that require some additional
information, such as semantic maps, instance maps, including Vid2vid [43],
Seg2vid [31], and FVS [45]. However, these assumptions restrict these methods
to be only applicable when this additional context is accessible, degrading
their potential generalization ability. We use Multi-scale Structural
Similarity Index Measure (MS-SSIM) [44] and LPIPS [48] as evaluation metrics.
Higher MS-SSIM and lower LPIPS indicate better performance.

Ground Truth | DVF [20] | Ours
---|---|---
|  |  |  |
|  |  |  |
$t+1$ | $t+1$ | $t+3$ | $t+1$ | $t+3$
Figure 5: Multi-frame prediction on Vimeo90K. Since Vimeo90K contains triplets
and we use two frames as input, there is no ground-truth corresponding to
$t+3$.

Input Ours Input Ours

Figure 6: The next frame prediction performance on diverse data. The left
image is the last observed frame, and we use red arrows to indicate the motion
from the input frames. The right image is the predicted next frame by our
method.

Quantitative results. Although our method does not use the training set in
Cityscapes and KITTI, our method can still produce outstanding results. As
shown in Table 2, our method can outperform RGB-based methods by a large
margin in both short-term and long-term video prediction. Our method improves
DVF [20] by 12.75%, 13.98%, 13.06% in MS-SSIM, 62.81%, 48.02%, 38.07% in of
LPIPS on the $t+1$, $t+3$, $t+5$ predictions. Moreover, compared with the
methods utilizing semantic or instance segmentation, our method can still
outperform FVS [45] by 6.11%, 7.09%, 6.24% in MS-SSIM on Cityscapes, and
33.21%, 17.55%, 13.58% in LPIPS on KITTI, for the $t+1$, $t+3$, $t+5$
predictions.

The quantitative results demonstrate that our method without external training
on the training set of Cityscapes and KITTI, can achieve better performance in
both short-term and long-term video prediction. This is because our method is
based on the powerful constraints from VFI: recent VFI models can produce
superior interpolation results that can be arguably treated as ground truth.
While other methods use handcrafted loss functions, these methods may overfit
to the major motion exhibited in the training set, the ”zooming-in” effect,
which is caused by the moving of the forwarding car, instead of learning the
real motion.

Qualitative Results. In Fig. 3, our method is compared to recent video
prediction methods, DVF [20] (RGB) and FVS [45] (RGB + semantic + instance).
DVF [20] tends to be dominated by “zooming-in” motion without predicting the
true motion, since the driving datasets are captured by a forward moving
camera. FVS [45] uses a handcrafted 2D affine transformation to approximate
the motion of moving cars. However, complex motion, including nonrigid
deformation and 3D rotation, can not be captured by 2D affine transformation.
Moreover, DVF and FVS rely on semantic and instance segmentation, and their
performance degrades when these assumptions do not hold. Visual comparisons on
Cityscapes are presented in the supplement.

###  4.2 Evaluation on Diverse Datasets

Since our optimization framework does not need external training, our method
can be generalized to any video at any resolution. Meanwhile, previous
external methods trained on dataset A may have performance degradation when
applied to dataset B, caused by the domain gap between A and B. To demonstrate
the generality of our method, we conduct a cross-dataset evaluation on diverse
datasets.

Datasets. We evaluate our methods on multiple datasets, including DAVIS [34],
Middlebury-Other [1], and Vimeo90K [47] datasets. DAVIS [34]: there are 30
sequences in the validation set with resolutions around $854\times 480$ .
Middlebury [1]: there are 10 videos with resolutions around $640\times 480$.
Vimeo90K [47]: there are 3782 triplets in the test set with a resolution of
$448\times 256$. By taking one clip every ten clips, we form the test set.

Components | SSIM$\uparrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$
---|---|---|---
Zero | 0.7719 | 23.79 | 0.1230
Noise | 0.7669 | 23.64 | 0.1228
Without $\mathcal{L}_{img}$ | 0.8939 | 28.86 | 0.0660
Without $\mathcal{L}_{cons}$ | 0.8732 | 28.43 | 0.1221
$\mathcal{L}_{img}$ with MSE | 0.8877 | 28.97 | 0.1232
With $\mathcal{L}_{interp}$ | 0.8963 | 28.87 | 0.0693
Long-term $\mathcal{L}_{img}$ | 0.6978 | 21.75 | 0.1657
W/o flow inpainting | 0.8882 | 28.94 | 0.1139
Full Model | 0.8975 | 29.10 | 0.0646

Table 4: The ablation study.

Baselines. We compare our method with two latest external methods, DVF [20]
and DYAN [18], which take only RGB frames as input, since they can be applied
to these datasets. Other methods that require additional assumptions cannot be
compared because their assumptions do not hold. We test these two models on
these datasets using their pretrained model on UCF101 [38].

Quantitative results. As exhibited in Table 3, our method outperforms DYAN
[18] by 12.90%, 15.0% in DAVIS, and by 23.97%, 28.00% on Middlebury in LPIPS
for $t+1$, $t+3$ predictions. Note that our method is still robust in long-
term prediction. On the Vimeo90K dataset, our method outperforms DVF [20] by
53.56% in LPIPS for next-frame prediction. Although these two baselines may
perform well on UCF-101, there is a domain gap problem between UCF-101 and
these test videos. This domain gap phenomenon commonly exists in video
prediction tasks, so most video prediction methods [22, 42, 20, 18, 31, 45, 4]
usually train a separate model for each dataset and even employ different
assumptions for each dataset. Differently, the domain gap in the VFI task does
not appear to be an issue. VFI methods [11, 33] can use one dataset for
training and can produce excellent results on various datasets. With the
powerful constraints provided by VFI, our method does not have the domain gap
problem (no external training): each sequence is independently optimized by
employing the FVI network as a constraint.

Qualitative results. As shown in Fig. 4 and Fig. 5, our method yields better
prediction results than baselines. The baselines produce distortion artifacts
around object boundary or fail at prediction when motion is complicated.
Meanwhile, our method can robustly predict future frames. As the Vimeo90K
dataset only provides three frames, we take the first two frames as input for
future frame prediction (there is only ground truth for the $t+1$ prediction).
Our method can produce high-quality prediction results in the long term, as
shown in Fig. 5. The intricate details, such as the hair of the woman, remain
clear in our results.

To demonstrate the generality of our method, we collect some real videos from
YouTube, such as the movie clips as shown in Fig. 6. We also present some
visual results on the BAIR Pushing [6] and Penn Action [49] datasets. Our
method can produce strong results on diverse data.

###  4.3 Ablation Study

We conduct an ablation study to demonstrate the importance of each component,
as shown in Table 4. Our ablation study is conducted on Cityscapes. We use the
performance of the next-frame prediction to evaluate different components.
Visual comparisons are presented in the supplement.

Initialization. If we set the optimization target as the prediction frame,
then the optimization cannot converge. Thus, we choose to optimize optical
flow, which eases the optimization process since pixels can be copied from
observed frames. If we initialize optical flow as zeros or Gaussian noise,
then our performance becomes worse. If we initialize the optical flow as the
copy of $f_{t\rightarrow t-1}$, then the performance is similar to our full
model. This demonstrates that good initialization helps the optimization to
converge.

Loss functions. We conduct experiments with loss functions with several
variants. Without $\mathcal{L}_{img}$: supervised only by
$\mathcal{L}_{cons}$. Without $\mathcal{L}_{cons}$: supervised only by
$\mathcal{L}_{img}$. $\mathcal{L}_{img}$ with MSE: using MSE loss rather than
$L_{1}$ loss in Eq. 8. With $\mathcal{L}_{interp}$: use the interpolation
result of VFI $I^{G}_{t}$ rather than the intermediate output $I^{G}_{t+1}$ in
Eq. 8. Long-term $\mathcal{L}_{img}$: setting the input frame length as 4 and
add a long-term constraint between $x_{t-3}$, $\tilde{x}_{t+1}$ and $x_{t-1}$.
The results show that the combination of $\mathcal{L}_{img}$ and
$\mathcal{L}_{cons}$ performs best. $\mathcal{L}_{img}$ provides a more direct
constraint than $\mathcal{L}_{interp}$. Long-term $\mathcal{L}_{img}$ has
performance degradation because when the motion is too large, the accuracy of
VFI decreases .

Flow inpainting. If we remove the optical flow inpainting procedure, then our
performance also drops because optical flow inpainting effectively corrects
invalid flow values.

Other pretrained VFI models. We also try utilizing Super SloMo [13] as our VFI
backbone and find this method also works for our framework. For the next frame
prediction on Cityscapes, the MS-SSIM for Super SloMo [13] and RIFE [11] is
0.9199 and 0.9454. Thus, we choose RIFE [11] as our VFI backbone.

Figure 7: Our model with different optimization iterations.

###  4.4 Convergence Analysis

As shown in Fig. 7, we conduct the convergence analysis on Cityscapes with a
resolution of 256 $\times$ 512 . In the first 400 iterations, the optimization
converges fast. After 400 iterations, the prediction result gradually
improves. The prediction result is still slowly improving after 3000
iterations.

##  5 Conclusion

We propose the first video prediction optimization method by casting the video
prediction problem as a VFI based optimization problem, which addresses the
domain gap issue in most video prediction methods. Our method can outperform
state-of-the-art methods and can be adapted to any video at any resolution.
Although our method relieves domain gap problem and presents impressive
performance, optimizing every frame by our method costs more time than other
external learning-based methods. The comparison between our and other methods
in terms of the model size and inference time is presented in the supplement.
As we observe, most run time in our model is spent on the gradient propagation
inside the VFI network [11], which inspires us to design a more efficient
backbone for acceleration in the future.

## References

  * [1] Simon Baker, Daniel Scharstein, J. P. Lewis, Stefan Roth, Michael J. Black, and Richard Szeliski.  A database and evaluation methodology for optical flow.  In ICCV, 2007.
  * [2] Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang.  Depth-aware video frame interpolation.  In CVPR, 2019.
  * [3] Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang.  MEMC-Net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement.  TPAMI, 43(3):933–948, Mar. 2021.
  * [4] Xinzhu Bei, Yanchao Yang, and Stefano Soatto.  Learning semantic-aware dynamics for video prediction.  In CVPR, 2021.
  * [5] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In CVPR, 2016.
  * [6] Chelsea Finn, Ian Goodfellow, and Sergey Levine.  Unsupervised learning for physical interaction through video prediction.  In NeurIPS, 2016.
  * [7] Hang Gao, Huazhe Xu, Qi-Zhi Cai, Ruth Wang, Fisher Yu, and Trevor Darrell.  Disentangling propagation and generation for video prediction.  In ICCV, 2019.
  * [8] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In CVPR, 2016.
  * [9] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  I. J. Robotics Res., 2013.
  * [10] Shurui Gui, Chaoyue Wang, Qihua Chen, and Dacheng Tao.  FeatureFlow: Robust video interpolation via structure-to-texture generation.  In CVPR, 2020.
  * [11] Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou.  RIFE: real-time intermediate flow estimation for video frame interpolation.  arXiv preprint arXiv:2011.06294, 2020.
  * [12] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  NIPS, 2015.
  * [13] Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik G. Learned-Miller, and Jan Kautz.  Super slomo: High quality estimation of multiple intermediate frames for video interpolation.  In CVPR, 2018.
  * [14] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [15] Wonkwang Lee, Whie Jung, Han Zhang, Ting Chen, Jing Yu Koh, Thomas E. Huang, Hyungsuk Yoon, Honglak Lee, and Seunghoon Hong.  Revisiting hierarchical approach for persistent long-term video prediction.  In ICLR, 2021.
  * [16] Chenyang Lei, Yazhou Xing, and Qifeng Chen.  Blind video temporal consistency via deep video prior.  In NeurIPS, 2020.
  * [17] Chenyang Lei, Yazhou Xing, Hao Ouyang, and Qifeng Chen.  Deep video prior for video consistency and propagation.  TPAMI, To Appear.
  * [18] Wenqian Liu, Abhishek Sharma, Octavia Camps, and Mario Sznaier.  Dyan: A dynamical atoms-based network for video prediction.  In ECCV, 2018.
  * [19] Yu-Lun Liu, Yi-Tung Liao, Yen-Yu Lin, and Yung-Yu Chuang.  Deep video frame interpolation using cyclic frame generation.  In AAAI, 2019.
  * [20] Ziwei Liu, Raymond Yeh, Yiming Liu Xiaoou Tang, , and Aseem Agarwala.  Video frame synthesis using deep voxel flow.  In ICCV, 2017.
  * [21] Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala.  Video frame synthesis using deep voxel flow.  In ICCV, 2017.
  * [22] William Lotter, Gabriel Kreiman, and David Cox.  Deep predictive coding networks for video prediction and unsupervised learning.  In ICLR, 2017.
  * [23] Pauline Luc, Natalia Neverova, Camille Couprie, Jacob Verbeek, and Yann LeCun.  Predicting deeper into the future of semantic segmentation.  In ICCV, 2017.
  * [24] Simone Meyer, Abdelaziz Djelouah, Brian McWilliams, Alexander Sorkine-Hornung, Markus Gross, and Christopher Schroers.  PhaseNet for video frame interpolation.  In CVPR, 2018.
  * [25] Simone Meyer, Oliver Wang, Henning Zimmer, Max Grosse, and Alexander Sorkine-Hornung.  Phase-based frame interpolation for video.  In CVPR, 2015.
  * [26] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In European conference on computer vision, pages 405–421. Springer, 2020.
  * [27] Simon Niklaus and Feng Liu.  Context-aware synthesis for video frame interpolation.  In CVPR, 2018.
  * [28] Simon Niklaus and Feng Liu.  Softmax splatting for video frame interpolation.  In CVPR, 2020.
  * [29] Simon Niklaus, Long Mai, and Feng Liu.  Video frame interpolation via adaptive convolution.  In CVPR, 2017.
  * [30] Simon Niklaus, Long Mai, and Feng Liu.  Video frame interpolation via adaptive separable convolution.  In ICCV, 2017.
  * [31] Junting Pan, Chengyu Wang, Xu Jia, Jing Shao, Lu Sheng, Junjie Yan, and Xiaogang Wang.  Video generation from single semantic label map.  In CVPR, 2019.
  * [32] Junheum Park, Keunsoo Ko, Chul Lee, and Chang-Su Kim.  BMBC: Bilateral motion estimation with bilateral cost volume for video interpolation.  In ECCV, 2020.
  * [33] Junheum Park, Chul Lee, and Chang-Su Kim.  Asymmetric bilateral motion estimation for video frame interpolation.  In ICCV, 2021.
  * [34] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbelaez, Alexander Sorkine-Hornung, and Luc Van Gool.  The 2017 DAVIS challenge on video object segmentation.  CoRR, abs/1704.00675, 2017.
  * [35] Xiaojuan Qi, Zhengzhe Liu, Qifeng Chen, and Jiaya Jia.  3d motion decomposition for RGBD future dynamic scene synthesis.  In CVPR, 2019.
  * [36] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli.  Singan: Learning a generative model from a single natural image.  In ICCV, 2019.
  * [37] Hyeonjun Sim, Jihyong Oh, and Munchurl Kim.  XVFI: Extreme video frame interpolation.  In ICCV, 2021.
  * [38] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  UCF101: A dataset of 101 human actions classes from videos in the wild.  CoRR, abs/1212.0402, 2012.
  * [39] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov.  Unsupervised learning of video representations using lstms.  In ICML, 2015.
  * [40] Zachary Teed and Jia Deng.  RAFT: recurrent all-pairs field transforms for optical flow.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV, 2020.
  * [41] Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V. Le, and Honglak Lee.  High fidelity video prediction with large stochastic recurrent neural networks.  In NeurIPS, 2018.
  * [42] Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee.  Decomposing motion and content for natural video sequence prediction.  In ICLR, 2017.
  * [43] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  Video-to-video synthesis.  In NeurIPS, 2018.
  * [44] Zhou Wang, Eero P. Simoncelli, and Alan C. Bovik.  Multiscale structural similarity for image quality assessment.  In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pages 1398–1402. IEEE, 2003.
  * [45] Yue Wu, Rongrong Gao, Jaesik Park, and Qifeng Chen.  Future video synthesis with object motion prediction.  In CVPR, 2020.
  * [46] Xiangyu Xu, Li Siyao, Wenxiu Sun, Qian Yin, and Ming-Hsuan Yang.  Quadratic video interpolation.  In NeurIPS, 2019.
  * [47] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman.  Video enhancement with task-oriented flow.  International Journal of Computer Vision (IJCV), 127(8):1106–1125, 2019.
  * [48] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [49] Weiyu Zhang, Menglong Zhu, and Konstantinos Derpanis.  From actemes to action: A strongly-supervised representation for detailed action understanding.  In ICCV, 2013.


11institutetext: Aarhus University, Aarhus, Denmark

# Learning by Design: Structuring and Documenting the Human Choices in Machine
Learning Development

Simon Aagaard Enni  0000-0002-1544-4371 Ira Assent  0000-0002-1091-9948

###### Abstract

The influence of machine learning (ML) is quickly spreading, and a number of
recent technological innovations have applied ML as a central technology.
However, ML development still requires a substantial amount of human expertise
to be successful. The deliberation and expert judgment applied during ML
development cannot be revisited or scrutinized if not properly documented, and
this hinders the further adoption of ML technologies–especially in safety
critical situations.

In this paper, we present a method consisting of eight design questions, that
outline the deliberation and normative choices going into creating a ML model.
Our method affords several benefits, such as supporting critical assessment
through methodological transparency, aiding in model debugging, and anchoring
model explanations by committing to a pre hoc expectation of the model’s
behavior. We believe that our method can help ML practitioners structure and
justify their choices and assumptions when developing ML models, and that it
can help bridge a gap between those inside and outside the ML field in
understanding how and why ML models are designed and developed the way they
are.

###### Keywords:

Machine learning Documentation Development Methodology

##  1 Introduction

Machine learning (ML) has led to some of the most influential innovations in
contemporary computing even leading some to categorize ML as the beginning of
a new data-driven industrial and scientific revolution [23]. With so much at
stake, critical discussions about the proper role of ML in society and about
the safety and ethics of ML technologies emerge [7, 18, 33, 35, 43]. In ML, a
machine learner is developed using carefully chosen biases and data to search
for a ML model, that performs some desired function, such as classification,
rare-event detection, or clustering given previously unseen input data. While
a ML model functions mostly automatically once learned, the process of
creating an effective machine learner requires a lot of human labor and
deliberation [8]. The main benefit of recognizing this indirect human
influence is in making explicit the deliberation and design going into
creating ML models, which allows us to systematize ML development,
documentation, critical investigations, maintenance and revision. Recent
critical discussions have revealed a serious need for such systematization, as
critics liken the current state of ML to alchemy, based on ”a substantial
amount of ’black art’” and ”folklore and magic spells” [5]. The discourse
surrounding ML as somehow magical and unknowable, especially coming from
insiders in the field, is a risk to the professional and scientific integrity
of the ML field and therefore should be taken seriously to avoid myth-making,
unsubstantiated hype, and the inevitable resulting disappointment [34]. In the
interest of the scientific and professional integrity of the ML field, we
present a design method that illustrates the different kinds of design choices
made by the developers of machine learners and associated design questions
that guide these choices. In this way, we argue that ML models are not just
learned, but also designed, albeit indirectly.

Chief among the benefits afforded by our method is that it makes the process
of creating machine learners more transparent, explicit, and interpretable,
supporting critical assessment by noting design choices and their rationales
in humanly readable form. It furthermore creates a documentation of the
intended behavior of the ML model that serves as a pre hoc explanation and
justification, similar to the preregistration of an experimental setup [19].
This description can be used to anchor explanations of model behavior later on
in the process and enables separating errors in implementation from errors in
the design of the model, thereby aiding with debugging. At this level of
abstraction, ML models can be approached by domain experts as well as ML
specialists, allowing for tighter collaboration and discussion.

As new legislation and research moves away from a reactive ’informed
consent’-based paradigm towards protection by design, as outlined in article
25 of the GDPR,111Article 25 can be read at https://gdpr-info.eu/art-25-gdpr/
\- last accessed on 02-11-2020. our method can furthermore be used to
structure and document such protective design. Privacy by design [26],
fairness by design [2, 10], and accountability by design [25, 3] are all
examples of this movement which could benefit from our method. Such approaches
attempt to ensure that models are designed with values in mind that are
aligned with those of society more broadly, illustrating the necessity of a
structured and explicit design strategy for ML models.

##  2 Related work

There have been attempts to organize the workflow of ML development and the
related discipline of data mining, that has resulted in process models [37,
41, 12, 1, 20]. Our contribution in this paper differs from those by focusing
not on how to organize a team of ML developers, or on outlining the different
phases of development for ML systems, but instead on documenting, justifying,
and deliberating on the model- and system-specific design choices made in ML
development. Our model distinguishes different technical design choices and
their associated design questions; regardless of workflow and makes different
delineations as a result.

The work of Yoram Reich in applying ML to civil engineering problems deserves
a special mention [37]. While his method also focuses on ML modeling, it has a
different application and focus. First off, Reich applies ML model to solve
civil engineering problems, and thereby his analysis focuses on matching ML
methods to engineering problems. Our approach is instead created as an aid in
documenting and deliberating on the design of and assumptions made when
creating general ML models.

The field of software testing has recently seen an upsurge of research in ML
testing with the purpose of detecting ML ”bugs”, defined as ”the differences
between existing and required behaviors of an ML system” [44]. In a related
vein, explainable AI (xAI) has also seen a resurgence in interest, among other
things as a common-sense driven way of evaluating the reasonableness of ML
models’ behavior through manual inspection [30, 9]. Our contributions are
complementary to both these fields, as the documentation provided through our
method can aid both in identifying the required behaviors of an ML system for
the ML testing field and in producing pre hoc explanations for systems that
can contextualize later post hoc explanations from xAI.

Mitchell et al. recently presented a principled documentation technique for ML
models in the form of Model Cards [31]. Models Cards are focused on
documenting facts about finished models, in particular performance and
fairness properties, whereas our model is made to document and guide the
design of the model as it is being developed. However, the documentation
derived from our method can be added to a Model Card, making the methods
complementary.

##  3 Approaching a Problem with ML: Pitfalls and Promises

Perhaps the greatest strength of ML is that it enables programming-by-example,
allowing for models to be produced for tasks where algorithmic solutions
either do not exist or are insufficient for practical use. Instead, a
description of the task is given along with examples demonstrating the
behavior that the model should learn to mimic for the given task. T. Mitchell
thus presented this canonical definition of ML:

> A computer program is said to learn from experience $E$ with respect to some
> class of tasks $T$ and performance measure $P$, if its performance at tasks
> in $T$, as measured by $P$, improves with experience $E$. [32, p. 2]

For learning to be possible, $T$, $E$, and $P$ must be carefully designed and
quantified such that the abstract pedagogical issue of ’learning’ is reduced
to the concrete problem of iteratively and quantitatively improving a measure,
$P$, with more experience, $E$, i.e., an optimization problem. It is here we
see the necessity of the design aspect of ML; it is through design that the
specifics of the task are described and made ”learnable.”

In the following we make an important distinction between a machine learner
and a ML model. A machine learner is a program implemented and designed by a
human programmer that produces a ML model through a process of learning from
training data. It is the design of the machine learner that we are concerned
with in this paper, as this design indirectly determines the functionality and
behavior of the ML model.

However, solving problems by learning solutions introduces pitfalls as the
task that the machine learner is meant to learn a model for is typically
under-specified. This is a natural result of trying to learn a general model
from examples and has to be handled by carefully chosen assumptions. Poor
choices here might lead to the resulting ML model being misaligned with the
intention of the developers in a number of ways; such as by displaying a high
rate of error, making socially discriminatory decisions, or relying on
unstable or spurious correlations. However, as mentioned previously,
assumptions in ML are often poorly documented, instead deferring to
professional hunches, best practices, and ”black art” when making the
decisions. In Section 4, we will present a model outlining the different
design choices made when designing a machine learner, which can help both
those inside and outside the ML field understand and communicate these
choices, shedding light on the ”black art” of ML.

##  4 The indirect design of machine learning models

In this section, we present a method outlining eight different categories of
design choices that are made in the development of a machine learner, five of
which spell out the technical make-up of the learner itself. Each of these
represent a response to a particular question about the desired functionality
of the ML model and can be seen in Figure 1. The remainder of this section
will cover each design choice in turn. In order to make our points more
concrete, we will introduce a running example of a fictitious ML system
designed for object detection where a design choice is made and justified in
each section. We choose to model this example on the well-documented AlexNet
[24], but we stress that our example is fictitious, and that the example
motivations and deliberations for the individual design choices are our own
for purely illustrative purposes. The two right-most columns in Figure 1
display an overview of the choices and their justifications in this example.

Figure 1: An illustration of the different design choices made in the creation
of a ML model and the design questions they are responses to. The first two
columns on the left outline the design questions and the two columns on the
right show the particular choices in our running example with corresponding
justifications.

###  4.1 High-level objective

> What is the model meant to achieve?

The high-level objective is the task that the ML model is meant to perform
once completed, and it motivates the rest of the design choices. This
objective is not necessarily meant to be a technical description, but rather a
declaration of the intent of the developers in designing the ML model. It can
be abstract, such as detecting objects in images or it could be very specific,
such as finding anomalies in the water consumption data of a specific
supermarket chain to detect faults and reduce water consumption [42]. The
high-level objective has to be specific enough to support decision making in
other sections towards achieving this goal. An objective such as ”detecting
objects in images” may be too vague in this regard, as it is not even clear
which kind of objects should be detected in which kind of images, let alone
for which purpose these objects should be detected. The anomaly example above
[42] is more specific and lends itself well to documenting a ML design
process. Since the kind of data and its source is known and the purpose of the
finished system is already declared, a lot of the following design choices can
be clearly motivated. While it is not known in advance what constitutes an
anomaly in the data, it is known what these anomalies would be used for, so
the search can be focused on those anomalies that, e.g., can be related to
faults and excessive water consumption.

Ideally the objective should specify (1) what kinds of data the model is
expected to work with, (2) what task the model is expected to perform, (3) why
the model is made for that task, and (4) which context the model is expected
to be deployed in. While every ML project is motivated by some high-level
objective, such objectives often change and are iteratively improved as the
project progresses and the feasibility of the objective is tested [36].

In our running example, we choose to design a ML model for detecting common
objects in pictures in order to classify and categorize pictures from social
media based on their contents. The model is thus expected to work with digital
photographs of varying resolution, and we would expect that the model should
be deployed on personal computers to help regular users of social media.

###  4.2 Training data

> What are good examples of desired behavior?

Ideally, the training data will supply sufficiently diverse examples to
eliminate any other models than the ones that work as intended. Furthermore,
in order to successfully generalize to new situations, a lot of data are often
needed, as the confidence and accuracy of model predictions tend to improve
with more data [32, 17]. High diversity eliminates models built on spurious
patterns and large amounts of data allow for better statistical bounds on the
expected error of the resulting model. Training data are chosen to exemplify
desired behavior in the learned model, and the resulting model may become
misaligned if the data are not sufficiently representative of this behavior.

There has long been awareness of the dangers of poor data [22], and the adage
”garbage in, garbage out” is often used to describe the futility of doing good
analysis with bad data. The data are not just meant to be free from errors,
but also to clearly illustrate the task that the ML model is meant to learn.
Explicitly asking ”what are good examples of desired behavior” as part of the
design process encourages reflection and enables critical scrutiny and
stakeholder inquiries. The chosen training data are contextualized as
deliberately selected information with an explicit purpose.

In our running example, we choose to use the ImageNet data set [39], as it
constitutes one of the largest collections of pictures tagged with the objects
depicted in them. ImageNet contains millions of pictures scraped from
different websites classified into hundreds of categories, and we expect these
pictures to be representative of what the model would encounter when deployed
in practice. We furthermore augment the data by introducing horizontal
reflections, small translations, and variations to color intensity, as we
expect the class of each picture to be invariant to such transformations, and
such augmentations are expected to reduce overfitting [24].

###  4.3 Data representation

> What are the relevant and irrelevant features of the chosen examples?

Once the training data are chosen they have to be represented in a way that
facilitates learning. This representation serves the purpose of highlighting
what information is relevant to the task and removing distractions. Often, the
data used in ML are represented as feature vectors via. feature engineering
[8, 29] where a series of quantified vector representations are tested through
a process of trial and error. However, the data may be represented in any
number of ways that the developer believes illustrates their important aspects
well, such as tensors for image and video data which preserve the locality of
pixel data, or graphs for network data, which preserve and highlight the
structure of the analyzed network.

A well-designed data representation excludes irrelevant information while
retaining as much relevant information as possible. However, what counts as
relevant and irrelevant information is not defined a priori. Recent advances
in representation learning have partially automated this process, when the
relevant information can be described with more abstract biases [29]. However,
it is important to carefully consider which features would facilitate the
learning process and which would hinder it. If important information is
removed by mistake, or if too much irrelevant information drowns out the
relevant bits, the model may learn spurious correlations between irrelevant
features. Furthermore, choices of model type and data set naturally constrain
what kinds of data representations are feasible for the particular task, as
the chosen representation has to fit them both.

In our running example, choice of data and model type indeed impacts
representation. We choose to represent the data as tensors with 256x256 pixels
and 3 color channels owing to two primary considerations. First, we use a
convolutional neural network (CNN) for classification, and such models require
input of a fixed dimensionality. Second, the tensor representation preserves
the relative location of pixels, which we expect to be beneficial to detecting
objects that extend over adjacent pixels.

###  4.4 Model type and architecture

> Which types of patterns is the model expected to use and how complex are
> they?

For any data set there are many potential models consistent with the data, and
choosing any one model over the rest requires specifying an a priori
preference in the form of an inductive bias [32]. With a bad inductive bias,
the chosen model might generalize in ways that are not aligned with the
interests of the developers even if they are consistent with the training
data. Such misalignment might show itself merely as bad performance, but it
may also be more subtle and problematic such as relying on unstable or
spurious correlations [4] or making socially discriminatory generalizations
[2]. The first part of the inductive bias is given by the model type and
architecture, and the second part is discussed in Section 4.5.

Choosing a good model type comes down to reasoning about the nature of the
relation between input and output in the chosen task. While any ML project
implicitly assumes the existence of some statistical regularity governing the
relation between input and output, choosing a particular model type biases the
learner towards learning specific kinds of patterns such as those that can be
represented by a weighted sum of the feature values in the case of a linear
model. Assumptions of linearity may not hold for many problems and often more
advanced models with subtler biases are used to represent complex dependencies
between input and output. Increased complexity comes at a cost, however, both
with respect to the amount of data required for successful learning and the
number of computations needed at each step in the learning process [32, 17].
Thus, the computational budget available might become a deciding factor in the
choice of model type and architecture, and wherever this is the case, it is
important to document such decisions, so that they may be reevaluated if the
availability of computational resources change.

A poor choice of model type and architecture may result in the model being
misaligned with the developers’ intentions if important patterns in the input-
output relationship cannot be captured by the model. If sequential order of
the data is paramount, such as in speech recognition, but a model type with an
assumption of no such sequences in the data is used, the model will most
likely fail to capture anything but weak and spurious patterns. This shows the
importance of careful design, as a good choice of model type and architecture
depends strongly on what is known about the relation between input and output
and which patterns to expect in the data. Documenting the reasoning behind the
model design by answering which patterns are expected in the data and which
structure the model is expected to exploit allows later studies to critically
scrutinize this fundamental aspect. For example, CNNs trained on ImageNet were
investigated on account of the assumption that they learn hierarchical
representations of increasingly complex shapes. Instead, experiments exposed a
texture bias in the models that undermined this assumption [14].

In our example, we choose a CNN with 5 convolution layers designed to detect a
hierarchical representation of pixel patterns as we expect that many features
central to object recognition such as compact shapes and textures are defined
by correlations in neighboring pixels. These layers are followed by 3 fully
connected layers to learn a classification function from the features detected
by the convolutional layers. ReLU activations are used throughout, as these
have been shown, e.g., to avoid exploding gradients [24, 15].

###  4.5 Optimization objective

> How are competing models compared?

While the chosen model type and architecture determine which general patterns
are expected, constituting the first part of the inductive bias, the
optimization objective determines how different parameterizations of the same
model are compared, constituting the second part of the inductive bias.
Choosing the optimization objective translates learning into a problem, where
each parameterization of the model maps to an associated value. In other
words, it determines how different instantiations of the same model can be
ranked and thereby it determines what is considered a successful model. In
order for the translation from abstract task to concrete objective to be
successful, the optimization objective should account for the different kinds
of errors the model can make and their relative values as well as any
constraints the parameters of the model are subject to. However, as in the
choice of model type, the optimization objective also has a computational
aspect to it.

It is worth noting that the optimization objective can account for more things
than just predictive power, e.g., classifiers can be optimized for increased
fairness [11], or they can be optimized to shift attention to patterns
considered more sensible [38]. If important goals or constraints are not
accounted for in the optimization objective, the learner might output a model
that is inconsistent with those.

In the running example we choose a multinomial regression objective with
weight decay regularization. With this choice the model optimizes its chances
of making the correct predictions across the training set, and the
regularization reduces overfitting due to the large number of parameters in
the model [17].

###  4.6 Learning algorithm

> How is the search for models performed?

Where the optimization objective defines criteria to evaluate the relative
value of model parameterizations, the learning algorithm decides how the
search for the optimal model is performed. It does not, however, imply any
behavior for the model in its own right. The learning algorithm is evaluated
on its efficiency and effectiveness–the parameters of the optimal model do not
change based on the method used to arrive at them. Two different learning
algorithms might still arrive at parameterizations with different performance
and at a different computational cost, though, as there are often no guarantee
that the learning algorithm arrives at the single global optimum.

However, since the efficiency of the learning algorithm is an important factor
in deciding the computational cost of training a ML model, it is not uncommon
to revisit other design choices to make them compatible with a powerful
learning algorithm, even if they represent the task less accurately as a
result. This is the case, e.g., for gradient descent where the loss-function,
and thereby also the model’s prediction function, must be differentiable,
which also constrains the input data to be a vector, matrix, or tensor of
fixed dimensionality. An example of compromise could be choosing a convex
optimization objective that less accurately represents the targeted problem,
but which guarantees that gradient descent converges towards the global
optimum. By explicitly noting such assumptions and considerations going into
the choice of a particular learning algorithm it can easily be revisited and
adjusted if necessary.

In the running example we choose to train the network using stochastic
gradient descent with momentum, since this method has previously shown good
performance on similar CNN models and other deep neural networks [40]. We add
dropout [21] to the first two fully connected layers to further reduce the
risk of overfitting by forcing neurons to learn to detect features that do not
require the presence of particular other neurons [24].

###  4.7 Output

> How should the output be presented?

In order to perform its task, the model’s output often has to be changed to
some useful format. The model type and/or optimization objective often
requires a continuous output from the model to facilitate optimization, and
this output can be interpreted, e.g., as the probability of seeing the
positive class, in the case of a logistic regression [17]. However, what is
needed from a model is often classification rather than a distribution of
probability over classes, and the output must be post-processed to achieve
this functionality. In other cases the numeric output might have to be changed
to, e.g., a risk scale of low, medium and high [28].

Thus a choice has to be made as to how the output is presented to the user and
while it does not affect the behavior of the model directly, it might have
important implications for its functionality. In particular it affects how the
system can be tested, as the form of the output might decide which outputs are
errors and which are not.

In our running example we make a simple modification by outputting the class
with the highest predicted probability as the predicted class. We make this
change because the system is meant to be used for classifying images, and this
requires returning a single predicted class for each input.

###  4.8 Performance evaluation

> How is the performance of the model evaluated in its actual context of use?

The final choice in creating a ML model is the evaluation scheme gauging the
performance of the trained model. Evaluating a ML model on held-back
validation data often informs and guides the design of its next iteration, and
thereby directly affects the design of the model [17, 27]. However, the
importance of a proper experimental design is often underappreciated,
especially when experimental designs are inherited from previous projects
[13].

Two important aspects must be be given particular attention: the quality of
the test data and the choice of evaluation metric. Ideally, the testing data
would be an unbiased and representative sample of the data the model is
expected to encounter in practice to get as good of an estimate of performance
as possible [32]. Thus, getting a testing data set of high quality requires
intimate knowledge of the problem that the model is made to address to account
for label skews, biases, concept drift and more. To ensure that all such
potential problems are accounted for it may be necessary to measure test data
oneself, if budget and practical considerations allow for it.

The second issue can become similarly subtle, as choosing a good evaluation
metric depends on the intended use-case of the learned model [13]. Knowing how
and for what the model is going to be used is necessary to choose a measure
that makes sense for this purpose. The performance of a ML model might also
have several qualitatively different facets that cannot be meaningfully
aggregated into a single measure of performance such as recall and precision.
Another example of this problem is when a model is required to be ”fair”,
which requires choosing and justifying a particular measure of fairness from a
vast population of fairness measures [6].

In the running example, we measure the top-1 accuracy, to test the efficiency
of the classification rule chosen in the output presentation, and the top-5
accuracy to test how often the model is close to being right, both measured on
held-back testing data. We will also manually inspect a random sample of the
top-5 results in order to see which specific classes the model has trouble
distinguishing. These measures relate to the high-level objective as we wish
to know which picture classes will likely contain the most errors and if the
classification could improve if we return more than one label per picture.

##  5 Conclusion and discussion

We presented a method consisting of eight design questions, the answers to
which serve to describe the desired behavior of the ML model learned by a
machine learner. This method can shed light on the deliberation and the
normative choices that are made when constructing machine learners. By
elevating the design process to the level of abstraction concerning the
properties of the input-output relation modeled by the ML model rather than
the level of technical and mathematical specifics, this method might make ML
more approachable to actors both inside and outside the ML field, thereby
supporting a higher degree of collaboration and discussion.

By noting the design choices and their rationales, the desired ML model is
described in humanly readable terms and ML developers thereby produce a pre
hoc explanation of the expected behavior and justification of the model,
similar to the preregistration of a scientific experimental setup [19]. This
description can help making expectations concrete and tangible and can anchor
later post hoc explanations [16] of model behavior. Describing the desired
model by iteratively refining answers to the eight design questions, generates
a stated justification which can act as an anchor for discussions and
maintenance. It thereby gives a lens through which to explain, debug, or
contest the model, opening up a discussion not just of the effectiveness of
these choices but also of their appropriateness.

## References

  * [1] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: Proc. ICSE-SEIP ’19. pp. 291–300. IEEE (2019). https://doi.org/10.1109/ICSE-SEIP.2019.00042
  * [2] Barocas, S., Selbst, A.D.: Big data’s disparate impact. California Law Review 104 (2016)
  * [3] Berendt, B., Preibusch, S.: Toward accountable discrimination-aware data mining: the importance of keeping the human in the loop - and under the looking glass. Big data 5(2), 135–152 (2017). https://doi.org/10.1089/big.2016.0055
  * [4] Calude, C.S., Longo, G.: The deluge of spurious correlations in big data. Foundations of science 22(3), 595–612 (2017). https://doi.org/10.1007/s10699-016-9489-4
  * [5] Campolo, A., Crawford, K.: Enchanted determinism: Power without responsibility in artificial intelligence. ESTS 6, 1–19 (2020). https://doi.org/10.17351/ests2020.277
  * [6] Corbett-Davies, S., Goel, S.: The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023 (2018)
  * [7] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016). https://doi.org/10.1007/s13347-015-0211-1
  * [8] Domingos, P.M.: A few useful things to know about machine learning. Commun. ACM 55(10), 78–87 (2012). https://doi.org/10.1145/2347736.2347755
  * [9] Doshi-Velez, F., Kim, B.: Considerations for evaluation and generalization in interpretable machine learning. In: Explainable and Interpretable Models in Computer Vision and Machine Learning, pp. 3–17. Springer (2018). https://doi.org/10.1007/978-3-319-98131-4_1
  * [10] Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. In: Proceedings of the 3rd innovations in theoretical computer science conference. pp. 214–226 (2012)
  * [11] Enni, S., Assent, I.: Using balancing terms to avoid discrimination in classification. In: Proc IEEE Int Conf Data Min. pp. 947–952. IEEE (2018). https://doi.org/10.1109/ICDM.2018.00116
  * [12] Fayyad, U., Piatetsky-Shapiro, G., Smyth, P.: The kdd process for extracting useful knowledge from volumes of data. Commun. ACM 39(11), 27–34 (nov 1996)
  * [13] Flach, P.: Performance evaluation in machine learning: The good, the bad, the ugly, and the way forward. In: Proc. AAAI-19. vol. 33, pp. 9808–9814 (2019)
  * [14] Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., Brendel, W.: Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. vol. abs/1811.12231 (2018), http://arxiv.org/abs/1811.12231
  * [15] Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. Proceedings of Machine Learning Research, vol. 15, pp. 315–323. JMLR Workshop and Conference Proceedings, Fort Lauderdale, FL, USA (11–13 Apr 2011)
  * [16] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM computing surveys (CSUR) 51(5), 93 (2018)
  * [17] Hastie, T., Tibshirani, R., Friedman, J.: The elements of statistical learning: data mining, inference, and prediction, Springer Series in Statistics. Springer New York (2009)
  * [18] Hildebrandt, M.: The new imbroglio - living with machine algorithms. The Art of Ethics in the Information Society (2016). https://doi.org/10.5117/9789462984493
  * [19] Hildebrandt, M.: Privacy as protection of the incomputable self: From agnostic to agonistic machine learning. Forthcoming in Theoretical Inquiries of Law 19(1) (2019)
  * [20] Hill, C., Bellamy, R., Erickson, T., Burnett, M.: Trials and tribulations of developers of intelligent systems: A field study. In: Proc. VL/HCC. pp. 162–170. IEEE (2016)
  * [21] Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 (2012)
  * [22] Kim, W., Choi, B.J., Hong, E.K., Kim, S.K., Lee, D.: A taxonomy of dirty data. Data mining and knowledge discovery 7(1), 81–99 (2003). https://doi.org/10.1023/A:1021564703268
  * [23] Kitchin, R.: Big data, new epistemologies and paradigm shifts. Big data & society 1(1) (2014). https://doi.org/10.1177/2053951714528481
  * [24] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Adv. Neural Inf. Process Syst. pp. 1097–1105 (2012)
  * [25] Kroll, J.A., Barocas, S., Felten, E.W., Reidenberg, J.R., Robinson, D.G., Yu, H.: Accountable algorithms. U. Pa. L. Rev. 165, 633 (2017)
  * [26] Langheinrich, M.: Privacy by design - principles of privacy-aware ubiquitous systems. In: International conference on Ubiquitous Computing. pp. 273–291. Springer (2001)
  * [27] Langley, P.: The changing science of machine learning. Machine Learning 82(3), 275–279 (2011). https://doi.org/10.1007/s10994-011-5242-y
  * [28] Larson, J., et al.: How we analyzed the compas recidivism algorithm (2016), https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm [Online; posted May-23-2016]
  * [29] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature 521(7553), 436 (2015)
  * [30] Lipton, Z.C.: The mythos of model interpretability. Queue 16(3), 31–57 (2018)
  * [31] Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I.D., Gebru, T.: Model cards for model reporting. In: Proc FAT*. pp. 220–229 (2019)
  * [32] Mitchell, T.M.: Machine Learning. McGraw-Hill, Inc., New York, NY, USA, 1 edn. (1997)
  * [33] Mittelstadt, B.D., Allo, P., Taddeo, M., Wachter, S., Floridi, L.: The ethics of algorithms: Mapping the debate. Big Data & Society 3(2) (2016). https://doi.org/10.1177/2053951716679679
  * [34] Moss, E., Schüür, F.: How modes of myth-making affect the particulars of ds/ml adoption in industry. In: Ethnographic Praxis in Industry Conference Proceedings. vol. 2018, pp. 264–280. Wiley Online Library (2018). https://doi.org/10.1111/1559-8918.2018.01207
  * [35] O’Neil, C.: Weapons of math destruction: How big data increases inequality and threatens democracy. Broadway Books (2017)
  * [36] Passi, S., Sengers, P.: Making data science systems work. Big Data & Society 7(2) (2020). https://doi.org/10.1177/2053951720939605
  * [37] Reich, Y.: Machine learning techniques for civil engineering problems. Computer-Aided Civil and Infrastructure Engineering 12(4), 295–310 (1997)
  * [38] Ross, A.S., Hughes, M.C.H., Doshi-Velez, F.: Right for the right reasons: Training differentiable models by constraining their explanations. In: Proc. IJCAI-17. pp. 2662–2670 (2017)
  * [39] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. IJCV 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
  * [40] Schmidhuber, J.: Deep learning in neural networks: An overview. Neural networks 61, 85–117 (2015). https://doi.org/10.1016/j.neunet.2014.09.003
  * [41] Shearer, C.: The crisp-dm model: the new blueprint for data mining. Journal of data warehousing 5(4), 13–22 (2000)
  * [42] Vercruyssen, V., Meert, W., Verbruggen, G., Maes, K., Baumer, R., Davis, J.: Semi-supervised anomaly detection with an application to water analytics. In: Proc IEEE Int Conf Data Min. pp. 527–536 (2018). https://doi.org/10.1109/ICDM.2018.00068
  * [43] Zarsky, T.Z.: Automated prediction: Perception, law, and policy. Communications of the ACM 55(9), 33–35 (2012). https://doi.org/10.1145/2330667.2330678
  * [44] Zhang, J.M., Harman, M., Ma, L., Liu, Y.: Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering (2020)