# Weighted multi-level deep learning analysis and framework for processing
breast cancer WSIs

Peter Bokor  Lukas Hudec  Ondrej Fabian  Wanda Benesova We gratefully
acknowledge the support of Siemens Healthineers Slovakia. The concepts and
information presented in this paper are based on research, and they are not
commercially available.Peter Bokor (e-mail: bokorpet98@gmail.com), Lukas Hudec
(e-mail<EMAIL_ADDRESS>corresponding author) and Wanda Benesova
(e-mail<EMAIL_ADDRESS>are with the Slovak University of
Technology, Faculty of Informatics and Information Technologies, Institute of
Computer Engineering and Applied Informatics, Ilkovicova 2, Bratislava,
Slovakia, 84216Ondrej Fabian (e-mail<EMAIL_ADDRESS>is with the Institute for
Clinical and Experimental Medicine, Clinical and Transplant Pathology Centre,
Videnska 1958/9, Prague 4, Czech Republic, 140 21 and the Charles University
and Thomayer University Hospital, 3rd Faculty of Medicine, Department of
Pathology and Molecular medicine, Videnska 800, Prague 4, Czech Republic, 140
59

###### Abstract

Prevention and early diagnosis of breast cancer (BC) is an essential
prerequisite for the selection of proper treatment. The substantial pressure
due to the increase of demand for faster and more precise diagnostic results
drives for automatic solutions. In the past decade, deep learning techniques
have demonstrated their power over several domains, and Computer-Aided (CAD)
diagnostic became one of them. However, when it comes to the analysis of Whole
Slide Images (WSIs), most of the existing works compute predictions from
levels independently. This is, however, in contrast to the histopathologist
expert approach who requires to see a global architecture of tissue structures
important in BC classification.

We present a deep learning-based solution and framework for processing WSIs
based on a novel approach utilizing the advantages of image levels. We apply
the weighing of information extracted from several levels into the final
classification of the malignancy. Our results demonstrate the profitability of
global information with an increase of accuracy from 72.2% to 84.8%.

Breast Cancer, Whole Slide Images, Voted Classification, Pyramid Approach,
Deep Learning, Medical Imaging

##  1 Introduction

Breast cancer is the most common form of cancer among women. In 2020, there
were 2.3 million women diagnosed and 680 thousand died globally of breast
cancer. After the last 5 years of diagnosing breast cancer, at the end of
2020, there were 7.8 million women alive and treated with this disease. Half
of the cases, there is no proven significant indicator that could have
predicted the risk of breast cancer other than age and gender [1]. Most of the
developed countries were able to increase the number of survival by prevention
and early detection. An essential part of the diagnostic process is the
microscopic analysis of tissue extracted by biopsy and stained by hematoxylin
and eosin, the dye that highlights cells from the stroma. A pathologist then
states a histopathological diagnosis of breast cancer by recognizing the
glandular epithelial structures. Unfortunately, this manual process is time-
consuming and difficult due to the variance of stained data, similarity of
benign and malignant structures, the vast size of histopathology slides, and
other environmental influences. The psychological pressure, demand for fast
diagnosis, a large number of postponed examinations, especially in the post-
covid era, all lead to an increase in demand for an unbiased histopathology
analysis support system. It should be free of environmental influence,
capable, reliable, and robust enough to assist pathologists with their
analyses.

We propose an end-to-end 3 steps solution for the automatic processing of
Whole Slide Images (WSI). The first step is an effective preprocessing of the
WSI images with our proposed framework. The further steps are based on using a
sequence of neural network models trained for specific tasks which we call
Experts. The last step using one of the mentioned Experts is a novel
architecture weighing the influence of different microscopic magnification
levels into final classification. An overview of our proposed framework is
shown in Figure 1.

We design a module, WSI Analyzer, that processes Whole Slide Images and
manages their organization. It matches annotation to the WSI, and
hierarchically organizes the data. WSIs processed by WSI Analyzer are
subsequently used by another designed module - WSI Generator which produces
large sets of data based on multiple parameters.

WSI Analyzer can be also used to produce a completely new analysis, using an
Expert. The Expert then applies their knowledge to a new WSI which results in
analysis. The produced analysis contains information about the predicted
diagnosis that doctors can use as a helping tool or diagnostic support.

We propose several Experts using binary and multi-class classification to
generate sufficient and precise diagnostic predictions. We propose an Expert
with a novel domain approach for the multi-class classification, that brings
attention to different levels of magnification for recognition of different
tumor types. All of our experiments are performed with a publicly available
dataset BACH ICIAR 2018.

We evaluate our approach quantitatively and empirically. We compare it against
state-of-the-art methods in the domain and each other, with a focus on the
enhancement.

Our work and contributions include:

  1. 1.

Complex analysis of breast tissue WSIs using multiple deep learning models,
taking into account contextual and detailed information, simulating analysis
approach of a histopathologist.

  2. 2.

A deep learning model with a special architecture capable of enhancing multi-
class classification prediction results.

  3. 3.

An automatic universal framework for processing large datasets and large data
that are natural in form of WSIs. The framework is designed to speed-up the
extraction of necessary data to train deep learning models to achieve state-
of-the-art results.

Figure 1: Overview of our WSI processing system - Horizontal flow = training,
vertical flow = analysis of new WSIs. The WSIs and their annotations are
processed using WSI Analyzer. Annotation regions from external XML files are
matched to WSI, bad quality regions are assessed and windows of different
magnifications are selected from the WSI. WSI Analyzers are used to provide
images with annotations to WSI Generators, which are used to train Experts.
Leveraging the flexibility and customizability of the Analyzers and
Generators, a general Expert may be trained for numerous amount of tasks. The
Experts are then used to produce complex analyses of unknown WSIs by combining
simpler partial analyses. Best viewed in color.

##  2 Related Work

Many recent works focus on creating automatic digital pathology systems that
support the classification and segmentation of histopathology microscopy data.
The approaches could be divided into two streams according to the data used:
typical microscopy tissue images; more modern WSI slices containing a very
high-resolution scan of a magnified tissue sample in a pyramidal structure.
The main effort is to build effective automated analysis systems [2] with
precise predictions. The most promising approaches are based on Deep learning.

Alongside the ICIAR 2018 dataset, Spanhol et al. [3] create a large dataset
called BreakHis consisting of 7909 images containing eight breast tumor types
and also provide baseline results using traditional computer vision methods
for feature extractions and traditional machine learning methods for
classification. Spanhol, in their paper, stresses the complex nature of
histopathology data due to the large visual disunity of structures within
tumor types, thus making the classification task even more difficult. Most of
the recent literature base their research on the BreakHis dataset, as it
contains image slices from 82 patients and four magnification levels.

Bayramoglu et al. [4] focus their research on magnification invariant
classification of breast histopathology images using a modern approach
leveraging the power of deep learning to replace the whole traditional
pipeline. Concluding their work, the authors remark that CNNs provide more
promising results than handcrafted features in breast cancer classification.

Recently Attalah et al. [5] applied five more advanced models for better
extraction of texture features, InceptionResnet v2 among them proven as one of
the best, with two handcrafted features for breast cancer classification. They
also apply an AutoEncoder network to learn optimal representation and reduce
the dimensionality of feature vectors created by a fusion of extracted
features. They reached a test accuracy of 83.984% on the ICIAR dataset.

Celik et al. [6], on the other hand, experimented with other advanced models
to recognize Invasive Ductal Carcinoma in histopathological tissue images.
Their DenseNet and ResNet models were pre-trained on ImageNet and finetuned on
a dataset with 227542 samples of $50\times 50$ pixels. The best achieved F1
score was close to 94% and balanced accuracy of 91.57%.

Other approaches considered the magnification very relevant and tried to
modify their approaches to exploit most of the context information provided.

Gandomkar et al. [7] proposed a sequential classification framework that
finetunes a separate ResNet network, pre-trained on ImageNet, on image patches
for each magnification factor from the BreakHis dataset. The first stage is
binary categorization to malignant/benign classes and then into eight
corresponding subclasses. According to the highest confidence from all
predictions on each patient’s microscopy samples, their approach predicted the
final diagnosis. Even though they downsampled the images from $700\times 460$
to $341\times 224$, they achieved 96.25% accuracy for eight class
categorization. The magnification level accuracy ranged from 82.4% to 97.96%.

There were experiments to create specialized neural network architecture with
many exciting approaches to increase its attention on context and local
information. One such work [8] introduces an attention block that employs
channel and spatial attention to provide more stable classification results.
However, their model’s quality and the success rate are questionable due to
their training graphs which indicate that their network is overfitted and
their validation subset is insufficient.

Han et al. [9] propose a class structure-based deep convolutional neural
network (CSDCNN) capable of overcoming the problems of similar structures in
different classes by optimizing the distance of feature vectors of different
classes in feature space. They achieve state-of-the-art results both in binary
and multi-class classification. Furthermore, they point out that CSDCNN can
classify whole slide images by fully preserving global information contained
within the images and avoiding the limitations of patch extraction methods.

WSIs offer better utilization of context information from microscopy images
with their pyramidal multi-scale structure. Recently, Tripathi et al. [10]
used annotated WSIs and 400 patch samples from the Bach ICIAR dataset. They
state that if the prediction needs to be made for the whole tumor or gland,
the network cannot lose spatial nor structural information. They transformed
the WSI into a sequence of patches and extracted features using GoogleNet pre-
trained on the ImageNet dataset. This transformation and feature reduction
allowed them to apply a bidirectional long-short-term memory (BiLSTM) network
architecture. It is questionable if the model suffers the Clever Hans effect
due to ImageNet pretraining, but the approach achieved 84.02% overall
accuracy.

More complex analysis using saliency prediction on each magnification level is
presented in Gecer et al. [11]. Four fully convolutional networks detect
saliency maps representing the histopathologists’ diagnostic regions of
interest. Instead of classifying all patches, the following networks receive
only the most salient regions, significantly reducing time complexity. Then
the majority voting or trained SVM evaluates downsampled classification and
saliency maps to predict the final class for the whole WSI.

Another speed-up by removing non-malignant regions proposed Ni et al. [12] in
their segmentation network using parallel DeepLab network branch classifying
pixel-wise malignancy. This patch/pixel-wise approach required a specific
hierarchy aware loss function considering the contribution of structures of
each segment and image samples with a single label region.

The most similar approach to our proposed method has been published very
recently by Feng et al. [13]. They segment WSIs of liver cancer biopsy samples
and classify them into final diagnoses applicable to the whole slides. They
experimented with several segmentation networks, and the best results achieved
U-Net. A similar pyramid approach is applied to multi-scale WSI generated by
Gaussian pyramid down-sampling. Then seven separate pre-trained U-Nets are
finetuned on individual magnification levels. The method cuts the slide into
12 patches of $448\times 448$ and shifts them in 3 directions to eliminate the
checkerboard artifacts from combined segmentation by weighting. The reverse
pyramid operation projects upsampled segmented WSI from all levels into
original magnification. The final segmentation map is generated by majority
voting of individual thresholded segmentation maps from each level.

According to the discussions with doctors and existing literature, assisting
doctors with meaningful analysis would help speed up their workflow. Analyzing
only magnified and high-detailed tissue samples is not enough for the
pathologist, as these cut-outs often contain little meaningful information
about the tissue structures. Furthermore, most existing approaches present
pathologists only unexplained classification results produced by a deep
learning model. Like doctors, recognizing malignant structures from these
small, highly-magnified regions are also difficult for the network, which does
not see the context that could provide valuable additional information. The
more promising approaches analyze WSIs instead of partial cut-outs. However,
as stated in Feng et al. [13] the problematic step in the analysis is to
combine the individual segmentation maps from all magnifications. The problems
occur in selecting the optimal threshold value for pixel-wise probability and
the static number of votes in majority voting.

Our framework focuses on analyzing the WSIs and provides multiple contextual
information to every magnified window, thus creating a pyramidal analysis by
neural network classifying window samples - slide level multi-class
segmentation. We introduce an adaptive weighting of magnification levels
correcting their contribution to the final combined classification.
Pathologists can use the created analysis for further diagnosis, as we
recognize four classes in the Bach ICIAR dataset. Furthermore, we aim to
employ a method to solve the inter-intraclass variance, leveraging different
magnification levels.

##  3 Framework for WSI Analysis

Due to its vast size and multiple magnification levels, the WSI contains a
massive amount of data. Accessing this data, binding annotation to regions,
and using it efficiently (generating a batch for training a model must not
take too long) is a challenging task. To address this challenge, we propose a
two-stage universal framework for complex WSI analysis consisting of Data
Processing and Data Analysis pipelines. To get the data from WSIs slides, we
use method calls from a Python version of OpenSlide library.

The data processing procedure consists of two main units - the WSI Analyzer
and the WSI Generator. WSI Analyzer is used to manage WSI and its annotation
and, upon request, produce image patches with associated annotations. These
images may subsequently be used by a WSI Generator to train a machine learning
model.

The data analysis pipeline produces analyses for new, previously unseen WSIs.
As the new slices do not have annotations with diagnoses, WSI Analyzer uses a
machine learning model capable of analyzing images to generate new
annotations. WSI Analyser uses the model to infer annotations by classifying
pyramidal regions of the WSI with multi-magnifications analysis. WSI Analyser
may use multiple models to produce complex combined analysis.

Analyzing the whole WSI using multiple magnification levels yields a complex
multi-magnifications analysis capturing both high-magnification details and
the lower-magnification architecture of tissue which provides a context.
Analyzing a WSI slice this way is comparable to the analysis performed by a
pathologist.

###  3.1 Dataset

The proposed Expert models were trained on the ICIAR 2018 [14] dataset. The
dataset consists of 10 annotated WSIs and 20 unannotated. Due to the small
amount of data and necessary structures, we utilized nine WSIs for the
training of our Experts, and the remaining one was used for evaluation. The
annotations associated with the WSIs contain three classes of damaged tissue -
benign, in situ, and invasive carcinomas. Visualization of the annotation may
be seen in Figure 10 (left image), where each of the RGB channels represents a
single class, the black is normal tissue or background. The annotation may
also be used to divide the WSI into tumorous and normal regions, producing a
binary analysis (shown in Figure 9, right). A usual approach to unify the H&E
staining colors is normalization. However, we do not normalize data to
experiment with our approach’s robustness and spare the computing resources to
speedup the analysis process.

###  3.2 WSI Analyzer

The primary function of the WSI Analyzer is providing easy access to multiple
magnifications of microscopy scans contained within a WSI and registering them
to their annotations. WSI Analyzer procedures split the WSI into smaller
window blocks using a complete Quadtree (T-pyramid) in order to assess each of
the windows’ quality meta-information. This structure is stored in memory for
future access to the images and contains information about the positions,
sizes, types, and information quality (background+fat vs. possibly malignant
tissue; we can exclude images that may not contain diagnostically important
information) of each window block.

The window blocks contain patches from all magnification levels (six in ICIAR
2018), creating a pyramidal structure, so we can properly utilize the whole
WSI and extract as much information as possible. The generation of multi-
magnifications T-pyramid structure is visualized in Figure 2. Using the
T-pyramid structure, we can easily combine specific information - acquired
from images with higher magnification with contextual information - from
images with lower magnifications.

Figure 2: Visualization of first three levels of the T-pyramid. Best viewed in
color.

####  3.2.1 Annotation Using the R-Tree

With WSI and its annotation as an input, the WSI Analyzer constructs an R-tree
of the regions contained in the annotation. The R-tree is a search tree over
axis-aligned rectangular regions, mainly used for spatial indexing and
accessing geographical data, which are not dissimilar to our annotated
regions. Using this structure provides a convenient way of performing spatial
queries on images in the form of square-shaped regions, we call Quads.
Spatially intersecting these square regions with the R-tree index, where
annotation information about regions of the WSI is stored (as shown in Figure
3), we achieve a fast way to assess whether an image belongs to at least one
tumor region, and if so, how much area of the image belongs to each region.
Having access to such data, we can create actual annotations for each image
within the T-pyramid by assigning each Quad a class. The resolution of
generated patches is arbitrary, and we used the size $256\times 256$ pixels
for patches from every magnification level. The smaller patch size of
$128\times 128$ contains four times less information than $256\times 256$ and
lacks the context, and larger patches increase the demand for computing
resources, which is not feasible in hospital integration. Tripathi [10] and
several other works used this resolution and noted similar reasons.

Figure 3: Visualization of first-level square patches intersected with
annotation regions. Green windows represent patches, different colors of
regions mean different tumor type. Best viewed in color.

####  3.2.2 Producing Images and Annotation

WSI Analyzers’ input parameters allow it to select square tissue patches,
Quads, along with their assessments to create a batch of images with their
corresponding annotations based on multiple parameters: classes to use,
magnification levels, information quality, batch size, custom preprocess
function applied to each image, size of extracted images. A Parametrized Batch
Producer is created to manage the selection of data from each class contained
within the WSI according to different parameters. The combination of the
outputs from these producers then generates a well-balanced mini-batch
containing samples from each class. Parametrized Batch Producers may be used
with different already mentioned parameters, fine-tuning the batch production
to all needs of well-balanced training of our proposed neural network and
others trained on WSIs.

###  3.3 WSI Generator

We design WSI Generators to produce training data processable by most of the
known Deep Learning frameworks and suitable for the training of neural network
models. WSI Generators can receive data from multiple WSI Analyzers so that
the network model can be trained on a large set of different images. This
training strategy is essential for the good generalization of the model. When
the Generator object is initialized, all samples from the same classes from
all used WSI Analyzers (and all used WSIs) are unified and considered into
creating the mini-batches. The unification is necessary to ensure that all
classes are present for training. The selection and labeling of the classes
are also required conditions for consistent one-hot encoding.

The Generator procedures are designed to be applied with different machine
learning tasks - multiclass classification, binary classification, and even
segmentation, and can extract and generate tissue patches with different
labels used by different models.

##  4 Deep Learning Experts

In addition to the data preparation, WSI Analyzer’s purpose is to analyze and
predict diagnosis annotations for regions of new WSIs. The prediction unit of
WSI Analyser has to be at least one Expert that is provided during
implementation calls while creating the WSI Analyzer. When processing
unannotated WSI, there is no information to construct R-tree with labels. The
labels and quality type are assessed from predictions by analysis experts
models. Other processing steps are the same as in previous use-cases.

After the WSI analysis process is over, the results are recorded into
corresponding T-pyramid Quads. The analysis process creates Quad pyramids of
all available magnifications combining details with low magnification
architectures, and the Quad structure may resemble an organized 3D graph.
Multiple Experts generate multiple analyses assessments, which are contained
inside each graph node. The final aggregated analysis depends on the applied
Experts and results in a complex analysis of the slide. Analyses may also be
generated with every Expert separately.

###  4.1 Binary Classification Expert

The first Expert we design is inspired by the Mask R-CNN’s region proposal
approach described and demonstrated by Li et al.[15] in a similar domain and
produces binary classification of the WSI. We apply a simple binary classifier
to divide the WSI’s T-pyramids into two sets - one that contains
diagnostically essential structures of damaged tissue and one that does not
and mostly consists of only healthy tissue and fat.

We train and use this first Expert to quickly and correctly reduce the size of
many T-pyramids generated from whole WSI. The resulting binary map may
theoretically provide enough clues to the doctor while deciding on a
diagnosis. Furthermore, it proposes regions with suspected damage for further
analysis applying other, more specialized Experts.

####  4.1.1 Binary Classification Expert Design

After several experiments with a custom simple convolutional network, we
decided to rely on a network with already existing and well-established
architecture InceptionV3 [16], with 1 Dense(512) and 1 Dropout(drop_rate=0.3)
layer added at the end of the network. Scheme of the architecture is shown in
Figure 4. Out of multiple experiments using different parameters and
configurations, we achieved the best validation accuracy of 0.8662. The
optimizer was Adam with the learning rate of $1e^{-6}$, the mini-batch size
used was 32, and the model trained for 29 epochs before early stopped. WSI
Generator produced 28800 samples for training.

Figure 4: Architecture of the Binary Expert. Best viewed in color.

###  4.2 Multi-Class Classification Expert

After filtering out redundant structures, we can focus purely on regions
marked as damaged and analyze them further in detail. For this purpose, we
design a Multi-Class Classification Expert. The Expert can distinguish between
3 tumor classes contained within the dataset we use - Benign, Carcinoma in
Situ, and Invasive Carcinoma.

####  4.2.1 Multi-Class Classification Expert Design

Considering the visual structural problems present within breast tissue
microscopy, mentioned by both Spanhol [3] and Han [9], and also other authors,
we utilize the strength of a robust architecture - InceptionResnetV2 [17].
This architecture provides enough capacity to observe and recognize the
differences and similarities between and within the classes to perform
satisfactory multi-class classification. Following the same pattern as in the
previous experiment, we added 1 Dense(512) and 1 Dropout(drop_rate=0.3) layer
at the network’s end. The scheme of the architecture is in Figure 5. The
optimizer was Adam with the learning rate of $1e^{-7}$; the mini-batch size
used was 18; the model was trained for 167 epochs before early-stopped and
achieved the best validation accuracy of 0.62. WSI Generator produced 486000
samples for the training.

Figure 5: Architecture of the Multi-Class Expert. Best viewed in color.

###  4.3 Weighing Predictions Expert

The main contribution of our paper is the Weighing Prediction Expert.
Different magnifications highlight various features that can contribute
differently to determine individual tumor classes. Some tumor classes depend
more on contextual information, while in other cases, details from higher
magnifications are more relevant. Predictions for each region from each
magnification level are assigned with different weights that determine the
influence on the final combined diagnosis. However, it is difficult to set the
correct weights manually because each tumor type or even specific tumor
structure can be better comprehensible from different magnification. We
propose a Weighing Expert neural network that can learn to assign different
weights to individual magnified Quad regions based on the present structures.

####  4.3.1 Weighing Predictions Expert Design

The Expert consists of six separate networks that do not share weights, and
the six products of networks are concatenated at the end to compute the final
value. The simplified schema of the architecture may is in Figure 6. Each
network takes an individual input that is an image from one of six levels of
T-pyramids created within the WSI Analyzer. The output is $6D$ vector - a
predicted weight for each level. The training process computes the loss
function from a multi-class label of ground truth annotations and six softmax
labels containing the previous Expert’s predictions weighted by the predicted
$6D$ vector.

  1. 1.

The loss for $N$ classes (3 classes in our work) is computed in 5 steps:

  2. 2.

The weights vector is extended to the shape of $6\times N$.

  3. 3.

The weights vector element-wise multiplies the previous Expert’s predictions
producing $6\times N$ matrix of weighted predictions.

$W_{pred}=weights\odot predictions$

  4. 4.

The summation of the weighted predictions produces $ND$ vector representing
weighted multi-class label - $W_{sum}$, with each class represented in this
vector having a value in range $\langle 0,\infty)$.

  5. 5.

Softmax to these labels produces $W_{Softmax}$

  6. 6.

Finally, the loss is computed as a categorical cross-entropy between the $gt$
labels and $W_{Softmax}$ values.

$Loss=-\sum_{i=1}^{N}gt_{i}.log(W_{Softmax_{i}})$

The optimizer was Adam with the learning rate of $1e^{-4}$. The mini-batch
size was 128; the model trained for 55 epochs before early stopped; the best
loss achieved was 0.5814, and the best evaluation loss of 1.154. The
classification evaluation metrics are later in the results section.

Figure 6: Simplified architecture of the Weighing Expert. The digits within
Conv layers represent number of filters. Conv layers have a stride of 2,
therefore the image resolution is halved with each convolution. The shown
architecture until last Conv(64) is used for all six network branches, and a
softmax of their summed outputs produces the final $6D$ vector of weights.
Best viewed in color.

##  5 Results

In this section, we evaluate the trained Experts. First, we evaluate
individual Experts’ performance, focusing purely on the Expert’s designated
task. In the second part, we evaluate classification on the whole WSI
exploiting the context information. All used evaluation metrics of different
Experts are shown in Table 1. All training runs were performed on an 6GB GPU
Nvidia GTX 1060 Max-Q design.

Table 1: The best results achieved by our Experts. Strategy is the strategy
used for evaluation: Balanced = subset of balanced classes; Slide = the
selected regions of one whole slide; Flip is the measure of the flip
operation.

Expert | Classes | Strategy | Accuracy | Precision | Recall | Specificity
---|---|---|---|---|---|---
Binary | 2 | Balanced | 88.6% | 88.2% | 88% | 89.2%
Multi-Class | 3 | Balanced | 72.2% | 58.5% | 58.3% | 79.1%
Slide | 85.3% | 49.5% | 64.3% | 82.4%
Weighing | 3 | Flip | 84.8% | 77.5% | 43.4% | 96.4%
Slide | 89.8% | 55.1% | 65.5% | 84.5%

###  5.1 Expert Level Results

First, we present the results of the network that represents each Expert. We
evaluate the Expert to assess how well the Experts handle the designated tasks
regardless of the context information. All of the results, compared to other
state-of-the-art approaches, are shown in Table 2.

Table 2: The best achieved results of analyzed methods for classification.
Last row are the best results achieved using our training strategy with state-
of-the-art nets.

Authors | Approach |  | Number
---
of classes
Accuracy
Spanhol et al.[3] |  | PFTAS descriptor +
---
Traditional classifiers
2 | 83.33%
Spanhol et al.[18] | DCNN | 2 | 87.28%
Bayramoglu et al.[4] | DCNN, magnification independent | 2 | 82.1%
DCNN, magnification specific | 2 | 80.66%
Han et al.[9] | CSDCNN | 2 | 96.25%
CSDCNN + Aug | 8 | 93,88%
Tripathi et al. [10] | BiLSTM | 3 | 84.02%
Our Solution | InceptionV3 | 2 | 88.3%
InceptionResnetV2 | 3 | 85.3%
InceptionResnetV2 + WeighingNet | 3 | 89.8%

####  5.1.1 Binary Classification Expert Results

The Binary Expert achieved an accuracy of 88.3%, a precision of 91%, and a
recall of 86% when evaluated as means of five test runs, each using 20480
previously unseen images balanced across both classes. Judging from these
metrics, we can see that the Binary Expert quite successfully distinguishes
between healthy and damaged tissue. These results are satisfactory enough for
the Expert to filter out redundant and diagnostically unnecessary structures
and mark regions required for further processing.

####  5.1.2 Multi-class Classification Expert Results

The multi-class Expert achieved an accuracy of 62.5%, with 63% precision and
62% recall. The test was performed the same way with five test runs, each
20480 previously unused images, evenly distributed among all classes. Figure 7
shows the confusion matrix for three different predicted classes. The results
show room for improvement, but as previously stated, the difficult nature of
breast histopathology visuals limits the performance of any deep learning
model. Contrary to Han [9], who uses special architecture and loss function,
we try to deal with this limit using a Weighing Expert.

Figure 7: Confusion matrix for the Multi-Class Expert. Best viewed in color.

####  5.1.3 Weighing Predictions Expert Results

In order to evaluate whether the Weighing Expert is working correctly, we
define the action of changing one class to another as Flip and the action of
not changing the class as No Flip. Using the non-weighted predictions, the
weighted predictions, and the ground truth annotations, we may define four
scenarios:

  * •

Correct Flip - the predicted class changes from an incorrect one to a correct
one (equivalent to true positive)

  * •

Incorrect Flips - the predicted class changes from a correct one to an
incorrect one (false positive)

  * •

Correct No Flips - the predicted class does not change and stays correct (true
negative)

  * •

Incorrect No Flips - the predicted class does not change and stays incorrect,
or the predicted class changes, but from an incorrect one to an incorrect one
(false negative).

Therefore, we can compute all usual metrics using true positives, true
negatives, false positives, and false negatives.

We performed the evaluation using 12123 samples from previously unseen WSI
where the metric measured the success rate of correctly correcting the
previous Expert’s outputs, in other words, if the class probability was
Flipped when it was necessary. The Weighting Prediction Expert scored an
accuracy of 76%, a precision of 77%, and a recall of 36%.

Weighting Prediction Expert achieved an accuracy of 75.8%; a precision of 53%;
and a recall of 74% in the measurement of the classification performance
comparing weighted predictions to the ground truth labels.

Figure 8: ROC of the Binary Expert achieved on the Evaluation slide. The
dotted line = random guessing. Best viewed in color.

###  5.2 WSI Level Results

Now we evaluate the performance of our framework combining the results of each
Expert on the classification of all regions of one WSI. The optimal
quantitative metric for this evaluation is Intersection over Union (IoU),
computed as

$IoU=\frac{gt\cap predicted}{gt\cup predicted}$

We also empirically evaluate each analysis result from a histopathologist’s
perspective, using the analysis as assistance.

####  5.2.1 Binary Expert WSI Level Results

Figure 8 shows the ROC of the Binary Expert. Based on several achieved scores:
0.98 AUC, 0.74 IoU, empirical evaluation using GT annotation, and the results
of Binary Expert analysis shown in Figure 9, we can establish that the Expert
can distinguish between healthy and damaged tissue and filter stroma regions
without significant structures. The identified possibly damaged stroma regions
can be proposed to other Experts for more complex analysis.

Figure 9: Left: GT binary annotations of the Slide. Right: Analysis performed
by the Binary Expert. Higher intensity of the color means higher confidence of
the Expert. Best viewed in color.

####  5.2.2 Multi-Class Expert WSI Level Results

Figure 10: Left: The GT annotations. Red channel represents Benign tumor,
Green channel is In Situ tumor and Blue channel is region with Invasive tumor.
Right: Analysis produced by the Multi-Class Expert. Best viewed in color.

Figure 10 shows the analysis results by the Multi-class Expert separately with
the ground truth (GT) annotations of searched tumor classes. IoU scores (micro
and macro average) of the whole slide are shown in Table 3. The ”Multi-Class”
strategy processed only GT tumor regions, hence the high
classification/segmentation score. The ”Multi-Class Pipeline” strategy used
”Binary Expert” first and then analyzed regions marked as damaged with
confidence greater than 0.3. The evaluation proved that the Binary Expert
region proposals are precise enough to help make the area analyzed by
subsequent Experts smaller, thus saving time and resources. Furthermore, we
can see that the Expert can pick up the differences between tumor classes and
create a complex analysis. The last strategy, called ”Weighing Pipeline”, used
our weighing network to determine the contribution of individual
magnifications to the final classification.

Table 3: IoU scores achieved by the Multi-Class Expert, the Multi-Class Expert Pipeline and the Weighted Pipeline when evaluated using a previously unseen WSI. Class | Strategy | Micro Average | Macro Average | Binary
---|---|---|---|---
All | Multi-Class | 0.93 | 0.55 | -
Multi-Class Pipeline | 0.9 | 0.47 | -
Weighing Pipeline | 0.9 | 0.47 | -
Benign | Multi-Class | 0.94 | 0.66 | 0.35
Multi-Class Pipeline | 0.88 | 0.54 | 0.14
Weighing Pipeline | 0.88 | 0.54 | 0.14
In Situ | Multi-Class | 0.95 | 0.54 | 0.11
Multi-Class Pipeline | 0.95 | 0.54 | 0.1
Weighing Pipeline | 0.96 | 0.53 | 0.07
Invasive | Multi-Class | 0.9 | 0.85 | 0.77
Multi-Class Pipeline | 0.86 | 0.8 | 0.7
Weighing Pipeline | 0.87 | 0.82 | 0.73

Similarly to the results of Multi-Class Expert evaluation (Section 5.1.2), we
see that even though the Expert is often able to analyze regions correctly,
confusion of the classes caused by all of the mentioned problems is still
present, even on WSI level.

####  5.2.3 Weighing Expert WSI Level Results

Figure 11: Analysis predictions produced by the Multi-Class Expert (left) and
the same analysis predictions weighted by the Weighing Expert (right). Red
channel represents Benign tumor, Green channel is In Situ tumor and Blue
channel is region with Invasive tumor. Best viewed in color.

Figure 11 shows the comparison of the analysis produced by the Multi-Class
Expert and the same analysis weighted by the Weighing Expert. The improvement
of the IoU is visible in Table 3 when comparing the simple Multi-Class versus
Weighing Pipeline strategy. The Weighing Expert network improves the
classifications and the metric - IoU.

The empirical evaluation of Weighing Expert’s performance by comparing the
weighted analysis (Figure 11, right) to non-weighted (Figure 11, left) and
ground truth annotations (Figure 10, right) shows improvements in predictions.
The Weighing Expert corrects classes in the areas with wrong predictions,
specifically: in the middle, a large area of the In Situ (green) region, and
Benign (red) changed correctly to Invasive carcinoma (blue) class region; we
also detected some minor changes in the top-left areas after closer
inspection. These results suggest that the Expert is working properly, but the
automatic evaluation metrics tell us that there is still room for improvement.

##  6 Discussion

As already mentioned, there are two types of publicly available datasets: the
WSI images annotated by histologists (ICIAR [14]); and selected samples from
one magnification as a chosen trade-off between context and detail [3, 19,
20]. During our work, we noted problems with both dataset types.

The pre-selected magnification with generated samples suffers from information
loss. It lacks information about the architecture of the analyzed structure.
We showed the digital cut-out samples to several doctors individually, and
they all agreed they could not state the diagnosis because they did not see
where in the whole slice the structure was and what its surroundings were
concerning the tissue morphology.

With this knowledge, the second problem emerges. Even when its surroundings
are visible, it is vague to mark the border of the tumor precisely. Almost all
of the existing annotations are only humanly possible approximations with
Bayesian error. With the help of histopathologists, we performed our own
annotations to mark borders more precisely than the original ICIAR 2018
annotations.

When the network analyzes a tiny detailed region with its architectural
context, it can mark the regions precisely in even more detail than human
experts, even if it is not necessary for the final diagnosis. Human experts
often enclose large regions containing other tumor types in the dataset
annotations because it is less manual work to select only the largest tumor
region. Therefore, we do not mean that the network overperforms the
histologist, but that the network only marks more small regions because it
operates on the window ($256\times 256$ pixels on all magnifications) size
level. In Figure 12, we can recognize predictions of benign tumor areas on the
left border of the invasive region. Those errors are caused mainly through
approximated annotations and similarity of stroma in benign tumor regions and
borders of invasive carcinoma. On the other hand, when it processes samples
from high magnification with much detail, sometimes the detail can overweight
the context. Then sparse regions of different classes can appear within a
large block, which we can observe in Zoom level 5 - many in situ and large
benign regions classified in invasive tissue. The ground truth annotations
show that benign regions are regions with fat or ducts, and in situ carcinoma
is predicted without high-level context - tissue architecture.

Our weighing expert should correct these misinterpretations and unify the
region if it contains only a particular tumor type, as can be seen in Figure
12. Unifying the annotation regions and the annotation should more resemble
the work of a histopathologist as is also stated in [12]. It is visible in
Table 3, in Confusion Matrix 7 and in Figure 12 that our multi-class expert
and Weighing Expert are leaning towards the Invasive carcinoma at the expense
of In Situ carcinoma. The In Situ is the only class that the results have
worsen in binary classification (one class against rest). We consider this
behavior a lesser problem because invasive carcinoma is much more dangerous
and more deadly [12], which also relates to the unification of large regions.

In comparing our multi-scale adaptive weighted voting approach to the majority
of related work analyzing only microscopy patches of single magnification, our
method is more general, and it better utilizes a large scale of WSI data.
Furthermore, our selected patches of size $256\times 256$ are a standard size
but contain more information than $50\times 50$ patches used in Celik et al.
[6] and helped to avoid overfitting.

The usual strategy of combining the evaluated patches is majority voting [13],
or thresholded or highest confidence [7], which Feng et al. [13] identified as
problematic. Instead, our weighing network learns to change the contribution
power of each magnification adaptively and improves the overall performance of
the multi-class classifier pipeline from 72% to 84%.

Figure 12: Images from left to right: classification on magnification level 0
(lowest); on magnification level 5 (highest); unification and reclassification
by weighing Expert; ground truth annotation over tissue slice. The
classification on the highest detail level inserts a high class-noise, which
is later partially corrected by Weighing Expert. Unfortunately, our Weighing
Expert reclassified In Situ carcinoma to Invasive as seen in the top right
corner, where high detail classifications were correct, even though correctly
unified region in the bottom right corner. Best viewed in color.

##  7 Conclusion

In this paper, we present a universal framework for complex WSI analysis,
using various approaches based on deep learning models in the form of Experts.
We also present multiple Experts: the Binary Expert for distinguishing between
healthy and damaged parts of the WSI and proposing damaged regions to other
Experts for further analysis; the Multi-Class Classification Expert for
classifying regions as different carcinoma classes and the Weighing Expert for
better utilization of multiple magnifications in WSI file. As a result of
using the combined Experts, we obtain a complex and unbiased analysis of the
whole WSI that can serve as a reference or a helping tool for pathologists.

Our binary classification approach using our custom data preparation and
generation and custom trained well-known model of InceptionV3 architecture are
comparable to other state-of-the-art approaches presented by Spanhol [18],
[3], and Bayramoglu [4], Tripathi [10], and Feng [13].

Our multi-class classification results are not yet on par with the research of
Han [9] and other related works. However, we propose an approach easy to
interpret for histopathologists and similar to their workflow. Furthermore,
increasing the number of trainable data, better hyper-parameter optimization
of the InceptionResnetV2, further exploration of the Weighing Expert’s
capabilities, and increasing the number of parameters for each network the
Expert contains to enhance the learning capacity may help to achieve state-of-
the-art results in this domain as well in the future.

The universality of our designed framework allows the creation of an arbitrary
number of Experts, each analyzing a specific part of the WSI, leading to even
more complex and precise analyses. For example, an Expert for segmentation of
the tissue, distinguishing even better between the classes, especially on the
tumor border regions where different cancer types overlap, would be valuable,
although hard to train due to the visual structure of histopathology images.
Another Expert possibility would be a counter of mitoses on the WSI level,
which is often required to grade the cancer stage.

## References

  * [1] “Breast cancer,” Mar 2021. [Online]. Available: https://www.who.int/news-room/fact-sheets/detail/breast-cancer
  * [2] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. van der Laak, B. van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” _Medical Image Analysis_ , vol. 42, no. December 2012, pp. 60–88, 2017.
  * [3] F. A. Spanhol, P. R. Cavalin, L. S. Oliveira, C. Petitjean, and L. Heutte, “A Dataset for Breast Cancer Histopathological Image Classificatio,” _IEEE Transactions on Biomedical Engineering_ , vol. 63, no. 7, pp. 1455–1462, 2016.
  * [4] N. Bayramoglu, “Deep Learning for Magnification Independent Breast Cancer Histopathology Image Classification,” pp. 2440–2445, 2016.
  * [5] O. Attallah, F. Anwar, N. M. Ghanem, and M. A. Ismail, “Histo-CADx: duo cascaded fusion stages for breast cancer diagnosis from histopathological images,” _PeerJ Computer Science_ , vol. 7, p. e493, 2021.
  * [6] Y. Celik, M. Talo, O. Yildirim, M. Karabatak, and U. R. Acharya, “Automated invasive ductal carcinoma detection based using deep transfer learning with whole-slide images,” _Pattern Recognition Letters_ , vol. 133, pp. 232–239, 2020. [Online]. Available: https://doi.org/10.1016/j.patrec.2020.03.011
  * [7] Z. Gandomkar, P. C. Brennan, and C. Mello-Thoms, “MuDeRN: Multi-category classification of breast histopathological image using deep residual networks,” _Artificial Intelligence in Medicine_ , vol. 88, pp. 14–24, 2018\. [Online]. Available: https://doi.org/10.1016/j.artmed.2018.04.005
  * [8] M. Toğaçar, K. B. Özkurt, B. Ergen, and Z. Cömert, “BreastNet: A novel convolutional neural network model through histopathological images for the diagnosis of breast cancer,” _Physica A: Statistical Mechanics and its Applications_ , vol. 545, p. 123592, 2020. [Online]. Available: https://doi.org/10.1016/j.physa.2019.123592
  * [9] Z. Han, B. Wei, Y. Zheng, Y. Yin, K. Li, and S. Li, “Breast Cancer Multi-classification from Histopathological Images with Structured Deep Learning Model,” _Scientific Reports_ , vol. 7, no. 1, p. 4172, 2017. [Online]. Available: https://doi.org/10.1038/s41598-017-04075-z
  * [10] S. Tripathi, S. K. Singh, and H. K. Lee, “An end-to-end breast tumour classification model using context-based patch modelling – A BiLSTM approach for image classification,” _Computerized Medical Imaging and Graphics_ , vol. 87, p. 101838, jan 2021.
  * [11] B. Gecer, S. Aksoy, E. Mercan, L. G. Shapiro, D. L. Weaver, and J. G. Elmore, “Detection and classification of cancer in whole slide breast histopathology images using deep convolutional networks,” _Pattern Recognition_ , vol. 84, pp. 345–356, dec 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0031320318302577
  * [12] H. Ni, H. Liu, K. Wang, X. Wang, X. Zhou, and Y. Qian, “WSI-Net: Branch-Based and Hierarchy-Aware Network for Segmentation and Classification of Breast Histopathological Whole-Slide Images,” in _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , vol. 11861 LNCS. Springer, oct 2019, pp. 36–44. [Online]. Available: https://doi.org/10.1007/978-3-030-32692-0_5
  * [13] Y. Feng, A. Hafiane, and H. Laurent, “A deep learning based multiscale approach to segment the areas of interest in whole slide images,” _Computerized Medical Imaging and Graphics_ , vol. 90, p. 101923, 2021.
  * [14] G. Aresta, T. Araújo, S. Kwok, S. S. Chennamsetty, M. Safwan, V. Alex, B. Marami, M. Prastawa, M. Chan, M. Donovan, G. Fernandez, J. Zeineh, M. Kohl, C. Walz, F. Ludwig, S. Braunewell, M. Baust, Q. D. Vu, M. N. N. To, E. Kim, J. T. Kwak, S. Galal, V. Sanchez-Freire, N. Brancati, M. Frucci, D. Riccio, Y. Wang, L. Sun, K. Ma, J. Fang, I. Kone, L. Boulmane, A. Campilho, C. Eloy, A. Polónia, and P. Aguiar, “Bach: Grand challenge on breast cancer histology images,” _Medical Image Analysis_ , vol. 56, pp. 122–139, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1361841518307941
  * [15] W. Li, J. Li, K. V. Sarma, K. C. Ho, S. Shen, B. S. Knudsen, A. Gertych, and C. W. Arnold, “Path R-CNN for Prostate Cancer Diagnosis and Gleason Grading of Histological Images,” _IEEE Transactions on Medical Imaging_ , vol. 38, no. 4, pp. 945–954, 2019.
  * [16] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [17] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 31, no. 1, 2017.
  * [18] F. A. Spanhol, L. S. Oliveira, C. Petitjean, and L. Heutte, “Breast cancer histopathological image classification using Convolutional Neural Networks,” in _2016 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 07 2016, pp. 2560–2567. [Online]. Available: http://ieeexplore.ieee.org/document/7727519/
  * [19] B. Ehteshami Bejnordi, M. Veta, P. Johannes van Diest, B. van Ginneken, N. Karssemeijer, G. Litjens, J. A. W. M. van der Laak, , and the CAMELYON16 Consortium, “Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer,” _JAMA_ , vol. 318, no. 22, pp. 2199–2210, 12 2017. [Online]. Available: https://doi.org/10.1001/jama.2017.14585
  * [20] B. S. Veeling, J. Linmans, J. Winkens, T. Cohen, and M. Welling, “Rotation equivariant cnns for digital pathology,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2018, pp. 210–218.


Jinsoo<EMAIL_ADDRESS>Jaesik<EMAIL_ADDRESS>In So
<EMAIL_ADDRESS>KAIST
Republic of Korea POSTECH
Republic of Korea Self-Supervised Real-time Video Stabilization

# Self-Supervised Real-time Video Stabilization

###### Abstract

Videos are a popular media form, where online video streaming has recently
gathered much popularity. In this work, we propose a novel method of real-time
video stabilization - transforming a shaky video to a stabilized video as if
it were stabilized via gimbals in real-time. Our framework is trainable in a
self-supervised manner, which does not require data captured with special
hardware setups (i.e., two cameras on a stereo rig or additional motion
sensors). Our framework consists of a transformation estimator between given
frames for global stability adjustments, followed by scene parallax reduction
module via spatially smoothed optical flow for further stability. Then, a
margin inpainting module fills in the missing margin regions created during
stabilization to reduce the amount of post-cropping. These sequential steps
reduce distortion and margin cropping to a minimum while enhancing stability.
Hence, our approach outperforms state-of-the-art real-time video stabilization
methods as well as offline methods that require camera trajectory
optimization. Our method procedure takes approximately 24.3 ms yielding 41 fps
regardless of resolution (e.g., 480p or 1080p).

##  1 Introduction

Due to the recent popularity in social networking services (SNS), videos have
become a popular media form, demanding higher visual quality as time
progresses. Recently, cameras (including smartphones) make use of hardware
configurations to produce stabilized videos. One such method is the Optical
Image Stabilizer (OIS), which negates instability caused by hand movements via
adjusting the optical lens positions. Another mechanism is the Electronic
Image Stabilizer (EIS) that is designed to compensate for more substantial
motion. However, these methods require specialized motion-sensing hardware
synchronized with image capture, and may lead to significant cropping of the
frame boundaries of the original video, which results in an inevitable zoom-in
effect. Fig. 2 shows a real example of a video captured with OIS and EIS.

Figure 1: Illustration of approach. Given a shaky camera path (black line
profile), our method applies global transforms(top row). Then, spatially
smoothed flow warping is applied for further stability(2nd row). Lastly, the
image margins are inpainted to minimize missing regions (last row).

To cover such limitations, software approaches have been developed typically
for offline purposes, namely post-processing of existing videos [Liu et
al.(2013b)Liu, Yuan, Tan, and Sun, Liu et al.(2014)Liu, Yuan, Tan, and Sun,
Grundmann et al.(2011)Grundmann, Kwatra, and Essa]. Offline approaches have
shown success with stabilization quality, but the computational speed is an
issue. More recently, due to the growing popularity in online streaming, works
on real-time methods [Liu et al.(2017)Liu, Li, Zhu, and Zeng, Liu et
al.(2016)Liu, Tan, Yuan, Sun, and Zeng, Wang et al.(2018)Wang, Yang, Lin,
Shamir, Zhang, Lu, and Hu, Xu et al.(2018)Xu, Hu, Wang, Mu, and Hu] have been
introduced. Especially, deep learning-based methods [Wang et al.(2018)Wang,
Yang, Lin, Shamir, Zhang, Lu, and Hu, Xu et al.(2018)Xu, Hu, Wang, Mu, and Hu]
utilize the fast feed-forward mechanism and a supervised approach to realize
real-time performance. Compared to offline methods, however, the real-time
approaches exhibit a slight loss of stability and increased distortion
artifacts. Moreover, supervised learning methods require capturing
simultaneous ground truth footages for supervised learning, which is a time-
consuming process.

A desirable video stabilization can be characterized by minimal cropping, low
visual distortion, and high stability. In this work, we propose a deep
learning-based approach to enable real-time video stabilization with these
aspects. The idea is to introduce a cascade module which consists of a coarse
(global) stabilizer and a fine (detail) stabilizer to handle complex camera
motion in the wild effectively. The coarse stabilizer estimates the rigid
inter-frame transformations among the multiple (unstable) frame inputs for
global adjustment of subsequent frames. Next, a fine stabilizer applies the
spatially smoothed optical flow between input frames to handle parallax and
spatially varying instability. Both stabilizers use moving average filters to
suppress noisy camera motion without explicitly optimizing camera
trajectories. After the stabilization step, the inpainting module fills in the
blank margins induced by the stabilization process. Fig. 1 shows an
illustration of the process.

Figure 2: Example of the cropping ratios among OIS+EIS, our method and
original frames taken by a Galaxy S10. OIS+EIS exhibits _zoom-in effects_.

A notable aspect of our method is that our approach does not require any
ground truth videos for deep architecture training, but yet manages to produce
strong stabilization effects in a self-supervised manner. This distinguishes
our method with supervised methods [Wang et al.(2018)Wang, Yang, Lin, Shamir,
Zhang, Lu, and Hu, Xu et al.(2018)Xu, Hu, Wang, Mu, and Hu] that require sets
of two videos, one of which is unstable while the other is physically
stabilized with gimbals, captured simultaneously. The proposed network is
fully convolutional and is trained end-to-end, where our model achieves real-
time regardless of input video size (e.g., 480p or 1080p) with a single GPU.
The contributions of our work are listed as follows.

  * •

We propose a real-time approach for video stabilization. The input video is
stabilized via a single feed-forward through the constituent modules.

  * •

We introduce a self-supervised training method for video stabilization.
Therefore, existing videos can be used for network training.

  * •

Our method induces low levels of cropping, low visual distortion, and high
stability. The proposed approach outperforms the state-of-the-art methods both
quantitatively and qualitatively.

##  2 Related Work

Video stabilization approaches consist of offline and real-time methods.
Offline methods are characterized by strong stabilization effects while having
a high computational load. On the other hand, real-time methods are
computationally efficient but convey weaker stabilization effects compared to
offline methods. We briefly address offline and real-time approaches in the
following.

Previous works on offline methods deal with techniques including Structure
from Motion (SfM) [Liu et al.(2009)Liu, Gleicher, Jin, and Agarwala], depth
information [Liu et al.(2012)Liu, Wang, Yuan, Bu, Tan, and Sun], 3D plane
constraints [Zhou et al.(2013)Zhou, Jin, and Ma], 3D reconstruction [Buehler
et al.(2001)Buehler, Bosse, and McMillan], light fields [Smith et
al.(2009)Smith, Zhang, Jin, and Agarwala], gyroscopes [Bell et al.(2014)Bell,
Troccoli, and Pulli, Karpenko et al.(2011)Karpenko, Jacobs, Baek, and Levoy,
Ovrén and Forssén(2015)], and partial 3D information [Goldstein and
Fattal(2012), Liu et al.(2011)Liu, Gleicher, Wang, Jin, and Agarwala, Liu et
al.(2013a)Liu, Niu, and Jin]. Image-based methods have also shown sufficient
quality [Chen et al.(2008)Chen, Lee, Huang, and Lin, Gleicher and Liu(2007),
Huang et al.(2018)Huang, Wei, and Zhang, Matsushita et al.(2006)Matsushita,
Ofek, Ge, Tang, and Shum, Yu and Ramamoorthi(2020)]. Grundmann  _et al_
[Grundmann et al.(2011)Grundmann, Kwatra, and Essa] apply L1-norm optimization
for camera path computation and extend the approach to handle rolling shutter
effects [Grundmann et al.(2012)Grundmann, Kwatra, Castro, and Essa]. Liu  _et
al_ [Liu et al.(2013b)Liu, Yuan, Tan, and Sun] model camera paths for each
image patches, and extend the idea to model the entire pixel profiles [Liu et
al.(2014)Liu, Yuan, Tan, and Sun]. This work shows that spatially smoothed
flow is useful for stabilizing frames, which our approach builds upon.
Recently, Choi and Kweon [Choi and Kweon(2020)] introduce an iterative frame
interpolation approach to video stabilization.

Real-time methods typically employ more efficient computations while using
historical frames to estimate stabilization parameters. Liu  _et al_ [Liu et
al.(2016)Liu, Tan, Yuan, Sun, and Zeng] proposes computing sparse mesh
profiles via applying median filters to pixel profiles. This idea is extended
further to video coding [Liu et al.(2017)Liu, Li, Zhu, and Zeng]. Limitations
of these methods are the slight wobbling artifacts and relatively low
stabilization quality compared to offline methods. Recently, deep learning-
based algorithms [Liu et al.(2021)Liu, Lai, Yang, Chuang, and Huang] take
advantage of fast feed-forward capabilities. Wang  _et al_ [Wang et
al.(2018)Wang, Yang, Lin, Shamir, Zhang, Lu, and Hu] propose a supervised
learning approach to video stabilization by defining the novel stability and
temporal loss terms. Similarly, Xu  _et al_ [Xu et al.(2018)Xu, Hu, Wang, Mu,
and Hu] proposes training an adversarial network in a supervised manner, which
estimates transformation parameters to generate stabilized frames. Compared
with those approaches, our approach does not require stable videos for
training. Recently, Yu and Ramamoorthi [Yu and Ramamoorthi(2020)] propose a
learning method using principal components of optical flow information. Our
approach estimate smooth warping field via neural network and produces high-
quality stabilization.

The core module of our pipeline is related to deep image tranformation
estimation and image inpainting. Deep homography estimation [DeTone et
al.(2016)DeTone, Malisiewicz, and Rabinovich, Nguyen et al.(2018)Nguyen, Chen,
Shivakumar, Taylor, and Kumar] enables transformation matrix estimation
between two frames without explicitly extracting image features. On the other
hand, advances in image inpainting tasks employ specific deep architecture
components for natural inpainting quality [Liu et al.(2018)Liu, Reda, Shih,
Wang, Tao, and Catanzaro, Yu et al.(2019)Yu, Lin, Yang, Shen, Lu, and Huang].
Although image boundary inpainting is a difficult task, our deep margin
inpainter utilizes multiple adjacent frames to fill in frame boundaries.

Figure 3: An overview of the Coarse-Net, Fine-Net and Margin-Net during
testing. The Coarse-Net computes the transform parameters among input frames
with respect to the center image $I_{i}$, which are then averaged to transform
$I_{i}$ to $C_{i}$. The Fine-Net compute the mean of the flow maps to warp
$C_{i}$, generating $F_{i}$. Finally, the Margin-Net takes $F_{i}$ and
neighboring input frames to generate $F_{i}^{m}$, which is applied with the
margin mask to generate the final output frame $O_{i}$.

##  3 Architecture

We aim to achieve high-quality stabilization and high computational speed via
three sequential modules, namely the coarse stabilizer, fine stabilizer, and
margin inpainter. The overall process of our method is illustrated in Fig. 3.
The coarse (first) stabilizer estimates the rigid transformations among frames
to globally transform a frame for stabilization. The fine (second) stabilizer
estimates the spatially smoothed flow among frames to adjust the remaining
instability. Finally, the margin inpainter fills in the frame boundaries to
reduce the need for cropping. Unlike prior work that runs offline, our
approach does not involve any optimization, camera pose estimation. Instead,
sequential feed-forward passes of the coarse and fine networks produce the
stabilized frames, which has been shown to be effective [Yu and
Ramamoorthi(2020)].

###  3.1 Coarse stabilizer

Our approach first estimates the image transformations between video
frames.111We empirically found the 3-DoF rigid transformation
$\bigl{[}\begin{smallmatrix}R({\theta)}&\mathbf{t}\\\
0&1\end{smallmatrix}\bigr{]}$ is surprisingly effective for video stability,
compared with transformations having higher DoF. The supplement summarizes our
experiment results on other type of transformations that include scale and/or
shear. Please note that the combination of Coarse stabilizer and Fine
stabilizer and their moving average can effectively handle challenging motions
(zoom, parallax, and so on), as shown in Fig. 6, 7. The network for coarse
motion estimation (Coarse-Net) is built upon the U-Net architecture
[Ronneberger et al.(2015)Ronneberger, Fischer, and Brox] followed by fully
connected (FC) layers to estimate the transformation parameters. The Coarse-
Net takes an optical flow $\mathcal{F}_{j\rightarrow i}$ from image pairs
$(I_{j},I_{i})$ and produces the transformation parameters that can relieve
abrupt motion. The network is shown in Fig. 3 (a).

With this procedure, an $i$-th image $I_{i}$ has the estimated transformation
parameters $\mathbf{p}_{j\rightarrow i}$ of the adjacent time stamps
$j\in\mathcal{N}$. They are averaged to produce a transformation matrix T. It
is obtained from $\overline{\mathbf{p}}_{i}$ as defined below:

$\overline{\mathbf{p}}_{i}:=\frac{1}{2N}\sum_{j=-N,j\neq
i}^{N}\mathbf{p}_{i+j\rightarrow i}.$ (1)

T adjusts the target frame to produce $C_{i}=\textrm{T}(I_{i})$ for
stabilization.222For convenience, we denote T as a transformation matrix as
well as a function that transforms image using T. This procedure is equivalent
to applying the moving average filter [Smith et al.(1997)] to the relative
displacements between frames within the window $2N$. Since the moving average
filter is a low-pass filter, it leads to noise suppression effects. This
procedure enables finding the transformation to stabilize the target frame,
while avoiding expensive camera pose estimation and optimization.

###  3.2 Fine stabilizer

The image transformations estimated by the coarse stabilizer alone is not
enough to handle parallax and spatially varying instability. To stabilize such
spatially varying instability, we employ _spatially smoothed optical flow_.
This stage is inspired by Liu _et al_ [Liu et al.(2014)Liu, Yuan, Tan, and
Sun]’s work. It demonstrates spatially smoothed pixel trajectories are what
essentially underlies between stabilized frames.

In practice, the smooth flow should be robust to foreground object movement.
For example, a moving person in a scene cannot interfere with the optical flow
that is induced solely by camera movement. Prior methods implement the smooth
optical flow by identifying pixels with discontinuous flow vectors [Liu et
al.(2014)Liu, Yuan, Tan, and Sun], which can be a slow process. Instead, we
utilize a simple neural network generating the smoothed optical flow through a
single feed-forward.

In this regard, the architecture of the fine stabilizer (Fine-Net) produces
smooth flow with a tiny U-shaped network. Fine-Net applies subsequent down-
sampling of the features for filtering noisy motions. In addition, the average
pooling layers in the network further induces spatial smoothness. Thus, by
design, the Fine-Net does not produce high-frequency flow. Fine-Net is shown
in Fig. 3 (b), and examples of smooth flow are shown in Fig. 4. It shows that
output flow is agnostic to small foreground movement.333In Fig. 4, we use the
full-sized images for the raw optical flow computation for the illustration.
Fine-Net uses resized video instead for optical flow computation as discussed
in Sec. LABEL:sec:efficient_implementation.

Likewise in the coarse stabilizer, the moving average is applied to a group of
smooth flow maps in a sliding window manner as Eq. (1). Averaged flow map
generated from a window of $C_{i}$ frames, produce a _stabilizing warp_ W for
frame $C_{i}$, resulting in $F_{i}$. Unlike the recent work by Yu and
Ramamoorthi [Yu and Ramamoorthi(2020)], our approach does not require an
additional PCA component, and solves the problem with simple network
architectures.

Figure 4: The Fine-Net produces spatially smoothed flow maps via a single feed
forward operation that is purely induced by camera movement. The raw optical
flow responds to the foreground, while our network is agnostic to it.

###  3.3 Margin inpainter

As frames are adjusted to stabilized positions, missing blank regions are
inevitably created at the frame margins. This is due to temporally unseen
content during camera shakes. Thus, a typical post-processing, cropping the
boundary of frame was needed in previous approaches [Liu et al.(2016)Liu, Tan,
Yuan, Sun, and Zeng, Liu et al.(2013b)Liu, Yuan, Tan, and Sun, Wang et
al.(2018)Wang, Yang, Lin, Shamir, Zhang, Lu, and Hu] to conceal missing blank
regions. However, excessive cropping leads to loss of original content and the
zoom-in effect as a consequence. We attempt to minimize such cropping via the
margin inpainter.

Our approach takes five frames, namely $I_{i-2}$, $I_{i-1}$, $F_{i}$,
$I_{i+1}$, and $I_{i+2}$, where $F_{i}$ is the warped image obtained from
Fine-Net. For motion compensation of the adjacent frames, the smooth flow maps
estimated from the Fine stabilizer are used to warp adjacent frames to the
stabilized center frame $F_{i}$. The aligned image frames are fed as input to
the Margin-Net, as shown in Fig. 3 (c).

Inspired by the gated convolutions [Yu et al.(2019)Yu, Lin, Yang, Shen, Lu,
and Huang], we implement the pixel gates via convolutional layers that act as
spatial attention modules. Adopting gated convolutions leads to effectively
handling the arbitrary shapes of the boundary masks, which is beneficial
compared to the equivariant nature of vanilla convolutional layers.

As a result, the Margin-Net learns how to combine adjacent frames to fill in
the missing area of $F_{i}$, and produces the inpainted image $I_{i}^{m}$. The
Margin-Net concatenates the original $I_{i}^{m}$ and warped frames $F_{i}$ as
input to generate the inpainted frame $O_{i}$.

Figure 5: Overview of all loss functions for self-supervised training. Losses
for Coarse-Net (yellow box), Fine-Net (green box), and Margin-Net (blue box)
are shown. For each network, the original image $I_{i}$ is being self-
augmented using the random image transforms (gray box), and it is used for the
loss function shown in red. For the data augmentation, 256$\times$256 patches
are used for training from the original image in the training phase.

##  4 Self-Supervised Learning

To train Coarse-Net, Fine-Net, and Margin-Net appropriately, we propose a
novel self-supervised learning scheme. This distinguishes our approach with
supervised methods [Wang et al.(2018)Wang, Yang, Lin, Shamir, Zhang, Lu, and
Hu, Xu et al.(2018)Xu, Hu, Wang, Mu, and Hu]. We carefully design the training
methods for each module444It is worth to note that all three networks in our
pipeline can handle an arbitrary size of images – Coarse-Net and Fine-Net take
rescaled flow fields as input, and Margin-Net is fully convolutional. With
this in mind, we use smaller patches of $256\times 256$ pix. of the video
frames, to augment data and to reduce memory consumption during training. The
cropped frames can be flipped horizontally or vertically for further
augmentations., which is a crucial aspect of making our approach work. An
overview of the loss functions is depicted in Fig. 5.

###  4.1 Training Coarse-Net

To train the Coarse-Net, we generate a pair of an original image and a
randomly transformed image. The random transformation parameters $\mathbf{p}$
(e.g., rotation, translation) is sampled from a bounded uniform distribution
ranging up to 30 degrees and 50 pixels respectively.

Learning by random augmentation. As the first step, the image $I_{i}$ is
transformed via the random rigid transformation matrix, yielding
$\mathrm{T}(I_{i})$. Then, the flow map between $I_{i}$ and
$\mathrm{T}(I_{i})$ is computed and fed to the Coarse-Net which estimates the
parameter vector $\mathbf{p}^{\prime}$. $I_{i}$ is then transformed with
$\mathbf{p}^{\prime}$ and results in $C_{i}$. We use the L1 loss function
between the randomly transformed frame $\mathrm{T}(I_{i})$ and the estimated
transformed frame $C_{i}$ defined as:

${\color[rgb]{1,0,0}\mathcal{L}_{t}}=\left\|\mathrm{T}(I_{i})-C_{i}\right\|_{1}$
(2)

In addition, since we know the exact (randomly generated) ground truth (GT)
parameter values, we can penalize the discrepancy between the GT parameter
vector $\mathbf{p}$ and the estimated vector $\mathbf{p}^{\prime}$. The loss
is
${\color[rgb]{1,0,0}\mathcal{L}_{p}}=\left\|\mathbf{p}-\mathbf{p}^{\prime}\right\|_{1}$.

Learning by predicting rigid transformation. $\mathcal{L}_{t}$ and
$\mathcal{L}_{p}$ show losses using a rigidly transformed version of the same
frame. In addition to these synthetic augmentations, another type of loss
teaches how to roughly align temporally adjacent image pairs.

For this purpose, we collect adjacent frames of $I_{i}$ towards each of the
$K$ neighboring frames from both sides555We use $K=2$ in our experiment., such
as $I_{i-K},...,I_{i-1}$ and $I_{i+1},...,I_{i+K}$. In this training phase,
the optical flow is computed via inputs $(I_{i},I_{i+k})$, and the Coarse-Net
estimates the transform parameters $\mathbf{p}_{i+k}$. $\mathbf{p}_{i+k}$
transforms $I_{i}$ and produces $C_{i+k}$. Now we apply the sum of L1 losses
using $I_{i}$ and $C_{i+k}$,
${\color[rgb]{1,0,0}\mathcal{L}_{n}}=\sum\limits_{k\in\pm[1,K]}\left\|I_{i+k}-C_{i+k}\right\|_{1}$.
This loss function makes Coarse-Net to learn rigid transform parameters for
roughly aligning adjacent frames. This loss term increases the robustness for
handling dynamic scenes.

Thus, the overall training loss for the Coarse-Net is the weighted sum of the
three losses
$L_{C}=\alpha\mathcal{L}_{t}+\beta\mathcal{L}_{p}+\gamma\mathcal{L}_{n}$,
where $\alpha=\beta=1.0$ and $\gamma=0.1$.

###  4.2 Training Fine-Net

Learning by random augmentation. Training Fine-Net is similar to the training
steps for Coarse-Net. We reuse the randomly transformed image
$\mathrm{T}(I_{i})$ in Eq. (2) and compute the optical flow between
$\mathrm{T}(I_{i})$ and $I_{i}$. This flow map is fed through the Fine-Net,
producing a _smoothed flow map_. Frame $I_{i}$ is then warped by performing
the backward warping technique, using the computed smooth flow, producing the
warped frame $\textrm{W}(I_{i})$. The L1 loss is applied between
$\mathrm{T}(I_{i})$ and the estimated frame $\textrm{W}(I_{i})$, such as
${\color[rgb]{1,0,0}\mathcal{L}_{f}}=\left\|\mathrm{T}(I_{i})-\textrm{W}(I_{i})\right\|_{1}$.

This loss helps the network to learn a relatively simple warping map which
provides a warm start for training. Note that the _spatially smoothed_ flow
maps are produced by the Fine-Net’s architecture design, as shown in Fig. 3
(b).

Learning by predicting smoothed flow. The Fine-Net aims to estimate smooth
flow maps between neighboring frames and $I_{i}$. We apply the sum of L1
losses to the $K$ frames as follows:
${\color[rgb]{1,0,0}\mathcal{L}_{N}}=\sum\limits_{k\in\pm[1,K]}\left\|I_{i+k}-F_{i+k}\right\|_{1}$,
where $F_{i+k}$ is the frame $I_{i}$ warped towards $I_{i+k}$ using the Fine-
Net’s predicted smoothed flow. The overall training loss for the Fine-Net is
the weighted sum of the two losses
$\mathcal{L}_{F}=\alpha\mathcal{L}_{f}+\gamma\mathcal{L}_{N}$, where
$\alpha=1.0$ and $\gamma=0.1$.

###  4.3 Training Margin-Net

The task of the Margin-Net is to fill in natural content in the blank margins
of $F_{i}$, as shown in Fig. 3 (c). To train this network, we prepare a pair
of images, with or without blank margins. In order to obtain such images, we
reuse $\mathrm{T}(I_{i})$ in Eq. (2). In addition, the same random transform
is applied to the _full-sized_ image of the dataset, and then cropping is
applied. In contrast to $\mathrm{T}(I_{i})$, this procedure _preserves_ the
content at the frame boundaries, as shown in Fig 5. Let us call this image
$\textrm{TC}(I_{i})$.

Learning to inpaint. Using the stabilized frame $F_{i}$ and warped images of
$[I_{i-2}$, $I_{i-1}$, $I_{i+1}$, $I_{i+2}]$ as input, the Margin-Net outputs
an inpainted frame $I_{i}^{m}$, utilizing the information from the given
adjacent frames. The Margin-Net is trained by the L1 loss between the
transformed frame $\textrm{TC}(I_{i})$ and the inpainted image $I_{i}^{m}$:
${\color[rgb]{1,0,0}\mathcal{L}_{g}}=\left\|\textrm{TC}(I_{i})-I_{i}^{m}\right\|_{1}$.

Furthermore, we can isolate the loss to focus only on the inpainted region by
applying a mask to both $\textrm{TC}(I_{i})$ and $I_{i}^{m}$:
${\color[rgb]{1,0,0}\mathcal{L}_{l}}=\left\|m\odot(\textrm{TC}(I_{i})-I_{i}^{m})\right\|_{1}$,
where $m$ is the inpainting margin mask and $\odot$ denotes the element-wise
product. Since the stabilized frame $F_{i}$ is used as input, a mask
indicating the inpainting region can be computed via the estimated transform
parameter and flow map from the preceding Coarse and Fine-Net. Using these two
losses together increases the training stability:
$\mathcal{L}_{M}=\mathcal{L}_{g}+\mathcal{L}_{l}$.

##  5 Experiments

For thorough evaluation, we conduct an extensive quantitative comparison to
both offline and state-of-the-art real-time methods, visual comparisons,
analysis against a commercial product that runs offline, and ablation studies.
We also present results on full resolution videos, namely nHD
(640$\times$360), HD (1280$\times$720), and FHD (1920$\times$1080). For
approaches without public code, we refer to the reported numbers in each
paper. For quantitative comparison, we use three commonly used metrics [Liu et
al.(2017)Liu, Li, Zhu, and Zeng, Liu et al.(2016)Liu, Tan, Yuan, Sun, and
Zeng, Liu et al.(2013b)Liu, Yuan, Tan, and Sun, Wang et al.(2018)Wang, Yang,
Lin, Shamir, Zhang, Lu, and Hu, Xu et al.(2018)Xu, Hu, Wang, Mu, and Hu] to
assess video stabilization quality, namely the _cropping ratio_ , _distortion
value_ , and _stability score_. For details including implementation details,
please refer to the supplementary material.

Test videos. Prior arts use videos that are publicly available from Liu  _et
al_ [Liu et al.(2013b)Liu, Yuan, Tan, and Sun]. However, it is important to
note that _no prior work validates all videos_ provided by Liu  _et al_ [Liu
et al.(2013b)Liu, Yuan, Tan, and Sun] for video stabilization. Instead, each
method uses an arbitrary subset of the videos. To configure the same
experimental setup and to respect the numbers reported by state-of-the-arts,
we use the _union_ of the video sets that were tested by them [Goldstein and
Fattal(2012), Grundmann et al.(2011)Grundmann, Kwatra, and Essa, Liu et
al.(2009)Liu, Gleicher, Jin, and Agarwala, Liu et al.(2011)Liu, Gleicher,
Wang, Jin, and Agarwala, Liu et al.(2017)Liu, Li, Zhu, and Zeng, Liu et
al.(2016)Liu, Tan, Yuan, Sun, and Zeng, Liu et al.(2013b)Liu, Yuan, Tan, and
Sun, Liu et al.(2014)Liu, Yuan, Tan, and Sun, Wang et al.(2018)Wang, Yang,
Lin, Shamir, Zhang, Lu, and Hu, Xu et al.(2018)Xu, Hu, Wang, Mu, and Hu].
Therefore, some videos that are not tested by an approach is not displayed in
this paper.

Figure 6: Quantitative comparison with the state-of-the-art video
stabilization methods [Goldstein and Fattal(2012), Grundmann et
al.(2011)Grundmann, Kwatra, and Essa, Liu et al.(2009)Liu, Gleicher, Jin, and
Agarwala, Liu et al.(2011)Liu, Gleicher, Wang, Jin, and Agarwala, Liu et
al.(2017)Liu, Li, Zhu, and Zeng, Liu et al.(2016)Liu, Tan, Yuan, Sun, and
Zeng, Liu et al.(2013b)Liu, Yuan, Tan, and Sun, Liu et al.(2014)Liu, Yuan,
Tan, and Sun, Wang et al.(2018)Wang, Yang, Lin, Shamir, Zhang, Lu, and Hu, Xu
et al.(2018)Xu, Hu, Wang, Mu, and Hu]. The results are evaluated with the
three metrics: cropping ratio, distortion value, and stability score. Higher
values indicate better performance.  Figure 7: Visual comparison with
_offline_ approaches [Grundmann et al.(2012)Grundmann, Kwatra, Castro, and
Essa, Liu et al.(2016)Liu, Tan, Yuan, Sun, and Zeng, Liu et al.(2013b)Liu,
Yuan, Tan, and Sun, Wang et al.(2018)Wang, Yang, Lin, Shamir, Zhang, Lu, and
Hu] in terms of zoom-in effect. Pink bounding boxes of the same size are
magnified to show image regions at the same respective locations. Our approach
shows the least amount of distortion and zoom-in effect. Please see our
supplementary video for visual results.

The union of video set consists of 19 video clips, and it covers
forward/backward motion (#1-5, 9, 10, 18), side motion (#6, 7, 11, 12, 14)
zoom-in and zoom-out (#8, 17), the rolling-shutter artifact (#13, 15, 16, 19).
The thumbnails of the video clips are shown in Fig. 6. Note that our approach
is validated with the union of the video sets tested by all prior works, and
we emphasize that this is the _most extensive collection of comparisons_ in
the video stabilization literature. Furthermore, our network is trained on the
DAVIS dataset [Perazzi et al.(2016)Perazzi, Pont-Tuset, McWilliams, Van Gool,
Gross, and Sorkine-Hornung] different from the 19 test clips, which verifies
the generalization of our approach.

Evaluation result. As shown in Fig. 6, the comparison with offline methods
[Goldstein and Fattal(2012), Grundmann et al.(2011)Grundmann, Kwatra, and
Essa, Liu et al.(2009)Liu, Gleicher, Jin, and Agarwala, Liu et al.(2011)Liu,
Gleicher, Wang, Jin, and Agarwala, Liu et al.(2013b)Liu, Yuan, Tan, and Sun,
Liu et al.(2014)Liu, Yuan, Tan, and Sun], online (real-time) methods [Liu et
al.(2017)Liu, Li, Zhu, and Zeng, Liu et al.(2016)Liu, Tan, Yuan, Sun, and
Zeng, Wang et al.(2018)Wang, Yang, Lin, Shamir, Zhang, Lu, and Hu, Xu et
al.(2018)Xu, Hu, Wang, Mu, and Hu], and ours indicates that the proposed
method shows favorable performance for the majority of videos. In particular,
our real-time method shows comparable cropping ratio to the recent approach
proposed by Choi and Kweon [Choi and Kweon(2020)], although their approach
runs offline and is designed to prevent any cropping.

We also conduct comparisons with a widely used commercial product, Adobe
Premiere (Pro CC 2017), and two recent approaches by Choi and Kweon [Choi and
Kweon(2020)] and by Yu and Ramanmoorthi [Yu and Ramamoorthi(2020)]. Please
refer to the supplementary material.

Ablation study. We conduct an extensive analysis of stabilization scores with
various combination of proposed modules (Coarse-Net, Fine-Net, Margin-Net),
window sizes, regarding resizing the frame inputs, types of transformations
for Coarse-Net (translation, rotation, scale, and shear). Please see the
supplement for the details.

Visual comparison. Although it is not fair to directly compare our real-time
method to offline methods, we present visual comparisons to the state-of-the-
art offline methods [Grundmann et al.(2012)Grundmann, Kwatra, Castro, and
Essa, Liu et al.(2016)Liu, Tan, Yuan, Sun, and Zeng, Liu et al.(2013b)Liu,
Yuan, Tan, and Sun, Wang et al.(2018)Wang, Yang, Lin, Shamir, Zhang, Lu, and
Hu] in Fig. 7. Notice that our method closely resembles the input content,
while state-of-the-art methods convey enlargements due to the zoom-in effect.
Furthermore, we can observe moderate levels of blur introduced by the other
methods, whereas our approach does not exhibit such artifacts. For detailed
visual results, please refer to the supplement.

##  6 Conclusion

We propose an unsupervised learning algorithm for video stabilization that
runs in real-time. Our approach consists of simple and efficient modules, such
as Coarse-Net for image transform estimation, and Fine-Net for estimating the
smoothed flow, and Margin-Net to compensate cropped contents. By combining all
the modules, our approach outperforms both offline and real-time state-of-the-
art methods on various videos (that shows zoom, rotation, parallax, and so on)
and is robust to severe camera movements.

##  7 Acknowledgement

This work was supported by the Institute for Information & Communications
Technology Promotion (2017-0-01772) grant funded by the Korea government. This
work was also supported by Institute of Information and communications
Technology Planning and evaluation (IITP) grant funded by the Korea government
(MSIT) (No.2021-0-02068, AI Innovation Hub)

## References

  * [Bell et al.(2014)Bell, Troccoli, and Pulli] Steven Bell, Alejandro Troccoli, and Kari Pulli.  A non-linear filter for gyroscope-based video stabilization.  In _European Conference on Computer Vision (ECCV)_ , 2014.
  * [Buehler et al.(2001)Buehler, Bosse, and McMillan] Chris Buehler, Michael Bosse, and Leonard McMillan.  Non-metric image-based rendering for video stabilization.  In _IEEE Computer Vision and Pattern Recognition (CVPR)_ , volume 2, pages 609–614, 2001.
  * [Chen et al.(2008)Chen, Lee, Huang, and Lin] Bing-Yu Chen, Ken-Yi Lee, Wei-Ting Huang, and Jong-Shan Lin.  Capturing intention-based full-frame video stabilization.  In _Computer Graphics Forum_ , volume 27, pages 1805–1814, 2008.
  * [Choi and Kweon(2020)] Jinsoo Choi and In So Kweon.  Deep iterative frame interpolation for full-frame video stabilization.  _ACM Transactions on Graphics (TOG)_ , 39(1):1–9, 2020.
  * [DeTone et al.(2016)DeTone, Malisiewicz, and Rabinovich] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Deep image homography estimation.  _arXiv preprint arXiv:1606.03798_ , 2016.
  * [Gleicher and Liu(2007)] Michael L Gleicher and Feng Liu.  Re-cinematography: improving the camera dynamics of casual video.  In _ACM Multimedia_ , pages 27–36, 2007.
  * [Goldstein and Fattal(2012)] Amit Goldstein and Raanan Fattal.  Video stabilization using epipolar geometry.  _ACM Transactions on Graphics (TOG)_ , 31(5):126, 2012.
  * [Grundmann et al.(2011)Grundmann, Kwatra, and Essa] Matthias Grundmann, Vivek Kwatra, and Irfan Essa.  Auto-directed video stabilization with robust l1 optimal camera paths.  In _IEEE Computer Vision and Pattern Recognition (CVPR)_ , pages 225–232, 2011.
  * [Grundmann et al.(2012)Grundmann, Kwatra, Castro, and Essa] Matthias Grundmann, Vivek Kwatra, Daniel Castro, and Irfan Essa.  Calibration-free rolling shutter removal.  In _IEEE ICCP_ , pages 1–8, 2012.
  * [Huang et al.(2018)Huang, Wei, and Zhang] Hua Huang, Xiao-Xiang Wei, and Lei Zhang.  Encoding shaky videos by integrating efficient video stabilization.  _IEEE TCSVT_ , 2018.
  * [Karpenko et al.(2011)Karpenko, Jacobs, Baek, and Levoy] Alexandre Karpenko, David Jacobs, Jongmin Baek, and Marc Levoy.  Digital video stabilization and rolling shutter correction using gyroscopes.  _CSTR_ , 1:2, 2011.
  * [Liu et al.(2009)Liu, Gleicher, Jin, and Agarwala] Feng Liu, Michael Gleicher, Hailin Jin, and Aseem Agarwala.  Content-preserving warps for 3d video stabilization.  _ACM Transactions on Graphics (TOG)_ , 28(3):44, 2009.
  * [Liu et al.(2011)Liu, Gleicher, Wang, Jin, and Agarwala] Feng Liu, Michael Gleicher, Jue Wang, Hailin Jin, and Aseem Agarwala.  Subspace video stabilization.  _ACM Transactions on Graphics (TOG)_ , 30(1):4, 2011.
  * [Liu et al.(2013a)Liu, Niu, and Jin] Feng Liu, Yuzhen Niu, and Hailin Jin.  Joint subspace stabilization for stereoscopic video.  In _IEEE International Conference on Computer Vision (ICCV)_ , pages 73–80, 2013a.
  * [Liu et al.(2018)Liu, Reda, Shih, Wang, Tao, and Catanzaro] Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro.  Image inpainting for irregular holes using partial convolutions.  In _European Conference on Computer Vision (ECCV)_ , 2018.
  * [Liu et al.(2012)Liu, Wang, Yuan, Bu, Tan, and Sun] Shuaicheng Liu, Yinting Wang, Lu Yuan, Jiajun Bu, Ping Tan, and Jian Sun.  Video stabilization with a depth camera.  In _IEEE Computer Vision and Pattern Recognition (CVPR)_ , pages 89–95, 2012.
  * [Liu et al.(2013b)Liu, Yuan, Tan, and Sun] Shuaicheng Liu, Lu Yuan, Ping Tan, and Jian Sun.  Bundled camera paths for video stabilization.  _ACM Transactions on Graphics (TOG)_ , 32(4):78, 2013b.
  * [Liu et al.(2014)Liu, Yuan, Tan, and Sun] Shuaicheng Liu, Lu Yuan, Ping Tan, and Jian Sun.  Steadyflow: Spatially smooth optical flow for video stabilization.  In _IEEE Computer Vision and Pattern Recognition (CVPR)_ , pages 4209–4216, 2014.
  * [Liu et al.(2016)Liu, Tan, Yuan, Sun, and Zeng] Shuaicheng Liu, Ping Tan, Lu Yuan, Jian Sun, and Bing Zeng.  Meshflow: Minimum latency online video stabilization.  In _European Conference on Computer Vision (ECCV)_ , pages 800–815, 2016.
  * [Liu et al.(2017)Liu, Li, Zhu, and Zeng] Shuaicheng Liu, Mingyu Li, Shuyuan Zhu, and Bing Zeng.  Codingflow: enable video coding for video stabilization.  _IEEE Transactions on Image Processing (TIP)_ , 26(7):3291–3302, 2017.
  * [Liu et al.(2021)Liu, Lai, Yang, Chuang, and Huang] Yu-Lun Liu, Wei-Sheng Lai, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang.  Hybrid neural fusion for full-frame video stabilization.  _IEEE Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [Matsushita et al.(2006)Matsushita, Ofek, Ge, Tang, and Shum] Yasuyuki Matsushita, Eyal Ofek, Weina Ge, Xiaoou Tang, and Heung-Yeung Shum.  Full-frame video stabilization with motion inpainting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)_ , 2006.
  * [Nguyen et al.(2018)Nguyen, Chen, Shivakumar, Taylor, and Kumar] Ty Nguyen, Steven W Chen, Shreyas S Shivakumar, Camillo Jose Taylor, and Vijay Kumar.  Unsupervised deep homography: A fast and robust homography estimation model.  _IEEE Robotics and Automation Letters_ , 3(3):2346–2353, 2018.
  * [Ovrén and Forssén(2015)] Hannes Ovrén and Per-Erik Forssén.  Gyroscope-based video stabilisation with auto-calibration.  In _IEEE International Conference on Robotics and Automation (ICRA)_ , pages 2090–2097, 2015.
  * [Perazzi et al.(2016)Perazzi, Pont-Tuset, McWilliams, Van Gool, Gross, and Sorkine-Hornung] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  In _IEEE Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [Ronneberger et al.(2015)Ronneberger, Fischer, and Brox] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical Image Computing and Computer Assisted Intervention_ , pages 234–241. Springer, 2015.
  * [Smith et al.(2009)Smith, Zhang, Jin, and Agarwala] Brandon M Smith, Li Zhang, Hailin Jin, and Aseem Agarwala.  Light field video stabilization.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2009\.
  * [Smith et al.(1997)] Steven W Smith et al.  The scientist and engineer’s guide to digital signal processing.  1997\.
  * [Wang et al.(2018)Wang, Yang, Lin, Shamir, Zhang, Lu, and Hu] Miao Wang, Guo-Ye Yang, Jin-Kun Lin, Ariel Shamir, Song-Hai Zhang, Shao-Ping Lu, and Shi-Min Hu.  Deep online video stabilization with multi-grid warping transformation learning.  _IEEE Transactions on Image Processing (TIP)_ , 2018.
  * [Xu et al.(2018)Xu, Hu, Wang, Mu, and Hu] Sen-Zhe Xu, Jun Hu, Miao Wang, Tai-Jiang Mu, and Shi-Min Hu.  Deep video stabilization using adversarial networks.  In _Computer Graphics Forum_ , volume 37, pages 267–276, 2018.
  * [Yu et al.(2019)Yu, Lin, Yang, Shen, Lu, and Huang] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Free-form image inpainting with gated convolution.  _IEEE International Conference on Computer Vision (ICCV)_ , 2019.
  * [Yu and Ramamoorthi(2020)] Jiyang Yu and Ravi Ramamoorthi.  Learning video stabilization using optical flow.  In _IEEE Computer Vision and Pattern Recognition (CVPR)_ , pages 8159–8167, 2020.
  * [Zhou et al.(2013)Zhou, Jin, and Ma] Zihan Zhou, Hailin Jin, and Yi Ma.  Plane-based content preserving warps for video stabilization.  In _IEEE Computer Vision and Pattern Recognition (CVPR)_ , pages 2299–2306, 2013.


§ INTRODUCTION

§ BACKGROUND & RELATED WORK

§ ADAPTIVE FINE-TUNING

§ EXPERIMENTS

§ RESULTS

§ SUMMARY

[1]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe
Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien
Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest,
and Alexander M. Rush.
Transformers: State-of-the-art natural language processing.
In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations, pages 38–45, Online,
October 2020. Association for Computational Linguistics.

[2]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language
understanding, 2019.

[3]
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi.
Revisiting few-sample bert fine-tuning, 2021.

[4]
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow.
On the stability of fine-tuning bert: Misconceptions, explanations,
and strong baselines, 2021.

[5]
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi,
and Noah Smith.
Fine-tuning pretrained language models: Weight initializations, data
orders, and early stopping, 2020.

[6]
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith.
Show your work: Improved reporting of experimental results, 2019.

[7]
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning.
Electra: Pre-training text encoders as discriminators rather than
generators, 2020.

[8]
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang.
Mixout: Effective regularization to finetune large-scale pretrained
language models, 2020.

[9]
Erik F. Tjong Kim Sang and Fien De Meulder.
Introduction to the CoNLL-2003 shared task: Language-independent
named entity recognition.
In Proceedings of the Seventh Conference on Natural Language
Learning at HLT-NAACL 2003, pages 142–147, 2003.

[10]
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.
Distilbert, a distilled version of bert: smaller, faster, cheaper and
ArXiv, abs/1910.01108, 2019.

[11]

[12]
Martin Malmsten, Love Börjeson, and Chris Haffenden.
Playing with words at the national library of sweden – making a
swedish bert, 2020.

[13]
Lars Ahrenberg, Johan Frid, and Leif-Joran Olsson.
A new resource for swedish named-entity recognition, 2020.

[14]
Alejandro Piad-Morffis, Yoan Gutiérrez, Hian Cañizares-Diaz,
Suilan Estevez-Velarde, Yudivián Almeida-Cruz, Rafael Muñoz,
and Andrés Montoyo.
Overview of the ehealth knowledge discovery challenge at iberlef

[15]

[16]
Lance A. Ramshaw and Mitchell P. Marcus.
Text chunking using transformation-based learning, 1995.

[17]
Felix Stollenwerk.
nerblackbox: a python package to fine-tune transformer-based language
models for named entity recognition, 2021.

[18]

[19]

[20]
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon
Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
Scaling laws for neural language models, 2020.

[21]

[22]
Skipper Seabold and Josef Perktold.
statsmodels: Econometric and statistical modeling with python.
In 9th Python in Science Conference, 2010.

§ HYPERPARAMETERS

§ NOTATION

§ MODEL EVALUATION

§ REPRODUCTION OF LITERATURE RESULTS

§ COMPUTATIONAL EFFORT

§ RESULTS: GENERALIZATION

§ VARIANTS OF THE ADAPTIVE FINE-TUNING APPROACH

§ ABLATION STUDIES

§ LEARNING CURVES

§ DEPENDENCY OF THE MODEL PERFORMANCE ON THE TRAINING DATASET SIZE AND TRAINING EPOCHS

§ DEPENDENCY OF THE MODEL PERFORMANCE ON THE VALIDATION DATASET SIZE

§ ADAPTIVE FINE-TUNING IN PRACTICE: USING THE VALIDATION DATASET FOR TRAINING