11institutetext: University of British Columbia, Vancouver, BC, Canada
22institutetext: University of Victoria, Victoria, BC, Canada 33institutetext:
University of Campinas, Campinas, Brazil 44institutetext: Simon Fraser
University, Burnaby, BC, Canada
44email<EMAIL_ADDRESS>
#  BiasPruner: Debiased Continual Learning for Medical Image Classification

Nourhan Bayasi 11 0000-0003-4653-6081 Jamil Fayyad 22 0000-0003-1553-8754
Alceu Bissoto 33 0000-0003-2293-6160 Ghassan Hamarneh 44 0000-0001-5040-7448
Rafeef Garbi 11 0000-0001-6224-0876

###### Abstract

Continual Learning (CL) is crucial for enabling networks to dynamically adapt
as they learn new tasks sequentially, accommodating new data and classes
without catastrophic forgetting. Diverging from conventional perspectives on
CL, our paper introduces a new perspective wherein forgetting could actually
benefit the sequential learning paradigm. Specifically, we present BiasPruner,
a CL framework that intentionally forgets spurious correlations in the
training data that could lead to shortcut learning. Utilizing a new bias score
that measures the contribution of each unit in the network to learning
spurious features, BiasPruner prunes those units with the highest bias scores
to form a debiased subnetwork preserved for a given task. As BiasPruner learns
a new task, it constructs a new debiased subnetwork, potentially incorporating
units from previous subnetworks, which improves adaptation and performance on
the new task. During inference, BiasPruner employs a simple task-agnostic
approach to select the best debiased subnetwork for predictions. We conduct
experiments on three medical datasets for skin lesion classification and chest
X-Ray classification and demonstrate that BiasPruner consistently outperforms
SOTA CL methods in terms of classification performance and fairness. Our code
is available here.

###### Keywords:

Continual Learning Debias Pruning Shortcut Learning.

##  1 Introduction

Humans inherently learn in a continual manner, acquiring new concepts over
time without forgetting previous ones. In contrast, deep learning models
encounter the challenge of catastrophic forgetting [17], wherein learning new
data can override previously acquired knowledge. This issue becomes especially
pronounced within the medical domain, given the ever-evolving nature of
medical data, the variations in acquisition protocols, the utilization of
diverse devices for obtaining medical images, and other factors that
contribute to shifts in data distributions or the introduction of new disease
classes over time. As a result, continual learning (CL) [25, 27] has emerged
as a promising solution, allowing a network to learn continually over a
sequence of presented data while forgetting as little as possible about
previous knowledge. Several CL methods have emerged within the medical field
to address the challenge of forgetting. Replay-based methods [22, 13] store a
subset of data samples and replay them to retain old information,
regularization-based methods [15] impose restrictions on the network parameter
updates to preserve prior knowledge while learning new tasks, and
architecture-based methods, assign specialized architectural components for
each task within the network [1, 2] or expand them to accommodate new tasks
[9].

While previous CL methods achieved success, they have yet to consider a more
realistic setting in which dataset bias exists. In medical imaging, bias could
manifest through an imbalanced distribution of sensitive attributes (e.g.,
gender, age, ethnicity) [4]. Even slight imbalances induce spurious
correlations between attributes and the classification target (diagnosis) [3],
creating an illusion of predictive power that models can exploit. Leveraging
such information compromises the network’s generalization ability, amplifying
societal biases over misrepresented populations in data (e.g., detecting
melanoma in individuals with dark skin tones. In CL, learning spurious
correlations poses a significant challenge due to bias transfer, where biases
learned by a model can be transferred to a downstream task even if it has
unbiased data [23]. Since CL involves learning a sequence of tasks, the bias
transfer can potentially be amplified. Moreover, recent work [5]
mathematically proved that handling bias becomes substantially harder when
tasks are presented sequentially compared to joint training.

To address this gap and tackle bias in CL, we propose BiasPruner, a fixed-size
network capable of learning sequentially and fairly over time by dedicating a
unique debiased subnetwork for each task. BiasPruner leverages a newly
proposed bias score to measure the contribution of each unit in the network to
learning spurious features. Units with high bias scores are pruned to form a
task-specific debiased subnetwork, which is kept frozen to avoid forgetting,
whereas the remaining pruned units are subsequently offered for learning new
tasks. Fig. 1 presents an overview of our method. We evaluate our solution on
three medical imaging classification datasets, each with different bias
attributes. Our results demonstrate BiasPruner’s superior performance in both
classification accuracy and fairness. While a few recent methods have
addressed fairness in CL [14, 16, 6], BiasPruner, to the best of our
knowledge, is the first work in the medical field covering different
benchmarks and bias attributes in a class-incremental setup. Crucially,
BiasPruner does not require dataset biases to be explicitly annotated. This is
particularly relevant in healthcare, where identifying biases is complex and
costly, compounded by patient data privacy concerns [19].

Figure 1: (Left) BiasPruner learns sequentially, allocating a subnetwork for
each task. (Right) BiasPruner evaluates each network unit’s contribution to
learning spurious features from biased training data, assigning bias scores.
High-score units are pruned, and the subnetwork is finetuned on both easy and
hard samples.

##  2 Methodology

BiasPruner employs a fixed-size network, $f$, capable of learning $T$ tasks
sequentially, one at a time, where $T$ is not pre-determined, without
forgetting any of the previously learned tasks. During training the $t$-th
domain, where $t\in\\{1,2,\ldots,T\\}$, the network does not have access to
old data, i.e., it exclusively receives biased training data
$D_{t}={(x_{i},y_{i})}$ specific to the current task, where ${(x_{i},y_{i})}$
represent the training samples, consisting of a total of $N_{t}$ images and
$y_{i}\in\mathcal{C}_{t}$ classes (note the subscript $t$ emphasizing that the
set of classes may change, including adding new classes, for new tasks). For
clarity, we employ the symbol $c$ to denote any class within the set
$\mathcal{C}_{t}$. BiasPruner creates a debiased subnetwork for the $t$-th
task by pruning units in the network that are mostly correlated with unknown
bias(es) in $D_{t}$. Furthermore, BiasPruner transfers knowledge through
pruning of the original network, including units of previously created
subnetworks, for each new task. At inference, BiasPruner identifies the
optimal subnetwork for predictions on a given data in a task-agnostic setup;
i.e., information about the task origin of a test image is unknown or
unavailable.

2.1. Detecting Spurious Features through Bias Scoring. Given a biased dataset
$D_{t}$, one of the key causes of learning shortcut predictions occurs when
the model finds it easier to learn spurious features rather than the intended
ones [19]. Consequently, we propose to intentionally encourage the network $f$
to quickly fit on the easier features from the training data of $D_{t}$. To
achieve this, we adopt the generalized cross entropy (GCE) [29] loss,
$\mathcal{L}_{\mathrm{GCE}}$, which was originally proposed to address noisy
labels by fitting on the easier clean data and slowly memorizing the hard
noisy samples. The GCE loss is formulated as follows:

$\mathcal{L}_{\mathrm{GCE}}(p(x;\theta),y)=\frac{1-p_{y}(x;\theta)^{q}}{q}$

where $q\in(0,1]$ is a hyperparameter controlling the degree of bias
amplification, $p(x;\theta)$ and $p_{y}(x;\theta)$ are the softmax output of
the network and its probability assigned to the target label $y$,
respectively. Due to the GCE loss’s gradient, which up-weights samples with a
high probability of predicting the correct target, the network quickly becomes
biased to easier samples and learns shortcuts [21].

Once the network is biased, it becomes logical to identify the units that have
contributed the most to learning the shortcut in each class. To achieve this,
we partition the training data $\\{x,y\\}$ into two groups per groundtruth
class $c$: The biased sample set, $\mathcal{E}_{c}^{t}$, consists of
$(x_{i},y_{i})$ pairs that are correctly classified by the biased network with
a probability $p_{y,i}\geq\tau$; i.e., samples that are $\mathcal{E}$asier for
the network to learn. Similarly, the unbiased sample set,
$\mathcal{H}_{c}^{t}$, comprises $(x_{i},y_{i})$ pairs that are misclassified
by the biased network; i.e., samples that are $\mathcal{H}$arder to learn, as
follows (Refer to supplementary material, Fig. 4, for visualizations):

$\mathcal{E}_{c}^{t}=\\{i\,|\,y_{i}=c_{i}\,~{}\&~{}\,p_{y,i}\geq\tau\\}~{}~{},~{}~{}\mathcal{H}_{c}^{t}=\\{i\,|\,y_{i}\neq
c_{i}\,~{}\&~{}\,p_{y,i}\geq\tau\\}.$

Next, a we define a bias score, $\mathcal{S}^{t}_{c,n}$, for each unit $n$ in
the biased network relative to a given class $c$ by analyzing each unit’s ReLU
activation, $a_{i}^{n}$, as follows:

$\mathcal{S}^{t}_{c,n}=\frac{1}{\left|\mathcal{E}_{c}^{t}\right|}\sum_{i\in\mathcal{E}_{c}^{t}}\text{Var}\left(a_{i}^{n}\right)-\frac{1}{\left|\mathcal{H}_{c}^{t}\right|}\sum_{i\in\mathcal{H}_{c}^{t}}\text{Var}\left(a_{i}^{n}\right).$

$\text{Var}\left(a_{i}^{n}\right)$ represents the variance of the feature map
$a_{i}^{n}$ over its spatial dimensions $(w,h)$. The final unit-based bias
score $\bar{\mathcal{S}^{t}_{n}}$ is calculated by averaging the results over
all class-specific scores. Units that respond more strongly to biased samples
(c.f. $\mathcal{E}_{c}^{t}$) than to unbiased samples (c.f.
$\mathcal{H}_{c}^{t}$) are assigned higher bias scores, designating them as
the main contributors to learning shortcuts in the network.

2.2 Forming Subnetworks by Bias-aware Pruning and Finetuning. To ensure
fairness in CL, we form a task-specific, debiased subnetwork, ${f}_{t}$, by
selectively removing the units responsible for learning the bias in $D_{t}$.
The pruning involves removing the top $\gamma\%$ of units, which includes the
output feature maps with the highest bias scores and their corresponding
filters, leaving ($1-\gamma\%$) for each $f_{t}$. To counteract potential
performance drop post-pruning while prioritizing improved performance on
harder-to-learn samples, we propose a new weighted cross entropy loss,
$\mathcal{L}_{\mathrm{WCE}}$, for fine-tuning $f_{t}$ on $D_{t}$ over a few
epochs:

$\mathcal{L}_{\mathrm{WCE}}(x)=\mathcal{W}(x)\cdot\mathcal{L}_{\mathrm{CE}}\left({f}(x),y\right),~{}~{}~{}\text{where}~{}~{}~{}\mathcal{W}(x)=\exp\left(\alpha\cdot\mathcal{L}_{\mathrm{GCE}}(x)\right).$

$\alpha\in(0,1)$ is a trainable parameter, and $\mathcal{L}_{\mathrm{GCE}}(x)$
is the sample’s GCE loss value determined as discussed in Sec. 2.1. With this
weighted function, the influence of training samples in the finetuning process
varies according to their bias alignment; i.e., easy samples (c.f.,
$\mathcal{L}_{\mathrm{GCE}}(x)$ is small) are down-weighted, whereas hard
samples (c.f., $\mathcal{L}_{\mathrm{GCE}}(x)$ is large) are up-weighted,
exponentially.

2.3. Debiased Knowledge Transfer for Enhanced Task Adaptation. When learning a
new task, BiasPruner facilitates knowledge transfer (KT), which is achieved by
pruning the entire original network $f$ to create the new task-specific
subnetwork, including both free units and pre-assigned debiased subnetworks of
previous tasks. To avoid forgetting the previously acquired knowledge, the
subnetworks associated with prior tasks are kept frozen and only the free
units are updated to learn the new task.

2.4. Task-agnostic Inference. BiasPruner addresses a practical scenario where
the task identity of a test image is unknown during inference. In other words,
the specific task to which an image belongs to is not explicitly provided.
Given a test batch of size $s$ as $\mathbf{X}^{\text{test }}$, we employ a
‘maxoutput’ strategy for task prediction [7], which involves identifying the
task with the maximum output response:
$t^{*}=\underset{t=1,2,\ldots,T}{\arg\max}\sum_{i=1}^{s}\max\varphi_{t}\left(\theta_{t}\left(\mathbf{x}_{i}^{test}\right)\right)$,
where $\varphi_{t}$ is the fully connected layer of the $t$-th subnetwork.
Subsequently, we use the selected $t^{*}$ task to make the final prediction
$\hat{\mathbf{y}}$ based on the corresponding subnetwork;
$\hat{\mathbf{y}}={f}_{t^{*}}\left(\mathbf{X}^{\text{test }}\right)$.

##  3 Experiments and Results

Datasets. We selected/constructed datasets based on three primary
considerations: (a) the presence of a dataset bias that is spuriously
correlated with the disease classes; (b) The need for a variety of classes to
facilitate the CL setup; and (c) publicly available to ensure reproducibility.
Hence, we include Fitzpatrick17K (FITZ) [10], HAM10000 (HAM) [24] and NIH
ChestX-Ray14 (NIH) [26]. Each dataset has 114, 7 and 14 distinct classes,
respectively, that are split into 6, 3 and 3 tasks, respectively, with non-
overlapping classes, as shown in Table 1 (Refer to supplementary material for
dataset (Table 5) $\&$ bias (Fig. 5) details).

Table 1: Details on the multi-class disease datasets used in our experiments.  Dataset |  Number of images | Classes |  Tasks | Classes per task | Dataset bias
---|---|---|---|---|---
FITZ | 16,012 | 114 | 6 | $[19,19,19,19,19,19]$ |  Skin tone (I, II, III, IV, V, VI)
HAM | 8,678 | 7 | 3 | $[2,2,3]$ |  Age (age$\geq$60, age$<$60)
NIH | 19,993 | 14 | 3 | $[4,5,5]$ | Gender (male, female)

Evaluation Metrics. We assess the performance of BiasPruner using both the
accuracy and fairness metrics. We use the commonly used F1-score (F) and
balanced accuracy (ACC) metrics. We report the accuracy per sensitive
attribute (e.g., male, female) as well as overall class performance (Overall).
For fairness, we use the demographic parity ratio (DPR) and equal opportunity
difference (EOD) metrics. Similar to other CL methods, we report all metrics
at the end of learning (i.e., after training the model on all $T$ tasks),
averaged across all tasks.

Implementation Details. We use ResNet-50 [11] as the backbone for feature
extraction and a unified classifier for all tasks during inference. We use the
Adam optimizer with a batch of 32 images for 200 epochs to train BiasPruner
with $\mathcal{L}_{\mathrm{GCE}}$, having early stopping in case of
overfitting. We set $q$ in $\mathcal{L}_{\mathrm{GCE}}$ to 0.7 (default) and
the confidence threshold to $\tau=0.70$. We set the pruning ratio to
$\gamma=0.6$ for all tasks. For the finetuning with
$\mathcal{L}_{\mathrm{WCE}}$, we train the debiased subnetwork for 20 epochs,
and we saved the weights with the highest ACC and EOD on the validation set.
In all experiments, we report averaged results across three random task
orders, aiming to neutralize any potential impact of the order in which tasks
are processed during network training.

I. Quantitative Results on Skin-tone-biased Dataset (FITZ) are reported in
Table 2. First, we compare BiasPruner (Exp $\mathcal{D}$) against three common
CL baselines (Exp $\mathcal{A}$): JOINT, which consolidates data from all
tasks for joint model training; SINGLE, which trains separate models for each
task and deploys task-specific models during inference; and SeqFT, which
finetunes a single model on the current task without addressing forgetting. We
observe that SINGLE outperforms JOINT as each task is learned independently,
leading to improved classification and fairness results, and that SeqFT
exhibits a significant performance drop due to catastrophic forgetting.
Notably, BiasPruner outperforms baselines in terms of overall accuracy and
fairness, attributing this to its ability to reduce the training data bias and
transfer knowledge across the tasks.

Table 2: Classification performance and fairness on FITZ. Best results marked in bold (except upper-bound). Higher is better for all metrics except EOD. Exp | Method | F | ACC | DPR | EOD
---|---|---|---|---|---
Type-I | Type-II | Type-III | Type-IV | Type-V | Type-VI | Overall
Comparison against Baselines
$\mathcal{A}$ | JOINT | 0.256 | 0.269 | 0.304 | 0.335 | 0.309 | 0.365 | 0.245 | 0.324 | 0.137 | 0.298
SINGLE | 0.435 | 0.410 | 0.469 | 0.465 | 0.495 | 0.492 | 0.430 | 0.472 | 0.185 | 0.251
SeqFT | 0.188 | 0.187 | 0.261 | 0.299 | 0.254 | 0.214 | 0.192 | 0.221 | 0.051 | 0.721
Comparison against CL Methods
$\mathcal{B}$ | EWC | 0.325 | 0.254 | 0.356 | 0.355 | 0.401 | 0.412 | 0.244 | 0.324 | 0.212 | 0.342
PackNet | 0.433 | 0.366 | 0.402 | 0.445 | 0.447 | 0.479 | 0.319 | 0.414 | 0.154 | 0.425
SupSup | 0.451 | 0.254 | 0.298 | 0.441 | 0.452 | 0.436 | 0.410 | 0.425 | 0.162 | 0.431
Comparison against CL with Bias Mitigation Methods
$\mathcal{C}$ | EWC+S | 0.308 | 0.264 | 0.357 | 0.324 | 0.411 | 0.417 | 0.385 | 0.341 | 0.228 | 0.311
PackNet+S | 0.495 | 0.434 | 0.485 | 0.494 | 0.565 | 0.562 | 0.584 | 0.501 | 0.184 | 0.248
SupSup+S | 0.466 | 0.418 | 0.467 | 0.432 | 0.554 | 0.561 | 0.534 | 0.492 | 0.182 | 0.221
EWC+W | 0.321 | 0.251 | 0.356 | 0.334 | 0.392 | 0.401 | 0.398 | 0.346 | 0.216 | 0.298
PackNet+W | 0.527 | 0.405 | 0.477 | 0.480 | 0.529 | 0.546 | 0.524 | 0.472 | 0.144 | 0.246
SupSup+W | 0.457 | 0.425 | 0.451 | 0.448 | 0.530 | 0.561 | 0.544 | 0.508 | 0.178 | 0.254
Our Proposed Fair Continual Learning Method
$\mathcal{D}$ | BiasPruner | 0.540 | 0.457 | 0.502 | 0.435 | 0.551 | 0.563 | 0.584 | 0.512 | 0.331 | 0.202
[Upper-bound] Comparison against a Bias Mitigation Method
$\mathcal{E}$ | FairDisCo | 0.542 | 0.479 | 0.523 | 0.468 | 0.571 | 0.574 | 0.615 | 0.548 | 0.474 | 0.192

Secondly, we compare BiasPruner against three CL methods (Exp $\mathcal{B}$):
EWC [12], a regularization-based method, and PackNet [20] and SupSup [28],
both subnetwork-based like ours. We notice that these CL methods demonstrate
lower fairness compared to BiasPruner, which is expected as they overlook
dataset bias. Specifically, PackNet and SupSup exhibit higher accuracy but
lower fairness compared to EWC. This is mainly due to their subnetwork-based
nature, which can inadvertently worsen accuracy disparities, particularly
among specific subgroups, during the removal of unimportant parameters [18].

Thirdly, we enhance the competing CL methods by augmenting each of them with
pre-processing bias mitigation algorithms (Exp $\mathcal{C}$). Specifically,
we apply the Resampling Algorithm (S), which balances the dataset by
oversampling minorities and undersampling majorities within each pair of skin
label and tone. Additionally, we explore the Reweighting Algorithm (W) [8],
which assigns lower weights to images that have been disadvantaged or favored
to prevent the model from learning discriminatory features. While showing
improved accuracy and fairness compared to Exp $\mathcal{B}$, they fall short
of our BiasPruner’s performance.

Finally, we compare BiasPruner to FairDisCo [8], a (non-CL) bias mitigation
technique for medical applications, which uses bias annotations in training.
Therefore, it can set an upper bound on the performance. For a fair
comparison, we allow FairDisCo to learn each task independently and report the
average performance over all tasks (Exp $\mathcal{E}$). Despite not using bias
annotations, BiasPruner exhibits slightly lower but comparable performance to
FairDisCo.

II. Quantitative Results on Age- and Gender-biased Dataset (HAM & NIH,
respectively) are given in Table 3. BiasPruner (Exp $\mathcal{I}$) outperforms
other baselines (Exp $\mathcal{F}$), CL methods (Exp $\mathcal{G}$), and CL
methods with debiasing (Exp $\mathcal{H}$) in both overall task classification
and fairness.

Table 3: Classification performance and fairness on HAM and NIH. Best results marked in bold (except upper-bound).  Exp | Method | HAM | NIH
---|---|---|---
F | ACC | DPR | EOD | F | ACC | DPR | EOD
$<$60 |  $\geq$60 | Overall | M | F | Overall
Comparison against Baselines
$\mathcal{F}$ | JOINT | 0.755 | 0.781 | 0.665 | 0.738 | 0.239 | 0.320 | 0.282 | 0.306 | 0.259 | 0.285 | 0.706 | 0.325
SINGLE | 0.836 | 0.819 | 0.834 | 0.841 | 0.609 | 0.131 | 0.434 | 0.428 | 0.403 | 0.417 | 0.728 | 0.311
SeqFT | 0.431 | 0.372 | 0.404 | 0.416 | 0.201 | 0.558 | 0.219 | 0.251 | 0.217 | 0.231 | 0.246 | 0.544
Comparison against CL Methods
$\mathcal{G}$ | EWC | 0.788 | 0.773 | 0.804 | 0.772 | 0.561 | 0.360 | 0.398 | 0.428 | 0.405 | 0.417 | 0.562 | 0.264
PackNet | 0.824 | 0.807 | 0.799 | 0.808 | 0.620 | 0.302 | 0.434 | 0.47 | 0.444 | 0.458 | 0.588 | 0.284
SupSup | 0.831 | 0.788 | 0.845 | 0.822 | 0.625 | 0.296 | 0.448 | 0.451 | 0.441 | 0.445 | 0.571 | 0.293
Comparison against CL with Bias Mitigation Methods
$\mathcal{H}$ | EWC+S | 0.834 | 0.821 | 0.832 | 0.827 | 0.575 | 0.172 | 0.412 | 0.434 | 0.416 | 0.421 | 0.567 | 0.259
PackNet+S | 0.839 | 0.849 | 0.817 | 0.829 | 0.613 | 0.181 | 0.419 | 0.44 | 0.425 | 0.434 | 0.640 | 0.211
SupSup+S | 0.849 | 0.802 | 0.811 | 0.817 | 0.639 | 0.204 | 0.432 | 0.456 | 0.448 | 0.451 | 0.662 | 0.204
EWC+W | 0.791 | 0.778 | 0.784 | 0.781 | 0.544 | 0.168 | 0.418 | 0.441 | 0.423 | 0.432 | 0.569 | 0.251
PackNet+W | 0.814 | 0.877 | 0.819 | 0.842 | 0.549 | 0.189 | 0.443 | 0.462 | 0.456 | 0.459 | 0.704 | 0.192
SupSup+W | 0.846 | 0.797 | 0.809 | 0.803 | 0.536 | 0.213 | 0.458 | 0.481 | 0.463 | 0.474 | 0.731 | 0.184
Our Proposed Fair Continual Learning Method
$\mathcal{I}$ | BiasPruner | 0.860 | 0.851 | 0.852 | 0.858 | 0.642 | 0.127 | 0.488 | 0.525 | 0.484 | 0.507 | 0.821 | 0.188
[Upper-bound] Comparison against a Bias Mitigation Method
$\mathcal{J}$ | FairDisCo | 0.873 | 0.876 | 0.904 | 0.893 | 0.682 | 0.113 | 0.486 | 0.545 | 0.512 | 0.538 | 0.855 | 0.150

III. Ablation Studies analyze the impact of individual components in
BiasPruner (Table 4). In Exp $\mathcal{K}$, we train the model using CE loss
instead of GCE. In Exp $\mathcal{L}$, we randomly prune the network instead of
using our bias-based pruning. In Exp $\mathcal{M}$, we finetune the debiased
subnetworks in BiasPruner with CE loss without weighting it. In Exp
$\mathcal{N}$, we simulate the absence of knowledge transfer (KT) by
prohibiting any overlapping between the parameters $\theta_{t}$ and
$\theta_{t^{\prime}}$ for any two tasks $t$ and $t^{\prime}$. We observe that
1) the impact of $\mathcal{L}_{\mathrm{WCE}}$ (Exp $\mathcal{M}$) is
predominant, as fine-tuning with CE leads to the poorest performance in
accuracy and fairness, attributed to the risk of subnetworks potentially
relearning bias.

Table 4: Classification (Overall) $\&$ fairness (DPR) results of BiasPruner from ablation studies. Best results marked in bold. Exp |  $\mathcal{L}_{\mathrm{GCE}}$ |  Bias-aware Pruning |  $\mathcal{L}_{\mathrm{WCE}}$ |  KT | FITZ | HAM | NIH
---|---|---|---|---|---|---|---
Overall$\uparrow$ |  DPR$\uparrow$ |  Overall$\uparrow$ |  DPR$\uparrow$ |  Overall$\uparrow$ |  DPR$\uparrow$
$\mathcal{D,I}$ | ✓ | ✓ | ✓ | ✓ | 0.512 | 0.331 | 0.858 | 0.642 | 0.507 | 0.821
$\mathcal{K}$ | $\times$ | ✓ | ✓ | ✓ | 0.498 | 0.254 | 0.834 | 0.579 | 0.501 | 0.779
$\mathcal{L}$ | ✓ | $\times$ | ✓ | ✓ | 0.508 | 0.328 | 0.842 | 0.637 | 0.498 | 0.814
$\mathcal{M}$ | ✓ | ✓ | $\times$ | ✓ | 0.481 | 0.247 | 0.792 | 0.576 | 0.468 | 0.754
$\mathcal{N}$ | ✓ | ✓ | ✓ | $\times$ | 0.504 | 0.324 | 0.851 | 0.630 | 0.496 | 0.803
Figure 2: The overall (dashed) and DPR (dotted) performance of BiasPruner and
other methods over all the seen tasks after each training step in the
continual learning sequence, where $Ti$ refers to the $i$th task.

IV. A Sequential Analysis, as illustrated in Fig. 2, showcases the
consistently superior performance of BiasPruner over other methods in terms of
overall accuracy and DPR after each step in the continual learning sequence
across FITZ, HAM, and NIH datasets.

V. For Analysis of Model Biases (e.g., skin tone), we trained classifiers for
sensitive attribute detection on top of frozen feature extractors from three
networks: CE-based, GCE-based, and our BiasPruner, all pre-trained to diagnose
(Fig. 3). The better-than-chance ($\in$ [0.63, 0.828]) detection accuracy of
sensitive attributes, in CE- and GCE-based training, reveals that sensitive
data is embedded in the originally learned features, i.e., the presence of
bias. GCE, due to its loss function promoting shortcut learning, showed the
most bias. The high accuracy achieved by CE shows that even naïvely trained
models are susceptible to bias. In contrast, BiasPruner shows minimal bias,
reflected by its near-random AUC values ($\in$ [0.49, 0.67]) when detecting
sensitive information.

Figure 3: Sensitive attribute detection from frozen models pre-trained to
diagnose. BiasPruner low AUCs indicate that bias is not encoded in its
resulting features.

##  4 Conclusion

In this paper, we presented BiasPruner, a new continual learning (CL)
framework that leverages intentional forgetting to improve fairness and
mitigate catastrophic forgetting. By quantifying the influence of each network
unit on learning spurious features, BiasPruner identifies and prunes highly
biased units to construct a debiased subnetwork for each task. Experimental
evaluations on three classification datasets demonstrate that BiasPruner
consistently outperforms baselines and CL methods in classification
performance and fairness. Our results highlight the need to address dataset
bias in future CL designs and evaluations, due to the frequent fairness
shortcomings of current CL methods.

###  4.0.1 Acknowledgements

We thank NVIDIA for their hardware grant and the Natural Sciences and
Engineering Research Council (NSERC) of Canada for the Vanier PhD Fellowship.
A. Bissoto is funded by FAPESP (2019/19619-7, 2022/ 09606-8).

###  4.0.2

The authors have no competing interests to declare that are relevant to the
content of this article.

## References

  * [1] Bayasi, N., Du, S., Hamarneh, G., Garbi, R.: Continual-GEN: Continual group ensembling for domain-agnostic skin lesion classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 3–13. Springer (2023)
  * [2] Bayasi, N., Hamarneh, G., Garbi, R.: Culprit-Prune-Net: Efficient continual sequential multi-domain learning with application to skin lesion classification. In: Medical Image Computing and Computer Assisted Intervention (MICCAI). pp. 165–175. Springer (2021)
  * [3] Bissoto, A., Barata, C., Valle, E., Avila, S.: Even small correlation and diversity shifts pose dataset-bias issues. Pattern Recognition Letters (2024)
  * [4] Brown, A., Tomasev, N., Freyberg, J., Liu, Y., Karthikesalingam, A., Schrouff, J.: Detecting shortcut learning for fair medical ai using shortcut testing. Nature Communications 14(1), 4314 (2023)
  * [5] Busch, F.P., Kamath, R., Mitchell, R., Stammer, W., Kersting, K., Mundt, M.: Where is the truth? the risk of getting confounded in a continual world. arXiv preprint arXiv:2402.06434 (2024)
  * [6] Chowdhury, S.B.R., Chaturvedi, S.: Sustaining fairness via incremental learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 6797–6805 (2023)
  * [7] Dekhovich, A., Tax, D.M., Sluiter, M.H., Bessa, M.A.: Continual prune-and-select: class-incremental learning with specialized subnetworks. Applied Intelligence 53(14), 17849–17864 (2023)
  * [8] Du, S., Hers, B., Bayasi, N., Hamarneh, G., Garbi, R.: FairDisCo: Fairer ai in dermatology via disentanglement contrastive learning. In: European Conference on Computer Vision. pp. 185–202 (2022)
  * [9] González, C., Ranem, A., Othman, A., Mukhopadhyay, A.: Task-agnostic continual hippocampus segmentation for smooth population shifts. In: Domain Adaptation and Representation Transfer MICCAI Workshop. pp. 108–118 (2022)
  * [10] Groh, M., Harris, C., Soenksen, L., Lau, F., Han, R., Kim, A., Koochek, A., Badri, O.: Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1820–1828 (2021)
  * [11] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [12] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114(13), 3521–3526 (2017)
  * [13] Kiyasseh, D., Zhu, T., Clifton, D.: A clinical deep learning framework for continually learning from cardiac signals across diseases, time, modalities, and institutions. Nature Communications 12(1), 4221 (2021)
  * [14] Lee, D., Jung, S., Moon, T.: Continual learning in the presence of spurious correlation. arXiv preprint arXiv:2303.11863 (2023)
  * [15] Lenga, M., Schulz, H., Saalbach, A.: Continual learning for domain adaptation in chest x-ray classification. In: Medical Imaging with Deep Learning. pp. 413–423 (2020)
  * [16] Lesort, T.: Spurious features in continual learning. In: AAAI Bridge Program on Continual Causality. pp. 59–62 (2023)
  * [17] Lewandowsky, S., Li, S.C.: Catastrophic interference in neural networks: Causes, solutions, and data. In: Interference and inhibition in cognition, pp. 329–361 (1995)
  * [18] Lin, X., Kim, S., Joo, J.: FairGrape: Fairness-aware gradient pruning method for face attribute classification. In: European Conference on Computer Vision. pp. 414–432 (2022)
  * [19] Luo, L., Xu, D., Chen, H., Wong, T.T., Heng, P.A.: Pseudo bias-balanced learning for debiased chest x-ray classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 621–631 (2022)
  * [20] Mallya, A., Lazebnik, S.: Packnet: Adding multiple tasks to a single network by iterative pruning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 7765–7773 (2018)
  * [21] Nam, J., Cha, H., Ahn, S., Lee, J., Shin, J.: Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems 33, 20673–20684 (2020)
  * [22] Perkonigg, M., Hofmanninger, J., Herold, C.J., Brink, J.A., Pianykh, O., Prosch, H., Langs, G.: Dynamic memory to alleviate catastrophic forgetting in continual learning with medical imaging. Nature communications 12(1), 5678 (2021)
  * [23] Salman, H., Jain, S., Ilyas, A., Engstrom, L., Wong, E., Madry, A.: When does bias transfer in transfer learning? arXiv preprint arXiv:2207.02842 (2022)
  * [24] Tschandl, P., Rosendahl, C., Kittler, H.: The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5(1), 1–9 (2018)
  * [25] Wang, L., Zhang, X., Su, H., Zhu, J.: A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487 (2023)
  * [26] Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2097–2106 (2017)
  * [27] Wang, Z., Yang, E., Shen, L., Huang, H.: A comprehensive survey of forgetting in deep learning beyond continual learning. arXiv preprint arXiv:2307.09218 (2023)
  * [28] Wortsman, M., Ramanujan, V., Liu, R., Kembhavi, A., Rastegari, M., Yosinski, J., Farhadi, A.: Supermasks in superposition. Advances in Neural Information Processing Systems 33, 15173–15184 (2020)
  * [29] Zhang, Z., Sabuncu, M.: Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems 31 (2018)

BiasPruner: Debiased Continual Learning for Medical Image Classification
(Supplementary Material)

Figure 4: Visualization of easy (blue square) and hard (red square) samples
across the different benchmarks. 1st row: Images from FITZ (Task 4), where
each image is labeled by its Fitzpatrick skin tone, denoted as I, II, IV, V or
VI. 2nd row: Images from HAM (Task 3), where each image is labeled by age (age
$<$ or $\geq$ 60), denoted as $-$ or $+$, respectively. 3rd row: Images from
NIH (Task 1), where each image is labeled as male or female, denoted as M or
F, respectively. We notice that the hard samples represent the minority within
their respective tasks. Figure 5: Bias distribution across the different tasks
in FITZ, HAM and NIH. Table 5: Distribution of images across train,
validation, and test sets for each task in FITZ, HAM, and NIH. The $V$ value
between brackets next to Train represents the Cramer’s $V$ correlation between
the sensitive attribute (e.g., skin tone in FITZ) and disease classes.

| FITZ | HAM | NIH
---|---|---|---
| Task 1 | Task 2 | Task 3 | Task 4 | Task 5 | Task 6 | Task 1 | Task 2 | Task 3 | Task 1 | Task 2 | Task 3
Train ($V$) |  | 2,582
---
(0.311)
| 2,519
---
(0.287)
| 1,986
---
(0.277)
| 1,609
---
(0.333)
| 1,315
---
(0.293)
| 1,198
---
(0.328)
| 519
---
(0.187)
| 731
---
(0.112)
| 4,441
---
(0.199)
| 6,003
---
(0.259)
| 2,217
---
(0.242)
| 2,481
---
(0.264)
Val | 361 | 358 | 286 | 230 | 180 | 171 | 85 | 123 | 809 | 1,711 | 646 | 710
Test | 747 | 721 | 568 | 462 | 375 | 344 | 167 | 240 | 1,563 | 3,472 | 1,311 | 1,442
Total | 3,690 | 3,598 | 2,840 | 2,301 | 1,870 | 1,713 | 771 | 1,094 | 6,813 | 11,186 | 4,174 | 4,633

Table 6: Continuation of Table 2 –Standard deviation results. Exp | Method | F | ACC | DPR | EOD
---|---|---|---|---|---
Type-I | Type-II | Type-III | Type-IV | Type-V | Type-VI | Overall
Comparison against Baselines
$\mathcal{A}$ | JOINT | 0.81 | 0.28 | 0.06 | 0.85 | 0.28 | 0.35 | 0.89 | 0.69 | 0.66 | 0.51
SINGLE | 0.73 | 0.13 | 0.15 | 0.30 | 0.45 | 0.53 | 0.19 | 0.11 | 0.12 | 0.38
SeqFT | 2.24 | 1.68 | 2.11 | 2.48 | 2.34 | 2.58 | 2.64 | 2.33 | 2.56 | 2.84
Comparison against CL Methods
$\mathcal{B}$ | EWC | 1.55 | 1.03 | 1.19 | 1.15 | 1.83 | 1.78 | 1.21 | 1.11 | 1.22 | 1.05
PackNet | 0.21 | 0.57 | 0.41 | 0.93 | 0.25 | 0.81 | 0.35 | 0.69 | 0.76 | 0.49
SupSup | 0.98 | 0.66 | 0.52 | 0.63 | 0.55 | 0.82 | 0.54 | 0.44 | 0.38 | 0.79
Comparison against CL with Bias Mitigation Methods
$\mathcal{C}$ | EWC+S | 1.51 | 1.76 | 1.34 | 1.19 | 1.75 | 1.26 | 1.33 | 1.28 | 1.01 | 1.14
PackNet+S | 0.15 | 0.14 | 0.62 | 0.51 | 0.26 | 0.17 | 0.64 | 0.25 | 0.46 | 0.26
SupSup+S | 0.25 | 0.58 | 0.61 | 0.63 | 0.24 | 0.5 | 0.28 | 0.56 | 0.63 | 0.65
EWC+W | 1.37 | 1.27 | 1.44 | 1.68 | 1.34 | 1.25 | 1.42 | 1.62 | 2.53 | 1.48
PackNet+W | 0.28 | 0.34 | 0.58 | 0.35 | 0.22 | 0.23 | 0.31 | 0.37 | 0.34 | 0.71
SupSup+W | 0.63 | 0.94 | 0.41 | 0.37 | 0.82 | 0.21 | 0.52 | 0.16 | 0.35 | 0.81
Our Proposed Fair Continual Learning Method
$\mathcal{D}$ | BiasPruner | 0.54 | 0.38 | 0.15 | 0.84 | 0.44 | 0.39 | 0.71 | 0.24 | 0.33 | 0.41
[Upper-bound] Comparison against a Bias Mitigation Method
$\mathcal{E}$ | FairDisCo | 0.91 | 0.22 | 0.84 | 0.65 | 0.74 | 0.33 | 0.85 | 0.12 | 0.35 | 0.34
Table 7: Continuation of Table 3 –Standard deviation results.  Exp | Method | HAM | NIH
---|---|---|---
F | ACC | DPR | EOD | F | ACC | DPR | EOD
$<$60 |  $\geq$60 | Overall |  M | F | Overall
Comparison against Baselines
$\mathcal{F}$ | JOINT | 0.05 | 0.14 | 0.11 | 0.14 | 0.09 | 0.08 | 0.15 | 0.08 | 0.11 | 0.22 | 0.61 | 0.23
SINGLE | 0.53 | 0.72 | 0.23 | 0.91 | 0.34 | 0.82 | 0.91 | 0.79 | 0.44 | 0.27 | 0.22 | 0.42
SeqFT | 2.64 | 0.65 | 0.85 | 0.36 | 0.41 | 0.45 | 0.54 | 0.49 | 0.96 | 0.46 | 0.22 | 0.36
Comparison against CL Methods
$\mathcal{G}$ | EWC | 1.46 | 1.12 | 1.69 | 1.79 | 1.14 | 1.16 | 1.74 | 1.62 | 1.42 | 1.66 | 1.19 | 1.77
PackNet | 0.44 | 0.57 | 0.28 | 0.22 | 0.82 | 0.61 | 0.22 | 0.43 | 0.51 | 0.17 | 0.34 | 0.55
SupSup | 0.53 | 0.64 | 0.22 | 0.74 | 0.91 | 0.84 | 0.39 | 0.53 | 0.55 | 0.69 | 0.22 | 0.43
Comparison against CL with Bias Mitigation Methods
$\mathcal{H}$ | EWC+S | 1.64 | 1.83 | 1.72 | 1.11 | 1.96 | 1.66 | 1.36 | 1.19 | 1.57 | 1.21 | 1.44 | 1.36
PackNet+S | 0.43 | 0.37 | 0.64 | 0.87 | 0.74 | 0.22 | 0.42 | 0.33 | 0.35 | 0.19 | 0.43 | 0.35
SupSup+S | 0.18 | 0.44 | 0.61 | 0.86 | 0.34 | 0.63 | 0.91 | 0.53 | 0.28 | 0.39 | 0.38 | 0.19
EWC+W | 1.17 | 1.75 | 1.81 | 1.62 | 1.56 | 1.38 | 1.35 | 1.86 | 1.54 | 1.81 | 1.71 | 1.26
PackNet+W | 0.84 | 0.35 | 0.71 | 0.23 | 0.55 | 0.14 | 0.45 | 0.82 | 0.22 | 0.65 | 0.27 | 0.33
SupSup+W | 0.72 | 0.43 | 0.16 | 0.44 | 0.78 | 0.81 | 0.62 | 0.77 | 0.25 | 0.34 | 0.64 | 0.81
Our Proposed Fair Continual Learning Method
$\mathcal{I}$ | BiasPruner | 0.44 | 0.18 | 0.57 | 0.21 | 0.38 | 0.25 | 0.35 | 0.41 | 0.27 | 0.39 | 0.44 | 0.62
[Upper-bound] Comparison against a Bias Mitigation Method
$\mathcal{J}$ | FairDisCo | 0.92 | 0.64 | 0.38 | 0.39 | 0.22 | 0.37 | 0.72 | 0.21 | 0.34 | 0.61 | 0.66 | 0.74


∎

11institutetext: Aniruddha Tamhane 22institutetext: Johns Hopkins University
22email<EMAIL_ADDRESS>11institutetext: All authors are with The Johns
Hopkins University, Baltimore, MD.

# Multimodal and self-supervised representation learning for automatic gesture
recognition in surgical robotics

Aniruddha Tamhane  Jie Ying Wu  Mathias Unberath

###### Abstract

Self-supervised, multi-modal learning has been successful in holistic
representation of complex scenarios. This can be useful to consolidate
information from multiple modalities which have multiple, versatile uses. Its
application in surgical robotics can lead to simultaneously developing a
generalized machine understanding of the surgical process and reduce the
dependency on quality, expert annotations which are generally difficult to
obtain. We develop a self-supervised, multi-modal representation learning
paradigm that learns representations for surgical gestures from video and
kinematics. We use an encoder-decoder network configuration that encodes
representations from surgical videos and decodes them to yield kinematics. We
quantitatively demonstrate the efficacy of our learnt representations for
gesture recognition (with accuracy between 69.6 % and 77.8 %), transfer
learning across multiple tasks (with accuracy between 44.6 % and 64.8 %) and
surgeon skill classification (with accuracy between 76.8 % and 81.2 %).
Further, we qualitatively demonstrate that our self-supervised representations
cluster in semantically meaningful properties (surgeon skill and gestures).

##  1 Introduction

The use of robotic surgery has increased in the recent years. Robotic surgical
consoles such as the da Vinci® Surgical System guthart2000intuitive  developed
by Intuitive Surgical Inc. (Sunnyvale, CA). are being used in place of
traditional surgical tools such as laparoscopes, with the aim of improving
surgical quality and patient prognosis. These consoles change the fundamental
nature of the surgeon’s interaction with the patient. Studying human surgical
performance under such a setting is thus a very exciting field of research
that is crucial for improving our understanding of surgeon-robot interaction
as well as improving patient outcomes. The rich and accurate surgical data
available in multiple modes such as videos and kinematics can be used to gain
a holistic comprehension of the surgery as well as study human surgical
performance closely and accurately.
A lot of recent work in this domain has focused on task-specific, supervised
learning from a single modality. We argue that this approach has several
drawbacks. For one, a task-specific learning paradigm (such as supervised
skill detection, for example) enables a very narrow learning of the overall
surgical process, giving very little understanding beyond the specific
training task. Further, supervised learning enforces a dependence on expert
annotations that are expensive to obtain, subjective and possibly erroneous.
Finally, ignoring multiple modalities of information (such as video and
kinematics) can be detrimental to learning generalizeable, feature-rich
representations.
We argue that robotic surgery can be broken down into a sequence of gestures
or surgemes varadarajan2009data . Thus, it is possible to model a surgery
similar to language models mikolov2013distributed ; devlin2018bert  over the
vocabulary of the surgemes. In this work, we propose a self-supervised
learning algorithm that learns task-agnostic surgical gesture representations
from two modalities (viz. video and kinematics) that can generalize well
across multiple tasks.
Our contributions in this paper are as follows: 1) we provide a deep-learning
based architecture to learn self-supervised surgical gesture representations
from video and kinematics, 2) we quantitatively demonstrate the utility of our
representations across multiple tasks by achieving state-of-the-art accuracy
in gesture recognition and high accuracy in skill recognition, 3) we
qualitatively demonstrate the rich information stored in our representations
by visualizing our representations forming semantically meaningful clusters.

##  2 Related Work

We review research literature closely related to our research. This section is
organised as follows: we review state-of-the-art supervised learning
approaches in surgical robotics in Section 2.1, unsupervised learning
approaches in Section 2.2, state-of-the-art approaches in video activity
recognition in Section 2.3 and landmark works in self-supervised learning from
multimodal sources in 2.4.

###  2.1 Supervised Surgical Gesture Recognition

A vast majority of research on surgical robotics learning task focuses on
supervised learning. The focus of such work is to learn a specific surgical
task such as future prediction, gesture recognition etc from annotated data.
In sarikaya2019surgical , Sarikaya and Jannin parse the optical flow
corresponding to the surgical videos using a Convolutional Neural Network
based backbone to perform a supervised surgical gesture classification. They
highlight an important point that solely optical flow is a sufficient source
of information for learning to classify surgical gestures with a high
accuracy. We use this insight to extract the optical flow as a source of
domain-independent visual information. In mazomenos2018gesture , Mazomenos et
al use Recurrent Neural Networks (RNNs) to classify surgical gestures in a
supervised manner. Similarly, in dipietro2016recognizing , DiPietro et al also
applied unidirectional and bi-directional LSTMs for gesture recognition. RNNs,
though a natural and effective choice in parsing sequential data such as video
frames are computationally expensive and harder to train. In our work, we use
CNNs to parse the sequential frames, with each CNN channel dedicated to a
particular frame. In sarikaya2018joint , Sarikaya et al combine both
modalities i.e. optical flow and video for supervised surgeme recognition.
This work is also a prime example of integrating multimodal information to
improve supervised gesture recognition results.

###  2.2 Unsupervised Surgical Gesture Recognition

Unsupervised surgical gesture recognition is a research paradigm with a large
potential impact as it does not rely on large amounts of quality, expert
annotations like supervised learning. In dipietro2018unsupervised , diPietro
and Hager demonstrated that predicting the next surgical gesture using RNNs in
an unsupervised manner captures the latent information necessary for surgical
gesture recognition. This was demonstrated on the kinematics data in the
JIGSAWS dataset. Further, they showed that these embeddings naturally
clustered corresponding to distinct higher level activities. In this work,
diPietro and Hager have demonstrated the possibility of learning unsupervised
representations from surgical data that can perform on par with state-of-the-
art supervised learning paradigms on surgical gesture recognition tasks.
Further, they demonstrated the versatility of these learnt representations by
demonstrating their utility in other tasks such as information retrieval. We
however envisage that a unimodal source of information (i.e. kinematics) could
potentially have implied a very narrow, task specific learning of
representations. In our work, we use a 2D-CNN based encoder-decoder structure
to learn representations that encode information from multiple modalities (viz
video and kinematics) and perform well across multiple tasks such as gesture
recognition, surgeon skill classification and cross-task gesture recognition.
The experiments have been explained in detail in Section 6.

###  2.3 Video activity recognition

Surgical gesture recognition from video data can benefit immensely by
borrowing from milestone techniques from video activity recognition. Thus,
reviewing key papers in this domain is fruitful to our efforts. In
karpathy2014large , Karpathy et al review several video classification models
(slow, early and late fusion) to provide empirical evidence for the
superiority of the slow fusion paradigm. Several works such as ji20123d  treat
videos as 3D videos, and thus extend the notion of a 2D CNN to 3D. This gives
us the necessary insights for parsing sequential visual data using CNNs.
Simonyan and Zisserman take a different approach in simonyan2014two . They
pair each video frame with the corresponding optical flow and parse both
through parallel 2D CNN based backbones. This approach has biological
justifications, since it has been indicated in simonyan2014two  that human
vision processing occurs in a similar manner, with separate neural pathways
dedicated to processing static image and motion information. We have
experimented with a two-stream approach using parallel CNN-based video and
optical flow parsing streams. However, we finalized upon an encoder-decoder
based architecture since it was more suitable for our particular learning
task. The architecture is further described in Section 4. In girdhar2019video
, Giridhar et al use the transformers-based attention as described in
vaswani2017attention  to learn high attention regions in video frames prior to
action recognition. It is worth exploring visual attention based models for
surgical activity recognition. We did not explore this further since visual
attention was an unnecessary feature for the data provided in the JIGSAWS
dataset, since only the surgical instruments (which are the only objects of
interest) move in the given video. Thus, we obtain all the information for
motion localization from the optical flow.

###  2.4 Multimodal self-supervised learning

Multimodal, self-supervised learning is has shown great promise in learning .
In arandjelovic2017look , Arandjelovic and Zisserman learnt embeddings for
video and audio inputs such that corresponding embeddings for audio and video
cluster close to each other. This was achieved by training a two-stream, CNN
based network to binary classify embeddings for correspondence. Further, in
arandjelovic2018objects , they demonstrate that the self-supservised
embeddings store information on the source of sound in the video. This
information can be extracted by computing a simple correlation between the
audio embeddings and different regions of the video frame representation.

##  3 Problem Formulation

Our problem statement is to learn multi-modal representations for surgical
robotic activity from video and kinematics data using a self-supervised
learning. Formally, given a dataset
$\mathbb{D}=(\mathcal{V}_{i},\mathcal{K}_{i})_{i=1}^{i=n}$ which is a tuple of
surgical videos $\mathcal{V}$ and corresponding kinematics $\mathcal{K}$, we
seek a corresponding lower-dimensional representation
$r(T(\mathcal{V}_{i}),\mathcal{K}_{i})$, where $T$ is a transformation on
$\mathcal{V}$. In our case, $T$ is a function that extracts the optical flow.
Further, we wish $r(T(\mathcal{V}_{i}),\mathcal{K}_{i})_{i=1}^{i=n}$ retain
all the critical information pertaining to the surgery, such as the exact
surgical gesture, identity of the surgeon and the skill with which the segment
of surgery was performed. Self-supervised learning is a natural solution to
this problem, given its efficacy in learning holistic information. To this
end, we define our problem a maximization of critical information retention in
$r$. Thus, our loss criterion $\mathcal{L}$ is a function
$\mathcal{L}:\mathbb{R}^{d}\times\mathbb{R}^{d}\rightarrow\mathbb{R}$, where
$\mathcal{L}$ is some measure of information loss between the information
sources i.e. $(\mathcal{V},\mathcal{K})$ and $r$. Finally, we define our
objective as the following empirical loss minimization over the parameters
$\theta$:

$\min_{\theta}\frac{1}{n}\sum_{i=1}^{n}\mathcal{L}(r(T(\mathcal{V}_{i}),\mathcal{K}_{i};\theta),\mathcal{V}_{i},\mathcal{K}_{i})$
(1)

Learning the representation function $r$ that encodes information from the
surgical videos and kinematics can be achieved using an appropriate proxy
learning objective. On the basis of our experiments, we propose a 2-D
Convolutional Neural Network based encoder-decoder learning objective.
Further, to learn representations that are independent of video modalities
such as luminence, color scheme, contrast etc. we extract the optical flow
$\mathcal{O}_{i}$ from the video frames $\mathcal{V}_{i}$ through
transformation $T$. Finally, we demonstrate that this architecture is helpful
in learning representations that outperform the current state-of-the-art in
self-supervised surgical gesture recognition. Thus, our problem further
reduces to the following: encode information from optical flow in
representations $r(T(\mathcal{V}))$ and minimize the information loss
$\mathcal{L}$ between the decoded representations
$\mathcal{D}(r(T(\mathcal{V})))$ and kinematics $\mathcal{K}$. We empirically
choose the loss function $\mathcal{L}$ as the L2 norm of the difference,
denoted by $||\cdot||_{2}$.
Thus, we rewrite the following formulation equivalent to the one described in
in Equation 1 under the above mentioned assumptions as:

$\min_{\theta,\phi}\frac{1}{n}\sum_{i=1}^{n}||\mathcal{D}({r(T(\mathcal{V}_{i});\theta);\phi)}-\mathcal{K}_{i}||_{2}^{2}$
(2)

given an parameterized encoder function $r$ with parameters $\mathcal{\theta}$
and decoder function $\mathcal{D}$ with parameters $\phi$.

##  4 Training and Architecture

We observe that the choice of architecture is key to learning quality
representations. We initiated our inquiry with an architecture similar to AVE-
Net described in arandjelovic2018objects , replacing the optical flow parsing
stream with the one we describe in Section 4.1 and experimenting with an RNN-
based parser (LSTMs/GRUs) and a 1D CNN based parser for the kinematics stream.
We kept the training task identical to the original representations alignment
task. We observed that the 1D CNN worked better than the RNN for parsing the
kinematics. However, the results were not entirely satisfactory. We observe
that similarity in multiple surgical gestures makes the task of identifying a
one-to-one mapping from video to kinematics a hard task, thus making the
training objective unsuitable for our task. We transformed the training
objective from a alignment-based task to an encoder-decoder task, where the
objective is to extract the corresponding kinematics vectors from the optical
flows. Accordingly, we modified the kinematics decoder to the one described in
Section 4.2. We observe that this configuration worked very well. Finally, we
choose a simple 2D Convolutional Neural Network based architecture for our
Encoder and a fully connected layer based architecture for our decoder
$\mathcal{D}$. A visual schematic of the architecture has been provided in
Figure 1. A detailed description has been provided in the subsequent sections.

Figure 1: Multimodal encoder-decoder network architecture

###  4.1 Encoder

We choose a 2D Convolutional Neural Network based architecture to model our
encoder function, as shown in Figure 1. It has been argued in
sarikaya2019surgical  that optical flow is crucial and sufficient for
classifying gestures. Further, we argue that optical flow can potentially
filter out domain-specific information such as the video quality, contrast,
details about the surgical instruments etc. that is not key in understanding
the surgical process. Therefore, we add an optical flow extraction as a pre-
processing step prior to encoding the representations. We use the Farneback
algorithm farneback2003two  to extract the optical flow and a 2D CNN based
encoder backbone similar to the one used by simonyan2014two  to encode it.
Finally, we provide the network a context of roughly 1.67 seconds by sub-
sampling the optical flow fields from every alternate frame over 50 frames.
Our objective here is to capture a context over a sufficiently large time-
frame.

###  4.2 Decoder

We choose a simple FCN with ReLU activations as decoder as shown in Figure 1.
The decoder outputs the 25 kinematics vectors each corresponding to the
sampled optical flows. We keep the decoder network relatively shallow to
ensure maximum information retention in the representations yielded by the
encoder network.

###  4.3 Training

We train our network for an encoder-decoder task wherein the encoder network
encodes information from the videos (optical flow) into the representation,
which is thereby parsed by the decoder network to extract the corresponding
kinematics. This has been elucidated in Section 3. While training, we
uniformly select every alternate frame, thus selecting 25 frames out of 50,
yielding a context of 1.6 seconds given a frame frequency of 30 Hz. We train
our network to encode this set of 25 video frames into representations. We use
a Mean Squared Error (MSE) between the kinematics vectors and the decoder
output as training loss, as given in Equation 2. Using this training
methodology, we train our network for 1000 epochs on the Knot Tying, Needle
Passing and Suturing datasets available in the JIGSAWS dataset. We observe
that the training loss uniformly decreases to with each training epoch,
indicating a smooth learning process.

##  5 Dataset

We test our learning approach on the JIGSAWS dataset, which contains a series
of annotated clips of surgical activity further sub-divided into three
datasets (viz knot-tying, needle-passing and suturing) performed using the da
Vinci® Surgical System console guthart2000intuitive . The frame-level
annotations include surgical gestures, anonymized user (surgeon)
identification and surgical skill metrics based on years of surgical
experience. The kinematics available on a frame-level consists of a
76-dimensional vector that includes the $x,y,z$ coordinates of the left and
right tool tips, the corresponding linear and angular velocities, the rotation
matrix and the gripper angle velocities. A detailed description of the JIGSAWS
dataset is given in gao2014jhu .

##  6 Experiments and results

We test the quality of our learned representations for a number of tasks as a
measure of the holistic understanding of the surgical process. We first divide
the frames in each video clip according to the corresponding gestures. We then
uniformly sample 25 frames from each set of 50 gesture-frames. Given the
sampling frequency of 30 Hz, we encode a gesture context of about a 1.67
seconds. To extract domain-independent information (as explained in Section
4.1, we extract the optical flow from the sampled video frames. We further
sample the 25 corresponding kinematics vectors.
We train the end-to-end deep encoder-decoder network described in Section 4 to
encode the representations from the optical flow and decode the kinematics
vectors from the representations. To obtain a visual understanding of the
information encoded in these representations, we reduce their dimensionality
to a 2-dimensional plane using the U-MAP algorithm mcinnes2018umap . We choose
U-MAP in particular because of its computational efficiency and ability to
preserve global structure, which would be crucial to appreciating the
differences in surgical gestures. We then perform a gesture classification
experiment using a gradient boosting based classifier trained solely on the
representations. Further, we also perform a skill-classification experiment
using another gradient-boosting based classifier. Finally, we perform a
gesture classification experiment on a set of cross-task representations i.e.
representations generated for a particular surgical task (eg: suturing) using
an encoder trained on a different surgical task (eg: knot-tying). Further
details and results of the experiments have been discussed in the following
sub-sections.

###  6.1 Representations visualizations using U-MAP

We use the U-MAP algorithm to perform a dimensionality reduction on the learnt
representations. We choose U-MAP in particular because of its property to
preserver local as well as global distances in the lower dimensional latent-
space. We plot the two-dimensional projections of the representation from the
Knot Tying dataset, color-coded for different gestures in Figure 2 and for
different skill-levels in Figure 3. We generate similar plots for the Needle
Passing dataset in Figures 4, 3 and Suturing dataset in Figures 6, 7. It is
interesting to observe that the in each of the skill-based plots (Figures 7,
5, 3), the representations neatly cluster into two distinct skill-based
clusters corresponding to ”beginner” and ”expert” skill level, with the
representations from the ”intermediate” skill level spread across both these
clusters. A possible explanation for this phenomenon is that the skill
”intermediate” is a vague category and encompasses people with varying skill-
levels that may be closer to the ”beginners” or the ”experts”. Another
interesting observation from the gesture-based plots is that the gestures-
based plots (Figures 6, 4, 2) is that each individual gesture (denoted by a
different color) has two distinct clusters corresponding to a distinct skill
category. Thus, it is evident from this visualization that each gesture has a
unique representation depending on whether it has been performed by an expert
or a beginner.

Figure 2: Knot Tying representations (gesture) Figure 3: Knot Tying
representations (skill) Figure 4: Needle Passing representations (gesture)
Figure 5: Needle Passing representations (skill) Figure 6: Suturing
representations (gesture) Figure 7: Suturing representations (skill)

###  6.2 Skill recognition

We test the efficacy of our representations in learning surgeon skill levels.
In particular, we train the XGBoost, a gradient boosting based classifier
introduced in chen2016xgboost  on the representations for a 3-class
classification task on the self-reported user skill. We do not train the
encoder while training the classifier. We repeat our experiment for 5 randomly
chosen train-test splits. We report our findings as an aggregate of the five
random splits in Table 1 (in the format of mean $\pm$ standard deviation). We
observe that the results indicate a significant retention of information
related to surgeon skill. Further, a major portion of the classification error
is contributed by surgeons having an ”intermediate” skill. This reaffirms the
observation made in Section 6.1 that the embeddings falling in this particular
skill category follows a uniform spread from ”beginner” to ”expert” and does
not necessarily cluster tightly, unlike the other two skill levels. This makes
it significantly harder to classify.

Table 1: Skill recognition results Dataset | Accuracy | Precision | Recall | F-1
---|---|---|---|---
Knot-Tying | 0.768 $\pm$ 0.0303 | 0.766 $\pm$ 0.0336 | 0.768 $\pm$ 0.0303 | 0.758 $\pm$ 0.0311
Needle-Passing | 0.808 $\pm$ 0.0335 | 0.808 $\pm$ 0.0335 | 0.808 $\pm$ 0.0335 | 0.800 $\pm$ 0.03535
Suturing | 0.812 $\pm$ 0.0228 | 0.810 $\pm$ 0.0254 | 0.812 $\pm$ 0.0228 | 0.804 $\pm$ 0.0267

###  6.3 Gesture recognition

We test the efficacy of our representations in learning different gestures
from the videos. Similar to the skill recognition task described in Section
6.2, we train an XGBoost classifier on the representations for gesture
classification. We freeze the encoder weights while training the classifier.
We repeat our experiment for 5 randomly chosen train-test splits. We report
our findings as an aggregate of the five random splits in Table 2. We observe
that the results are promising indicating a good retention of gesture related
information in the representations. Further, we observe that quantitatively,
our gesture classification results outperform those reported in
dipietro2018unsupervised . We attribute this to the fact that our
representations encode multimodal information from both kinematics and video
streams.

Table 2: Gesture recognition results Dataset | Accuracy | Precision | Recall | F-1
---|---|---|---|---
Knot-Tying | 0.762 $\pm$ 0.0487 | 0.774 $\pm$ 0.0531 | 0.762 $\pm$ 0.0487 | 0.758 $\pm$ 0.0496
Needle-Passing | 0.696 $\pm$ 0.0328 | 0.694 $\pm$ 0.0376 | 0.696 $\pm$ 0.0305 | 0.674 $\pm$ 0.0351
Suturing | 0.778 $\pm$ 0.0239 | 0.782 $\pm$ 0.0259 | 0.778 $\pm$ 0.0239 | 0.754 $\pm$ 0.0261

###  6.4 Transfer learning for gesture recognition

We measure the robustness of our learned algorithm through a transfer-learning
based gesture recognition task. We generate representations using an encoder
network trained on one dataset (e. g. : Knot Tying) and use them for gesture
classification on the other two datasets (Needle Passing and Suturing in this
case) similar to the gesture recognition task described in Section 6.3. We
repeat the gesture classification experiment for 5 randomly chosen train-test
splits. As in the original gesture classification experiment, we train an
XGBoost classifier on the representations generated using the encoder
initialized with transferred weights. We do not retrain the encoder while
training the classifier. We report our findings as an aggregate of the results
of the 5 random split based experiments as follows: results obtained using an
encoder trained on the Needle Passing dataset are reported in Table 3, the
Knot Tying dataset are reported in Table 4 and the Suturing dataset are
reported in 5. We observe that there is a slight decrease in performance after
transfer learning. However, the results are comparable to the baseline results
and those obtained from other state-of-the-art unsupervised surgical learning
works. This indicates that promising results can be achieved by enabling end-
to-end training (including training the encoder) during the task-specific
learning.

Table 3: Transfer learning (from Needle Passing) for gesture recognition Dataset | Accuracy | Precision | Recall | F-1
---|---|---|---|---
Knot-Tying | 0.568 $\pm$ 0.03271 | 0.608 $\pm$ 0.05403 | 0.568 $\pm$ 0.03271 | 0.570 $\pm$ 0.03605
Suturing | 0.598 $\pm$ 0.01923 | 0.570 $\pm$ 0.02738 | 0.598 $\pm$ 0.01923 | 0.557 $\pm$ 0.02509
Table 4: Transfer learning (from Knot Tying) for gesture recognition Dataset | Accuracy | Precision | Recall | F-1
---|---|---|---|---
Needle-Passing | 0.446 $\pm$ 0.05319 | 0.464 $\pm$ 0.06188 | 0.444 $\pm$ 0.05176 | 0.430 $\pm$ 0.05148
Suturing | 0.624 $\pm$ 0.02302 | 0.634 $\pm$ 0.04560 | 0.624 $\pm$ 0.02302 | 0.590 $\pm$ 0.02345
Table 5: Transfer learning (from Suturing) for gesture recognition Dataset | Accuracy | Precision | Recall | F-1
---|---|---|---|---
Knot-Tying | 0.648 $\pm$ 0.04338 | 0.648 $\pm$ 0.04338 | 0.648 $\pm$ 0.04338 | 0.650 $\pm$ 0.04062
Needle-Passing | 0.448 $\pm$ 0.04658 | 0.458 $\pm$ 0.03898 | 0.446 $\pm$ 0.04505 | 0.434 $\pm$ 0.04605

##  7 Discussion and future work

Surgical robotics is an exciting area for applying multi-modal, self-
supervised learning techniques due to the complexity of the task, multiple
modes of information and lack of dependence on expert annotated data. The
choice of the how to achieve the self-supervised learning can be varied.
Previous works in literature have used varied proxy learning objectives (that
we denote by $\mathcal{L}$ in Equation 1). For example, the sequential nature
of the video/kinematics frames has been utilized to devise a future prediction
task in dipietro2018unsupervised , where the learned representations were used
in gesture recognition and information retrieval. In another example, the
audio-visual correspondence task as described in arandjelovic2018objects ;
arandjelovic2017look  has been used to improve audio classification and enable
sound localization in videos. This can be transformed into a kinematics/visual
correspondence task to learn the surgical representations. We observe that
this choice is sensitive to the dataset in and the nature of information that
is expected from the learnt representations. For example, we observe that the
video-kinematics alignment task using triplets loss as described in
arandjelovic2018objects  is a bad learning objective for surgical robotics
data as provided in the JIGSAWS dataset because surgery involves a repetition
of similar gestures that have similar kinematics. This creates a problem where
there is no easy one-to-one correspondence between the video and kinematics
sets, rendering the triplet loss based training to be very difficult. A
possible way to circumvent this problem is to generate negative samples in
each triplet that are sufficiently distant from the base sample by some
distance metric. Further, we observe that training RNNs takes more
computational resource and cannot be easily parallelized as compared to
training CNNs. Hence, we focus on using an architecture primarily built using
CNNs and fully-connected networks.
We introduce an encoder-decoder based self-supervised learning system that
transforms video stream into corresponding kinematics, as explained in detail
in Sections 4, 3 . We further demonstrate empirically that the learnt
representations effectively encode information on surgical gestures and
surgeon skill. Also, the representations are sufficiently robust across
multiple tasks, thus facilitating transfer learning. This also raises the
possibility that using an end-to-end, task-specific training methodology can
further improve the accuracy in all the mentioned tasks. Finally, we visually
observe that the representations form clusters corresponding to a specific
gesture and skill-level.
There are a few limitations to our work. One being that JIGSAWS dataset has a
single camera perspective in all the videos, thus making it impossible to test
the learning of the camera transformation matrix. Secondly, our encoder
network does not parse 3D visual information, which has been provided in the
JIGSAWS dataset through a left and right camera feed for each video. This
could possibly lead to learning information on new tasks such as depth
perception and also simultaneously improve accuracy on the currently existing
tasks. Finally, while our work does solve the surgeon skill classification
task, it does not address the specific differences in surgical behaviour (e.
g. differences in kinematics) that could potentially address the gap in
surgical skill and be used as milestones in surgeon training.
There also are additional applications of the proposed representation that
future work could explore, such as future task prediction, information
retrieval, surgical activity segmentation etc. Training these representations
on a sufficiently large and diverse surgical dataset can potentially lead to a
standardized architecture for parsing surgical activity that is highly
transferable across datasets and tasks. This could in turn enable a universal
technology that tracks surgical progress in real-time, giving feedback
regarding possible mistakes, surgical scene depth, next gesture suggestion
etc. with a high accuracy.

## References

  * (1) Arandjelovic, R., Zisserman, A.: Look, listen and learn.  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 609–617 (2017)
  * (2) Arandjelovic, R., Zisserman, A.: Objects that sound.  In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 435–451 (2018)
  * (3) Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system.  In: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785–794 (2016)
  * (4) Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805 (2018)
  * (5) DiPietro, R., Hager, G.D.: Unsupervised learning for surgical motion by learning to predict the future.  In: International conference on medical image computing and computer-assisted intervention, pp. 281–288. Springer (2018)
  * (6) DiPietro, R., Lea, C., Malpani, A., Ahmidi, N., Vedula, S.S., Lee, G.I., Lee, M.R., Hager, G.D.: Recognizing surgical activities with recurrent neural networks.  In: International conference on medical image computing and computer-assisted intervention, pp. 551–558. Springer (2016)
  * (7) Farnebäck, G.: Two-frame motion estimation based on polynomial expansion.  In: Scandinavian conference on Image analysis, pp. 363–370. Springer (2003)
  * (8) Gao, Y., Vedula, S.S., Reiley, C.E., Ahmidi, N., Varadarajan, B., Lin, H.C., Tao, L., Zappella, L., Béjar, B., Yuh, D.D., et al.: Jhu-isi gesture and skill assessment working set (jigsaws): A surgical activity dataset for human motion modeling.  In: MICCAI Workshop: M2CAI, vol. 3, p. 3 (2014)
  * (9) Girdhar, R., Carreira, J., Doersch, C., Zisserman, A.: Video action transformer network.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 244–253 (2019)
  * (10) Guthart, G.S., Salisbury, J.K.: The intuitive/sup tm/telesurgery system: overview and application.  In: Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), vol. 1, pp. 618–621. IEEE (2000)
  * (11) Ji, S., Xu, W., Yang, M., Yu, K.: 3d convolutional neural networks for human action recognition.  IEEE transactions on pattern analysis and machine intelligence 35(1), 221–231 (2012)
  * (12) Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L.: Large-scale video classification with convolutional neural networks.  In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1725–1732 (2014)
  * (13) Mazomenos, E., Watson, D., Kotorov, R., Stoyanov, D.: Gesture classification in robotic surgery using recurrent neural networks with kinematic information.  CRAS (2018)
  * (14) McInnes, L., Healy, J., Melville, J.: Umap: Uniform manifold approximation and projection for dimension reduction.  arXiv preprint arXiv:1802.03426 (2018)
  * (15) Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality.  In: Advances in neural information processing systems, pp. 3111–3119 (2013)
  * (16) Sarikaya, D., Guru, K.A., Corso, J.J.: Joint surgical gesture and task classification with multi-task and multimodal learning.  arXiv preprint arXiv:1805.00721 (2018)
  * (17) Sarikaya, D., Jannin, P.: Surgical gesture recognition with optical flow only.  arXiv preprint arXiv:1904.01143 (2019)
  * (18) Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos.  In: Advances in neural information processing systems, pp. 568–576 (2014)
  * (19) Varadarajan, B., Reiley, C., Lin, H., Khudanpur, S., Hager, G.: Data-derived models for segmentation with application to surgical assessment and training.  In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 426–434. Springer (2009)
  * (20) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need.  In: Advances in neural information processing systems, pp. 5998–6008 (2017)


$\displaystyle-\eta\Delta y_{s}+\nu_{0}y_{s}=f\text{ in }\Omega,\quad
y_{s}=0\text{ on }\partial\Omega.$ (A.2)

The weak formulation corresponding to (A.2) seeks $y_{s}\in H^{1}_{0}(\Omega)$
such that

$\displaystyle\eta\langle\nabla y_{s},\nabla\phi\rangle+\nu_{0}\langle
y_{s},\phi\rangle=\langle f,\phi\rangle\quad\text{ for all }\phi\in
H^{1}_{0}(\Omega).$

For $\nu_{0}\geq 0,$ using Poincaré inequality, it can be observed that
$\eta\|\nabla\phi\|^{2}+\nu_{0}\|\phi\|^{2}\geq\eta\|\nabla\phi\|^{2}.$ Thus,
an application of Lax-Milgram theorem [25, Theorem 3.1.4] leads to the
existence of a unique solution $y_{s}\in H^{1}_{0}(\Omega)$ and a elliptic
regularity [23, Theorem 4, Chapter 5] implies $y_{s}\in H^{2}(\Omega)$ with

$\displaystyle\|y_{s}\|_{H^{2}(\Omega)}\leq C\|f\|,$

for some $C>0.$

Step 2. Now, for a given $\psi\in H^{2}(\Omega),$ consider

$\displaystyle-\eta\Delta y_{s}+\nu_{0}y_{s}=f+g(\psi)\text{ in }\Omega,\quad
y_{s}=0\text{ on }\partial\Omega,$ (A.3)

where $g(\psi)=-\psi\textbf{v}\cdot\nabla\psi.$ Our aim is to show that (A.3)
has unique solution $y_{s}\in H^{2}(\Omega)\cap H^{1}_{0}(\Omega)$ by showing
$g(\psi)\in L^{2}(\Omega)$ and using Step 1. Note that,
$g(\psi)=\displaystyle\sum_{i=1}^{d}\psi v_{i}\frac{\partial\psi}{\partial
x_{i}},$ where $\textbf{v}=(v_{1},v_{2},\ldots,v_{d})^{T}\in\mathbb{R}^{d}.$ A
Hölders inequality leads to

$\displaystyle\left\|\psi v_{i}\frac{\partial\psi}{\partial
x_{i}}\right\|^{2}=\int_{\Omega}\left|\psi v_{i}\frac{\partial\psi}{\partial
x_{i}}\right|^{2}dx\leq\left(\int_{\Omega}|\psi|^{4}dx\right)^{1/2}\left(\int_{\Omega}\left|v_{i}\frac{\partial\psi}{\partial
x_{i}}\right|^{4}dx\right)^{1/2}=\|\psi\|_{L^{4}(\Omega)}^{2}\left\|v_{i}\frac{\partial\psi}{\partial
x_{i}}\right\|_{L^{4}(\Omega)}^{2}.$

for all $1\leq i\leq d.$ The Sobolev embedding (2.2) for
$\Omega\subset\mathbb{R}^{d},\,d\in\\{1,2,3\\}$ implies

$\displaystyle\|g(\psi)\|\leq\|\psi\|_{L^{4}(\Omega)}|\textbf{v}|\|\nabla\psi\|_{L^{4}(\Omega)}\leq
s_{0}^{2}|\textbf{v}|\|\psi\|_{H^{1}(\Omega)}\|\psi\|_{H^{2}(\Omega)}.$ (A.4)

Thus, for any given $\psi\in H^{2}(\Omega)\cap H^{1}_{0}(\Omega),$ $g(\psi)\in
L^{2}(\Omega),$ and hence using Step 1, there exists a unique solution
$y_{s}\in H^{2}(\Omega)\cap H^{1}_{0}(\Omega)$ satisfying

$\displaystyle\|y_{s}\|_{H^{2}(\Omega)}\leq
C\|f\|+s_{0}^{2}|\textbf{v}|\|\psi\|_{H^{1}(\Omega)}\|\psi\|_{H^{2}(\Omega)},$
(A.5)

for some $C>0$.

Step 3. For any $\rho>0,$ define $D_{\rho}=\\{y\in H^{2}(\Omega)\cap
H^{1}_{0}(\Omega)\,|\,\|y\|_{H^{2}(\Omega)}\leq\rho\\}.$ In this step, we find
a $\rho_{0}>0$ such that for all $0<\rho\leq\rho_{0},$ the map
$S:D_{\rho}\longrightarrow D_{\rho}$ defined by $S(\psi)=y_{s}^{\psi},$ where
$y_{s}^{\psi}$ is solution of (A.3), is well defined and contraction. For
$f\in L^{2}(\Omega)$ with $\|f\|_{L^{2}(\Omega)}\leq\frac{\rho}{2C}$ and
$\rho\leq\frac{1}{2|\textbf{v}|s_{0}^{2}},$ (A.5) implies

$\|S(\psi)\|_{H^{2}(\Omega)}=\|y_{s}^{\psi}\|_{H^{2}(\Omega)}\leq
C\|f\|_{L^{2}(\Omega)}+s_{0}^{2}|\textbf{v}|\|\psi\|_{H^{1}(\Omega)}\|\psi\|_{H^{2}(\Omega)}\leq\rho.$

Therefore, $S$ is a self map.

Now, to show contraction, let $\psi^{1},\,\psi^{2}\in D_{\rho}$ be given and
let $y_{s}^{\psi^{1}}$ and $y_{s}^{\psi^{2}}$ be the corresponding solutions
of (A.3). Then, $y_{s}^{\psi^{1}}-y_{s}^{\psi^{2}}$ satisfies

$\displaystyle-\eta\Delta(y_{s}^{\psi^{1}}-y_{s}^{\psi^{2}})+\nu_{0}(y_{s}^{\psi^{1}}-y_{s}^{\psi^{2}})=g(\psi^{1})-g(\psi^{2})\text{
in }\Omega,\,y_{s}^{\psi^{1}}-y_{s}^{\psi^{2}}=0\text{ on }\partial\Omega.$

From Step 1, we have

$\displaystyle\|S(\psi^{1})-S(\psi^{2})\|_{H^{2}(\Omega)}=\|y_{s}^{\psi^{1}}-y_{s}^{\psi^{2}}\|_{H^{2}(\Omega)}\leq
C\|g(\psi^{1})-g(\psi^{2})\|_{L^{2}(\Omega)}.$

Note that

$\displaystyle
g(\psi^{1})-g(\psi^{2})=\psi^{1}\textbf{v}\cdot\nabla\psi^{1}-\psi^{2}\textbf{v}\cdot\nabla\psi^{2}=\displaystyle\sum_{i=1}^{d}\left(\psi^{1}v_{i}\left(\frac{\partial\psi^{1}}{\partial
x_{i}}-\frac{\partial\psi^{2}}{\partial
x_{i}}\right)+(\psi^{1}-\psi^{2})v_{i}\frac{\partial\psi^{2}}{\partial
x_{i}}\right).$

We estimate $g(\psi^{1})-g(\psi^{2})$ as estimated $g(\psi)$ in (A.4) and
obtain

$\displaystyle\|g(\psi^{1})-g(\psi^{2})\|_{L^{2}(\Omega)}$ $\displaystyle\leq
s_{0}^{2}|\textbf{v}|\left(\|\psi^{1}\|_{H^{1}(\Omega)}\|\psi^{1}-\psi^{2}\|_{H^{2}(\Omega)}+\|\psi^{1}-\psi^{2}\|_{H^{1}(\Omega)}\|\psi^{2}\|_{H^{2}(\Omega)}\right)$
$\displaystyle\leq
2s_{0}^{2}|\textbf{v}|\rho\|\psi^{1}-\psi^{2}\|_{H^{2}(\Omega)}.$

Therefore, choosing $\rho_{0}=\frac{1}{4s_{0}^{2}|\textbf{v}|},$ we obtain for
all $0<\rho\leq\rho_{0},$

$\displaystyle\|S(\psi^{1})-S(\psi^{2})\|_{H^{2}(\Omega)}\leq
C\|g(\psi^{1})-g(\psi^{2})\|_{L^{2}(\Omega)}\leq\frac{1}{2}\|\psi^{1}-\psi^{2}\|_{H^{2}(\Omega)}.$

Hence, by using Banach fixed point theorem, we conclude the proposition. ∎

###  A.2. Proof of Theorem 3.2

We first determine $\theta_{0}\in(\frac{\pi}{2},\pi)$ such that (a) holds. Set
$\theta_{0}=\pi-\tan^{-1}\left(\frac{\alpha_{1}}{\eta/2}\right),$ where
$\alpha_{1}>0$ is as in (3.3). Note that $\theta_{0}\in(\frac{\pi}{2},\pi).$

(a) In the next three steps, we show that
$\Sigma^{c}(-\widehat{\nu};\theta_{0})\subset\rho(\mathcal{A})$ and the
resolvent estimate holds.

Step 1. ($\mu\in\Sigma^{c}(-\widehat{\nu};\theta_{0})$ with
$\Re(\mu)\geq-\widehat{\nu}$). Let $\mu\in\mathbb{C}$ be arbitrary with
$\Re(\mu)\geq-\widehat{\nu}.$ First, we show that for any given $g\in
L^{2}(\Omega),$ there exists a unique $z\in D(\mathcal{A})$ such that $(\mu
I-\mathcal{A})z=g.$ That is, we want to solve

$\displaystyle a(z,\phi)+\mu\langle z,\phi\rangle=\langle g,\phi\rangle\text{
for all }\phi\in H^{1}_{0}(\Omega),$ (A.6)

for $z,$ where $a(\cdot,\cdot)$ is as defined in (3.2). As
$\Re(\mu)\geq-\widehat{\nu},$ (3.6) implies
$\Re\left(a(\phi,\phi)+\mu\langle\phi,\phi\rangle\right)$
$\geq\Re\left(a(\phi,\phi)\right)-\widehat{\nu}\|\phi\|^{2}\geq\frac{\eta}{2}\|\nabla\phi\|^{2}.$
Therefore, by Lax-Milgram Theorem ([21, 15, Theorem 1, Section 1, Chapter
VII]) for any given $g\in L^{2}(\Omega),$ there exists a unique solution $z\in
H^{1}_{0}(\Omega)$ of (A.6). Note that, $(\mu I-\mathcal{A})z=g$ implies

$\displaystyle-\eta\Delta z+y_{s}\textbf{v}\cdot\nabla z+\textbf{v}\cdot\nabla
y_{s}z+(\mu+\nu_{0})z=g\text{ in }\Omega,\,z=0\text{ on }\partial\Omega.$
(A.7)

Using the regularity results for elliptic equation [25], we have $z\in
D(\mathcal{A}).$ Substitute $\phi=z$ in (A.6), use (3.6) and a Cauchy-Schwarz
inequality to obtain

$\displaystyle\frac{\eta}{2}\|\nabla z\|^{2}\leq\Re\left(a(z,z)+\mu\langle
z,z\rangle\right)\leq\|g\|\,\|z\|.$

This along with the Poincaré inequality leads to

$\displaystyle\|z\|\leq C\|g\|,$ (A.8)

for some $C=C(\eta,C_{p})>0.$

Step 2. Resolvent estimate for
$\Re(\mu)\geq-\widehat{\nu},\,\mu\neq-\widehat{\nu}$. Let
$\mu=-\widehat{\nu}+\rho e^{i\theta},$ $\rho>0$ and
$-\frac{\pi}{2}\leq\theta\leq\frac{\pi}{2}.$ Substitute
$\phi=e^{i\frac{\theta}{2}}z$ in (A.6) to obtain

$\displaystyle a(z,e^{i\frac{\theta}{2}}z)+(-\widehat{\nu}+\rho
e^{i\frac{\theta}{2}})\langle z,e^{i\frac{\theta}{2}}z\rangle=\langle
g,e^{i\frac{\theta}{2}}z\rangle.$ (A.9)

From the definition of $a(\cdot,\cdot)$ in (3.2) and from (A.9), we obtain

$\displaystyle\cos(\theta/2)\eta\|\nabla
z\|^{2}+(\nu_{0}-\widehat{\nu}+\rho)\cos(\theta/2)\|z\|^{2}$
$\displaystyle\leq\left|\Re\left(a(z,e^{i\frac{\theta}{2}}z)+(-\widehat{\nu}+\rho
e^{i\theta})\langle z,e^{i\frac{\theta}{2}}z\rangle\right)\right|$
$\displaystyle\qquad+\left|\langle y_{s}\textbf{v}\cdot\nabla
z,e^{i\frac{\theta}{2}}z\rangle+\langle\textbf{v}\cdot\nabla
y_{s}z,e^{i\frac{\theta}{2}}z\rangle\right|$
$\displaystyle\leq\|g\|\,\|z\|+\left|\langle y_{s}\textbf{v}\cdot\nabla
z,e^{i\frac{\theta}{2}}z\rangle+\langle\textbf{v}\cdot\nabla
y_{s}z,e^{i\frac{\theta}{2}}z\rangle\right|.$

This inequality and a similar estimate as used to obtain (3.5) lead to

$\displaystyle\cos(\theta/2)\left(\eta\|\nabla
z\|^{2}+(\nu_{0}-\widehat{\nu}+\rho)\|z\|^{2}\right)$
$\displaystyle\leq\|g\|\,\|z\|+\frac{\eta}{\sqrt{2}}\|\nabla
z\|^{2}+\frac{|\textbf{v}|^{2}}{\sqrt{2}\eta}(C_{2}^{2}+s_{0}^{4})\|y_{s}\|_{H^{2}(\Omega)}^{2}\|z\|^{2}.$

In view of (3.4), and the fact that
$\cos(\theta/2)\geq\cos(\pi/4)\geq\frac{1}{\sqrt{2}}$ (as
$-\frac{\pi}{2}\leq\theta\leq\frac{\pi}{2}$), we have

$\displaystyle\rho\cos(\theta/2)\|z\|^{2}\leq C\|g\|\,\|z\|.$

Noting that $\rho=|\mu+\widehat{\nu}|$ and
$\cos(\theta/2)\geq\cos(\pi/4)\geq\frac{1}{\sqrt{2}},$ we have

$\displaystyle\|R(\mu,\mathcal{A})\|_{\mathcal{L}(L^{2}(\Omega))}\leq\frac{C}{|\mu+\widehat{\nu}|}\text{
for all }\mu(\neq-\widehat{\nu})\text{ with }\Re(\mu)\geq-\widehat{\nu},$
(A.10)

for some positive constant $C$ independent of $\mu.$

Step 3. Case of any $\mu\in\Sigma^{c}(-\widehat{\nu};\theta_{0})$ with
$\Re(\mu)<-\widehat{\nu}$. The proof is analogous to Step 3 of [3, proof of
Theorem 3.4(a) in Section A.1].

(b) - (c) The proofs of (b) and (c) follow utilizing (a) and proof of [3,
Theorem 3.4]. ∎

## Declaration

Conflict of interest. This work was done when I was a Ph.D. student at the
Department of Mathematics, IIT Bombay, and during that time, I was supported
by an institute TA fellowship.

Acknowledgment. I am thankful to Prof. Neela Nataraj, Prof. Debanjana Mitra,
and Prof. Mythily Ramaswamy for their valuable insights, feedback, and
encouragement during various stages of this research. Their dedications and
expertise played a pivotal role in the successful completion of this study.

## References

  * [1] S. Agmon, _Lectures on elliptic boundary value problems_ , AMS Chelsea Publishing, Providence, RI, 2010, Prepared for publication by B. Frank Jones, Jr. with the assistance of George W. Batten, Jr., Revised edition of the 1965 original.
  * [2] W. Akram and D. Mitra, _Local stabilization of viscous Burgers equation with memory_ , Evol. Equ. Control Theory 11 (2022), no. 3, 939–973.
  * [3] W. Akram, D. Mitra, N. Nataraj, and M. Ramaswamy, _Feedback stabilization of a parabolic coupled system and its numerical study_ , Math. Control Relat. Fields 14 (2024), no. 2, 695–746.
  * [4] M. Badra, _Stabilisation par feedback et approximation des ´equations de navier–stokes_ , PhD. Thesis, Toulouse (2006), i–260.
  * [5] M. Badra and T. Takahashi, _Stabilization of parabolic nonlinear systems with finite dimensional feedback or dynamical controllers: application to the Navier-Stokes system_ , SIAM J. Control Optim. 49 (2011), no. 2, 420–463.
  * [6] by same author, _On the Fattorini criterion for approximate controllability and stabilizability of parabolic systems_ , ESAIM Control Optim. Calc. Var. 20 (2014), no. 3, 924–956.
  * [7] A. Balogh and M. Krstić, _Burgers’ equation with nonlinear boundary feedback: $H^{1}$ stability, well-posedness and simulation_, vol. 6, 2000, Dedicated to Professor Robert E. Skelton, pp. 189–200.
  * [8] V. Barbu, _Controllability and stabilization of parabolic equations_ , Progress in Nonlinear Differential Equations and their Applications, vol. 90, Birkhäuser/Springer, Cham, 2018, Subseries in Control.
  * [9] V. Barbu, I. Lasiecka, and R. Triggiani, _Tangential boundary stabilization of Navier-Stokes equations_ , Mem. Amer. Math. Soc. 181 (2006), no. 852, x+128.
  * [10] V. Barbu and R. Triggiani, _Internal stabilization of Navier-Stokes equations with finite-dimensional controllers_ , Indiana Univ. Math. J. 53 (2004), no. 5, 1443–1494.
  * [11] A. Bensoussan, G. Da Prato, M. C. Delfour, and S. K. Mitter, _Representation and control of infinite dimensional systems_ , second ed., Systems & Control: Foundations & Applications, Birkhäuser Boston, Inc., Boston, MA, 2007.
  * [12] T. Breiten and K. Kunisch, _Riccati-based feedback control of the monodomain equations with the FitzHugh-Nagumo model_ , SIAM J. Control Optim. 52 (2014), no. 6, 4057–4081.
  * [13] by same author, _Compensator design for the monodomain equations with the FitzHugh-Nagumo model_ , ESAIM Control Optim. Calc. Var. 23 (2017), no. 1, 241–262.
  * [14] J.-M. Buchot, J.-P. Raymond, and J. Tiago, _Coupling estimation and control for a two dimensional Burgers type equation_ , ESAIM Control Optim. Calc. Var. 21 (2015), no. 2, 535–560.
  * [15] J. A. Burns and S. Kang, _A control problem for burgers’ equation with bounded input/output_ , Nonlinear Dyn. 2 (1991), 235–262.
  * [16] Ch. I. Byrnes, D. S. Gilliam, and V. I. Shubov, _On the global dynamics of a controlled viscous Burgers’ equation_ , J. Dynam. Control Systems 4 (1998), no. 4, 457–519.
  * [17] J. Caldwell, P. Wanless, and A. E. Cook, _A finite element approach to Burgers’ equation_ , Appl. Math. Modelling 5 (1981), no. 3, 189–193.
  * [18] John R. Cannon, Richard E. Ewing, Yinnian He, and Yanping Lin, _A modified nonlinear Galerkin method for the viscoelastic fluid motion equations_ , Internat. J. Engrg. Sci. 37 (1999), no. 13, 1643–1662.
  * [19] Y. Chen and T. Zhang, _A weak Galerkin finite element method for Burgers’ equation_ , J. Comput. Appl. Math. 348 (2019), 103–119.
  * [20] Zhangxin Chen, _Finite element methods and their applications_ , Scientific Computation, Springer-Verlag, Berlin, 2005.
  * [21] R. Dautray and J.L. Lions, _Mathematical analysis and numerical methods for science and technology. Vol. 2_ , Springer-Verlag, Berlin (1988), xvi+561, Functional and variational methods, With the collaboration of Michel Artola, Marc Authier, Philippe Bénilan, Michel Cessenat, Jean Michel Combes, Hélène Lanchon, Bertrand Mercier, Claude Wild and Claude Zuily, Translated from the French by Ian N. Sneddon.
  * [22] A. Dogan, _A Galerkin finite element approach to Burgers’ equation_ , Appl. Math. Comput. 157 (2004), no. 2, 331–346.
  * [23] L. C. Evans, _Partial differential equations_ , second ed., Graduate Studies in Mathematics, vol. 19, American Mathematical Society, Providence, RI, 2010.
  * [24] T. Kato, _Perturbation theory of linear operators_ , Springer-Verlag, New York (1966).
  * [25] S. Kesavan, _Topics in functional analysis and applications_ , John Wiley & Sons, Inc., New York, 1989.
  * [26] M. Kroller and K. Kunisch, _Convergence rates for the feedback operators arising in the linear quadratic regulator problem governed by parabolic equations_ , SIAM J. Numer. Anal. 28 (1991), no. 5, 1350–1385.
  * [27] M. Krstic, _On global stabilization of Burgers’ equation by boundary control_ , Systems Control Lett. 37 (1999), no. 3, 123–141.
  * [28] S. Kundu and A. K. Pani, _Finite element approximation to global stabilization of the Burgers’ equation by Neumann boundary feedback control law_ , Adv. Comput. Math. 44 (2018), no. 2, 541–570.
  * [29] by same author, _Global stabilization of BBM-Burgers’ type equations by nonlinear boundary feedback control laws: theory and finite element error analysis_ , J. Sci. Comput. 81 (2019), no. 2, 845–880.
  * [30] by same author, _Global stabilization of two dimensional viscous Burgers’ equation by nonlinear Neumann boundary feedback control and its finite element analysis_ , J. Sci. Comput. 84 (2020), no. 3, Paper No. 45, 29\.
  * [31] I. Lasiecka and R. Triggiani, _The regulator problem for parabolic equations with Dirichlet boundary control. I. Riccati’s feedback synthesis and regularity of optimal solution_ , Appl. Math. Optim. 16 (1987), no. 2, 147–168.
  * [32] by same author, _The regulator problem for parabolic equations with Dirichlet boundary control. II. Galerkin approximation_ , Appl. Math. Optim. 16 (1987), no. 3, 187–216.
  * [33] by same author, _Numerical approximations of algebraic Riccati equations for abstract systems modelled by analytic semigroups, and applications_ , Math. Comp. 57 (1991), no. 196, 639–662, S13–S37.
  * [34] by same author, _Control theory for partial differential equations: continuous and approximation theories. I_ , 74 (2000), xxii+644+I4, Abstract parabolic systems.
  * [35] by same author, _Control theory for partial differential equations: continuous and approximation theories. II_ , 75 (2000), i–xxii and 645–1067 and I1–I4, Abstract hyperbolic-like systems over a finite time horizon.
  * [36] H. V. Ly, K. D. Mease, and E. S. Titi, _Distributed and boundary control of the viscous Burgers’ equation_ , Numer. Funct. Anal. Optim. 18 (1997), no. 1-2, 143–188.
  * [37] P. S. Mantri, N. Nataraj, and A. K. Pani, _A qualocation method for Burgers’ equation_ , J. Comput. Appl. Math. 213 (2008), no. 1, 1–13.
  * [38] D. Matthes and S. Plazotta, _A variational formulation of the BDF2 method for metric gradient flows_ , ESAIM Math. Model. Numer. Anal. 53 (2019), no. 1, 145–172.
  * [39] A. K. Pany, N. Nataraj, and S. Singh, _A new mixed finite element method for Burgers’ equation_ , J. Appl. Math. Comput. 23 (2007), no. 1-2, 43–55.
  * [40] J.-P. Raymond, _Feedback boundary stabilization of the two-dimensional Navier-Stokes equations_ , SIAM J. Control Optim. 45 (2006), no. 3, 790–828.
  * [41] J.-P. Raymond and L. Thevenet, _Boundary feedback stabilization of the two dimensional Navier-Stokes equations with finite dimensional controllers_ , Discrete Contin. Dyn. Syst. 27 (2010), no. 3, 1159–1187.
  * [42] M. Renardy and R. C. Rogers, _An introduction to partial differential equations_ , Springer-Verlag, New York 13 (2004), xiv+434.
  * [43] L. Thevenet, J.-M. Buchot, and J.-P. Raymond, _Nonlinear feedback stabilization of a two-dimensional Burgers equation_ , ESAIM Control Optim. Calc. Var. 16 (2010), no. 4, 929–955.
  * [44] V. Thomée, _Galerkin finite element methods for parabolic problems_ , 25 (2006), xii+370.
  * [45] R. Triggiani, _On the stabilizability problem in Banach space_ , J. Math. Anal. Appl. 52 (1975), no. 3, 383–403.
  * [46] M. Tucsnak and G. Weiss, _Observation and control for operator semigroups_ , (2009), xii+483.
  * [47] S. Volkwein, _Distributed control problems for the Burgers equation_ , Comput. Optim. Appl. 18 (2001), no. 2, 115–140.