# Scaling Up Adaptive Filter Optimizers

Jonah Casebeer, , Nicholas J. Bryan, , Paris Smaragdis J. Casebeer is with the
Department of Computer Science, University of Illinois at Urbana-Champaign,
Urbana, IL 61801 USA (e-mail: jonah.casebeer@ieee.org).
N. J. Bryan is with Adobe Research, San Francisco, CA, 94103 USA (e-mail:
<EMAIL_ADDRESS>
P. Smaragdis is with the Department of Computer Science and Department,
University of Illinois at Urbana-Champaign, Urbana, IL 61801 USA (e-mail:
<EMAIL_ADDRESS>
###### Abstract

We introduce a new online adaptive filtering method called supervised multi-
step adaptive filters (SMS-AF). Our method uses neural networks to control or
optimize linear multi-delay or multi-channel frequency-domain filters and can
flexibly scale-up performance at the cost of increased compute – a property
rarely addressed in the AF literature, but critical for many applications. To
do so, we extend recent work with a set of improvements including feature
pruning, a supervised loss, and multiple optimization steps per time-frame.
These improvements work in a cohesive manner to unlock scaling. Furthermore,
we show how our method relates to Kalman filtering and meta-adaptive
filtering, making it seamlessly applicable to a diverse set of AF tasks. We
evaluate our method on acoustic echo cancellation (AEC) and multi-channel
speech enhancement tasks and compare against several baselines on standard
synthetic and real-world datasets. Results show our method performance scales
with inference cost and model capacity, yields multi-dB performance gains for
both tasks, and is real-time capable on a single CPU core.

###### Index Terms:

adaptive filtering, supervised adaptive filtering, acoustic echo cancellation,
beamforming, learning to learn

##  I Introduction

Adaptive filters (AF) play an indispensable role in a wide array of signal
processing applications such as acoustic echo cancellation, equalization, and
interference suppression. AFs are parameterized by time-varying filter weights
and require an update or optimization rule to control them over time.
Improving the performance of AFs continues to pose an intricate challenge,
requiring a nuanced approach to optimizer design. Consequently, AF algorithm
designers have relied on mathematical insights to create tailored optimizers,
starting from the foundational development of the least mean squares algorithm
(LMS) [1] to the Kalman filter [2, 3, 4, 5].

Figure 1: Acoustic echo cancellation performance vs. model size, optimization
steps per time-frame (opt. steps), and supervision levels. Bubble size shows
real-time-factor (RTF) where smaller is faster, inner-shape shows opt. steps,
and the vertical dotted line separates unsupervised (left) and supervised
(right) approaches. SMS-AF, in bold on the far right, demonstrate robust
scaling performance in terms of parameters, and RTF.

In contrast, we have witnessed countless remarkable deep learning algorithm
advancements in other domains through the principle of “scaling” [6, 7, 8, 9].
The scaling approach involves improving an existing method by deploying
additional computational resources. Scaling methodologies are particularly
enticing, as they tap into the increasing computational capabilities of modern
smart devices, minimizing the need for labor-intensive manual tuning and
intervention. In the context of neural networks for online low-latency AFs
that can benefit from scaling, we find two general approaches: 1) model-based
methods that integrate deep neural networks (DNNs) into existing AF frameworks
to update optimizer statistics [10, 11, 12], step-size [13, 14], or other
quantities [15] and 2) model-free strategies that do not rely on an existing
AF strategy, learn AF optimizers using meta-learning in an end-to-end fashion
[16, 17, 18, 19], and yield state-of-the-art (SOTA) results [15, 19]. We focus
on the latter, given their recent success, but note scaling such approaches
have either been limited to high-latency regimes [18] or only marginally
improves results [15].

We propose a new online AF method called supervised multi-step AF (SMS-AF).
Our method integrates a series of algorithm improvements on top of recent
meta-learning methods [17, 18] that together enable scaling performance by
increasing model capacity and/or inference cost as shown in Fig. 1. We
evaluate our approach on the tasks of acoustic echo cancellation (AEC) and
generalized sidelobe canceller (GSC) speech enhancement and compare to recent
SOTA approaches. Results show that our scaling behavior translates to
substantial performance gains in all metrics across tasks and datasets and
delivers breakthrough AEC performance

Our contributions include: 1) A new general purpose AF method that allow us to
reliably improve performance by simply using more computation, 2) Design
insights for customizing our proposed method for the task of AEC and GSC, 3)
Empirical exploration of AF optimizer scaling showing our approach scales vs.
model size and optimizer step count, and 4) Insights as to how our approach
generalizes Kalman filtering.

##  II Background

###  II-A Adaptive Filters

An AF is an optimization procedure that seeks to adapt filter parameters to
fit an objective over time. AFs typically input a mixture
$\underline{\mathbf{d}}[\tau]$, adjust a time-varying linear filter
$h_{\bm{\theta}[\tau]}$ with parameters $\bm{\theta}[\tau]$ to remove noise
via knowledge from a reference signal $\underline{\mathbf{u}}[\tau]$, produce
estimate $\underline{\mathbf{y}}[\tau]$, and output an error signal
$\underline{\mathbf{e}}[\tau]$. We focus on multi-delay and/or multi-channel
frequency-domain filters (MDF) for low-latency processing. The filter
parameters are updated across time by minimizing a loss,
$\mathcal{L}(h_{\bm{\theta}[\tau-1]},\cdots)$, resulting in a per frame $\tau$
update rule,

$\bm{\theta}[\tau]=\bm{\theta}[\tau-1]+\bm{\Delta}[\tau].$ (1)

$\bm{\Delta}[\tau]$ can also be written as the output of an optimizer
$g_{\bm{\phi}}(\bm{\xi}[\tau])=\bm{\Delta}[\tau]$ with input $\bm{\xi}[\tau]$,
and parameters $\bm{\phi}$.

###  II-B Adaptive Filter Optimizers

The AF optimizer, $g_{\bm{\phi}}(\bm{\xi}[\tau])$ is key and can vary in
levels of sophistication. In the simple case, the optimizer can be a hand-
derived algorithm such as LMS. In this case, each parameter in $\bm{\theta}$
is updated independently, so $g$ accepts the gradient with respect to the loss
via $\bm{\xi}[\tau]$, and is only parameterized by the step-size
$\bm{\phi}=\\{\lambda\\}$, resulting in $g$ simply scaling the gradient. Most
AFs operate via the following steps [4]:

$\displaystyle\underline{\mathbf{e}}[\tau]$ $\displaystyle=$ $\displaystyle
h_{\bm{\theta}[\tau-1]}(\underline{\mathbf{d}}[\tau],\underline{\mathbf{u}}[\tau])$
(2) $\displaystyle\bm{\Delta}[\tau]$ $\displaystyle=$ $\displaystyle
g_{\bm{\phi}}(\bm{\xi}[\tau],\cdots)$ (3) $\displaystyle\bm{\theta}[\tau]$
$\displaystyle=$ $\displaystyle\bm{\theta}[\tau-1]+\bm{\Delta}[\tau],$ (4)

where (2) applies the filter, (3) updates the optimizer, and (4) updates the
filter parameters for the next frame. Optimizers typically use filter output
$\underline{\mathbf{e}}[\tau]$, produced via $\bm{\theta}[\tau-1]$, creating a
feedback loop. The Kalman filter (KF) extends the above via distinct “predict”
and “update” steps. In the KF predict step, (2)-(4) are run as normal. In the
KF update step, however, the filter output is reprocessed after (4) using the
latest data:

$\displaystyle\underline{\mathbf{e}}[\tau]$ $\displaystyle=$ $\displaystyle
h_{\bm{\theta}[\tau]}(\underline{\mathbf{d}}[\tau],\underline{\mathbf{u}}[\tau]).$
(5)

###  II-C Learned Optimizers

Historically, AFs are hand-derived, given a loss and filter. In contrast,
neural-AF optimizers can be trained via meta-learning (Meta-AF) [20, 16, 17].
Meta-AFs are trained to control MDF filters via recurrent neural networks
(RNNs) with parameters $\bm{\phi}$ that are trained to maximize AF performance
on a large dataset via an unsupervised (or self-supervised) meta-objective
$\mathcal{L}_{M}(\;g_{\bm{\phi}},\mathcal{L}(h_{\bm{\theta}},\cdots)\;)$ and
backpropagation through time (BPTT). A common meta-loss is,

$\displaystyle\mathcal{L}_{M}(\underline{\bar{\mathbf{e}}})$ $\displaystyle=$
$\displaystyle\ln E[\|\underline{\bar{\mathbf{e}}}[\tau]\|^{2}],$ (6)

where
$\underline{\bar{\mathbf{e}}}[\tau]=\mathrm{cat}(\underline{\mathbf{e}}[\tau],\cdots,\underline{\mathbf{e}}[\tau+L-1])\in\mathbb{R}^{RL}$,
$L$ is the truncation length, $R$ is the hop size, and $\mathrm{cat}$
concatenates.

Two important extensions to Meta-AF include 1) higher-order Meta-AF (HO-Meta-
AF) [18], which introduces learnable coupling modules to model groups of
filter parameters, reduce complexity, and improve performance for high-latency
single-block frequency-domain filters and 2) low-complexity neural Kalman
filtering (NKF) [15], which extends Meta-AF with a KF, a supervised loss, and
different training setup. We regard Meta-AF as SOTA for unsupervised AFs (see
Table IV [17]) and NKF as SOTA for supervised AFs [15].

##  III Scaling Up Learned Optimizers

As the foundation of our SMS-AF method, we combine Meta-AFs [17] with a
higher-order optimizer [18] with per-frequency inputs
$\bm{\xi}_{\mathrm{k}}[\tau]$, and then extend it with three task-agnostic
improvements and one task-specific change. Our training and inference methods
are summarized in Alg. 1.

###  III-A Scaling Up Feature Quality: Feature Pruning

Our first insight is to use only three features to control filter adaptation:
knowledge of the filter input, final filter output, and filter state. Compared
to past work [17] that uses,

$\displaystyle\bm{\xi}_{\mathrm{k}}[\tau]=[\nabla_{\mathrm{k}}[\tau],\mathbf{u}_{\mathrm{k}}[\tau],\mathbf{d}_{\mathrm{k}}[\tau],\mathbf{e}_{\mathrm{k}}[\tau],\mathbf{y}_{\mathrm{k}}[\tau]],$
(7)

where $\nabla_{\mathrm{k}}[\tau]$ is a gradient w.r.t. loss
$\mathcal{L}(\cdots)$, we use

$\displaystyle\bm{\xi}_{\mathrm{k}}[\tau]=[\mathbf{u}_{\mathrm{k}}[\tau],\mathbf{e}_{\mathrm{k}}[\tau],\bm{\theta}_{\mathrm{k}}[\tau]].$
(8)

Pruning reduces complexity by lowering input dimension and memory requirements
for the optimizer, while eliminating inference-time gradients, as shown in
line $8$ of Alg. 1.

###  III-B Scaling Up Supervision: Supervised Loss

Our second insight is to use a high-quality supervised loss, instead of an
unsupervised loss. Previous methods have explored supervised losses such as
frame-wise independent supervised losses for echoes [15] or oracle filter
parameters [11]. These methods treat frequency bins, adjacent frames, and
other channels as distinct optimization entities and have not scaled [15]. As
such, we compute our supervised loss in the time-domain after all AF
operations have been performed. This strategy is similar to (6), but with
supervision. Our supervision is non-causal; the loss at $\tau$ depends on
updates from $<\tau$, enabling the optimizer to learn anticipatory updates.
For AEC,

$\displaystyle\mathcal{L}_{S}(\mathbf{d}_{\mathbf{u}},\mathbf{e})$
$\displaystyle=$ $\displaystyle\ln
E[\|\underline{\bar{\mathbf{d}}}_{\mathbf{u}}[\tau]-\underline{\bar{\mathbf{e}}}[\tau]\|^{2}],$
(9)

where $\mathbf{d}_{\mathbf{u}}[\tau]=\mathbf{u}[\tau]\ast\mathbf{w}[\tau]$ is
the true echo. For GSC, we use scale-invariant signal-to-distortion ratio (SI-
SDR). Better loss functions exclusively impact the training phase, without
contributing to test-time complexity, making this change cost-free for
inference. This corresponds to line $22$ of Alg. 1.

###  III-C Scaling Up Feedback: Multi-Step Optimization

Our third insight is to leverage the iterative nature of optimizers by
executing multiple optimization steps per time frame. By doing so, we offer
our optimizers a more powerful feedback mechanism and use the most current
parameters for the filter output. Specifically, we run our optimizer update
via (2)-(4), (2)-(5), or looping over (2)-(5) multiple times. The first option
follows Meta-AF, the second option follows a typical KF, and the third extends
the KF. We denote the number of (2)-(4) iterations via $C$. Incorporating
multi-step optimization in Alg. 1 involves three changes. First, initializing
each frame’s filter and optimizer state with results from the last frame
(lines $3-4$). Second, iteratively progressing through steps withing a frame
(line $5$), while updating filter parameters/outputs, and optimizer state
(lines $8-10$). Last, running a final filter forward pass using the latest
parameters (line $11$). We find this approach to be a compelling alternative
to increasing the dimension of the optimizer, $H$. Notably, it avoids
increasing the parameter count, and it linearly scales complexity, in stark
contrast to the quadratic complexity effects associated with $H$.

Algorithm 1 Training and inference algorithm.

1:function
Forward($g_{\bm{\phi}},h_{\bm{\theta}},\underline{\mathbf{u}},\underline{\mathbf{d}}_{\mathrm{m}}$)

2: for $\tau\leftarrow 0\textrm{ to }L$ do

3: $\bm{\theta}_{\mathrm{k}}[\tau]$ $\leftarrow$
$\bm{\theta}_{\mathrm{k}}[\tau-1]$ $\triangleright$ Initialize with last
estimate

4: $\bm{\psi}_{\mathrm{k}}[\tau]$ $\leftarrow$
$\bm{\psi}_{\mathrm{k}}[\tau-1]$ $\triangleright$ Initialize with last state

5: for $\mathrm{c}\leftarrow 0\textrm{ to }C$ do $\triangleright$ For each PU
iteration

6: $\underline{\mathbf{e}}[\tau]$ $\leftarrow$
$h_{\bm{\theta}[\tau]}(\underline{\mathbf{d}}[\tau],\underline{\mathbf{u}}[\tau])$

7: for $\mathrm{k}\leftarrow 0\textrm{ to }K$ do $\triangleright$ Predict step

8: $\bm{\xi}_{\mathrm{k}}[\tau]$ $\leftarrow$
$[\mathbf{u}_{\mathrm{k}}[\tau],\mathbf{e}_{\mathrm{k}}[\tau],\bm{\theta}_{\mathrm{k}}[\tau]$

9: $(\bm{\Delta}_{\mathrm{k}}[\tau],\bm{\psi}_{\mathrm{k}}[\tau])$
$\leftarrow$
$g_{\bm{\phi}}(\bm{\xi}_{\mathrm{k}}[\tau],\bm{\psi}_{\mathrm{k}}[\tau])$

10: $\bm{\theta}_{\mathrm{k}}[\tau]$ $\leftarrow$
$\bm{\theta}_{\mathrm{k}}[\tau]+\bm{\Delta}_{\mathrm{k}}[\tau]$

11: $\underline{\mathbf{e}}[\tau]$ $\leftarrow$
$h_{\bm{\theta}[\tau]}(\underline{\mathbf{d}}[\tau],\underline{\mathbf{u}}[\tau])$
$\triangleright$ Update step

12: $\bar{\mathbf{e}}$ $\leftarrow$
Cat$(\underline{\mathbf{e}}[\tau]\;\forall\tau)$

13: return $\bar{\mathbf{e}},\bm{\psi}[\tau],h_{\bm{\theta}[\tau]}$

14:function Train($\mathcal{D}$)

15: $\bm{\phi}$ $\leftarrow$ [21] init

16: while $\bm{\phi}$ not Converged do

17: $\underline{\mathbf{u}},\underline{\mathbf{d}}_{\mathrm{m}}$ $\leftarrow$
Sample($\mathcal{D}$) $\triangleright$ Get batch from $\mathcal{D}$

18: $\bm{\theta},\bm{\psi}$ $\leftarrow$ $\mathbf{0},\mathbf{0}$

19: for $n\leftarrow 0\textrm{ to end }$ do

20: $\underline{\mathbf{u}},\underline{\mathbf{d}}_{\mathrm{m}}$ $\leftarrow$
NextL$(\underline{\mathbf{u}},\underline{\mathbf{d}}_{\mathrm{m}})$
$\triangleright$ Grab $L$ frames

21: $\underline{\mathbf{u}},\bm{\psi},h_{\bm{\theta}}$ $\leftarrow$
Forward($g_{\bm{\phi}},\bm{\psi},h_{\bm{\theta}},\underline{\mathbf{u}},\underline{\mathbf{d}}_{\mathrm{m}}$)

22: $L_{S}$ $\leftarrow$ $\mathcal{L}_{S}(\cdots)$ $\triangleright$ Task-
dependent objective

23: $\bm{\nabla}$ $\leftarrow$ Grad($L_{S}$, $\bm{\phi}$) $\triangleright$
Truncated BPTT

24: $\bm{\phi}[n+1]$ $\leftarrow$ MetaOpt($\bm{\phi}[n],\bm{\nabla}$)
$\triangleright$ i.e. Adam return $\hat{\bm{\phi}}$

###  III-D Modifications to Overlap-Save for AEC

When using overlap-save, we noticed artifacts due to rapid filter adaptation.
So, we applied a straightforward solution: overlap-add with a synthesis
window, but no analysis window.

###  III-E Perspectives

Our modifications are a notable departure from the Meta-AF methodology, but
still aim to learn a neural optimizer for AFs end-to-end. First, by pruning
inputs features and introducing a supervised loss, we eliminate the need for
explicit meta-learning, leading to a more streamlined BPTT training process.
Second, we replace the past unsupervised loss with a new, strong supervised
signal and loss, helping us scale up. Third, we leverage a multi-step
optimization scheme. This creates a generalization of the Kalman filter, where
all parameters are entirely learned, while retaining explicit predict and
update steps. This also effectively deepens our optimizer networks by sharing
parameters across layers in a depth-wise manner.

##  IV Experimental Design

###  IV-A Experiments

The goal of our experiments is to benchmark SMS-AF and demonstrate how it
scales. To do so, we perform an initial within method ablation, then study
scaling on AEC and GSC tasks and vary 1) optimizer model sizes with small (S),
medium (M), and large (L) models 2) an unsupervised (U) or supervised (S) loss
and 3) the number of predict (P) and update (U) steps per frame. Each
experiment is labeled with an identifier (e.g. S$\cdot$S$\cdot$PU), indicating
the size, supervision, and number of PU steps. We label baselines when
applicable.

###  IV-B AEC Configuration

Our AEC signal model is
$\underline{\mathbf{d}}[{\mathrm{t}}]=\underline{\mathbf{u}}[{\mathrm{t}}]\ast\underline{\mathbf{w}}+\underline{\mathbf{n}}[{\mathrm{t}}]+\underline{\mathbf{s}}[{\mathrm{t}}]$,
where $\underline{\mathbf{n}}$ stands for noise, and $\underline{\mathbf{s}}$
is speech. The goal is to recover the speech $\underline{\mathbf{s}}$ given
the far-end $\underline{\mathbf{u}}$, and mixture $\underline{\mathbf{d}}$.
This involves fitting a filter to mimic $\underline{\mathbf{w}}$. We use a
linear MDF filter with $8$ blocks, each of size $512$, a hop of $256$, and
construct the output using overlap-add with a Hann window. Our baselines are
NLMS, KF [22], Meta-AF [17], HO-Meta-AF [18], and Neural-Kalman Filter [15].
We also test several HO-Meta-AF model sizes as well as multi-step NLMS, KF,
and HO-Meta-AF. For training, we use the synthetic fold of the Microsoft AEC
Challenge [23]. Each scene has double-talk, near-end noise and loud-speaker
nonlinearities. We also evaluate on the real, crowd-sourced, blind test-set
[23]. We use echo return loss enhancement (ERLE) [24] to measure echo
reduction. To describe perceptual quality, we use AEC-MOS, a reference-free
model that predicts a 5 point score [23]. On real data, we prefix with an R,
use ERLE in single-talk and AEC-MOS in double-talk. To quantify complexity, we
use mega FLOP (MFLOP) counts, single core real-time-factor (RTF) equal to
processing over elapsed time, and model size.

###  IV-C GSC Configuration

For GSC, we use a single-block frequency-domain GSC beamformer. The signal
model at each of the $M$ microphones is
$\underline{\mathbf{u}}_{{\mathrm{m}}}[{\mathrm{t}}]=\underline{\mathbf{r}}_{{\mathrm{m}}}[{\mathrm{t}}]\ast\underline{\mathbf{s}}[{\mathrm{t}}]+\underline{\mathbf{n}}_{{\mathrm{m}}}[{\mathrm{t}}]$,
and $\underline{\mathbf{r}}_{{\mathrm{m}}}[{\mathrm{t}}]$ is the impulse
response from source to mic ${\mathrm{m}}$. The goal is to recover the clean
speech $\underline{\mathbf{s}}$ given the input signal
$\underline{\mathbf{u}}_{{\mathrm{m}}}[{\mathrm{t}}]$. This requires fitting a
filter to remove the effects of noise,
$\underline{\mathbf{n}}_{{\mathrm{m}}}$. We assume access to a steering vector
and compare against NLMS, recursive-lease-squares (RLS), and Meta-AF. We test
multiple HO-Meta-AF model sizes as well as multi-step NLMS, RLS, and HO-Meta-
AF. We used the CHIME-3 [25] dataset. For overall quality, we compute scale-
invariant signal-to-distortion ratio (SI-SDR) [26], and contrast signal-to-
interference ratio (SIR), and signal-to-artifact ratio (SAR) [27]. For
perceptual quality, we use Short-Time Objective Intelligibility (STOI) [28].

###  IV-D Optimizer Configuration

For AEC and GSC, we use higher-order Meta-AF optimizers with banded coupling,
and a group size of $5$ [18]. This amounts to a Conv1D layer, two GRU layers,
and a transposed Conv1D layer. To train, we use Adam with a batch of $16$, a
learning rate of $10^{-4}$, and randomize the truncation length $L$ with a
maximum of $128$. We apply gradient clipping and reduce the learning rate by
half if the validation performance does not improve for $10$ epochs, and stop
training after $30$ epochs with no improvement. We use log-MSE loss on the
echo for AEC, and SI-SDR loss on the clean speech for GSC. All models are
trained on one GPU. Note, our S, M, and L model sizes correspond to
$g_{\bm{\phi}}$ hidden state sizes of $16$, $32$, and $64$ with parameters
counts of about $5$K, $16$K, and $57$K.

##  V Results & Discussion

TABLE I: AEC performance vs. computational cost. Model |  | ERLE$\uparrow$ |  |  R-ERLE$\uparrow$ |  |  AEC- MOS$\uparrow$ |  |  R-AEC- MOS$\uparrow$ |  | MFLOPs$\downarrow$ |  | RTF$\downarrow$
---|---|---|---|---|---|---|---|---|---|---|---|---
Mixture |  | - |  | - |  | 2.33 |  | 2.69 |  | - |  | -
NLMS$\cdot$P |  | 4.37 |  | 1.74 |  | 3.16 |  | 3.06 |  | 0.08 |  | 0.31
KF$\cdot$P |  | 5.44 |  | 2.71 |  | 3.32 |  | 3.18 |  | 0.12 |  | 0.32
KF$\cdot$PU |  | 6.65 |  | 3.83 |  | 3.65 |  | 3.38 |  | 0.12 |  | 0.35
KF$\cdot$PUx2 |  | 7.15 |  | 5.58 |  | 3.89 |  | 3.66 |  | 0.18 |  | 0.39
M$\cdot$U$\cdot$P [17] |  | 6.30 |  | 2.99 |  | 3.62 |  | 3.31 |  | 9.02 |  | 0.41
S$\cdot$U$\cdot$P |  | 7.09 |  | 3.30 |  | 3.57 |  | 3.26 |  | 2.80 |  | 0.36
M$\cdot$U$\cdot$P |  | 7.85 |  | 3.75 |  | 3.62 |  | 3.30 |  | 7.07 |  | 0.39
L$\cdot$U$\cdot$P |  | 8.03 |  | 3.91 |  | 3.65 |  | 3.33 |  | 20.42 |  | 0.48
NKF$\cdot$S$\cdot$PU |  | 9.29 |  | 6.38 |  | 3.69 |  | 3.59 |  | 10.16 |  | -
S$\cdot$S$\cdot$P |  | 8.63 |  | 3.81 |  | 3.63 |  | 3.31 |  | 2.80 |  | 0.36
M$\cdot$S$\cdot$P |  | 9.84 |  | 4.62 |  | 3.73 |  | 3.40 |  | 7.07 |  | 0.39
L$\cdot$S$\cdot$P |  | 11.62 |  | 5.52 |  | 3.83 |  | 3.50 |  | 20.42 |  | 0.48
S$\cdot$S$\cdot$PU |  | 9.13 |  | 6.05 |  | 3.85 |  | 3.55 |  | 2.81 |  | 0.38
M$\cdot$S$\cdot$PU |  | 11.22 |  | 7.87 |  | 3.99 |  | 3.72 |  | 7.08 |  | 0.41
L$\cdot$S$\cdot$PU |  | 13.34 |  | 9.78 |  | 4.05 |  | 3.85 |  | 20.43 |  | 0.49
S$\cdot$S$\cdot$PUx2 |  | 9.80 |  | 6.96 |  | 3.85 |  | 3.66 |  | 5.56 |  | 0.50
M$\cdot$S$\cdot$PUx2 |  | 11.77 |  | 8.86 |  | 4.04 |  | 3.80 |  | 14.11 |  | 0.56
L$\cdot$S$\cdot$PUx2 |  | 14.25 |  | 11.15 |  | 4.12 |  | 3.94 |  | 40.81 |  | 0.69

###  V-A Initial Within Method Ablation

We perform an initial within-method ablation on the task of AEC to understand
our modifications. First, we compare an M$\cdot$U$\cdot$P variant trained with
the full feature vs. pruned feature set. The full feature set achieves an ERLE
of $6.39$dB (not shown), while the pruned set scores $7.85$dB, over a dB gain.
We expand on the pruned model and add supervision, resulting in an ERLE
improvement to $9.84$dB, a gain of nearly $2$dB. We then use multiple steps
per frame. Extending the supervised model with an update-step increases ERLE
to $11.22$dB, and doubling the iterations reaches $11.77$dB. To mitigate
clicking artifacts, we then add our modified OLA scheme. This change reduces
the ERLE by $\approx 1$ dB ERLE, but removes severe clicking artifacts.
Combined, this yields a $4$dB ERLE gain.

###  V-B AEC Scaling Ablation and Benchmarking

Next, we explore scaling in AEC as shown in Fig. 1 and Table I. We attempt to
scale up our baselines and then do so with our proposed model. When scaling
model size, we notice that scaling the unsupervised model from S to L
(S$\cdot$U$\cdot$P to L$\cdot$U$\cdot$P) results in a peak gain of $\approx
1$dB, to $8.03$dB ERLE. In contrast, scaling the supervised model from S to L
(S$\cdot$S$\cdot$P to L$\cdot$S$\cdot$P) yields larger gains, peaking at
$11.62$dB. When scaling optimization steps, we find the unsupervised models
from S$\cdot$U$\cdot$P to S$\cdot$U$\cdot$PUx2 results in marginal or even a
negative performances changes. In contrast, scaling from S$\cdot$S$\cdot$P to
S$\cdot$S$\cdot$PUx2 provides $+1$dB, and L$\cdot$S$\cdot$P to
L$\cdot$S$\cdot$PUx2 provides $+2.65$dB, showing supervision is crucial to
unlock the benefit of multiple opt. steps per frame. Our best$\cdot$performing
L$\cdot$S$\cdot$PUx2 scores over $14$dB ERLE, doubling the S$\cdot$U$\cdot$P
performance of $7.09$dB.

When benchmarking against competing methods, we note that SOTA sueprvised NKF
method is most comparable. Our S$\cdot$S$\cdot$PU model matches NKF
performance while using only one-fifth of the NKF MFLOP count. Our
M$\cdot$S$\cdot$PU model further enhances all metrics and uses fewer MFLOPs.
For our best-performing L$\cdot$S$\cdot$PUx2, we score $14.25$dB ERLE, a
$4.96$dB improvement over NKF. In perceptual metrics, our top-performing
L$\cdot$S$\cdot$PUx2 model achieves $11$dB in R-ERLE and a R-AEC-MOS of
$3.94$, while remaining real-time on a single CPU core. Surprisingly, RTF
scales non-linearly with MFLOPs and model size, showing untapped scaling
potential.

TABLE II: Beaforming performance vs. computational cost. Model |  | SI-SDR$\uparrow$ |  | SIR$\uparrow$ |  | SAR$\uparrow$ |  | STOI$\uparrow$ |  | MFLOPs$\downarrow$ |  | RTF$\downarrow$
---|---|---|---|---|---|---|---|---|---|---|---|---
Mixture |  | -0.71 |  | - |  | - |  | 0.674 |  | - |  | -
NLMS$\cdot$P |  | 8.60 |  | 16.21 |  | 9.78 |  | 0.905 |  | 0.43 |  | 0.36
NLMS$\cdot$PU |  | 8.84 |  | 16.54 |  | 10.00 |  | 0.910 |  | 0.47 |  | 0.47
RLS$\cdot$P |  | 9.84 |  | 16.70 |  | 9.70 |  | 0.919 |  | 0.53 |  | 0.50
RLS$\cdot$PU |  | 10.14 |  | 17.16 |  | 11.49 |  | 0.924 |  | 0.54 |  | 0.62
S$\cdot$U$\cdot$P |  | 12.20 |  | 22.57 |  | 12.79 |  | 0.931 |  | 4.70 |  | 0.41
M$\cdot$U$\cdot$P |  | 12.62 |  | 22.56 |  | 13.26 |  | 0.938 |  | 12.08 |  | 0.45
L$\cdot$U$\cdot$P |  | 12.45 |  | 22.43 |  | 13.09 |  | 0.935 |  | 36.35 |  | 0.53
S$\cdot$S$\cdot$P |  | 13.92 |  | 23.00 |  | 14.66 |  | 0.950 |  | 4.70 |  | 0.41
M$\cdot$S$\cdot$P |  | 14.34 |  | 23.45 |  | 15.07 |  | 0.953 |  | 12.08 |  | 0.45
L$\cdot$S$\cdot$P |  | 14.69 |  | 24.36 |  | 15.33 |  | 0.954 |  | 36.35 |  | 0.53
S$\cdot$S$\cdot$PU |  | 15.46 |  | 25.69 |  | 16.09 |  | 0.956 |  | 4.74 |  | 0.51
M$\cdot$S$\cdot$PU |  | 16.83 |  | 27.70 |  | 17.41 |  | 0.960 |  | 12.12 |  | 0.54
L$\cdot$S$\cdot$PU |  | 17.22 |  | 28.37 |  | 17.80 |  | 0.962 |  | 36.39 |  | 0.62
S$\cdot$S$\cdot$PUx2 |  | 15.67 |  | 25.89 |  | 16.35 |  | 0.956 |  | 9.07 |  | 0.70
M$\cdot$S$\cdot$PUx2 |  | 17.06 |  | 28.42 |  | 17.63 |  | 0.961 |  | 23.83 |  | 0.76
L$\cdot$S$\cdot$PUx2 |  | 17.72 |  | 29.52 |  | 18.25 |  | 0.964 |  | 72.37 |  | 0.91

###  V-C GSC Beamforming Scaling and Benchmarking

Beamforming results are in Table II. Notably, SMS-AF improvements apply
without any modifications. Here, all models assume access to a steering
vector, which can be challenging to estimate in practice. Again, supervision
and multi-step optimization yield significant performance gains. Our
S$\cdot$S$\cdot$P model outperforms all baselines including L$\cdot$U$\cdot$P
across all metrics. Our model scales reliably with the L$\cdot$S$\cdot$P
variant improving performance in all metrics. Scaling up the iterations to
S$\cdot$U$\cdot$PUx2 yields larger gains across all metrics. Our largest and
best model, L$\cdot$S$\cdot$PUx2 scores a remarkable 17.72 dB SI-SDR while
still being real-time. Of note, the L$\cdot$S$\cdot$PU model has the same RTF
as RLS$\cdot$PU, even though RLS uses fewer operations. Again, we show that
SMS-AF performance scales with both model capacity and optimization steps per
frame.

##  VI Conclusion

We introduce a method for a neural network-based adaptive filter optimizers
called supervised multi-step adaptive filters (SMS-AF). We extend meta-
adaptive filtering methods with several advances that combine to reliably
increase performance by leveraging more computation. We evaluate our method on
low latency, online AEC and GSC tasks, compare against many baselines and test
on both synthetic and real data. SMS-AF improves both subjective and objective
metrics, achieving $\approx 5$dB ERLE/SI-SDR gains compared to prior work, and
increases the performance ceiling across AEC and GSC. Furthermore, we relate
our work to the Kalman filter and meta-AFs, giving insight for many other
applications. We believe scaling-up AFs is a promising direction and hope our
results encourage future work on scalable, general purpose AFs.

## References

  * [1] Bernard Widrow and Marcian E. Hoff,  “Adaptive switching circuits,”  Tech. Rep., Stanford University, 1960.
  * [2] V. John Mathews,  “Adaptive polynomial filters,”  IEEE Signal Processing Magazine (SPM), 1991.
  * [3] José Antonio Apolinário, José Antonio Apolinário, and R Rautmann,  QRD-RLS adaptive filtering,  Springer, 2009.
  * [4] Simon S. Haykin,  Adaptive filter theory,  Pearson, 2008.
  * [5] Lawrence R. Rabiner, Bernard Gold, and CK Yuen,  Theory and application of digital signal processing,  Prentice-Hall, 2016.
  * [6] Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger,  “Broken neural scaling laws,”  in International Conference on Learning Representations (ICLR), 2022.
  * [7] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.,  “Training compute-optimal large language models,”  arXiv preprint arXiv:2203.15556, 2022.
  * [8] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie,  “A ConvNet for the 2020s,”  in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [9] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park,  “Scaling up gans for text-to-image synthesis,”  in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [10] Jonah Casebeer, Jacob Donley, Daniel Wong, Buye Xu, and Anurag Kumar,  “NICE-Beam: Neural integrated covariance estimators for time-varying beamformers,”  arXiv:2112.04613, 2021.
  * [11] Thomas Haubner, Andreas Brendel, and Walter Kellermann,  “End-to-end deep learning-based adaptation control for frequency-domain adaptive system identification,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
  * [12] Thomas Haubner and Walter Kellermann,  “Deep learning-based joint control of acoustic echo cancellation, beamforming and postfiltering,”  in IEEE European Signal Processing Conference (EUSIPCO), 2022.
  * [13] Amir Ivry, Israel Cohen, and Baruch Berdugo,  “Deep adaptation control for acoustic echo cancellation,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
  * [14] Behrad Soleimani, Henning Schepker, and Majid Mirbagheri,  “Neural-afc: Learning-based step-size control for adaptive feedback cancellation with closed-loop model training,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
  * [15] Dong Yang, Fei Jiang, Wei Wu, Xuefei Fang, and Muyong Cao,  “Low-complexity acoustic echo cancellation with neural kalman filtering,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
  * [16] Jonah Casebeer, Nicholas J. Bryan, and Paris Smaragdis,  “Auto-DSP: Learning to optimize acoustic echo cancellers,”  in IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2021.
  * [17] Jonah Casebeer, Nicholas J. Bryan, and Paris Smaragdis,  “Meta-AF: Meta-learning for adaptive filters,”  IEEE Transactions on Audio, Speech, and Language Processing (TASLP), 2022.
  * [18] Junkai Wu, Jonah Casebeer, Nicholas J. Bryan, and Paris Smaragdis,  “Meta-learning for adaptive filters with higher-order frequency dependencies,”  in IEEE International Workshop on Acoustic Signal Enhancement (IWAENC), 2022.
  * [19] Jonah Casebeer, Junkai Wu, and Paris Smaragdis,  “Meta-af echo cancellation for improved keyword spotting,”  arXiv:2312.10605, 2023.
  * [20] Marcin Andrychowicz, Misha Denil, Sergio Gómez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas,  “Learning to learn by gradient descent by gradient descent,”  in NeurIPS, 2016.
  * [21] Moritz Wolter and Angela Yao,  “Complex gated recurrent neural networks,”  in NeurIPS, 2018.
  * [22] Gerald Enzner and Peter Vary,  “Frequency-domain adaptive Kalman filter for acoustic echo control in hands-free telephones,”  Elsevier Signal Processing, 2006.
  * [23] Ross Cutler, Ando Saabas, Tanel Parnamaa, Marju Purin, Hannes Gamper, Sebastian Braun, Karsten Sorensen, and Robert Aichner,  “ICASSP 2022 acoustic echo cancellation challenge,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
  * [24] Gerald Enzner, Herbert Buchner, Alexis Favrot, and Fabian Kuech,  “Acoustic echo control,”  in Academic press library in signal processing. Elsevier, 2014.
  * [25] Jon Barker, Ricard Marxer, Emmanuel Vincent, and Shinji Watanabe,  “The third CHiME speech separation and recognition challenge: Dataset, task and baselines,”  in Automatic Speech Recongition and Understanding Workshop (ASRU). IEEE, 2015.
  * [26] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R. Hershey,  “SDR–half-baked or well done?,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019.
  * [27] Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte,  “Performance measurement in blind audio source separation,”  IEEE Transactions on Audio, Speech, and Language Processing (TASLP), 2006.
  * [28] Cees H. Taal, Richard C. Hendriks, Richard Heusdens, and Jesper Jensen,  “An algorithm for intelligibility prediction of time–frequency weighted noisy speech,”  IEEE Transactions on Audio, Speech, and Language Processing (TASLP), 2011.


# Class-Wise Principal Component Analysis for hyperspectral image feature
extraction

Dimitra Koumoutsou
<EMAIL_ADDRESS>Eleni Charou
<EMAIL_ADDRESS>
Georgios Siolas
<EMAIL_ADDRESS>
Giorgos Stamou
<EMAIL_ADDRESS>
###### Abstract

This paper introduces the Class-wise Principal Component Analysis, a
supervised feature extraction method for hyperspectral data. Hyperspectral
Imaging (HSI) has appeared in various fields in recent years, including Remote
Sensing. Realizing that information extraction tasks for hyperspectral images
are burdened by data-specific issues, we identify and address two major
problems. Those are the Curse of Dimensionality which occurs due to the high-
volume of the data cube and the class imbalance problem which is common in
hyperspectral datasets. Dimensionality reduction is an essential preprocessing
step to complement a hyperspectral image classification task. Therefore, we
propose a feature extraction algorithm for dimensionality reduction, based on
Principal Component Analysis (PCA). Evaluations are carried out on the Indian
Pines dataset to demonstrate that significant improvements are achieved when
using the reduced data in a classification task.

Keywords— Principal Component Analysis, feature extraction, class imbalance,
hyperspectral image, dimensionality reduction

##  1 Introduction

_Hyperspectral imaging_ is essentially the recording of a "scene" in multiple
spectra, ranging from the visible to the Near Infra-Red (NIR) [1].
Hyperspectral images are comprised of continuous and contiguous spectral bands
at hundreds of narrow wavelength intervals. The resulting image is what we
call “hyperspectral scene”, which is the depiction of a geographic location
that contains both spectral and spatial information. Especially in the
spectral domain, the resolution of the information is very high. Due to this
fact, hyperspectral data offer great potential, therefore robust processing
strategies need to be employed for maximizing information extraction.

The data recorded from a Hyperspectral sensor have a 3-dimensional structure,
which is called _data cube_ [2], as shown in Figure 1. The first 2 dimensions
correspond to the spatial information, while the third dimension is the
spectral dimension. Hyperspectral data have a high resolution in both the
spectral and spatial domain. Because they contain spatial information for
hundreds of contiguous spectral bands, the hyperspectral cube is a high-volume
data cube.

_Class imbalance_ in a dataset is a common problem problem which can be
described formally as an unequal distribution of samples per class label [3].
Classes with fewer samples are likely to be under-represented in a
classification model, affecting its accuracy. There is also the possibility
that classes with few samples are more important for an application depending
on the desired task than their statistical insignificance suggests. In this
case, the model’s accuracy may seem acceptable, because all the samples play
an equal role, even if the majority belongs to one or few classes. The
minority of the “weak” class samples hardly affect the accuracy metric, making
it misguiding for evaluation.

Figure 1: The hyperspectral cube [4]. The first 2 dimensions correspond to the
spatial information, while the third dimension is the spectral dimension

The _Curse of Dimensionality_ , or Hughes effect, was first mentioned by
Richard Bellman in 1961 in his work in Dynamic Programming, as the term used
to describe the issue of “an increase in sparseness and dissimilarity of data”
which occurs as dimensionality increases [5]. Although it would seem that
denser information collected by sensors should result in more accurate
classification outcomes, this is not always the case with hyperspectral data.
The problem is that an increase in dimensionality will actually deteriorate
classification, affecting clustering, and the categorization of objects,
materials or targets in general. Large amounts of data do not always coincide
with a large amount of information; in fact, useful information is not present
in every data point of the hyperspectral cube. Information is sparse and
because spectral bands are recorded at very narrow wavelength intervals,
neighboring bands are most likely correlated. Therefore, using the original
recorded data cube does not only slow down the information extraction process;
it actually deteriorates it. This is partly because a significant amount of
data in the recording is actually noise, but also because training a neural
network with redundant data often leads to the model overfitting the training
data.

_Dimensionality Reduction (DR)_ [6] is the branch of mathematics used to
harvest the vast amount of information stored in hyperspectral data, while
dealing with the Curse of Dimensionality issue. Dimensionality Estimation and
Reduction is the process of transforming a cube with $(X,Y)$ spatial
dimensions, and $L$ spectral dimensions, represented as a $3-$D vector as
$(H,X,L)$, to a lower-dimensional cube $(H,Y,N)$, where $N<L$. Since many
channels in a hyperspectral image are correlated and/or noisy, redundant data
could be eliminated from datasets. The challenge of Dimensionality Reduction
is to dispose of unnecessary or low quality information, while retaining the
total of the significant features about the targets of interest. In Remote
Sensing HSI applications Dimensionality Reduction is used as a preprocessing
step, mainly in two main forms, band selection [7] or feature extraction [8].
So far, literature findings indicate that the latter is the optimal approach
and is an indispensable preprocessing step when analyzing hyperspectral data.
A powerful feature extraction algorithm should retain only useful information
and discard all redundancies in the dataset, so implementing it as a
preprocessing step in a classification model is expected to reduce
computational time and improve overall performance.

##  2 Principal Component Analysis (PCA)

Principal Component Analysis (PCA) [9] is also called Karhunen–Loève
transform. It is a powerful technique of multivariate analysis which seeks for
the projection that best represents data in the least square sense. PCA is
commonly used for dimensionality reduction in an automatic fashion. This
method is based on rotating the original data into a set of axes that
maximizes the variability in the first few axes. This is done by finding a new
coordinate system in the hyper-dimensional vector space where the data
exhibits no correlation [10]. Let,

$\displaystyle X_{i}=[x_{1},x_{2},...x_{N}]_{i}^{T}$ (2.1) $\displaystyle
m=\frac{1}{M}\displaystyle\sum\limits_{i=1}^{M}[x_{1},x_{2},...x_{N}]_{i}^{T}$
(2.2) $\displaystyle
C_{x}=\frac{1}{M}\displaystyle\sum\limits_{i=1}^{M}(x_{i}-M)(x_{i}-M)^{T}$
(2.3)

Where,
$X_{i}$ is a hyperspectral image vector of all $N$ dimensions of a given
pixel, located at $i$,
$x_{i}$ is a pixel at the i-th spectral band,
$M$ is the total number of pixels in each band,
$m$ is the mean value of all the pixels in the image,
$N$ is the number of spectral dimensions in the original data,
$C_{x}$ is the covariance matrix, which is diagonal or uncorrelated in the new
coordinate system.

A linear transformation, $U$, that transforms the original hyperspectral data,
$X_{i}$, into the new coordinate system $Y$ is then calculated. The original
covariance matrix $C_{x}$ becomes a diagonalized covariance matrix. The
solution becomes a generalized eigenvalue problem of the form:

$C_{x}=UDU^{T}$ (2.4)

where

$D=diag(\lambda_{1},\lambda_{2},..,\lambda_{N})$ $U=(u_{1},u_{2},..,u_{N})$

Here,
$D$ is the diagonal matrix of $\lambda_{I}$
$\lambda_{I}$ are the eigen-values of $C_{x}$
$C_{x}$ is the covariance matrix
$U$ is the orthogonal matrix, which is the matrix of $u_{i}$
$u_{i}$ are the eigenvectors.

Finally, the PCA transform is computed by

$Y_{i}=U^{T}X_{i}$ (2.5)

where

$i=1,2,\dots,M$

Each of the vectors $Y_{i}$ contains the reduced information of the original
hyperspectral image. The number of the components used in this final step
depends on the intended information extraction task and will be thoroughly
investigated in our experiments section.

##  3 Proposed Method

We introduce _Class-Wise Principal Component Analysis (CW-PCA)_ , which is a
supervised alternative to the classic PCA algorithm described in the previous
Section.

The motivation behind this new approach is that the features that contain most
of the variance are not necessarily those that contain the information we wish
to preserve. Besides, we should further consider the issue of class imbalance.
The classes that are already ill-represented in the original space are prone
to further elimination in the transformed subspace. The challenge that should
be addressed is the following; if the spectral information that is most
informative about a class is discarded during the dimensionality reduction
process, there is a risk of the class being misclassified. Although this could
occur both for “strong” and “weak” classes, it is only the weak ones that
would suffer the greatest hit. Since those classes are not predominant, they
heavily rely on the abundance of spectral features and the ability of a fit
classifier to recognize their unique spectral signatures. If that spectral
signature is altered in a way that discards the uniqueness of the
object/material, nothing else can be used to their advantage. On that account,
extracting principal components from the entire dataset poses a risk. We wish
to find a method to drastically reduce dimensionality with minimal loss of
information and maximal discarding of redundancy. To ensure that the low-
dimensional subspace bears all of the important features of the original
dataset, we introduce the concept of CW-PCA, which is outlined in Figure 2.

Figure 2: Outline for the proposed feature extraction technique, Class-Wise
Principcal Component Analysis (CW-PCA).

Let $(X,Y,L)$ be the dimensions of the original image where $(X,Y)$ denote the
spatial information and $C$ is the number of bands. Assuming there is a known
number of classes in the dataset ($N-$ classes), we may reduce dimensionality
from $L$ to a multiple of $N$. To implement this method, we first split the
original scene into $N$ subscenes. Each of them has the same dimensions
$(X,Y,L)$ as the original scene, but only has positive values in the
coordinates of the $N-$th class samples. Essentially, we create $N$ non-
overlapping subscenes each of whom only carries samples from one class. Then,
PCA is applied separately on each of these subscenes and one (or more)
Principal Components are extracted from each subscene. At the final step, the
Principal Components (PCs) are concatenated along the spectral dimension. For
example, if only the first PC is kept from each class-scene, at this point we
should have $N$ Principal Components, each with dimensions $(X,Y,1)$, and the
concatenated vector-subspace would be $(X,Y,N)$ dimensional. It is evident
from the described process that the number of PCs in the resulting subspace
should be a multiple of the number of classes. More specifically, if the
$m-$th first PCs are selected from each class, the resulting spectral
dimension is $m\cdot N$.

##  4 Experiments

Four feature extraction methods are tested, three of which are already used in
hyperspectral imaging for dimensionality reduction. Then, we propose a new
method, Class-wise PCA, which we will examine in more depth to determine
whether it can actually contribute to resolving the two problems we discussed:
dimensionality reduction and class imbalance. Although class imbalance can be
solved at data-level (for example with data augmenation techniques [11]), we
want to investigate if using a supervised method to incorporate class
information at the feature extraction stage can have an effect on
classification accuracy, especially in class-wise accuracy.

To rank the three methods against each other and evaluate our proposed method
accordingly, we use all four different methods to reduce dimensionality and
perform classification on the data. So far as unsupervised methods are
concernes, experimental results prove that 15 PCA components are enough to
convey the information needed to achieve maximum accuracy. On the other hand,
when transforming data with a supervised approach, class information is
considered in the transformed lower-dimensional space. In the case of LDA this
also means that, at most, the number of bands in the transformed space can be
equal to the number of classes. As previously stated, CW-PCA transforms data
to a number of bands that is an integer multiple of the number of classes.
Holding on to the same reasoning, we will keep one component from each class
in the CW-PCA algorithm, which will give us equal number of bands as the LDA
algorithm. Since there are 16 classes in our dataset, and we found the optimal
number of components in the unsupervised algorithms to be 15, we can have a
fair comparison across the four methods as the reduction in dimensionality is
roughly the same.

###  4.1 Dataset

The Indian Pines dataset is a publically available hyperspectral scene [12].
It consists of $145\times 145$ pixels in the spatial dimension and 224
spectral bands, although the corrected version we use has $220$, in the range
of $400$ to $2500nm$. Data was captured with the Airborne Visible/Infrared
Imaging Spectrometer (AVIRIS) sensor [13] in North-Western Indiana, USA, with
a Ground Sample Distance (GSD) of $20m$ and are labeled with 16 classes of
vegetation not all of which are mutually exclusive; more specifically there
are two-thirds agriculture, and one-third forest or other natural vegetation.
The 16 labeled classes are

> ’Alfalfa’, ’Corn-notill’, ’Corn-mintill’, ’Corn’, ’Grass-pasture’, ’Grass-
> trees’, ’Grass-pasture-mowed’, ’Hay-windrowed’, ’Oats’, ’Soybean-notill’,
> ’Soybean-mintill’, ’Soybean-clean’, ’Wheat’, ’Woods’, ’Buildings-Grass-
> Trees-Drives’, ’Stone-Steel-Towers’

Figure 3: (a) Accuracy and (b) Loss curves for the training and validation
set, when performing classification on data reduced by PCA, ICA, LDA and CW-
PCA. The CW-PCA method converges faster and shows no risk of overfitting.

###  4.2 Dimensionality Reduction

The first aspect is the dimensionality reduction. Essentially, we want to
determine if reducing the spectral dimensions by up to 90%, from the original
220 bands to only 16 (for the Indian Pines dataset), will accurately convey
all useful information and discard redundancies. This will be revealed in the
classification outcomes in two ways. If the dimensions are not effectively
reduced, the redundant information will introduce noise to the model and skew
the results. In contrary, if we shrink the input too much, we may have extreme
loss of information which will simply not suffice to train a reliable model.
We can see how each methods performs in terms of classification accuracy in
Figure 3(a), while the loss curve for the train and validation sets is given
in Figure 3(b). ICA has the worst performance out of the four, both in terms
of convergence speed (over 70 epochs) and accuracy. The PCA and LDA are
similar, with the latter outperforming PCA slightly. Hence it appears that
including class information in the preprocessing steps is of essense. Finally,
the CW-PCA has an outstanding performance, with very fast convergence in 10
epochs, which is 50% faster than PCA and LDA. Also, from the loss curve we see
that even after 90 training epochs, there is not a slightest hint for
overfitting. A quantitative evaluation of the classification scores is given
in Table 1, which corroborates the quantitative information.

DR Method | OA | AA
---|---|---
ICA | 89.43 | 81.48
PCA | 91.53 | 84.79
LDA | 96.43 | 90.12
CW-PCA | 99.95 | 99.82

Table 1: Classification accuracy scores (OA and AA) for different methods of
feature extraction (PCA, ICA, LDA, CW-PCA)

.

Figure 4: Percentages of training samples per class for the Indian Pines
Dataset. We observe the magnitude of the class imbalance problem.

###  4.3 Class imbalance

In Figure 4 we observe the significant class imbalance problem for the Indian
Pines Dataset, which is a benchmark dataset for the majority of hyperspectral
image classification models in the literature. Next we evaluate the CW-PCA
method against the other three algorithms with respect to their contribution
in solving the class imbalance problem. Although the improvement is already
evident from the Average Accuracy metric in Table 1, which expressed the
class-wise accuracy of our model, we will proceed with some visual results
that highlight the superiority of CW-PCA against other methods. This indicates
that the high accuracy scores are not a result of few classes dominating the
results and misguiding the evaluation scheme.

To further investigate the issue of class imbalance using the class-wise
accuracy, we plot the accuracy per class. As can be seen in the class
imbalance plot in Figure 4, the weak classes for the Indian Pines dataset are
1, 7, 9 and 16. In Figure 5 we see that CW-PCA has almost 100% accuracy even
in the case of weak classes, while other methods yield significantly poorer
results. LDA seems to perform better at weak classes than the unsupervised
counterparts, however there are still inconsistencies, for example class 16
has a 26% offset than all other methods. To highlight and better observe how
weak classes were classified, we “zoom in” on those four classed for a close-
up. The bottom graph in Figure 4 underlines the improvement in weak classes
and proves that CW-PCA outperforms all other methods. Amongst the three other
methods (PCA, ICA, LDA), there is no clear ranking. Even if LDA appears better
in most cases the results are not always consistent. Consequently, even
amongst supervised methods, CW-PCA is the most reliable choice.

Figure 5: (top) Class-wise accuracy using CW-PCA for dimensionality reduction
versus PCA, ICA, LDA. Clearly CW-PCA outperforms all other methods, (bottom)
Class-wise accuracy for weak classes of the Indian Pines Dataset, which have
less than 1% of total training samples (classes 1, 7, 9, 16). Comparison of
using CW-PCA for dimensionality reduction versus PCA, ICA, LDA.

## References

  * [1] Gary Shaw and Dimitris Manolakis  “Signal processing for hyperspectral image exploitation”  In _IEEE Signal processing magazine_ 19.1 IEEE, 2002, pp. 12–16
  * [2] José M Bioucas-Dias et al.  “Hyperspectral remote sensing data analysis and future challenges”  In _IEEE Geoscience and remote sensing magazine_ 1.2 IEEE, 2013, pp. 6–36
  * [3] Nathalie Japkowicz  “The class imbalance problem: Significance and strategies”  In _Proc. of the Int’l Conf. on Artificial Intelligence_ 56, 2000  Citeseer
  * [4] “Hyperspectral Cube”  URL: https://en.wikipedia.org/wiki/Hyperspectral_imaging#/media/File:HyperspectralCube.jpg
  * [5] Richard E Bellman  “Adaptive control processes: a guided tour”  Princeton university press, 2015
  * [6] Laurens Van Der Maaten, Eric Postma and Jaap Van den Herik  “Dimensionality reduction: a comparative”  In _J Mach Learn Res_ 10.66-71, 2009, pp. 13
  * [7] Weiwei Sun and Qian Du  “Hyperspectral band selection: A review”  In _IEEE Geoscience and Remote Sensing Magazine_ 7.2 IEEE, 2019, pp. 118–139
  * [8] Wenzhi Zhao and Shihong Du  “Spectral–spatial feature extraction for hyperspectral image classification: A dimension reduction and deep learning approach”  In _IEEE Transactions on Geoscience and Remote Sensing_ 54.8 IEEE, 2016, pp. 4544–4554
  * [9] Karl Pearson  “LIII. On lines and planes of closest fit to systems of points in space”  In _The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science_ 2.11 Taylor & Francis, 1901, pp. 559–572
  * [10] Jonathon Shlens  “A tutorial on principal component analysis”  In _arXiv preprint arXiv:1404.1100_ , 2014
  * [11] Connor Shorten and Taghi M Khoshgoftaar  “A survey on image data augmentation for deep learning”  In _Journal of Big Data_ 6.1 Springer, 2019, pp. 1–48
  * [12] “Indian Pines Dataset”  URL: http://www.ehu.eus/ccwintco/index.php/Hyperspectral_Remote_Sensing_Scenes#Indian_Pines
  * [13] Robert O Green et al.  “Imaging spectroscopy and the airborne visible/infrared imaging spectrometer (AVIRIS)”  In _Remote sensing of environment_ 65.3 Elsevier, 1998, pp. 227–248


# Domain Adaptation for Measurements of Strong Gravitational Lenses

Paxson Swierc1
Department of Astronomy and Astrophysics
University of Chicago
<EMAIL_ADDRESS>
Megan Zhao1
Department of Astronomy and Astrophysics
University of Chicago
<EMAIL_ADDRESS>
Aleksandra Ćiprijanović
Computational Science and AI Directorate
Fermi National Accelerator Laboratory;
Department of Astronomy and Astrophysics
University of Chicago
<EMAIL_ADDRESS>
Brian Nord
Computational Science and AI Directorate
Fermi National Accelerator Laboratory;
Department of Astronomy and Astrophysics
University of Chicago;
Kavli Institute for Cosmological Physics
University of Chicago
<EMAIL_ADDRESS>
###### Abstract

Upcoming surveys are predicted to discover galaxy-scale strong lenses on the
order of 105, making deep learning methods necessary in lensing data analysis.
Currently, there is insufficient real lensing data to train deep learning
algorithms, but the alternative of training only on simulated data results in
poor performance on real data. Domain Adaptation may be able to bridge the gap
between simulated and real datasets. We utilize domain adaptation for the
estimation of Einstein radius ($\Theta_{E}$) in simulated galaxy-scale
gravitational lensing images with different levels of observational realism.
We evaluate two domain adaptation techniques - Domain Adversarial Neural
Networks (DANN) and Maximum Mean Discrepancy (MMD). We train on a source
domain of simulated lenses and apply it to a target domain of lenses simulated
to emulate noise conditions in the Dark Energy Survey (DES). We show that both
domain adaptation techniques can significantly improve the model performance
on the more complex target domain dataset. This work is the first application
of domain adaptation for a regression task in strong lensing imaging analysis.
Our results show the potential of using domain adaptation to perform analysis
of future survey data with a deep neural network trained on simulated data.

##  1 Introduction

11footnotetext: Equal contribution.

Strong gravitational lensing is a powerful probe of astrophysics, as well as
cosmology. Galaxy-scale strong lensing systems in the Sloan Lens ACS (SLACS)
[5] survey gave insights into the dark and baryonic matter profiles in massive
elliptical galaxies [5, 7]. In cosmology, strong lensing systems have been
used as an independent probe of the Hubble constant through time delays in
multiple images in the system [9, 32] and for constraining the dark energy
equation of state [14, 31].

Upcoming large-scale astronomical surveys, including the Rubin Observatory’s
Legacy Survey of Space and Time (LSST)[22], the Hyper Suprime-Cam Subaru
Strategic Program (HSC)[3], Euclid[26], and the Nancy Grace Roman Space
Telescope 222https://www.jpl.nasa.gov/missions/the-nancy-grace-roman-space-
telescope , will observe an unprecedented number of lensed systems. Collett
[13] predicted that $>10^{5}$ galaxy scale strong lensing systems will be
observed by LSST and Euclid. The large amount of data from these surveys will
require much more efficient analysis techniques, such as those utilizing
machine learning (ML) techniques. Currently discovered lensing systems are on
the order of a thousand, which is inadequate to train many ML algorithms,
forcing researchers to use simulated datasets of various forms, such as in
[37, 28, 35]. For classification tasks, it has been shown that the inevitable
gap between training and testing data causes complex, large-parameter ML
models trained on labeled simulation data to perform poorly (i.e., systematic
biases) on new, unlabeled observational data [e.g., 6]. The same issues with
classification are present in regression tasks, requiring an advancement to
mitigate biases in deep learning analyses.

Domain adaptation (DA) is a class of techniques used when the distribution of
training data differs from that of the testing data [15]. In DA, the training
distribution of data is the source data, while the data it will be tested on
is the target data. We are specifically evaluating unsupervised domain
adaptation [30], which is the problem scenario where one has access to labeled
source data, but target data is unlabeled. Unsupervised DA techniques adjust
training to include unlabeled target data, detect the difference between the
source and the target domain distributions, and aim to minimize it. In
astronomy, DA has been applied to multiple scenarios of classification [e.g.,
10, 12, 11, 4, 33]), generative models [e.g., 23, 25], and regression in one
scenario [20].

We apply two DA techniques to strong gravitational lensing analysis in this
work: adversarial training using Domain Adversarial Neural Networks (DANNs)
[19] and a distance-based method called Maximum Mean Discrepancy (MMD) [21].
We apply both techniques to the task of estimating the Einstein radius
($\Theta_{E}$) of galaxy-scale gravitational lensing systems. Both source and
target domain data are simulated using
deeplenstronomy111https://github.com/deepskies/deeplenstronomy [24]. The
source data is simulated strong lenses, with no noise. For target data, the
same strong lenses are simulated, but with emulated Dark Energy Survey
(DES)[1] conditions. We show that both techniques improve the model
performance on the target domain, demonstrating the capacity of using DA to
bridge the gap between simulated and real data.

Our manuscript is organized as follows. In Section 2 we describe the two DA
methods utilized in this work; in Section 3 we describe the data; in Section 4
we describe the neural network model; in Section 5 we present our results; and
in Section 6 we discuss our findings.

##  2 Methods

Both DA approaches used are unsupervised techniques: while they require
labeled data from the source domain, they leverage only unlabeled data from
the target domain. This mirrors the problem setting of training on labeled
simulated data, while only having access to unlabeled real data. We assume
that the two domains have the same conditional probability distributions of a
label $y$, given some data $x$, ${p}_{s}{(y|x)}={p}_{t}{(y|x)}$, but that the
marginal probability distribution of the data $x$ within the domains
themselves is not equal: ${p}_{s}{(x)}\neq{p}_{t}{(x)}$. This is known as a
covariate shift between domains [18]. Under this assumption, feature-based DA
techniques such as DANN and MMD aim to close the gap by learning a
transformation that maps source data into target data [18]. DANN and MMD find
this transformation by utilizing domain invariant features. In practice, a
Convolutional Neural Network (CNN) feature extractor learns to produce
invariant features that are identical between the two domains.

We use a total loss:
$\mathcal{L}_{Total}=\mathcal{L}_{MSE}+\lambda_{DA}\mathcal{L}_{DA}$, where
$\mathcal{L}_{MSE}$ is the Mean Squared Error (MSE) loss used to minimize the
error on the task the model is trained to perform (estimate the Einstein
radius of a gravitational lens). $\mathcal{L}_{DA}$ is either the MMD loss or
the DANN loss, and it minimizes the variance of extracted features between
domains. This loss is weighted by constant $\lambda_{DA}$.

DANN [19] uses a two-headed network, with an adversarial approach to find
invariant features. All data is fed into a feature extractor made of
convolutional layers, which is then forwarded to the two heads, both made of
linearly connected layers. The first head is the label predictor, which takes
features and makes predictions of the labels associated with them. This head
is only trained with features extracted from labeled source data and minimizes
$\mathcal{L}_{MSE}$. The second head is the domain classifier, which takes
features from both domains and makes predictions as to which domain those
features came from. We use a Negative Log-Likelihood Loss function (NLL) loss
as $\mathcal{L}_{DA}$. With DANNs, the goal is to learn domain-invariant
features, which will maximize $\mathcal{L}_{DA}$ loss and confuse the domain
classifier. To simultaneously optimize the the feature extractor to find
features that will enable classification of labeled data in the label
predictor head, and confuse the domain classifier head (rendering it unable to
differentiate between the domains), a gradient reversal layer is used in
between the feature extractor and the domain classifier. This layer does not
affect the data in forward propagation, but multiplies the gradient by -1 in
back propagation, flipping the gradient.

MMD [21] is a distance-based domain adaptation method that identifies
invariant features by minimizing the distance between the source and target
domain distributions. We employ kernel methods to calculate MMD distance,
which we use as our DA loss:

$\mathcal{L}_{DA}=\frac{1}{N(N-1)}\sum_{i!=j}k\left(x_{\mathrm{s}}(i),x_{\mathrm{s}}(j)\right)-k\left(x_{\mathrm{s}}(i),x_{\mathrm{t}}(j)\right)-\\\
k\left(x_{\mathrm{t}}(i),x_{\mathrm{s}}(j)\right)+k\left(x_{\mathrm{t}}(i),x_{\mathrm{t}}(j)\right),$

where $k$ is a kernel between samples $x_{\mathrm{s}}$ from the source domain
and samples $x_{\mathrm{t}}$ from the target domain, and $N$ is the total
number of samples across both domains. We follow [10, 38] and use a
combination of multiple Gaussian Radial Basis Functions (RBF) kernels.

##  3 Data

We use the deeplenstronomy [24] package, which is built on lenstronomy [8], to
generate galaxy-scale strong lens datasets with $40\times 40$-pixel images in
a source domain and in a target domain. For the source domain, we generate
images with no pixel noise. For the target domain, we add noise based on
DES[1] observing conditions, which include sky brightness and seeing, drawn
from empirical distributions of these values [2]. Both domains contain
$50,000$ lenses with Einstein radius varying from 0.5 arc seconds to 3.0 arc
seconds drawn from a uniform distribution. We divide the dataset into
training, validation, and testing with a ratio of 70%:10%:20%. Figure 1
provides example lensing systems from the source and target domain datasets.

Figure 1: Examples of our source (top) and target (bottom) domain images. The
Einstein radius increases from left to right.

##  4 Neural Network Architecture

We employ a baseline CNN with three convolution layers (each layer followed by
ReLu activation, batch normalization, and max pooling) and two linear layers.
Details of the CNN architecture are in Table 2 in the appendix. MSE loss is
used for the main regression task, and DA is performed using a DANN or MMD
loss. For both the DANN and MMD variations, latent space features are gathered
after the final max pooling layer. For the DANN, we use a gradient reversal
layer followed by the same two-linear layer architecture repeated for the
domain classifier. A softmax activation is applied to the domain classifier’s
output and NLL is used as the domain classifier loss. We use the Adam
optimizer for both the DANN and the MMD. Optimal hyperparameters were found
through cross-validation. For all models, the batch size is fixed at 32. For
training without DA we used a learning rate of $10^{-5}$, while both DANN and
MMD used $6\times 10^{-5}$. For MMD, we used DA loss weight
$\lambda_{DA}=1.4$. The DA loss weight for DANN was set adaptively as proposed
in [19]. It is initiated at 0 and gradually reaches 1 over the course of
training, using the weighting function $\lambda_{DA}=\frac{2}{1+\exp{(-10\cdot
p)}}-1$, where $p$ is fraction of training process completed, from 0 to 1. For
all models, convergence was reached after 30 epochs.

To evaluate the performance of the CNN, we use the R-squared (R2) score [36].
It is defined as $R^{2}=1-\frac{SS_{Regression}}{SS_{Total}}$, where
${SS_{Regression}=\sum{(\hat{y_{i}}-y_{i})^{2}}}$ and
${SS_{Total}=\sum{(\bar{y}-y_{i})^{2}}}$. Here, $\hat{y_{i}}$ is the model’s
prediction, $y_{i}$ is the corresponding true value, and $\bar{y}$ is the mean
of all true values. This metric shows how well the model can predict a
dependent variable (Einstein radius). A larger R2 indicates a better fit of
the data, with a perfect fit of $R\textsuperscript{2}=1.0$.

##  5 Results

| Source Domain | Target Domain
---|---|---
No DA | $0.98\pm 0.0011$ | $0.50\pm 0.23$
DANN | $0.97\pm 0.0052$ | $0.94\pm 0.011$
MMD | $0.98\pm 0.013$ | $0.95\pm 0.018$
Table 1: R2 score for training without DA in the top row, with DANN in the
middle row, and with MMD in the bottom row. Each score is the mean of ten
identical networks, initiated with different random seeds and trained under
the same procedure. Standard deviation is also given across these models.

Figure 2: Predictions against actual values of test data sampled from the
source domain (left panels) and target domain (right panels). Below each plot
are the fractional residuals for visualization. Brighter color represents a
higher density of samples, found using a kernel-density estimate with Gaussian
kernels [34]. Results are given without DA (top row), with DANN (middle row),
and with MMD (bottom row).

We report the R2 scores in the source and target test sets in Table 1.
Furthermore, in Figure 2 we visualise predictions against actual values, as
well as fractional residuals, which are calculated by
$Residual_{fract}=(\Theta_{E,Predicted}-\Theta_{E,True})/\Theta_{E,Predicted}$.
The fractional residuals help visualize how each prediction is potentially
biased.

Figure 2 and the R2 scores in Table 1 show that both the DANN and MMD are able
to significantly improve model performance on a more complex target domain
when trained with a simpler source domain (up to $0.45$ improvement in target
$R^{2}$ scores, making both source and target scores close to $1$) . The
fractional residuals in Figure 2 support these results, with both DANN and MMD
attaining target domain fractional residuals substantially smaller in
magnitude than the model without DA. It should also be noted from the
residuals that both DANN and MMD struggle the most with lower values of
Einstein radius in the target domain. The model without DA and MMD similarly
struggle on the low end of Einstein radius in the source domain, while DANN
alone performs similarly across the whole source domain. With the inclusion of
DA, DANN suffers a small trade-off in accuracy on the source domain. MMD may
not lose accuracy on the source data, compared to regular training without DA,
but has a higher degree of uncertainty due to its increased standard
deviation.

##  6 Discussion

These results show that both of the tested DA-based methods have the potential
to facilitate the analysis of real survey data in the future – i.e., reducing
bias when using labeled simulated data for training and new unlabeled real
data as the test set.

Currently, the existing number of real lens images are likely not enough to
use as a target domain for DANN or MMD. However, as future large-scale surveys
are released, many more real lens images will become available, with the help
of automated identification using deep learning. Studies such as [16], [27],
and [29] have demonstrated the viability of these methods. At that point, DANN
and MMD may be used for further analysis.

It should be noted that if noise were to be the only difference between
simulated and real data, it may be viable to just simulate noise in training
data. However, other domain shifts are likely to exist between real and
simulated data, which DANN and MMD would be employed to mitigate. To this end,
a further investigation into domain shifts beyond DES-survey noise emulation
is needed to ensure the gap from simulated to real data can be closed. Both
DANN and MMD would likely struggle to overcome the domain shift caused by
differences in data distributions of the parameters the network is trying to
infer. For instance, if the source and target domain lenses are drawn from
different distribution of Einstein radii, the network may have difficulty
estimating this parameter correctly. This calls for investigation into other
domain adaptation techniques, such as instance-based techniques [17, 18],
which will be a topic of our future investigations.

## Acknowledgments and Disclosure of Funding

This manuscript has been authored by Fermi Research Alliance, LLC under
Contract No. DE-AC02-07CH11359 with the U.S. Department of Energy, Office of
Science, Office of High Energy Physics. The authors of this paper have
committed themselves to performing this work in an equitable, inclusive, and
just environment, and we hold ourselves accountable, believing that the best
science is contingent on a good research environment.

We acknowledge the Deep Skies Lab as a community of multi-domain experts and
collaborators who have facilitated an environment of open discussion, idea-
generation, and collaboration. This community was important for the
development of this project.

Furthermore, we also thank the anonymous referees whose comments helped
improve this manuscript.

## References

  * [1] T. M. C. Abbott, F. B. Abdalla, A. Alarcon, J. Aleksić, S. Allam, S. Allen, A. Amara, J. Annis, J. Asorey, S. Avila, and et. al.  Dark energy survey year 1 results: Cosmological constraints from galaxy clustering and weak lensing.  Phys. Rev. D, 98:043526, Aug 2018.
  * [2] T. M. C. Abbott, F. B. Abdalla, S. Allam, A. Amara, J. Annis, J. Asorey, S. Avila, O. Ballester, M. Banerji, W. Barkhouse, and et al.  The dark energy survey: Data release 1.  The Astrophysical Journal Supplement Series, 239(2):18, nov 2018\.
  * [3] Hiroaki Aihara, Nobuo Arimoto, Robert Armstrong, Sté phane Arnouts, Neta A Bahcall, Steven Bickerton, James Bosch, Kevin Bundy, Peter L Capak, James H H Chan, and et. al.  The hyper suprime-cam SSP survey: Overview and survey design.  Publications of the Astronomical Society of Japan, 70(SP1), sep 2017.
  * [4] Stephon Alexander, Sergei Gleyzer, Pranath Reddy, Marcos Tidball, and Michael W Toomey.  Domain adaptation for simulation-based dark matter searches using strong gravitational lensing.  arXiv preprint arXiv:2112.12121, 2021.
  * [5] MW Auger, T Treu, AS Bolton, R Gavazzi, LVE Koopmans, PJ Marshall, LA Moustakas, and S Burles.  The sloan lens acs survey. x. stellar, dynamical, and total mass correlations of massive early-type galaxies.  The Astrophysical Journal, 724(1):511, 2010.
  * [6] Camille Avestruz, Nan Li, Hanjue Zhu, Matthew Lightman, Thomas E Collett, and Wentao Luo.  Automated lensing learner: automated strong lensing identification with a computer vision technique.  The Astrophysical Journal, 877(1):58, 2019.
  * [7] Matteo Barnabe, Oliver Czoske, Léon VE Koopmans, Tommaso Treu, and Adam S Bolton.  Two-dimensional kinematics of slacs lenses–iii. mass structure and dynamics of early-type lens galaxies beyond z$\simeq$ 0.1.  Monthly Notices of the Royal Astronomical Society, 415(3):2215–2232, 2011.
  * [8] Simon Birrer and Adam Amara.  lenstronomy: Multi-purpose gravitational lens modelling software package.  Physics of the Dark Universe, 22:189–201, 2018.
  * [9] Simon Birrer, Suhail Dhawan, and Anowar J. Shajib.  The hubble constant from strongly lensed supernovae with standardizable magnifications.  The Astrophysical Journal, 924(1):2, jan 2022.
  * [10] A Ćiprijanović, Diana Kafkes, Kathryn Downey, Sudney Jenkins, Gabriel N Perdue, Sandeep Madireddy, Travis Johnston, Gregory F Snyder, and Brian Nord.  Deepmerge–ii. building robust deep learning algorithms for merging galaxy identification across domains.  Monthly Notices of the Royal Astronomical Society, 506(1):677–691, 2021.
  * [11] A. Ćiprijanović, A. Lewis, K. Pedro, S. Madireddy, B. Nord, G. N. Perdue, and S. M. Wild.  DeepAstroUDA: semi-supervised universal domain adaptation for cross-survey galaxy morphology classification and anomaly detection.  Machine Learning: Science and Technology, 4(2):025013, June 2023\.
  * [12] Aleksandra Ćiprijanović, Diana Kafkes, Gregory Snyder, F. Javier Sánchez, Gabriel Nathan Perdue, Kevin Pedro, Brian Nord, Sandeep Madireddy, and Stefan M. Wild.  DeepAdversaries: examining the robustness of deep learning models for galaxy morphology classification.  Machine Learning: Science and Technology, 3(3):035007, September 2022.
  * [13] Thomas E. Collett.  THE POPULATION OF GALAXY–GALAXY STRONG LENSES IN FORTHCOMING OPTICAL IMAGING SURVEYS.  The Astrophysical Journal, 811(1):20, sep 2015.
  * [14] Thomas E Collett and Matthew W Auger.  Cosmological constraints from the double source plane lens sdssj0946+ 1006\.  Monthly Notices of the Royal Astronomical Society, 443(2):969–976, 2014.
  * [15] Gabriela Csurka.  Domain Adaptation for Visual Applications: A Comprehensive Survey.  arXiv e-prints, page arXiv:1702.05374, February 2017.
  * [16] Andrew Davies, Stephen Serjeant, and Jane M Bromley.  Using convolutional neural networks to identify gravitational lenses in astronomical images.  Monthly Notices of the Royal Astronomical Society, 487(4):5263–5271, May 2019.
  * [17] Antoine de Mathelin, Guillaume Richard, François Deheeger, Mathilde Mougeot, and Nicolas Vayatis.  Adversarial weighting for domain adaptation in regression.  In 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI), pages 49–56. IEEE, 2021.
  * [18] Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia.  A brief review of domain adaptation.  Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020, pages 877–894, 2021.
  * [19] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The journal of machine learning research, 17(1):2096–2030, 2016\.
  * [20] Sankalp Gilda, Antoine de Mathelin, Sabine Bellstedt, and Guillaume Richard.  Unsupervised Domain Adaptation for Constraining Star Formation Histories.  arXiv e-prints, page arXiv:2112.14072, December 2021.
  * [21] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola.  A kernel two-sample test.  Journal of Machine Learning Research, 13(25):723–773, 2012.
  * [22] Željko Ivezić, Steven M Kahn, J Anthony Tyson, Bob Abel, Emily Acosta, Robyn Allsman, David Alonso, Yusra AlSayyad, Scott F Anderson, John Andrew, et al.  Lsst: from science drivers to reference design and anticipated data products.  The Astrophysical Journal, 873(2):111, 2019.
  * [23] Qiufan Lin, Dominique Fouchez, and Jerome Pasquet.  Galaxy image translation with semi-supervised noise-reconstructed generative adversarial networks.  In 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, January 2021.
  * [24] Robert Morgan, Brian Nord, Simon Birrer, Joshua Yao-Yu Lin, and Jason Poh.  deeplenstronomy: A dataset simulation package for strong gravitational lensing.  arXiv preprint arXiv:2102.02830, 2021.
  * [25] Teaghan O’Briain, Yuan-Sen Ting, Sébastien Fabbro, Kwang M. Yi, Kim Venn, and Spencer Bialek.  Cycle-starnet: Bridging the gap between theory and data by leveraging large data sets.  The Astrophysical Journal, 906(2):130, January 2021.
  * [26] Will Percival, Michael Balogh, Dick Bond, Jo Bovy, Raymond Carlberg, Scott Chapman, Patrick Cote, Nicolas Cowan, Sebastien Fabbro, Laura Ferrarese, and et al.  The Euclid Mission.  In Canadian Long Range Plan for Astronomy and Astrophysics White Papers, volume 2020, page 20, October 2019.
  * [27] C E Petrillo, C Tortora, S Chatterjee, G Vernardos, L V E Koopmans, G Verdoes Kleijn, N R Napolitano, G Covone, L S Kelvin, and A M Hopkins.  Testing convolutional neural networks for finding strong gravitational lenses in kids.  Monthly Notices of the Royal Astronomical Society, October 2018\.
  * [28] S Rezaei, J P McKean, M Biehl, W de Roo, and A Lafontaine.  A machine learning based approach to gravitational lens identification with the international LOFAR telescope.  Monthly Notices of the Royal Astronomical Society, 517(1):1156–1170, jul 2022.
  * [29] C Schaefer, M Geiger, T Kuntzer, and JP Kneib.  Deep convolutional neural networks as strong gravitational lens detectors.  arXiv preprint arXiv:1705.07132, 2017.
  * [30] Manuel Schwonberg, Joshua Niemeijer, Jan-Aike Termöhlen, Jörg P. Schäfer, Nico M. Schmidt, Hanno Gottschalk, and Tim Fingscheidt.  Survey on Unsupervised Domain Adaptation for Semantic Segmentation for Visual Perception in Automated Driving.  arXiv e-prints, page arXiv:2304.11928, April 2023.
  * [31] Mauro Sereno.  Probing the dark energy with strong lensing by clusters of galaxies.  arXiv preprint astro-ph/0209210, 2002.
  * [32] SH Suyu, PJ Marshall, MW Auger, S Hilbert, RD Blandford, LVE Koopmans, CD Fassnacht, and T Treu.  Dissecting the gravitational lens b1608+ 656. ii. precision measurements of the hubble constant, spatial curvature, and the dark energy equation of state.  The Astrophysical Journal, 711(1):201, 2010.
  * [33] Ricardo Vilalta, Kinjal Dhar Gupta, Dainis Boumber, and Mikhail M. Meskhi.  A general approach to domain adaptation with applications in astronomy.  Publications of the Astronomical Society of the Pacific, 131(1004):108008, sep 2019.
  * [34] Stanisław Węglarczyk.  Kernel density estimation and its application.  In ITM web of conferences, volume 23, page 00037. EDP Sciences, 2018\.
  * [35] Joshua Wilde, Stephen Serjeant, Jane M Bromley, Hugh Dickinson, Lé on V E Koopmans, and R Benton Metcalf.  Detecting gravitational lenses using machine learning: exploring interpretability and sensitivity to rare lensing configurations.  Monthly Notices of the Royal Astronomical Society, 512(3):3464–3479, feb 2022.
  * [36] S Wright.  Correlation and causation.  Journal of agricultural research, 20(7):557, 1921.
  * [37] E Zaborowski, A Drlica-Wagner, F Ashmead, JF Wu, R Morgan, CR Bom, AJ Shajib, S Birrer, W Cerny, L Buckley-Geer, et al.  Identification of galaxy-galaxy strong lens candidates in the decam local volume exploration survey using machine learning.  arXiv preprint arXiv:2210.10802, 2022.
  * [38] Yinghua Zhang, Yu Zhang, Ying Wei, Kun Bai, Yangqiu Song, and Qiang Yang.  Fisher deep domain adaptation, 2020.

## Appendix

Layers | Properties | Output Shape | Parameters
---|---|---|---
input | 1*40*40 | (1, 40, 40) | 0
Convolution (2D) |  | Filters: 8
---
Kernel: 3 3
Activation: ReLu
(8, 42, 42) | 80
Batch Normalization | - | (8, 42, 42) | 16
MaxPooling |  | Kernel: 2 2
---
Stride: 2
(8, 21, 21) | 0
Convolution (2D) |  | Filters: 16
---
Kernel: 3 3
Activation: ReLu
(16, 21, 21) | 1,168
Batch Normalization | - | (16, 21, 21) | 32
MaxPooling |  | Kernel: 2 2
---
Stride: 2
(16, 10, 10) | 0
Convolution (2D) |  | Filters: 32
---
Kernel: 3 3
Activation: ReLu
(32, 10, 10) | 4,640
Batch Normalization | - | (32, 10, 10) | 64
MaxPooling |  | Kernel: 2 2
---
Stride: 2
(32, 5, 5) | 0
Flatten | - | (800) | -
Fully Connected | Activation: ReLu | (128) | 102,528
Fully Connected | Activation: ReLu | (1) | 129
Table 2: CNN architecture