# Single-ended Recovery of Optical Fiber Transmission Matrices using Neural
Networks

Yijie Zheng1,Terry Wright1,Wen Zhong2,3,Qing Yang2,3,George S. D. Gordon1*
1Optics and Photonics Research group, University of Nottingham, UK
2State Key Laboratory of Extreme Photonics and Instrumentation, College of
Optical Science and Engineering
International Research Center for Advanced Photonics, Zhejiang University,
Hangzhou, China
3Research Center for Humanoid Sensing, Zhejiang Lab, Hangzhou, China
*Corresponding Author<EMAIL_ADDRESS>0000-0002-6513-1584 0000-0002-7333-5106

###### Abstract

Ultra-thin multimode optical fiber imaging promises next-generation medical
endoscopes reaching high image resolution for deep tissues. However, current
technology suffers from severe optical distortion, as the fiber’s calibration
is sensitive to bending and temperature and thus requires in vivo re-
measurement with access to a single end only. We present a neural network
(NN)-based approach to reconstruct the fiber’s transmission matrix (TM) based
on multi-wavelength reflection-mode measurements. We train two different NN
architectures via a custom loss function insensitive to global phase-
degeneracy: a fully connected NN and convolutional U-Net. We reconstruct the
64 $\times$ 64 complex-valued fiber TMs through a simulated single-ended
optical fiber with $\leq$ 4% error and cross-validate on experimentally
measured TMs, demonstrating both wide-field and confocal scanning image
reconstruction with small error. Our TM recovery approach is 4500 times
faster, is more robust to fiber perturbation during characterization, and
operates with non-square TMs.

_K_ eywords Optical fiber imaging $\cdot$ Transmission matrix reconstruction
$\cdot$ Custom loss function $\cdot$ Neural network

##  1 INTRODUCTION

Ultra-thin endoscopes are a promising technique for enabling cell-scale
imaging in difficult-to-reach parts of the body, with the potential to improve
disease detection in organs such as the pancreas and ovaries. Commercial
products using imaging fiber bundles around 1mm in diameter are used in bile
ducts [1] and flexible and full-color imaging has been demonstrated using
distal scanning mechanisms that are typically around 2mm in diameter [2, 3,
4]. To further reduce the size of endoscopes, recent work has focused on
imaging through ultra-thin multimode fibers with diameters of 0.125mm and has
achieved _in vivo_ fluorescence imaging in brains of immobilized mice [5].
However, there are some key limitations of these imaging systems that use
ultra-thin optical fiber. First, the thinnest such imaging devices are made
using multimode fiber (MMF), which suffers from optical distortion that
changes whenever the fiber is perturbed, particularly for longer fibers
($>$1m) required to reach deep inside the human body [6]. Second, to calibrate
this distortion, practical endoscopes made using MMF or some types of fiber
bundle require measurement of their transmission matrix (TM) which requires
transmittg a set of well-defined modes of light from the proximal facet to the
distal facet where the resulting optical field must be measured. If
calibration is required immediately before use (e.g for _in vivo_ use), such
components would be required on the distal tip and would thus compromise the
ultra-thin form factor [7].

A number of methods have been proposed to calibrate fiber TMs without distal
access including guidestars [8, 9, 10], beacons that can be tracked [11, 12],
or reflective structures on the fiber tips [7, 13, 14]. Gordon et al. [14]
proposed a single-ended method of TM recovery based on the fiber system shown
in Figure 1, with a specially designed reflector stack that provides different
reflectances at different wavelengths. This approach avoids the need for
measurement at both proximal and distal end of the fiber and works for non-
unitary TMs. The reflection matrix,
$\mathbf{C_{\lambda}}\in\mathbb{C}^{M^{2}\times M^{2}}$, where images are
assumed to be $M\times M$ pixels, describes how an incident field
$\mathbf{E_{in}}\in\mathbb{C}^{M^{2}}$ is transformed via propagation through
the optical fiber, reflected by the reflector stack and finally transferred
back through the fiber into an output field
$\mathbf{E_{out}}\in\mathbb{C}^{M^{2}}$ at a wavelength of $\lambda$:

$\mathbf{C_{\lambda}}\mathbf{E_{in\lambda}}=\mathbf{E_{out\lambda}}$ (1)

The reflectance matrix, $\mathbf{C_{\lambda}}\in\mathbb{C}^{M^{2}\times
M^{2}}$, is obtained from a forward pass via the TM,
$\mathbf{A_{\lambda}}\in\mathbb{C}^{M^{2}\times M^{2}}$, reflection via the
reflector stack, $\mathbf{R_{\lambda}}\in\mathbb{C}^{M^{2}\times M^{2}}$, and
then a return pass via the TM,
$\mathbf{A_{\lambda}}^{\top}\in\mathbb{C}^{M^{2}\times M^{2}}$. It has been
previously been shown that the forward TM at wavelength $\lambda$,
,$\mathbf{A_{\lambda}}$, can be unambiguously reconstructed based on the
measured reflection matrices at 3 different wavelengths, with imaging
performed at a fourth wavelength [14]. This requires design of a special
reflector stack whose reflection matrix changes with wavelength at a
significantly faster rate than the fire TM, which may be achieved by
alternately stacking metasurface reflectors with long-pass filters. A first-
order dispersion model is assumed for the TM and has been experimentally
validated to be $>70\%$ accurate over a 5nm bandwidth for a 1-2m length MMF
using 110 modes, avoiding degeneracies arising from matrix logarithms [14],
although recent work has shown MMF dispersion modeling over a much greater
bandwidth [15]. This leads to a set of 3 non-linear quadratic-form equations:

$\mathbf{C_{\lambda_{1}}}=\mathbf{{A}_{\lambda_{1}}^{T}}\mathbf{R_{\lambda_{1}}}\mathbf{A_{\lambda_{1}}}$
(2)
$\mathbf{C_{\lambda_{2}}}={(e^{(\log\mathbf{{A_{\lambda_{1}}}}\frac{\lambda_{1}}{\lambda_{2}})})}^{T}\mathbf{R_{\lambda_{2}}}\mathbf{(}e^{(\log\mathbf{{A_{\lambda_{1}}}}\frac{\lambda_{1}}{\lambda_{2}})})$
(3)
$\mathbf{C_{\lambda_{3}}}={(e^{(\log\mathbf{{A_{\lambda_{1}}}}\frac{\lambda_{1}}{\lambda_{3}})})}^{T}\mathbf{R_{\lambda_{3}}}\mathbf{(}e^{(\log\mathbf{{A_{\lambda_{1}}}}\frac{\lambda_{1}}{\lambda_{3}})})$
(4)

where
$\mathbf{e^{(\log\mathbf{{A_{\lambda_{1}}}}\frac{\lambda_{1}}{\lambda_{2}})}}$
is the TM adjusted for a wavelength $\lambda_{2}$.

Currently, these equations are solved by using an iterative approach which
relies on optimization of the entire TM [14]. This therefore scales in
complexity with the square of the matrix dimension, incurring significant
computational time, especially for large matrices. In practice, the TM shows
high sensitivity to bending and temperature so in a practical usage scenario
would need to be measured very frequently and reconstructed immediately prior
to imaging. Large computational times are therefore not practical.

Considering this, there are several methods that have been developed in order
to reduce the computational time for fiber imaging. These methods typically
exploit prior knowledge about the fibers to improve or speed up TM
reconstruction. For example, Li et al. [16] proposed a compressed sampling
method based on the optical TM to reconstruct full-size TM of a multimode
fiber supporting 754 modes at compression ratios down to 5% with good
fidelity. Similarly, look-up tables can be used with a reflective beacon for
high-speed TM estimation over an experimentally sampled prior space [12].
Huang et al. [17] retrieved the optical TM of a multimode fiber using the
extended Kalman filter, enabling faster reconstruction.

Recently, there has been work on using deep learning approaches, involving
convolutional neural networks, to reconstruct images via multimode fibers both
in transmission and reflection modes [18, 19, 20]. These methods have the
advantage of being fast, and also learning and utilising prior information
about the fiber properties and the objects being imaged. However, their
performance typically degrades significantly under fiber perturbation because
they do not have access to reflection calibration measurements required to
unambiguously resolve a TM. Further, because such approaches seek to
approximate the forward propagation of light and often only consider amplitude
image recovery, they often rely on classical mean-squared error loss functions
for training.

In order to incorporate reflection calibration measurements following fiber
perturbation, it may instead be advantageous to use AI approaches to
reconstruct a TM rather than an image, though there has been relatively little
work in this area. When reconstructing a TM comprising complex numbers, a
particular type of degeneracy arises that is not well handled by conventional
AI loss functions: a global phase factor. In many physical problems, including
the recovery of TMs for the purposes of image reconstruction and phase-
hologram generation, global phase factors are not relevant as they do not
affect the perceived performance of the system: it is the _relative_ phase
between pixels that must be preserved. The relative phase between rows or
columns of fiber TMs or reflection matrices is typically preserved using a
reference beam e.g. interferometry [21] or referencing to a fiber mode [22].
Global phase may have a physical interpretation related to the physical length
of the fiber, but in practice it is often arbitrary unless great care is
taken. For example in interferometric systems the global phase is likely to be
arbitrary unless the optical path lengths of the reference and sample arms are
perfectly matched, which is very challenging for multimode fibers. Further,
the global phase often drifts during practical experiments [21], and
approaches using phase-retrieval produce entirely arbitrary global phase
values [22]. Therefore, in many practical situations, conventional loss
functions will convert arbitrary shifts in the global phase to large changes
in loss, which can confound minimization algorithms used to fit AI models and
cause overfitting.

In this paper, we therefore present a method of implementing single-ended
recovery of an optical fiber TM by solving Eqn. 2– Eqn. 4 based on three
reflection matrix measurements at three different wavelengths. Specifically,
we present two different neural network architectures, fully connected neural
network (FCNN) and convolutional U-net based neural networks, and demonstrate
the performance of both. As a necessary step, we account for the global phase
factor of the entire TM by developing a custom global phase insensitive loss
function that avoids degeneracies introduced by conventional loss functions
such as mean-squared error (MSE). We first validate our model by recovering
$64\times 64$ complex-valued fiber TMs through a simulated single-ended
optical fiber system (shown in Figure 1) with $\leq 4\%$ error for both FCNN
and convolutional U-net architectures. We then demonstrate image
reconstruction through fiber based on recovered TMs for two different imaging
modalities: widefield imaging, achieving $\leq 9\%$ error, and confocal
imaging, achieving $\leq 5\%$ error. We highlight several advantages of this
TM recovery approach compared to previous TM recovery methods. Firstly, once
the model is trained ($\sim$100 hours), it only requires $\sim$1 second for
reconstruction, which is 4500 times faster than pre-existing iterative
approaches. Secondly, the conventional method [14] can only reconstruct square
TM cases, whereas this method is compatible with non-square-shaped TM with
$\leq 8\%$ error, useful for many practical cases where optical systems may
have different mode bases at proximal and distal ends. Third, the requirements
for the reflectors in terms of matrix properties are related and then can have
arbitrary distributions of eigenvalues.

Figure 1: Single-ended optical fiber imaging system for transmission matrix
(TM) recovery. The object, which will produce an image
$\mathbf{X}\in\mathbb{C}^{M\times M}$ is placed at the distal facet. Light
with a field of $\mathbf{E_{in}}\in\mathbb{C}^{M^{2}}$ propagates from the
proximal facet through the optical fiber, with the forward TM of the optical
fiber defined as $\mathbf{A_{\lambda}}\in\mathbb{C}^{M^{2}\times M^{2}}$ at
the wavelength, $\lambda$. A reflector stack with a three-layer structure is
placed at the distal facet, producing a reflector matrix
$\mathbf{R_{\lambda}}\in\mathbb{C}^{M^{2}\times M^{2}}$ at the wavelength
$\lambda$. There are four different wavelengths used, where wavelengths
$\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are used for characterization
and $\lambda_{4}$ for imaging. For each wavelength, light propagates through
one or more layers of the stack and is partially reflected by metasurfaces at
the interfaces between layers. This produces distinct reflector matrices at
each wavelength, $R_{\lambda_{1..3}}$, which can be addressed using a tuneable
laser. At wavelength $\lambda_{4}$ light passes fully through the stack so
that imaging can be performed. Faithful image reconstruction require
correction for the TM of the reflector stack using pre-calibrated values at
this wavelength [14]. Reflection matrices,
$\mathbf{C_{\lambda}}\in\mathbb{C}^{M^{2}\times M^{2}}$, can be measured at
relevant wavelengths to recover the TM.

##  2 RESULTS

###  2.1 Simulated TM recovery

This TM recovery model was trained on a simulated dataset comprising 900,000
sets of simulated reflection matrices, $\mathbf{C}_{\lambda}$ at 3
wavelengths, $\lambda_{1}=850$nm, $\lambda_{2}=852$nm and $\lambda_{3}=854$nm,
as input and a complex-valued non-unitary TM at wavelength $\lambda_{1}$,
$\mathbf{A}_{\lambda_{1}}$ as output. Performance was monitored during
training using 200,000 validation data that were not used in training. Final
performance figures for the model were evaluated using an additional 100,000
data that were not part of the validation set. Figure 2(a) shows the training
and validation loss in training the FCNN model over 2500 epochs using
different loss functions, namely conventional mean absolute error (MAE), and
our global-phase insensitive custom loss function (Eqn.11). Our global-phase
insensitive loss functions show a decreasing loss in both training (yellow
line) and validation (purple line) in the first 2000 epochs and convergence
after 2500 epochs, whereas the MAE loss function exhibits fluctuating non-
converging loss values for both training (blue line) and validation (red
line). An example of a reconstructed TM predicted by the FCNN model at
different epochs is shown inset in Figure 2(a). It can be seen that the
predicted TM approaches the target TM from 300 epochs to 2500 epochs, enabled
by the custom loss function.

Figure 2 (b) compares the TM result predicted by our two different neural
network architectures using different loss functions. Both FCNN and
convolutional U-net-based neural networks cannot recover TM when using the MAE
loss function but are capable of recovering TM using the global phase
insensitive loss function, with an average loss (metric defined in Eqn. 8) of
$\leq 4\%$ over 100,000 test TMs. Compared to the previous iterative approach,
which requires nearly 10 hours to converge with an average loss of $\leq
1.5\%$, our method shows a larger $\leq 4\%$ average loss, but images retain
acceptable visual quality of reconstructed images (shown in Result Section
Widefield image reconstruction based on recovered TM). Furthermore, we also
evaluated the computational resource usage of the two different neural network
architectures as shown in Figure2 (c). Training was implemented using
Tensorflow 2.0 running on an NVIDIA Tesla V100 GPU. Compared to FCNN, the
convolutional U-net shows significant advantages in memory usage, requiring
1000 times fewer trainable parameters, and in convergence time, which is
reduced by $20\%$. However, it shows $0.7\%$ larger average loss on the test
set. Both FCNN and convolutional U-net can recover TM at a loss $\leq 4\%$
(standard deviation 0.44% and 0.56% respectively), with $\sim$1s prediction
time.

Figure 2: Simulated TMs recovered by training Neural Network (NN) models. (a)
Training and validation loss tracking plot using MAE and custom loss function
(Eqn. 11) of fully-connected neural network (FCNN) model. Blue and red line
represent training and validation loss using mean absolute error (MAE) loss
function. Yellow and purple line represent training and validation loss using
our global-phase insensitve loss function. TM recovery results over different
epochs are shown inset. (b) TM recovery results using two different neural
network architectures (i.e. FCNN and convolutional U-net networks), with two
different loss functions, namely MAE and custom loss function in Eqn. 11. (c)
Comparison between FCNN and convolutional U-net architecture in aspects of
average loss, standard deviation, training time, prediction time, number of
converging epochs and number of trainable parameters. In complex-valued
colormap, hue represents phase and lightness/darkness represents amplitude.

###  2.2 Widefield image reconstruction based on recovered TM

Next, we examine a widefield imaging modality to evaluate the performance of
recovered TMs at a fourth wavelength (the imaging wavelength),
$\lambda_{4}=854.5$nm, where we assume a pixel basis at the fiber output. We
considered 3 example images denoted $\mathbf{x}\in\mathbb{C}^{8\times 8}$: an
amplitude-only image with a ‘space invader’ pattern, a phase-only digit with a
uniform amplitude and a random complex-valued image. Random noise was added to
better simulate expected behaviour in a real system (shown in Eqn. 7). Figure
3 shows the image reconstruction results based on recovered TM using FCNN and
convolutional U-net networks. It can be seen that all three images can be
successfully reconstructed based on recovered TMs using both neural network
models. Using the IMMAE (image MAE metric defined in Eqn. 9) we achieved error
$\leq 9\%$ and using SSIM (structural similarity index measure metric defined
in Eqn. 10 we achieve similarity $\geq 83\%$. Because of the superior error
performance of the FCNN approach, for the remainder of this work we choose
this model to demonstrate proof-of-principle performance of neural network TM
recovery under a range of realistic scenarios. However, when scaling to larger
size TMs, the convolutional U-net may be more favourable due to fewer
trainable parameters.

Figure 3: Widefield image reconstruction based on recovered transmission
matrices (TMs) using fully-connected neural network (FCNN) and convolutional
U-net networks. We considered 3 example images: an amplitude-only image with a
‘space invader’ pattern, a phase-only digit with a uniform amplitude and a
random complex-valued image. Two metrics are defined to evaluate the image
reconstruction performance, namely image mean absolute error, IMMAE (Eqn. 9)
and structural similarity index measure, SSIM (Eqn. 10).In complex-valued
colormap, hue represents phase and lightness/darkness represents amplitude.

###  2.3 Effect of fiber perturbation widefield image reconstruction

We then evaluate the robustness of our TM recovery model by simulating the
effect of the TM changing mid-way through fiber characterisation.
Specifically, we simulate a situation in which the reflection matrices
measured at the first two wavelengths ($\mathbf{C_{1}}$ and $\mathbf{C_{2}}$)
are recorded using a fixed TM, but the reflection matrix measured at the third
wavelength $\mathbf{C_{3}}$ is recorded initially using this same TM but the
final few columns are generated using a different TM. Therefore, we simulated
10 sets of $64\times 64$ reflection matrices with five different perturbation
rates indicating the numbers of columns swapped (2/64, 4/64, 8/64, 16/64, and
32/64). Figure 4 shows the results of recovering the TM based on these
perturbed measurements, and then using this perturbed TM estimate to perform
widefield image reconstructions. This is repeated for different fiber
perturbation rates based on our pre-trained TM recovery FCNN model. It can be
seen that our TM recovery model is compatible with optical fibers with a small
perturbation rate (below 6%) with TM average loss $\leq 8\%$ (standard
deviation $\leq 0.82\%$), IMMAE $\leq 19\%$ and SSIM $\geq 76\%$ but
performance degrades above this.

Figure 4: Effect of simulated perturbations of fiber transmission matrices
(TMs) during reflection-mode characterization. Perturbations are simulated by
swapping columns between two different TMs at wavelength $\lambda_{3}$. Two
metrics are defined to evaluate the image reconstruction performance based on
recovered TM using fully-connected neural network (FCNN), namely image mean
absolute error, IMMAE (Eqn. 9) and structural similarity index measure, SSIM
(Eqn. 10). a) Estimated perturbed TM. b) Error in estimated TM. c) Image
reconstruction of amplitude-only target (uniform phase) using estimated
perturbed TM. d) Image reconstruction of phase-only target (uniform amplitude)
using estimated perturbed TM. e) Image reconstruction of complex amplitude and
phase target using estimated perturbed TM.In complex-valued colormap, hue
represents phase and lightness/darkness represents amplitude.

###  2.4 Confocal image reconstruction based on recovered TM

Many fiber imaging systems use a confocal imaging approach, rather than
widefield, because of its superior resolution. We therefore simulated confocal
scanning imaging reconstruction through our recovered TM. For this, we used a
simulated LP basis based on a typical fiber profile (see Methods). We
addressed $128\times 128$ spot positions and found the average percentage of
power in the focus to be 48.5% using recovered TM, which compares favourably
to that achieved using an identity matrix as the TM (49.9%) as shown in Figure
5(a). Next, we reconstructed confocal images of three target images, denoted
$\mathbf{x}\in\mathbb{C}^{128\times 128}$ that contain the same pattern shown
in Result Section Widefield image reconstruction based on recovered TM by
integrating total reflected power for each spot position. We used
$\lambda_{4}=845.5$nm here for imaging. Figure 5(b)-(d) shows the confocal
image results using FCNN model. It can be seen that all three types of
confocal images can be successfully reconstructed based on recovered TMs, with
IMMAE $\leq 5\%$ and SSIM $\geq 90\%$. Also, it shows $3\%$ points less image
error and $4\%$ points higher similarity between reconstructed and target
confocal images compared to widefield imaging.

Figure 5: Spot scanning and confocal image reconstruction based on recovered
transmission matrices (TMs) using fully-connected neural network (FCNN) model.
The spot and 2D full samples are generated using $128\times 128$ targets.
Power ratio ($pr$) is defined to calculate the average percentage of power in
the focus by scanning $128\times 128$ approximate spot positions. Two metrics
are defined to evaluate the image reconstruction performance, namely image
mean absolute error, IMMAE (Eqn. 9) and structural similarity index measure,
SSIM (Eqn. 10). a) Creation of a focussed spot in perfect scenario and band-
limited scenario with 64 modes using both actual (target) and recovered TMs.
b) Image reconstruction using confocal scanning for amplitude-only image, c)
Phase-only image and d) Random complex valued image.In complex-valued
colormap, hue represents phase and lightness/darkness represents amplitude.

###  2.5 Non-square TM recovery

We next examine the practical case of non-square TMs, e.g. where the desired
representation at the distal end of a fiber might be different from that used
at the proximal end and may have more elements. This might be the case, for
example, when measuring forward TMs to use to train such a network. To recover
a TM $\mathbf{A}\in\mathbb{C}^{M_{d}\times M_{p}}$, we require that the
reflection matrix, $\mathbf{C}\in\mathbb{C}^{M_{p}\times M_{p}}$ and that the
reflector matrix, $\mathbf{R}\in\mathbb{C}^{M_{d}\times M_{d}}$. ${M_{p}}$ and
${M_{d}}$ represent the number of elements used for the basis representation
at the proximal and distal ends of the fiber respectively. Figure 6 shows one
example of recovered non-square-shaped TM $\in\mathbb{C}^{6\times 12}$ using
FCNN, with the average loss of 3.96% (standard deviation 0.38%). ‘Wide’ TMs
(with $M_{d}<M_{p}$) may be over-constrained by larger dimension $\mathbf{C}$
producing stable solutions, while ‘tall’ TMs (with $M_{p}<M_{d}$) may be
underconstrained producing degenerate solutions.

Figure 6: Non-square shaped transmission matrix (TM) $\in\mathbb{C}^{6\times
12}$ recovered by our TM recovery model using fully-connected neural network
(FCNN) architecture. a) TM recovered by FCNN model. b) Target TM. c) Error
between recovered and target TMs. In complex-valued colormap, hue represents
phase and lightness/darkness represents amplitude.

###  2.6 Reflector matrix conditioning

Explicit conditions on the reflector matrices, $\mathbf{R}$, namely having
distinct eigenvalues, have been a requirement in previous approaches [14, 7].
Here, we test our TM recovery FCNN model with different reflector matrix
conditioning by varying the number of distinct eigenvalues, as shown in
Supplementary Figure 1. We examined the recovered TMs for 3 conditionings of
$\mathbf{R}$: 1 distinct eigenvalue (unitary, all eigenvalues the same), 2
distinct eigenvalues and 6 distinct eigenvalues. The average recovery loss
achieved is 5.02% (standard deviation 0.43%), 4.96% (standard deviation
0.41%), 3.88% (standard deviation 0.37%) respectively, demonstrating that the
recovery process is compatible with various reflector matrices conditions.

###  2.7 Computational resource usage

As the dimension of recovered images increases, we expect an increase in the
TM dimension thus requiring more computational resources. Empirically measured
computational resources are plotted in log-scale in Figure 7 (a)-(c): minimum
training data, minimum memory usage, and converging time respectively. All
indicate a quadratic relationship to the image dimension $M$ for both FCNN and
convolutional U-net models. For practical imaging applications we would desire
at least $32\times 32$ image resolution, giving a $1024\times 1024$ TM, which
would require training with $>$10 million examples, leading to memory
consumption $>$1.5TB for the FCNN. By comparison, the convolutional U-net
would require only 1.1TB of memory consumption. Compared to FCNN,
convolutional U-net shows potential advantages in using 25% fewer memory
resources and 20% less training data within 15% less training time. Figure
7(d) compares the prediction time using our neural network model with the
conventional methods using iterative optimization approaches [14], where our
FCNN model shows less reconstruction time ($\sim$1s vs. 1920s for a 12
$\times$ 12 TM).

Figure 7: Empirically measured computational resources plotted in log-scale.
(a) Minimum training data versus the number of image dimensions, plotting in
log-scale.(b) Minimum memory usage versus the number of image dimensions,
plotting in log-scale. (c) Converging time versus the number of image
dimensions. (d) Prediction time of using our transmission matrix (TM) recovery
model and conventional method, plotting in log-scale.

###  2.8 Validation with experimental TMs

Finally, to demonstrate applicability to practical systems we cross validated
our model on experimentally measured TMs recorded under a representative range
of likely fibre conformations [12]. The graph of loss vs. epoch for the
different training steps used to adapt the model pre-trained on simulated data
is shown in Supplementary Figure 2. The model is first re-trained from step 1
with additional inputs using random matrices to avoid overfitting to the prior
distribution of simulated matrices (Step 2). We next continued training the
model on a small subset of the experimental data to aid domain transferability
(Step 3). By creating ‘submatrices’ we are effectively implementing a
downsampling basis change and so expect some change in correlations betweens
TM elements. However, by including random matrices in our training set we
actively prevent the model from learning a strong prior distribution over
these correlations and thus overfitting.

The final model achieves recovery of simulated TMs with an average loss of
3.38% (standard deviation 0.48%) and experimental matrices with 3.42%
(standard deviation 0.57%), suggesting that it is applicable to realistic
fibre TMs under various conformations. An example of experimental TM used for
the test at each step is shown inset in Supplementary Figure 2, where the
predicted TM is getting closer to the target TM from Step 1 to Step 3.

##  3 DISCUSSION

We have demonstrated the successful reconstruction of forward fiber TMs based
on reflection-mode measurements at multiple wavelengths using a neural
network-based approach encompassing two architectures: a fully-connected
neural network and a convolutional U-Net. Previous work applying neural
networks to fibers has focussed on image reconstruction as the end goal, but
we instead focus on TM reconstruction. Such an approach is more flexible as
the inputs to the network are calibration measurements that reflect a fiber’s
deformation state at any given time – previous image reconstruction approaches
have instead learned a static representation of the fiber TM encoded in the
neural network weights. Using our approach, the recovered TM will be accurate
up to the most recent calibration measurements and can be used for high-speed
image recovery via conventional matrix operations.

One major challenge of recovering the TM is the presence of degenerate global
phase shift. Previous work on image reconstruction has addressed this problem
by training separate networks for amplitude and phase recovery in purely real
space and accepting relatively poor performance for phase recovery [18]. Here,
we present a loss function that is insensitive to this global phase degeneracy
and show a high degree of convergence compared to conventional MAE metrics. We
believe this metric could also find applications in neural network based
computer-generated holography or phase retrieval.

There are several major advantages to our neural network approach compared to
previous iterative approaches [14]. First, the prediction time is very fast
(typically $\sim$1s), over 4500x faster than the existing iterative approach,
which makes this a feasible approach for future real-time imaging, Training
the network is much slower, but this only need to be done once per fiber for a
fixed reflector as a one-off calibration step. Second, our approach shows
robustness to the fiber TM changing part way through characterization
measurements, as is likely to happen during real _in vivo_ usage, and can
tolerate up to 6% column swaps between different reflection matrices. This
performance could be further improved by re-training the network with
perturbed examples as input, allowing it to learn an ‘error correction’
strategy. Third, the approach can reconstruct non-square TMs. This is
important because the sampling basis of the light on the proximal facet is
often the pixel basis of the camera but at the distal end it may be a mode
basis of the fiber (e.g. LP modes). The fibre may support many fewer modes
than the camera has pixels. Therefore, to optimize speed and imaging
performance it is often desirable to retrieve a TM in the mode basis of the
fiber that can easily be addressed using our camera coordinates: hence a non-
square TM. Finally, our recovery process is compatible with various reflector
matrices conditions whereas in previous models the eigenvalues are required to
be distinct [14, 7]. However, it is noted that eigenvalues are unlikely to be
identical in any experimental system due to small mode-dependent power losses,
but the removal of this condition may increase robustness to noise.

Several trade-offs should be considered when using this approach. The first
trade-off is the need for large amounts of experimental transmission and
reflection measurements to train the network. This can be addressed by
augmenting experimental data with large amount of simulated fiber data: we
have previously found good agreement between simulated and experimental
matrices [14]. To implement these simulations, the reflector matrices would
need to be known in advance e.g. by performing TM measurements of a fiber,
attaching a reflector, then measuring reflection matrix with minimal fiber
perturbation. Alternatively, it may be possible to devise a method of reliably
manufacturing reflectors with consistent and highly reproducible properties.
Our simulations assume limited fiber bandwidth to avoid degeneracies arising
from matrix logarithms, but this bandwidth could be extended using a nonlinear
model of the fiber TM over a much broader wavelength range [15]. This joint
simulation-experiment approach borrows ideas from data augmentation and domain
transfer [23, 24, 25]. The use of adaptive loss functions, such as in
generative-adversarial networks, may further enable convergence using
relatively small experimental datasets, or else help to generate further
training data. In these latter scenarios it could be envisioned that the
reflector need not be characterized in advanced and can be inferred from a
small number of experimental measurements.

The second key trade-off is that the training process is very memory-intensive
for dealing with large sizes of TM that are typically encountered in imaging
applications e.g. $1024\times 1024$ would require over 1TB for training the
recovery model. The convolutional U-net architecture has much ($\times$1000)
fewer trainable parameters, which reduces memory usage somewhat, although the
large dimension of each input TM also has a large influence on the memory
usage. However, using convolutional architectures may come at the expense of
slightly increased error (a few percent) as we observed in our TM
reconstruction results. One possible solution is to develop matrix compression
techniques such as auto-encoder models to represent matrices in a low-
dimensional latent space. Reducing batch size will also reduce memory usage
but can lead to greater fluctuations and poorer convergence.

We anticipate this neural network-based TM recovery model will lead to
machine-learning models for complex-valued data, for example in holographic
imaging and projection and phase retrieval, where both phase control and speed
are required.

##  4 CONCLUSIONS

Overall, our model for reconstructing $64\times 64$ complex-valued fiber TMs
through a single-ended optical fiber system achieves $\leq 4\%$ error for both
FCNN and convolutional U-net based neural network architectures. By re-
training our FCNN model we also show that it can reconstruct experimentally
measured $64\times 64$ TMs with $3.42\%$ error. Using our recovered TMs we
demonstrate image reconstruction of complex objects in two imaging modalities:
8$\times 8$ pixel widefield imaging, achieving IMMAE $\leq 9\%$ and SSIM $\geq
83\%$, and confocal scanning, achieving $48.5\%$ power focused in the spot and
$\leq 5\%$ IMMAE and $\geq 90\%$ SSIM.

##  5 METHODS

###  5.1 Simulated data generation

We present a TM recovery method that uses neural networks, instead of using
iterative approaches [14], to solve the Eqn. 2 – Eqn. 4. Figure 8 shows the
schematic of this TM recovery model. Specifically, we first simulated $N$
optical fiber TMs, $\mathbf{A_{\lambda_{1}}}\in\mathbb{C}^{M^{2}\times
M^{2}}$, at a wavelength of ${\lambda_{1}}$ as the ground truth. Here we use
$M=8$ and $\lambda_{1}=850nm$. Our simulation model aims to recreate typical
properties found in practical fiber TMs. First, we assume that TMs are sparse
in some commonly used basis e.g. LP modes for multimode fibers or pixel basis
for multicore fibers [26]. Second, we assume TMs can be arranged such that the
majority of power intensity lies along the main diagonal with additional power
spread along sub-diagonals, which is also typically observed when using bases
closely matched to the fiber eigenbasis[27]. Third, we assume that realistic
TMs are slightly non-unitary, with mode-dependent loss values (i.e. condition
numbers) in the range of 3-5 [27, 26]. To meet these assumptions, we generate
a uniformly distributed random tri-diagonal matrix,
$\mathbf{B}\in\mathbb{C}^{64\times 64}$, which has non-zero elements only at
the main diagonal, subdiagonal and superdiagonal. We then compute the left
singular matrix $\mathbf{U}\in\mathbb{C}^{64\times 64}$ and right singular
matrix $\mathbf{V}\in\mathbb{C}^{64\times 64}$ via singular value
decomposition (SVD). We then apply a new singular value distribution via a
matrix $\mathbf{S_{new}}\in\mathbb{R}^{64\times 64}$, a diagonal matrix that
contains random values along its diagonal ranging from 0.5 to 2.5 to simulate
our expected TM. We then construct the TM at $\lambda_{1}$ as:

$\mathbf{A_{\lambda_{1}}}=\mathbf{U}*\mathbf{S_{new}}*\mathbf{V}^{T}$ (5)

Next, we apply Eqn. 2 \- Eqn. 4 to simulate corresponding TMs,
$\mathbf{A_{\lambda_{1}}}$ and $\mathbf{A_{\lambda_{1}}}$ at wavelengths
$\lambda_{2}=852nm$ and $\lambda_{3}=854nm$. Here, we use wavelengths
$\lambda_{1}=850$nm, $\lambda_{2}=852$nm and $\lambda_{3}=854$nm as physically
realistic values within the TM bandwidth of a typical endoscope length fiber
($\sim$2m) [14].

Following this, we simulate reflectors on the fibre distal tip by generating
three complex-valued matrices with complex uniform randomly distributed
elements (-1 to 1 and -$i$ to $i$). These three matrices then form our
reflector matrices, $\mathbf{R_{\lambda_{1}}}\in\mathbb{C}^{M^{2}\times
M^{2}}$, $\mathbf{R_{\lambda_{2}}}\in\mathbb{C}^{M^{2}\times M^{2}}$, and
$\mathbf{R_{\lambda_{3}}}\in\mathbb{C}^{M^{2}\times M^{2}}$ at wavelengths
${\lambda_{1}}$, $\lambda_{2}$ and $\lambda_{3}$ respectively. By generating
reflectors in this way we ensure with high probability, according to random
matrix theory, that the eigenvalues are distinct [28].

Finally, we combine the TMs and reflector matrices to generate $N$ sets of
complex-valued reflection matrices
$\mathbf{C_{\lambda_{1}}}\in\mathbb{C}^{64\times 64}$,
$\mathbf{C_{\lambda_{2}}}\in\mathbb{C}^{64\times 64}$, and
$\mathbf{C_{\lambda_{3}}}\in\mathbb{C}^{64\times 64}$ at three different
wavelengths $\lambda_{1}=850$nm, $\lambda_{2}=852$nm and $\lambda_{3}=854$nm.

To feed this data to our neural network, which only accepts real numbers, we
convert inputs and outputs from complex to real-valued data. A $2\times 2$
complex-valued matrix can be represented by a $4\times 4$ real-valued matrix
as follows [29]:

$\begin{bmatrix}{a+bi}&{c+di}\\\
{e+fi}&{g+hi}\end{bmatrix}\mapsto\begin{bmatrix}a&{-b}&c&{-d}\\\ b&a&d&c\\\
e&{-f}&g&{-h}\\\ f&e&h&g\\\ \end{bmatrix}$ (6)

where, $\mapsto$ indicates an isomorphism.

Finally, the input of the model, three
$\mathbf{C_{\lambda}}\in\mathbb{C}^{128\times 128}$ at different wavelengths
are normalized using in the range from -1 and 1. Each set of 3 reflection
matrices, $\mathbf{C_{\lambda_{1..3}}}$ represents a single input to our
neural network model. We split the $N$ data into training, validation and test
sets with an 8:2:1 ratio. The validation set is expected to provide unbiased
evaluations and stopping criteria on unseen data, and test set aims to examine
the generalization performance of the model on unseen data. Test set is
independent to the validation set and contains unseen data that is not used
during training process. The model is trained with the ADAM optimizer using
our custom-defined loss function. Python was used for model training and
MATLAB was used for data pre-processing and post-processing because of its
ease of use for complex matrix computations.

Figure 8: Schematic of transmission matrix (TM) recovery model. This contains
(a) data generation, (b) data pre-processing and (c) model training. $N$ pairs
of TM are firstly simulated as the ground truth. $\lambda_{1}$, $\lambda_{2}$
and $\lambda_{3}$ represent three different wavelengths (in our case, 850nm,
852nm and 854nm). The input of the model is all real-valued matrices
concatenated with reflection matrices at three different wavelengths. L
represents custom loss function and w represents the weight updated by the
optimizer.

###  5.2 Imaging modalities

We validate our system via two commonly used fiber imaging modalities: wide-
field and confocal scanning. For wide-field imaging we assume a pixel basis at
the fiber distal end, and that a ground-truth complex image
$\mathbf{X}\in\mathbb{C}^{M^{2}}$ passes via the simulated optical fiber
ground-truth TM at a fourth wavelength $\lambda_{4}=854.5$nm. We then use the
recovered TM at a fourth wavelength $\lambda_{4}=854.5$nm to estimate original
image and compare this to the ground-truth. For simplicity, we assume the
object is located at the distal fiber facet with no loss coupling into and out
of the fibre, and we neglect the loss in transferring through the reflector
stack but consider random Gaussian noise to better simulate expected behaviour
in a real system. Theoretically, the reconstructed widefield image,
$\mathbf{\hat{X}_{j}}\in\mathbb{C}^{M^{2}}$ can be calculated by:

$\mathbf{\hat{X_{j}}}=\mathbf{(\hat{A}_{\lambda_{4}}}^{T})^{-1}(\mathbf{{A_{\lambda_{4}}}}^{T}\mathbf{X_{j}}+\mathbf{Z})$
(7)

where, $\mathbf{Z}$ is random Gaussian noise with a power of 2% of the target
signal. $\mathbf{\hat{X}_{j}}$ is the reconstructed image, $\mathbf{X_{j}}$ is
the target image, $\mathbf{\hat{A}_{\lambda_{4}}}$ and
$\mathbf{A_{\lambda_{4}}}$ are the recovered TM and target TM at wavelength
$\lambda_{4}$ respectively.

For confocal imaging, which is commonly used in fiber imaging [30], we assume
an LP mode basis at the fiber output, which is a realistic and widely used
fiber basis [31]. To implement this, we first simulated a multimode fiber with
a core radius of $30\mu m$, length of $1.5m$ and a numerical aperture (NA) of
0.24 and used 64 of the available modes as our basis for creating a spot to be
scanned. We used our recovered TM to estimate the required proximal field to
create a spot then examined the expected spot by forward propagating via the
ground truth TM. We next scanned $128\times 128$ approximate spot positions
and found the average percentage of power within the FWHM of the spot. Using
this approach we reconstructed confocal images of three target images, denoted
$\mathbf{x}\in\mathbb{C}^{128\times 128}$ that contain the same pattern shown
in Result Section Widefield image reconstruction based on recovered TM by
integrating total reflected power for each spot position.

###  5.3 Metrics

We next define loss metrics to compare the complex overlap integral between
the target and recovered results. However, due to the degenerate global phase
factor, this complex overlap integral is multi-valued. Therefore, the loss
metrics defined here first normalize to correct for the global phase term (see
Methods Section Global phase insentive loss function) before calculating the
complex correlation. This is an essential step to gauge the accuracy both for
TM recovery and image reconstruction in widefield and confocal modalities.

To gauge the accuracy of our TM recovery we define an average loss metric by
calculating the average mean absolute error (MAE) of each validated TM in the
test data:

$\mathrm{average\ loss\
(\mathbf{\hat{A}_{t}},\mathbf{A_{t}})}=\frac{1}{N_{test}}\times\frac{1}{4M^{2}}\sum_{n=1}^{N_{test}}\sum_{t=1}^{4M^{2}}|\mathbf{\hat{A}_{t}}-\mathbf{A_{t}}e^{i\phi_{c}}|$
(8)

where, $\mathbf{\hat{A}_{t}}$ is the recovered TM, $\mathbf{A_{t}}$ is the
target TM, $e^{i\phi_{c}}$ is the global phase normalization term, $4M^{2}$ is
the total number of TM elements, and $N_{test}$ is the number of data used for
testing.

To gauge the accuracy of widefield reconstructed image based on the recovered
TM, we define two metrics, namely the image MAE (IMMAE) and complex-valued
based structural similarity index measure (SSIM) of each reconstructed image:

$\mathrm{IMMAE\
(\mathbf{\hat{X}_{j}},\mathbf{X_{j}})}=\frac{1}{M^{2}}\sum_{j=1}^{M^{2}}|\mathbf{\hat{X}_{j}}-\mathbf{X_{j}}e^{i\phi_{c}}|$
(9) $\mathrm{SSIM\
(\mathbf{\hat{X}_{j}},\mathbf{X_{j}})}=\frac{2|\sum_{j=1}^{M^{2}}\mathbf{\hat{X}_{j}}\mathbf{X_{j}}^{*}e^{i\phi_{c}}|+K}{\sum_{j=1}^{M^{2}}|\mathbf{\hat{X}_{j}|}^{2}+\sum_{j=1}^{M^{2}}|\mathbf{X_{j}|}^{2}+K}$
(10)

where, $M^{2}$ is the total number of pixels of image $\mathbf{X_{j}}$,
$\mathbf{X_{j}}^{*}$ is the complex conjugate of $\mathbf{X_{j}}$, $K=0.03$ is
a positive constant to improve the robustness when the local signal to noise
ratios are low. These two metrics can also be extended to evaluate the
performance of recovered confocal images.

In the confocal case, we compare recovered images $\mathbf{\hat{Y}}$ against
the confocal imaging scenario with an identity TM, $\mathbf{Y}$, using the two
metrics (IMMAE and SSIM) introduced in Eqn. 9 and Eqn. 10. This aims to do a
fair comparison considering the effect of conversion between fiber mode basis
and pixel basis.

###  5.4 Network architectures

We defined two neural network models: a fully-connected neural network (FCNN)
and convolution U-net-based neural network as shown in Figure 9. The FCNN is a
ten-layer densely connected neural network (eight hidden layers), including
32,768 neurons in first and last hidden layers and 8192 neurons in other
layers, all with Leaky ReLU activation function. Figure 9 shows the FCNN
architecture, where reflection matrices
$\mathbf{C_{\lambda_{1}}}\in\mathbb{R}^{128\times 128}$,
$\mathbf{C_{\lambda_{2}}}\in\mathbb{R}^{128\times 128}$,
$\mathbf{C_{\lambda_{3}}}\in\mathbb{R}^{128\times 128}$ are flattened into
$1D$ arrays and then concatenated as the input of the model (with the size of
$49152\times 1$) and TM $\mathbf{A_{\lambda_{1}}}\in\mathbb{R}^{128\times
128}$, flattened into $1D$ array as the output (with the size of $16384\times
1$). Batch normalization layers were defined between every dense layer and
dropout layers at the rate of $0.2$ were defined after the first two dense
layers. Also two skip connections were developed in order to prevent the model
overfitting. The model was trained iteratively with the global phase-
insensitive custom loss function used. The training dataset for recovering
$64\times 64$ TM consisted of 500,000 matrices and the model was run for 2500
epochs, taking 182.5 hours using Tensorflow 2.0 running on a NVIDIA Tesla V100
GPU. The Adam optimizer was used with a learning rate of 0.004 in a decay rate
of $1e^{-4}$.

Next, we developed a U-net-based model that used encoder-decoder architecture,
including seven Conv2D and DeConv2D layers respectively and two MaxPooling and
UpSampling layers respectively with LeakyRelu activation function in each
layer. Figure 9 shows this architecture, where reflection matrices
$\mathbf{C_{\lambda_{1}}}\in\mathbb{R}^{128\times 128}$,
$\mathbf{C_{\lambda_{2}}}\in\mathbb{R}^{128\times 128}$,
$\mathbf{C_{\lambda_{3}}}\in\mathbb{R}^{128\times 128}$ are defined in three
channels as the input of the model (with the size of $128\times 128\times 3$)
and TM $\mathbf{A_{\lambda_{1}}}\in\mathbb{R}^{128\times 128}$, as the output
(with the size of $128\times 128\times 1$). Batch normalization layers were
defined between every layer and dropout layers at the rate of $0.2$ were
defined after the second and last second Conv layers. Also three skip
connections were developed in order to prevent the model being overfitted. The
model was trained iteratively with the global phase insensitive custom loss
function defined. Also, 2200 epochs were used for training 400,000 training
datasets using 143h. The Adam optimizer was used with a learning rate of 0.004
in a decay rate of 1e-4.

Figure 9: Architectures of two different neural network models used for TM
recovery. (a) Fully-connected neural network, (b) Convolutional U-net

###  5.5 Global phase insensitive loss function

Widely-used conventional loss functions such as mean absolute error (MAE) or
mean squared error (MSE) calculate the absolute difference between predicted
and target output values. However, there is a class of problems whose
solutions trained by deep learning models are degenerate within a global phase
factor, but whose relative phase between pixels must be preserved. This class
includes problems where complex TMs are reconstructed and relative phase, but
not global phase, could extend to phase-hologram generation algorithms where
replay-field phase is relavant [32]. This is depicted visually in
Supplementary Figure 3(a), which shows one example of a pair of predicted and
target matrices with complex entries depicted as vectors. Supplementary Figure
3(b) shows the complex error between these two matrices when using MAE as the
loss function. Due to the global phase shift, we observe that the vectors have
large magnitudes, which will lead to an overall very large MAE when their
magnitudes are summed. In the limiting case (e.g. when the phase shift is of
$\pi$) where the predicted and target matrices are identical, this global
phase shift can result in a normalized MAE of 100% when the true value should
be 0%. To avoid this problem, we propose a custom loss function termed a
‘global phase insensitive’ loss function that normalizes for this global phase
factor:

$L(\mathbf{\widehat{A_{t}}}(w),\mathbf{A_{t}})=\sum_{t=1}^{4M^{2}}\left|\mathbf{\widehat{A_{t}}}(w)-\mathbf{A_{t}}e^{i\phi(\mathrm{Tr}(\mathbf{A_{t}}^{H}\mathbf{\widehat{A_{t}}}(w))}\right|+\frac{\alpha}{2}{\|w\|}^{2}$
(11)

where, $\mathbf{\widehat{A_{t}}}(w)\in\mathbb{C}^{M^{2}\times M^{2}}$ and
$\mathbf{A_{t}}\in\mathbb{C}^{M^{2}\times M^{2}}$ represent predicted and
target output value with regards to weight, $w$, respectively, $\sum$
represents summation over all matrix elements, $\phi$ represents the argument
function for a complex number input. We add an $\ell_{2}$ regularization term
to encourage generalization of the model with regularization parameter,
$\alpha$ = $10^{-4}$. This formula implicitly weights the phase contributions
by the product of magnitudes of the respective elements in
$\mathbf{\widehat{A_{t}}}(w)$ and $\mathbf{{A_{t}}}$, which upon convergence
will approximately equal the squared magnitude of the target.

The rationale for this is that when the optimization algorithm approaches a
minimum, the remaining error for each complex element will be entirely due to
aleatoric uncertainty, e.g. a circularly symmetric zero-mean complex Gaussian
distribution [33]. To estimate the correction factor, the element-wise complex
errors can be summed, as shown in Supplementary Figure 3(c). This will produce
an overall complex factor that has the desired global phase shift, shown in
Supplementary Figure 3(d). The predicted output value can be corrected by
multiplying by this phase factor as shown in Supplementary Figure 3(e), the
result of which is then used to compute further parameter updates in the
gradient descent algorithm. It can be seen that the complex error in
Supplementary Figure 3(f) between the predicted and target output value is
reduced to a minimum after removing the phase factor compared to that
calculated by MAE. We then compared the absolute values of the complex error
calculated by MAE (green bar) and our customized global phase insensitive loss
function (blue bar) respectively over 100,000 pairs of predicted and desired
TM as shown in Supplementary Figure 3(g). The error using the custom loss
function is more than two times smaller than that of the conventional loss
function (MAE).

###  5.6 Experimental measurement and updated model setup

The optical layout of our experimental setup is provided in [12]. Measuring
the TM, treated as a linear operator, involves input modulation and output
recovery. To achieve input modulation, the incident light was collimated into
MMF by an objective lens and a 4$f$ configuration after reflecting from
digital micro-mirror device (DMD). In our specific case, the input fields are
implemented using the Hadamard basis, providing an orthogonal set of modulated
inputs. For the complex output recovery, images of the output facet of the
fiber are captured using a movable calibration module. The light from the
multimode fiber (MMF) is combined with the reference signal using a beam
splitter. The reference light is directed to the camera through a single-mode
fiber, and then the off-axis holography is recorded by CCD camera after
magnifying by a microscope objective. By employing this optical layout, we are
able to accurately measure the TM and gain insights into the propagation
characteristics of light through the MMF. Using this approach, we measured
experimental TMs for 164 different bending conformations and selected 64 rows
and 64 columns of the raw measurements to create suitably sized matrices for
training our model. TMs are measured at a wavelength of 488 nm and we use
these as our base $A_{\lambda_{1}}$ at wavelength $\lambda_{1}$. We then scale
this matrix to $\lambda_{2-3}$ using the method in Eqn. 3 and Eqn. 4, which
has been validated previously on multi-wavelength TM datasets [14].

For a small size matrix ($8\times 8$) we find that the model trained on
simulation data is sufficient to reconstruct cropped $8\times 8$ experimental
matrices. However, for the larger $64\times 64$ matrices, the model pre-
trained on simulated data is prone to overfitting as it tends to also learn
the prior distribution of TMs, and the subsequent loss is around $9.24\%$
(standard deviation 0.78%). This learning of a prior can be advantageous in
some cases where prior information is essential to resolve degeneracies, but
to avoid that here we next re-trained the model with an augmented dataset
containing 600,000 additional random matrices with elements drawn from complex
uniform distribution. This reduced loss to an average value of around $5.07\%$
(standard deviation of 0.67%), but a degree of overfitting is still present
and so tweaked the model by continuing the training on 100 experimental TM,
keeping 64 aside for validation and testing. Visually, these experimental TMs
are structured differently to our simulated TM due to their use of a different
basis. However, the training on random matrices seems to encourage
generalisation to different matrix structures.

## DATA AVAILABILITY

The data presented in this study are available from the following source:
https://doi.org/10.17639/nott.7334

## CODE AVAILABILITY

The code for this study is available from the following source:
https://doi.org/10.17639/nott.7334.

## AUTHOR CONTRIBUTIONS

YZ: Idea conception, code implementation, result analysis, manuscript writing.
TW: Code implementation for confocal imaging. WZ, QY: Collection of
experimental data, manuscript writing. GG: Idea conception, project
supervision, manuscript writing.

## COMPETING INTERESTS

The authors declare no competing interests.

## ACKNOWLEDGEMENT

The authors acknowledge support from a UKRI Future Leaders Fellowship
(MR/T041951/1) and an EPSRC Ph.D. studentship.

## References

  * [1] Cancer Research UK.  Boston scientific, portfolio-group.
  * [2] Cameron M. Lee, Christoph J. Engelbrecht, Timothy D. Soper, Fritjof Helmchen, and Eric J. Seibel.  Scanning fiber endoscopy with highly flexible, 1 mm catheterscopes for wide-field, full-color imaging.  Journal of Biophotonics, 3:385–407, 3 2010.
  * [3] Gavrielle R Untracht, Karol Karnowski, and David D Sampson.  Imaging the small with the small: Prospects for photonics in micro-endomicroscopy for minimally invasive cellular-resolution bioimaging.  APL Photonics, 6(6):060901, 2021.
  * [4] Kyungmin Hwang, Yeong-Hyeon Seo, Daniel Y Kim, Jinhyo Ahn, Soyoung Lee, Kyung Hee Han, Koun-Hee Lee, Sangyong Jon, Pilhan Kim, Kate E Yu, et al.  Handheld endomicroscope using a fiber-optic harmonograph enables real-time and in vivo confocal imaging of living cell morphology and capillary perfusion.  Microsystems & Nanoengineering, 6(1):72, 2020.
  * [5] Turtaev et al.  High-fidelity multimode fibre-based endoscopy for deep brain in vivo imaging.  Light: Science & Applications, 7(1):1–8, 2018.
  * [6] Demetri Psaltis and Christophe Moser.  Imaging with multimode fibers.  Optics and Photonics News, 27:24, 1 2016.
  * [7] Ruo Yu Gu, Reza Nasiri Mahalati, and Joseph M. Kahn.  Design of flexible multi-mode fiber endoscope.  Optics Express, 23:26905, 10 2015.
  * [8] Shuhui Li, Simon AR Horsley, Tomáš Tyc, Tomáš Čižmár, and David B Phillips.  Memory effect assisted imaging through multimode optical fibres.  Nature Communications, 12(1):3751, 2021.
  * [9] Ang Li, Wenxuan Liang, Honghua Guan, Yung-Tian A Gau, Dwight E Bergles, and Xingde Li.  Focus scanning with feedback-control for fiber-optic nonlinear endomicroscopy.  Biomedical optics express, 8(5):2519–2527, 2017.
  * [10] Uri Weiss and Ori Katz.  Two-photon lensless micro-endoscopy with in-situ wavefront correction.  Optics Express, 26(22):28808–28817, 2018.
  * [11] Salma Farahi, David Ziegler, Ioannis N Papadopoulos, Demetri Psaltis, and Christophe Moser.  Dynamic bending compensation while focusing through a multimode fiber.  Optics express, 21(19):22504–22514, 2013.
  * [12] Zhong Wen, Zhenyu Dong, Qilin Deng, Chenlei Pang, Clemens F Kaminski, Xiaorong Xu, Huihui Yan, Liqiang Wang, Songguo Liu, Jianbin Tang, et al.  Single multimode fibre for in vivo light-field-encoded endoscopic imaging.  Nature Photonics, pages 1–9, 2023.
  * [13] Haoshuo Chen, Nicolas K Fontaine, Roland Ryf, David T Neilson, and Peter Winzer.  Remote spatio-temporal focusing over multimode fiber enabled by single-ended channel estimation.  IEEE Journal of Selected Topics in Quantum Electronics, 26(4):1–9, 2020.
  * [14] George SD Gordon, Milana Gataric, Alberto Gil CP Ramos, Ralf Mouthaan, Calum Williams, Jonghee Yoon, Timothy D Wilkinson, and Sarah E Bohndiek.  Characterizing optical fiber transmission matrices using metasurface reflector stacks for lensless imaging without distal access.  Physical Review X, 9(4):041050, 2019.
  * [15] Szu-Yu Lee, Vicente J Parot, Brett E Bouma, and Martin Villiger.  Efficient dispersion modeling in optical multimode fiber.  Light: Science & Applications, 12(1):31, 2023.
  * [16] Shuhui Li, Charles Saunders, Daniel J Lum, John Murray-Bruce, Vivek K Goyal, Tomáš Čižmár, and David B Phillips.  Compressively sampling the optical transmission matrix of a multimode fibre.  Light: science & applications, 10(1):88, 2021.
  * [17] Guoqiang Huang, Daixuan Wu, Jiawei Luo, Yin Huang, and Yuecheng Shen.  Retrieving the optical transmission matrix of a multimode fiber using the extended kalman filter.  Optics Express, 28:9487, 3 2020.
  * [18] Babak Rahmani, Damien Loterie, Georgia Konstantinou, Demetri Psaltis, and Christophe Moser.  Multimode optical fiber transmission with a deep learning network.  Light: science & applications, 7(1):69, 2018.
  * [19] Yifan Liu, Panpan Yu, Yijing Wu, Ziqiang Wang, Yinmei Li, Jinyang Liang, Puxiang Lai, and Lei Gong.  Single-shot wide-field imaging in reflection by using a single multimode fiber.  Applied Physics Letters, 122(6):063701, 2023.
  * [20] Pengfei Fan, Yufei Wang, Michael Ruddlesden, Xuechun Wang, Mohamed A Thaha, Jiasong Sun, Chao Zuo, and Lei Su.  Deep learning enabled scalable calibration of a dynamically deformed multimode fiber.  Advanced Photonics Research, 3(10):2100304, 2022.
  * [21] Ralf Mouthaan, Peter J Christopher, George SD Gordon, Timothy D Wilkinson, and Tijmen G Euser.  Robust correction of interferometer phase drift in transmission matrix measurements.  Applied Optics, 61(15):4315–4321, 2022.
  * [22] George SD Gordon, James Joseph, Travis Sawyer, Alexander J Macfaden, Calum Williams, Timothy D Wilkinson, and Sarah E Bohndiek.  Full-field quantitative phase and polarisation-resolved imaging through an optical fibre bundle.  Optics express, 27(17):23929–23947, 2019.
  * [23] Sebastian J Wirkert, Anant S Vemuri, Hannes G Kenngott, Sara Moccia, Michael Götz, Benjamin FB Mayer, Klaus H Maier-Hein, Daniel S Elson, and Lena Maier-Hein.  Physiological parameter estimation from multispectral images unleashed.  In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part III 20, pages 134–141. Springer, 2017\.
  * [24] Ahmed Osman, Jane Crowley, and GS D Gordon.  Training generative adversarial networks for optical property mapping using synthetic image data.  Biomedical Optics Express, 13(10):5171–5186, 2022.
  * [25] Connor Shorten and Taghi M Khoshgoftaar.  A survey on image data augmentation for deep learning.  Journal of big data, 6(1):1–48, 2019.
  * [26] George SD Gordon, Ralf Mouthaan, Timothy D Wilkinson, and Sarah E Bohndiek.  Coherent imaging through multicore fibres with applications in endoscopy.  Journal of Lightwave Technology, 37(22):5733–5745, 2019.
  * [27] Joel Carpenter, Benjamin J Eggleton, and Jochen Schröder.  110x110 optical mode transfer matrix inversion.  Optics express, 22(1):96–101, 2014.
  * [28] Ludwig Arnold.  On wigner’s semicircle law for the eigenvalues of random matrices.  Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, 19(3):191–198, 1971.
  * [29] Gerald N Hile and Pertti Lounesto.  Matrix representations of clifford algebras.  Linear algebra and its applications, 128:51–63, 1990.
  * [30] Damien Loterie, Salma Farahi, Ioannis Papadopoulos, Alexandre Goy, Demetri Psaltis, and Christophe Moser.  Digital confocal microscopy through a multimode fiber.  Optics Express, 23:23845, 9 2015.
  * [31] Martin Plöschner, Tomáš Tyc, and Tomáš Čižmár.  Seeing through chaos in multimode fibres.  Nature Photonics, 9(8):529–535, 2015.
  * [32] Yijie Zheng and George Gordon.  Global phase insensitive loss function for deep learning in holographic imaging and projection applications.  In AI and Optical Data Sciences IV, volume 12438, pages 288–293. SPIE, 2023.
  * [33] Dhaivat Bhatt.  Variational aleatoric uncertainty calibration in neural regression.  2021\.


# An empirical study of weakly supervised audio tagging embeddings for general
audio representations

###### Abstract

We study the usability of pre-trained weakly supervised audio tagging (AT)
models as feature extractors for general audio representations. We mainly
analyze the feasibility of transferring those embeddings to other tasks within
the speech and sound domains. Specifically, we benchmark weakly supervised
pre-trained models (MobileNetV2 and EfficientNet-B0) against modern self-
supervised learning methods (BYOL-A) as feature extractors. Fourteen
downstream tasks are used for evaluation ranging from music instrument
classification to language classification. Our results indicate that AT pre-
trained models are an excellent transfer learning choice for music, event, and
emotion recognition tasks. Further, finetuning AT models can also benefit
speech-related tasks such as keyword spotting and intent classification.

##  1 Introduction

The recognition of audio patterns is a vital part of enabling machines an
understanding of our world. Automatic audio pattern recognition is a broad
research field, which can be split into two main categories: sound-related and
speech-related. Speech-related tasks mainly focus on the content of spoken
languages i.e., who/what/when is something spoken? On the other hand, the
sound domain focuses on classification, i.e., Bird songs, acoustic scenes,
emotion, sound events. Most prior works focused on either of those domains [1,
2, 3] i.e., models trained in the sound domain were also evaluated in the
sound domain.

This work aims to provide an empirical study on the performance of models
trained in the sound domain and evaluated on both sound and speech domains. We
believe that audio tagging (AT) is a reasonable pre-training task for sound
and speech-domain tasks since AT incorporates unconstrained speech data,
spoken in different languages and a variety of accents as well as sound events
such as music. Recent challenges such as the holistic evaluation of audio
representations (HEAR) 2021 111For further information please visit
https://neuralaudio.ai/. [4] have aimed to further push the efforts in
researching universal audio embeddings. To the best of our knowledge, there
has been no comprehensive work that compared self-supervised approaches to
their weakly supervised counterparts in the sound domain. Thus, this work aims
to investigate the usage of AT pre-trained neural networks as a general audio
representation, especially we aim to answer whether AT pre-trained models are
useful for speech-related tasks. Our contributions are as follows:

  * •

Explore AT pre-trained models as a common back-bone for sound and speech
downstream tasks.

  * •

A comparison of weakly supervised pretraining against modern self-supervised
approaches.

  * •

We provide insights into the limits of sound domain pretrained models to their
application in the speech domain.

##  2 Related work

Figure 1: In this work, we investigate the use of (upstream) Audio tagging
pre-trained models as general audio representations in two domains: sound and
speech.

In terms of weakly supervised pre-training, the works in [3, 5, 6, 7] proposed
novel model architectures for weakly supervised AT and explored the
performance on a limited amount of downstream tasks.

Further, the works [8, 9, 10, 11, 12, 13] explored self-supervised learning
with a linear classifier for sound classification, focusing on sound-domain
downstream tasks. Initial work for cross-domain evaluation has been done in
[1, 9], but those works mainly compared against other self-supervised or
traditional weakly-supervised methods. The work in [14] explored the effects
of weakly-supervised audio tagging pretraining to sound-domain downstream
tasks, such as music tagging and acoustic scene classification.

The paper is organized as follows. Section 3 describes our proposed framework
and experimental details are laid out in Section 4. Results are presented in
Section 5 and a conclusion is provided in Section 6.

##  3 Framework

Our approach trains a standard multi-label multi-class convolutional neural
network (CNN) model on the $5200\text{\,}\mathrm{h}$ long Audioset [15]
containing 527 labels. Given a training audio clip signal $\boldsymbol{x}$
(i.e., of length 10 s) and a multi-hot label
$\boldsymbol{y}\in\\{0,1\\}^{527}$, the training objective is to predict
$\boldsymbol{y}$ using a model $\mathcal{F}$, such that
$\mathcal{F}(\boldsymbol{x})\mapsto\boldsymbol{y}$. The model is trained to
optimize the binary cross entropy (BCE), defined as:

$\mathcal{L}_{\text{BCE}}=\boldsymbol{y}\log\boldsymbol{\hat{y}}+(\boldsymbol{1}-\boldsymbol{y})\log(\boldsymbol{1}-\boldsymbol{\hat{y}}),$
(1)

between the model predictions $\boldsymbol{\hat{y}}$, which are commonly
processed using a sigmoid activation function, and the ground truth
$\boldsymbol{y}$. We use MobilenetV2 [16] (MBv2) and an Efficientnet-B0 [17]
(Eff-B0) as our neural network back-bones ($\mathcal{F}$), both of which are
off the shelf models deployed in the vision domain and can be adapted to the
sound domain with little effort.

Different from off the shelf available vision-based model, our audio-domain
MBv2 and Eff-B0 networks use a decision-level output pooling function, which
produces a single output vector $\hat{\boldsymbol{y}}\in\mathbb{R}^{527}$,
from its per-timestep outputs $\hat{\boldsymbol{y}}_{1:T}$ as
$\hat{\boldsymbol{y}}=\frac{1}{T}\sum_{t}^{T}\boldsymbol{\hat{y}}_{t}$ for the
(pooled) audio duration $T$.

Then, we explore transfer learning to downstream tasks by either freezing
(only train a linear classifier) or finetuning (train entire model) the pre-
trained CNN model, as can be seen in Figure 1.

##  4 Experiments

###  4.1 Datasets

This work uses a single dataset for pretraining purposes, being Audioset [15]
and 14 different downstream datasets for finetuning. A short introduction for
each dataset is provided.

Dataset | #Train | #Eval | # Labels | Metric
---|---|---|---|---
Audioset [15] | 1,904,746 | 18,229 | 527 | mAP
US8k† [18] | 8,732 | - | 10 | Accuracy
FSD18 [19] | 8,530 | 1,600 | 41 | mAP@3
FSD19-C [20] | 4,473 | 4,481 | 80 |  lwlwrap
FSD19-N [20] | 17,833 | 4,481 | 80 |  lwlwrap
FSD50k [21] | 36,796 | 10,231 | 200 | mAP
ESC-50† [22] | 2,000 | - | 10 | Accuracy
NS [23] | 283,704 | 4,097 | 10 | Accuracy
MSoS [24] | 1,350 | 50 | 5 | Accuracy
RAV† [25] | 1,400 | - | 8 | Accuracy
SCV1/11 [26] | 51,088 | 6,835 | 11 | Accuracy
SCV1/30 [26] | 51,088 | 6,835 | 30 | Accuracy
SCV2 [26] | 84,843 | 11,005 | 35 | Accuracy
FSC [27] | 23,132 | 3,793 | 31 | Accuracy
Vox1 [28] | 138,361 | 8,251 | 1251 | Accuracy
VL [29] | 2,536,954 | 1,609 | 107 | ErrorRate
Table 1: Description of datasets used in this work with their respective
evaluation metrics and their respective number of samples for training
(#Train) and evaluation (#Eval). Datasets denoted with a † are k-fold cross-
validated. Audioset is used as the upstream dataset for pretraining purposes
exclusively. mAP represents mean average precision.

####  4.1.1 Sound domain datasets

Three Freesound (FSD) datasets are used in this work, all of which focus on
(multi-label) audio tagging. We deploy FSDKaggle2018 [19] (FSD18),
FSDKaggle2019 [20] (FSD19) and FSD50k [21] as our downstream datasets. All
recordings in the FSD datasets were obtained from Freesound and annotated with
different vocabulary sizes. FSD19 dataset has two optional training sets, a
clean curated one (FSD19-C) and a large noisy one (FSD19-N). ESC-50 is an
environmental sound dataset [22] consisting of 50 distinct sound labels. The
Making Sense of Sounds [24] (MSoS) dataset focuses on classifying five
categories types. NSynth [23] contains musical notes, labeled with the
instrument family with overall 11 classes. The Ryerson Audio-Visual Database
of Emotional Speech [25] (RAVDESS) contains eight different emotions portrayed
by professional actors. UrbanSound8k [18] (US8k) contains ten acoustic scene
classes in urban environments.

####  4.1.2 Speech domain datasets

Keyword spotting (KWS) detects a fixed set of predefined keywords within a
spoken utterance. We use the Google Speech Commands (SPC) V1 and V2 datasets
[26] for this task, which contains 30 and 35 keywords, respectively. For V1,
we use the common 11 class subset, where the original 30 classes have been
reduced to 10 common keywords + “Unknown”. The largest downstream dataset in
this work is the VoxLingua107 [29] (VL) dataset, containing over 6000 hours of
data across 107 languages, which is aimed at language identification. Intent
classification is evaluated on the Fluent Speech Commands [27] (FSC) dataset,
which requires categorizing utterances into (31) predefined slots. Speaker
identification (SID) is a classification task aiming to identify a person by
their voice. The VoxCeleb1 [28] (Vox1) dataset is used for SID. The data
statistics and evaluation metrics for all downstream datasets are displayed in
Table 1.

###  4.2 Setup

Regarding front-end feature extraction, all audio clips are converted to a 16
kHz sampling rate. As a common backbone, we use log-Mel spectrograms (LMS)
with 64 bins extracted every 10 ms with a window of 32ms. For all experiments,
we apply batch-wise zero padding to the longest clip within a batch. We use a
batch-size of 32 for Audioset pretraining and 64 for all downstream tasks. The
neural network back-end is implemented in Pytorch [30].

####  4.2.1 Audioset pretraining details

For training our models on Audioset, a label-balancing sampling strategy is
used [31, 3], which selects samples from the training dataset such that each
batch contains at least one label instance. We use Adam optimization with a
starting learning rate of 1e-4 using standard binary cross-entropy (BCE) as
the criterion and a batch size of 32.

Audioset contains two training sets, being the 60 h long balanced set and the
5,200 h long unbalanced set. We train our models on the full (balanced +
unbalanced) training set for at most 100 epochs. We checkpoint every
$\approx\frac{1}{6}$ epoch, where the model is evaluated on the balanced
subset. An early stop of 5 checkpoints is adopted and the four best ranking
validation checkpoints are weight averaged and tested on the public evaluation
set.

Our augmentation procedure is split into wave-domain and spectral-domain. In
the wave-domain, we use polarity inversion, gain, and randomly shifting with
their default settings as defined in torch-
audiomentations222https://github.com/asteroid-team/torch-audiomentations. In
the spectral-domain, we use mixup [32] with $\beta(1,1)$ and SpecAug [33] with
a time mask of 96 frames and a frequency mask of 12 bins.

####  4.2.2 Downstream training details

For all downstream tasks, we run the experiments for at most 150 epochs with
an early stop of 15 epochs. Each downstream task saves the best model
checkpoint given its respective main metric (see Table 1). We use Adam
optimization with a starting learning rate of 1e-4 with no learning rate
scheduling. After training, we weight-average the four best checkpoints on the
held-out validation dataset and evaluate the averaged model on the evaluation
set. Note that for datasets that do not provide an independent cross-
validation dataset (VL, MSoS), we use a 90%/10% split of the available
development data for training and validation, respectively. Further, for tasks
that are evaluated over a K-Fold split (US8k, ESC-50, RAV), we use the pre-
defined train/test split for training and cross-validation respectively. On
the K-Fold datasets, an average result across all folds is reported.

For tasks with variable-length input (FSD-50k, Vox1, VL, FSD2019), we
preprocess the training dataset by first segmenting each training clip to trim
to a maximal length. We chose 10 seconds for FSD18 and FSDK50k datasets since
their sample duration ranges between 0.3 and 30s and 4 seconds for the Vox1,
FSD19, FSC, and VL datasets. Note that each segment inherits its corresponding
clip label. We use a batch size of 1 during evaluation, in order to avoid any
performance bias caused by zero-padding or clip-segmentation. Note data
augmentation is not applied for downstream tasks. Cross-entropy is the default
training criterion for all classification, which is replaced by BCE for multi-
label datasets (FSD50k, FSD18, FSD19).

##  5 Results

Model | #Params (M) |  mAP $\uparrow$ |  mAUC $\uparrow$ |  d’ $\uparrow$
---|---|---|---|---
MBv2 | 3.0 | 40.6 | 97.1 | 2.695
Eff-B0 | 4.6 | 41.7 | 96.4 | 2.546
MBv1 [3] | 4.7 | 38.9 | 97.0 | 2.653
MBv2 [3] | 4.0 | 38.3 | 96.8 | 2.624
Eff-B2 [6] | 13.6 | 43.0 | - | -
CNN14 [3] | 80.7 | 43.1 | 97.3 | 2.732
Table 2: Baseline results of our models (rows 1-2) on the public Audioset evaluation set against other approaches (rows 3-6) from literature. Higher is better for all metrics and the best results are in boldface. Dataset | MBv2 | Eff-B0
---|---|---
✐ | ❄ | 🔥 | ✐ | ❄ | 🔥
MSoS $\uparrow$ | 56.85 | 90.40 | 93.55 | 58.80 | 89.80 | 93.75
NS $\uparrow$ | 33.56 | 66.50 | 79.93 | 78.82 | 64.65 | 77.92
US8K $\uparrow$ | 74.77 | 83.27 | 84.29 | 75.03 | 83.00 | 85.22
RAV $\uparrow$ | 46.54 | 50.90 | 66.66 | 11.66 | 48.68 | 70.00
ESC-50 $\uparrow$ | 59.90 | 92.20 | 93.70 | 58.17 | 92.10 | 94.48
FSD18 $\uparrow$ | 85.60 | 87.31 | 94.43 | 78.96 | 91.02 | 91.12
FSD19-C $\uparrow$ | 56.16 | 68.84 | 72.96 | 54.95 | 67.86 | 72.92
FSD19-N $\uparrow$ | 38.17 | 53.57 | 49.97 | 38.46 | 54.89 | 49.93
FSD50k $\uparrow$ | 38.03 | 44.41 | 54.62 | 38.24 | 44.66 | 57.20
Vox1 $\uparrow$ | 44.86 | 19.02 | 48.43 | 62.23 | 13.54 | 54.71
VL $\downarrow$ | 19.83 | 92.85 | 26.17 | 19.71 | 93.41 | 20.39
FSC $\uparrow$ | 99.12 | 17.37 | 99.47 | 97.94 | 13.58 | 98.04
SPCV1 $\uparrow$ | 97.62 | 69.28 | 98.27 | 97.75 | 68.76 | 98.30
SPCV2 $\uparrow$ | 96.18 | 49.13 | 96.81 | 96.35 | 42.85 | 96.86
Table 3: Transfer learning results on the proposed downstream tasks. Both
models are either trained from scratch (✐), frozen (❄), where only the
classifier is trained, or finetuned (🔥). All results represent an average of 4
trials for each task (except VL). The best result for each dataset is marked
in bold. $\uparrow$ represents higher is better and $\downarrow$ lower is
better.

Paradigm | Method |  MSoS $\uparrow$ |  NS $\uparrow$ |  US8K $\uparrow$ |  RAV $\uparrow$ |  ESC-50 $\uparrow$ |  FSD18 $\uparrow$ |  FSD19-C $\uparrow$ |  FSD19-N $\uparrow$ |  FSD50k $\uparrow$ |
---|---|---|---|---|---|---|---|---|---|---|---
SSL |  BYOL-A [1] | - | 74.10 | 79.10 | - | - | - | - | - | - |
BYOL-A∗ ❄ | 78.90 | 75.27 | 79.78 | 65.48 | 86.45 | 90.06 | 64.07 | 36.28 | 31.10 |
BYOL-A∗ 🔥 | 80.75 | 80.04 | 78.90 | 65.17 | 82.86 | 89.36 | 63.12 | 38.51 | 40.38 |
COLA [1] | - | 70.20 | 78.50 | - | - | - | - | - | - |
Weak |  WEANET [34] | - | - | - | - | 94.10 | - | 72.80 | 50.30 | - |
CNN14 [3] ❄ | 88.60 | - | - | 39.70 | 90.80 | - | - | - | - |
CNN14 [3] 🔥 | 96.00 | - | - | 72.10 | 94.70 | - | - | - | - |
MBv2 ❄ | 90.40 | 66.50 | 83.27 | 50.90 | 92.20 | 87.31 | 68.84 | 53.57 | 44.41 |
MBv2 🔥 | 93.55 | 79.93 | 84.29 | 66.66 | 93.70 | 94.43 | 72.96 | 49.97 | 54.62 |
Eff-B0 ❄ | 89.90 | 64.65 | 83.00 | 48.68 | 92.10 | 91.02 | 67.86 | 54.89 | 44.66 |
Eff-B0 🔥 | 93.75 | 77.92 | 85.22 | 70.00 | 94.48 | 91.12 | 72.92 | 49.93 | 57.20 |

Table 4: Comparison of the proposed approach against other works in literature
on sound-domain based tasks. Models denoted with ∗ have been finetuned from a
publicly available model. Best results in bold.

###  5.1 Upstream results on Audioset

We provide a performance comparison of our baseline models against current
state-of-the-art (SOTA) methods in Table 2. Standard evaluation on Audioset
includes mean average precision (mAP) as its most significant metric, whereas
mean area under the curve (mAUC) and d-prime (d’) are secondary metrics. Our
two utilized models provide excellent performance, especially when considering
the few parameters.

###  5.2 Transfer learning results

The results of the proposed models for transferring to other tasks can be seen
in Table 3. Results are categorized by their respective domain type (top =
audio, bottom = speech).

As it can be seen, for all sound-related tasks, training a simple classifier
on top of our pre-trained model (❄) enhances performance again training from
scratch (✐). Large performance gains are observed for music/sound
classification (MSoS, NS), audio tagging (FSD, ESC), and acoustic scene (US8k)
datasets. Smaller, but considerable gains are also observed for emotion
detection (RAV).

Unfortunately, regarding most speech-related tasks (Vox1, SPCV1/2), both of
our pre-trained model’s performance largely drops if weights are frozen (❄).
This can largely be explained due to the differences in audio pattern
recognition and speech recognition: The spectral diversity for audio pattern
tasks is much richer than for speech-related tasks. Further, even though the
pretraining dataset, Audioset, contains speech, the corresponding labels are
coarse and have no connection to the content of speech e.g., “Speech” and
“Conversation” are similar concepts of human language, but lack detailed
content information.

Freeing all weights during the finetune process (🔥), performance on SPCV1/2
and FSC datasets improve against the baseline from scratch. This suggests that
audio tagging pre-trained models are capable of enhancing performance on
speech tasks, although the performance gains are limited. Results on the VL
dataset indicate that training from scratch is superior to a pre-trained
alternative, which is likely due to the dataset’s sufficient training size
(6000 h).

###  5.3 Comparison to other approaches

We compare our weakly-supervised approach against the state-of-the-art CNN14
model and the modern self-supervised bootstrap your own audio latent (Byol-A)
[1] method for sound-related downstream tasks in Table 4. We choose the
publicly available BYOL-A model (“64x96d2048”) pre-trained on Audioset with a
comparable parameter size of 5M.

As the results in Table 4 indicate, pretrained SSL embeddings perform well on
most downstream sound-tasks, but are mostly outperformed by weakly supervised
approaches. For the FSD19-N and FSD50k datasets, we observe a significant
performance difference (38.51 vs. 54.89 and 40.38 vs. 57.20) between BYOL-A
and our weakly trained Eff-B0. When comparing our models against CNN14, we
observe that both of our models obtain a superior performance when weights are
frozen (MSoS, RAV, ESC-50), which indicates that higher mAP performance on
Audioset might not transfer to other tasks. Even though CNN14’s fine-tuning
performance is superior to our models, its parameter size of 80 M (see Table
2) slows down the finetuning process drastically, leading to comparatively
long training times.

| Method |  Vox1 $\uparrow$ |  VL $\downarrow$ |  SPCV1 $\uparrow$ |  SPCV2 $\uparrow$ |  FSC $\uparrow$
---|---|---|---|---|---|---
SSL |  BYOL-A [1] | 40.10 | - | - | 92.20 | -
BYOL-A∗ ❄ | 30.53 | 94.10 | 93.11 | 90.67 | 67.10
BYOL-A∗ 🔥 | 57.91 | 34.18 | 97.89 | 96.65 | 97.78
COLA [8] | 30.40 | - | - | 76.70 | -
W2V [2] | 56.56 | - | - | - | 84.92
W2V2.0 [2] | 75.18 | - | - | - | 84.92
Weak |  MBv2 ❄ | 19.02 | 92.85 | 69.28 | 49.13 | 17.37
MBv2 🔥 | 48.43 | 26.17 | 98.27 | 96.81 | 99.47
Eff-B0 ❄ | 13.54 | 93.41 | 68.76 | 42.85 | 13.58
Eff-B0 🔥 | 54.71 | 20.39 | 98.30 | 96.86 | 98.04

Table 5: Comparison of our weakly-supervised approach against Wav2Vec (W2V),
COLA and BYOL-A in literature on speech-domain based tasks. Models denoted
with ∗ have been finetuned from a publicly available model.

Further, results in Table 5 indicate that except for Vox1, finetuned weakly
supervised pre-trained models can outperform SSL approaches by a large margin
on the SPCV1/2 and FSC tasks. Moreover, we observe a large performance gap
between Wav2Vec and all other approaches on the Vox1 dataset, which indicates
that our weakly-supervised pretraining and BYOL-A do not sufficiently process
inter-utterance information, such as speech identity.

##  6 Conclusion

This paper explored an empirical evaluation of pre-trained AT models as
possible candidates for general audio representations. A MobileNetV2 and an
EfficientNet-B0 were used as back-bones for finetuning on 14 datasets split
into two domains. The results indicate that within both sound and speech
domains, AT pre-training can improve performance.

We observe that freezing pretrained parameters for tasks within the sound
domain generally improves performance, while on the contrary models trained
from scratch are preferred over models with frozen parameters. However, when
finetuning on speech datasets, we observe significant gains against training
from scratch and even other self-supervised methods. Given the results of this
work, we provide suggestions regarding transfer learning within speech and
sound domains:

  * •

For sound-domain tasks, AT-pretraining largely benefits performance and should
therefore be preferred.

  * •

AT-pretraining can be used for downstream speech-domain tasks only when
parameter finetuning is possible.

  * •

Self-supervised pretraining (BYOL-A) offers a well-rounded performance across
both speech and sound domains without finetuning. However, additional
finetuning only offers marginal performance improvements for most sound-domain
tasks.

This paper provides empirical evidence that AT pre-trained models can be used
across domains for speech-related tasks and displays the performance gap
between current self-supervised solutions and weakly-supervised ones.

## References

  * [1] Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino,  “Byol for audio: Self-supervised learning for general-purpose audio representation,”  in 2021 International Joint Conference on Neural Networks (IJCNN), 2021, pp. 1–8.
  * [2] Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung yi Lee,  “SUPERB: Speech Processing Universal PERformance Benchmark,”  in Proc. Interspeech 2021, 2021, pp. 1194–1198.
  * [3] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley,  “PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition,”  IEEE/ACM Transactions on Audio Speech and Language Processing, vol. 28, pp. 2880–2894, dec 2020.
  * [4] Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W Schuller, Christian J Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, et al.,  “Hear 2021: Holistic evaluation of audio representations,”  arXiv preprint arXiv:2203.03022, 2022.
  * [5] Yuan Gong, Yu-An Chung, and James Glass,  “AST: Audio Spectrogram Transformer,”  in Proc. Interspeech 2021, 2021, pp. 571–575.
  * [6] Yuan Gong, Yu-An Chung, and James Glass,  “Psla: Improving audio event classification with pretraining, sampling, labeling, and aggregation,”  arXiv preprint arXiv:2102.01243, 2021.
  * [7] Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov,  “Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection,”  in ICASSP 2022.
  * [8] Aaqib Saeed, David Grangier, and Neil Zeghidour,  “Contrastive learning of general-purpose audio representations,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 3875–3879.
  * [9] Luyu Wang and Aaron van den Oord,  “Multi-format contrastive learning of audio representations,”  arXiv preprint arXiv:2103.06508, 2021.
  * [10] Luyu Wang, Pauline Luc, Adrià Recasens, Jean-Baptiste Alayrac, and Aäron van den Oord,  “Multimodal self-supervised learning of general audio representations,”  CoRR, vol. abs/2104.12807, 2021.
  * [11] Xavier Favory, Konstantinos Drossos, Tuomas Virtanen, and Xavier Serra,  “Coala: Co-aligned autoencoders for learning semantically enriched audio representations,”  in International Conference on Machine Learning (ICML), 2020.
  * [12] Marco Tagliasacchi, Beat Gfeller, Félix de Chaumont Quitry, and Dominik Roblek,  “Pre-training audio representations with self-supervision,”  IEEE Signal Processing Letters, vol. 27, pp. 600–604, 2020.
  * [13] Yuan Gong, Cheng-I Jeff Lai, Yu-An Chung, and James Glass,  “Ssast: Self-supervised audio spectrogram transformer,”  arXiv preprint arXiv:2110.09784, 2021.
  * [14] Anurag Kumar, Yun Wang, Vamsi Krishna Ithapu, and Christian Fuegen,  “Do Sound Event Representations Generalize to Other Audio Tasks? A Case Study in Audio Transfer Learning,”  in Proc. Interspeech 2021, 2021, pp. 1214–1218.
  * [15] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter,  “Audio set: An ontology and human-labeled dataset for audio events,”  in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 776–780.
  * [16] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen,  “Mobilenetv2: Inverted residuals and linear bottlenecks,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4510–4520.
  * [17] Mingxing Tan and Quoc Le,  “Efficientnet: Rethinking model scaling for convolutional neural networks,”  in International Conference on Machine Learning. PMLR, 2019, pp. 6105–6114.
  * [18] Justin Salamon, Christopher Jacoby, and Juan Pablo Bello,  “A dataset and taxonomy for urban sound research,”  in Proceedings of the 22nd ACM international conference on Multimedia, 2014, pp. 1041–1044.
  * [19] Eduardo Fonseca, Jordi Pons Puig, Xavier Favory, Frederic Font Corbera, Dmitry Bogdanov, Andres Ferraro, Sergio Oramas, Alastair Porter, and Xavier Serra,  “Freesound datasets: a platform for the creation of open audio datasets,”  in Proceedings of the 18th ISMIR Conference, p. 486-93. International Society for Music Information Retrieval (ISMIR), 2017.
  * [20] Eduardo Fonseca, Manoj Plakal, Frederic Font, Daniel P. W. Ellis, and Xavier Serra,  “Audio tagging with noisy labels and minimal supervision,”  in Submitted to DCASE2019 Workshop, NY, USA, 2019.
  * [21] Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra,  “Fsd50k: an open dataset of human-labeled sound events,”  arXiv preprint arXiv:2010.00475, 2020.
  * [22] Karol J Piczak,  “Esc: Dataset for environmental sound classification,”  in Proceedings of the 23rd ACM international conference on Multimedia, 2015, pp. 1015–1018.
  * [23] Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan,  “Neural audio synthesis of musical notes with wavenet autoencoders,”  in International Conference on Machine Learning. PMLR, 2017, pp. 1068–1077.
  * [24] Christian Kroos, Oliver Bones, Yin Cao, Lara Harris, Philip JB Jackson, William J Davies, Wenwu Wang, Trevor J Cox, and Mark D Plumbley,  “Generalisation in environmental sound classification: the ‘making sense of sounds’ data set and challenge,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 8082–8086.
  * [25] Steven R Livingstone, Katlyn Peck, and Frank A Russo,  “Ravdess: The ryerson audio-visual database of emotional speech and song,”  in Annual meeting of the canadian society for brain, behaviour and cognitive science, 2012, pp. 205–211.
  * [26] Pete Warden,  “Speech commands: A dataset for limited-vocabulary speech recognition,”  arXiv preprint arXiv:1804.03209, 2018.
  * [27] Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio,  “Speech model pre-training for end-to-end spoken language understanding,”  arXiv preprint arXiv:1904.03670, 2019.
  * [28] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman,  “Voxceleb: a large-scale speaker identification dataset,”  arXiv preprint arXiv:1706.08612, 2017.
  * [29] Jörgen Valk and Tanel Alumäe,  “Voxlingua107: a dataset for spoken language recognition,”  in 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 652–658.
  * [30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala,  “PyTorch: An Imperative Style, High-Performance Deep Learning Library,”  in Advances in Neural Information Processing Systems 32, H Wallach, H Larochelle, A Beygelzimer, F Alché-Buc, E Fox, and R Garnett, Eds., pp. 8026–8037. Curran Associates, Inc., 2019.
  * [31] Heinrich Dinkel, Mengyue Wu, and Kai Yu,  “Towards duration robust weakly supervised sound event detection,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 887–900, 2021.
  * [32] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz,  “mixup: Beyond empirical risk minimization,”  in International Conference on Learning Representations, 2018.
  * [33] Daniel S. Park, William Chan, Yu Zhang, Chung Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  in Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. 2019, vol. 2019-September, pp. 2613–2617, International Speech Communication Association.
  * [34] Anurag Kumar and Vamsi Ithapu,  “A sequential self teaching approach for improving generalization in sound event recognition,”  in International Conference on Machine Learning. PMLR, 2020, pp. 5447–5457.