# AutoSpeed: A Linked Autoencoder Approach for Pulse-Echo Speed-of-Sound
Imaging for Medical Ultrasound

Farnaz Khun Jush  Pattern Recognition Lab, Friedrich-Alexander-University,
Erlangen, Germany  Technology Excellence, Siemens Healthcare GmbH, Erlangen,
Germany  Markus Biele  Technology Excellence, Siemens Healthcare GmbH,
Erlangen, Germany  Peter M. Dueppenbecker  Technology Excellence, Siemens
Healthcare GmbH, Erlangen, Germany  Andreas Maier  Pattern Recognition Lab,
Friedrich-Alexander-University, Erlangen, Germany

###### Abstract

Quantitative ultrasound, e.g., speed-of-sound (SoS) in tissues, provides
information about tissue properties that have diagnostic value. Recent studies
showed the possibility of extracting SoS information from pulse-echo
ultrasound raw data (a.k.a. RF data) using deep neural networks that are fully
trained on simulated data. These methods take sensor domain data, i.e., RF
data, as input and train a network in an end-to-end fashion to learn the
implicit mapping between the RF data domain and SoS domain. However, such
networks are prone to overfitting to simulated data which results in poor
performance and instability when tested on measured data. We propose a novel
method for SoS mapping employing learned representations from two linked
autoencoders. We test our approach on simulated and measured data acquired
from human breast mimicking phantoms. We show that SoS mapping is possible
using linked autoencoders. The proposed method has a Mean Absolute Percentage
Error (MAPE) of $2.39\%$ on the simulated data. On the measured data, the
predictions of the proposed method are close to the expected values with MAPE
of $1.1$ $\%$. Compared to an end-to-end trained network, the proposed method
shows higher stability and reproducibility.

_K_ eywords Convolutional Autoencoder $\cdot$ Speed-of-sound Imaging $\cdot$
Representation Learning

##  1 Introduction

B-mode imaging is a qualitative approach and its outcome is highly dependent
on the operator’s expertise. Quantitative values, e.g., speed-of-sound (SoS)
in tissue, can ease interpretation and provide additional information about
tissue properties. In particular, SoS is proven to carry diagnostic
information for differentiating tissue types in breast cancer screening [1, 2,
3].

Prior works attempted to reconstruct SoS from ultrasound pulse-echo data. For
instance, [4] used a passive reflector to record the time of flight of the
signals and reconstructed SoS by solving a limited angle inverse problem.
Although a passive reflector provides an acoustic mirror effect and a timing
reference, placing such a reflector is not feasible in many clinical setups.
[5] used spatial phase shifts of beamformed echoes obtained under varying
steering angles. One of the problems of this approach is that small variations
in the inputs, for instance in the presence of measurement or phase noises can
make the setup unstable [6]. Thus, [6] included an a priori model based on
B-mode image segmentation in a Bayesian framework. This model relies on manual
segmentation of the B-mode images which might not be practical in clinical
setups. In the pulse-echo setup, model-based SoS reconstruction is non-trivial
due to limitations in the number of transmission angles and limited frequency
bandwidths, thus, SoS reconstruction by analytical or optimization approaches
requires carefully chosen regularization and optimization methods, prior
knowledge, and complex fine-tuning [7].

During data acquisition, the sensor encodes an intermediate representation of
the object under examination in the sensor domain. The reconstruction then is
performed by inversion of the corresponding encoding function [8]. However,
the exact inverse function is not available a priori, thus, reconstruction
problems need to approximate the function. Recently, deep neural networks are
vastly being used to solve reconstruction problems. Opposed to analytical and
optimization methods where the inverse function is approximated in multiple
stages of signal processing and/or optimization steps, in deep learning
approaches, the network tries to solve the problem by learning the
corresponding mapping between the sensor domain and reconstruction domain [8].
Therefore, during the training, a low-dimensional representation of the data
in both domains is implicitly learned. This concept was first proposed in [8],
a.k.a. AUTOMAP, to perform a robust reconstruction of multiple sensor-domain
data. The idea is derived from domain transfer approaches, yet, no mechanism
is implemented to actually guarantee the robustness of the solutions found by
such black-box networks. Consequently, these methods are proved to be
unstable, especially when the test data has perturbations or structural
differences compared to training data [9].

Similar techniques are employed for SoS reconstruction from ultrasound echo
data. [10, 11, 12, 13, 14, 15, 16] investigated encoder-decoder networks with
multiple or single steering angles for SoS reconstruction. In these studies,
an encoder-decoder network takes sensor domain data as input and directly
reconstructs the SoS map in the output. Since there is no known gold standard
method to create SoS GT from measured pulse-echo ultrasound data, all the
investigated methods rely on simulated data. As such, transfer of these
methods to clinical setups is challenging and their robustness is still under
debate [10, 15]. These methods are prone to overfitting the distribution of
the simulated data and thus perform unsatisfactorily when tested on real data
[15].

In this study, we propose a novel approach for SoS mapping from a single
plane-wave acquisition. Instead of the end-to-end encoder-decoder approaches
previously proposed, where an implicit mapping between sensor domain and SoS
domain was learned, following the known operator paradigm [17], we propose a
method to break the problem into two steps: In the training phase, firstly, we
encode the sensor domain and the SoS domain data in an intermediate low-
dimensional representation using two linked autoencoders; secondly, we find a
mapping between two representations. For inference, without any further
training, these two steps are joined to create a network that takes the RF
data as input and returns the SoS map in the output. An overview is
demonstrated in Fig. 1. We then compare the proposed method with a baseline
encoder-decoder network both on simulated and measured data acquired from
human breast mimicking phantom.

(a)  (b)

Figure 1: Overview of the proposed method: $X$ being RF domain, $Y$ being SoS
domain, $f$ encoder and $g$ decoder
(a) Training: firstly, two domain are encoded to an intermediate low-
dimensional representation by training two linked autoencoders jointly:
$X\rightarrow f_{1}(X)=M\rightarrow g_{1}(f_{1}(X))=X^{\prime}\approx X$ and
$Y\rightarrow f_{2}(Y)=N\rightarrow g_{2}(f_{2}(Y))=Y^{\prime}\approx Y$ ;
Secondly, a mapping is performed between two representations during the
training ($M\rightarrow f_{3}(M)=M^{\prime}\approx N$) and in an additional
fine-tuning step ($M\rightarrow f_{4}(M)=M^{\prime}\approx N$);
(b) Inference: without further training, two steps are joined to create a
network that maps RF domain to SoS domain: $X\rightarrow f_{1}(X)=M\rightarrow
f_{4}(f_{1}(X))=M^{\prime}\rightarrow g_{2}(f_{4}(f_{1}(X)))=Y^{\prime}$.

##  2 Methods

###  2.1 Network Architecture, Training

Autoencoders [18, 19, 20] are vastly being used to learn low-dimensional
latent representations. An autoencoder is a neural network that maps its input
to itself utilizing an intermediate representation, the so-called latent
space. Autoencoders thus learn to map input data to the latent space, and
latent space data back to the input domain [21, 22, 23]. Hence, to learn the
efficient representation from two domains, i.e., sensor domain (representing
RF data) and SoS domain, autoencoders can be employed. Convolutional
autoencoders combine local convolution connections with autoencoders, thus,
preserving spatial locality [21].

As such, to extract an efficient representation from two domains, firstly two
linked convolutional autoencoders are trained jointly. Secondly, the encoded
latent spaces are extracted and a final step of training is performed to fine-
tune the mapping of RF data and SoS map latent spaces.

####  2.1.1 Linked Autoencoders

Two convolutional autoencoders that are connected via their latent spaces are
trained to extract robust features of RF data and SoS map. During training
efficient representations of the SoS domain and RF domain are encoded in the
latent spaces of the autoencoders. However, these representations can have
different interpretations. To bridge the gap between the aforementioned
representations, they are connected via a fully connected layer that is
trained simultaneously with two autoencoders. This way, the autoencoders are
optimized in a manner that the latent spaces of two networks indicate the
closest possible representations. Fig. 2 (a) demonstrates the model
architecture.

The autoencoders take the input vectors of RF and SoS domain as $x$ and $y$,
respectively, and map them to the hidden representations $m$ and $n$, where
$m=f_{\theta_{1}}(x)=\xi(x\ast W_{1}+b_{1})$ and
$n=f_{\theta_{2}}(y)=\xi(y\ast W_{2}+b_{2})$ are parameterized by
$\theta_{1}=\\{W_{1},b_{1}\\}$ and $\theta_{2}=\\{W_{2},b_{2}\\}$,
respectively. $W_{1,2}=\\{w_{j},j=1,2,..,k\\}$, where $w_{j}$ is the weight of
convolution kernel $j$ and $k$ is the kernel size [19, 22]. $b_{1}$ and
$b_{2}$ are biases. $f_{\theta}$ is the encoder. $\xi$ is the activation
function [24].

The encoded representations, $m$ and $n$, are then used to reconstruct
$x^{\prime}$ and $y^{\prime}$, where
$x^{\prime}=g_{\theta^{\prime}_{1}}(m)=\xi(m\ast
W^{\prime}_{1}+b^{\prime}_{1})$ and
$y^{\prime}=g_{\theta^{\prime}_{2}}(n)=\xi(n\ast
W^{\prime}_{2}+b^{\prime}_{2})$ are similarly parametrized by
$\theta^{\prime}_{1}=\\{W^{\prime}_{1},b^{\prime}_{1}\\}$ and
$\theta^{\prime}_{2}=\\{W^{\prime}_{2},b^{\prime}_{2}\\}$, respectively.
$b^{\prime}_{1}$ and $b^{\prime}_{2}$ are biases.
$W^{\prime}_{1,2}=\\{w^{\prime}_{j},j=1,2,..,k\\}$, where $w^{\prime}_{j}$ is
the weight of convolution kernel $j$ and $k$ is the kernel size [19, 22].
$g_{\theta^{\prime}}$ is the decoder.

Each training $x^{(i)}$ and $y^{(i)}$ is thus mapped to a corresponding
$m^{(i)}$ and $n^{(i)}$ and reconstructs $x^{\prime(i)}$ and $y^{\prime(i)}$.

Simultaneously, the latent spaces are being optimized to match: $m$ is mapped
to $m^{\prime}$ via a fully connected layer,
$m^{\prime}=f_{\theta_{3}}(m)=\xi(mW_{3}+b_{3})$, $\theta_{3}$ is
parameterized by $W_{3}$, the weight, and $b_{3}$, the bias. During the
training $m^{\prime},n$ are being optimized in the loss function to match
closely, where $m^{\prime}=f_{\theta_{3}}(f_{\theta_{1}}(x))$ and
$n=f_{\theta_{2}}(y)$. The parameters of the model are optimized to minimize
the following cost function:

$\displaystyle
J_{LAE}=\frac{1}{n}\sum_{i=1}^{N}L(x^{(i)},x^{\prime(i)})+L(y^{(i)},y^{\prime(i)})+L(m^{\prime(i)},n^{(i)})$
(1)
$\displaystyle=\frac{1}{n}\sum_{i=1}^{N}L(x^{(i)},g_{\theta^{\prime}_{1}}(f_{\theta_{1}}(x^{(i)}))+L(y^{(i)},g_{\theta^{\prime}_{2}}(f_{\theta_{2}}(y^{(i)}))$
$\displaystyle+L(f_{\theta_{3}}((f_{\theta_{1}}(x^{(i)})),f_{\theta_{2}}(y^{(i)}))$

where L is the loss function, here the Mean Squared Error (MSE),
$L(x,x^{\prime})=\leavevmode\nobreak\ ||x-x^{\prime}||^{2}$, is used. Note
that for simplification, vector representation is used in the text but Fig. 2
shows matrix representation.

(a)

(b)

Figure 2: Network Architecture, Training: (a): Linked Autoencoders: two
convolutional autoencoders connected via their latent spaces, the linked
autoencoders are trained jointly to encode two domains where the intermediate
representation match as closely as possible. One autoencoder takes the SoS map
as input and the other takes RF data as input. The RF data is a matrix of size
$192\times 2048$ ($192$ channels, $2048$ samples with a sampling frequency of
$40$ $MHz$). The SoS map is a matrix of size $384\times 384$ which translates
to $3.84$ $cm$ in lateral and axial directions. Operations in each layer are
color-coded. (b): IRM-Layer: the intermediate representation from the RF
domain is mapped to the intermediate representation from the SoS domain via a
fully connected layer. The color blocks show the operations in each layer. The
network is implemented using Tensorflow 2.2 [25].

####  2.1.2 Fine-tuning (IRM-Layer)

After convergence of the linked autoencoders, the latent spaces of two
autoencoders are extracted. To fine-tune the mapping of the latent spaces, a
fully connected layer which will be referred to as the Intermediate
Representation Mapper Layer (IRM-Layer) is trained separately. This layer
takes the latent space of the RF autoencoder as input ($n$) and the latent
space of the SoS autoencoder as output ($m$) similar to the mapping of latent
spaces during linked autoencoder training.

Thus, $m$ is mapped to $m^{\prime}$, where
$m^{\prime}=f_{\theta_{4}}(m)=\xi(mW_{4}+b_{4})$. $\theta_{4}$ is
parameterized by $W_{4}$, the weight and $b_{4}$ the bias. The following cost
function is optimized:

$J_{IRM}=\frac{1}{n}\sum_{i=1}^{N}L(n,m^{\prime})=\frac{1}{n}\sum_{i=1}^{N}L(n,f_{\theta_{4}}(m))$
(2)

IRM-Layer represents a mapping function between two latent spaces $m$ and $n$.
Fig. 2 (b) shows the schematic.

###  2.2 Network Architecture, Inference

For the inference, we propose a network henceforth will be referred to as
AutoSpeed. AutoSpeed is an encoder-decoder network that its encoder is
detached from the trained RF autoencoder and its decoder is detached from the
trained SoS autoencoder. Extracted encoder and decoder paths are connected via
the trained IRM-Layer. Note that no retraining step is performed to employ the
AutoSpeed, the layers are simply taken from the trained linked networks and
connected via the trained IRM-Layer.

In the encoder, $x$ is mapped to $m$, where
$m=f_{\theta_{1}}(x)=\xi(x*W_{1}+b_{1})$. $b_{1}$ and $W_{1}$ are bias and
weights of the encoder of the RF autoencoder. The IRM-Layer with weights and
bias $W_{4}$ and $b_{4}$ is then being used to map $m$ to $m^{\prime}$:
$m^{\prime}=f_{\theta_{4}}(m)=\xi(mW_{4}+b_{4})$.

In the decoder, $m^{\prime}$ is mapped to
$y^{\prime}=g_{\theta^{\prime}_{2}}(m^{\prime})=\xi(m^{\prime}*W_{2}+b_{2})$,
where $W_{2}$ and $b_{2}$ are the weights and biases are taken from decoder of
the SoS autoencoder. Fig. 3 shows the schematic of AutoSpeed.

Figure 3: Network Architecture, Inference (AutoSpeed): The network consists of
an encoding path that is extracted from the encoding path of the trained RF
Autoencoder and a decoding path that is extracted from the trained SoS
Autoencoder, two paths are connected via a fully connected layer, i.e., IRM-
Layer, which is trained during the fine-tuning step. No further training is
required to employ AutoSpeed. AutoSpeed takes RF data as input and returns the
SoS map in the output.

##  3 Results

###  3.1 Dataset

For SoS reconstruction, acquiring sufficient measured data alongside its
corresponding GT for training a deep neural network is challenging because
there is no known gold-standard method capable of creating exact GT for
reflection data (pulse-echo ultrasound) and there are only a few phantoms
available with known heterogeneous SoS. Therefore, for deep learning-based
approaches using simulated data for training is a common practice [7, 10, 11,
12, 13, 14, 15, 16]. K-Wave toolbox [26] (Version 1.3) is used for the
simulation of training data. K-Wave is a MATLAB toolbox that allows the
simulation of linear and non-linear wave propagation with arbitrary
heterogeneous parameters [26].

The LightABVS system that has been previously introduced for pulse-echo
ultrasound imaging applications is used for data acquisition [27]. LightABVS
system consists of a linear probe with 192 active channels, a pitch of
$200\leavevmode\nobreak\ \mu m$, and a center frequency of $5$ $MHz$. The size
of the medium is considered to be $3.8\leavevmode\nobreak\ cm$ in depth and
$7.6\leavevmode\nobreak\ cm$ in the lateral direction (twice the size of the
probe head) and the probe head is placed above the central section of the
medium. A single plane-wave with zero degrees is used for the simulations [11,
13, 15]. The medium is simulated on a 2D grid of size $1536\times 3072$.

The simulation setup is based on the Combined setup proposed by [15], where a
joint set of datasets including both geometrical and natural tissue models are
simulated for training data generation:

  * •

Ellipsoids: The geometrical mediums are based on a simplified model for organs
proposed by [10] and adopted for breast use-case in [13, 15]. This setup
considers a medium with homogeneous background and embeds multiple ellipsoids
in random places inside the homogeneous background. Random SoS values in the
range $[1300-1700]\leavevmode\nobreak\ m/s$ are assigned to the medium
structures in the SoS domain.

  * •

T2US: The natural tissue structures are extracted from Tomosynthesis images.
This setup is based on the T2US setup proposed in [15]. Random patches from
Tomosynthesis images with benign or malignant lesions are used to create SoS
maps with SoS values in the range $[1300-1700]\leavevmode\nobreak\ m/s$.

The attenuation value is set to $0.75$ $dB/MHz.cm$ and the mass density of
$1020$ $kg/m^{3}$ based on the property of breast tissue [28], Table B.1.
Further details of this setup can be found in [15].

###  3.2 Training

####  3.2.1 Linked Autoencoder

The linked autoencoder is trained for 200 epochs with SGD optimizer with a
batch size of $8$ on an Nvidia RTX Quadro 8000 GPU. During training, $6000$
training cases were used ($4800$ training, $1200$ validation). The test set
contains 150 cases. The size of the dataset is chosen based on [10, 13, 15].

RF Autoencoder: Over 10 training runs, the RF autoencoder converges to an
average Root Mean Squared Error (RMSE) of $6.06\pm 0.31$ (SoS$\pm$SD where SD
is the standard deviation) and Mean Absolute Error (MAE) of $3.94\pm 0.29$,
where the input RF data is in the range $[-1024,1024]$. It is noteworthy that
during training experiments we observed that normalization hinders the
convergence of RF Autoencoder, thus, the data is not normalized.

SoS Autoencoder: The SoS autoencoder converges to RMSE of $7.89\pm 3.23$ $m/s$
and MAE of $6.68\pm 3.03$ $m/s$, where the input range is $[1300,1700]$ $m/s$.

Intermediate Layer: The intermediate representation converges to RMSE of
$1.14\pm 0.28$ and MAE of $0.93\pm 0.36$, where the output of mid-layers is in
the range $[-15,30]$.

####  3.2.2 Fine-tuning (IRM-Layer)

The IRM-layer is trained for 150 epochs using the Adam optimizer [29], where
it converges to RMSE of $0.68\pm 0.28$ and MAE of $0.55\pm 0.19$ over 10
training runs.

###  3.3 Inference

####  3.3.1 Simulated data

On the test dataset, AutoSpeed has an average RMSE of $47.98\pm 4.15$ $m/s$,
MAE of $37.26\pm 3.56$ $m/s$ and and MAPE of $2.39\pm 0.22\%$ over 10 training
runs.

We set up a network based on [15] and compared the results of AutoSpeed with
this network. Hereafter, we will refer to the baseline network as En-De-Net.
The En-De-Net converges to RMSE of $23.64\pm 0.7$ $m/s$, MAE $17.32\pm 0.7$
$m/s$, and MAPE of $1.11\pm 0.05\%$. The error ranges of both networks are in
the typical range of SoS errors $(1-2\%)$ [30, 16, 10, 15]. However, on the
simulated data the En-De-Net has a lower error rate.

####  3.3.2 Measured data

Single Frames: We acquired ultrasound RF data for a CIRS multi-modality breast
phantom with the LightABVS system [27]. $30$ cases with visible dense masses
are acquired, where masses are placed in different locations inside the region
of interest (ROI). The phantom contains an anechoic skin-mimicking layer and
hyperechoic dense masses embedded in homogeneous background. Because we lack
measured SoS ground truth data, we compare to the manufacturer’s
specifications. The predicted SoS in the background region and dense lesions
region is in the range specified by the manufacturer in Table 1, Expected
value. Each inclusion is manually segmented based on the B-mode image to
compare SoS values predicted inside the inclusion and in the background by two
networks. Since the simulation model does not include anechoic tissue
characteristics, the skin-mimicking layer is excluded in the error computation
in Table 1.

Fig. 4 shows predicted SoS values inside the inclusion and in the background
for 30 cases compared to the expected value. For 30 cases, the mean predicted
SoS value in the background is $1535$ $m/s$ with SD of $6$ $m/s$ and $1527$
$m/s$ with SD of $19$ $m/s$ for AutoSpeed and En-De-Net, respectively. The
mean value of predicted SoS inside the inclusion for AutoSpeed is $1561$ $m/s$
with SD $11$ $m/s$ and for En-De-Net is $1545$ $m/s$ with SD of $43$ $m/s$
(Table 1). Fig. 5 shows 8 sample cases of this dataset. Quantitatively, the
mean SoS values inside the inclusion and in the background are close to the
expected values (Table 1). However, En-De-Net shows higher SD. The high SD for
En-De-Net means that this network in some cases suffers from inverted SoS
values, fails to find the inclusion and/or the predicted SoS values for the
same kind of inclusion varies among shots. This effect can be seen in Fig. 5,
for example, in cases #4, #5, #6, and #7 which are very similar cases, En-De-
Net shows varying SoS values in the inclusion region.

Table 1: Mean SoS values in the background and inside dense inclusion acquired
from CIRS multi-modality breast phantom, $\pm$ shows the SD value. The
predicted values are computed on $30$ cases, all with inclusion in different
locations.

| Background ($m/s$) | Inclusion ($m/s$)
---|---|---
Expected value | $1520\pm 10$ | $1580\pm 20$
AutoSpeed | $1535\pm 6$ | $1561\pm 11$
En-De-Net | $1527\pm 19$ | $1545\pm 45$

Figure 4: Predicted SoS inside dense inclusion and in the background for 30
measured phantom cases.

Qualitatively, both networks in most cases can find the inclusion area with
SoS contrast ($76\%$ and $90\%$ for En-Co-Net and AutoSpeed, respectively, for
30 cases). However, if En-De-Net finds the area with SoS contrast often the
predictions have clearer margins. AutoSpeed shows more stable SoS predictions
inside the inclusion area but in some cases shows SoS contrast in regions
outside of the inclusion area, we noticed that these regions are the regions
with high reflections compared to the background, for example, where highly
reflective small structures are present. Therefore, AutoSpeed is sensitive to
these kinds of reflective scatterers. On the other hand, it is more likely
that the predictions from one shot to another shot vary for the En-De-Net,
meaning the predicted values inside the same inclusion contrary to expectation
changes in different locations, e.g., Fig. 5 case #2 vs. case #3 or even in
the very similar filed of views e.g., Fig. 5 case #5 and #5 vs. case #7. The
following section shows the extreme case of this phenomenon, where multiple
shots of the exact field of view with mechanically fixed transducer and
phantom are investigated.

| #1 | #2 | #3 | #4 | #5 | #6 | #7 | #8 |  |
---|---|---|---|---|---|---|---|---|---|---
B-mode |  |  |  |  |  |  |  |  |  |
En-De-Net SoS Map |  |  |  |  |  |  |  |  |  |
En-De-Net SoS, B-mode Overlay |  |  |  |  |  |  |  |  |  |
AutoSpeed SoS Map |  |  |  |  |  |  |  |  |  |
AutoSpeed SoS, B-mode Overlay |  |  |  |  |  |  |  |  |  |
Figure 5: Predicted SoS maps by En-De-Net and AutoSpeed for 8 cases where a
dense inclusion is present in the B-mode image. The inclusion is marked by red
arrows in B-mode images.

Multiple Frames: Here, we perform an experiment to demonstrate the stability
and reproducibility of AutoSpeed predictions compared to En-De-Net when tested
on multiple shots of the same field of view. We acquired 200 frames from the
same field of view when the probe head and phantom is mechanically fixed.
Since the field of view is the same, the expectation is that the networks
predict consistent SoS values inside the inclusion and in the background.
Additionally, we expect that the SoS values inside the inclusion have low SD.
Fig. 6 shows 4 cases of the aforementioned dataset.

Although both networks follow the geometry in the corresponding b-mode images,
could find the inclusion and the skin layer and background margin correctly,
AutoSpeed shows more consistency. In Fig. 6, Frame 1 green arrows in the En-
De-Net prediction image show the inconsistent regions that appear differently
from frame to frame. For example, on the left side of the image, the SoS
values are inverted from one frame to another frame. On the right side of the
image, there is a region detected with SoS contrast, but the margin and the
SoS values in the corresponding region vary. The same phenomenon can be seen
in the inclusion area in the middle. Whereas, the predictions of the AutoSpeed
network are consistent throughout all the frames.

| Frame 1 | Frame 2 | Frame 3 | Frame 4 |
---|---|---|---|---|---
B-mode |  |  |  |  |
Absolute Difference B-mode |  |  |  |  |
En-De-Net SoS Map |  |  |  |  |
AutoSpeed SoS Map |  |  |  |  |
Figure 6: 1st row: reconstructed b-mode images of 4 consecutive frames, 2nd row: absolute difference matrices of each frame with its previous frame (frame 0 is not shown here), 3rd row: predicted SoS for the consecutive shots for En-De-Net, 4th row: predicted SoS for AutoSpeed Network. Predicted SoS maps by AutoSpeed are more consistent for the same field of view compared to the predictions of En-De-Net. Inside Inclusion | Background
---|---
|
(a) | (b)
|
(c) | (d)
Figure 7: AutoSpeed vs. En-De-Net, over 200 consecutive frames from the same
field of view: (a): Comparison of average predicted SoS values inside the
inclusion, (b): Comparison of average predicted SoS values in the background,
(c): Comparison of the standard deviation of the predictions inside the
inclusion, (d): Comparison of the standard deviation of the predictions in the
background; average predicted SoS values by AutoSpeed both inside the
inclusion and in the background is more consistent through all the frames
compared to En-De-Net.

Fig. 7 compares the mean SoS values inside the inclusion and in the background
region for two networks and the standard deviations of predicted values inside
the inclusion area and in the background for each frame. Based on Fig. 7 (a)
and (b), AutoSpeed has highly consistent predictions both inside the inclusion
and in the background in consecutive frames of the same field of view.
Additionally, Fig. 7 (c) and (d) show that the SoS values in each frame inside
the inclusion and in the background have consistent uniformity for AutoSpeed
compared to En-De-Net. Thus, we can conclude that AutoSpeed can extract more
stable features compared to En-De-Net which performs an end-to-end mapping and
lower error rates on the simulated data do not necessarily translate to the
measured data.

##  4 Discussion and Conclusion

SoS reconstruction for pulse-echo ultrasound can be advantageous in clinical
studies, because it employs the available data acquisition setups used for
B-mode imaging and it provides quantitative measures that can be used to
distinguish different tissue types. Yet, it is still in its infancy stage and
there is no accepted gold standard approach for pulse-echo SoS reconstruction.
Analytical and optimization-based methods require prior and carefully chosen
regularization parameters and the proposed methods are often either not stable
and/or not feasible for real-time applications. In recent years, deep learning
techniques are widely used to solve inverse problems. These methods often
perform an par or even in cases outperform model-based methods. Additionally,
they move the computational burden to training phase, thus, in inference, they
can perform in real-time. But they need training data often alongside their
labels or GT maps. Hence, the lack of a gold standard method for SoS
reconstruction poses a challenge to develop deep learning techniques as well.
Because it is practically impossible to create sufficient GT data for deep
learning algorithms. Simulation toolboxes offer an alternative solution to
create training data. However, there is no guarantee that the networks trained
on simulated data are transferable to real data. As a result, incorporating
prior knowledge can increase the chance of success of such networks in real
data setups.

In this study, we proposed a novel approach for pulse-echo SoS imaging from a
single plane-wave acquisition. Inspired by the known parameter paradigm [17,
31], we hardcoded the domain transfer problem in a dual autoencoder approach,
similar to the idea behind Cycle-GAN [32]. We trained two linked autoencoders
to extract efficient representations from RF data and SoS maps. In a fine-
tuning step, a fully connected layer mapped the representation from the RF
data domain to the SoS domain. There is evidence that the functions learned by
networks can be decomposed into modules and those modules can be used in other
tasks even without any further training [31, 33, 34]. Thus, in the inference,
an encoder-decoder-like architecture is proposed in which the encoder is
detached from the RF data autoencoder and the decoder is detached from the SoS
autoencoder (both fully trained) and two paths are connected via a trained
fully connected layer (IRM-Layer). We tested the method on simulated and
measured data and made a comparison with an encoder-decoder network that is
trained end-to-end. We showed that SoS mapping is possible employing such a
setup and is more stable compared to the end-to-end mapping solution
previously proposed.

On the simulated data, the proposed network, i.e., AutoSpeed, reconstructs the
SoS maps with $37.26\pm 3.56$ $m/s$ MAE is in the typical range of SoS error
rates but the error rates are higher compared to the network that is trained
end-to-end, i.e., En-De-Net which is $17.32\pm 0.7$ $m/s$. [35] demonstrated
that breast fatty tissue and parenchyma have mean SoS values of $1422\pm 9$
$m/s$ and $1487\pm 21$ $m/s$, respectively. The SoS values for lesions is on
average higher, e.g., benign lesions have a mean SoS value of $1513\pm 27$
$m/s$ and for malignant lesions, the average SoS is $1548\pm 17$ $m/s$.
Another study [36] showed that the presence of breast lesions results in focal
increments $\Delta$SoS in comparison with the background tissue. Malignant
lesions show significantly higher $\Delta$SoS in comparison with begin lesoins
$\Delta SoS>41.64$ $m/s$. [36] reported $\Delta$SoS in range $[14-118]$ $m/s$
for malignant and $[7-41]$ $m/s$ for benign lesions. Therefore, theoretically,
the predicted values are relevant for clinical use cases.

Our experiments on the measured data demonstrated that outperforming on the
simulated data does not necessarily translate to the measured data setup and
the networks that are trained end-to-end can be prone to overfitting to the
distribution of simulated data. Whereas, our proposed method that employs the
learned representations from autoencoders is more stable, can extract
efficient features, and outperforms the previously proposed method in terms of
reproducibility. On the measured data setup, AutoSpeed could detect the
inclusions and correct margins, and the predicted SoS maps are close to the
expected range.

Initial results on phantom data are highly encouraging. AutoSpeed on the
measured data predicted SoS values in range $1535\pm 6$ $m/s$ and $1561\pm 11$
$m/s$ in the background region and inside the inclusion area, respectively,
where the corresponding expected values are $1520\pm 10$ $m/s$ and $1580\pm
20$ $m/s$. Nevertheless, further research is required to improve the results,
e.g., remove artifacts related to highly reflective scatterers, for instance,
by including similar structures in the tissue modeling, and improve training
to create cleaner margins, for example, by using multi-scale approaches.
Additionally, more research is required to transfer and prove the efficiency
of such methods in clinical setups.

## Disclaimer

The information in this paper is based on research results that are not
commercially available.

## Acknowledgments

We thank Prof. Michael Golatta and the university hospital of Heidelberg for
providing the Tomosynthesis dataset.

## References

  * [1] Mark Sak, Neb Duric, Peter Littrup, Lisa Bey-Knight, Haythem Ali, Patricia Vallieres, Mark E Sherman, and Gretchen L Gierach.  Using speed of sound imaging to characterize breast density.  Ultrasound in medicine & biology, 43(1):91–103, 2017.
  * [2] Sergio J Sanabria, Orcun Goksel, Katharina Martini, Serafino Forte, Thomas Frauenfelder, Rahel A Kubik-Huch, and Marga B Rominger.  Breast-density assessment with hand-held ultrasound: A novel biomarker to assess breast cancer risk and to tailor screening?  European radiology, 28(8):3165–3175, 2018.
  * [3] JS Schreiman, JJ Gisvold, James F Greenleaf, and RC Bahn.  Ultrasound transmission computed tomography of the breast.  Radiology, 150(2):523–530, 1984.
  * [4] Sergio J Sanabria, Marga B Rominger, and Orcun Goksel.  Speed-of-sound imaging based on reflector delineation.  IEEE Transactions on Biomedical Engineering, 66(7):1949–1962, 2018\.
  * [5] Patrick Stähli, Maju Kuriakose, Martin Frenz, and Michael Jaeger.  Improved forward model for quantitative pulse-echo speed-of-sound imaging.  Ultrasonics, 108:106168, 2020.
  * [6] Patrick Stähli, Martin Frenz, and Michael Jaeger.  Bayesian approach for a robust speed-of-sound reconstruction using pulse-echo ultrasound.  IEEE transactions on medical imaging, 40(2):457–467, 2020.
  * [7] Valery Vishnevskiy, Sergio J Sanabria, and Orcun Goksel.  Image reconstruction via variational network for real-time hand-held sound-speed imaging.  In International workshop on machine learning for medical image reconstruction, pages 120–128. Springer, 2018.
  * [8] Bo Zhu, Jeremiah Z Liu, Stephen F Cauley, Bruce R Rosen, and Matthew S Rosen.  Image reconstruction by domain-transform manifold learning.  Nature, 555(7697):487–492, 2018.
  * [9] Vegard Antun, Francesco Renna, Clarice Poon, Ben Adcock, and Anders C Hansen.  On instabilities of deep learning in image reconstruction and the potential costs of ai.  Proceedings of the National Academy of Sciences, 117(48):30088–30095, 2020.
  * [10] Micha Feigin, Daniel Freedman, and Brian W Anthony.  A deep learning framework for single-sided sound speed inversion in medical ultrasound.  IEEE Transactions on Biomedical Engineering, 67(4):1142–1151, 2019\.
  * [11] Micha Feigin, Manuel Zwecker, Daniel Freedman, and Brian W Anthony.  Detecting muscle activation using ultrasound speed of sound inversion with deep learning.  In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 2092–2095. IEEE, 2020\.
  * [12] Marvin Heller and Georg Schmitz.  Deep learning-based speed-of-sound reconstruction for single-sided pulse-echo ultrasound using a coherency measure as input feature.  In 2021 IEEE International Ultrasonics Symposium (IUS), pages 1–4. IEEE, 2021.
  * [13] Farnaz Khun Jush, Markus Biele, Peter Michael Dueppenbecker, Oliver Schmidt, and Andreas Maier.  Dnn-based speed-of-sound reconstruction for automated breast ultrasound.  In 2020 IEEE International Ultrasonics Symposium (IUS), pages 1–7. IEEE, 2020.
  * [14] Farnaz Khun Jush, Peter Michael Dueppenbecker, and Andreas Maier.  Data-driven speed-of-sound reconstruction for medical ultrasound: Impacts of training data format and imperfections on convergence.  In Annual Conference on Medical Image Understanding and Analysis, pages 140–150. Springer, 2021.
  * [15] Farnaz Khun Jush, Markus Biele, Peter M. Dueppenbecker, and Andreas Maier.  Deep learning for ultrasound speed-of-sound reconstruction: Impacts of training data diversity on stability and robustness, 2022.
  * [16] SeokHwan Oh, Myeong-Gee Kim, Youngmin Kim, Hyuksool Kwon, and Hyeon-Min Bae.  A neural framework for multi-variable lesion quantification through b-mode style transfer.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 222–231. Springer, 2021.
  * [17] Andreas K Maier, Christopher Syben, Bernhard Stimpel, Tobias Würfl, Mathis Hoffmann, Frank Schebesch, Weilin Fu, Leonid Mill, Lasse Kling, and Silke Christiansen.  Learning with known operators reduces maximum error bounds.  Nature machine intelligence, 1(8):373–380, 2019.
  * [18] Yoshua Bengio.  Learning deep architectures for AI.  Now Publishers Inc, 2009.
  * [19] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  * [20] Andreas Maier, Christopher Syben, Tobias Lasser, and Christian Riess.  A gentle introduction to deep learning in medical image processing.  Zeitschrift für Medizinische Physik, 29(2):86–101, 2019.
  * [21] Min Chen, Xiaobo Shi, Yin Zhang, Di Wu, and Mohsen Guizani.  Deep features learning for medical image analysis with convolutional autoencoder neural network.  IEEE Transactions on Big Data, 2017.
  * [22] Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber.  Stacked convolutional auto-encoders for hierarchical feature extraction.  In International conference on artificial neural networks, pages 52–59. Springer, 2011.
  * [23] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.  Extracting and composing robust features with denoising autoencoders.  In Proceedings of the 25th international conference on Machine learning, pages 1096–1103, 2008.
  * [24] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li.  Empirical evaluation of rectified activations in convolutional network.  arXiv preprint arXiv:1505.00853, 2015.
  * [25] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  Software available from tensorflow.org.
  * [26] Bradley E Treeby and Benjamin T Cox.  k-wave: Matlab toolbox for the simulation and reconstruction of photoacoustic wave fields.  Journal of biomedical optics, 15(2):021314, 2010.
  * [27] Pascal A Hager, Farnaz Kuhn Jush, Markus Biele, Peter M Düppenbecker, Oliver Schmidt, and Luca Benini.  Lightabvs: A digital ultrasound transducer for multi-modality automated breast volume scanning.  In 2019 IEEE International Ultrasonics Symposium (IUS), 2019.
  * [28] Thomas L Szabo.  Diagnostic ultrasound imaging: inside out.  Academic Press, 2004.
  * [29] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [30] Christopher Rowland Hill, Jeff C Bamber, and Gail R ter Haar.  Physical principles of medical ultrasonics, 2004.
  * [31] Andreas Maier, Harald Köstler, Marco Heisig, Patrick Krauss, and Seung Hee Yang.  Known operator learning and hybrid machine learning in medical imaging—a review of the past, the present, and the future.  Progress in Biomedical Engineering, 2022.
  * [32] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
  * [33] Weilin Fu, Katharina Breininger, Roman Schaffert, Nishant Ravikumar, and Andreas Maier.  A divide-and-conquer approach towards understanding deep networks.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 183–191. Springer, 2019.
  * [34] Weilin Fu, Lennart Husvogt, Stefan Ploner, James G Fujimoto, and Andreas Maier.  Modularization of deep networks allows cross-modality reuse.  In Bildverarbeitung für die Medizin 2020, pages 274–279. Springer, 2020.
  * [35] Cuiping Li, Nebojsa Duric, Peter Littrup, and Lianjie Huang.  In vivo breast sound-speed imaging with ultrasound tomography.  Ultrasound in medicine & biology, 35(10):1615–1628, 2009.
  * [36] Lisa Ruby, Sergio J Sanabria, Katharina Martini, Konstantin J Dedes, Denise Vorburger, Ece Oezkan, Thomas Frauenfelder, Orcun Goksel, and Marga B Rominger.  Breast cancer assessment with pulse-echo speed of sound ultrasound from intrinsic tissue reflections: Proof-of-concept.  Investigative radiology, 54(7):419–427, 2019.