11institutetext: Laboratory for Applications of Synchrotron Radiation (LAS),
KIT, Germany 22institutetext: Institute for Photon Science and Synchrotron
Radiation (IPS), KIT, Germany 33institutetext: Engineering Mathematics and
Computing Lab (EMCL), Heidelberg University, Germany

# Optimizing the Procedure of CT Segmentation Labeling

Yaroslav Zharov 1133 Tilo Baumbach 1122 Vincent Heuveline 33

###### Abstract

In Computed Tomography, machine learning is often used for automated data
processing. However, increasing model complexity is accompanied by
increasingly large volume datasets, which in turn increases the cost of model
training. Unlike most work that mitigates this by advancing model
architectures and training algorithms, we consider the annotation procedure
and its effect on the model performance. We assume three main virtues of a
good dataset collected for a model training to be _label quality_ ,
_diversity_ , and _completeness_. We compare the effects of those virtues on
the model performance using open medical CT datasets and conclude, that
_quality_ is more important than _diversity_ early during labeling; the
_diversity_ , in turn, is more important than _completeness_. Based on this
conclusion and additional experiments, we propose a labeling procedure for the
segmentation of tomographic images to minimize efforts spent on labeling while
maximizing the model performance.

###### Keywords:

Segmentation Computed Tomography Labeling.

##  1 Introduction

Neural Networks for computer vision has achieved results on par with human
experts [2]. However, these models require large datasets to be trained. Which
leads to constant pressure to collect and label more training data. The
bottleneck of collecting and sharing data becomes less prominent with
increased automation, cheaper storage, and faster networks. Although, labeling
becomes more expensive as the data resolution grows and tasks become more
complex. Subject-matter experts (e.g., medical practitioners) are, therefore,
required to spend many hours as labeling experts at the cost of other duties.

Multiple vectors of research are focused on the question, of how to minimize
the amount of labeled data required for training: transfer learning [13],
self-supervised learning [4], active learning [10], etc. While only active
learning modifies the labeling procedure, others mostly assume the dataset
given. In application areas of Computed Tomography (CT), the number of data
sets is rapidly increasing due to advances in instrumentation and efficient
digital pixel array detectors [7]. Due to the diversity of biological or
technical specimens, and due to strict data requirements of the medical area
(e.g., anonymity and disease representation), task-specific datasets are often
collected. To help the community meet this growing demand for datasets, we
consider the question of how to acquire the best dataset with minimal effort.
To define what the best dataset is, we note that the dataset collection is not
the end of the process, but rather an intermediate step to produce a good
model. The value of the dataset collected to train a model, therefore, can be
measured as the performance of a model trained on this dataset. We focus this
paper on medical CT segmentation, however, our conclusions hold for other CT
segmentation tasks, since we never impose assumptions that are specific to the
medical imaging data.

We assume, that the main virtues of a good dataset collected for training are
_quality_ , _diversity_ , and _completeness_. The _quality_ itself could be
separated into the data quality and the label quality. In case of the data
collected for model training, the data quality is defined by the application
of the model. We will, therefore, exclude it from the consideration. The
labeling quality is defined by the imperfectness of the annotation [8], where
the segmentation labels are not accurately aligned with the actual anatomy
presented in the images. Hereinafter the _quality_ refers to the label
quality. The _diversity_ is understood as the ability to represent the general
variety of samples controlled by known parameters (e.g., patient’s age, sex,
and anamnesis). We understand the _completeness_ as the ability to represent
the natural fluctuations of the human morphology (e.g., even twins can have
slightly different morphological features). From the point of view of the data
manifold, _diversity_ is the ability to sparsely cover the whole manifold with
representative prototypical examples. In contrast, _completeness_ is the
ability to populate it densely and represent the distribution nuances.

In this work, we want to identify the relative importance of these virtues for
model performance. If we have all three virtues fulfilled to the maximal
extent in a dataset, no model is required, since any possible example is
already incorporated to a dataset, and prediction function is just a lookup
table. In practice, however, we train models to interpolate the full manifold
from sparse points provided in the training dataset. Of course, improving a
dataset by each virtue makes a better model. However, given the limited time,
experts balance them guided by intuition and often even implicitly.

In contrast to the concurrent work [5], we focus on the segmentation task
specifically, and consider only the fully supervised training, as opposed to
the weak supervision as a way to vary label granularity. We also propose a
labeling procedure that optimizes the effort.

##  2 Method

###  2.1 Datasets Preparation & Model Training

In this study we choose the brain tumor, heart, and liver tasks from the
Medical Decathlon segmentation datasets [1]. For the sake of simplicity, we
joined all available classes, to represent binary segmentation, however, our
private experience shows that the results hold for a multiclass segmentation.
For each dataset, we took the openly available markup and split substracted
20% of the available data as the test set. The test is selected once for a
dataset and never altered. We other 80% train+val set, as it will be split
again later on.

In medical datasets, it is typical to have a relatively small amount of
volumes collected from a representative variety of patients [9]. Hence, we
assume that the portion of random volumes used for training could be a proxy
to the _diversity_ , and we specify it as a number $\in(0,1]$ representing
this portion. Although this subsampling also affects _completeness_ , as we
show in Section 3.3 and especially in Figure 5, the model responds differently
to diversity and completeness. We conclude that this is a plausible and
sufficient proxy for the purposes of qualitative comparisons presented in this
paper. We follow [12], and always train a 2D model on slices, instead of a 3D
one on volumes. Based on assumption that adjacent slices represent small
variations of roughly the same morphology, we use the portion of the slices
used for training as a proxy for the _completeness_ , which is reported as
number $\in(0,1]$ as well. Finally, as a proxy for the _quality_ of the
dataset, we take a subset of equidistant slices and interpolate the labels
between them using the nearest neighbor approach. Varying the distance between
slices, and therefore the interpolation errors, allows us to manipulate the
label _quality_ , which we report as a percent $\in[0,100]$ representing the
IoU between the label after interpolation and the original label.

To measure the model performance for some virtue value, we:

  1. 1.

modify the train+val part of the dataset to model some virtue:

     * •

sample a portion of volumes to model _diversity_ ;

     * •

sample a portion of slices containing a mask to model _completeness_ ;

     * •

sample an equidistant set of slices and interpolate markup between them to
model _quality_.

  2. 2.

split the resulting data into train and val, at a ratio of 80 to 20.

  3. 3.

upsample the train in such a way, that the amount of labeled slices is always
equal to 80% of labeled slices in the original train+val set;

  4. 4.

fit the model on train, select the best snapshot on val, and measure the model
quality on test.

We hypothesized that, while tuning the model and optimizer hyperparameters can
change the model performance, it will not change the relative importance of
different dataset virtues for the model performance. Therefore, we always
train the same model (UNet [11] with ResNet-18 [3] as the backbone), with the
same optimizer (Adam [6] with $3e-4$ learning rate), for the same amount of
epochs (100 epochs and 10 epochs long cooldown of the early stopping). For
each measurement, the median of 5 runs on random train+val splits is reported.

###  2.2 Results Interpretation

The target of optimization of the labeling procedure is to obtain the model
with the best performance given a certain available effort budget for
labeling. For example, an expert can roughly segment 10 volumes, or spend the
same time, precisely segmenting 3 volumes. For experiments, we devise custom
proxies of the effort measure. We leave the empirical measurement of the
effort (e.g., as elapsed time) for future research, however, we consulted with
experts involved in the segmentation process, and they concur with our
estimation.

To find the optimal strategy we consider a plot, where the horizontal axis
describes labeling efforts spent, and the vertical axis represents the model
performance (to make the plots clearer we normalize the model performance to
the quality of the model trained on the unaltered data). For the same amount
of effort spent pursuing different virtues, we will have different model
qualities. The optimal strategy of labeling is represented by a convex
polyline that passes through the points on the plot in such a way that no
points lie above it. Following this trajectory provides the best possible
dataset at any given moment.

To understand the optimal strategy, we consider another plot. On the
horizontal axis, we plot the model performance, and on the vertical axes–the
value of the compared virtues. For each point on the optimal trajectory, we
add one point per virtue in comparison. Therefore, each vertically aligned set
of points represents the virtues required to achieve a specific model
performance. This way, moving along the horizontal axis, we can see which
virtue should be pursued earlier on, to stay on the optimal trajectory.

##  3 Results

###  3.1 What is More Important, _Quality_ or _Diversity_?

To compare _quality_ and _diversity_ , we define effort as the portion of the
volumes used (as a measure of _diversity_) multiplied by _quality_. E.g.,
$0.1$ of the volumes segmented with $80\%$ IoU will result in $8\%$ effort.

Figure 1: Optimal trajectory w.r.t. _quality_ and _diversity_. The variation
of the _quality_ is represented by color, of _diversity_ by the marker shape.

The sampled plot with the optimal trajectory is shown in Figure 1. From this
plot, we observe that the optimal trajectory connects the high-quality points,
while low-quality points always fall far below the line.

Figure 2: Importance of the _diversity_ and _quality_. The orange line
represents label _quality_ , and blue– _diversity_.

We show in Figure 2 how _quality_ and _diversity_ drive the optimal
trajectory. From this plot we conclude, that _quality_ is more important early
on, even though we never use IoU worse than 75% (which could be admissible
quality for small area labels). However, as _quality_ reaches around 90%,
increasing _diversity_ becomes as important or even more important than
increasing _quality_.

###  3.2 How Much Labeling _Quality_ is Enough?

Increasing labeling quality up to 100% is challenging if not impossible. But
where is a meaningful threshold of the labeling quality, after which the model
performance stagnates? To investigate it, we plot the model performance
against the labeling quality (see Figure 3). We also plot the performance of
the model trained on sparsely segmented slices (each 5th, 10th, and 15th). We
conclude, that, first, if one can not achieve good interpolation quality, it
may be even harmful to interpolate, and, second, in accordance with the
previous section–one should aim for 90% quality of the labeling before aiming
for either _completeness_ or _diversity_.

Figure 3: Saturation of the model performance with the increase of the label
_quality_. Additionally, the performance of the model trained on a small
portion of $1.0$ quality slices is represented with horizontal lines.

###  3.3 What is More Important, _Diversity_ or _Completeness_?

To compare _diversity_ and _completeness_ , we define the effort as a total
percentage of slices segmented. E.g., if we sample 6 volumes from 10 available
(_diversity_ = $0.6$) and segmented each 10th slice (_completeness_ = $0.1$),
then the effort is $0.06$.

Figure 4: Optimal trajectory w.r.t. _completeness_ and _diversity_. Variation
of _diversity_ is represented by color, of _completeness_ by the marker shape.

The optimal trajectory plot is shown in Figure 4. The brightest demonstration
of the importance of _diversity_ is in the right bottom parts: the $0.2$ of
_diversity_ with $1$ of _completeness_ is much worse than vice versa.

Figure 5: Importance of the _diversity_ and _completeness_. The orange line
represents label _completeness_ , and blue– _diversity_.

In Figure 5 we demonstrate the importance of _diversity_ and _completeness_
for the optimal trajectory. Not only _diversity_ is more important early on,
but _completeness_ contributes less to the model performance (note the steeper
growth of the _completeness_ with the performance growth). Therefore, finding
more diverse examples and segmenting more volumes should be preferred to
segmenting more random variations and segmenting slices more
densely/interpolating them.

###  3.4 How Much Data _Diversity_ is Enough?

Although, how to define a reasonable limit to stop searching for _diversity_ ,
and start increasing _completeness_? We plot the performance of the model
w.r.t. _diversity_ in Figure 6, each line represents a different set of
virtues. Since the total amount of the data possibly available is unknown, we
can not define numerical limit. Instead, we note that all lines saturate ca.
at the same point of increasing diversity. Hence we can define where to stop
increasing _diversity_ , and start _increasing_ completeness, by continuously
updating a segmentation model, while expanding the dataset.

Figure 6: Saturation of the model performance with the increase of the
_diversity_. No matter how other virtues vary (different colors), saturation
comes at the same point.

##  4 Conclusion

In this paper, we have compared the importance of different ways to spend
labeling efforts and presented a way to optimize the segmentation labeling
procedure. We minimized the effort required to obtain the model of a specific
quality. In general, we conclude that _quality_ is more important than
_diversity_ , which is more important than _completeness_. Based on our
experiments, we propose the following procedure to minimize the effort during
labeling volumetric data for segmentation:

  1. 1.

Start with segmenting slices, without interpolation. Aim for maximal quality
affordable without pixel hunting, at least 90%.

  2. 2.

Decide on your time budget and distribute slices to segment as evenly through
diverse volumes as possible. Though, keep in mind, that the structure of
interest may impose a minimal slice number per volume to capture all parts of
the structure.

  3. 3.

Train a model as early in the process as possible. This allows, first,
deciding which areas require more markup (by means of active learning, or just
by an expert assessment of predictions), and, second, recognizing the moment
when model performance starts to saturate w.r.t. _diversity_.

  4. 4.

After hitting the saturation w.r.t. the _diversity_ , increase the
_completeness_ either by adding more volumes or by interpolating more slices
to squeeze the last performance percent.

We leave a rigorous study of the exact conversion between theoretical effort
metrics presented in this paper and empirical efforts (e.g., time spent) for
future research.

#### Acknowledgment.

Data used in this paper was published in [1] and is available under a Creative
Commons license CC-BY-SA4.0.

## References

  * [1] Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B.A., Litjens, G., Menze, B., Ronneberger, O., Summers, R.M., van Ginneken, B., Bilello, M., Bilic, P., Christ, P.F., Do, R.K., Gollub, M.J., Heckers, S.H., Huisman, H., Jarnagin, W.R., McHugo, M.K., Napel, S., Pernicka, J.S., Rhode, K., Tobon-Gomez, C., Vorontsov, E., Meakin, J.A., Ourselin, S., Wiesenfarth, M., Arbeláez, P., Bae, B., Chen, S., Daza, L., Feng, J., He, B., Isensee, F., Ji, Y., Jia, F., Kim, I., Maier-Hein, K., Merhof, D., Pai, A., Park, B., Perslev, M., Rezaiifar, R., Rippel, O., Sarasua, I., Shen, W., Son, J., Wachinger, C., Wang, L., Wang, Y., Xia, Y., Xu, D., Xu, Z., Zheng, Y., Simpson, A.L., Maier-Hein, L., Cardoso, M.J.: The Medical Segmentation Decathlon. Nature Communications 13(1) (6 2022). https://doi.org/10.1038/s41467-022-30695-9, http://arxiv.org/abs/2106.05735
  * [2] Avetisian, M., Kokh, V., Tuzhilin, A., Umerenkov, D.: Radiologist-level stroke classification on non-contrast CT scans with Deep U-Net (3 2020). https://doi.org/10.1007/978-3-030-32248-9_91, http://arxiv.org/abs/2003.14287
  * [3] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2016-Decem, 770–778 (12 2016). https://doi.org/10.1109/CVPR.2016.90, http://arxiv.org/abs/1512.03385
  * [4] Jing, L., Tian, Y.: Self-Supervised Visual Feature Learning with Deep Neural Networks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(11), 4037–4058 (2 2021). https://doi.org/10.1109/TPAMI.2020.2992393, http://arxiv.org/abs/1902.06162
  * [5] Kim, T.S., Jang, G., Lee, S., Kooi, T.: Did You Get What You Paid For? Rethinking Annotation Cost of Deep Learning Based Computer Aided Detection in Chest Radiographs. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 13433 LNCS, 261–270 (9 2022). https://doi.org/10.1007/978-3-031-16437-8_25, http://arxiv.org/abs/2209.15314
  * [6] Kingma, D.P., Ba, J.L.: Adam: A method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings (2015)
  * [7] Kiryati, N., Landau, Y.: Dataset growth in medical image analysis research. Journal of Imaging 7(8), 155 (8 2021). https://doi.org/10.3390/jimaging7080155, https://www.mdpi.com/2313-433X/7/8/155
  * [8] Lösel, P.D., van de Kamp, T., Jayme, A., Ershov, A., Faragó, T., Pichler, O., Tan Jerome, N., Aadepu, N., Bremer, S., Chilingaryan, S.A., Heethoff, M., Kopmann, A., Odar, J., Schmelzle, S., Zuber, M., Wittbrodt, J., Baumbach, T., Heuveline, V.: Introducing Biomedisa as an open-source online platform for biomedical image segmentation. Nature Communications 11(1), 5577 (12 2020). https://doi.org/10.1038/s41467-020-19303-w, http://www.nature.com/articles/s41467-020-19303-w
  * [9] Luca, A.R., Ursuleanu, T.F., Gheorghe, L., Grigorovici, R., Iancu, S., Hlusneac, M., Grigorovici, A.: Impact of quality, type and volume of data used by deep learning models in the analysis of medical images. Informatics in Medicine Unlocked 29, 100911 (2022). https://doi.org/10.1016/j.imu.2022.100911, https://linkinghub.elsevier.com/retrieve/pii/S2352914822000612
  * [10] Ren, P., Xiao, Y., Chang, X., Huang, P.Y., Li, Z., Gupta, B.B., Chen, X., Wang, X.: A Survey of Deep Active Learning. ACM Computing Surveys 54(9) (8 2022). https://doi.org/10.1145/3472291, http://arxiv.org/abs/2009.00236
  * [11] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 9351, 234–241 (5 2015). https://doi.org/10.1007/978-3-319-24574-4_28, http://arxiv.org/abs/1505.04597
  * [12] Zettler, N., Mastmeyer, A.: Comparison of 2D vs. 3D U-Net Organ Segmentation in abdominal 3D CT images. Computer Science Research Notes 3101, 41–50 (7 2021). https://doi.org/10.24132/CSRN.2021.3101.5, http://arxiv.org/abs/2107.04062
  * [13] Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., He, Q.: A Comprehensive Survey on Transfer Learning (11 2019), http://arxiv.org/abs/1911.02685


11institutetext: Department of Physics and Astronomy, Iowa State University,
Ames, Iowa 50011, USA

# Polarization in heavy ion collisions: a theoretical review

Matteo Buzzegoli
Presented at the 20th International Conference on Strangeness in Quark Matter
(SQM 2022)11<EMAIL_ADDRESS>
###### Abstract

In these proceedings I discuss the recent progress in the theory of spin
polarization in relativistic fluids. To date, a number of studies have begun
to examine the impact of the shear tensor on the local spin polarization and
whether this contribution can restore agreement between the measurements and
the predictions obtained from a polarization induced by the gradients of the
plasma. I present the derivation of the spin polarization vector of a fermion
at local thermal equilibrium and I discuss the role of pseudo-gauge
transformations and of dissipative effects. I list what we can learn from the
polarization measured at lower energies. Finally, I discuss possible
applications of spin polarization measurements in relativistic heavy ion
collisions.

##  1 Introduction

Particles in a rotating medium tends to align their spin in the direction of
the total angular momentum due to the spin-rotation coupling. It was then
expected that fermions produced in relativistic heavy-ion collisions will have
a global spin polarization proportional to the vorticity of the fluid
Liang:2004ph ; Becattini:2007nd ; Becattini:2007sr . Based on this idea, one
obtains that the spin polarization vector of a fermion is given by a Cooper-
Frye like formula Becattini:2013fla :

$S^{\mu}(p)=-\frac{1}{8m}\epsilon^{\mu\rho\sigma\tau}p_{\tau}\frac{\int_{\Sigma}{\rm
d}\Sigma\cdot p\;n_{F}(1-n_{F})\varpi_{\rho\sigma}}{\int_{\Sigma}{\rm
d}\Sigma\cdot p\;n_{F}},$ (1)

where

$\varpi_{\mu\nu}=-\frac{1}{2}\left(\partial_{\mu}\beta_{\nu}-\partial_{\nu}\beta_{\mu}\right)$
(2)

is the thermal vorticity tensor and $n_{F}=\left({\rm e}^{\beta\cdot
p-\mu/T}+1\right)^{-1}$ is the Fermi distribution function with
$\beta^{\mu}=u^{\mu}/T$ the four-temperature vector and $\mu$ a chemical
potential. The predictions for global polarization obtained from this formula
were confirmed in 2017 with $\Lambda$ hyperons produced in Au-Au collisions at
$\sqrt{s_{NN}}=200$ Gev star ; Adam:2018ivw , see for instance
Becattini:2020ngo  for a review.

However, models based on the vorticity induced spin polarization (1) are not
able to reproduce later measurements of local $\Lambda$ spin polarization,
that is as a function of the particle momentum Adam:2019srw ; Niida:2018hfw ;
ALICE:2021pzu . This is known as local polarization sign puzzle because the
models predicted the opposite sign for the longitudinal (along the beam axis)
component of the spin polarization vector. Once it was established that this
discrepancy can not be explained with feed-down corrections Xia:2019fjf ;
Becattini:2019ntv , the sign puzzle was a strong motivation for theoretical
investigation as something relevant was clearly missing in the theoretical
description. Further investigations, that I review in next section, indeed
revealed that spin polarization can also be induced by the shear flow of the
fluid and, for heavy-ion collisions, this turned out to be a relevant
contribution.

##  2 Spin polarization at local thermal equilibrium

In this section I review the derivation of the spin polarization vector of a
fermion in a fluid that reached the local thermal equilibrium. The spin
polarization vector is obtained from the particle branch of the Wigner
function using (see the review Becattini:2020sww  for details)

$S^{\mu}_{\varpi}(p)=\frac{1}{2}\frac{\int_{\Sigma_{FO}}{\rm d}\Sigma\cdot
p\;{\rm
tr}_{4}\left[\gamma^{\mu}\gamma^{5}W_{+}(x,p)\right]}{\int_{\Sigma_{FO}}{\rm
d}\Sigma\cdot p\;{\rm tr}_{4}\left[W_{+}(x,p)\right]},$ (3)

where, for heavy-ion collision applications, the hypersurface $\Sigma_{FO}$ is
the decoupling hypersurface, where the system can be described with the quasi-
free hadronic effective fields. Hence, $\Lambda$ particles being quasi-free,
we can use the Wigner function of a free Dirac field:

$W(x,k)_{AB}=\frac{1}{(2\pi)^{4}}\\!\int\\!{\rm d}^{4}y\,{\rm e}^{-{\rm
i}k\cdot y}\langle:\bar{\Psi}_{B}(x+y/2)\Psi_{A}(x-y/2):\rangle,$ (4)

where the brackets denotes the thermal average with the statistical operator
$\hat{\rho}$: $\langle\hat{X}\rangle={\rm tr}\left(\hat{\rho}\hat{X}\right)$.

For a relativistic system whose underlying microscopic theory is a quantum
field theory, the statistical operator describing the thermal states can be
obtained using the Zubarev method Becattini:2019dxo . By maximazing the
entropy of the system at the moment of thermalization and imposing constrains
on the momenta and energy densities, one obtains the covariant statistical
operator at the time of decoupling

$\widehat{\rho}=\frac{1}{Z}\exp\Bigg{[}-\int_{\Sigma_{FO}}{\rm
d}\Sigma_{\mu}\left(\widehat{T}_{B}^{\mu\nu}\beta_{\nu}-\zeta\,\widehat{j}^{\mu}\right)+\int_{\Theta}{\rm
d}\Theta\left(\widehat{T}_{B}^{\mu\nu}\nabla_{\mu}\beta_{\nu}-\widehat{j}^{\mu}\nabla_{\mu}\zeta\right)\Bigg{]},$
(5)

where $\widehat{T}_{B}^{\mu\nu}$ is the Belinfante stress-energy tensor (SET)
operator, $\widehat{j}$ is a conserved current, and $\zeta=\mu/T$. One can
show that the second term in the exponent gives rise to dissipative effects,
while the first term gives the non-dissipative local thermal equilibrium. I
postpone the discussion of the dissipative part in a later section and here I
focus on the local equilibrium statistical operator:

$\widehat{\rho}\simeq\widehat{\rho}_{{\rm
LE}}=\frac{1}{Z}\exp\left[-\int_{\Sigma_{FO}}\\!\\!{\rm
d}\Sigma_{\mu}\left(\widehat{T}_{B}^{\mu\nu}\beta_{\nu}-\zeta\,\widehat{j}^{\mu}\right)\right].$
(6)

The derivation of the spin polarization at local thermal equilibrium from Eq.
(3) reduces to the evaluation of the Wigner function at local equilibrium

$W(x,k)_{LE}=\frac{1}{Z}{\rm tr}\left(\exp\left[-\int_{\Sigma_{FO}}\\!\\!{\rm
d}\Sigma_{\mu}(y)\left(\widehat{T}_{B}^{\mu\nu}(y)\beta_{\nu}(y)-\zeta(y)\,\widehat{j}^{\mu}(y)\right)\right]\widehat{W}(x,k)\right).$
(7)

For a fluid in the hydrodynamic regime, a good approximation of (7) is
obtained expanding the thermodynamic fields as a series of gradients around
the point $x$ and using the linear response theory to obtain the Wigner
function mean value. Keeping the first order in derivatives of the four-
temperature and of the chemical potential

$\beta_{\nu}(y)\simeq\beta_{\nu}(x)+\partial_{\lambda}\beta_{\nu}(x)(y-x)^{\lambda},\quad\zeta(y)\simeq\zeta_{\nu}(x)+\partial_{\lambda}\zeta(x)(y-x)^{\lambda}\,,$
(8)

one finds

$\begin{split}\widehat{\rho}_{LE}\simeq\frac{1}{Z}\exp&\left[-\beta_{\nu}(x)\widehat{P}^{\nu}+\zeta(x)\widehat{Q}+\frac{1}{2}\varpi_{\mu\nu}(x)\widehat{J}^{\mu\nu}_{x}-\frac{1}{2}\xi_{\mu\nu}(x)\widehat{Q}^{\mu\nu}_{x}\right.\\\
&\left.+\partial_{\lambda}\zeta(x)\\!\\!\\!\int\\!\\!\\!{\rm
d}\Sigma_{\mu}(y-x)^{\lambda}\widehat{j}^{\mu}(y)+\cdots\right],\end{split}$
(9)

where

$\widehat{J}^{\mu\nu}_{x}=\\!\int\\!{\rm
d}\Sigma_{\lambda}\left[(y-x)^{\mu}\widehat{T}_{B}^{\lambda\nu}(y)-(y-x)^{\nu}\widehat{T}_{B}^{\lambda\mu}(y)\right],$
(10)

is the conserved angular momentum operator (the generator of boosts and
rotations) and

$\widehat{Q}^{\mu\nu}_{x}=\\!\int\\!{\rm
d}\Sigma_{\lambda}\left[(y-x)^{\mu}\widehat{T}_{B}^{\lambda\nu}(y)+(y-x)^{\nu}\widehat{T}_{B}^{\lambda\mu}(y)\right],$
(11)

is a non-conserved symmetric quadrupole like operator. Notice that
$\widehat{J}^{\mu\nu}$ couples with the thermal vorticity $\varpi$, while
$\widehat{Q}^{\mu\nu}$ couples with the thermal shear tensor defined as

$\xi_{\mu\nu}=\frac{1}{2}\left(\partial_{\mu}\beta_{\nu}+\partial_{\nu}\beta_{\mu}\right).$
(12)

The spin polarization vector obtained from (3) using the linear response
theory on the operators appearing in (9) is

$S^{\mu}(p)=-\epsilon^{\mu\rho\sigma\tau}p_{\tau}\frac{\int_{\Sigma}{\rm
d}\Sigma\cdot
p\;n_{F}(1-n_{F})\left[\varpi_{\rho\sigma}+2\,\hat{t}_{\rho}\frac{p^{\lambda}}{\varepsilon}\xi_{\lambda\sigma}-\frac{\hat{t}_{\rho}\partial_{\sigma}\zeta}{2\varepsilon}\right]}{8m\int_{\Sigma}{\rm
d}\Sigma\cdot p\;n_{F}},$ (13)

where $\hat{t}$ is the time direction in the laboratory frame. The first term
is the vorticity induced polarization (1), the second is the recently found
shear induced polarization Becattini:2021suc ; Becattini:2021iol ; Liu:2021uhn
; Liu:2021nyg ; Yi:2021ryh , and the last one is the contribution from the
gradient of chemical potential obtained in Liu:2020dxg ; Fu:2022myl ,
sometimes called spin Hall effect term.

###  2.1 Thermal shear and solution of the sign puzzle

The effect of thermal shear in spin polarization was studied in ref.
Becattini:2021iol ; Yi:2021ryh ; Fu:2021pok ; Ryu:2021lnx ; Florkowski:2021xvy
; Sun:2021nsg ; Alzhrani:2022dpi ; Wu:2022mkr . These analyses revealed the
following facts. The thermal shear tensor at the freeze-out hypersurface can
be larger than the thermal vorticity. Because of its momentum dependence (see
Eq. (13)), the thermal shear does not contribute to global polarization but it
was found to give a large contribution to local spin polarization. Predictions
including the thermal shear are closer to the experimental data but they are
still lacking a good quantitative agreement.

In order to reproduce the experimental data, no new effects are needed. It is
sufficient to improve the approximation used to derive the spin polarization
spectrum (13). Since the available experimental data for local spin
polarization was taken at 200 GeV and at 5 TeV, at such high energies it is
expected that the hadronization happens at the same temperature. In that case
the statistical operator is given by the Isothermal Local Equilibrium
operator:

$\widehat{\rho}_{\rm ILE}=\frac{1}{Z}\exp\left[-\int_{\Sigma_{FO}}\\!\\!{\rm
d}\Sigma_{\mu}\widehat{T}_{B}^{\mu\nu}\left(\frac{u_{\nu}}{T_{\rm
FO}}\right)\right]=\frac{1}{Z}\exp\left[-\frac{1}{T_{\rm
FO}}\int_{\Sigma_{FO}}\\!\\!{\rm
d}\Sigma_{\mu}\widehat{T}_{B}^{\mu\nu}u_{\nu}\right],$ (14)

because the temperature $T_{\rm FO}$ is constant along the decoupling
hypersurface. In this setting, expanding the temperature in a Taylor series
together with $\beta$ as in Eq. (8) will only introduce unnecessary mistakes.
The appropriate spin polarization spectrum at high energies involves only the
fluid velocity gradients Becattini:2021iol :

$S_{\rm
ILE}^{\mu}(p)=-\epsilon^{\mu\rho\sigma\tau}p_{\tau}\frac{\int_{\Sigma}{\rm
d}\Sigma\cdot
p\;n_{F}(1-n_{F})\left[\omega_{\rho\sigma}+2\,\hat{t}_{\rho}\frac{p^{\lambda}}{\varepsilon}\Xi_{\lambda\sigma}\right]}{8mT_{\rm
FO}\int_{\Sigma}{\rm d}\Sigma\cdot p\;n_{F}},$ (15)

where

$\omega_{\rho\sigma}=\frac{1}{2}\left(\partial_{\sigma}u_{\rho}-\partial_{\rho}u_{\sigma}\right),\quad\Xi_{\rho\sigma}=\frac{1}{2}\left(\partial_{\sigma}u_{\rho}+\partial_{\rho}u_{\sigma}\right)$
(16)

denote the kinematic vorticity and shear tensors. The predictions of (15) are
in quantitative agreement with the data taken at 200 GeV Becattini:2021iol .
Preliminary results presented at this conference Palermo:sqm22  showed an
improvement of the agreement when including the feed-down effects and that the
predictions of (15) are also in quantitative agreement with the local spin
polarization measured by ALICE ALICE:2021pzu . Future analyses of high energy
spin polarization should use the isothermal settings.

What also emerged from these analyses is that the local spin polarization is
sensitive to the properties of the plasma, which ultimately determines the
gradients of the thermo-hydrodynamic fields in (13). This feature allows to
use spin polarization as a probe of the quark gluon plasma properties.

###  2.2 Low energies

With new data of spin polarization at very low energy STAR:2021beb , we can
further test the models and see whether they can reproduce the maximum of spin
polarization Ivanov:2019ern ; Ivanov:2022ble ; Deng:2020ygd ; Deng:2021miw ;
Guo:2021udq  expected to occurs around $\sqrt{S_{NN}}=7$ GeV. This energy
range is also important for polarization in the FAIR and NICA facilities. At
lower energies we also observe a larger deviation between the polarization of
$\Lambda$ and $\bar{\Lambda}$, which allows to probe the gradients of baryonic
chemical potential Liu:2020dxg ; Fu:2022myl  and the effect of the magnetic
field Guo:2019joy . Physics of rotating matter at these energies was also
discussed in this conference Liao:sqm22 .

###  2.3 Spin tensor and pseudo-gauge dependence

For the first time, the importance of the spin degrees of freedom in the
determination of the spin polarization vector, opens the possibility to have a
direct experimental test of a local description of spin, i.e. a spin tensor.
Indeed one can include a spin tensor in hydrodynamic equations
Florkowski:2018fap ; Hattori:2019lfp ; Hongo:2021ona ; Hongo:2022izs ;
Bhadury:2020cop ; Bhadury:2021oat ; She:2021lhe  and let it evolve with the
rest. In the same way that the temperature is a thermodynamic potential for
energy, the thermodynamic quantity associated with spin is called spin
potential: $\Omega_{\mu\nu}$, and it might be present in spin polarization.

The definition of spin tensor is not unique in special relativity but is given
up to a transformation called pseudo-gauge transformation. For instance, from
the Belinfante pseudo-gauge with SET $\widehat{T}^{\mu\nu}_{B}$ and vanishing
spin tensor, a generic pseudo-gauge $\Phi$ is obtained with:

$\begin{split}\widehat{T}^{\mu\nu}_{\Phi}=&\widehat{T}^{\mu\nu}_{B}+\frac{1}{2}\nabla_{\lambda}\left(\widehat{\Phi}^{\lambda,\mu\nu}-\widehat{\Phi}^{\mu,\lambda\nu}-\widehat{\Phi}^{\nu,\lambda\mu}\right),\\\
\widehat{\mathcal{S}}^{\lambda,\mu\nu}_{\Phi}=&-\widehat{\Phi}^{\lambda,\mu\nu}+\nabla_{\rho}\widehat{Z}^{\mu\nu,\lambda\rho},\end{split}$
(17)

where $\widehat{\Phi}^{\lambda,\mu\nu}=-\widehat{\Phi}^{\lambda,\nu\mu}$ and
$\widehat{Z}^{\mu\nu,\lambda\rho}=-\widehat{Z}^{\nu\mu,\lambda\rho}=-\widehat{Z}^{\mu\nu,\rho\lambda}$.
This raises the question whether a pseudo-gauge transformation affects the
results for spin polarization. It was indeed realized that the local
equilibrium form of statistical operator is different for different pseudo-
gauges Becattini:2018duy :

$\widehat{\rho}_{\rm
LTE}^{\,\Phi}=\frac{1}{\mathcal{Z}}\exp\left\\{\\!-\\!\int\\!\\!{\rm
d}\Sigma_{\mu}\\!\\!\left[\widehat{T}^{\mu\nu}_{B}\beta_{\nu}\\!-\\!\frac{1}{2}\left(\varpi_{\lambda\nu}-\Omega_{\lambda\nu}\right)\widehat{\Phi}^{\mu,\lambda\nu}-\xi_{\lambda\nu}\widehat{\Phi}^{\lambda,\mu\nu}-\frac{1}{2}\Omega_{\lambda\nu}\nabla_{\rho}\widehat{Z}^{\lambda\nu,\mu\rho}-\widehat{j}^{\mu}\zeta\right]\right\\}.$
(18)

Starting from this, the different predictions of the spin polarization
spectrum for several pseudo-gauge were obtained Buzzegoli:2021wlg . This
difference is not only related to the contribution of the spin potential but
also of the thermal shear. This feature is not unique of the statistical
mechanics approach and the role of pseudo-gauge was studied in different
approaches Speranza:2020ilk ; Fukushima:2020ucl ; Das:2021aar ; Li:2020eon ;
Weickgenannt:2022jes . This is an unique opportunity to study the physical
meaning of pseudo-gauge transformations and possibly to shed light on the role
of spin potential in gravitational phenomena.

##  3 Dissipative effects and relativistic kinetic theory with spin

The current data and the inclusion of the thermal shear support the local
equilibrium picture with a small contribution of dissipative effects. However,
one should check how much the dissipative phenomena affect the spin
polarization. These corrections in the spin polarization formula have been
found for the massless Shi:2020htn  and massive Weickgenannt:2022zxs  Dirac
field, but so far there are no analyses on the impact of these effects on the
spin polarization of $\Lambda$ particles in heavy-ion collisions. If the
dissipative effects will be found to be relevant, this would imply a
separation of scales between the interaction time and the time in which the
spin degrees of freedom equilibrates Hongo:2021ona ; Hongo:2022izs ;
Hidaka:2017auj . In this scenario the inclusion of a spin tensor in
hydrodynamics mentioned above will be crucial.

An estimate of the magnitude of the dissipative effect might not be easy, as
it was realized that, in addition to usual transport coefficients, in a
hydrodynamic with spin additional transport coefficients related to spin
emerge Weickgenannt:2022zxs ; Hongo:2021ona ; Hongo:2022izs ; Hidaka:2017auj .
The values of these new transport coefficients for the QGP can be fixed by
fitting the data of spin polarization, but in this way one inevitably loose
the predictive power of the current theory.

A powerful approach to study dissipative effects and the role of spin in
hydrodynamics is the relativistic kinetic theory. The Wigner-function
formalism in particular is convenient to preserve all the covariant properties
of the system and to find results as an expansion in $\hbar$. Motivated by
spin polarization, this approach recently experienced a rich and fast
development Weickgenannt:2019dks ; Gao:2019znl ; Hattori:2019ahi ; Liu:2020flb
; Weickgenannt:2020aaf ; Weickgenannt:2021cuo .

In this regard, I would like to draw attention to an often overlook assumption
needed to solve the Wigner equation in kinetic theory with collisions:

$\left[\gamma\cdot\left(p+{\rm
i}\frac{\hbar}{2}\partial\right)-m\right]W_{\alpha\beta}=\hbar\,\mathcal{C}_{\alpha\beta}.$
(19)

Wigner equation itself is not enough to determine all the properties of the
system. For instance, changing $\widehat{\Phi}$ in (18) results in different
Wigner functions all solving Eq. (19). Indeed the determination of the non
equilibrium, dissipative Wigner function from (19) requires the input of the
equilibrium Wigner function. For a system in absence of vorticity, the
equilibrium form is well known. But the general form of global equilibrium
Wigner function in the presence of vorticity was obtained only recently
Becattini:2020qol ; Palermo:2021hlf . Usually the ansatz for the equilibrium
form used in kinetic theory agrees with the correct form only at first order
in thermal vorticity. For the current applications is enough, but one have to
keep it in mind when deriving higher order results.

##  4 Applications of spin polarization in heavy ion collisions

Spin is a new element in heavy-ion collisions and as a commonly accepted
theoretical framework is being built, several ideas on how to use it as a
diagnostic tool for the QGP have been proposed. What makes spin polarization a
good probe to the QGP properties is its dependence on the gradients of the
thermo-hydrodynamics fields (fluid velocity, temperature and baryonic chemical
potential).

For instance, reproducing the vorticity structure of the plasma requires
knowledge of the properties and evolution of the QGP. It was then proposed to
use spin correlations of two $\Lambda$ hyperons to probe the vorticity
structure of the fluid Pang:2016igs ; Xia:2018tes ; Ryu:2021lnx . In this way
one acquires more constraints on the properties of the QGP.

Correlations of helicity, that is the projection of spin along the momentum of
the particle, of two hyperons in the same event can also reveal the presence
of local parity violation in hot QCD matter Du:2008zzb ; Becattini:2020xbh ;
Gao:2021rom . Local parity violations should create an imbalance of right-
handed and left-handed quarks in the plasma, described with an axial chemical
potential $\mu_{A}$. Currently the chiral magnetic effect is used to reveal
the presence of $\mu_{A}$ Kharzeev:2004ey . But the axial imbalance should
also add a contribution to spin polarization (13) as Becattini:2020xbh

$S^{\mu}(p)=S^{\mu}_{\rm hydro}(p)+\frac{g_{h}}{2}\frac{\int_{\Sigma}{\rm
d}\Sigma\cdot p\;(\mu_{A}/T)n_{F}\left(1-n_{F}\right)}{\int_{\Sigma}{\rm
d}\Sigma\cdot p\;n_{F}}\frac{\varepsilon
p^{\mu}-m^{2}\hat{t}^{\mu}}{m\varepsilon},$ (20)

resulting into a peculiar (parity breaking) contribution to helicity
correlations.

Spin polarization can be used to study anything that produce large enough
gradients. This is the case of a jet crossing the plasma. The vorticity field
generated by it contains information about the energy lost by the jet and can
be studied with spin polarization measurements in reference to the jet axis
Serenone:2021zef .

The results derived so far for spin polarization do not include interactions.
Vorticity induced polarization is a consequence of spin-rotation coupling
$g_{\Omega}\vec{S}\cdot\vec{\Omega}$, and Einstein Equivalence Principle (EEP)
prevents spin-rotation coupling to receive radiative corrections:
$g_{\Omega}\equiv 1$ kobzarev1962gravitational . However, the breaking of
Lorentz covariance at finite temperature also breaks the EEP and this allows
interactions to affect the spin-rotation coupling Buzzegoli:2021jeh . For
strong interactions the leading order correction is expected to be

$g_{\Omega}=1-\frac{N_{c}^{2}-1}{2}\frac{1}{6}\frac{g^{2}T^{2}}{m^{2}},\quad
T\ll m$ (21)

which, in a typical event, results in a reduction of about 40% in the spin
polarization of the strange quark. Whether we can detected this consequence of
the breaking of EEP in the energy dependence of $\Lambda$ polarization is
still under investigation.

##  5 Conclusions

Spin polarization is a new topic in heavy ion collisions that just started to
show its potential. It can be used to answer many questions left unanswered in
relativistic hydrodynamics and to probe the properties of hot QCD matter. When
all the relevant gradients are included and more accurate approximations
adopted, the description of the QGP as a fluid close to local equilibrium with
small viscous corrections is receiving confirmation as a solid model in heavy
ion collisions. Right now, theory and experiments in this field are rapidly
expanding and spin will be used as a tool to investigate fundamental physics
in the QGP and beyond.

Acknowledgments. M.B. is supported by the US Department of Energy under Grant
No. DE-FG02-87ER40371.

## References

  * (1) Z.T. Liang, X.N. Wang, Phys. Rev. Lett. 94, 102301 (2005), [Erratum: Phys.Rev.Lett. 96, 039901 (2006)], nucl-th/0410079
  * (2) F. Becattini, F. Piccinini, Annals Phys. 323, 2452 (2008), 0710.5694
  * (3) F. Becattini, F. Piccinini, J. Rizzo, Phys. Rev. C 77, 024906 (2008), 0711.1253
  * (4) F. Becattini, V. Chandra, L. Del Zanna, E. Grossi, Annals Phys. 338, 32 (2013), 1303.3431
  * (5) L. Adamczyk et al. (STAR), Nature 548, 62 (2017), 1701.06657
  * (6) J. Adam et al. (STAR), Phys. Rev. C 98, 014910 (2018), 1805.04400
  * (7) F. Becattini, M.A. Lisa, Ann. Rev. Nucl. Part. Sci. 70, 395 (2020), 2003.03640
  * (8) J. Adam et al. (STAR), Phys. Rev. Lett. 123, 132301 (2019), 1905.11917
  * (9) T. Niida (STAR), Nucl. Phys. A 982, 511 (2019), 1808.10482
  * (10) S. Acharya et al. (ALICE), Phys. Rev. Lett. 128, 172005 (2022), 2107.11183
  * (11) X.L. Xia, H. Li, X.G. Huang, H.Z. Huang, Phys. Rev. C 100, 014913 (2019), 1905.03120
  * (12) F. Becattini, G. Cao, E. Speranza, Eur. Phys. J. C 79, 741 (2019), 1905.03123
  * (13) F. Becattini, Lect. Notes Phys. 987, 15 (2021), 2004.04050
  * (14) F. Becattini, M. Buzzegoli, E. Grossi, Particles 2, 197 (2019), 1902.01089
  * (15) F. Becattini, M. Buzzegoli, A. Palermo, Phys. Lett. B 820, 136519 (2021), 2103.10917
  * (16) F. Becattini, M. Buzzegoli, G. Inghirami, I. Karpenko, A. Palermo, Phys. Rev. Lett. 127, 272302 (2021), 2103.14621
  * (17) S.Y.F. Liu, Y. Yin, JHEP 07, 188 (2021), 2103.09200
  * (18) Y.C. Liu, X.G. Huang, Sci. China Phys. Mech. Astron. 65, 272011 (2022), 2109.15301
  * (19) C. Yi, S. Pu, D.L. Yang, Phys. Rev. C 104, 064901 (2021), 2106.00238
  * (20) S.Y.F. Liu, Y. Yin, Phys. Rev. D 104, 054043 (2021), 2006.12421
  * (21) B. Fu, L. Pang, H. Song, Y. Yin (2022), 2201.12970
  * (22) B. Fu, S.Y.F. Liu, L. Pang, H. Song, Y. Yin, Phys. Rev. Lett. 127, 142301 (2021), 2103.10403
  * (23) S. Ryu, V. Jupic, C. Shen, Phys. Rev. C 104, 054908 (2021), 2106.08125
  * (24) W. Florkowski, A. Kumar, A. Mazeliauskas, R. Ryblewski, Phys. Rev. C 105, 064901 (2022), 2112.02799
  * (25) Y. Sun, Z. Zhang, C.M. Ko, W. Zhao, Phys. Rev. C 105, 034911 (2022), 2112.14410
  * (26) S. Alzhrani, S. Ryu, C. Shen, arXiv:2203.15718 (2022), 2203.15718
  * (27) X.Y. Wu, C. Yi, G.Y. Qin, S. Pu, Phys. Rev. C 105, 064909 (2022), 2204.02218
  * (28) A. Palermo, F. Becattini, M. Buzzegoli, G. Inghirami, I. Karpenko, _Local equilibrium and Lambda polarization in high energy heavy ion collisions_ , in _These proceedings_ (2022), 2208.09874
  * (29) M.S. Abdallah et al. (STAR), Phys. Rev. C 104, L061901 (2021), 2108.00044
  * (30) Y.B. Ivanov, V.D. Toneev, A.A. Soldatov, Phys. Rev. C 100, 014908 (2019), 1903.05455
  * (31) Y.B. Ivanov, A.A. Soldatov, Phys. Rev. C 105, 034915 (2022), 2201.04527
  * (32) X.G. Deng, X.G. Huang, Y.G. Ma, S. Zhang, Phys. Rev. C 101, 064908 (2020), 2001.01371
  * (33) X.G. Deng, X.G. Huang, Y.G. Ma (2021), 2109.09956
  * (34) Y. Guo, J. Liao, E. Wang, H. Xing, H. Zhang, Phys. Rev. C 104, L041902 (2021), 2105.13481
  * (35) Y. Guo, S. Shi, S. Feng, J. Liao, Phys. Lett. B 798, 134929 (2019), 1905.12613
  * (36) J. Liao, _Novel Effects of Rotational Polarization in Relativistic Nuclear Collisions_ , in _These proceedings_ (2022)
  * (37) W. Florkowski, A. Kumar, R. Ryblewski, Prog. Part. Nucl. Phys. 108, 103709 (2019), 1811.04409
  * (38) K. Hattori, M. Hongo, X.G. Huang, M. Matsuo, H. Taya, Phys. Lett. B 795, 100 (2019), 1901.06615
  * (39) M. Hongo, X.G. Huang, M. Kaminski, M. Stephanov, H.U. Yee, JHEP 11, 150 (2021), 2107.14231
  * (40) M. Hongo, X.G. Huang, M. Kaminski, M. Stephanov, H.U. Yee (2022), 2201.12390
  * (41) S. Bhadury, W. Florkowski, A. Jaiswal, A. Kumar, R. Ryblewski, Phys. Rev. D 103, 014030 (2021), 2008.10976
  * (42) S. Bhadury, J. Bhatt, A. Jaiswal, A. Kumar, Eur. Phys. J. ST 230, 655 (2021), 2101.11964
  * (43) D. She, A. Huang, D. Hou, J. Liao (2021), 2105.04060
  * (44) F. Becattini, W. Florkowski, E. Speranza, Phys. Lett. B 789, 419 (2019), 1807.10994
  * (45) M. Buzzegoli, Phys. Rev. C 105, 044907 (2022), 2109.12084
  * (46) E. Speranza, N. Weickgenannt, Eur. Phys. J. A 57, 155 (2021), 2007.00138
  * (47) K. Fukushima, S. Pu, Phys. Lett. B 817, 136346 (2021), 2010.01608
  * (48) A. Das, W. Florkowski, R. Ryblewski, R. Singh, Phys. Rev. D 103, L091502 (2021), 2103.01013
  * (49) S. Li, M.A. Stephanov, H.U. Yee, Phys. Rev. Lett. 127, 082302 (2021), 2011.12318
  * (50) N. Weickgenannt, D. Wagner, E. Speranza, Phys. Rev. D 105, 116026 (2022), 2204.01797
  * (51) S. Shi, C. Gale, S. Jeon, Phys. Rev. C 103, 044906 (2021), 2008.08618
  * (52) N. Weickgenannt, D. Wagner, E. Speranza, D. Rischke (2022), 2203.04766
  * (53) Y. Hidaka, S. Pu, D.L. Yang, Phys. Rev. D 97, 016004 (2018), 1710.00278
  * (54) N. Weickgenannt, X.L. Sheng, E. Speranza, Q. Wang, D.H. Rischke, Phys. Rev. D 100, 056018 (2019), 1902.06513
  * (55) J.H. Gao, Z.T. Liang, Phys. Rev. D 100, 056021 (2019), 1902.06510
  * (56) K. Hattori, Y. Hidaka, D.L. Yang, Phys. Rev. D 100, 096011 (2019), 1903.01653
  * (57) Y.C. Liu, K. Mameda, X.G. Huang, Chin. Phys. C 44, 094101 (2020), [Erratum: Chin.Phys.C 45, 089001 (2021)], 2002.03753
  * (58) N. Weickgenannt, E. Speranza, X.l. Sheng, Q. Wang, D.H. Rischke, Phys. Rev. Lett. 127, 052301 (2021), 2005.01506
  * (59) N. Weickgenannt, E. Speranza, X.l. Sheng, Q. Wang, D.H. Rischke, Phys. Rev. D 104, 016022 (2021), 2103.04896
  * (60) F. Becattini, M. Buzzegoli, A. Palermo, JHEP 02, 101 (2021), 2007.08249
  * (61) A. Palermo, M. Buzzegoli, F. Becattini, JHEP 10, 077 (2021), 2106.08340
  * (62) L.G. Pang, H. Petersen, Q. Wang, X.N. Wang, Phys. Rev. Lett. 117, 192301 (2016), 1605.04024
  * (63) X.L. Xia, H. Li, Z.B. Tang, Q. Wang, Phys. Rev. C 98, 024905 (2018), 1803.00867
  * (64) F. Du, L.E. Finch, J. Sandweiss, Phys. Rev. C 78, 044908 (2008)
  * (65) F. Becattini, M. Buzzegoli, A. Palermo, G. Prokhorov, Phys. Lett. B 822, 136706 (2021), 2009.13449
  * (66) J.H. Gao, Phys. Rev. D 104, 076016 (2021), 2105.08293
  * (67) D. Kharzeev, Phys. Lett. B 633, 260 (2006), hep-ph/0406125
  * (68) W.M. Serenone, J.a.G.P. Barbon, D.D. Chinellato, M.A. Lisa, C. Shen, J. Takahashi, G. Torrieri, Phys. Lett. B 820, 136500 (2021), 2102.11919
  * (69) I.Y. Kobzarev, L. Okun, Zh. Eksperim. i Teor. Fiz. 43 (1962)
  * (70) M. Buzzegoli, D.E. Kharzeev, Phys. Rev. D 103, 116005 (2021), 2102.01676


# A Dual Adversarial Calibration Framework for Automatic Fetal Brain Biometry

Yuan Gao1 Lok Hin Lee111footnotemark: 1 Richard Droste1,211footnotemark: 1
Rachel Craik1,3 Sridevi Beriwal1
Aris Papageorghiou1 Alison Noble1
1 University of Oxford 2 Amazon 3 Kings College London Equal
contribution.Corresponding author<EMAIL_ADDRESS>
###### Abstract

This paper presents a novel approach to automatic fetal brain biometry
motivated by needs in low- and medium- income countries. Specifically, we
leverage high-end (HE) ultrasound images to build a biometry solution for low-
cost (LC) point-of-care ultrasound images. We propose a novel unsupervised
domain adaptation approach to train deep models to be invariant to significant
image distribution shift between the image types. Our proposed method, which
employs a Dual Adversarial Calibration (DAC) framework, consists of
adversarial pathways which enforce model invariance to; i) adversarial
perturbations in the feature space derived from LC images, and ii) appearance
domain discrepancy. Our Dual Adversarial Calibration method estimates
transcerebellar diameter and head circumference on images from low-cost
ultrasound devices with a mean absolute error (MAE) of 2.43mm and 1.65mm,
compared with 7.28 mm and 5.65 mm respectively for SOTA.

##  1 Introduction

Pregnancy dating is a crucial part of obstetric care because antenatal care
and interventions aimed at improving pregnancy outcome rely on knowledge of
the gestational age (GA). Measurements of the size of specific fetal head
anatomies is routinely performed to estimate and validate GA. Biometries used
include the fetal skull Head Circumference (HC) and the Transcerebellar
Diameter (TCD) [1], easily measured on ultrasound images from high-end (HE)
machines. US images from HE imaging machines have high imaging contrast, high
imaging definition and low speckle noise compared to low-cost (LC) ultrasound
images which are acquired with point-of-care (POC) ultrasound probes with
greater varied image appearance and hence quality [5]. However, HE imaging may
not be available in resource-constrained areas.

Previous literature has considered automated approaches for fetal brain
biometry on HE 2D ultrasound (US) images for GA estimation. [20] use contour
detection and graph cuts for HC estimation. [14] propose a regional
convlutional neural network for detection of key anatomical structures. [21]
use U-Nets for HC segmentation and measurement. More recently, [24] directly
regresses HC measurements from ultrasound images without segmentation and [13]
directly regresses GA from fetal head images using a Bayesian neural network.
However, these methods estimate on mid-end US images, which may not be
available in resource-constrained settings.

In this paper, we consider jointly learning HC and TCD automated fetal
biometry from partially labelled US images acquired with a HE ultrasound
machine (GE Voluson E8) combined with unlabelled data from a LC POC ultrasound
probe (Konted C10R) for biometry on LC images. This is clinically relevant in
a case where interobserver variation on ground truth labelling on LC images is
high, due to the reduced image quality and fuzzy edges, and can be useful
where a central corpus of well-labelled HE images are available for use for
validation and inference on LC images. The core assumption is that domain
invariant representations for feature extraction can be jointly learned from
LC and HE ultrasound images, and unsupervised learning can calibrate the model
in the LC domain so to produce consistent predictions. To this end, we propose
a Dual Adversarial Calibration (DAC) approach exploiting two adversarial
pathways. One pathway forces predictions from LC and HE US images to lie on
the same output manifold by training a segmentation network with a
discriminator which learns to classify between them. The other pathway forces
the LC output to be invariant to self-paced adversarial noise perturbations.
Additionally, we propose a novel asymmetric domain augmentation technique
specifically designed to cope with the appearance discrepancy between HE and
LC images. Experimental results presented show that our proposed approach
significantly improves the performance of HC and TCD biometry on LC US images
compared to a neural network trained on HE images alone. Ablation experiments
also reveal that our dual adversarial pathways lead to a network that is able
to learn useful feature representations that are domain invariant.

Figure 1: Overview of the training procedure for our proposed dual adversarial
calibration network, which is trained with two adversarial signals
simultaneously. 1) a Domain Calibration (DC) pathway designed to learn domain-
invariant segmentation, and 2) a Segmentation Calibration (SC) pathway
designed to learn robust segmentation on the unlabelled LC images. SAU: Strong
Augmentation; WAU: Weak Augmentation; SS: Supervised Segmentation (on HE
images); BU: Bilinear Upsampling. UEP, LEP explained in section 3.2.

###  1.1 Contributions.

Our contributions are three-fold: 1) we leverage the supervised learned
knowledge from HE ultrasound images to significantly improve biometry
estimation on LC ultrasound images; 2) we introduce a novel dual adversarial
unsupervised intra-modality domain and semantic transfer for this purpose; 3)
our results are shown to be competitive compared to SOTA for both automated HC
and TCD estimation on LC US images.

##  2 Related Work

###  2.1 Medical Cross-Modality Domain Adaptation.

Medical cross-modality domain adaptation aims to retain network performance
from the distribution change from an image resulting from one imaging modality
to another. Examples of imaging modalities include magnetic resonance imaging
(MRI), computed tomography (CT) and ultrasound (US) imaging. Prior literature
focused on cross-modality domain adaptation between CT/MR [7, 11, 3, 6]. In
detail, [7] uses an adversarial domain adaptation module to map target input
features to the output domain space. [11] use a tumour-preserving cycle-
consistency loss to map CT and MRI images before training a U-Net to segment
lung MRI scans. [3] use a shared encoder space with adversarial based domain
adaptations for the segmentation of cardiac structures. [6] share
convolutional kernels between MRI and CT images, but use modality specific
normalization layers to improve cross-modality performance. Comparatively,
literature on cross-modality domain adaptation involving US is limited. [12]
generate synthetic MR fetal head images from US scans, but validation is
limited as only appearance is evaluated without segmentation results.

###  2.2 Intra-Modality US Domain Adaptation

Previous literature on intra-modality US domain adaptation focused on
adaptation between different HE imaging devices [4, 15, 18]. [4] use a u-net
for left atrium segmentation in 3D ultrasound, and incorporate the imaging
device used as prior knowledge during inference. [15] use a hierarchical style
transfer network to modify image appearances from a target domain to the
source domain for fetal head and abdomen segmentation. [18] considered intra-
domain ultrasound adaptation for image classification using mutual information
minimization. However, in all the examples above, the domain gap in considered
was limited as both source and target domain images were acquired with HE
ultrasound machines with similar imaging capabilities.

###  2.3 Low-Cost Ultrasound Probe Image Analysis

Three recent papers investigate learning from low-cost POC probes [8, 23, 17].
Gao et al.[8] developed an image quality assessment framework to identify
frames from LC US that can be used for manual TCD measurement by a
sonographer. Van den Heuvel et al.[23] investigate gestational age estimation
on downsampled HE images to evaluate GA estimation on degraded images, but do
not directly evaluate from LC POC probes. Maraci et al.[17] segments the
cerebellum for TCD measurement, but the LC US image led to poor segmentation
performance.

##  3 Method

Figure 2: Asymmetrical Domain Augmentation. SAU: Strong Augmentation; WAU:
Weak Augmentation. ${\dagger}$: Extra augmentations included in SAU.

In this section, we present the Dual Adversarial Calibration framework in
detail, which is illustrated in Fig.1. The framework consists of four main
parts: 1) a domain dependent asymmetrical augmentation module of the input
space; 2) a segmentation network consisting of a shared encoder-decoder
framework; 3) a domain calibration (DC) adversarial pathway for semantic
transfer on the predicted output space; 4) a segmentation calibration (SC)
cycle pathway for unsupervised semantic transfer.

###  3.1 Asymmetrical Domain Augmentation.

We observe that in practice, HE data are fairly consistent in imaging quality
and appearance, whereas LC data can vary quite substantially. Key anatomies,
such as the cerebellum and the thalamus are always clearly visible on HE data,
but may not be so on LC data, as imaged structures do not have clear edges and
acoustic artefacts such as shadows and speckle can lead to further image
degradation. We therefore augment each input domain asymmetrically.
Specifically, weak augmentation is applied on LC data to maximize network
generalization whilst preserving the spatial visibility of key anatomies by
the inclusion of linear and non-linear grid distortions, and horizontal
flipping. Strong augmentation is applied on HE data to simulate noisy images
including random gamma, random brightness and contrast adjustment, image
compression and artificial speckle noise. We qualitatively observe these data
augmentations act to decrease domain gap between HE and LC data, as shown in
Figure 2, which leads to a stronger response to domain calibration.

###  3.2 Core Segmentation Network.

We formulate fetal biometry via segmentation tasks. The HC is derived from
segmentation of the fetal skull. However in LC US images, segmentation of the
cerebellum is challenging due to low image contrast and noise from signal
attenuation. We therefore adopt a strategy inspired by TCD measurement in
real-clinical practice, where two points are placed on the cerebellum boundary
edge for TC diameter estimation. Thus, the network predicts the upper extreme
point (UEP) and lower extreme point (LEP) of the cerebellum, from which the
TCD is estimated.

We therefore target two image analysis tasks - fetal skull segmentation for
the HC, and UEP and LEP detection for TCD estimation. We employ a U-Net based
segmentation network, represented by $\\{G(x;\theta),x\in(x_{LC},x_{HE})\\}$
where $x_{LC}$ and $x_{HE}$ are examples of HE and LC input US images
respectively and $\theta$ represents model parameters. We use ResNet18 [9] as
an encoder-decoder backbone with residual blocks (fine to coarse: 64, 64, 128,
256 ,512 feature channels) for the encoder and residual blocks with 2D
bilinear upsampling (coarse to fine: 256, 128, 64, 32, 16 feature channels)
for the decoder with skip connections between the encoder-decoder. Our
segmentation head consists of a single $3\times 3$ 2D convolutional layer to
map the decoded feature maps (16 channels) to 3 channels, corresponding to the
skull, UEP and LEP predictions. We then apply a pixel-wise sigmoid function to
the segmentation output to obtain a pixel-wise probability map for each
anatomy. To address class imbalance, we use a DICE loss to train the
segmentation model defined as:

Figure 3: The pipeline from segmentation to the final biometry.

$L_{seg}=1-\frac{2\sum_{h,w,c}y_{HE}p_{HE}}{\sum_{h,w,c}y^{2}_{HE}+\sum_{h,w,c}p^{2}_{HE}}$
(1)

where $y_{HE}\epsilon\mathbb{R}^{h\times w\times c}$ and
$p_{HE}\epsilon\mathbb{R}^{h\times w\times c}$ are the ground truth
annotations and the pixel-wise probability maps for HE images, h, w and c are
height, width and number of classes respectively (c=3 in our experiments).

The whole pipeline from input to final biometric estimation is depicted in
Figure 3. We got the segmentation probability maps for each structure i.e.
Skull, UEP and LEP. For computing HC, we segment the entire fetal head,
retrieve a skull contour from the probability map, then fit an ellipse to the
contour, obtain the center, major and minor axis, and rotation of the fitted
ellipse. For computing TCD, we perform non-maximum suppression to find the
pixel with greatest probability for UEP and LEP, then draw a line between the
two estimated point locations.

###  3.3 Domain Calibration Pathway.

We observe that $x_{HE}$ and $x_{LC}$ have very different intensity
distributions. This reflects in imaging quality discrepancy, and networks
trained on one do not perform well on the other. To address this, we introduce
an adversarial pathway to calibrate the underlying output space to be
invariant to the input domain [22]. Specifically, the output space can be
modelled as a low-dimensional manifold that contains simple representations
and rich semantic information about target anatomies. By minimizing the
distance between HE US predictions ${p}_{HE}=G(x_{HE};\theta)$ and LC US
predictions ${p}_{LC}=G(x_{LC};\theta)$, the model can learn specific target
anatomical regions of interest from a low dimensional representational space.
We adopt a Least-Square GAN [16] loss for domain adaptation. The domain
adaptation is modelled with the objective:

$\begin{split}\operatorname*{argmin}_{\varphi}L_{adv}(D)&:=\frac{1}{2}\operatorname{\mathbb{E}}_{p\sim{p}_{out}(HE)}[(D({p}_{HE};\varphi)-1)^{2}]\\\
&+\frac{1}{2}\operatorname{\mathbb{E}}_{p\sim{p}_{out}(LC)}[(D(p_{LC};\varphi))^{2}]\end{split}$
(2)

$\operatorname*{argmin}_{\theta}L_{adv}(G):=\frac{1}{2}\operatorname{\mathbb{E}}_{p\sim{p}_{out}(LC)}[(D({p}_{LC};\varphi)-1)^{2}]$
(3)

Here $D(p;\phi)$ represents the input probality map ${p}$ to our discriminator
parameterised by $\phi$. The discriminator consists of five convolutional
layers (fine-to-coarse: 64, 128, 256, 512, 1 feature channels) with a kernel
size of 3 and stride of 1 and each followed by a leaky ReLU parameterised by
0.2 and a max pooling layer.

Figure 4: Examples of fetal brain biometry planes acquired by different
devices and pixel intensity distributions. HE scanner: GE Voluson E8, LC
probe: Konted C10R. TC: Transcerebellar planes, TV: Transventricular planes,
TT: Transthalamic planes, UEP: Upper Extreme Point, LEP: Lower Extreme Point,
TCD: Transcerebellar Diameter, HC: Head Circumference, BPD: Biparietal
Diameter, and OFD: Occipito-Frontal Diameter.

###  3.4 Segmentation Calibration Pathway.

The generator $G(x,\theta)$ is trained on supervised segmentation with the
labelled source input $x_{HE}$ with available ground truth $y_{HE}$. We
further regularize the model using unlabelled $x_{LC}$ images by using
adaptive perturbations by noting that the predicted segmentation should be
locally smooth to adversarial perturbations in the input $x_{LC}$ images [19].
As shown in Fig.1, we generate an augmented LC input tuple
$X=\\{(x_{LC},x_{LC}+\xi),\xi\sim N(0,1)\\}$ and prediction tuple
$P=(p_{LC},\hat{p}_{LC})$ from the segmentation network $G(X;\theta)$. $\xi$
is self-paced and updated with a cycle pathway by maximizing the distance
between $(p_{LC},\hat{p}_{LC})$, generating $\xi_{adv}$, an adversarial
perturbation, then minimising the Kullbeck-Leibler loss
$KL\left[G(x;\theta)\parallel G(x+\xi_{adv})\right]$ to reduce network
sensitivity to noise perturbations. To do this we compute the derivative of
$L_{D}$ w.r.t. $\xi_{adv}$ defined as
$g_{\xi}=\bigtriangledown_{\xi_{adv}}L_{D}$ evaluated with backpropagation,
and the perturbation as
$\xi_{adv}=\epsilon\frac{g_{\xi}}{\left\|g_{\xi}\right\|_{2}}$, where
$\epsilon$ is a hyper-parameter that controls the strength of perturbation.
The goal is therefore for the network to be robust to the perturbations so as
to produce consistent outputs from the perturbed inputs. The distance loss
$L_{D}$ is therefore defined as:

$\begin{split}&\operatorname*{argmin}_{\theta}L_{D}(x,\xi,\theta):=\\\
&\operatorname{\mathbb{E}}_{x\sim
p_{data}(LC)}KL[[G(x_{LC};\theta)]\parallel[G(x_{LC}+\xi_{adv};\theta)]]\\\
&\text{s.t.}\
\xi_{adv}:=\operatorname*{argmax}_{\xi}\left\\{L_{D}(x_{LC},\xi,\theta);\left\|\xi\right\|_{2}\leq\epsilon\right\\}\end{split}$
(4)

###  3.5 Optimisation.

Our model optimization is performed in a two-step process during model
training. The domain discriminator is first optimized by minimizing the loss
$L_{adv}(D)$ while the generator is frozen. The generator is then subsequently
optimized by optimizing the joint loss $L_{joint}$, defined as:

$\centering\begin{split}&L_{joint}(x_{HE},x_{LC},\xi,\theta)=\\\
&L_{seg}(X_{HE},\theta)+\alpha L_{D}(X_{LC},\xi,\theta)+\beta
L_{adv}(G)\end{split}\@add@centering$ (5)

where the hyperparameters $\alpha$ and $\beta$ determine the strength of
segmentation and domain calibration respectively.

##  4 Experiments and Results

Table 1: Dataset description for training and evaluation of our model. Planes
are split between test and training on a subject basis to prevent data
leakage.

| No. of Subjects | Acquisition Device | Training (Seg. Labels) | Testing (Biometry [mm])
---|---|---|---|---
|  |  | TC Planes | TC Planes (HC) | TT/TV Planes (TCD)
HE Dataset | 540 | GE Volusen E8 | 519 (Labelled HC & TCD) | - | -
LE Dataset | 560 | Konted GEN1 C10R | 418 (Unlabelled) | 387 (Labelled) | 526 (Labelled)

###  4.1 Datasets.

We have two different datasets acquired from two different clinical studies,
examples of which can be seen in Fig. 4. HE and LC images were acquired from a
clinical US scanner and a POC probe respectively described in Table 1, along
with the distribution of scans used for training and testing. The difference
in image quality is apparent in Fig. 4. This includes reduced imaging contrast
between tissues of high echogenicity (fetal skull) and low echogenicity
(internal brain tissue), reduced clarity of internal brain structures such as
the cerebellum and ventricles, reduced edge sharpness as can be seen from the
boundary of the skull. All of these features are clinically used for
determination of TCD and HC. Furthermore, image resolution is reduced and
there is increased noise in LC images. We used TC planes from the HE dataset
segmented with HC and TCD measurement points along with unlabelled TC planes
from the LC dataset to train our model. The performance of our model was then
evaluated by comparison between biometry from inferred segmentations and the
biometry extracted by an expert sonographer.

Images from each source differed in input resolution (784$\times$1008 px HE,
228$\times$378 px LC). All images were resized to 448$\times$576px. Ground
truth pixel maps for the fetal skull were labelled $x_{i}\in\\{0,1\\}$ for
pixels inside and outside the skull respectively. For TCD estimation, instead
of labelling the entire cerebellum, a Gaussian kernel was centered on the
sonographer annotated key points i.e. UEP and LEP used as the ground truth for
training. As seen in table 2, we achieve mean TCD error of 2.43 mm and mean HC
error of 1.65mm on the best performing model.

###  4.2 Network Training.

Our model was implemented with pytorch 1.4.0 and trained on a single Quadro
RTX 5000 GPU. $G(\theta)$ was optimized with Nesterov accelerated SGD with an
initial learning rate of 0.1, momentum of 0.9 and a weight decay of $10^{-3}$.
$D(\phi)$ was optimized with Adam with an initial learning rate of $10^{-4}$
and weight decay of $10^{-4}$. Both $D(\phi)$ and $G(\theta)$ were optimized
for 70 epochs with a minibatch size of 4. Both learning rates were multiplied
by a factor of 0.1 after epochs 40 and 60. Grid search was performed on a
logarithmic scale for $\alpha$ and $\beta$; $\alpha=10^{-1}$ and
$\beta=10^{-3}$ gave the best performance.

Table 2: Ablation study of our method and comparison to SOTA for HC and TCD
estimation on LC test dataset. Mean Absolute Error (MAE) $\pm$ std is
reported.

Method | Aug. | DC, Adpt. Loss, Adpt. Space | SC |  | TCD
---
(mean $\pm$ SD [mm])
| HC
---
(mean $\pm$ SD [mm])
1 W/o | ✓, Weak | - | - | 46.63$\pm$8.64 | 36.27$\pm$7.81
2 W/o | ✓, Strong | - | - | 30.96$\pm$7.46 | 22.52$\pm$7.54
3 DC | ✓, Asymmetrical | ✓, V-GAN Loss, out. space | - | 13.80$\pm$3.91 | 10.21$\pm$3.63
4 DC | ✓, Asymmetrical | ✓, LS-GAN Loss, feat. space | - | 8.64$\pm$1.42 | 6.53$\pm$1.11
5 DC | ✓, Asymmetrical | ✓, LS-GAN Loss, out. space | - | 7.93$\pm$1.64 | 4.31$\pm$1.25
6 DAC | ✓, Asymmetrical | ✓, LS-GAN Loss, feat. space | ✓ | 4.62$\pm$0.46 | 3.26$\pm$0.43
7 DAC | ✓, Asymmetrical | ✓, LS-GAN Loss, out. space | ✓ | 2.43$\pm$0.37 | 1.65$\pm$0.31
CycleGAN [25] | ✓, Asymmetrical | ✓, Cycle-GAN Loss, in. space | - | 14.79$\pm$3.20 | 9.69$\pm$3.19
CyCADA [10] | ✓, Asymmetrical | ✓, CyCADA Loss, in.$\&$feat. space | - | 7.28$\pm$2.72 | 5.65$\pm$2.25
UCMDA [7] | ✓, Asymmetrical | ✓, Adversarial loss, in feat. space | - | 8.13$\pm$2.30 | 6.74$\pm$2.19
SIFA [2] | ✓, Asymmetrical | ✓, SIFA Loss, in.$\&$feat.$\&$out. space | - | 8.69$\pm$1.21 | 6.32$\pm$1.04

  * •

* Note: W/o: Without Adaptation; DC: Domain Calibration; SC: Segmentation Calibration; DAC: Dual Adversarial Calibration.

###  4.3 Results and Ablation Study

Figure 5: Biometry distributions on LC test dataset from different models
numbered 1-7 as in Table 2. (a) predicted HCs (b) predicted TCDs for different
models. GT: expert manual biometry. Figure 6: Example outputs from different
models numbered 1-7 as in Table 2, evaluated on LC test images (Blue box: TCD,
purple box: HC). The GT column represents the ground truth.

To better understand the individual contributions from each component of our
model, we perform an ablation study which composed of three settings: 1) we
train the segmentation network with labelled HE images and the directly apply
to LC images at test time; 2) we investigate domain adaptation from either
feature or output space by applying adversarial training; 3) in addition to
domain adaptation, we incorporate the self-paced unsupervised branch for
segmentation calibration. We show in Table 2 ( 1, 2) which are models trained
with HE images directly evaluated on LC images. Direct estimation of
biometries from LC images from a model trained with HE images performs poorly
as expected due to the substantial domain shift. Inclusion of adversarial
calibration ( 3, 4, 5) significantly improves biometry estimation. We also
find that training with a LS-GAN adversarial loss outperforms a Vanilla GAN
loss, and adversarial calibration on the output space leads to better
localization compare to feature space ( 5, 7 vs. 4, 6), which suggests that
the output manifold is a suitable rich environment for domain calibration.
Further addition of adversarial segmentation calibration ( 6, 7) leads to the
best performance with MAE of 1.65mm for HC and 2.43mm for TCD estimation.

Also, as can be seen in Fig. 5 (a) and (b), without domain adaptation i.e. 1,
2, there are a number of outliers and the biomertry is significantly out of
the distribution compared to the expert’s manual biometry for both the HC and
TCD. By incorporating the adversarial training 3, 5, the model’s predicted
distribution is shifted towards the ground truth. DAC 7 results in the most
closed distribution to the expert biomerty. Additionally, we found HC
predicted by DAC model tends to be better agreed with expert measurements,
compared to TCD and the HC mean absolute error is smaller than TCD’s. This may
because the appearance of the TCD landmark is prone to be affected by image
quality and artefacts, however, the skull signal is bright and consistent,
which is less affected by the imaging quality changes.

###  4.4 Comparison with SOTA.

We compare our proposed model with current SOTA in the lower rows of Table 2.
We found that domain adaptation methods that adapt from the input space
(CycleGAN [25]) give a high estimation error, as only style is preserved by
internal image content and anatomies are not well conserved, especially for TC
images. Taking into account the feature space (CyCADA [10]) leads to better
performance, but continues to misjudge points of the cerebellum. We also
investigated the SOTA methods for unsupervised cross-modality domain
adaptation, UCMDA and SIFA [7, 2]. UCMDA[7] proposed a domain critic module
(DCM) that minimizes difference between domains in the feature space. SIFA
uses a shared encoder and performs domain adaptation in the output space, and
use cycle-GAN based identity loss to learn domain invariant features. We find
that these cross modality models not necessarily works on our intra-modality
adaptation task and underperform compared to our best performing model 7. This
may because intra-modality domain shift is relatively moderate compared to
cross-modality and the cross-modality models are too complex to fit on the
moderate domain shift. Our proposed DAC design focus on aligning the intra-
modality domain shift by simply incorporating a self-paced distribution
alignment process. The self-paced calibration helps to align the intra-domain
feature space and make models more robust against artefacts and variation in
imaging quality, introduced by LC probes.

###  4.5 Qualitative Results.

As shown in Fig. 6, without domain or segmentation adversarial calibration,
models ( 1, 2) fail to localize at least one of the TCD measurement points and
fail to segment the skull for HC. We find that including domain calibration (
3, 5) leads to noisy probability maps for the TCD measurement points, but the
final prediction (after non-maximum suppression) become more accurate. Using a
LS-GAN loss leads to smoother adversarial calibration for domain adaptation
and which leads to more localized prediction of the TCD measurement points ( 5
vs. 3). In addition to domain calibration, we find that segmentation
calibration (reflected in Table2 5 vs. 7) increases the performance of TCD
estimation more than HC estimation. This suggests that adversarial
segmentation perturbation helps the network to localize of small targets in a
noisy environment. Our complete model 7 outperforms all of the above. It
correctly identifies the TCD measurement points and segments the fetal skull,
even in challenging examples.

##  5 Conclusion

This paper addresses the problem of domain adaptation from clinical high-end
ultrasound images to low cost point-of-care ultrasound images with greater
varied imaging quality and increased noise. We proposed a novel dual
adversarial calibration framework which enables the network to learn invariant
features to both image types, and leverages high-end ultrasound images to
enable a solution for accurate automatic biometry on LC US images. Our
approach outperforms current SOTA for domain calibration and semi-supervised
learning methods applied to this task.

#### Acknowledgements

We acknowledge the ERC (ERC-ADG-2015 694 project PULSE), the EPSRC
(EP/R013853/1, EP/T028572/1) and the MRC (MR/P027938/1).

## References

  * [1] Martin R Chavez et al.  Fetal transcerebellar diameter measurement for prediction of gestational age at the extremes of fetal growth.  JUM, 2007.
  * [2] Cheng Chen et al.  Semantic-aware generative adversarial nets for unsupervised domain adaptation in chest x-ray segmentation.  In MLMI, 2018.
  * [3] Cheng Chen et al.  Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation.  In AAAI, 2019.
  * [4] Markus A Degel, Nassir Navab, and Shadi Albarqouni.  Domain and geometry agnostic cnns for left atrium segmentation in 3d ultrasound.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 630–637. Springer, 2018.
  * [5] Christoph F Dietrich et al.  Point of care ultrasound: a wfumb position paper.  Ultrasound in medicine & biology, 2017.
  * [6] Qi Dou, Quande Liu, Pheng Ann Heng, and Ben Glocker.  Unpaired multi-modal segmentation via knowledge distillation.  IEEE transactions on medical imaging, 39(7):2415–2425, 2020.
  * [7] Qi Dou, Cheng Ouyang, Cheng Chen, Hao Chen, and Pheng-Ann Heng.  Unsupervised cross-modality domain adaptation of convnets for biomedical image segmentations with adversarial loss.  arXiv preprint arXiv:1804.10916, 2018.
  * [8] Yuan Gao, Sridevi Beriwal, Rachel Craik, Aris T Papageorghiou, and J Alison Noble.  Label efficient localization of fetal brain biometry planes in ultrasound through metric learning.  In Medical Ultrasound, and Preterm, Perinatal and Paediatric Image Analysis, pages 126–135. Springer, 2020.
  * [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [10] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  CyCADA: Cycle-consistent adversarial domain adaptation.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1989–1998, Stockholmsmässan, Stockholm Sweden, 10–15 Jul 2018. PMLR.
  * [11] Jue Jiang, Yu-Chi Hu, Neelam Tyagi, Pengpeng Zhang, Andreas Rimner, Gig S Mageras, Joseph O Deasy, and Harini Veeraraghavan.  Tumor-aware, adversarial domain adaptation from ct to mri for lung cancer segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 777–785. Springer, 2018.
  * [12] Jianbo Jiao, Ana IL Namburete, Aris T Papageorghiou, and J Alison Noble.  Self-supervised ultrasound to mri fetal brain image synthesis.  IEEE Transactions on Medical Imaging, 39(12):4413–4424, 2020.
  * [13] Lok Hin Lee, Elizabeth Bradburn, Aris T Papageorghiou, and J Alison Noble.  Calibrated bayesian neural networks to estimate gestational age and its uncertainty on fetal brain ultrasound images.  In Medical Ultrasound, and Preterm, Perinatal and Paediatric Image Analysis, pages 13–22. Springer, 2020.
  * [14] Zehui Lin, Shengli Li, Dong Ni, Yimei Liao, Huaxuan Wen, Jie Du, Siping Chen, Tianfu Wang, and Baiying Lei.  Multi-task learning for quality assessment of fetal head ultrasound images.  Medical image analysis, 58:101548, 2019.
  * [15] Zhendong Liu, Xiaoqiong Huang, Xin Yang, Rui Gao, Rui Li, Yuanji Zhang, Yankai Huang, Guangquan Zhou, Yi Xiong, Alejandro F Frangi, et al.  Generalize ultrasound image segmentation via instant and plug & play style transfer.  In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 419–423. IEEE, 2021.
  * [16] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley.  Least squares generative adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2794–2802, 2017.
  * [17] Mohammed A Maraci, Mohammad Yaqub, Rachel Craik, Sridevi Beriwal, Alice Self, Peter von Dadelszen, Aris Papageorghiou, and J Alison Noble.  Toward point-of-care ultrasound estimation of fetal gestational age from the trans-cerebellar diameter using cnn-based ultrasound image analysis.  Journal of Medical Imaging, 7(1):014501, 2020.
  * [18] Qingjie Meng et al.  Mutual information-based disentangled neural networks for classifying unseen categories in different domains: Application to fetal ultrasound imaging.  TMI, 2020.
  * [19] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii.  Distributional smoothing with virtual adversarial training.  arXiv preprint arXiv:1507.00677, 2015.
  * [20] Sylvia Rueda, , et al.  Evaluation and comparison of current fetal ultrasound image segmentation methods for biometric measurements: a grand challenge.  TMI, 2013.
  * [21] Zahra Sobhaninia, Shima Rafiei, Ali Emami, Nader Karimi, Kayvan Najarian, Shadrokh Samavi, and SM Reza Soroushmehr.  Fetal ultrasound image segmentation for measuring biometric parameters using multi-task deep learning.  In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 6545–6548. IEEE, 2019\.
  * [22] Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker.  Learning to adapt structured output space for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7472–7481, 2018.
  * [23] Thomas LA van den Heuvel, Hezkiel Petros, Stefano Santini, Chris L de Korte, and Bram van Ginneken.  Automated fetal head detection and circumference estimation from free-hand ultrasound sweeps using deep learning in resource-limited countries.  Ultrasound in medicine & biology, 45(3):773–785, 2019.
  * [24] Jing Zhang, Caroline Petitjean, Pierre Lopez, and Samia Ainouz.  Direct estimation of fetal head circumference from ultrasound images based on regression cnn.  In Medical Imaging with Deep Learning, pages 914–922. PMLR, 2020\.
  * [25] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.