aainstitutetext: Department of Physics and Astronomy, University of
California, Riverside, CA 92521, USAbbinstitutetext: Physics Division,
Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USAccinstitutetext:
Berkeley Institute for Data Science, University of California, Berkeley, CA
94720, USA

# Designing Observables for Measurements
with Deep Learning

Owen Long b,c and Benjamin Nachman

###### Abstract

Many analyses in particle and nuclear physics use simulations to infer
fundamental, effective, or phenomenological parameters of the underlying
physics models. When the inference is performed with unfolded cross sections,
the observables are designed using physics intuition and heuristics. We
propose to design optimal observables with machine learning. Unfolded,
differential cross sections in a neural network output contain the most
information about parameters of interest and can be well-measured by
construction. We demonstrate this idea using two physics models for inclusive
measurements in deep inelastic scattering.

##  1 Introduction

Simulations are widely used for parameter estimation in particle and nuclear
physics. A typical analysis will follow one of two paths: forward-folding or
unfolding. In the forward-folding pipeline, the target physics model must be
specified at the time of inference. We focus on unfolding, where detector
distortions are corrected in a first step (unfolding) and then the resulting
cross section can be analyzed in the context of many models by any end users.
In the unfolding pipeline, the first step is to identify observables sensitive
to a given parameter(s). These are typically identified using physical
reasoning. Then, the differential cross sections of these observables are
measured, which includes unfolding with uncertainty quantification. Finally,
the measured cross sections are fit to simulation templates with different
values of the target parameters. This approach has been deployed to measure
fundamental parameters like the top quark mass CMS:2022kqg  and the strong
coupling constant $\alpha_{s}(m_{Z})$ H1:2017bml ; Collaboration:2023xha  as
well as parton distribution functions H1:2015ubc ; Harland-Lang:2016yfn ;
Hou:2019efy ; NNPDF:2021njg  and effective or phenomenological parameters in
parton shower Monte Carlo programs Campbell:2022qmc .

A key drawback of the standard pipeline is that the observables are
constructed manually. There is no guarantee that the observables are maximally
sensitive to the target parameters. Additionally, the observables are usually
chosen based on particle-level information alone and so detector distortions
may not be small. Such distortions can reduce the sensitivity to the target
parameter once they are corrected for by unfolding. In some cases, the
particle-level observable must be chosen manually because it must be
calculable precisely in perturbation theory. There have been proposals to
optimize the detector-level observable for a given particle-level observable
Arratia:2022wny  since they do not need to be the same. Alternatively, one
could measure the full phase space and project out the desired observable
after the fact Arratia:2021otl  Datta:2018mwd ; bunse2018unification ;
Ruhe2019MiningFS ; Andreassen:2019cjw ; Bellagente:2019uyp ; 1800956 ;
Vandegar:2020yvw ; Andreassen:2021zzk ; Howard:2021pos ; Backes:2022vmn ;
Chan:2023tbf ; Shmakov:2023kjj ; Alghamdi:2023emm ; H1:2021wkz ;
H1prelim-22-031 ; H1:2023fzk ; H1prelim-21-031 ; LHCb:2022rky ;
Komiske:2022vxg ; Song:2023sxb . When the inference is directly based on Monte
Carlo simulations, the particle-level observable is not fixed by perturbative
calculability.

We propose to use machine learning for designing observables that are
maximally sensitive to a given parameter(s) while also being minimally
sensitive to detector distortions. Simultaneous optimization ensures that we
only use regions of phase space that are measurable. A tailored loss function
is used to train neural networks. We envision that this approach could be used
for any case where simulations are used for parameter estimation. For
concreteness, we demonstrate the new technique to the case of differentiating
two parton shower Monte Carlo models of deep inelastic scattering. While
neither model is expected to match data exactly, the availability of many
events with corresponding detailed simulations makes this a useful benchmark
problem. We do not focus on the parameter estimation step itself, but there
are many proposals for doing this optimally with machine learning
Brehmer:2018kdj ; Brehmer:2018eca ; Brehmer:2018hga ; DeCastro:2018psv ;
Elwood:2018qsr ; Andreassen:2019nnm ; Brehmer:2019xox ; Wunsch:2020iuh ;
Ghosh:2021roe ; GomezAmbrosio:2022mpm ; Simpson:2022suz .

This paper is organized as follows. Section 2 introduces our approach to
observable construction. The datasets used for demonstrating the new method
are introduced in Sec. 3. Results with these datasets are presented in Sec. 4.
The paper ends with conclusions and outlook in Sec. 5.

##  2 Methodology

We begin by constructing new observables that are simultaneously sensitive to
a parameter while also being minimally sensitive to detector effects. We
introduce the method with a toy model for continuous parameter estimation,
which demonstrates the essential ideas in a simplified context. This is
followed by a more complete binary classification example using simulated deep
inelastic scattering events from the H1 experiment at HERA, where the goal is
to be maximally sensitive to distinguishing two datasets.

###  2.1  Toy example for continuous parameter estimation

For each event, we have pairs $(Z,X)$ where $Z$ represents the particle-level
observable and $X$ represents the detector-level observable. Capital letters
represent random variables while lower-case letter represent realizations of
the corresponding random variables. We consider the case $X$ and $Z$ have the
same structure, i.e. they are both sets of 4-vectors. This is the standard
case where $X$ is a set of energy-flow objects that are meant to correspond to
the 4-vectors of particles before being distorted by the detector.
Furthermore, we fix the same definition of the observable at particle and
detector level. The training samples are generated with a uniform distribution
for the parameter of interest $\mu$, so each event is specified by
$(\mu_{i},z_{i},x_{i})$. Then, we parameterize the observable $f$ as a neural
network and optimize the following loss function:

$\displaystyle
L[f]=\sum_{i}(f(x_{i})-\mu_{i})^{2}+\lambda\sum_{i}(f(x_{i})-f(z_{i}))^{2}\,,$
(1)

where the form of both terms is the usual mean squared error loss used in
regression tasks. The first term trains the regression to predict the
parameter of interest $\mu$ while the second term trains the network to make
the predictions given detector level features $x$ and particle level features
$z$ similar. The hyperparameter $\lambda$ must be tuned and controls the trade
off between sensitivity to the parameter of interest $\mu$ and sensitivity to
detector effects.

The loss function in Eq. 1 is similar to the setting of decorrelation, where a
classifier is trained to be independent from a given feature Blance:2019ibf ;
Englert:2018cfo ; Louppe:2016ylz ; Dolen:2016kst ; Moult:2017okx ;
Stevens:2013dya ; Shimmin:2017mfk ; Bradshaw:2019ipy ; ATL-PHYS-PUB-2018-014 ;
DiscoFever ; Wunsch:2019qbo ; Rogozhnikov:2014zea ; 10.1088/2632-2153/ab9023 ;
Kasieczka:2020pil ; Kitouni:2020xgb ; Estrade:2019gzk ; Aguilar-
Saavedra:2017rzt ; aguilarsaavedra2020mass . One could apply decorrelation
techniques in this case to ensure the classifier is not able to distinguish
between features at detector level and at particle level. However, this will
only ensure that the probability density for $f$ is the same for particle
level and detector level. To be well-measured, we need more than statistical
similarity between distributions - we need them to be similar event by event.
The final term in Eq. 1 is designed for exactly this purpose.

All deep neural networks are implemented in Keras chollet2015keras /TensorFlow
tensorflow  and optimized using Adam adam . The network models use two hidden
layers with 50 nodes per layer and Rectified Linear Unit (ReLU) activation
functions for intermediate layers and a linear activation function for the
last layer.

Figure 1:  Input features and resolution model for toy regression example. Two
different experimental resolution functions, A and B, are shown.

Figure 1 illustrates the input features and resolution model for the toy
study. Two particle-level features $z_{0}$ and $z_{1}$ are modeled as normal
distributions: $Z_{0}\sim\mathcal{N}(\mu,0.5)$ and
$Z_{1}\sim\mathcal{N}(\mu,0.1)$, where feature 1 is significantly more
sensitive to the parameter of interest $\mu$. The experimental resolution on
the features is given by $X_{0}\sim\mathcal{N}(Z_{0},0.1)$ and
$X_{1}\sim\mathcal{N}(Z_{1},0.5)$ so that feature 0 is well measured, while
feature 1 has a relatively poor resolution. For this model, the net
experimental sensitivity to $\mu$ is the same for both features, but feature 0
is much less sensitive to detector effects. Our proposed method will take this
into account in the training of the neural network.

Figure 2:  Results of toy regression example as a function of the $\lambda$
hyperparameter.

To demonstrate the sensitivity to uncertainties associated with detector
effects, we make predictions using $f$ trained with resolution model A on a
sample generated with resolution model B, shown in Figure 1, where the width
is increased by a factor of 1.4 for both features and a bias of 0.2 is
introduced for the $x_{1}$ feature. Figure 2 shows the results as a function
of the $\lambda$ hyperparameter. With $\lambda=0$, which corresponds to the
usual approach for this type of regression task, the correlation between each
of the detector level features $x_{0}$ and $x_{1}$ and the network prediction
$f$ is the same, reflecting the fact that $x_{0}$ and $x_{1}$ have the same
sensitivity to $\mu$. As $\lambda$ increases, more emphasis is placed on
feature $x_{0}$, which is well measured. The resolution of the regression,
given by the RMS of $f$, starts at the expected value of about
$\sqrt{0.5^{2}+0.1^{2}}/\sqrt{2}$ for resolution model A and $\lambda=0$ and
increases with $\lambda$ as the network relies more on $x_{0}$ for the
prediction. The bias in the prediction for resolution model B is large for
$\lambda=0$ but falls significantly with increasing $\lambda$.

###  2.2  Full example for binary classification

In the binary case, we have two datasets generated from simulation 1 (sim. 1)
and simulation 2 (sim. 2). The loss function for classification is given by

$\displaystyle L[f]=-\sum_{i\in\text{sim.
1}}\log(f(z_{i}))-\sum_{i\in\text{sim.
2}}\log(1-f(z_{i}))+\lambda\sum_{i\in\text{sim. 1 \&
2}}(f(x_{i})-f(z_{i}))^{2}\,,$ (2)

where the first two terms represent the usual binary cross entropy loss
function for classification and the third term represents the usual mean
squared error loss term for regression tasks. As in the regression case, the
hyperparameter $\lambda$ must be tuned and controls the trade off between
sensitivity to the dataset and sensitivity to detector effects. The network
model is the same as in the previous example except that the final layer uses
a sigmoid activation function. The binary case is a special case of the
previous section where there are only two values of the parameter of interest.
It may also be effective to train the binary case for a continuous parameter
using two extreme values of the parameter. In this paper, we use high-quality,
well-curated datasets from the binary case because of their availability, but
it would be interesting to explore the continuous case in the future.

To determine the efficacy of the new observable, we unfold it with TUnfold
version 17.9 Schmitt:2012kp  through the interface included in the Root 6.24
Brun:1997pa  distribution. The response matrix is defined from a 2D binning
the NN output, given detector level and particle level features. The matrix
uses 24 and 12 bins for detector and particle inputs, respectively, which
gives reasonable stability and cross-bin correlations in the unfolding
results. The ultimate test is to show that the difference between sim. 1 at
detector-level unfolded with sim. 2 for the response matrix and the particle
level sim. 1 (or vice versa) is smaller than the difference between sim. 1 and
sim. 2 at particle level. In other words, this test shows that the ability to
distinguish sim. 1 and sim. 2 significantly exceeds the modeling uncertainty
from the unfolding.

##  3 Datasets

We use deep inelastic scattering events from high-energy electron-proton
collisions to demonstrate the performance of the new approach. These simulated
data are from the H1 experiment at HERA H1:1996jzy ; H1:1996prr  and are used
in the same way as Ref. Arratia:2021tsq . They are briefly described in the
following.

Two parton shower Monte Carlo programs provide the particle-level simulation:
Rapgap 3.1 Jung:1993gf  or Djangoh 1.4 Charchula:1994kf . The energies of the
incoming beams are $E_{e}=27.6$ GeV and $E_{p}=920$ GeV, for the lepton and
proton, respectively, matching the running conditions of HERA II. Radiation
from Quantum Electrodynamic processes is simulated by Heracles routines
Spiesberger:237380 ; Kwiatkowski:1990cx ; Kwiatkowski:1990es  in both cases.
The outgoing particles from these two datasets are then fed into a Geant 3
geant4 -based detector simulation.

Following the detector simulation, events are reconstructed with an energy-
flow algorithm energyflowthesis ; energyflowthesis2 ; energyflowthesis3  and
the scattered electron is reconstructed using the default H1 approach
H1:2012qti ; H1:2014cbm ; H1:2021wkz . Mis-measured backgrounds are suppressed
with standard selections H1:2012qti ; H1:2014cbm . This whole process makes
use of the modernized H1 computing environment at DESY Britzger:2021xcx . Each
dataset is comprised of approximately 10 million events.

Figure 3:  Particle level distributions of the nine NN input features for the
Djangoh and Rapgap generators.

Figure 3 shows histograms of the nine features used as input for the neural
network training. These features include the energy $E$, longitudinal momentum
$p_{z}$, transverse momentum $p_{T}$, and pseudorapidity $\eta$ of the
scattered electron and the total Hadronic Final State (HFS), as well as the
difference in azimuthal angle between the two $\Delta\phi(e,{\rm HFS})$. The
HFS is quite sensitive to the $\eta$ acceptance of the detector. In order to
have HFS features that are comparable for the particle and detector
definitions, we only use generated final state particles with $|\eta|<3.8$ in
the definition of the particle-level HFS 4-vector. Both simulations provide
event weights that must be used for physics analysis. In our study, we do not
weight the simulated events in order to maximize the effective statistics of
the samples in the neural network training. The electron feature distributions
agree very well for the two simulations, while there are some visible
differences in the HFS features.

##  4 Results

Figure 4:  Neural Network output distributions for four values of the
$\lambda$ hyperparameter, which sets the scale for the Detector - Particle
disagreement penalty in the loss function. The top row shows the results for
$\lambda=0$, where there is no penalty if the NN predictions for Detector-
level input features and Particle-level input features disagree. The bottom
three rows show increasing values of $\lambda$ : 1, 20, and 100.

We now apply the method introduced in Sec. 2.2 to the DIS dataset described in
Sec. 3. Figure 4 shows the results of four neural network trainings with
different values of $\lambda$, which sets the relative weight of the MSE term
in the loss function (Eq. 2) that controls the sensitivity to detector
effects. With $\lambda=0$, the classification performance for particle-level
inputs is strong, while there are significant disagreements between the
particle and detector level neural network outputs. As $\lambda$ increases,
the particle and detector level agreement improves at the cost of weaker
classification performance. In what follows, we will use the network trained
with $\lambda=100$. For a parameter estimation task, the entire distribution
will be used for inference and therefore excellent event-by-event
classification is not required.

Figure 5:  Results of unfolding the NN output. The top (bottom) row shows the
unfolding for the Rapgap (Djangoh) generator. The left column shows the
response matrix for the unfolding, where the distribution of the NN output
given the Detector-based input features (horizontal axis) is normalized to
unit area for each bin of the NN output given the Particle-based input
features (vertical axis). The center column shows the unfolded distribution
compared to the true distribution. The right column shows the matrix of
correlation coefficients from the unfolding.

Next, we investigate how these spectra are preserved after unfolding. Figure 5
shows the results of unfolding the neural network output, given detector-level
features, to give the neural network output distribution for particle-level
features. The input distribution for the unfolding is $10^{5}$ events randomly
chosen from a histogram of the neural network output for detector-level inputs
from the simulation. The unfolding response matrices for the two simulations
agree fairly well and are concentrated along the diagonal. The output of the
unfolding shows very good agreement with the true distribution of the neural
network output given particle-level inputs, demonstrating acceptable closure
for the unfolding. The correlations in the unfolding result are mostly between
neighboring bins of the distribution.

Figure 6:  Results of testing the model dependence of the unfolding the neural
network output distribution by varying the response matrix used in the
unfolding. The top row normalizes the unfolded distribution with the true
distribution from the same simulation, testing the unfolding closure (points)
and unfolding model dependence (red histogram). The bottom row normalizes the
unfolded distribution with the true distribution from the other simulation,
showing the ability to distinguish the models.

One of the biggest challenges for the $\lambda=0$ case is that it is highly
sensitive to regions of phase space that are not well-constrained by the
detector. As a result, the output of the unfolding is highly dependent on the
simulation used in the unfolding (prior). Figure 6 shows the model dependence
of the unfolding and the ability of the neural network to perform the model
classification task. The top row of the figure shows the results of the
closure tests compared with a measure of the unfolding model dependence, where
we perform the unfolding with the response matrix from the other simulation.
The unfolded distributions have been normalized using the true distribution
from the same simulation, giving an expected flat distribution consistent with
1. The model dependence is small and generally less than 10%. The bottom row
shows the unfolded distributions instead normalized by the true distribution
from the other simulation. The degree to which the distribution deviates from
unity is a measure of the model discrimination power of the network. The
results show significant deviations from unity, indicating that the neural
network can distinguish the two simulations. The size of the deviations from
unity is large compared to the size of the model dependence of the unfolding.

Figure 7:  Results of testing the model dependence of the unfolding the HFS
$\eta$ distribution by varying the response matrix used in the unfolding. The
top row normalizes the unfolded distribution with the true distribution from
the same simulation, testing the unfolding closure (points) and unfolding
model dependence (red histogram). The bottom row normalizes the unfolded
distribution with the true distribution from the other simulation, showing the
ability to distinguish the models.

Figure 3 shows that some of the HFS variables in the input features may be
able to distinguish the two models directly. Figure 7 shows the results of
running the unfolding procedure using the HFS $\eta$ distribution for model
discrimination, where the model dependence is significantly larger, compared
to the neural network output. The shape is distorted, including deviations up
to 20%, when the response matrix from the other simulation was used in the
unfolding. Since the modeling uncertainty is comparable to or larger than the
size of the effect we are trying to probe, such observables are much less
useful than the neural network output for the inference task.

##  5 Conclusions and Outlook

Unfolded differential cross section measurements are a standard approach to
making data available for downstream inference tasks. While some measurements
can be used for a variety of tasks, often, there is a single goal that
motivate the result. In these cases, we advocate to design observables that
are tailored to the physics goal using machine learning. The output of a
neural network trained specifically for the downstream task is an observable
and its differential cross section likely contains more information than
classical observables. We have proposed a new loss function for training the
network so that the resulting observable can be well measured. We anticipate
that our new approach could be useful for a variety of scientific goals,
including measurements of fundamental parameters like the top quark mass and
tuning Monte Carlo event generators.

There are a number of ways this approach could be extended in the future. We
require that the observable have the same definition at particle level and
detector level, while additional information at detector-level like
resolutions may be useful to improve precision. A complementary strategy would
be to use all the available information to unfold the full phase space
Arratia:2021otl . Such techniques may improve the precision by integrating all
of the relevant information at detector level, but they may compromise
specific sensitivity by being broad and have no direct constraints on
measurability. It would be interesting to compare our tailored approach to
full phase space methods in the future.

## Code availability

The code in this work can be found in: https://github.com/owen234/designer-
obs-paper.

## Acknowledgments

We thank Miguel Arratia and Daniel Britzger for useful discussions and
feedback on the manuscript. Additionally, we thank our colleagues from the H1
Collaboration for allowing us to use the simulated MC event samples. Thanks to
DESY-IT and the MPI für Physik for providing some computing infrastructure and
supporting the data preservation project of the HERA experiments. B.N. was
supported by the Department of Energy, Office of Science under contract number
DE-AC02-05CH11231.

## References

  * (1) CMS Collaboration, A. Tumasyan, et al., Measurement of the differential $\hbox{t}\overline{\hbox{t}}$ production cross section as a function of the jet mass and extraction of the top quark mass in hadronic decays of boosted top quarks, Eur. Phys. J. C 83 (7) (2023) 560.  arXiv:2211.01456, doi:10.1140/epjc/s10052-023-11587-8.
  * (2) H1 Collaboration, V. Andreev, et al., Determination of the strong coupling constant $\alpha_{s}(m_{Z})$ in next-to-next-to-leading order QCD using H1 jet cross section measurements, Eur. Phys. J. C 77 (11) (2017) 791, [Erratum: Eur.Phys.J.C 81, 738 (2021)].  arXiv:1709.07251, doi:10.1140/epjc/s10052-017-5314-7.
  * (3) ZEUS Collaboration, Z. Collaboration, Measurement of jet production in deep inelastic scattering and NNLO determination of the strong coupling at ZEUS (9 2023).  arXiv:2309.02889.
  * (4) H1, ZEUS Collaboration, H. Abramowicz, et al., Combination of measurements of inclusive deep inelastic ${e^{\pm}p}$ scattering cross sections and QCD analysis of HERA data, Eur. Phys. J. C 75 (2015).  arXiv:1506.06042, doi:10.1140/epjc/s10052-015-3710-4.
  * (5) L. A. Harland-Lang, A. D. Martin, P. Motylinski, R. S. Thorne, The impact of the final HERA combined data on PDFs obtained from a global fit, Eur. Phys. J. C 76 (4) (2016) 186.  arXiv:1601.03413, doi:10.1140/epjc/s10052-016-4020-1.
  * (6) T.-J. Hou, et al., New CTEQ global analysis of quantum chromodynamics with high-precision data from the LHC, Phys. Rev. D 103 (1) (2021) 014013.  arXiv:1912.10053, doi:10.1103/PhysRevD.103.014013.
  * (7) NNPDF Collaboration, R. D. Ball, et al., The path to proton structure at 1% accuracy, Eur. Phys. J. C 82 (5) (2022) 428.  arXiv:2109.02653, doi:10.1140/epjc/s10052-022-10328-7.
  * (8) J. M. Campbell, et al., Event Generators for High-Energy Physics Experiments, in: Snowmass 2021, 2022.  arXiv:2203.11110.
  * (9) M. Arratia, D. Britzger, O. Long, B. Nachman, Optimizing observables with machine learning for better unfolding, JINST 17 (07) (2022) P07009.  arXiv:2203.16722, doi:10.1088/1748-0221/17/07/P07009.
  * (10) M. Arratia, et al., Publishing unbinned differential cross section results, JINST 17 (01) (2022) P01024.  arXiv:2109.13243, doi:10.1088/1748-0221/17/01/P01024.
  * (11) K. Datta, D. Kar, D. Roy, Unfolding with Generative Adversarial Networks (2018).  arXiv:1806.00433.
  * (12) M. Bunse, N. Piatkowski, T. Ruhe, W. Rhode, K. Morik, Unification of deconvolution algorithms for Cherenkov astronomy, in: 5th International Conference on Data Science and Advanced Analytics (DSAA), IEEE, 2018, pp. 21–30.
  * (13) T. Ruhe, T. Voigt, M. Wornowizki, M. Börner, W. Rhode, K. Morik, Mining for spectra - the dortmund spectrum estimation algorithm, 2019.
  * (14) A. Andreassen, P. T. Komiske, E. M. Metodiev, B. Nachman, J. Thaler, OmniFold: A Method to Simultaneously Unfold All Observables, Phys. Rev. Lett. 124 (2020) 182001.  arXiv:1911.09107, doi:10.1103/PhysRevLett.124.182001.
  * (15) M. Bellagente, A. Butter, G. Kasieczka, T. Plehn, R. Winterhalder, How to GAN away Detector Effects (2019).  arXiv:1912.00477, doi:10.21468/SciPostPhys.8.4.070.
  * (16) M. Bellagente, A. Butter, G. Kasieczka, T. Plehn, A. Rousselot, R. Winterhalder, Invertible Networks or Partons to Detector and Back Again (Jun 2020).  arXiv:2006.06685, doi:10.21468/SciPostPhys.9.5.074.
  * (17) M. Vandegar, M. Kagan, A. Wehenkel, G. Louppe, Neural Empirical Bayes: Source Distribution Estimation and its Applications to Simulation-Based Inference, in: A. Banerjee, K. Fukumizu (Eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, Vol. 130 of Proceedings of Machine Learning Research, PMLR, 2021, pp. 2107–2115.  arXiv:2011.05836.
URL https://proceedings.mlr.press/v130/vandegar21a.html

  * (18) A. Andreassen, P. T. Komiske, E. M. Metodiev, B. Nachman, A. Suresh, J. Thaler, Scaffolding Simulations with Deep Learning for High-dimensional Deconvolution, in: 9th International Conference on Learning Representations, 2021.  arXiv:2105.04448.
  * (19) J. N. Howard, S. Mandt, D. Whiteson, Y. Yang, Foundations of a Fast, Data-Driven, Machine-Learned Simulator (Jan 2021).  arXiv:2101.08944.
  * (20) M. Backes, A. Butter, M. Dunford, B. Malaescu, An unfolding method based on conditional Invertible Neural Networks (cINN) using iterative training (12 2022).  arXiv:2212.08674.
  * (21) J. Chan, B. Nachman, Unbinned profiled unfolding, Phys. Rev. D 108 (1) (2023) 016002\.  arXiv:2302.05390, doi:10.1103/PhysRevD.108.016002.
  * (22) A. Shmakov, K. Greif, M. Fenton, A. Ghosh, P. Baldi, D. Whiteson, End-To-End Latent Variational Diffusion Models for Inverse Problems in High Energy Physics (5 2023).  arXiv:2305.10399.
  * (23) T. Alghamdi, et al., Toward a generative modeling analysis of CLAS exclusive $2\pi$ photoproduction (7 2023).  arXiv:2307.04450.
  * (24) H1 Collaboration, V. Andreev, et al., Measurement of lepton-jet correlation in deep-inelastic scattering with the H1 detector using machine learning for unfolding (Aug 2021).  arXiv:2108.12376.
  * (25) H1 Collaboration, Machine learning-assisted measurement of multi-differential lepton-jet correlations in deep-inelastic scattering with the H1 detector, H1prelim-22-031 (2022).
URL
https://www-h1.desy.de/h1/www/publications/htmlsplit/H1prelim-22-031.long.html

  * (26) H1 Collaboration, V. Andreev, et al., Unbinned Deep Learning Jet Substructure Measurement in High $Q^{2}$ ep collisions at HERA (3 2023).  arXiv:2303.13620.
  * (27) H1 Collaboration, Measurement of lepton-jet correlations in high $Q^{2}$ neutral-current DIS with the H1 detector at HERA, H1prelim-21-031 (2021).
URL
https://www-h1.desy.de/h1/www/publications/htmlsplit/H1prelim-21-031.long.html

  * (28) LHCb Collaboration, Multidifferential study of identified charged hadron distributions in $Z$-tagged jets in proton-proton collisions at $\sqrt{s}=$13 TeV (8 2022).  arXiv:2208.11691.
  * (29) P. T. Komiske, S. Kryhin, J. Thaler, Disentangling quarks and gluons in CMS open data, Phys. Rev. D 106 (9) (2022) 094021.  arXiv:2205.04459, doi:10.1103/PhysRevD.106.094021.
  * (30) STAR Collaboration, Y. Song, Measurement of CollinearDrop jet mass and its correlation with SoftDrop groomed jet substructure observables in $\sqrt{s}=200$ GeV $pp$ collisions by STAR (7 2023).  arXiv:2307.07718.
  * (31) J. Brehmer, K. Cranmer, G. Louppe, J. Pavez, Constraining Effective Field Theories with Machine Learning, Phys. Rev. Lett. 121 (11) (2018) 111801.  arXiv:1805.00013, doi:10.1103/PhysRevLett.121.111801.
  * (32) J. Brehmer, K. Cranmer, G. Louppe, J. Pavez, A Guide to Constraining Effective Field Theories with Machine Learning, Phys. Rev. D 98 (5) (2018) 052004.  arXiv:1805.00020, doi:10.1103/PhysRevD.98.052004.
  * (33) J. Brehmer, G. Louppe, J. Pavez, K. Cranmer, Mining gold from implicit models to improve likelihood-free inference, Proc. Nat. Acad. Sci. 117 (10) (2020) 5242–5249.  arXiv:1805.12244, doi:10.1073/pnas.1915980117.
  * (34) P. De Castro, T. Dorigo, INFERNO: Inference-Aware Neural Optimisation, Comput. Phys. Commun. 244 (2019) 170–179.  arXiv:1806.04743, doi:10.1016/j.cpc.2019.06.007.
  * (35) A. Elwood, D. Krücker, Direct optimisation of the discovery significance when training neural networks to search for new physics in particle colliders (6 2018).  arXiv:1806.00322.
  * (36) A. Andreassen, B. Nachman, Neural Networks for Full Phase-space Reweighting and Parameter Tuning, Phys. Rev. D 101 (9) (2020) 091901.  arXiv:1907.08209, doi:10.1103/PhysRevD.101.091901.
  * (37) J. Brehmer, F. Kling, I. Espejo, K. Cranmer, MadMiner: Machine learning-based inference for particle physics, Comput. Softw. Big Sci. 4 (1) (2020) 3.  arXiv:1907.10621, doi:10.1007/s41781-020-0035-2.
  * (38) S. Wunsch, S. Jörger, R. Wolf, G. Quast, Optimal Statistical Inference in the Presence of Systematic Uncertainties Using Neural Network Optimization Based on Binned Poisson Likelihoods with Nuisance Parameters, Comput. Softw. Big Sci. 5 (1) (2021) 4.  arXiv:2003.07186, doi:10.1007/s41781-020-00049-5.
  * (39) A. Ghosh, B. Nachman, D. Whiteson, Uncertainty-aware machine learning for high energy physics, Phys. Rev. D 104 (5) (2021) 056026.  arXiv:2105.08742, doi:10.1103/PhysRevD.104.056026.
  * (40) R. Gomez Ambrosio, J. ter Hoeve, M. Madigan, J. Rojo, V. Sanz, Unbinned multivariate observables for global SMEFT analyses from machine learning, JHEP 03 (2023) 033.  arXiv:2211.02058, doi:10.1007/JHEP03(2023)033.
  * (41) N. Simpson, L. Heinrich, neos: End-to-End-Optimised Summary Statistics for High Energy Physics, J. Phys. Conf. Ser. 2438 (1) (2023) 012105.  arXiv:2203.05570, doi:10.1088/1742-6596/2438/1/012105.
  * (42) A. Blance, M. Spannowsky, P. Waite, Adversarially-trained autoencoders for robust unsupervised new physics searches, JHEP 10 (2019) 047.  arXiv:1905.10384, doi:10.1007/JHEP10(2019)047.
  * (43) C. Englert, P. Galler, P. Harris, M. Spannowsky, Machine Learning Uncertainties with Adversarial Neural Networks, Eur. Phys. J. C79 (1) (2019) 4\.  arXiv:1807.08763, doi:10.1140/epjc/s10052-018-6511-8.
  * (44) G. Louppe, M. Kagan, K. Cranmer, Learning to Pivot with Adversarial Networks, in: I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems, Vol. 30, Curran Associates, Inc., 2017\.  arXiv:1611.01046.
URL
https://papers.nips.cc/paper/2017/hash/48ab2f9b45957ab574cf005eb8a76760-Abstract.html

  * (45) J. Dolen, P. Harris, S. Marzani, S. Rappoccio, N. Tran, Thinking outside the ROCs: Designing Decorrelated Taggers (DDT) for jet substructure, JHEP 05 (2016) 156.  arXiv:1603.00027, doi:10.1007/JHEP05(2016)156.
  * (46) I. Moult, B. Nachman, D. Neill, Convolved Substructure: Analytically Decorrelating Jet Substructure Observables (2017).  arXiv:1710.06859.
  * (47) J. Stevens, M. Williams, uBoost: A boosting method for producing uniform selection efficiencies from multivariate classifiers, JINST 8 (2013) P12013.  arXiv:1305.7248, doi:10.1088/1748-0221/8/12/P12013.
  * (48) C. Shimmin, P. Sadowski, P. Baldi, E. Weik, D. Whiteson, E. Goul, A. Sogaard, Decorrelated Jet Substructure Tagging using Adversarial Neural Networks, Phys. Rev. D96 (7) (2017) 074034.  arXiv:1703.03507, doi:10.1103/PhysRevD.96.074034.
  * (49) L. Bradshaw, R. K. Mishra, A. Mitridate, B. Ostdiek, Mass Agnostic Jet Taggers (2019).  arXiv:1908.08959, doi:10.21468/SciPostPhys.8.1.011.
  * (50) Performance of mass-decorrelated jet substructure observables for hadronic two-body decay tagging in ATLAS, Tech. Rep. ATL-PHYS-PUB-2018-014, CERN, Geneva (Jul 2018).
URL https://cds.cern.ch/record/2630973

  * (51) G. Kasieczka, D. Shih, DisCo Fever: Robust Networks Through Distance Correlation (2020).  arXiv:2001.05310, doi:10.1103/PhysRevLett.125.122001.
  * (52) S. Wunsch, S. Jórger, R. Wolf, G. Quast, Reducing the dependence of the neural network function to systematic uncertainties in the input space (2019).  arXiv:1907.11674, doi:10.1007/s41781-020-00037-9.
  * (53) A. Rogozhnikov, A. Bukva, V. Gligorov, A. Ustyuzhanin, M. Williams, New approaches for boosting to uniformity, JINST 10 (03) (2015) T03002.  arXiv:1410.4140, doi:10.1088/1748-0221/10/03/T03002.
  * (54) C. Collaboration, A deep neural network to search for new long-lived particles decaying to jets, Machine Learning: Science and Technology (2020).  arXiv:1912.12238, doi:10.1088/2632-2153/ab9023.
  * (55) G. Kasieczka, B. Nachman, M. D. Schwartz, D. Shih, ABCDisCo: Automating the ABCD Method with Machine Learning (Jul 2020).  arXiv:2007.14400, doi:10.1103/PhysRevD.103.035021.
  * (56) O. Kitouni, B. Nachman, C. Weisser, M. Williams, Enhancing searches for resonances with machine learning and moment decomposition (Oct 2020).  arXiv:2010.09745.
  * (57) V. Estrade, C. Germain, I. Guyon, D. Rousseau, Systematic aware learning - A case study in High Energy Physics, EPJ Web Conf. 214 (2019) 06024.  doi:10.1051/epjconf/201921406024.
  * (58) J. A. Aguilar-Saavedra, J. H. Collins, R. K. Mishra, A generic anti-QCD jet tagger, JHEP 11 (2017) 163.  arXiv:1709.01087, doi:10.1007/JHEP11(2017)163.
  * (59) J. A. Aguilar-Saavedra, F. R. Joaquim, J. F. Seabra, Mass Unspecific Supervised Tagging (MUST) for boosted jets (2020).  arXiv:2008.12792, doi:10.1007/JHEP03(2021)012.
  * (60) F. Chollet, Keras, https://github.com/fchollet/keras (2015).
  * (61) M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., Tensorflow: A system for large-scale machine learning., in: OSDI, Vol. 16, 2016, pp. 265–283.
  * (62) D. Kingma, J. Ba, Adam: A method for stochastic optimization (2014).  arXiv:1412.6980.
  * (63) S. Schmitt, TUnfold: an algorithm for correcting migration effects in high energy physics, JINST 7 (2012) T10003.  arXiv:1205.6201, doi:10.1088/1748-0221/7/10/T10003.
  * (64) R. Brun, F. Rademakers, ROOT: An object oriented data analysis framework, Nucl. Instrum. Meth. A 389 (1997) 81–86.  doi:10.1016/S0168-9002(97)00048-X.
  * (65) H1 Collaboration, I. Abt, et al., The Tracking, calorimeter and muon detectors of the H1 experiment at HERA, Nucl. Instrum. Meth. A 386 (1997) 348–396.  doi:10.1016/S0168-9002(96)00894-7.
  * (66) H1 Collaboration, I. Abt, et al., The H1 detector at HERA, Nucl. Instrum. Meth. A 386 (1997) 310–347.  doi:10.1016/S0168-9002(96)00893-5.
  * (67) M. Arratia, D. Britzger, O. Long, B. Nachman, Reconstructing the kinematics of deep inelastic scattering with deep learning, Nucl. Instrum. Meth. A 1025 (2022) 166164.  arXiv:2110.05505, doi:10.1016/j.nima.2021.166164.
  * (68) H. Jung, Hard diffractive scattering in high-energy e p collisions and the Monte Carlo generator RAPGAP, Comput. Phys. Commun. 86 (1995) 147–161.  doi:10.1016/0010-4655(94)00150-Z.
  * (69) K. Charchula, G. A. Schuler, H. Spiesberger, Combined QED and QCD radiative effects in deep inelastic lepton - proton scattering: The Monte Carlo generator DJANGO6, Comput. Phys. Commun. 81 (1994) 381–402.  doi:10.1016/0010-4655(94)90086-8.
  * (70) H. Spiesberger, et al., Radiative corrections at HERA (1992) 798–839.
URL https://cds.cern.ch/record/237380

  * (71) A. Kwiatkowski, H. Spiesberger, H. J. Mohring, Characteristics of radiative events in deep inelastic _ep_ scattering at HERA, Z. Phys. C 50 (1991) 165–178.  doi:10.1007/BF01558572.
  * (72) A. Kwiatkowski, H. Spiesberger, H. J. Mohring, Heracles: An Event Generator for $ep$ Interactions at HERA Energies Including Radiative Processes: Version 1.0, Comput. Phys. Commun. 69 (1992) 155–172.  doi:10.1016/0010-4655(92)90136-M.
  * (73) S. Agostinelli, et al., Geant4 \- a simulation toolkit, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 506 (3) (2003) 250 – 303.  doi:https://doi.org/10.1016/S0168-9002(03)01368-8.
URL http://www.sciencedirect.com/science/article/pii/S0168900203013688

  * (74) M. Peez, Search for deviations from the standard model in high transverse energy processes at the electron proton collider HERA. (Thesis, Univ. Lyon), Ph.D. thesis (Jun 2003).
  * (75) S. Hellwig, Untersuchung der $D^{*}$ \- $\pi$ slow Double Tagging Methode in Charmanalysen, Diploma thesis, Univ. Hamburg (Jun 2004).
  * (76) B. Portheault, First measurement of charged and neutral current cross sections with the polarized positron beam at HERA II and QCD-electroweak analyses. (Thesis, Univ. Paris XI), Ph.D. thesis (Mar 2005).
  * (77) H1 Collaboration, F. D. Aaron, et al., Inclusive Deep Inelastic Scattering at High $Q^{2}$ with Longitudinally Polarised Lepton Beams at HERA, JHEP 09 (2012) 061.  arXiv:1206.7007, doi:10.1007/JHEP09(2012)061.
  * (78) H1 Collaboration, V. Andreev, et al., Measurement of multijet production in $ep$ collisions at high $Q^{2}$ and determination of the strong coupling $\alpha_{s}$, Eur. Phys. J. C 75 (2) (2015) 65.  arXiv:1406.4709, doi:10.1140/epjc/s10052-014-3223-6.
  * (79) D. Britzger, S. Levonian, S. Schmitt, D. South, Preservation through modernisation: The software of the H1 experiment at HERA, EPJ Web Conf. 251 (2021) 03004.  arXiv:2106.11058, doi:10.1051/epjconf/202125103004.


11institutetext: Department of Computer Science, Vanderbilt
University22institutetext: Department of Electrical and Computer Engineering,
Vanderbilt University
22email<EMAIL_ADDRESS>
# Enhancing Data Diversity for Self-training Based Unsupervised Cross-modality
Vestibular Schwannoma and Cochlea Segmentation

Han Liu equal contribution11 Yubo Fan* 11 Ipek Oguz 1122 Benoit M. Dawant 22

###### Abstract

Automatic segmentation of vestibular schwannoma (VS) and cochlea from magnetic
resonance imaging can facilitate VS treatment planning. Unsupervised
segmentation methods have shown promising results without requiring the time-
consuming and laborious manual labeling process. In this paper, we present an
approach for VS and cochlea segmentation in an unsupervised domain adaptation
setting. Specifically, we first develop a cross-site cross-modality unpaired
image translation strategy to enrich the diversity of the synthesized data.
Then, we devise a rule-based offline augmentation technique to further
minimize the domain gap. Lastly, we adopt a self-configuring segmentation
framework empowered by self-training to obtain the final results. On the
CrossMoDA 2022 validation leaderboard, our method has achieved competitive VS
and cochlea segmentation performance with mean Dice scores of 0.8178 ± 0.0803
and 0.8433 ± 0.0293, respectively.

###### Keywords:

Vestibular schwannoma Cochlea Unsupervised domain adaptation Self-training

##  1 Introduction

Vestibular schwannoma (VS) is a benign tumor that stems from an overproduction
of Schwann cells. It develops from the vestibular nerve which connects the
brain and the inner ear and its common symptoms include hearing loss,
dizziness, and tinnitus [19]. Magnetic resonance imaging (MRI) is crucial for
diagnosis and surveillance of VS and contrast enhanced T1 weighted (ceT1) MRI
is currently the most commonly used protocol. However, this involves
gadolinium, which may produce side effects ranging from mild to severe. As a
possible noncontrast and lower-cost alternative, high-resolution T2-weighted
(hrT2) imaging has shown promises for follow-up surveillance scans [2, 17,
15].

To facilitate the clinical workflow, automatic methods to segment the VS in
ceT1 and hrT2 have recently emerged [17, 20, 24]. However, training supervised
VS segmentation models requires manual annotation, which is expensive and
time-consuming. Weakly-supervised [4] and unsupervised VS segmentation methods
[5] have thus drawn increasing interest in the community. In the CrossMoDA
2021 challenge [5], participants were given the task of unsupervised cross-
modality VS and cochlea segmentation, i.e., segmenting these two structures in
hrT2, but annotations were only provided in unpaired ceT1 images in the
training set. In the CrossMoDA 2022 challenge, an additional set of ceT1 and
hrT2 images are acquired at another MRI site. Therefore, two types of domain
gaps exist in this challenge due to the difference in (1) acquisition sites,
i.e., site A vs. site B, and (2) MRI modalities, i.e., ceT1 vs. hrT2, as shown
in Figure 1. Both domain gaps need to be addressed to achieve robust
segmentation performance in CrossMoDA 2022.

In this paper, we present our solution to the segmentation task of CrossMoDA
2022. We approach this task as an unsupervised domain adaptation (UDA) problem
where we first train cross-site unpaired image translation models to generate
pseudo target domain (hrT2) images, then apply a rule-based augmentation to
the pseudo hrT2 images, and finally train nnU-Net [9] segmentation models
using a self-training scheme. The results on the challenge leaderboard showed
that our method has achieved promising segmentation performances on both VS
and cochlea.

Figure 1: Two types of domain gaps exist in CrossMoDA 2022 due to the
difference in (1) acquisition sites and (2) MRI modalities.

##  2 Related Work

###  2.1 Unsupervised Domain Adaptation (UDA)

The performance of machine learning models can be affected by data
distribution shift between the training/source dataset and test/target dataset
[7]. UDA refers to the task of improving model performance on target domain
data when their label is not available. Feature alignment-based methods aim to
learn domain-invariant features across different domains. Domain Adversarial
Neural Network (DANN) [6] is a representative architecture that utilizes
adversarial learning between a feature extractor and a domain discriminator.
The domain discriminator learns to differentiate whether the extracted
features are from the source or target domain, and the feature extractor
learns domain-invariant features such that it can fool the discriminator.
Another way to align features is by optimizing some divergence metrics [14,
21] between the source and target domains. With the emergence of unpaired
image-to-image translation approaches such as CycleGAN [23] and UNIT [13],
image-level alignment is also used to tackle the UDA problem [8]. By utilizing
a cycle-consistency loss between the input and the reconstructed images, the
models are trained without paired data. Then, image-level alignment can be
achieved by generating pseudo target domain images from the source domain
images. In this work, we adopt image-level alignment paradigm to bridge the
domain gap between ceT1 and hrT2.

###  2.2 Self-training in UDA

Self-training strategies have shown promising results in the field of UDA by
fully utilizing the unlabeled target domain data. It has been shown to improve
the performance of the segmentation models by fine-tuning them on the target
images with pseudo labels. Zou et al. [25] propose a confidence regularized
self-training framework which encourages the smoothness of the network output
and reduces the confidence in false positives during training. Yu et al. [22]
use a portion of the pseudo labels with high probability to iteratively fine-
tune the model and achieve superior performance on a UDA dataset. This
strategy is also used by the top teams in CrossMoDA 2021 [5], further
demonstrating its effectiveness in the UDA problem.

##  3 Methods

###  3.1 Overview

In this study, we propose to tackle the UDA segmentation problem by following
the popular “synthesis-then-segmentation” training strategy [3, 18, 1, 12].
Specifically, we perform unpaired image translation to synthesize the pseudo
target domain images in a cross-site and cross-modality fashion. Then we
devise a rule-based offline augmentation method to further increase the data
variability. Lastly, we adopt the nnU-Net framework and perform self-training
to train a target-domain segmentation model with both the synthetic and real
target domain data.

Figure 2: The diagram of cross-site cross-modality unpaired image translation.

###  3.2 Cross-site Cross-modality Unpaired Image Translation

Since our task is to segment the VS and cochlea in hrT2 images while their
labels are only available in the unpaired ceT1 images, we adopt an unpaired
image translation method, i.e., CycleGAN [23], to bridge the gap between the
two modalities and synthesize pseudo hrT2 from ceT1. Then the pseudo hrT2
images with the annotations from the corresponding ceT1 can be utilized to
train segmentation models. However, MRIs (both ceT1 and hrT2) in the provided
dataset were acquired from two different sites, and we observe slight
appearance differences within each modality (especially hrT2) across sites. To
enrich the diversity of the pseudo hrT2 images, we synthesize them using ceT1
within and across sites. Specifically, as shown in Figure 2, five CycleGAN
models are trained with different source/target domain data configurations to
generate pseudo hrT2 images. We use subscripts to denote images from site A or
site B, e.g., hrT2A means hrT2 from site A. Compared to only generating
within-site pseudo images, i.e., pseudo hrT2A from ceT1A (CycleGAN #1) and
pseudo hrT2B from ceT1B (CycleGAN #4), our cross-site training scheme can
generate twice more pseudo hrT2 data for the downstream task of nnU-Net
segmentation. Note that CycleGAN #5 has the same network architecture as the
other four, and the only difference is that its training set contains images
from both site A and site B.

###  3.3 Rule-based Offline Augmentation for VS and Cochlea

Thanks to the unpaired image translation, the domain gap between ceT1 and hrT2
can be substantially reduced at the image-level by generating pseudo hrT2
images. However, there is still a domain gap between the pseudo and real hrT2,
as we observe that the cochlea and VS in the pseudo hrT2 images do not have
the same intensity characteristics as in the real hrT2 images. To overcome
this issue, we propose to adjust the intensities of the VS and cochlea regions
in pseudo hrT2 images to further minimize the domain gap, as shown in Figure
3. As suggested by [1], for VS, we reduce the signal intensity of the voxels
that are labeled as VS by 50% to further increase the heterogeneity of VS
signals [1]. This VS augmentation is randomly applied to 50% of our pseudo
hrT2 images. Cochleae typically have high signal intensities in hrT2 images,
with values that were empirically found to be within the [85th, 95th]
intensity percentile. However, we observe that the generated cochlea in pseudo
hrT2 images can have low intensities or can even be absent. To improve the
appearance of pseudo hrT2 images with a mean cochlea intensity lower than the
85th intensity percentile, we replace the original intensity of the cochlea
voxels by a value randomly sampled from a uniform distribution, which is
bounded by the 85th and 95th intensity percentile of that image. Lastly, the
augmented cochlea region is smoothed by a 3D Gaussian kernel to further refine
the cochlea appearance. The cochlea augmentation is applied to all the pseudo
hrT2 images.

Figure 3: An illustration of the rule-based offline augmentation technique on
two cases. Orange and blue arrows represent the augmented cochlea and VS,
respectively.

###  3.4 Segmentation with nnU-Net and Self-training

With the augmented pseudo hrT2 images, we can train a segmentation model
supervised by the paired ceT1 labels. Here, we utilize a popular self-
configuring segmentation framework, i.e., nnU-Net, for supervised learning.
However, with this approach the real hrT2 images are not involved in training
the segmentation model due to the lack of labels. To tackle this problem, we
adopt a self-training strategy to make use of the unlabeled real hrT2 images.
Specifically, we firstly use the model trained on pseudo hrT2 images to obtain
the pseudo labels of the unlabeled real hrT2 images. Next, we re-train the
nnU-Net with the combined data, i.e., pseudo hrT2 images with real ceT1 labels
and real hrT2 images with pseudo labels. Note that we perform self-training
several (in our case, 3) times, since we observe improvements on the
validation leaderboard each time the network is re-trained with the updated
pseudo labels produced by self-training.

##  4 Experiments and Results

###  4.1 Dataset

The dataset was released by the MICCAI challenge CrossMoDA 2022 [16]. 105 ceT1
and 105 hrT2 unpaired images were obtained on a 32-channel Siemens Avanto 1.5T
scanner, which are called “London data” or site A data. Another 105 ceT1 and
105 hrT2 unpaired images were obtained on a Philips Ingenia 1.5T scanner,
which are called “Tilburg data” or site B data. Image resolutions are
different for images from different sequences and sites. The manually
segmented mask of the VS and cochlea for the 210 ceT1 images are provided.

###  4.2 Pre-processing and Network Training

We observe that the fields of view and resolutions vary substantially across
different sites and modalities. Inspired by [12], to discard the irrelevant
brain regions in our task, we first crop each MRI scan into a region of
interest (ROI) by rigid registration with an atlas. Specifically, we use 4
atlases (2 sites × 2 modalities) and we register each scan with the atlas from
the same site and modality. Moreover, to avoid losing information from high-
resolution images, we resample all the images and masks to the highest
resolution of the images in the challenge dataset, i.e., [0.4102, 0.4102, 1.0]
mm.

For CycleGANs, we train the 5 models (as in Figure 2) in both 2D and 3D
settings. The patch size for 2D CycleGANs is 256 $\times$ 256 and for 3D is
112 $\times$ 112 $\times$ 24\. Random cropping or zero padding is used when
the dimension of the resampled ROI does not match the patch size during
training. We use Adam optimizer [10] with a fixed learning rate of 2e-4. The
training is stopped after 100 epochs for the 2D CycleGANs and after 1000
epochs for the 3D CycleGANs. Sliding window inference with an overlapping
ratio of 0.8 is used to generate the final synthesis results. Synthesized 2D
slices are then merged into the 3D volumes based on their original positions.

For nnU-Net training, we use the network architecture and patch size provided
by the 3D fullres mode. Five-fold cross-validation is used. The Dice loss +
cross-entropy loss is used as the loss function. The SGD optimizer with an
initial learning rate of 1e-2 is used and the learning rate is decayed by a
polynomial function. We do not apply connected component analysis to post-
process the segmentation results, as we find that the impact of this post-
processing operation on the segmentation results varies across folds.

Overall, we use 4 training stages. In stage 1, we train the nnU-Net with the
pseudo hrT2 images generated from 2D and 3D CycleGANs. In stage 2, we replace
the images from 2D CycleGANs with real hrT2 images for self-training. In stage
3, we apply offline VS augmentation and oversample the pseudo hrT2 images with
small tumors, as we observe on the validation leaderboard that the small-tumor
segmentation performance of our network is low. We oversample the pseudo hrT2
images with small tumors based on the VS labels from ceT1 till the training
data size reaches 1000 (maximum number that nnU-Net allows). In stage 4, we
perform another round of self-training and use the ensemble model from stages
2, 3, and 4 as our final model by averaging softmax probabilities.

Figure 4: Synthesis results of the cross-site cross-modality unpaired image
translation. First row: source domain (ceT1) image. Second row: pseudo target
domain (hrT2) images. Third row: real target domain (hrT2) images as
reference.

###  4.3 Experimental Results

####  4.3.1 Synthesis Results.

In Figure 4, we qualitatively show the effectiveness of the cross-site cross-
modality unpaired image translation. We select a representative ceT1 image
from site A and generate the pseudo hrT2 images using CycleGAN #1, #2, and #5,
which generate pseudo hrT2 from site A, site B, and a combination of site A
and site B, respectively. Two representative real hrT2 images from site A and
from site B are provided in Figure 4 for comparison. We observe the
consistency of the overall image contrast characteristics between the pseudo
and real images from the same site.

####  4.3.2 Segmentation Results.

For quantitative evaluation, we submit our segmentation results to the
validation leaderboard, where the Dice score and average symmetric surface
distance (ASSD) between segmentation results and the ground truth are
computed. In Table 1, we show the segmentation results for both VS and cochlea
obtained at each training stage (the configurations are described in detail in
Section 3.2). We notice that the largest improvement is observed from stage 1
to stage 2, when the model is trained on real target domain images for the
first time. Moreover, the effectiveness of our proposed offline VS
augmentation and small-tumor oversampling is demonstrated by the improvement
observed from stage 2 to stage 3. Lastly, the mean Dice scores and ASSDs
achieved by our final ensemble model are 0.8178 ± 0.0803 and 0.8433 ± 0.0293,
and 0.6673 ± 0.2713 mm and 0.2053 ± 0.1489 mm for VS and cochlea,
respectively. For qualitative evaluation, we visualize the segmentation
results in Figure 5 for four representative cases (a and b are from site A; c
and d are from site B). We posit that the poor VS segmentation, i.e., f and h,
is due to the uncommon appearances of VS in the validation set, which can be
difficult to synthesize if such appearances rarely exist in our target domain
training dataset.

Figure 5: Qualitative results on the validation set. The first row and the second row correspond to the hrT2 images and their segmentation results, respectively. Note that a, b and c, d are from different sites. Table 1: Quantitative results on the validation leaderboard |  | Dice$\uparrow$ (%) | ASSD$\downarrow$ (mm)
---|---|---|---
| Stage | VS | Cochlea | VS | Cochlea
| 1 | 73.54±24.26 | 81.91±4.33 | 1.89±8.41 | 0.25±0.15
| 2 | 77.97±18.29 | 82.42±4.01 | 0.72±0.29 | 0.23±0.15
| 3 | 80.37±9.64 | 84.46±2.97 | 1.14±1.73 | 0.22±0.18
| ensemble | 81.78±8.03 | 84.33±2.93 | 0.67±0.27 | 0.21±0.15

##  5 Discussion and Conclusion

In the CrossMoDA 2021 challenge, we have observed that there were three
important components used by the top teams including (1) segmentation via nnU-
Net [1, 3, 18], (2) self-training with real target domain data [18, 3], and
(3) data augmentation for VS and cochlea [1, 11]. Based on this observation,
our proposed method for CrossMoDA 2022 also incorporates these components to
achieve competitive performance on the leaderboard. To address the additional
domain gap (multi-site MRI) in CrossMoDA 2022, we synthesize site-specific
pseudo target domain images with multiple CycleGAN models. Nevertheless, we
conjecture that the segmentation performance might be further improved if we
also train site-specific target domain segmentation models; this will be
investigated.

In conclusion, we proposed a solution to tackle the UDA problem for VS and
cochlea segmentation in CrossMoDA 2022. We developed a cross-site cross-
modality unpaired image translation strategy to enrich the diversity of the
synthesized data and a rule-based offline augmentation method to further
minimize the domain gap. Lastly, we empowered the nnU-Net by self-training to
make use of the unlabeled data. According to the validation leaderboard, our
method has achieved a promising segmentation performance on both VS and
cochlea.

##  6 Acknowledgements

This work was supported in part by the NIH grants NIDCD-DC014462, NIDCD-
DC008408, NIBIB-T32EB021937, and by the Advanced Computing Center for Research
and Education (ACCRE) of Vanderbilt University. This publication contents are
solely the responsibility of the authors and do not necessarily represent the
official views of the founding institutes.

## References

  * [1] Choi, J.W.: Using out-of-the-box frameworks for unpaired image translation and image segmentation for the crossmoda challenge. arXiv e-prints pp. arXiv–2110 (2021)
  * [2] Coelho, D.H., Tang, Y., Suddarth, B., Mamdani, M.: Mri surveillance of vestibular schwannomas without contrast enhancement: clinical and economic evaluation. The Laryngoscope 128(1), 202–209 (2018)
  * [3] Dong, H., Yu, F., Zhao, J., Dong, B., Zhang, L.: Unsupervised domain adaptation in semantic segmentation based on pixel alignment and self-training. arXiv preprint arXiv:2109.14219 (2021)
  * [4] Dorent, R., Joutard, S., Shapey, J., Bisdas, S., Kitchen, N., Bradford, R., Saeed, S., Modat, M., Ourselin, S., Vercauteren, T.: Scribble-based domain adaptation via co-segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 479–489. Springer (2020)
  * [5] Dorent, R., Kujawa, A., Ivory, M., Bakas, S., Rieke, N., Joutard, S., Glocker, B., Cardoso, J., Modat, M., Batmanghelich, K., Belkov, A., Calisto, M.B., Choi, J.W., Dawant, B.M., Dong, H., Escalera, S., Fan, Y., Hansen, L., Heinrich, M.P., Joshi, S., Kashtanova, V., Kim, H.G., Kondo, S., Kruse, C.N., Lai-Yuen, S.K., Li, H., Liu, H., Ly, B., Oguz, I., Shin, H., Shirokikh, B., Su, Z., Wang, G., Wu, J., Xu, Y., Yao, K., Zhang, L., Ourselin, S., Shapey, J., Vercauteren, T.: Crossmoda 2021 challenge: Benchmark of cross-modality domain adaptation techniques for vestibular schwannoma and cochlea segmentation (2022)
  * [6] Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V.: Domain-Adversarial Training of Neural Networks, p. 189–209. Advances in Computer Vision and Pattern Recognition, Springer International Publishing, Cham (2017), http://link.springer.com/10.1007/978-3-319-58347-1_10
  * [7] Guan, H., Liu, M.: Domain adaptation for medical image analysis: A survey. IEEE Transactions on Biomedical Engineering 69(3), 1173–1185 (Mar 2022). https://doi.org/10.1109/TBME.2021.3117407
  * [8] Hoffman, J., Tzeng, E., Park, T., Zhu, J.Y., Isola, P., Saenko, K., Efros, A., Darrell, T.: Cycada: Cycle-consistent adversarial domain adaptation. In: Proceedings of the 35th International Conference on Machine Learning. p. 1989–1998. PMLR (Jul 2018), https://proceedings.mlr.press/v80/hoffman18a.html
  * [9] Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)
  * [10] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv arXiv:1412.6980 (Jan 2017), http://arxiv.org/abs/1412.6980, arXiv:1412.6980 [cs]
  * [11] Li, H., Hu, D., Zhu, Q., Larson, K.E., Zhang, H., Oguz, I.: Unsupervised cross-modality domain adaptation for segmenting vestibular schwannoma and cochlea with data augmentation and model ensemble. In: International MICCAI Brainlesion Workshop. pp. 518–528. Springer (2022)
  * [12] Liu, H., Fan, Y., Cui, C., Su, D., McNeil, A., Dawant, B.M.: Unsupervised Domain Adaptation for Vestibular Schwannoma and Cochlea Segmentation via Semi-supervised Learning and Label Fusion. In: Crimi, A., Bakas, S. (eds.) Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. pp. 529–539. Springer International Publishing, Cham (2022)
  * [13] Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017), https://proceedings.neurips.cc/paper/2017/hash/dc6a6489640ca02b0d42dabeb8e46bb7-Abstract.html
  * [14] Long, M., Cao, Y., Wang, J., Jordan, M.I.: Learning transferable features with deep adaptation networks. arXiv arXiv:1502.02791 (May 2015), http://arxiv.org/abs/1502.02791, arXiv:1502.02791 [cs]
  * [15] Pizzini, F.B., Sarno, A., Galazzo, I.B., Fiorino, F., Aragno, A.M., Ciceri, E., Ghimenton, C., Mansueto, G.: Usefulness of high resolution t2-weighted images in the evaluation and surveillance of vestibular schwannomas? is gadolinium needed? Otology & Neurotology 41(1), e103–e110 (2020)
  * [16] Shapey, J., Kujawa, A., Dorent, R., Wang, G., Dimitriadis, A., Grishchuk, D., Paddick, I., Kitchen, N., Bradford, R., Saeed, S.R., et al.: Segmentation of vestibular schwannoma from mri, an open annotated dataset and baseline algorithm. Scientific Data 8(1), 1–6 (2021)
  * [17] Shapey, J., Wang, G., Dorent, R., Dimitriadis, A., Li, W., Paddick, I., Kitchen, N., Bisdas, S., Saeed, S.R., Ourselin, S., et al.: An artificial intelligence framework for automatic segmentation and volumetry of vestibular schwannomas from contrast-enhanced t1-weighted and high-resolution t2-weighted mri. Journal of neurosurgery 134(1), 171–179 (2019)
  * [18] Shin, H., Kim, H., Kim, S., Jun, Y., Eo, T., Hwang, D.: Cosmos: Cross-modality unsupervised domain adaptation for 3d medical image segmentation based on target-aware domain translation and iterative self-training. arXiv preprint arXiv:2203.16557 (2022)
  * [19] Vestibular schwannoma (acoustic neuroma) and neurofibromatosis, https://www.nidcd.nih.gov/health/vestibular-schwannoma-acoustic-neuroma-and-neurofibromatosis
  * [20] Wang, G., Shapey, J., Li, W., Dorent, R., Dimitriadis, A., Bisdas, S., Paddick, I., Bradford, R., Zhang, S., Ourselin, S., et al.: Automatic segmentation of vestibular schwannoma from t2-weighted mri by deep spatial attention with hardness-weighted loss. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 264–272. Springer (2019)
  * [21] Yan, H., Ding, Y., Li, P., Wang, Q., Xu, Y., Zuo, W.: Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). p. 945–954. IEEE, Honolulu, HI (Jul 2017). https://doi.org/10.1109/CVPR.2017.107, http://ieeexplore.ieee.org/document/8099590/
  * [22] Yu, F., Zhang, M., Dong, H., Hu, S., Dong, B., Zhang, L.: Dast: Unsupervised domain adaptation in semantic segmentation based on discriminator attention and self-training. Proceedings of the AAAI Conference on Artificial Intelligence 35(1212), 10754–10762 (May 2021). https://doi.org/10.1609/aaai.v35i12.17285
  * [23] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2223–2232 (2017)
  * [24] Zhu, Q., Li, H., Cass, N.D., Lindquist, N.R., Tawfik, K.O., Oguz, I.: Acoustic neuroma segmentation using ensembled convolutional neural networks. In: Medical Imaging 2022: Biomedical Applications in Molecular, Structural, and Functional Imaging. vol. 12036, pp. 228–234. SPIE (2022)
  * [25] Zou, Y., Yu, Z., Kumar, B., Wang, J.: Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In: Proceedings of the European conference on computer vision (ECCV). pp. 289–305 (2018)