# Bengali Document Layout Analysis with Detectron2

Md Ataullha1, Mahedi Hassan Rabby2, Mushfiqur Rahman3, Tahsina Bintay Azam4
All authors contributed equally to this work. Department of Computer Science
and Engineering
Shahjalal University of Science and Technology
Sylhet, Bangladesh
Email: {ataullha81, mahedi33, mushfiqur83<EMAIL_ADDRESS>
###### Abstract

Document digitization is vital for preserving historical records, efficient
document management, and advancing OCR (Optical Character Recognition)
research. Document Layout Analysis (DLA) involves segmenting documents into
meaningful units like text boxes, paragraphs, images, and tables. Challenges
arise when dealing with diverse layouts, historical documents, and unique
scripts like Bengali, hindered by the lack of comprehensive Bengali DLA
datasets.

We improved the accuracy of the DLA model for Bengali documents by utilizing
advanced Mask R-CNN models available in the Detectron2 library. Our evaluation
involved three variants: Mask R-CNN R-50, R-101, and X-101, both with and
without pretrained weights from PubLayNet, on the BaDLAD dataset, which
contains human-annotated Bengali documents in four categories: text boxes,
paragraphs, images, and tables.

Results show the effectiveness of these models in accurately segmenting
Bengali documents. We discuss speed-accuracy trade-offs and underscore the
significance of pretrained weights. Our findings expand the applicability of
Mask R-CNN in document layout analysis, efficient document management, and OCR
research while suggesting future avenues for fine-tuning and data
augmentation.

###### Index Terms:

Mask R-CNN, Instance Segmentation, Transfer Learning

##  I Introduction

Document Layout Analysis (DLA) is a fundamental challenge in the digital age,
enabling structured information extraction from unstructured documents. Deep
Learning, especially Mask R-CNN within the Detectron2 [1] framework, offers
promising solutions for automating DLA tasks.

This paper explores Mask R-CNN’s potential in DLA, using the BaDLAD [2]
dataset —a diverse collection of Bengali documents. We meticulously evaluate
three Mask R-CNN configurations, considering factors like architecture,
efficiency, and feature extraction capabilities. Our aim is to identify a
model that excels in predicting bounding polygon segments (masks) for document
layout components.

Our findings showcase an optimal configuration, Mask R-CNN R-101, which
leverages pre-trained weights to achieve high accuracy in mask prediction.
This configuration also boasts efficiency during inference. We employ
thresholding and Run-Length Encoding (RLE) for post-processing to enhance
practical utility.

This paper contributes valuable insights to the field of deep learning-based
Document Layout Analysis, offering a potent solution for understanding
document layout components. Our findings have implications for research and
practical applications.

##  II Methodology

###  II-A Deep Learning Pipeline with Detectron2

Our deep learning pipeline leverages the robust capabilities of Detectron2, an
advanced library developed by Facebook AI Research. This state-of-the-art
library incorporates cutting-edge detection and segmentation algorithms,
establishing itself as a formidable tool for addressing intricate document
layout analysis tasks. By harnessing the power of Detectron2, our pipeline
excels at predicting and segmenting diverse document layout components,
accurately defined by bounding polygons. This approach endows our pipeline
with adaptability and versatility, making it a potent solution for real-world
scenarios that encompass varying document types and layouts.

###  II-B Model Selection

Our model selection process involves a meticulous evaluation of three distinct
COCO-InstanceSegmentation models from the Detectron2 model zoo. Each model was
carefully chosen based on its architectural attributes, advantages, and
potential drawbacks. These models include:

TABLE I: COCO INSTANCE SEGMENTATION BASELINE MODELS WITH MASK R-CNN [1] Name | lrsched | traintime(s/iter) | inferencetime(s/im) | trainmem(GB) | boxAP | maskAP
---|---|---|---|---|---|---
R50-FPN | 3x | 0.261 | 0.043 | 3.4 | 41.0 | 37.2
R101-FPN | 3x | 0.340 | 0.056 | 4.6 | 42.9 | 38.6
X101-FPN | 3x | 0.690 | 0.103 | 7.2 | 44.3 | 39.5

####  II-B1 mask-rcnn-R-50-FPN-3x

Architecture: This model adopts the Mask R-CNN architecture, featuring a
ResNet-50 backbone and Feature Pyramid Network (FPN). It is pre-trained on the
COCO dataset, striking a balance between computational efficiency and feature
extraction capability.

Advantages: The ResNet-50 backbone offers a blend of computational efficiency
and adept feature extraction. FPN contributes to multi-scale object detection
and segmentation.

Disadvantages: The smaller backbone may limit the model’s capacity to capture
intricate features, potentially affecting performance on tasks with complex
document layouts.

####  II-B2 mask-rcnn-R-101-FPN-3x

Architecture: This model employs the Mask R-CNN architecture, featuring a
ResNet-101 backbone coupled with a Feature Pyramid Network (FPN) for
comprehensive feature extraction. The ”3x” designation indicates that the
model is trained for three times the default number of iterations. It is pre-
trained on the COCO dataset and excels in instance segmentation tasks.

Advantages: The ResNet-101 backbone boasts robust feature extraction
capabilities, complemented by FPN’s proficiency in handling objects of varying
scales. Pre-training on COCO enhances its generalization potential.

Disadvantages: The model’s larger backbone and FPN introduce higher
computational demands, potentially resulting in extended inference times.
Fine-tuning may be necessary for optimal performance in document layout
analysis.

####  II-B3 mask-rcnn-X-101-32x8d-FPN-3x

Architecture: This model builds upon the Mask R-CNN architecture,
incorporating a more intricate ResNeXt-101-32x8d backbone and FPN. It is also
pre-trained on COCO, offering heightened feature extraction capabilities.

Advantages: The ResNeXt-101-32x8d backbone enhances feature extraction, while
FPN adeptly handles multi-scale object analysis.

Disadvantages: Similar to the previous model, the larger backbone and FPN
result in higher computational requirements and extended inference times,
potentially necessitating substantial computational resources.

In summary, our model selection process encompasses a range of models, each
tailored to specific requirements and trade-offs. These models are integrated
into our deep learning pipeline, enabling us to effectively tackle the
complexities of document layout analysis, ensuring superior performance and
adaptability across diverse document types and layouts.

###  II-C Baseline Model Performance

The performance of baseline models for COCO instance segmentation using Mask
R-CNN. These models are trained with different backbone architectures
(R50-FPN, R101-FPN, and X101-FPN) and follow a 3x learning rate schedule. We
present key performance metrics that are crucial for assessing the
effectiveness of these models in instance segmentation tasks.

Table I provides detailed information about each baseline model’s training
time per iteration, inference time per image, memory usage during training, as
well as their Box Average Precision (AP) and Mask AP scores. These metrics
offer insights into the computational efficiency and accuracy of the models,
shedding light on their suitability for various real-world applications in
instance segmentation.

###  II-D Prepossessing

After model selection, our next critical step involves preprocessing the
BaDLAD dataset, which comprises around 34,000 meticulously annotated Bengali
documents. These documents span a wide range of content categories, including
newspapers, official documents, notices, reports, novels, comics, magazines,
and historical records of the liberation war. The annotations for each
document delineate regions corresponding to four distinct classes: Paragraph,
Text Box, Image, and Table. Our core objective is to construct a robust deep
learning pipeline capable of analyzing PNG-format document images and
accurately predicting bounding polygon segments, known as masks, for these
specified classes.

Figure 1: Binary mask of a document

Our training dataset consists of 20,365 images, meticulously organized in the
COCO format. This careful arrangement ensures seamless integration with the
Detectron2 library, a critical component of our pipeline. Our preprocessing
challenge revolves around predicting masks that intricately outline the
bounding polygons for each class within the image. These masks are represented
as 2D matrices aligned with the image pixels, where the value 0 represents the
background category (indicating the absence of specific layout elements), and
the value 1 signifies the presence of the corresponding class.

In the Figure 1, we can see an example of a mask, where the white regions
represent the presence of a specific layout element (e.g., text box or
paragraph), and the black regions denote the background. This preprocessing
step is crucial to prepare the data for training our deep learning models
effectively.

###  II-E Training

After selecting our models and preprocessing the BaDLAD dataset, we proceeded
with the training process to develop accurate and robust deep learning models
for document layout analysis. Here, we outline the key aspects of our training
process.

###  II-F Model Configuration and Hyperparameters

We considered various configurations for our Mask R-CNN models, including the
use of pretraining and different hyperparameters. These configurations are
detailed in the Table II

TABLE II: PERFORMANCE OF DIFFERENT MASK R-CNN CONFIGURATIONS BASED ON BADLAD DATASET Model | Pretrained | Learning Rate | Warmup Iters | Max Iters | Train Split | Dice Score
---|---|---|---|---|---|---
Mask R-CNN R-50 | False | 0.0015 | 500 | 17000 | 0.95 | 0.87769
| True | 0.001 | 1800 | 18000 | 0.95 | 0.88099
Mask R-CNN R-101 | False | 0.001 | 5 | 64000 | 1 | 0.89191
| True | 0.001 | 5 | 25000 | 1 | 0.89082
Mask R-CNN X-101 | False | 0.001 | 5 | 25000 | 0.99 | 0.87803
| True | 0.001 | 1000 | 28255 | 0.99 | 0.88711

In this table, we compare the performance of different Mask R-CNN
configurations on the BaDLAD dataset. We considered whether the models were
pretrained or not, the learning rate, warmup iterations, maximum iterations,
training split, and the Dice Score, which reflects the model’s accuracy in
predicting object masks.

###  II-G Utilizing Pretrained Path

For our training, we leveraged pretrained weights from the PubLayNet dataset
[3] as a starting point. This approach allowed our models to inherit knowledge
from a related dataset, enhancing their performance and accelerating
convergence during training.

###  II-H Training Data Split

We split our dataset into training and validation subsets, ensuring that 95%
(for some models, 99%) of the data was used for training. This division
allowed us to train the models on a substantial portion of the dataset while
reserving a smaller portion for validation to monitor the model’s performance.

###  II-I Training Duration

The training duration varied based on the model and configuration. We
conducted training for a specified number of iterations (Max Iters) while
monitoring the model’s performance on the validation set. Training was halted
when the model achieved satisfactory results, preventing overfitting.

###  II-J Dice Score

The Dice Score metric was employed to evaluate the quality of the predicted
object masks. A higher Dice Score indicates better accuracy in predicting
object boundaries. The models’ performance was assessed based on this metric.

In summary, our training process involved configuring the models, utilizing
pretrained weights, splitting the dataset for training and validation, and
monitoring performance using the Dice Score. This iterative process allowed us
to fine-tune the models for accurate document layout analysis on the BaDLAD
dataset.

###  II-K Post-processing

After the training of our Mask R-CNN models on the BaDLAD dataset, we
conducted a crucial post-processing step to improve the accuracy of our
predictions. This step involved the application of a threshold value to
differentiate between foreground and background pixels within the predicted
masks.

We set a threshold of 60% to classify pixels in the predicted masks. Pixels
with a probability value greater than or equal to 60% were considered part of
the foreground, corresponding to the specified layout class, while pixels with
probabilities below this threshold were categorized as background. This
thresholding process was instrumental in enhancing prediction accuracy by
reducing false positives and noise.

Furthermore, after applying the threshold, we adopted the Run-Length Encoding
(RLE) technique to efficiently represent the predicted masks. RLE is a
compression algorithm that encodes sequences of consecutive pixel values,
making it an ideal choice for encoding binary masks.

By encoding our predicted masks using RLE, we achieved a more compact
representation of mask information. This encoded format facilitated seamless
integration into various downstream applications, such as document retrieval,
content extraction, or archival systems.

In summary, our post-processing step, involving the 60% threshold and
subsequent prediction of the RLE of the mask, played a pivotal role in
refining predictions and enabling efficient data representation for practical
applications in document layout analysis.

##  III Results and Discussion

In this section, we present the outcomes of our experiments with different
Mask R-CNN configurations on the BaDLAD dataset and discuss their implications
for future work.

###  III-A Performance of Different Mask R-CNN Configurations

We conducted extensive experiments with various Mask R-CNN configurations, as
detailed below. Our goal was to evaluate their performance on instance
segmentation tasks using the BaDLAD dataset. Notably, we explored both
pretrained and non-pretrained paths for different models.

####  III-A1 Mask R-CNN R-50

When trained without a pretrained path, this configuration achieved a Dice
Score of 0.87769. However, when pretrained weights were utilized, it exhibited
improved performance, reaching a Dice Score of 0.88099. This demonstrates the
benefit of transfer learning for this model.

####  III-A2 Mask R-CNN R-101

This model, when pretrained, delivered a remarkable Dice Score of 0.89082.
Even without pretraining, it achieved a high Dice Score of 0.89191. These
results highlight the effectiveness of this configuration in instance
segmentation tasks.

####  III-A3 Mask R-CNN X-101

Without pretrained weights, this configuration attained a Dice Score of
0.87803. However, leveraging pretrained weights significantly improved its
performance, resulting in a Dice Score of 0.88711. This underscores the
importance of pretraining for this model.

####  III-A4 Optimal Configuration

After thorough experimentation, our findings point to the superiority of the
Mask R-CNN R-101 configuration without pretrained weights for our dataset.
This particular setup delivered outstanding results, boasting a Dice Score of
0.89191 (even achieving 0.89283 during the Dl sprint 2.0 competition [4] on
the entire test dataset). Impressively, it exhibited remarkable efficiency
during inference, processing all 1625 images in a mere 46 minutes and 24
seconds.

Figure 2: Inference results using Mask R-CNN R-101

In Figure 2, you can observe the inference results using the optimal Mask
R-CNN R-101 configuration. The efficient processing time and high Dice Score
make this configuration suitable for real-world document layout analysis
tasks.

##  IV Future Work

While our non-pretrained Mask R-CNN R-101 configuration exhibited impressive
performance on the BaDLAD dataset, we acknowledge the scope for enhancements.
Subsequent research endeavors could investigate the potential of Mask R-CNN
R-101 with pretrained weights, coupled with extended training iterations,
building upon the promising outcomes observed in our experiments. Furthermore,
the incorporation of data augmentation strategies, including rotation and
flipping, holds promise for further enhancing model efficacy, particularly
when dealing with datasets that encompass horizontal image orientations.

##  V Conclusion

In conclusion, our research underscores the vital role of choosing the right
model and setup for document layout analysis. Our study showcased the
impressive capabilities of Mask R-CNN R-101 with pretrained weights, which not
only achieved remarkable accuracy but also worked efficiently. These results
suggest that exploring pretrained paths and implementing data augmentation
techniques could further boost performance. We are confident that our
discoveries will drive advancements in document analysis methods, paving the
way for practical use in real-world contexts, particularly with datasets like
BaDLAD.

## References

  * [1] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2,” https://github.com/facebookresearch/detectron2, 2019.
  * [2] M. I. H. Shihab, M. R. Hasan, M. R. Emon, S. M. Hossen, M. N. Ansary, I. Ahmed, F. R. Rakib, S. E. Dhruvo, S. S. Dip, A. H. Pavel _et al._ , “Badlad: A large multi-domain bengali document layout analysis dataset,” in _International Conference on Document Analysis and Recognition_. Springer, 2023, pp. 326–341.
  * [3] X. Zhong, J. Tang, and A. J. Yepes, “Publaynet: largest dataset ever for document layout analysis,” in _2019 International Conference on Document Analysis and Recognition (ICDAR)_. IEEE, Sep. 2019, pp. 1015–1022.
  * [4] M. A. Haider, S. Khondker, Sameen53, Sushmit, and Tahsin, “Dl sprint 2.0 - buet cse fest 2023,” https://kaggle.com/competitions/dlsprint2, 2023.


11institutetext: Vanderbilt University, Nashville, TN, USA22institutetext:
Siemens Healthineers, Princeton, NJ, USA
22email<EMAIL_ADDRESS>
# Learning Site-specific Styles for Multi-institutional Unsupervised Cross-
modality Domain Adaptation

Han Liu 11 Yubo Fan 11 Zhoubing Xu 22 Benoit M. Dawant 11 Ipek Oguz 11

###### Abstract

Unsupervised cross-modality domain adaptation is a challenging task in medical
image analysis, and it becomes more challenging when source and target domain
data are collected from multiple institutions. In this paper, we present our
solution to tackle the multi-institutional unsupervised domain adaptation for
the crossMoDA 2023 challenge. First, we perform unpaired image translation to
translate the source domain images to the target domain, where we design a
dynamic network to generate synthetic target domain images with controllable,
site-specific styles. Afterwards, we train a segmentation model using the
synthetic images and further reduce the domain gap by self-training. Our
solution achieved the $1^{st}$ place during both the validation and testing
phases of the challenge. The code repository is publicly available at
https://github.com/MedICL-VU/crossmoda2023.

###### Keywords:

MRI Vestibular schwannoma Cochlea Multi-institutional Unsupervised domain
adaptation Style transfer Dynamic network

##  1 Introduction

The crossMoDA challenges111https://crossmoda-challenge.ml/ [6] aim to tackle
the unsupervised cross-modality segmentation of vestibular schwannoma (VS) and
cochleae on MRI scans. Specifically, participants are provided with the
labeled source domain data, i.e., contrast-enhanced T1-weighted (ceT1) images,
and the unlabeled target domain data, i.e., high-resolution T2-weighted (hrT2)
images. The goal of this challenge is to train a segmentation model for the
target domain hrT2 images. The crossMoDA 2023 extends the previous editions by
introducing (1) a sub-segmentation task for the VS (intra- and extra-meatal
components)[28] and (2) more heterogeneous data collected from multiple
institutions. The schematic problem description of the crossMoDA 2023 is
illustrated in Fig. 1. Specifically, the organizers partition the multi-
institutional images into 3 sub-datasets, namely ETZ, LDN, and UKM. It can be
observed that the hrT2 images from different sub-datasets have significantly
different appearances and thus it is critical to ensure the robustness of our
segmentation model on the multi-institutional data.

As the images within the same sub-dataset have relatively consistent styles,
we assume that the images within each sub-dataset are collected from the same
site. Note that this assumption is not accurate for the UKM sub-dataset as it
includes images collected from multiple sites. However, by considering the UKM
images as collected from the same site, we will show that our generative model
can learn a UKM-specific style that can be used to diversify the styles of our
synthetic images. In this paper, our contributions are summarized as follows:

  * •

We revisit the top-performing solutions of the previous crossMoDA challenges
and analyze the factors contributing to their success.

  * •

To addresses the intra-domain variability in multi-institutional UDA, we
propose a dynamic network to generate synthetic images with controllable,
site-specific styles, which are used to train the downstream segmentation
model for improved robustness and generalizability.

  * •

Our proposed method achieves the $1^{st}$ place during both the validation and
testing phases of the crossMoDA 2023 challenge.

Figure 1: Schematic problem description of the crossMoDA 2023 challenge. The
task of this challenge is cross-modality unsupervised domain adaptation (UDA),
where source domain and target domain are contrast-enhanced T1 (ceT1) and
high-resolution T2 (hrT2), respectively. Note that both source and target
domain data are collected from multiple institutions, leading to additional
challenges to the UDA tasks, which primarily focus on the inter-domain gap
rather than the intra-domain variability.

##  2 Related Works

###  2.1 CrossMoDA Challenges: 2021-2023

While numerous domain adaptation techniques have been proposed for image
segmentation, most of these techniques have only been validated either on
private datasets or on small public datasets, and mostly addressed single-
class segmentation tasks. The crossMoDA challenge[6] introduced the first
large and multi-class dataset for cross-modality domain adaptation for medical
image segmentation. In the 2021 edition, source and target domain data were
collected from a single scanner and the participants were asked to segment the
cochleae and the whole VS in hrT2 images, i.e., a 2-class segmentation task.
With the same task, the 2022 edition included additional data from another
scanner for both source and target domain datasets, making the domain
adaptation task more challenging by introducing intra-domain variability. The
2023 edition further enlarged the datasets by including multi-institutional,
heterogeneous data for both domains and introduced a sub-segmentation for the
VS (intra- and extra-meatal components), leading to a 3-class segmentation
task with significant intra-domain variability.

Figure 2: The training strategy of the image-level domain alignment approaches
for UDA.

###  2.2 Top Solutions in crossMoDA 2021 and 2022

The top solutions in the 2021 and 2022 editions are mainly based on the image-
level domain alignment approach. As illustrated in Fig. 2, it typically
consists of three steps. In step 1, unpaired image translation is used to
translate ceT1 images to synthetic hrT2 images. The most commonly used
techniques include cycleGAN[32], CUT[24], QS-Attn[11] with either 2D or 3D
backbones. In step 2, the synthetic hrT2 images and the associated ceT1 labels
are used to train a segmentation model. In step 3, to further reduce the
domain gap between synthetic and real hrT2, the unlabeled real hrT2 are used
to train the segmentation model via self-training. Specifically, the network
trained in step 2 is used to firstly generate the pseudo labels on the real
hrT2 images. Then the synthetic and real hrT2 images are combined to re-train
a segmentation network. This self-training process can be repeated iteratively
by using the most updated pseudo labels generated by the network trained at
the previous iteration.

Based on the image-level domain alignment strategy, the top teams have
proposed a variety of techniques to further improve the performance. In the
2021 edition, the 1st place team[27] proposed to add segmentation decoders to
the generators of the 2D cycleGAN to better synthesize the VS and the cochlea.
Additionally, they visually inspected the pseudo labels to select the most
reliable ones for self-training. The 2nd place team proposed PAST[5], where 2D
NICE-GAN[3] was used for image synthesis and self-training with pixel-level
pseudo label filtering was used for segmentation. The 3rd place team[4] used
the CUT model for image synthesis and proposed an offline data augmentation
technique to simulate the heterogeneous signal intensity of VS. In the 2022
edition, the 1st place team built upon the PAST algorithm and added extra
segmentation heads for NICE-GAN. Moreover, to address the intra-domain
variability, they trained separate segmentation models for different sites and
structures. The 2nd place team[15] proposed to improve the image synthesis via
multi-view image translation, where the cycleGAN and the QS-Attn were used in
parallel. The 3rd place team[25] proposed to improve the generalizability of
the segmentation model by generating diverse appearances of VS via SinGAN[26].

In summary, the top solutions in 2021 and 2022 editions demonstrated three
promising directions to improve the image-level domain alignment: (1) better
synthetic hrT2 images in step 1, (2) higher-quality pseudo labels for self-
training in step 3, and (3) local intensity augmentation for VS in step 2 and
3.

##  3 Methods

Motivated by the previous works[19, 18, 27, 5, 9, 31], we propose to tackle
the UDA problem by reducing the domain gap at the image-level, and follow the
3-step strategy as presented in Sec. 2.2. Since the quality of synthetic hrT2
images is critical to the performance of the downstream segmentation task, our
key innovations are mainly focused on the step 1, i.e., unpaired image
translation. To address the intra-domain variability, we propose to generate
synthetic hrT2 images with site-specific styles, which are then used to train
the segmentation model for improved robustness to various hrT2 styles. The
details of our novel techniques for image translation are provided as follows.

Figure 3: Illustration of our proposed label-assisted intensity
transformation. The VS (yellow arrow) and cochleae (blue arrows) have opposite
intensity profiles in ceT1 and hrT2 images.

###  3.1 Label-assisted Intensity Transformation

The VS and the cochleae have significantly different intensity profiles in
ceT1 and hrT2. As shown in Fig. 3, the cochleae have weak signals and the VS
has strong signals in ceT1 images, but the opposite is true in hrT2. Our
preliminary experiment shows that the synthesis network with the original ceT1
as input may fail to capture the appearance difference of these structures
between the two modalities. To address this problem, we propose to transform
the intensity profiles of VS and cochlea in ceT1 images before feeding them to
the synthesis network. After we perform regular preprocessing steps, which
include rescaling to [-1, 1] range (see Sec. 3.5.1), we replace the intensity
values of the cochleae by 1.0, i.e., the maximum value of the preprocessed
image. In addition, we decrease the intensity values of the VS by
$\mu_{VS}+0.5$, where $\mu_{VS}$ is the mean intensity of the VS. With the
transformed ceT1, we reduce the appearance difference of the VS and cochleae
across two modalities before performing the synthesis task.

Figure 4: Illustration of our dynamic generator used for the unpaired image
translation. T is the label-assisted intensity transformation. Given a site
code (a one-hot vector), our dynamic network is trained to generate site-
specific affine parameters for the last instance normalization layer, which is
then used to control the output hrT2 styles.

###  3.2 Anatomy-aware Image Synthesis

We adopt the QS-Attn [11] and extend it to 3D for volumetric unpaired image
translation. 3D QS-Attn is used because (1) compared to 2D networks, 3D
networks can generate synthetic images with better slice-to-slice continuity
by exploiting the intra-slice information, and (2) compared to CycleGAN[32],
QS-Attn is less memory-intensive and thus more suitable for 3D networks.

As shown in Fig. 4, we propose to improve the image synthesis by making the
generator focus more on the anatomical structures in the downstream
segmentation task, i.e., the VS and the cochleae. To this end, we add an extra
segmentation decoder $D_{seg}$ to the generator such that our generator learns
to synthesize hrT2 images and segment these structures jointly. As
demonstrated in [27], this multi-task learning paradigm can help better
preserve the shape of the structures-of-interest (SOI) in the synthetic
images. Moreover, we employ another segmentation network $S$ to segment SOI
from the synthetic hrT2 images, further encouraging the generated SOI to have
semantically meaningful boundaries.

###  3.3 Site-specific Styles

To ensure the robustness to different hrT2 styles, we propose to generate the
synthetic hrT2 images with site-specific styles to train the segmentation
model. Inspired by [20], we propose to modify the synthesis decoder to a
dynamic network, where the style of the output hrT2 image is conditioned on a
given site prior. Specifically, we replace the last instance normalization
(IN) layer of the synthesis decoder by a dynamic instance normalization (DIN)
layer. This is motivated by previous studies[7, 12, 16] where the IN layers
are shown to effectively control the styles of images. We encode the site
condition as a one-hot vector $c$, which is passed to a controller (a 3D
convolutional layer with a kernel size of $1\times 1\times 1$) to generate
site-specific affine parameters $\gamma^{s}$ and $\beta^{s}$ for IN.
Therefore, we can train a single unified synthesis network on all hrT2 images
with a controllable output style, as shown in Fig. 5.

Figure 5: Synthetic hrT2 images with site-specific styles. In top three rows,
each row displays a representative ceT1 image being transformed to hrT2 with
different site-specific styles. The bottom row displays real hrT2 images from
three different sites, which are used as references for style comparison. Each
column corresponds to the same site-specific style and the associated site
code is shown on the top left corner at the bottom row. Figure 6: Examples of
hrT2 styles generated by style interpolation. During inference, arbitrary site
codes can be used as the site condition to generate unseen hrT2 styles. In
each example, the site code is shown at the top left corner .

###  3.4 Oversampling Hard Samples by Style Interpolation

Based on the segmentation results from the validation set, we observe that the
VS with either (1) tiny/no extra-meatal components, or (2) large extra-meatal
components with heterogeneous appearance, are more challenging to segment. We
refer to these cases as hard samples. We find that such hard samples are
indeed under-represented in the source domain dataset and their associated
synthetic hrT2 may need to be oversampled for balanced training. In practice,
we select the hard samples based on the aforementioned two rules with the help
of source domain labels. Inspired by style interpolation [7, 12, 10], we
propose to generate more diverse hrT2 styles for oversampling by feeding the
controller with unseen site codes. As shown in Fig. 6, we oversample each hard
sample by translating the same ceT1 image into a variety of unseen hrT2
styles, further enriching the diversity of our synthetic dataset.

###  3.5 Implementation Details

####  3.5.1 Preprocessing

All MR scans are set to the RAI orientation, resampled to the median voxel
size of the dataset, i.e., $0.41\times 0.41\times 1$ mm3, and further cropped
into $256\times 144\times 32$ based on the positions of the cochleae, which
are computed by a segmentation network additionally trained on real hrT2
images with pseudo-labeled cochleae. The cropped volumes are used for all the
synthesis and segmentation tasks. For image synthesis, we normalize both ceT1
and hrT2 images using Z-score normalization, clip the intensity values to the
[0, 99.9th] percentile, and rescale the values to [-1, 1].

####  3.5.2 Synthesis

The backbone of our dynamic generator is a 3D 9-block ResNet. Due to the limit
of GPU memory, the input is a 3D patch with a size of $256\times 144\times 8$
randomly cropped from the preprocessed image. We use overlapping sliding
windows for inference. During training, we apply on-the-fly data augmentation
including random contrast adjustment ($p=0.4$ for ceT1 and $p=0.1$ for hrT2;
smaller $p$ for hrT2 to preserve the site-specific style) and randomly
flipping on the LR direction ($p=0.5$). The loss function for image synthesis
is expressed as:
$L_{G}=L_{QS}+\lambda_{1}L_{seg}^{ceT1}+\lambda_{2}L_{seg}^{hrT2}+\lambda_{3}L_{edge}$,
where $L_{QS}=L_{adv}+L_{con}^{ceT1}+L_{con}^{hrT2}$ is the default loss
function of QS-Attn. $L_{seg}^{ceT1}$ and $L_{seg}^{hrT2}$ are the
segmentation losses for $D_{seg}$ and $S$, respectively. We also adopt an edge
loss $L_{edge}$[30, 8, 22] to encourage the edge consistency between the input
and the output so that the texture within the VS and the cochlea boundary can
be well preserved. We use $\lambda_{1}=0.5$, $\lambda_{2}=0.5$ and
$\lambda_{3}=1$. We train the network for 400 epochs with a learning rate of
$2e-4$ and another 400 epochs with linear decay policy. For other
hyperparameters, we use the default settings of the QS-Attn.

####  3.5.3 Segmentation

We use the nnU-Net V2[14] with 3D fullres configuration for all our
segmentation tasks. We build upon the default nnUNetTrainer and make the
following modifications. First, we only enable random flipping along LR
direction. Second, we introduce two local intensity augmentation functions to
only augment the intensity values of the VS and the cochlea. Specifically, we
randomly multiply the VS intensity with $u\sim U(1.2,2)$. In addition, we
randomly reduce the cochleae intensity by $v\sim U(0.5,1)$, since previous
study indicates that the cochleae ipsilateral to VS may have weaker signals in
hrT2[2]. We follow [21] to train segmentation models and perform two rounds of
self-training. Previous studies suggest that image-level pseudo label
filtering can be incorporated into self-training to avoid performance
degradation caused by unreliable pseudo labels[13, 29, 23]. Therefore, we
remove the real hrT2 images with unreliable pseudo labels from our training
set throughout the self-training process, where the pseudo labels with no
tumor prediction or with multiple tumor components on both sides are
considered unreliable. Note that the reliability of pseudo labels can be
determined by connected component analysis and thus the entire process is
fully automatic. Lastly, we use model ensemble by averaging the predictions
from 11 models to further boost the performance. These models include 3
standard nnU-Net models trained with different seeds and 8 customized nnU-Net
models with the following configurations: 2 different backbone architectures
(U-Net or ResU-Net) $\times$ 2 different augmentation strategies (strong or
weak local intensity augmentation for VS and cochleae) $\times$ 2 different
sets of unseen site codes for style interpolation.

##  4 Experiments and Results

We use the dataset1 1 1 https://www.synapse.org/#!Synapse:syn51317912 provided
by the crossMoDA 2023 challenge[17, 28]. Dice score and average symmetric
surface distance (ASSD) for extra-meatal VS, intra-meatal VS, and cochleae, as
well as the boundary ASSD (denoted as ‘bound’) are used for quantitative
evaluation.

Table 1: Quantitative results during the validation phase (96 cases). Bold represents the best scores. The three rows at the bottom are our ablation studies. ST: self-training. Tr: modified nnUNetTrainer. OS: oversampling Method | Dice$\uparrow$ (%) | ASSD$\downarrow$ (mm)
---|---|---
|  extra |  intra |  cochlea |  extra |  intra |  cochlea |  bound
Ours (ensemble) |  85.75 |  74.36 |  84.07 |  0.45 |  0.44 |  0.20 |  0.51
Ours (single) |  85.08 |  73.34 |  84.44 |  0.48 |  0.45 |  0.20 |  0.53
Team A |  83.63 |  70.57 |  83.55 |  0.50 |  0.59 |  0.23 |  4.76
Team B |  72.75 |  56.94 |  86.66 |  17.62 |  16.80 |  0.18 |  32.79
Team C |  81.32 |  59.79 |  83.56 |  8.57 |  7.67 |  0.22 |  20.78
w/o ST |  84.06 |  71.42 |  82.78 |  0.50 |  0.59 |  0.21 |  0.54
w/o (ST, Tr) |  81.74 |  68.92 |  82.81 |  8.58 |  7.93 |  0.53 |  8.69
w/o (ST, Tr, OS) |  79.09 |  64.15 |  81.94 |  8.68 |  11.80 |  0.56 |  12.86
Table 2: Quantitative results during the testing phase (341 cases). Method | Dice$\uparrow$ (%) | ASSD$\downarrow$ (mm)
---|---|---
|  extra |  intra |  cochlea |  extra |  intra |  cochlea |  bound
Ours |  84.9 |  72.8 |  83.6 |  0.452 |  0.496 |  0.201 |  0.675
Team A |  80.8 |  69.9 |  84.4 |  0.593 |  0.581 |  0.207 |  1.985
Team B |  78.6 |  60.7 |  84.3 |  6.552 |  9.711 |  0.246 |  18.575
Team C |  78.4 |  64.6 |  81.4 |  1.625 |  4.036 |  1.316 |  9.953
Team D |  63.7 |  55.8 |  75.0 |  20.806 |  27.814 |  12.776 |  24.089
Team E |  67.6 |  56.3 |  76.7 |  13.874 |  18.607 |  11.026 |  35.848

###  4.1 Quantitative Results

In Table 1, we report the evaluation metrics on the validation leaderboard.
Our method (a single model) achieves the $1^{st}$ place on validation
leaderboard and model ensembling can slightly improve the performance.
Moreover, we perform ablation studies on the validation set to investigate the
effectiveness of self-training, our modified nnUNetTrainer, and the
oversampling strategy. The results show that each component can effectively
improve the segmentation performance. As shown in Table 2, during the testing
phase, our method outperforms other methods in all evaluation metrics except
the Dice score of cochleae. We note that during both validation and testing
phases our method achieves significantly smaller boundary ASSD, i.e., the
distance between the intra-meatal and extra-meatal boundary. This demonstrates
its superiority in identifying the anatomical separation between the two tumor
components.

Figure 7: Qualitative results of the representative cases from the validation
sets. (a) Unsatisfactory segmentation results. (b) Satisfactory segmentation
results. Dice scores of the intra- and extra-meatal VS are displayed.

###  4.2 Qualitative Results

Representative examples of results obtained with images in the validation set
are shown in Fig. 7. In Fig. 7 (a), we can observe that even with our
oversampling technique (Sec. 3.4), the segmentation results on some hard cases
remain unsatisfactory. For example, UKM_150 and LDN_185 include VS with
tiny/no extra-meatal components and VS with large extra-meatal components and
heterogeneous textures, respectively. Moreover, the field of view and the
image quality may also have a negative impact on the segmentation performance,
e.g., UKM_174. In Fig. 7 (b), even though the hrT2 images may have very
different styles, our model can produce good segmentation results for the VS
whose shapes are common in the training set, indicating that generating site-
specific styles is a promising way to improve model robustness for multi-
institutional data.

##  5 Discussion and Conclusion

In this paper, we have presented our solution for the crossMoDA 2023 challenge
to tackle the multi-institutional UDA problem. Specifically, we have generated
synthetic hrT2 images with site-specific styles to improve the robustness of
the segmentation model. The results obtained during both the validation and
testing phases show that our method has achieved superior performance against
other competitors. Notably, the boundary ASSD achieved by our method is much
smaller than the ones achieved by other methods. This suggests that our method
is more reliable than other approaches for the follow-up clinical analyses,
for which the clear separation between intra- and extra-components is crucial.
For instance, the size and volume features extracted from the extra-meatal VS
are considered as the most sensitive radiomic features for the evaluation of
VS growth[1].

Though our solution has achieved promising performance, we believe there are
several interesting directions to further improve our method. First, by
generating site-specific styles, we assume that the images in each sub-dataset
are collected from the same site and have relatively consistent appearances.
However, this assumption is not strictly accurate for the UKM sub-dataset,
where the images are collected from multiple sites and scanners. Indeed, we
find that the images in the UKM sub-dataset may have significantly different
appearances, which cannot be simply represented by a single site-specific
style. Therefore, an interesting direction for future studies is to transform
the site-specific style to the image-specific style, i.e., the generated style
is conditioned on a reference real hrT2 image. Second, though we can produce
some synthetic styles by feeding the dynamic generator with unseen site codes
(Sec. 3.4), the generated styles and the associated codes do not have strong
correspondence and thus our style interpolation process is not explainable.
The underlying reason may be that the dynamic generator is optimized to learn
only 3 discrete site-specific styles, leading to a discontinuous latent space
of styles. In the future, we will explore some regularization techniques to
make the latent space more continuous. This would permit to not only generate
site-specific styles, but also more diverse and explainable synthetic styles
via style interpolation.

##  6 Acknowledgements

This work was supported in part by the National Science Foundation grant
2220401 and the National Institutes of Health grant T32EB021937.

## References

  * [1] Baccianella, S., Esuli, A., Sebastiani, F.: Evaluation measures for ordinal regression. In: 2009 Ninth international conference on intelligent systems design and applications. pp. 283–287. IEEE (2009)
  * [2] Cass, N.D., Fan, Y., Lindquist, N.R., Dawant, B.M., Tawfik, K.O.: Automated whole cochlear t2 signal demonstrates weak correlation with hearing loss in observed vestibular schwannoma. Audiology & Neuro-otology pp. 1–11 (2023)
  * [3] Chen, R., Huang, W., Huang, B., Sun, F., Fang, B.: Reusing discriminators for encoding: Towards unsupervised image-to-image translation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8168–8177 (2020)
  * [4] Choi, J.: Using out-of-the-box frameworks for unpaired image translation and image segmentation for the crossmoda challenge. arXiv preprint arXiv:2110.01607 (2021)
  * [5] Dong, H., Yu, F., Zhao, J., Dong, B., Zhang, L.: Unsupervised domain adaptation in semantic segmentation based on pixel alignment and self-training. arXiv preprint arXiv:2109.14219 (2021)
  * [6] Dorent, R., Kujawa, A., Ivory, M., Bakas, S., Rieke, N., Joutard, S., Glocker, B., Cardoso, J., Modat, M., Batmanghelich, K., et al.: Crossmoda 2021 challenge: Benchmark of cross-modality domain adaptation techniques for vestibular schwannoma and cochlea segmentation. Medical Image Analysis 83, 102628 (2023)
  * [7] Dumoulin, V., Shlens, J., Kudlur, M.: A learned representation for artistic style. In: International Conference on Learning Representations (2017), https://openreview.net/forum?id=BJO-BuT1g
  * [8] Fan, Y., Khan, M.M., Liu, H., Noble, J.H., Labadie, R.F., Dawant, B.M.: Temporal bone ct synthesis for mr-only cochlear implant preoperative planning. In: Medical Imaging 2023: Image-Guided Procedures, Robotic Interventions, and Modeling. vol. 12466, pp. 358–363. SPIE (2023)
  * [9] Han, L., Huang, Y., Tan, T., Mann, R.: Unsupervised cross-modality domain adaptation for vestibular schwannoma segmentation and koos grade prediction based on semi-supervised contrastive learning. arXiv preprint arXiv:2210.04255 (2022)
  * [10] Hu, D., Li, H., Liu, H., Yao, X., Wang, J., Oguz, I.: Map: Domain generalization via meta-learning on anatomy-consistent pseudo-modalities. arXiv preprint arXiv:2309.01286 (2023)
  * [11] Hu, X., Zhou, X., Huang, Q., Shi, Z., Sun, L., Li, Q.: Qs-attn: Query-selected attention for contrastive learning in i2i translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18291–18300 (2022)
  * [12] Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE international conference on computer vision. pp. 1501–1510 (2017)
  * [13] Huang, Z., Wang, H., Ye, J., Niu, J., Tu, C., Yang, Y., Du, S., Deng, Z., Gu, L., He, J.: Revisiting nnu-net for iterative pseudo labeling and efficient sliding window inference. In: Fast and Low-Resource Semi-supervised Abdominal Organ Segmentation: MICCAI 2022 Challenge, FLARE 2022, Held in Conjunction with MICCAI 2022, Singapore, September 22, 2022, Proceedings, pp. 178–189. Springer (2023)
  * [14] Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)
  * [15] Kang, B., Nam, H., Han, J.W., Heo, K.S., Kam, T.E.: Multi-view cross-modality mr image translation for vestibular schwannoma and cochlea segmentation. arXiv preprint arXiv:2303.14998 (2023)
  * [16] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4401–4410 (2019)
  * [17] Kujawa, A., Dorent, R., Connor, S., Thomson, S., Ivory, M., Vahedi, A., Guilhem, E., Bradford, R., Kitchen, N., Bisdas, S., et al.: Deep learning for automatic segmentation of vestibular schwannoma: A retrospective study from multi-centre routine mri. medRxiv pp. 2022–08 (2022)
  * [18] Li, H., Hu, D., Zhu, Q., Larson, K.E., Zhang, H., Oguz, I.: Unsupervised cross-modality domain adaptation for segmenting vestibular schwannoma and cochlea with data augmentation and model ensemble. In: International MICCAI Brainlesion Workshop. pp. 518–528. Springer (2021)
  * [19] Liu, H., Fan, Y., Cui, C., Su, D., McNeil, A., Dawant, B.M.: Unsupervised domain adaptation for vestibular schwannoma and cochlea segmentation via semi-supervised learning and label fusion. In: International MICCAI Brainlesion Workshop. pp. 529–539. Springer (2021)
  * [20] Liu, H., Fan, Y., Li, H., Wang, J., Hu, D., Cui, C., Lee, H.H., Zhang, H., Oguz, I.: Moddrop++: A dynamic filter network with intra-subject co-training for multiple sclerosis lesion segmentation with missing modalities. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 444–453. Springer (2022)
  * [21] Liu, H., Fan, Y., Oguz, I., Dawant, B.M.: Enhancing data diversity for self-training based unsupervised cross-modality vestibular schwannoma and cochlea segmentation. arXiv preprint arXiv:2209.11879 (2022)
  * [22] Liu, H., Sigona, M.K., Manuel, T.J., Chen, L.M., Dawant, B.M., Caskey, C.F.: Evaluation of synthetically generated computed tomography for use in transcranial focused ultrasound procedures. Journal of Medical Imaging 10(5), 055001–055001 (2023)
  * [23] Liu, H., Xu, Z., Gao, R., Li, H., Wang, J., Chabin, G., Oguz, I., Grbic, S.: Cosst: Multi-organ segmentation with partially labeled datasets using comprehensive supervisions and self-training. arXiv preprint arXiv:2304.14030 (2023)
  * [24] Park, T., Efros, A.A., Zhang, R., Zhu, J.Y.: Contrastive learning for unpaired image-to-image translation. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16. pp. 319–345. Springer (2020)
  * [25] Sallé, G., Conze, P.H., Bert, J., Boussion, N., Visvikis, D., Jaouen, V.: Cross-modal tumor segmentation using generative blending augmentation and self training. arXiv preprint arXiv:2304.01705 (2023)
  * [26] Shaham, T.R., Dekel, T., Michaeli, T.: Singan: Learning a generative model from a single natural image. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4570–4580 (2019)
  * [27] Shin, H., Kim, H., Kim, S., Jun, Y., Eo, T., Hwang, D.: Cosmos: Cross-modality unsupervised domain adaptation for 3d medical image segmentation based on target-aware domain translation and iterative self-training. arXiv preprint arXiv:2203.16557 (2022)
  * [28] Wijethilake, N., Kujawa, A., Dorent, R., Asad, M., Oviedova, A., Vercauteren, T., Shapey, J.: Boundary distance loss for intra-/extra-meatal segmentation of vestibular schwannoma. In: International Workshop on Machine Learning in Clinical Neuroimaging. pp. 73–82. Springer (2022)
  * [29] Yang, L., Zhuo, W., Qi, L., Shi, Y., Gao, Y.: St++: Make self-training work better for semi-supervised semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4268–4277 (2022)
  * [30] Yu, B., Zhou, L., Wang, L., Shi, Y., Fripp, J., Bourgeat, P.: Ea-gans: edge-aware generative adversarial networks for cross-modality mr image synthesis. IEEE transactions on medical imaging 38(7), 1750–1762 (2019)
  * [31] Zhao, Z., Xu, K., Yeo, H.Z., Yang, X., Guan, C.: Ms-mt: Multi-scale mean teacher with contrastive unpaired translation for cross-modality vestibular schwannoma and cochlea segmentation. arXiv preprint arXiv:2303.15826 (2023)
  * [32] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2223–2232 (2017)