# LMVE at SemEval-2020 Task 4: Commonsense Validation and Explanation using
Pretraining Language Model

Shilei Liu1, Yu Guo1, Bochao Li1 Feiliang Ren12∗
1School of Computer Science and Engineering, Northeastern University
2Key Laboratory of Data Analytics and Optimization for Smart
Industry (Northeastern University), Ministry of Education
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

00footnotetext: ∗Corresponding author.

This paper describes our submission111Our team name on leaderboard is NEUKG,
and the first three authors contributed equally to this work. to subtask a and
b of _SemEval-2020 Task 4_. For subtask a, we use a ALBERT based model with
improved input form to pick out the common sense statement from two statement
candidates. For subtask b, we use a multiple choice model enhanced by _hint
sentence_ mechanism to select the reason from given options about why a
statement is against common sense. Besides, we propose a novel transfer
learning strategy between subtasks which help improve the performance. The
accuracy scores of our system are 95.6 / 94.9 on official test set222These
scores were obtained during the _Post-Evaluation_ phase, and our scores in the
_Evaluation_ phase are 90.04 / 93.8. and rank 7th / 2nd on _Post-Evaluation_
leaderboard.

##  1 Introduction

22footnotetext:  This work is licensed under a Creative Commons Attribution
4.0 International Licence. Licence details:
http://creativecommons.org/licenses/by/4.0/.

Common sense verification and explanation is an important and challenging task
in artificial intelligence and natural language processing. This is a simple
task for human beings, because human beings can make full use of external
knowledge accumulated in their daily lives. However, common sense verification
and reasoning is difficult for machines. According to Wang [Wang et al.,
2019], even some state-of-the-art language models such as ELMO[Peters et al.,
2018] and BERT[Devlin et al., 2019] have very poor performance. So it is
crucial to integrate the ability of commonsense-aware to natural language
understanding model[Davis, 2017].

SemEval-2020 task4[Wang et al., 2020] aims to improve the ability of common
sense judgment for model, and we participated in two subtasks of this task.
The dataset of SemEval-2020 task4 named ComVE. Each instance in ComVE is
composed of 5 sentences $\left\langle
s_{1},s_{2},o_{1},o_{2},o_{3}\right\rangle$. $s_{1}$ and $s_{2}$ will be used
for subtask a, and $s_{1}$ or $s_{2}$ with $\left\langle
o_{1},o_{2},o_{3}\right\rangle$ will be used for subtask b.

Subtask a(also known as _Sen-Making_ task) aims to test a model’s ability of
commonsense validation. Specifically, given two statements $\left\langle
s_{1},s_{2}\right\rangle$ whose lexical and syntactic are similar, the object
of _Sen-Making_ model is to determine which statement is common sense(compared
to another one). For example, $s_{1}$ is _put the elephant in the
refrigerator_ and $s_{2}$ is _put the turkey in the refrigerator_ , a good
model needs to judge that the latter is more common sense.

Subtask b(also known as _Explanation_ task) is a multiple choice task that
aims to find the key reason why a given statement does not make sense. For
example, given a sentence that violates common sense with three options
$\left\langle s,o_{1},o_{2},o_{3}\right\rangle$, where $s$ is _he put an
elephant into the fridge_ , $o_{1}$ is _an elephant is much bigger than a
fridge_ , $o_{2}$ is _elephants are usually gray while fridges are usually
white_ , and $o_{3}$ is _an elephant cannot eat a fridge_ , the model needs to
judge that $o_{1}$ is the correct option.

The official baseline of _Sen-Making_ use a pretraining language model(PLM) to
dynamic encoding the two input sentence, and use a simple full connection
neural network to calculate the perplexities respectively, and choosing the
one with lower scores as the correct one. We believe that the baseline method
treats two sentences independently and ignores the inner relationship between
the two sentences, so we propose a novel model structure that fully considers
the interaction between statements.

The official baseline of _Explanation_ treats the task as BERT-like multiple
choice task[Devlin et al., 2019]. We think that the baseline model doesn’t
make full use of the input data. So we design a structure to incorporate
another statement that is common sense to existing model.

In addition, we believe that fine-tuning on similar subtask can improve the
performance of the current subtask because there are many commonalities
between the two subtasks, so we propose a novel transfer learning mechanism
between _Sen-Making_ and _Explanation_.

The proposed system named LMVE, it is a neural network model(includes two sub-
modules to solve both subtask a and b) bases on large scale pretraining
language model.

Our contributions are as follows:

  * •

First, we propose _subtask level transfer learning_ that help share
information between subtasks.

  * •

Second, we propose a novel structure to calculate the perplexity of sentence,
which takes into account the interaction between sentences in a pair.

  * •

Third, we propose the _hint sentence_ mechanism that will help improve the
performance of multiple choice task.(subtask b).

##  2 System Description

We consider our model for both _Sen-Making_ and _Explanation_ as two parts:
encoder and decoder. Encoder is mainly used for getting the contextual
representation of input sentence tokens. In recent years, some pretraining
language models including BERT[Devlin et al., 2019], RoBERTa[Liu et al., 2019]
and ALBERT[Lan et al., 2020] have been proven beneficial for many natural
language processing (NLP) tasks[Rajpurkar et al., 2016, Bowman et al., 2015].
These pretrained models have learned general-purpose language representations
on a large amount of unlabeled data, therefore, adapting these models to the
downstream tasks can bring a good initialization for parameters and avoid
training from scratch[Xu et al., 2020]. So we tried some popular PLMs as
encoders. Decoder consists of several simple linear layers whose number of
parameters are far less than encoder, and the role of decoder is to fuse the
output of encoder and predict the answer.

###  2.1 LMVE for _Sen-Making_ Task

Figure 1: The model architecture for _Sen-Making_ task, (a) is official
baseline and (b) is ours. The yellow point denotes the vector representation
of the output sequence(same as below).

Figure 1(a) is the official baseline model[Wang et al., 2019], which regards
two sentences as independent individuals. But in ComVE, there are certain
similarities(lexical and grammatical) between the two statements, so we think
that the interaction between the two sentences is helpful to improve the
performance of the model. Figure 1(b) gives an overview of our model for _Sen-
Making_ task which is mainly composed of three modules including token
encoding, feature fusion and answer prediction.

Encoding: Let $\\{x^{1}_{0},...,x^{1}_{p}\\}$ and
$\\{x^{2}_{0},...,x^{2}_{q}\\}$ represent one-hot vectors for the first
sentence and the second sentence in an instance, we first concatenate them and
add some special tokens like Figure 1(b), then we will get two sequences
$\\{x_{[CLS]},x^{1}_{0},...,x^{1}_{p},x_{[SEP]},x^{2}_{0},...,x^{2}_{q},x_{[SEP]}\\}$
and
$\\{x_{[CLS]},x^{2}_{0},...,x^{2}_{q},x_{[SEP]},x^{1}_{0},...,x^{1}_{p},x_{[SEP]}\\}$.
The two sequences will be fed into ALBERT respectively. We use $U^{i}\in
R^{d\times n}$ and $V^{i}\in R^{d\times n}$ denote outputs of $i$-th
transformer block in ALBERT, where $d$ is the hidden size of model, $n$ is the
sequence length, and $i\in\\{0,...,L-1\\}$.

Fusion: Some pretraining language model(BERT et al.) usually take the first
token (corresponds to [CLS]) of the output of last transformer block as the
representation of a sequence, but we use the weighted sum of the
representation of first token in all transformer block outputs as the final
representation333Subsequent experimental results show that the performance
using the last 4 layers is the best.. The following equations describe the
process of fusion:

$\displaystyle r_{i}^{0}$ $\displaystyle=U_{0}^{i}\qquad r_{i}^{1}=V_{0}^{i}$
(1) $\displaystyle\alpha_{i}$
$\displaystyle=\frac{\exp(\mathbf{\omega_{i}})}{\sum_{j}\exp(\mathbf{\omega_{j}})}$
(2) $\displaystyle x_{k}$ $\displaystyle=\sum_{i=0}^{L-1}\alpha_{i}r_{i}^{k}$
(3)

where $\mathbf{\omega}\in\mathbb{R}^{d}$ is a trainable parameter and
$i\in[0,L-1]$. We can regard $x_{k}$ as the representation of $k$-th
statement($k\in\\{0,1\\}$).

Answer Prediction: This module maps the output of the fusion layer to a
probability distribution of answer. Given $\mathbf{w}\in\mathbb{R}^{d}$ and
$b\in\mathbb{R}$ as learnable parameters, it calculates the answer possibility
as

$p\left(k\right)=softmax\left(\mathbf{w}\boldsymbol{x}_{k}+b\right)$ (4)

We define the training loss as cross entropy loss function:

$\mathcal{L}_{a}=-\frac{1}{N}\sum_{i=1}^{N}p(y_{i})$ (5)

where $N$ is the number of samples in the dataset and $y_{i}\in\\{0,1\\}$.

###  2.2 Hint Sentence mechanism

Before formally introducing our model for _Explanation_ task, let’s first
introduce the hint sentence mechanism.

In official baseline[Wang et al., 2019], the against common sense statement
will be concatenated with three options respectively and fed into the model.
We believe that this form of input does not make full use of the data in the
ComVE. Specifically, for an instance $\left\langle
s_{1},s_{2},o_{1},o_{2},o_{3}\right\rangle$ in ComVE, it is assumed that
$s_{1}$ does not conform to common sense, then $\left\langle
s_{2},o_{1},o_{2},o_{3}\right\rangle$ will be used to train the baseline model
or to predict answer. However, in this process, $s_{1}$ was abandoned. We
believe that another common sense statement($s_{1}$) in statement pair
contains some useful information, and should be incorporated into our model.

So we propose hint sentence mechanism: A hint sentence is common sense and its
lexical and syntactic are similar to the given against common sense statement
and they differ by only few words. In other words, we call the another
sentence in the sentence pair a _hint sentence_.

The process of how the _hint sentence_ is integrated into the existing model
can be referred to next section and Figure 2. The results of ablation
experiment(Sec 3.6) show that hint sentence mechanism can greatly improve the
performance of our model for _Explanation_ task.

###  2.3 LVME for _Explanation_ Task

Figure 2: The model architecture for _Explanation_ task.

Figure 2 gives an overview of our model for _Explanation_ task. it also has
three modules.

Let $\\{s_{0},...,s_{p}\\}$, $\\{h_{0},...,h_{q}\\}$ and
$\\{o^{i}_{0},...,o^{i}_{r_{i}}\\}$ represent one-hot vectors for the input
statement, hint sentence and $i$-th option in an instance, where
$i\in\\{0,1,2\\}$ and $p$, $q$ and $r$ is the length of them, we first
concatenate them and add some special tokens like Figure 2, then we will get
three sequences
$\\{x_{[CLS]},s_{0},...,s_{p},x_{[SEP]},h_{0},...,h_{q},x_{[SEP]},o^{i}_{0},...,o^{i}_{r_{i}},x_{[SEP]}\\}$.
Then the three sequences will be fed into ALBERT respectively.

Similar to last sub-section, each sequence will get a representation vector
after fusion, and then the three representation vector will pass a linear
layer like Equation 4 to calculate the probability distributions of answer.

We define training loss as

$\mathcal{L}_{b}=-\frac{1}{N}\sum_{i=1}^{N}p(y_{i})$ (6)

where $N$ is the number of samples in the dataset and $y_{i}\in\\{0,1,2\\}$ is
true label.

###  2.4 Subtask Level Transfer Learning

Figure 3: The process of subtask level transfer learning.

Transfer learning is a research problem in machine learning that focuses on
storing knowledge gained while solving one problem and applying it to a
different but related problem. PLM is a typical example of transfer learning
and we call it _task level transfer learning_.

_Sen-Making_ task and  _Explanation_ task are both generalized multiple choice
tasks, and there is an association between the input data for them, so we
believe that in _SemEval-2020 Task 4_ , fine-tuning on similar subtask can
improve the performance of the current subtask.

_Subtask level transfer learning_ refers to use the encoder after fine-tuning
on subtask a(_Sen-Making_) to train subtask b(_Explanation_) and vice versa.
The process of Subtask level transfer learning are shown in Figure 3.

##  3 Experiments and Analysis

Figure 4: Learning curve on the training dataset.

###  3.1 Dataset

ComVE include 10000 samples in train set and 1000 samples in dev/test set for
both _Sen-Making_ and _Explanation_ task. The average length of two statements
in the _Sen-Making_ task are both 8.26, exactly the same. The average length
of true reasons is 7.63 in _Explanation_ task.

It should be noted that in _SemEval-2020 Task 4_ , the test set of
_Explanation_ task is issued only after _Sen-Making_ task is completed, so it
is impossible to use the test set of _Explanation_ task to reverse deduce the
answer of subtask a test set.

###  3.2 Baseline

To verify the effectiveness of our model, we used ALBERT to replace the BERT
in the official baseline, leaving the rest unchanged. We do not perform
subtask level transfer learning (Sec 2.4) on them.

###  3.3 Preprocessing

Data Augmentation: To enhance the robustness of our model, we use Google
Sheets444https://www.google.com/sheets/about to perform back translation
technology on original texts to get augmented texts. Specifically, given a
training sample $\left\langle s_{1},s_{2},o_{1},o_{2},o_{3}\right\rangle$ we
first translate the original statements $s_{1},s_{2}$ to _French_ and then
translate them back to _English_ (denoted as $\hat{s_{1}},\hat{s_{2}}$).
$\left\langle\hat{s_{1}},\hat{s_{2}},o_{1},o_{2},o_{3}\right\rangle$ will add
to training dataset as a new sample. the size of the dataset has doubled after
augmentation.

Tokenization: We employ the tokenizer that comes with the HuggingFace[Wolf et
al., 2019] PyTorch implementation of ALBERT. The tokenizer lowercases the
input and applies the SentencePiece encoding[Kudo, 2018] to split input words
into most frequent subwords present in the pre-training corpus. Non-English
characters will be removed.

###  3.4 Implementation Details

We use the _Transformers_ 555https://github.com/huggingface/transformers
toolkit to implemented our model and tune the hyper-parameters according to
validation performance on the development set. The hidden size is equal to the
corresponding PLM. To train our model, we employ the AdamW
algorithm[Loshchilov and Hutter, 2019] with the initial learning rate as 2e-5
and the mini-batch size as 48.

We also prepared an ensemble model consisting of 7 models for _Sen-Making_
task and 19 for _Explanation_ task with different hyperparameter settings and
random seeds. We used majority voting strategy to fuse the candidate
predictions of different models together.

Model | Params | _Sen-Making_ | _Explanation_
---|---|---|---
Random | - | 49.52 | 32.77
BERTbase[Devlin et al., 2019] | 117M | 88.56 | 85.32
BERTlarge[Devlin et al., 2019] | 340M | 86.55 | 90.12
XLNet[Yang et al., 2019] | 340M | 90.33 | 91.07
SpanBERT[Joshi et al., 2019] | 340M | 89.46 | 90.47
RoBERTa[Liu et al., 2019] | 355M | 93.56 | 92.37
ALBERTbase[Lan et al., 2020] | 12M | 86.63 | 84.37
ALBERTlarge[Lan et al., 2020] | 18M | 88.01 | 89.72
ALBERTxlarge[Lan et al., 2020] | 60M | 92.03 | 92.45
Ours(ALBERTxxlarge) | 235M | 95.68 | 95.48
Our-ensemble | - | 95.91 | 96.39
Table 1: Performance with different encoder.

###  3.5 Main Result

The result of our model for subtask a and subtask b are summarized in Table 1.
We have tried different pretraining language model as our encoder, and found
that ALBERT based model achieves the state-of-the-art performance.

Figure 4 shows a learning curve computed over the provided training data with
testing against the development set, and we can see that in the case of _low-
resource_ (only use 10%-20% training data of target task), the performance of
introducing subtask level transfer learning is significantly higher than
original implementation.

_Sen-Making_ Model | Acc | $\Delta_{a}$
---|---|---
Our-single | 95.68 | -
w/o method b† | 94.88 | 0.80
w/o data augmentation | 95.43 | 0.25
w/o weighted sum fusion | 95.32 | 0.36
w/o subtask level transfer | 94.85 | 0.83
Baseline | 93.24 | 2.44

_Explanation_ Model | Acc | $\Delta_{b}$
---|---|---
Our-single | 95.48 | -
w/o hint sentence | 93.47 | 2.01
w/o data augmentation | 95.39 | 0.09
w/o weighted sum fusion | 95.11 | 0.37
w/o subtask level transfer | 94.98 | 0.50
Baseline | 93.12 | 2.36

Table 2: Ablation study on model components. $\dagger$ means we use the model
structure as Figure 1(a) and _Baseline_ means the model in Sec 3.2.

###  3.6 Ablation Study

To get better insight into our model architecture, we conduct an ablation
study on dev set of ComVE, and the results are shown in Table 2.

From the results we can see that _subtask level transfer learning_ has a
relatively large contribution for both subtask a and b, which confirms our
hypothesis that fine-tuning on similar task can improve the performance of the
current task. Data augmentation and weighted sum fusion also have minor
contributions due to the more robust dataset and more robust model.

For subtask a, we can see that compared with baseline method(Figure 1(a)),
concatenating another sentence as input can have a higher performance. We
speculate that the reason is traditional method treat two statements as
independent individuals, and our method takes into account the inherent
connection between the two statements.

For subtask b, we can see from Table 2 that _hint sentence_ makes a great
contribution the overall improvement. We think the reason is a common sense
statement with a similar grammar and syntax does help the model to determine
why the input sentence is against common sense.

##  4 Conclusions

This paper introduces our system for commonsense validation and explanation.
For _Sen-Making_ task, we use a novel pretraining language model based
architecture to pick out one of the two given statements that is against
common sense. For _Explanation_ task, we use a hint sentence mechanism to
improve the performance greatly. In addition, we propose a subtask level
transfer learning to share information between subtasks.

As future work, we plan to integrate the external knowledge base(such as
ConceptNet666http://www.conceptnet.io/) into commonsense inference.

## Acknowledgements

We thank the reviewers for their helpful comments. This work is supported by
the National Natural Science Foundation of China (No. 61572120) and the
Fundamental Research Funds for the Central Universities (No.N181602013).

## References

  * [Bowman et al., 2015]  Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning.  2015\.  A large annotated corpus for learning natural language inference.  In Lluís Màrquez, Chris Callison-Burch, Jian Su, Daniele Pighin, and Yuval Marton, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 632–642. The Association for Computational Linguistics.
  * [Davis, 2017]  Ernest Davis.  2017\.  Logical formalizations of commonsense reasoning: A survey.  J. Artif. Intell. Res., 59:651–723.
  * [Devlin et al., 2019]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2019\.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * [Joshi et al., 2019]  Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy.  2019\.  Spanbert: Improving pre-training by representing and predicting spans.  CoRR, abs/1907.10529.
  * [Kudo, 2018]  Taku Kudo.  2018\.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 66–75. Association for Computational Linguistics.
  * [Lan et al., 2020]  Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  2020\.  {ALBERT}: A Lite {BERT} for Self-supervised Learning of Language Representations.  In International Conference on Learning Representations.
  * [Liu et al., 2019]  Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  2019\.  Roberta: A robustly optimized BERT pretraining approach.  CoRR, abs/1907.11692.
  * [Loshchilov and Hutter, 2019]  Ilya Loshchilov and Frank Hutter.  2019\.  Decoupled weight decay regularization.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
  * [Peters et al., 2018]  Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  2018\.  Deep Contextualized Word Representations.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * [Rajpurkar et al., 2016]  Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  2016\.  Squad: 100, 000+ questions for machine comprehension of text.  In Jian Su, Xavier Carreras, and Kevin Duh, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383–2392. The Association for Computational Linguistics.
  * [Wang et al., 2019]  Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao.  2019\.  Does it make sense? and why? A pilot study for sense making and explanation.  In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4020–4026. Association for Computational Linguistics.
  * [Wang et al., 2020]  Cunxiang Wang, Shuailong Liang, Yili Jin, Yilong Wang, Xiaodan Zhu, and Yue Zhang.  2020\.  SemEval-2020 task 4: Commonsense validation and explanation.  In Proceedings of The 14th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  * [Wolf et al., 2019]  Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew.  2019\.  Huggingface’s transformers: State-of-the-art natural language processing.  ArXiv, abs/1910.03771.
  * [Xu et al., 2020]  Yige Xu, Xipeng Qiu, Ligao Zhou, and Xuanjing Huang.  2020\.  Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation.  arXiv:2002.10345 [cs], February.  arXiv: 2002.10345.
  * [Yang et al., 2019]  Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le.  2019\.  XLNet: Generalized Autoregressive Pretraining for Language Understanding.  In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 5754–5764.


# 3D MRI Synthesis with Slice-Based Latent Diffusion Models: Improving Tumor
Segmentation Tasks in Data-Scarce Regimes

###### Abstract

Despite the increasing use of deep learning in medical image segmentation, the
limited availability of annotated training data remains a major challenge due
to the time-consuming data acquisition and privacy regulations. In the context
of segmentation tasks, providing both medical images and their corresponding
target masks is essential. However, conventional data augmentation approaches
mainly focus on image synthesis. In this study, we propose a novel slice-based
latent diffusion architecture designed to address the complexities of
volumetric data generation in a slice-by-slice fashion. This approach extends
the joint distribution modeling of medical images and their associated masks,
allowing a simultaneous generation of both under data-scarce regimes. Our
approach mitigates the computational complexity and memory expensiveness
typically associated with diffusion models. Furthermore, our architecture can
be conditioned by tumor characteristics, including size, shape, and relative
position, thereby providing a diverse range of tumor variations. Experiments
on a segmentation task using the BRATS2022 confirm the effectiveness of the
synthesized volumes and masks for data augmentation. Code is available here :
https://github.com/Arksyd96/synthesis-with-slice-based-ldm

Index Terms—  Data Augmentation, Diffusion Models, Generative Modeling, MRI

##  1 Introduction

Deep learning has witnessed remarkable growth in medical imaging,
demonstrating its notable effectiveness segmentation tasks across various
imaging modalities, including MRI [1, 2]. However, the ongoing challenge of
limited access to annotated medical imaging data is a major challenge,
primarily due to the rarity of certain pathologies and rigorous medical
privacy regulations, consequently leading to a laborious and time-consuming
manual delineation of tumor masks by medical professionals. In this context,
data augmentation has emerged as an inseparable part of deep learning,
enabling models to overcome the limitations associated with a scarcity of
training samples and generalize more effectively the data. However, when
dealing with complex medical imaging structures, conventional augmentation
techniques such as rotations, cropping or noise injection, may introduce
deformations, resulting in deviations from the true data distribution. To
address these challenges, advanced deep learning-based data augmentation
techniques have been proposed, striving to generate synthetic samples that
closely resemble real data while preserving the semantic integrity of the
medical images [3, 4]. These models also offer privacy preservation and data
anonymization.

Generative Adversarial Networks (GANs) [5] have found widespread applications
in medical imaging [6, 3] and have been advocated in numerous literature
reviews for data augmentation due to their ability to generate realistic
images [7]. However, GANs exhibit certain limitations, including learning
instability, convergence issues, and the well-documented problem of mode
collapse [8], where the generator produces a limited range of samples. In
contrast, Variational Autoencoders (VAEs) [9] have been proposed as an
alternative to GANs, offering a more stable training process and a more
efficient inference procedure. However, VAEs are also known to produce blurry
images [3] and are incapable of generating high-resolution images. Recently,
diffusion models have emerged as a promising method for image synthesis,
offering superior image quality and realism compared to GANs while maintaining
a good mode coverage [10]. This has led to the rise of these models and the
development of various alternatives, such as the Latent Diffusion Model (LDM)
[11]. Although these models provide an attractive solution to the challenge of
limited training data, a common issue arises from their high computational
cost and demanding memory requirements, making them impractical for 3D medical
image synthesis. This holds particularly true for diffusion-based models,
which are more resource-intensive, presenting challenges in their integration
into clinical routines, especially for real-time tasks like data harmonization
or imputation. Beyond this, generative models also require a significant
amount of data, limiting their feasibility in medical imaging.

Fig. 1: Illustration of the proposed architecture. Initially, an MRI volume
and its associated 3D mask are decomposed into multiple pairs of 2D slices and
masks, denoted as $\\{x_{i},m_{i}\\}$. These are fed into the encoder
$\mathcal{E}$. $l(.)$ and $\tau(.)$ are the positional and condition
embedders, respectively. $c$ represents the tumor features vector. See section
2.2.

Recent studies have primarily concentrated on image generation or translation
[12, 13], which, in the context of tumor segmentation, is insufficient. The
importance lies in generating both images and their corresponding tumor masks,
as these masks serve as ground truth for segmentation tasks, adding complexity
and cost to the generation process, as we must generalize not only the medical
image but also the associated mask. In this study, we introduce a lightweight
variant of latent diffusion models (named SBLDM), employing a slice-by-slice
approach for the simultaneous generation of medical images and corresponding
segmentation masks. Our architecture is trained under data limitations, and we
demonstrate its efficacy in augmenting training data for segmentation tasks.
Moreover, our model allows precise control of tumor size, shape, and relative
position—enabling the generation of a diverse range of tumor variations. This
conditionning also serves as regularization to our model. Our evaluation
encompasses the quality of generated images, followed by a comprehensive
assessment in the context of 3D segmentation tasks using synthesized volumes.

This paper contains three main contributions:

  * •

Proposition of an efficient, slice-by-slice diffusion model for the
simultaneous generation of high-quanlity medical images and associated
segmentation masks with tumor feature controlling.

  * •

Highlighting the strength of our approach in data-scarce environments, in
contrast to the data-intensive nature of GAN-based architectures.

  * •

Comprehensive evaluation showcasing the effectiveness of our high-quality
synthesized MRIs in enhancing segmentation tasks.

##  2 Method

###  2.1 Diffusion models

Diffusion models [14, 15] are a subset of generative models based on a
forward-and-backward diffusion process. This stochastic process can be thought
of as a parameterized Markov chain with a fixed number of time steps, denoted
as $T$. During the forward, Gaussian noise is gradually added to an initial
data point $x_{0}\sim q(x_{0})$, following a predefined variance scheduler
$\beta_{1},\dots,\beta_{T}$:

$q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\mathbf{I})$
(1)

During backward, the model is trained to reverse the forward process starting
with a Gaussian noise $x_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, and
reconstructing it into the initial data distribution $q(x_{0})$ with learned
parameters $\theta$. This process can be expressed as:

$p_{\theta}(x_{t-1}|x_{t})=\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t},t);\sigma^{2}_{t})$
(2)

During training, the model attempts to predict the added noise to $x_{0}$,
denoted as $\epsilon$, and extracts it at each step from $x_{t}$ to recreate
the original sample. To learn the parameters $\theta$, such that
$p_{\theta}(x_{t-1}|x_{t})$ approximates $q(x_{t-1}|x_{t})$, maximum
likelihood estimation with variational inference is used, that is similar to
VAEs [9] maximizing the evidence lower bound (ELBO). The loss function is
finally defined as the mean squared error between the added noise $\epsilon$
and the predicted $\hat{\epsilon}$.

Latent diffusion models (LDM) [11] represent a variant of diffusion models
that introduces a two-stage process. Initially, data is projected into a
lower-dimensional latent space, typically learned through an autoencoder. The
diffusion model then operates in this latent space, generating new latent
variables. These latents are subsequently transformed back into pixel space
using a decoder.

###  2.2 Slice-based latent diffusion model (SBLDM)

We propose a new method in response to the difficulties encountered in
training 3D diffusion models, arising from the considerable computational
expenses and memory constraints, as well as the need for substantial data
quantities to avoid overfitting. Our methodology leverages a 2-dimensional VAE
with a positional embedder to encode the volumetric data in a slice-by-slice
manner. Decomposing volumes into individual slices enables the construction of
a larger 2D dataset with increased variance and greater diversity in modes.
The accurate positioning of each slice facilitates the use of 2D autoencoders
for 3D volume generation, enhancing the autoencoder’s capacity to focus on
individual slices, and leading to improved generalization.

Our architecture is based on a latent diffusion model that jointly generates
3D MRI volumes and the corresponding tumor mask (Figure 1). A positional
embedder $l(.)$ is introduced in our architecture, allowing it to acquire an
understanding of the relative position of each slice within the volume. This
additional layer of supervision equips the autoencoder with more spatial
awareness. The encoder $\mathcal{E}$ is modeled as conditional distribution
$q_{\phi}(z_{i}|x_{i},m_{i},l(i))$ where $x_{i},m_{i}\in\mathbb{R}^{W\times
H}$ are the image and its corresponding segmentation mask at slice $i\leq D$,
and $l(i)$ is the embedding of the slice $i$. The encoder projects samples
into a lower-dimensional representation
$z_{i}\in\mathbb{R}^{W^{{}^{\prime}}\times H^{{}^{\prime}}}$ that encaspulates
common characteristics between the image and its associated segmentation mask
for a given slice. On the other hand, the decoder $\mathcal{D}$ can be seen as
conditional joint distribution $p_{\theta}(x_{i},m_{i}|z_{i},l(i))$ that
reconstructs the image and masks pairs given the latent representationsand the
relative position of the slice. Subsequently, all these individual $z_{i}$ are
amalgamated to form a 3D latent space, denoted as
$Z=\cup_{i}z_{i}\in\mathbb{R}^{W^{{}^{\prime}}\times H^{{}^{\prime}}\times
D}$. Subsequently, a diffusion model is trained to capture not only the
broader latent variable distribution but also the implicit volumetric
dimension introduced through the concatenation of the latent representations.

###  2.3 Conditionning on the tumor characteristics

We further propose to control our model based on tumor characteristics,
allowing us to control the size, shape, and relative position of the tumor.
This conditionning also serves as regularization to our model, improving
supervision and mitigating overfitting. Additionally, this conditioning helps
address scenarios where models might generate tumor-free volumes. By
specifying the position, we reinforce the constraint of adding a visible tumor
to the synthesized data. The tumor’s size and shape are quantified through
parameters such as voxel volume, surface area, and sphericity. Meanwhile, its
relative position is determined by the coordinates of its center of mass
$(x,y,z)$ and its dimensions $(w,h,d)$, collectively forming a bounding box
around the tumor. To enable this level of control, we leverage a conditioning
vector, which is passed through the Multilayer Perceptron $\tau$ (Figure 1) to
encode these parameters into a feature vector. This feature vector is
subsequently fused with the main latent representation $Z$ during the
diffusion process using a scale-shift norm.

##  3 Experimentations

###  3.1 Dataset

We evaluate the efficacy of our proposed method using the publicly available
dataset: BRAin Tumor Segmentation (BRATS2022) [16] proposes multi-modal MRIs
with a volume shape of 240$\times$240$\times$155 and a voxel resolution of
1$\times$1$\times$1 $mm^{3}$. The images are skull-stripped and co-registered
to the same anatomical template. The ground truth segmentation masks are
provided for the tumor core (TC), enhancing tumor (ET), and whole tumor (WT)
regions forming three tumor labels. In our experiments, we only consider FLAIR
modality and the WT region.

###  3.2 Implementation details

We deliberately limited our training set to only 100 volumes to simulate a
data-scarce scenario, and evaluations are made on an another set of 100
volumes. To accommodate memory limitations for comparative methods, all
volumes were resized to $192\times 192\times 96$ dimensions. We employ a VAE
as an autoencoder with a downsampling factor of 4 and a U-Net [17] for the
diffusion. Our architecture excludes attention modules and utilizes only one
residual block per resolution. The experiments were conducted on an NVIDIA
GeForce RTX A6000 GPU with 48GB of VRAM, using the Adam optimizer. We employed
a learning rate of $1e-5$. For the segmentation task, we utilize the nnUNet
[18] framework with default settings, including for the standard data
augmentation.

###  3.3 Quantitative results

We conducted a quantitative comparison of the generated volumes using our
method against two other state-of-the-art techniques. This includes a 3D Least
Squares GAN (3D-LSGAN) [19] with a backbone inspired from Deep Convolutional
GAN [20] and a 3D version of the original latent diffusion model (3D-LDM)
[11], wherein the autoencoder is defined as a VAE-GAN [21] with a downsampling
factor of 4. Standard 3D pixel-space diffusion models (DDPM) [15] were
excluded due to their memory consumption and unreasonable sampling time. The
Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR), as
well as the number of parameters and sampling time of each method are chosen
for evaluation. Our results demonstrate that our method achieves the highest
SSIM of 0.731 and the top PSNR of 21.701 (see Table 1). All accomplished while
maintaining an efficient parameter count. Despite its notable efficiency in
terms of architecture and sampling time, the 3D-LSGAN experiences the most
significant quality impact, primarily due to the data scarcity issue, which
makes it impractical for augmentation. GAN-based architectures are notably
data-intensive, in contrast to likelihood-based models like 3D-LDM and our
approach. To enhance sampling time, we implemented a DDIM sampling scheme [22]
with our method, limiting the number of steps to 50. This optimization results
in notable time savings, with only a negligible loss in quality.

Methods | PSNR $\uparrow$ | SSIM $\uparrow$ | #params $\downarrow$ | Sampling time $\downarrow$
---|---|---|---|---
3D-LSGAN | 20.091 | 0.601 | 71M | 0.002s
3D-LDM | 21.034 | 0.677 | 728M | 13.324s
SBLDM (Ours) | 21.466 | 0.731 | 159M | 30.900s
SBLDM (DDIM) | 21.701 | 0.726 | 159M | 1.965s

Table 1: Quantitative performance of the proposed generative models on the
BRATS datasets.  Fig. 2: Comparison of images generated using our proposed
method and other generative models on the BRATS datasets.

###  3.4 Qualitative results

We present a qualitative comparison between samples synthesized using
different methods from an axial view in Figure 2. Our method’s samples exhibit
a higher level of realism and fine-grained details, being nearly
indistinguishable from real volumes. The mode-coverage is further enriched by
the conditionning we offer during the generation process. In contrast, the
3D-LDM’s autoencoder trained with only 100 volumes, produces images that are
slightly blurrier and lack fine-grained details, indicative of autoencoder
underfitting. As for the 3D-LSGAN, the results are subpar, with minimal brain
structure and limited details, accompanied by visible image artifacts. Given
the scarcity of data, mode collapse is challenging to mitigate at this stage.
We do not present images from sagittal and coronal views due to their low
quality on the BRATS dataset, rendering them less informative. In Figure 3, we
illustrate some conditioned generations of MRIs and corresponding masks using
our method. By varying the voxel volume parameter, we generate tumors ranging
from small to large while staying within the same brain region. These results
demonstrate not only the brain’s variability with each generation but also our
method’s successful adherence to size and position constraints.

###  3.5 Using synthetic volumes on a segmentation task

We conducted segmentation model training using synthetically generated data,
highlighting their potential as effective data augmentors. Our augmentation
pipeline involves data generation from the restricted training set and its
combination with the synthetically generated images, thereby creating an
augmented dataset, as outlined in [3].

To evaluate our approach, we initially report results with the original
restricted dataset and its augmented version with a factor of $\times$5\.
Then, we enlarge the real dataset with an additional 100 synthesized volumes,
generated through each respective method. We also train the nnUNet using only
synthetic volumes to measure their contribution and individual impact. Our
findings, presented in Table 2, underscore the ability of our synthesized
images to improve the segmentation task results, achieving a DSC score of
$0.815$ and IoU of $0.715$. This significantly outperforms the corresponding
scores for other methods. Notably, the GAN-based architecture appears to
deteriorate the results compared to the baseline. This decline can be
attributed to the poor quality of the volumes and the occurrence of mode-
collapse. Furthermore, we combine both standard augmentation approaches and
our synthetic volumes, which leads to even more substantial improvements. We
set the number of synthetic volumes to 100 (factor $\times 2$), as we observe
that beyond this threshold, the improvement in DSC reaches a plateau. This
observation can be attributed to the limited number of modes covered by the
synthetic images.

Fig. 3: Figure showcasing volumes with varying tumor Sizes, ranging from 0.0
to 1.0. The tumor position is fixed.

Methods |  DSC $\uparrow$ |  IoU $\uparrow$
---|---|---
Real volumes |  0.714$\pm$0.05 |  0.592$\pm$0.04
Augmented volumes ($\times$5) |  0.806$\pm$0.02 |  0.716$\pm$0.02
Synth only | 3D-LSGAN |  0.355$\pm$0.13 |  0.302$\pm$0.11
| 3D-LDM |  0.529$\pm$0.07 |  0.401$\pm$0.06
| SBLDM (ours) |  0.673$\pm$0.04 |  0.551$\pm$0.04
Real + Synth | 3D-LSGAN |  0.623$\pm$0.08 |  0.514$\pm$0.07
| 3D-LDM |  0.705$\pm$0.03 |  0.583$\pm$0.02
| SBLDM (ours) |  0.815$\pm$0.02 |  0.715$\pm$0.02
Real + SBLDM synth + Aug. |  0.834$\pm$0.01 |  0.739$\pm$0.01

Table 2: Quantitative performance of the segmentation task in Dice coefficient
(DSC), and Intersection-over-Union (IoU).

##  4 Conclusion

In this paper, we present a slice-based latent diffusion model for the joint
synthesis of 3D MRI volumes and their segmentation masks in data-scarce
regimes. Our approach offers practical advantages, requiring fewer
computational and memory resources compared to traditional pixel-space and
standard latent diffusion models. Our future work will focus on adapting our
approach for multi-modal MRI synthesis.

## References

  * [1] Alexander Selvikvåg Lundervold and Arvid Lundervold,  “An overview of deep learning in medical imaging focusing on mri,”  Zeitschrift für Medizinische Physik, vol. 29, no. 2, pp. 102–127, 2019.
  * [2] Tongxue Zhou, Stéphane Canu, Pierre Vera, and Su Ruan,  “Latent correlation representation learning for brain tumor segmentation with missing mri modalities,”  IEEE Transactions on Image Processing, vol. 30, pp. 4263–4274, 2021.
  * [3] Aghiles Kebaili, Jérôme Lapuyade-Lahorgue, and Su Ruan,  “Deep learning approaches for data augmentation in medical imaging: A review,”  Journal of Imaging, vol. 9, no. 4, pp. 81, 2023.
  * [4] Ying Song, Shuangjia Zheng, Liang Li, Xiang Zhang, Xiaodong Zhang, et al.,  “Deep learning enables accurate diagnosis of novel coronavirus (covid-19) with ct images,”  IEEE/ACM transactions on computational biology and bioinformatics, vol. 18, no. 6, pp. 2775–2780, 2021.
  * [5] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio,  “Generative adversarial networks,”  arXiv preprint arXiv:1406.2661, 2014.
  * [6] Jinbao Wang, Guoyang Xie, et al.,  “Fedmed-gan: Federated domain translation on unsupervised cross-modality brain image synthesis,”  Neurocomputing, vol. 546, pp. 126282, 2023.
  * [7] Yizhou Chen, Xu-Hua Yang, Zihan Wei, Ali Asghar Heidari, et al.,  “Generative adversarial networks in medical image augmentation: a review,”  Computers in Biology and Medicine, p. 105382, 2022.
  * [8] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin,  “Which training methods for gans do actually converge?,”  in International conference on machine learning. PMLR, 2018, pp. 3481–3490.
  * [9] Diederik P Kingma and Max Welling,  “Auto-encoding variational bayes,”  arXiv preprint arXiv:1312.6114, 2013.
  * [10] Prafulla Dhariwal and Alexander Nichol,  “Diffusion models beat gans on image synthesis,”  Advances in Neural Information Processing Systems, vol. 34, pp. 8780–8794, 2021.
  * [11] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer,  “High-resolution image synthesis with latent diffusion models,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10684–10695.
  * [12] Zolnamar Dorjsembe, Hsing-Kuo Pao, Sodtavilan Odonchimed, and Furen Xiao,  “Conditional diffusion models for semantic 3d medical image synthesis,”  arXiv preprint arXiv:2305.18453, 2023.
  * [13] Lan Jiang, Ye Mao, Xiangfeng Wang, Xi Chen, and Chao Li,  “Cola-diff: Conditional latent diffusion model for multi-modal mri synthesis,”  in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2023, pp. 398–408.
  * [14] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli,  “Deep unsupervised learning using nonequilibrium thermodynamics,”  in International Conference on Machine Learning. PMLR, 2015, pp. 2256–2265.
  * [15] Jonathan Ho, Ajay Jain, and Pieter Abbeel,  “Denoising diffusion probabilistic models,”  Advances in Neural Information Processing Systems, vol. 33, pp. 6840–6851, 2020.
  * [16] Ujjwal Baid et al.,  “The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification,”  arXiv preprint arXiv:2107.02314, 2021.
  * [17] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241.
  * [18] Fabian Isensee et al.,  “nnu-net: Self-adapting framework for u-net-based medical image segmentation,”  arXiv preprint arXiv:1809.10486, 2018.
  * [19] H Xie, RYK Lau, W Zhen, and SP Smolley,  “Least squares generative adversarial networks,”  arXiv, 2016.
  * [20] Alec Radford, Luke Metz, and Soumith Chintala,  “Unsupervised representation learning with deep convolutional generative adversarial networks,”  arXiv preprint arXiv:1511.06434, 2015.
  * [21] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, and Ole Winther,  “Autoencoding beyond pixels using a learned similarity metric. corr abs/1512.09300 (2015),”  arXiv preprint arxiv:1512.09300, 2015.
  * [22] Jiaming Song, Chenlin Meng, and Stefano Ermon,  “Denoising diffusion implicit models,”  arXiv preprint arXiv:2010.02502, 2020.