# Improving Cross-Lingual Reading Comprehension with Self-Training

Wei-Cheng Huang∗
National Taiwan University
<EMAIL_ADDRESS>
&Chien-yu Huang
National Taiwan University
<EMAIL_ADDRESS>
&Hung-yi Lee
National Taiwan University
<EMAIL_ADDRESS>∗ These authors contributed equally.

###### Abstract

Substantial improvements have been made in machine reading comprehension,
where the machine answers questions based on a given context. Current state-
of-the-art models even surpass human performance on several benchmarks.
However, their abilities in the cross-lingual scenario are still to be
explored. Previous works have revealed the abilities of pre-trained
multilingual models for zero-shot cross-lingual reading comprehension. In this
paper, we further utilized unlabeled data to improve the performance. The
model is first supervised-trained on source language corpus, and then self-
trained with unlabeled target language data. The experiment results showed
improvements for all languages, and we also analyzed how self-training
benefits cross-lingual reading comprehension in qualitative aspects.

##  1 Introduction

Machine reading comprehension has gained much attention because of its
practical applications, and substantial improvements have been made recently.
Pre-trained models such as BERT Devlin et al. (2019) and GPT family Radford et
al. ; Brown et al. (2020) achieved state-of-the-art performance, or even
surpassed the humans, in several benchmarks, including SQuAD Rajpurkar et al.
(2016, 2018), HotpotQA Yang et al. (2018), and CoQA Reddy et al. (2019).

However, previous works mostly focused on monolingual reading comprehension,
and large-scale data is usually available only in English. Several
multilingual datasets were then created for development Cui et al. (2019); Liu
et al. (2019); Lewis et al. (2019). Also, techniques such as transfer learning
or domain adaptation thus become important for cross-lingual reading
comprehension. Hsu et al. (2019) showed the ability of multilingual BERT
(m-BERT) for zero-shot cross-lingual reading comprehension. They first trained
an m-BERT on source language reading comprehension corpus, and the model is
then able to be applied to those in another target language without further
fine-tuning.

On the other hand, self-training has been attractive in knowledge transfer. In
self-training, a model is first trained on a labeled dataset. Then, the
trained model gives pseudo-labels to unlabeled data. A subset of these pseudo-
labeled data is selected into the labeled dataset for the next training epoch.
This procedure is iterated several times.

This is the first paper using self-training to improve cross-lingual reading
comprehension. In real applications, we may still have access to sufficient
target language data, but without well-annotated answers. We thus propose to
utilize these data by self-training. Model is first trained on a source
language corpus with annotation and then self-trained on target language data
where only contexts and questions are available. Pseudo-labels are dynamically
obtained depending on the model’s confidence during self-training. Experiment
results showed improvements among all corpora in four different languages,
compared to previous zero-shot performance. We also analyzed how self-training
works for cross-lingual reading comprehension.

##  2 Related Works

###  2.1 Cross-lingual transfer learning

While there are hundreds of languages in the world, large-scale datasets that
are crucial for machine learning, are usually only available in English.
Transfer learning for cross-lingual understanding thus becomes an important
issue. Kim et al. (2017) utilized adversarial training to obtain language-
general information without ancillary resources and achieved better
performance on POS tagging. Schuster et al. (2019) used a machine translation
encoder to obtain cross-lingual contextualized representations and it
outperformed pre-trained multilingual ELMo Peters et al. (2018) on text
classification and sequence labeling.

With the development of pre-trained models such as BERT Devlin et al. (2019),
zero-shot cross-lingual transfer became more feasible. Such cross-lingual
transfer has been studied in several tasks like text classification Eriguchi
et al. (2018), dependency parsing Wang et al. (2019), text summarization Duan
et al. (2019), and reading comprehension Hsu et al. (2019).

###  2.2 Self-training

Self-training has succeeded in several domains. Zoph et al. (2020) showed the
generality and flexibility of self-training and achieved better performance
compared to pre-trained models in computer vision. Such techniques have also
been applied for speech recognition Kahn et al. (2020); Chen et al. (2020) and
speech translation Pino et al. (2020).

In NLP, self-training has also been studied and applied in several aspects. Du
et al. (2020) showed that self-training is complementary to pre-training in
standard text classification benchmarks. Dong and de Melo (2019) used self-
training on cross-lingual document and sentiment classification. However, the
effectiveness of self-training on cross-lingual reading comprehension remained
unknown, since it requires more reasoning and understanding compared to simple
text classification, which led to the idea of self-training on cross-lingual
reading comprehension as proposed here.

##  3 Proposed Approach

A well-labeled reading comprehension dataset $D$ is composed of at least three
components: contexts $C=\\{c_{1},c_{2},\ldots,c_{n}\\}$, questions
$Q=\\{q_{11},q_{12},\ldots,q_{nm}\\}$, and corresponding answers
$A=\\{a_{11},a_{12},\ldots,a_{nm}\\}$, where $q_{ij}$ denotes the $j$-th
question of $i$-th context and $a_{ij}$ is the corresponding answer. In this
paper, we assume that we have access to a labeled dataset
$D_{S}=\\{C_{S},Q_{S},A_{S}\\}$ in source language $S$ and a unlabeled dataset
$D_{T}=\\{C_{T},Q_{T}\\}$ in target language $T$. Given a pre-trained
multilingual model, we first fine-tune it on the labeled dataset $D_{S}$
(fine-tuning stage), and then use $D_{T}$ for self-training (self-training
stage).

###  3.1 Self-training process

The complete two-stage process works as specified in Algorithm 1. First, in
fine-tuning stage, the pre-trained multilingual model $M_{P}$ is fine-tuned on
the labeled source language dataset $D_{S}$ in a supervised manner. The model,
denoted as $M_{0}$, is then used as the starting point in the next stage. In
the self-training stage, the training and labeling process iterate several
times. For each iteration $i$, we first generate pseudo-labels for $D_{T}$
with model $M_{i}$ (Sec. 3.2), and train $M_{0}$ on $D_{T}$ with these labels.

Algorithm 1 Cross-lingual self-training process.

1:

2:$M_{P}$: Pre-trained multilingual model.

3:$D_{S}$: Labeled dataset in the source language.

4:$D_{T}$: Unlabeled dataset in the target language.

5:$N$: Number of self-training iterations.

6:$\theta$: Threshold for filtering pseudo-labels.

7:$M_{0}\leftarrow\text{train}(M_{P},D_{S})$

8:$D_{0}\leftarrow\text{label}(M_{0},D_{T},\theta)$

9:for each $i\in[1,N]$ do

10: $M_{i}\leftarrow\text{train}(M_{0},D_{i-1})$

11: $D_{i}\leftarrow\text{label}(M_{i},D_{T},\theta)$

12:end for

13:return $M_{N}$

###  3.2 Labeling process

To obtain the pseudo-labels, we run the unlabeled set through the model.
Algorithm 2 explains the labeling process. For each unlabeled example
$(c_{i},q_{ij})$, the model gives $p$ answers
$A_{ij}=\\{a_{ij1},a_{ij2},\ldots,a_{ijp}\\}$ with corresponding confidences
$T_{ij}=\\{\tau_{ij1},\tau_{ij2},\ldots,\tau_{ijp}\\}$. We then filter data by
a threshold $\theta$. Only examples whose greatest confidence exceeds the
threshold are preserved. For each question, the answer with the greatest
confidence is used as the pseudo-label.

Algorithm 2 Pseudo-label generation.

1:

2:$M$: Model used in self-training process.

3:$D_{T}$: Unlabeled dataset in the target language.

4:$\theta$: Confidence threshold.

5:$D^{{}^{\prime}}\leftarrow\emptyset$

6:for $c_{i}\in C_{T}$ do

7: for $q_{ij}\in Q_{T}$ do

8: $A_{ij},T_{ij}\leftarrow M(c_{i},q_{ij})$

9: if $\max_{\tau}T_{ij}\geq\theta$ then

10: $k\leftarrow\operatorname*{arg\,max}T_{ij}$

11: $D^{{}^{\prime}}\leftarrow
D^{{}^{\prime}}\cup\\{(c_{i},q_{ij},a_{ijk})\\}$

12: end if

13: end for

14:end for

15:return $D^{{}^{\prime}}$

##  4 Experimental Settings

###  4.1 Model

We utilized publicly-available pre-trained m-BERT Wolf et al. (2020) in our
experiment. The m-BERT is a transformer-based encoder pre-trained on
multilingual Wikipedia and can perform well on several cross-lingual tasks.

###  4.2 Datasets

We used 4 datasets in different languages in the experiments: SQuAD v1.1
Rajpurkar et al. (2016) in English, FQuAD d’Hoffschmidt et al. (2020) in
French, KorQuAD v1.0 Lim et al. (2019) in Korean, and DRCD Shao et al. (2018)
in traditional Chinese. These datasets were collected from Wikipedia in
different languages, and all of the questions are answerable. However, the
test set of FQuAD is not available, and we thus evaluated our approaches on
development sets for all languages.

###  4.3 Pseudo-labeling

In extraction-based reading comprehension, the model answers the question by
yielding two probability distributions over the context tokens. One is for the
start position, and the other is for the end position. The answer is then
extracted by selecting the span between the start and end positions.

Here we consider the probabilities as a measure of confidence. More
specifically, we took the distributions of start and end tokens as input, then
filtered out impossible candidates (for example, candidates with end token
before start token), and performed beam search with the summation of start/end
distributions as confidence. The threshold $\theta$ was set to 0.7, which
means only the examples whose most confident answer with a score greater than
0.7 were used in the self-training stage.

Dataset | EM | F1 | Dataset | EM | F1
---|---|---|---|---|---
Source | Target | Zero-shot | Self-trained | Zero-shot | Self-trained | Source | Target | Zero-shot | Self-trained | Zero-shot | Self-trained
SQuAD | FQuAD | 53.39 | 57.59 | 72.78 | 75.75 | DRCD | SQuAD | 56.33 | 66.26 | 69.22 | 77.49
DRCD | 67.00 | 71.25 | 82.94 | 86.65 | FQuAD | 35.10 | 42.57 | 54.83 | 63.52
KorQuAD | 53.22 | 61.07 | 75.63 | 83.99 | KorQuAD | 56.62 | 61.71 | 79.77 | 85.70
FQuAD | SQuAD | 68.56 | 72.41 | 79.84 | 82.67 | KorQuAD | SQuAD | 61.69 | 67.01 | 71.74 | 76.80
DRCD | 49.74 | 65.38 | 72.90 | 81.33 | FQuAD | 39.46 | 42.35 | 56.24 | 59.02
KorQuAD | 55.68 | 58.76 | 73.62 | 82.17 | DRCD | 76.14 | 77.89 | 84.11 | 85.77

Table 1:  Comparison of the performance (EM and F1) between zero-shot and
self-trained m-BERTs on different datasets. The pre-trained m-BERT was first
fine-tuned on the source language, and then self-trained and evaluated on the
target language.

##  5 Results

###  5.1 Quantitative results

Following the convention, we used exact match (EM) and F1 scores to evaluate
the performance. Since we used 4 datasets in the experiments, there are
therefore 12 source-target pairs to be evaluated. The notation source-target
denotes that the model was fine-tuned on the source dataset, and then self-
trained and evaluated on the target dataset. Table 1 presents the results on
all 12 pairs across the four datasets. The m-BERT was first trained on the
source language dataset, self-trained on the training set of the target
language, and evaluated on the development set of the target language. Zero-
shot means the m-BERT is only trained on the source language without self-
training. We see that both EM and F1 were enhanced across all pairs with the
help of self-training, compared to the original zero-shot ones (+1.75 to
+15.64 on EM and +1.66 to +8.69 on F1). The consistent improvements show that
our approach is general but not limited to specific languages. Table 2 lists
the F1 scores in each iteration when m-BERT was first fine-tuned on SQuAD and
self-trained on other datasets. In each iteration, almost all models were made
better, and the first iteration benefited the most, as there are obvious gaps
between zero-shot and iteration 1 for all models on both EM and F1 scores.

Target | Zero-shot | Iter. 1 | Iter. 2 | Iter. 3
---|---|---|---|---
FQuAD | 72.78 | 74.19 | 74.96 | 75.75
DRCD | 82.94 | 85.55 | 85.59 | 86.64
KorQuAD | 75.63 | 82.00 | 83.27 | 83.99
Table 2: Comparison of F1 scores of each iteration when m-BERT was first fine-
tuned on SQuAD and self-trained on different datasets.

###  5.2 Qualitative analysis

We then investigated how our approach enhances the performance. Specifically,
we studied the cross-lingual transfer on SQuAD, DRCD, and FQuAD in terms of
the question and answer types. We classified questions and answers with the
same procedure mentioned in their original papers, and the details are
described in the appendix. KorQuAD was excluded here because the question and
answer types were not specified clearly in the original paper, and we did not
found a publicly-available reliable tagging toolkit for Korean.

Figure 1: The zero-shot F1 scores with corresponding improvements in first
iteration in self-training stage for different types of questions.

We first analyzed how the scores varied concerning its original zero-shot
scores. We wondered whether the larger amount of labels brings more benefits
to their corresponding types of questions. However, we did not see an obvious
trend here, which indicates that the amount of pseudo-labels is not the key to
the improvements for different question types (refer to the appendix for the
figures).

We then looked into the relation between the zero-shot scores and
corresponding improvements for different types of questions in the first
iteration of self-training, as shown in Figure 1. We see that self-training
mostly improved the examples that were not answered well in the zero-shot
inference (low F1 score on $x$-axis with good improvement on $y$-axis). We
also present the results for answer types but in the appendix due to the space
limit.

Question Types | DRCD-SQuAD | FQuAD-SQuAD
---|---|---
Who | 5.45 | -0.3
What | 5.89 | 1.95
When | 16.98 | -0.05
Where | 4.44 | 3.04
Why | 8.27 | 1.77
Which | 4.07 | 0.46
How | 8.41 | 6.12
Other | 3.70 | 2.63
Table 3: The F1 improvements ($\Delta$F1) of each question type in the first iteration when self-trained on SQuAD. Question Types | SQuAD-DRCD | FQuAD-DRCD
---|---|---
Who | 4.26 | 9.83
What | 3.16 | 8.21
When | 5.64 | 10.59
Where | 1.14 | 3.41
Why | 0.66 | 0.42
Which | 1.64 | 3.3
How | 8.55 | 6.19
Other | 2.19 | 4.02
Table 4: The F1 improvements ($\Delta$F1) of each question type in the first iteration when self-trained on DRCD. Question Types | SQuAD-FQuAD | DRCD-FQuAD
---|---|---
Who | 1.32 | 3.65
What (quoi) | -0.16 | 6.08
What (que) | -0.16 | 7.54
When | -3.03 | 4.25
Where | 12.90 | 2.28
Why | 1.42 | -13.17
How | 4.41 | 12.18
How many | 6.57 | 8.24
Other | 1.00 | 5.74
Table 5: The F1 improvements ($\Delta$F1) of each question type in the first
iteration when self-trained on FQuAD.

We further compared the improvements of each question and answer type in the
first iteration. Table 3, 4 and 5 show the results for question types, and
those for answer types are in the appendix. The question types vary in the
tables due to different categories in their original papers. We found that in
the SQuAD-DRCD pair, ”How” and ”When” for questions and ”Numeric” for answers
improved the most in terms of F1 scores. Similar results were obtained in the
DRCD-SQuAD pair with ”When” and ”How” questions and ”Date” and ”Verb” answers.
In the FQuAD-SQuAD pair, ”How” questions and ”Adjective” and ”Other Numeric”
answers improved the most, but ”Where” improved instead of ”When”. This
suggests our approach helps with the numeric questions and date answers the
most. Also, such improvements between DRCD and SQuAD were more than those
between FQuAD and SQuAD. We believe this is because their formats in Chinese
much differs from those in English.

##  6 Conclusion

This paper presents the first self-training approach to improve cross-lingual
machine reading comprehension. The experiments were conducted on large-scale
datasets in four different languages. The results showed that our approach
improved the performance significantly compared to the baseline with 1 – 16 EM
and 1 – 8 F1 scores. We also analyzed how self-training improves cross-lingual
reading comprehension in several aspects and found that improvements are
correlated to zero-shot performance but not the number of pseudo-labels.

## Acknowledgments

We are grateful to Taiwan Computing Cloud (https://www.twcc.ai/) for providing
computation resources.

## References

  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Chen et al. (2020) Yi-Chen Chen, Zhaojun Yang, Ching-Feng Yeh, Mahaveer Jain, and Michael L Seltzer. 2020.  Aipnet: Generative adversarial pre-training of accent-invariant networks for end-to-end speech recognition.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6979–6983. IEEE.
  * Cui et al. (2019) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2019\.  Cross-lingual machine reading comprehension.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1586–1595.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * d’Hoffschmidt et al. (2020) Martin d’Hoffschmidt, Maxime Vidal, Wacim Belblidia, and Tom Brendlé. 2020.  Fquad: French question answering dataset.  _arXiv preprint arXiv:2002.06071_.
  * Dong and de Melo (2019) Xin Luna Dong and Gerard de Melo. 2019.  A robust self-learning framework for cross-lingual text classification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6307–6311.
  * Du et al. (2020) Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoyanov, and Alexis Conneau. 2020.  Self-training improves pre-training for natural language understanding.  _arXiv preprint arXiv:2010.02194_.
  * Duan et al. (2019) Xiangyu Duan, Mingming Yin, Min Zhang, Boxing Chen, and Weihua Luo. 2019.  Zero-shot cross-lingual abstractive sentence summarization through teaching generation and attention.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3162–3172.
  * Eriguchi et al. (2018) Akiko Eriguchi, Melvin Johnson, Orhan Firat, Hideto Kazawa, and Wolfgang Macherey. 2018.  Zero-shot cross-lingual classification using multilingual neural machine translation.  _arXiv preprint arXiv:1809.04686_.
  * Hsu et al. (2019) Tsung-Yuan Hsu, Chi-Liang Liu, and Hung-yi Lee. 2019.  Zero-shot reading comprehension by cross-lingual transfer learning with multi-lingual language representation model.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5935–5942.
  * Kahn et al. (2020) Jacob Kahn, Ann Lee, and Awni Hannun. 2020.  Self-training for end-to-end speech recognition.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7084–7088. IEEE.
  * Kim et al. (2017) Joo-Kyung Kim, Young-Bum Kim, Ruhi Sarikaya, and Eric Fosler-Lussier. 2017.  Cross-lingual transfer learning for pos tagging without cross-lingual resources.  In _Proceedings of the 2017 conference on empirical methods in natural language processing_ , pages 2832–2838.
  * Lewis et al. (2019) Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019.  Mlqa: Evaluating cross-lingual extractive question answering.  _arXiv preprint arXiv:1910.07475_.
  * Lim et al. (2019) Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019.  Korquad1. 0: Korean qa dataset for machine reading comprehension.  _arXiv preprint arXiv:1909.07005_.
  * Liu et al. (2019) Jiahua Liu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2019.  Xqa: A cross-lingual open-domain question answering dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2358–2368.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Manning et al. (2014) Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014.  The stanford corenlp natural language processing toolkit.  In _Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations_ , pages 55–60.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237.
  * Pino et al. (2020) Juan Pino, Qiantong Xu, Xutai Ma, Mohammad Javad Dousti, and Yun Tang. 2020.  Self-training for end-to-end speech translation.  _Proc. Interspeech 2020_ , pages 1476–1480.
  * (20) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for squad.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D Manning. 2019.  Coqa: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266.
  * Schuster et al. (2019) Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019.  Cross-lingual transfer learning for multilingual task oriented dialog.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3795–3805.
  * Shao et al. (2018) Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai. 2018.  Drcd: a chinese machine reading comprehension dataset.  _arXiv preprint arXiv:1806.00920_.
  * Wang et al. (2019) Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019.  Cross-lingual bert transformation for zero-shot dependency parsing.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5725–5731.
  * Wolf et al. (2020) Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380.
  * Zoph et al. (2020) Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. 2020.  Rethinking pre-training and self-training.  _Advances in Neural Information Processing Systems_ , 33.

##  Appendix A Implementation Details

For each experiment, we first fine-tuned m-BERT on the training set in the
source language for 3 epochs. Then, the training set in the target language
was used as the unlabeled set for self-training for 3 iterations. Then, in
each iteration, the model was also trained for 3 epochs. Last, the performance
was evaluated on the development set in the target language.

For both fine-tuning and self-training stage, we trained the model using AdamW
Loshchilov and Hutter (2018) with the linear scheduler without warm-up. The
batch size was 32, and the learning rate was $5\times 10^{-5}$.

The maximum input context and question length of the model was set to 384 and
64 respectively, and the document stride was 128. During inference, the
maximum length of machine-generated predictions was 30, and we used a beam
size of 20 when computing predictions.

##  Appendix B Classifying Questions and Answers

#### SQuAD

We classified the questions based on their word: What, When, Where, Which,
Why, Who, How, and other. As for the answers, following Rajpurkar et al.
(2016), we leveraged constituency parses, POS tags, and NER tags with Stanford
CoreNLP Manning et al. (2014).

#### DRCD

For DRCD, we mostly followed Shao et al. (2018) to categorize the questions
and answers. However, we used our own set of keywords and measure words
because the keywords and the measure words were not given in the original
paper.

#### FQuAD

We grouped the questions in the development set of FQuAD by their beginning
interrogative words, the same as d’Hoffschmidt et al. (2020) did. Nonetheless,
since the regular expressions and rules to classify the answers were not
specified in their paper, we followed a similar procedure in SQuAD to
categorize them.

##  Appendix C Improvement Analysis

Figure 2: The relation between F1 improvements and the amount of pseudo-labels
for different question types in the first iteration of self-training stage.
Figure 3: The relation between F1 improvements and the amount of pseudo-labels
for different answer types in the first iteration of self-training stage.
Figure 4: The change of F1 scores in the first iteration in self-training
stage for different types of answers.

Types | DRCD-SQuAD | FQuAD-SQuAD
---|---|---
Questions
Who | 5.45 | -0.3
What | 5.89 | 1.95
When | 16.98 | -0.05
Where | 4.44 | 3.04
Why | 8.27 | 1.77
Which | 4.07 | 0.46
How | 8.41 | 6.12
Other | 3.70 | 2.63
Answers
Date | 13.13 | -0.3
Other Numeric | 6.77 | 4.16
Person | 6.62 | 2.43
Location | 6.72 | 3.14
Other Entity | 2.85 | 0.55
Common Noun Phrase | 6.10 | 1.79
Verb Phrase | 8.43 | 4.48
Adjective Phrase | 3.87 | 3.21
Clause | 3.49 | 1.67
Other | 7.05 | 2.83

Table 6: The F1 improvements ($\Delta$F1) of each question and answer type in the first iteration when self-trained on SQuAD. Types | SQuAD-DRCD | FQuAD-DRCD
---|---|---
Questions
Who | 4.26 | 9.83
What | 3.16 | 8.21
When | 5.64 | 10.59
Where | 1.14 | 3.41
Why | 0.66 | 0.42
Which | 1.64 | 3.3
How | 8.55 | 6.19
Other | 2.19 | 4.02
Answers
Numeric | 2.09 | 5.02
Entity | 2.81 | 5.47
Description | 2.26 | 3.14
Table 7: The F1 improvements ($\Delta$F1) of each question and answer type in
the first iteration when self-trained on DRCD.

Types | SQuAD-FQuAD | DRCD-FQuAD
---|---|---
Questions
Who | 1.32 | 3.65
What (quoi) | -0.16 | 6.08
What (que) | -0.16 | 7.54
When | -3.03 | 4.25
Where | 12.90 | 2.28
Why | 1.42 | -13.17
How | 4.41 | 12.18
How many | 6.57 | 8.24
Other | 1.00 | 5.74
Answers
Date | 2.42 | 6.65
Other Numeric | -0.29 | 4.64
Person | 0.88 | 4.18
Location | 0.63 | 8.06
Other Proper Nouns | 2.01 | 5.24
Common Noun | 2.21 | 7.55
Verb | -0.05 | 5.89
Adjective | 4.04 | 12.5
Other | 1.47 | 1.87

Table 8: The F1 improvements ($\Delta$F1) of each question and answer type in
the first iteration when self-trained on FQuAD.

Figure 2 and Figure 3 respectively show the relations between F1 improvements
and the number of pseudo-labels of different question or answer types in the
first self-training iteration. The numbers and content of types vary due to
different categories in their original papers. We can see that the number of
pseudo-labels does not strongly related to the improvements on F1 score. On
the other hand, Figure 4 displays the improvements of F1 scores for different
types of answers. It shows a similar trend with those in question types, but
is not as obvious.

Table 6, 7 and 8 show the improvements of F1 scores for each question and
answer types for pairs of datasets in the first iteration in self-training
stage. The numbers and content of types vary because the categories were
different in their original papers.

Dataset | Threshold $\theta$
---|---
Source | Target | 0.5 | 0.6 | 0.7 | 0.8 | 0.9
SQuAD | FQuAD | 75.21 | 75.66 | 75.75 | 74.07 | 73.01
DRCD | 86.07 | 86.10 | 86.64 | 85.80 | 85.62
KorQuAD | 83.19 | 83.46 | 83.99 | 83.79 | 82.80

Table 9:  Comparison of the F1 scores on different datasets when different
thresholds were used in pseudo-labeling.

##  Appendix D Confidence Threshold

Table 9 lists the F1 scores of the models which were fine-tuned on SQuAD then
self-trained on other datasets with different thresholds in pseudo-labeling.
The best performances were obtained when $\theta=0.7$.


# M2MRF: Many-to-Many Reassembly of Features for Tiny Lesion Segmentation in
Fundus Images

Qing Liu  Haotian Liu  Wei Ke  Yixiong Liang This work is supported in part by
the High Performance Computing Center of Central South University. Qing Liu is
partially supported by the National Natural Science Foundation of China (NSFC
No. 62006249), Changsha Municipal Natural Science Foundation (kq2014135) and
Hunan Provincial Natural Science Foundation of China (2021JJ40788). Wei Ke is
partially supported by the National Natural Science Foundation of China (NSFC
No. 62006182). Yixiong Liang is partially supported by the Hunan Provincial
Science and Technology Innovation Leading Plan (No.2020GK2019). Qing Liu,
Haotian Liu and Yixiong Liang are with School of Computer Science, Central
South University, Changsha, Hunan, 410083, China (email:
<EMAIL_ADDRESS>(yxliang, haotianliu)@csu.edu.cn). Wei Ke is with
Xi’an Jiaotong University, Xi’an, Shaanxi, China (email: wei.ke@xjtu.edu.cn).
Qing Liu and Haotian Liu are equal contribution. Wei Ke and Yixiong Liang are
co-corresponding authors.

###### Abstract

Feature reassembly is an essential component in modern CNN-based segmentation
approaches, which includes feature downsampling and upsampling operators.
Existing operators reassemble multiple features from a small predefined region
into one for each target location independently. This may result in loss of
spatial information, which could vanish activations caused by tiny lesions
particularly when they cluster together. In this paper, we propose a many-to-
many reassembly of features (M2MRF). It reassembles features in a dimension-
reduced feature space and simultaneously aggregates multiple features inside a
large predefined region into multiple target features. In this way, long range
spatial dependencies are captured to maintain activations on tiny lesions.
Experimental results on two lesion segmentation benchmarks, i.e. DDR and
IDRiD, show that (1) our M2MRF outperforms existing feature reassembly
operators; (2) equipped with our M2MRF, the HRNetv2 is able to achieve
significant better performance to CNN-based segmentation methods and
competitive even better performance to two recent transformer-based
segmentation methods. Our code is made publicly available at
https://github.com/CVIU-CSU/M2MRF-Lesion-Segmentation.

feature reassembly, upsampling operator, downsampling operator, tiny lesion
segmentation, fundus image analysis

##  1 Introduction

This paper focuses on segmentation of tiny lesions, e.g. soft exudates (SEs),
hard exudates (EXs), microaneurysms (MAs) and hemorrhages (HEs) (see Fig. 1)
in colour fundus images, which is an important prerequisite for enabling
computers to assist human doctors for clinical purpose. It falls into the
research area of semantic segmentation.

With the rise of deep convolutional neural networks (CNN), many CNN-based
approaches such as [long2015fully, ronneberger2015u, xie2015holistically,
chen2017deeplab, zhao2017pyramid, chen2018encoder, wang2020deep] have been
proposed for semantic segmentation and achieved extraordinary progress. On
Cityscapes [cordts2016cityscapes] with 19 classes, the mean of class-wise
intersection over union (mIoU) by a state-of-the-art approach HRNetV2
[wang2020deep] has been improved to 81%. However, when fine-tuning HRNetV2
[wang2020deep] for four-class lesion segmentation in fundus images, the mIoU
on IDRiD [porwal2020idrid] decreases to 47%. Why is lesion segmentation so
much harder than natural scene image segmentation?

Figure 1: Example for colour fundus images and lesions. Four types of lesions
i.e. SEs, EXs, MAs and HEs are delineated in yellow, red, blue and green
respectively. Many lesions are extremely small and the size variance across
them are enormous

Two possible factors behind performance gap are the extreme small size of
lesions and large scale variation across them. Interestingly, almost 50%
lesions are less than 256 pixels in images of size $4288\times 2848$ from
IDRiD [porwal2020idrid] dataset provided by the 2018 ISBI grand challenge.
Such tiny size of lesions rises an extreme challenge for CNN-based
segmentation approaches to learn discriminative representations with enough
spatial information. To make matters worse, the smallest 10% lesions in IDRiD
[porwal2020idrid] only contain less than 74 pixels while the largest 10%
lesions contain more than 1920 pixels. Tiny lesions require CNNs maintain as
enough local information as possible within a small receptive field while
large lesions require CNNs exploit long range contexts over a large receptive
field. This demand contradiction raised by enormous size variation presents
another challenge for CNN-based approaches. Moreover, in fundus images,
multiple lesions with large size variation always coexist and cluster
together, which makes the representation learning more challenge.

Figure 2: Illustration of our motivation. Here we take the downsampling with
rate of 2 for example. Input contains two EXs, which are delineated with red
lines. Feature activations mainly caused by background and EXs are represented
by circles in light orange and red respectively. With the input activations,
we want to preserve activations of both EXs. To generate activation
$\mathbf{y}$, M2ORF first finds the source feature $\mathbf{x}$ in input
activations, and then determines a small local region $\Omega_{\mathbf{x}}$
centred on $\mathbf{x}$, and finally reassembles features inside
$\Omega_{\mathbf{x}}$ with importance weighting kernel
$\mathbf{W}_{\Omega_{\mathbf{x}}}$ into target activation $\mathbf{y}$, as
illustrated in (a). M2ORF ignores the coexist of multiple lesions, which may
vanish tiny lesion activations. Instead, our M2MRF directly learns to
reassemble activations inside a large local region $\mathbf{P}$ to generate
multiple output activations $\mathbf{Q}$ simultaneously which enables subtle
activations be enhanced mutually and well preserved, as illustrated in (b).
Here $\mathbf{W}_{\mathbf{P}}$ is the importance weighting kernel in M2MRF.

Taking an in-depth look at modern CNN-based segmentation approaches
[long2015fully, ronneberger2015u, xie2015holistically, chen2017deeplab,
zhao2017pyramid, chen2018encoder, sun2019deep, wang2020deep], the challenges
of applying them to lesion segmentation are mainly caused by the repetitive
utilisation of operators for reassembly of features (RF operators for short),
i.e. downsampling and upsampling operators. Downsampling operator is the key
contributor to enlarge CNNs’ receptive fields via reducing the spatial
resolution of feature maps [chen2017deeplab, gao2019lip]. Conversely,
upsampling operator is essential to recover the spatial resolution for better
segmentation. Strided max-pooling and strided convolution
[springenberg2014striving, sun2019deep, wang2020deep] are two widely-adopted
operators for downsampling while bilinear interpolation and deconvolution
[zeiler2010deconvolutional][zeiler2014visualizing, long2015fully,
noh2015learning] are two common choices for feature upsampling. Those RF
operators assume that features in target feature map are independent of each
other and follow a many-to-one pipeline. They first determine a small local
region in input feature map, according to a target location in output feature
map. Then, they reassemble multiple features in the small local region with
importance weighting kernels into one as the output feature for the target
location. However, during the downsampling process, resembling features inside
small local region independently for each target location ignores the coexist
of multiple tiny lesions and limits ability to capture long-range dependencies
for mutual activation enhancement between tiny lesions. These further lead to
dilution or even vanishing of subtle activations caused by tiny lesions
[gao2019lip], as illustrated in Fig. 2 (a). Once subtle lesion activations are
vanished, it is almost impossible to recover them via upsampling, which
consequently results in miss identification.

In this paper, we aim to address above issues raised by M2ORF operators within
the context of tiny lesion segmentation. Basically, we pursue a RF operator
that is capable of 1) preserving subtle activations caused by tiny lesions; 2)
easily being integrated into existing CNN-based segmentation models. To this
end, we propose a lightweight RF operator, termed as Many-to-Many Reassembly
of Features (M2MRF), which considers the coexist of tiny lesions and enhances
their subtle activations mutually via reassembling multiple features inside a
large predefined region into multiple target features simultaneously (see
Fig.2(b)). We demonstrate the effectiveness of our M2MRF on two public tiny
lesion segmentation datasets, i.e. DDR [li2019diagnostic] and IDRiD
[porwal2020idrid]. Experiments show that our M2MRF outperforms state-of-the-
art RF operators. Particularly, compared with a strong segmentation baseline
HRNetv2 [wang2020deep], our M2MRF exhibits significant improvements with
negligible increase of parameters and inference time.

The rest of this paper is organised as follows. The most related works are
briefly reviewed in Section 2, and our proposed M2MRF and its application on
tiny lesion segmentation are described in Section 3. Section 4 presents the
experiments and analysis. The conclusion is presented in Section 5.

##  2 Related Work

Feature reassembly operators in deep networks. In existing deep networks
[long2015fully] [ronneberger2015u] [chen2017deeplab] [chen2018encoder]
[wang2020deep] [sun2019deep] [noh2015learning] [badrinarayanan2017segnet],
strided max-pooling [lecun1990handwritten] and strided convolution are two
widely-adopted operators for feature downsampling. Bilinear interpolation,
deconvolution and unpooling[zeiler2014visualizing] [zeiler2011adaptive] are
widely-adopted for feature upsampling. The general idea of these operators is
to generate a feature vector for each target location via reassembling
multiple features inside a predefined region with importance kernels.
Particularly, importance kernels for strided max-pooling, unpooling and
bilinear interpolation are hand-crafted and feature maps are processed
channel-by-channel efficiently. However, they ignore the context dependencies
across channels and the diversity of local patterns. Instead, the importance
kernels for strided convolution and deconvolution are learned. Their dimension
depends on the input features, which is always high in CNNs. This makes the
computation burden of reassembly heavy when large importance kernels are used.
Thus it is difficult to reassemble features from a large region.

Recently, novel ideas about learning-based RF operators emerge. LIP
[gao2019lip] learns adaptive importance kernels based on inputs to enhance the
discriminative features. In CARAFE [wang2019carafe] and CARAFE++
[wang2020carafeplus], content-aware kernels for each target position is
learned according to input features for feature reassembly. Similarly,
IndexNet [lu2019indices] [lu2020index] learns importance kernels from feature
encoder to guide the feature reassembly in both feature encoder and decoder.
Instead of reassembling features inside a predefined region, deformable RoI
pooling [dai2017deformable] reassembles features in an adaptive region which
is learned from the input features. To model the affinity information, A2U
[dai2021learning] is proposed to learn importance kernels according to second-
order features for feature reassembly. However, these learning-based operators
still follow the paradigm of many-to-one feature reassembly and are prone to
dilute or even vanish features of tiny lesions.

Tiny lesion segmentation in fundus images. The development originated from
one-type lesion segmentation with either hand-crafted features
[zhang2014exudate, liu2017location, du2020automatic] or deep features
[liu2021dsm] and has been evolved to multi-type lesion segmentation. For
example, in [li2019diagnostic], PSPNet [zhao2017pyramid] and DeeplabV3+
[chen2018encoder] are directly fine-tuned for multi-type lesion segmentation.
Guo et al. [guo2019seg] develop a network named L-Seg on the top of FCN
[long2015fully]. More recently, multi-task learning frameworks
[foo2020multi][wang2021deep] are proposed to exploit the correlation among
tasks of lesion segmentation and others to boost performance of all tasks. All
of those methods adopt many-to-one RF operators during feature encoding and
decoding and are prone to ignore tiny lesions. In [sun2021lesion], specially
designed lesion filters are proposed to discover lesions and suspicious lesion
regions are detected. However, this work is unable to provide fine lesion
regions and predict lesion classes. One thing worth to mention is that the
public datasets with pixel-level annotations also updates from one-type lesion
segmentation e.g. E-ophtha EX [decenciere2013teleophta] and e-ophtha MA
[decenciere2013teleophta] to multi-type lesion segmentation e.g. IDRiD
[porwal2020idrid] and DDR [li2019diagnostic].

Figure 3: An overview of our Many-to-Many Reassembly of Features (M2MRF).
$\\{\mathbf{P}_{l}\\}_{l=1}^{L}$ and $\\{\mathbf{Q}_{l}\\}_{l=1}^{L}$ are
feature patches of size $S_{h}\times S_{w}\times\frac{C}{r}$ and $\delta
S_{h}\times\delta S_{w}\times\frac{C}{r}$ respectively, where $L(=\lceil
H/S_{h}\rceil\cdot\lceil W/S_{w}\rceil)$ is the number of patches.
$\\{\mathbf{p}_{l}\\}_{l=1}^{L}$ and $\\{\mathbf{q}_{l}\\}_{l=1}^{L}$ are
feature vectors. In this figure, a feature map of size $H\times W\times C$ is
downsampled by a factor of $\delta(=1/2)$. (Best view in colour)

Deep semantic segmentation. The development can be mainly classified into
three veins. The first vein focuses on how to produce and aggregate multi-
scale representations. FCN [long2015fully] provides a natural solution which
reuses middle-level features to compensate for spatial details in high-level
features. UNet [ronneberger2015u] propagates low-level information to high-
levels via skip connections, which also inspires many variants specially for
medical image segmentation such as VNet [Diceloss], UNet++ [zhou2019unetpp],
MNet [fu2018joint] and SAN [liu2019spatial]. In [li2020gated], gated fully
fusion is proposed to selectively fuse multi-level features. The second vein
focuses on high-resolution representation learning. For example, in
[ronneberger2015u, noh2015learning, badrinarayanan2017segnet], an encoder-
decoder architectural style is adopted to gradually recover feature resolution
while HRNet [sun2019deep, wang2020deep] is specially designed to learn high-
resolution representations. The third vein introduces attention mechanism and
variant modules such as DANet [fu2019dual], DRANet [fu2020scene], CCNet
[huang2019ccnet], RecoNet [chen2020tensor], HANet [choi2020cars] and DNL
[yin2020disentangled] are developed to explore long range dependencies. More
recently, vision transformers such as Swin [liu2021swin], Twins [chu2021Twins]
and CSWin [dong2021cswin] are developed for segmentation. However, those
methods are designed for segmenting objects in proper size rather than tiny
size.

##  3 Method

In this section, we first give a simple analysis for M2ORF operators and then
detail our M2MRF. Finally we take HRNetV2 [wang2020deep] as an example and
present how to integrate our M2MRF into CNN-architectures for tiny lesion
segmentation.

###  3.1 Analysis for M2ORF Operators

Given the input feature map $\mathbf{X}\in\mathcal{R}^{H\times W\times C}$ and
sample rate $\delta$ where $\delta>0$, the goal of feature reassembly is to
generate target feature map $\mathbf{Y}\in\mathcal{R}^{\lfloor\delta
H\rfloor\times\lfloor\delta W\rfloor\times C}$ via finding a function mapping
$\Phi$ parametrised by importance kernels $\mathbf{W}$:

$\mathbf{Y}=\Phi(\mathbf{X};\mathbf{W})\;.$ (1)

Here $\delta<1$ for downsampling and $\delta>1$ for upsampling.

To make the feature reassembly computational efficient, most existing methods
degrade it to a many-to-one local sampling problem, i.e. reassembling multiple
features in a predefined local region to one target feature. Specifically, for
any target feature $\mathbf{y}\in\mathcal{R}^{C}$ at location
$(i^{\prime},j^{\prime})$ in $\mathbf{Y}$, most existing methods assume that
there is a corresponding source feature $\mathbf{x}\in\mathcal{R}^{C}$ at
location $(i,j)$ in $\mathbf{X}$, where $i=\lfloor i^{\prime}/\delta\rfloor$
and $j=\lfloor j^{\prime}/\delta\rfloor$. They follow three steps to obtain
$\mathbf{y}$ : (1) setting/learning a local region $\Omega_{\mathbf{x}}$
according to $\mathbf{x}\in\mathcal{R}^{C}$; (2) with $\Omega_{\mathbf{x}}$,
setting/learning corresponding importance kernels
$\mathbf{W}_{i^{\prime},j^{\prime}}$; (3) obtaining $\mathbf{y}$ via

$\mathbf{y}=\Phi(\mathbf{P}_{\Omega_{\mathbf{x}}};\mathbf{W}_{i^{\prime},j^{\prime}})\;,$
(2)

where $\mathbf{P}_{\Omega_{\mathbf{x}}}$ denotes features in
$\Omega_{\mathbf{x}}$.

Obviously, M2ORF operators assume that each target feature in $\mathbf{Y}$ is
independent and ignores the coexist of multiple lesions. Additionally,
considering the computational efficiency, previous RF operators such as
strided convolution, deconvolution, CARAFE++ [wang2020carafeplus] and A2U
[dai2021learning] usually consider a small size $\Omega_{\mathbf{x}}$, which
makes them fail to exploit long-range dependencies. As a result, activations
caused by tiny lesions are prone to be diluted or even vanished. To alleviate
the dilution of tiny lesion activations, one natural solution is to discard
the assumption of M2ORF and generate multiple features for multiple target
locations with features inside a large local region simultaneously. To this
end, we propose many-to-many reassembly of features (M2MRF).

###  3.2 Many-to-Many Reassembly of Features

Module overview. Fig. 3 illustrates an overview of reassembling the input
feature map $\mathbf{X}$ of size $H\times W\times C$ with our M2MRF operator
with sample rate $\delta$ to feature map $\mathbf{Y}$ of size $\lfloor\delta
H\rfloor\times\lfloor\delta W\rfloor\times C$. First, a channel compressor is
performed on $\mathbf{X}$ to reduce its channel from $C$ to $\frac{C}{r}$ for
computational efficiency. We denote the output as $\mathbf{X}_{CC}$. Then we
partition $\mathbf{X}_{CC}$ into feature patches
$\\{\mathbf{P}_{l}\\}_{l=1}^{L}$ of size $S_{h}\times S_{w}\times\frac{C}{r}$,
where $L=\lfloor H/S_{h}\rfloor\cdot\lfloor W/S_{w}\rfloor$. Our proposed
M2MRF is performed on each feature patch and outputs
$\\{\mathbf{Q}_{l}\\}_{l=1}^{L}$ of size $\lfloor\delta
H\rfloor\times\lfloor\delta W\rfloor\times\frac{C}{r}$. Thereafter, those
patches are merged into feature map $\mathbf{Y}_{CC}$ of size $\lfloor\delta
H\rfloor\times\lfloor\delta W\rfloor\times\frac{C}{r}$. Finally, we recover
the feature channel to $C$ via channel recover. For channel compressor and
recover, we simply implement them with a $1\times 1$ regular convolution
layer.

M2MRF. With a local feature patch
$\mathbf{P}\in\\{\mathbf{P}_{l}\\}_{l=1}^{L}$, the goal is to generate
$\mathbf{Q}$ of size $\lfloor\delta S_{h}\rfloor\times\lfloor\delta
S_{w}\rfloor\times\frac{C}{r}$:

$\mathbf{Q}=\Phi(\mathbf{P};\mathbf{W}_{patch})\;.$ (3)

Here we let $M=\lfloor\delta S_{h}\rfloor\times\lfloor\delta
S_{w}\rfloor,N=S_{h}\times S_{w}$, and treat this task as generating $M$
features $\mathbf{Q}=\\{\mathbf{y}_{m}\\}_{m=1}^{M}$ from $N$ source features
$\mathbf{P}=\\{\mathbf{x}_{n}\\}_{n=1}^{N}$ where $\mathbf{y}_{m}$,
$\mathbf{x}_{n}\in\mathcal{R}^{1\times\frac{C}{r}}$. To achieve this, one
option is to adopt linear projection, thus Eq.(3) can be expressed as:

$[\mathbf{y}_{1},\cdots,\mathbf{y}_{M}]=[\mathbf{x}_{1},\cdots,\mathbf{x}_{N}][\mathbf{W}_{1},\cdots,\mathbf{W}_{M}]\;,$
(4)

where $\mathbf{W}_{1},\cdots,\mathbf{W}_{M}$ are parameters of size
$\frac{NC}{r}\times\frac{C}{r}$ to be learned. Therefore we have
$\mathbf{W}_{patch}=[\mathbf{W}_{1},\cdots,\mathbf{W}_{M}]$ whose size is
$\frac{NC}{r}\times\frac{MC}{r}$. For simplicity, we denote
$\mathbf{p}=[\mathbf{x}_{1},\cdots,\mathbf{x}_{N}],\mathbf{q}=[\mathbf{y}_{1},\cdots,\mathbf{y}_{M}]$,
and rewrite Eq.(4) as:

$\mathbf{q}=\mathbf{p}\mathbf{W}_{patch}\;.$ (5)

To make use of long-range dependencies, $S_{h}S_{w}$ is required to be large.
Accordingly, $N$ and $M$ are large and $\mathbf{W}_{patch}$ would be a large
matrix. On one hand, it is always difficult to optimise such a large matrix.
On the other hand, storing a large matrix results in high memory consumption.
To reduce the number of parameters to be learned, we decompose
$\mathbf{W}_{patch}$ to two small matrices via a two-layer linear projections.
Thus Eq. (4) can be rewrote as:

$\mathbf{q}=(\mathbf{p}\mathbf{W}_{patch}^{\prime})\mathbf{W}_{patch}^{\prime\prime}\;,$
(6)

where
$\mathbf{W}_{p}^{\prime}\in\mathcal{R}^{\frac{NC}{r}\times\frac{NC}{\alpha
r}}$ and $\mathbf{W}_{p}^{\prime\prime}\in\mathcal{R}^{\frac{NC}{\alpha
r}\times\frac{MC}{r}}$ are parameters in the two linear projections, and
$\alpha\geq 1$ such that the matrix dimension of $\mathbf{W}_{patch}^{\prime}$
and $\mathbf{W}_{patch}^{\prime\prime}$ is less than $\mathbf{W}_{patch}$ far
away.

Figure 4: Two variants of our M2MRF. Here we take $\delta=4$ for upsampling
and $\delta=1/4$ for downsampling as examples.

###  3.3 M2MRF for Tiny Lesion Segmentation

As HRNetV2 [wang2020deep] is originally designed for high-resolution
representation learning and has achieved state-of-the-art performance on
semantic segmentation task, we adopt it as the baseline for tiny lesion
segmentation. In HRNetV2 [wang2020deep], RF operators involve in the fusion
module to exchange information across multi-resolution feature maps. Strided
convolution with stride of 2 is adopted for downsampling and bilinear
interpolation for upsampling. Repeating $t$ times of the strided convolution,
the feature resolution decreases to $1/2^{t}$. To build M2MRF variants, we
replace the repeated strided convolution and bilinear interpolation with our
M2MRF.

We propose two options to replace the repeated $t$ layers of strided
convolution with our M2MRF. One is to replace each strided convolution layer
with our proposed M2MRF with $\delta=1/2$ to gradually decrease the feature
resolution. We term it as cascade M2MRF. The other is to directly set $\delta$
to $1/2^{t}$ in M2MRF to replace the entirety of $t$ layers of strided
convolution. We term it as one-step M2MRF. Similarly, there are also two
options to replace bilinear interpolation layer with scale factor $2^{t}$. One
is to cascade M2MRF with $\delta=2$ to gradually increase the feature
resolution to $2^{t}$. The other is one-step M2MRF with $\delta=2^{t}$ which
directly increases the feature resolution to $2^{t}$. Taking $t=2$ for
example, Fig.4 illustrates the cascade M2MRF and one-step M2MRF for
downsampling and upsampling respectively.

With high-resolution feature maps by variants of HRNetV2 equipped with our
M2MRFs, we attach multiple binary classifiers on them to obtain probability
maps for each lesion class. Considering the extremely class imbalance in tiny
lesion segmentation, we adopt Dice loss [Diceloss] to train M2MRF-HRNetV2 as
well as the baseline.

##  4 Experiments and Discussions

To validate the effectiveness of our M2MRF module on tiny lesion segmentation,
we carry out experiments on two public datasets i.e. DDR [li2019diagnostic]
and IDRiD [porwal2020idrid]. For evaluation metrics, we follow
[porwal2020idrid] and utilize Area Under Precision-Recall curve (AUPR). We
also report mean F-score (mF) and mean IoU (mIoU) as they are widely adopted
for segmentation evaluation.

###  4.1 Implementation Details

####  4.1.1 Datasets and augmentation

DDR [li2019diagnostic] contains 757 colour fundus images of size ranging from
$1088\times 1920$ to $3456\times 5184$, among which 383 for training, 149 for
validation and $225$ for testing. In DDR [li2019diagnostic], 24154, 13035,
1354 and 10563 connected regions are annotated by experts as EXs, HEs, SEs and
MAs respectively. To our best knowledge, DDR [li2019diagnostic] is the largest
dataset for lesion segmentation in fundus images. IDRiD [porwal2020idrid]
contains 81 colour fundus images of size $4288\times 2848$ among which 54 for
training and 27 for testing. It is provided by a grand challenge on “Diabetic
Retinopathy – Segmentation and Grading” in 2018. In IDRiD [porwal2020idrid],
11716, 1903, 150 and 3505 connected regions are annotated by experts as EXs,
HEs, SEs and MAs respectively.

Before feeding images into models, we follow [wang2021deep] and downsample
images in DDR [li2019diagnostic] such that the long side is 1024. Thereafter,
zero padding is used on short side to enlarge its length to 1024. Following
[liu2021dsm, guo2019seg], images in IDRiD [porwal2020idrid] are resized to
$1440\times 960$. Three tricks are adopted to augment the training data:
multi-scale (0.5-2.0), rotation (90∘, 180∘ and 270∘) and flipping(horizontal
and vertical).

####  4.1.2 Experimental setting

M2MRF-HRNetV2 is built on the top of implementation of HRNetV2 [wang2020deep]
provided by MMSegmentation [mmseg2020]. We initialize parameters associated
with both M2MRF and dense classification layers with Gaussian distribution
with zeros mean and standard deviation of 0.01 and the rest with the pre-
trained model on ImageNet [krizhevsky2012imagenet]. SGD is used for parameters
optimisation. Hyper-parameters include: initial learning rate (0.01 poly
policy with power of 0.9), weight decay (0.0005), momentum (0.9), batch size
(4) and iteration epoch (60k on DDR and 40k on IDRiD).

###  4.2 Results on DDR and Analysis

####  4.2.1 Ablation study for M2MRF

There are four hyper-parameters in our M2MRF, i.e. the patch size $S_{h}$ and
$S_{w}$, $r$ in channel compressor, $\alpha$ in Eq.(6). For $S_{h}$ and
$S_{w}$, we directly let them equal and conduct experiments with setting
$\\{4,8,16\\}$. For $r$ and $\alpha$, we conduct experiments with setting $r$
to $\\{2,4,8\\}$ and $\alpha$ to $\\{32,64,128\\}$ and replacing the strided
convolution layer in HRNetV2 [wang2020deep] with our cascade M2MRF and
bilinear interpolation layer in HRNetV2 [wang2020deep] with one-step M2MRF.
Performance on DDR [li2019diagnostic] validation set is listed in Table 1,
from which we can see that $S_{h}=S_{w}=8$, $r=4$ and $\alpha=64$ yield best
performance. In what followed, except for extra illustration, $S_{h}=S_{w}=8$,
$r=4$ and $\alpha=64$ are default setting.

$S_{h},S_{w}$ | $r$ | $\alpha$ | mAUPR | mF | mIoU
---|---|---|---|---|---
4 | 4 | 64 | 60.93 | 59.77 | 43.33
8 | 4 | 64 | 61.49 | 60.26 | 43.80
16 | 4 | 64 | 60.69 | 59.42 | 42.93
8 | 2 | 64 | 61.13 | 59.89 | 43.45
8 | 4 | 64 | 61.49 | 60.26 | 43.80
8 | 8 | 64 | 61.43 | 60.18 | 43.78
8 | 4 | 32 | 60.34 | 59.24 | 42.71
8 | 4 | 64 | 61.49 | 60.26 | 43.80
8 | 4 | 128 | 61.31 | 60.06 | 43.63
Table 1: M2MRF(cascade/one-step) with different settings on DDR validation set [li2019diagnostic]. Results are averaged over three repetitions. operators | mAUPR | mF | mIoU | Param(M) | FPS
---|---|---|---|---|---
DS | StrideConv | 45.21 | 43.95 | 28.84 | 65.85 | 11.31
MaxPool | 45.97 | 44.28 | 29.17 | 59.45 | 11.88
LIP [gao2019lip] | 43.14 | 40.68 | 26.29 | 75.44 | 6.80
CARAFE++ [wang2020carafeplus] | 46.27 | 43.51 | 28.51 | 66.30 | 9.24
one-step M2MRF | 49.41 | 45.68 | 30.36 | 60.75 | 9.83
cascade M2MRF | 48.92 | 44.99 | 29.71 | 61.12 | 9.28
US | Bilinear | 45.21 | 43.95 | 28.84 | 65.85 | 11.31
Deconv | 46.24 | 44.59 | 29.37 | 73.12 | 10.86
CARAFE++[wang2020carafeplus] | 46.68 | 45.32 | 29.97 | 72.12 | 10.26
one-step M2MRF | 45.44 | 44.03 | 28.88 | 75.03 | 9.69
cascade M2MRF | 45.37 | 44.46 | 29.27 | 72.25 | 9.16
Table 2: Comparing with different downsampling (DS) and upsampling (US)
operators on DDR [li2019diagnostic] testing set. StrideConv as downsampling
operator and bilinear interpolation as upsampling operator are the default set
in vanilla HRNetV2 [wang2020deep]. The Best and Second Best results are
highlighted in boldface and underlined respectively. Results are averaged over
three repetitions. Figure 5: Visualization results on DDR test set
[li2019diagnostic] by variants of HRNetV2 with different downsampling
operators, i.e., our one-step M2MRF, cascade M2MRF, StrideConv [wang2020deep],
MaxPool, LIP [gao2019lip] and CARAFE++ [wang2020carafeplus]. Regions
delineated in green and blue are HEs and MAs by experts or segmentation
approaches. As red lesions, MAs and tiny HEs have deceptive appearances and
both exhibit as dark red dots. In GT map (b), challenge lesions are marked
with solid boxes. In visualised segmentation result maps (c-h), wrong and miss
identifications are marked with dotted boxes. Figure 6: Visualization results
on DDR test set [li2019diagnostic] by variants of HRNetV2 with different
upsampling operators, i.e., our one-step M2MRF, cascade M2MRF, bilinear
interpolation [wang2020deep], deconvolution and CARAFE++ [wang2020carafeplus].
Regions delineated in blue, red and yellow are MAs, EXs and SEs by experts or
segmentation approaches. In GT map (b), challenge lesions are marked with
solid boxes. In visualised segmentation result maps (c-g), wrong and miss
identifications are marked with dotted boxes.

####  4.2.2 Comparison to other RF operators

Here we verify the effectiveness of our proposed M2MRF on DDR
[li2019diagnostic] testing set. In what follows, we first compare our M2MRF
with downsampling and upsampling operators, and then RF operators that
can/must be used as pairs.

Comparison to downsampling and upsampling operators. For downsampling, we
replace strided convolution operator (StrideConv) in HRNetV2 [wang2020deep]
with three optional operators, i.e., strided max-pooling (MaxPool), LIP
[gao2019lip] and CARAFE++ [wang2020carafeplus], and our two variants of M2MRF,
i.e., cascade M2MRF and one-step M2MRF, and make comparisons on DDR
[li2019diagnostic] test set. Among those operators, MaxPool is rule-based and
parameter-free while rest are learning-based.

StrideConv is the default setting in vanilla HRNetV2 [wang2020deep], whose
kernel size is set to $3\times 3$ and stride is set to 2. For MaxPool,
$3\times 3$ max-pooling kernel with stride of 2 is adopted. For LIP
[gao2019lip] and CARAFE++ [wang2020carafeplus], the suggested settings by
original papers are adopted. We replace StrideConv in HRNetV2 [wang2020deep]
with those compared downsampling operators and apply to tiny lesion
segmentation. Table 2 reports their performances. We can see that: (1) Rule-
based downsampling operator, i.e., MaxPool, achieves better performance than
StrideConv slightly in terms of mAUPR, mF and mIoU consistently; (2) LIP
[gao2019lip] seriously degrades the segmentation performance. Possible reason
is that LIP [gao2019lip] adopts instance normalisation to normalise the
importance kernels globally over entire feature map of each channel, which may
heavily dilute subtle activations caused by tiny lesions and result in miss
identification on tiny lesions; (3) CARAFE++ [wang2020carafeplus] improves
mAUPR by 1.06% but slightly decreases mF by 0.44% and mIoU by 0.33%; (4) Our
two variants of M2MRF improve mAUPR, mF and mIoU consistently. In detail,
cascade M2MRF improves mAUPR, mF and mIoU by 3.71%, 1.04% and 0.87%
respectively while one-step improves mAUPR, mF and mIoU by 4.20%, 1.73% and
1.52% respectively.

We also list the parameters and inference speed of variants of HRNetV2
equipped with the compared downsampling operators in Table 2. Comparing with
baseline, our proposed M2MRFs have less parameters but scarifies slight
inference speed. Comparing with LIP [gao2019lip] and CARAFE++
[wang2020carafeplus], our proposed M2MRFs have less parameters and competitive
inference speed. Obviously, HRNetV2 [wang2020deep] equipped with MaxPool has
least parameters and achieves fastest inference speed as MaxPool is rule-based
and parameter-free.

RF operators | mAUPR | mF | mIoU | Param(M) | FPS
---|---|---|---|---|---
StrideConv/Bilinear | 45.21 | 43.95 | 28.84 | 65.85 | 11.31
CARAFE++[wang2020carafeplus] | 47.64 | 44.80 | 29.54 | 72.57 | 8.69
MaxPool/Unpooling | 48.81 | 46.17 | 30.73 | 59.45 | 11.62
IndexNet[lu2020index] | 48.06 | 45.64 | 30.28 | 70.33 | 10.75
A2U[dai2021learning] | 45.89 | 44.44 | 29.27 | 66.51 | 3.97
M2MRF-A | 49.94 | 46.60 | 31.16 | 69.93 | 9.21
M2MRF-B | 49.42 | 45.77 | 30.41 | 67.15 | 8.54
M2MRF-C | 48.94 | 45.40 | 30.09 | 70.30 | 8.43
M2MRF-D | 49.25 | 45.57 | 30.27 | 67.52 | 8.01
Table 3: Comparing with different paired RF operators on DDR [li2019diagnostic] testing set. M2MRF-A: (one-step/one-step), M2MRF-B:(one-step/cascade), M2MRF-C:(cascade/one-step), M2MRF-D: (cascade/cascade). The first row is the baseline i.e. vanilla HRNetV2 [wang2020deep]. The Best and Second Best results are highlighted in boldface and underlined respectively. All results are averaged over three repetitions. Figure 7: Visualization results on DDR test set [li2019diagnostic] by variants of HRNetV2 with paired RF operators: M2MRF-A, M2MRF-B, M2MRF-C, M2MRF-D, StrideConv/bilinear, MaxPool/UnPool, IndexNet [lu2020index], CARAFE++ [wang2020carafeplus] and A2U [dai2021learning]. Regions delineated in green, blue, red and yellow are HEs, MAs, EXs and SEs by experts or segmentation approaches. In GT map (b), challenge lesions are marked with solid boxes. In visualised segmentation result maps (c-k), wrong and miss identifications are marked with dotted boxes. Methods | AUPR | F | IoU
---|---|---|---
EX | HE | SE | MA | mAUPR | EX | HE | SE | MA | mF | EX | HE | SE | MA | mIoU
HED[xie2015holistically] | 61.40 | 43.19 | 46.68 | 20.61 | 42.97 | 56.63 | 42.61 | 45.50 | 22.43 | 41.79 | 39.50 | 27.09 | 29.46 | 12.63 | 27.17
DNL[yin2020disentangled] | 56.05 | 47.81 | 42.01 | 14.71 | 40.14 | 53.36 | 42.71 | 40.40 | 15.60 | 38.02 | 36.39 | 27.15 | 25.33 | 8.46 | 24.33
Deeplabv3+[chen2018encoder] | 62.32 | 40.79 | 41.83 | 24.39 | 42.34 | 58.59 | 37.97 | 41.83 | 25.40 | 40.95 | 41.44 | 23.44 | 26.46 | 14.55 | 26.47
PSPNet[zhao2017pyramid] | 57.04 | 42.71 | 42.32 | 14.85 | 39.23 | 54.35 | 39.37 | 42.08 | 16.09 | 37.97 | 37.31 | 24.51 | 26.64 | 8.75 | 24.31
SPNet[hou2020strip] | 44.10 | 38.22 | 32.93 | 12.37 | 31.91 | 38.78 | 24.13 | 34.00 | 13.74 | 27.66 | 24.19 | 13.76 | 20.55 | 7.38 | 16.47
L-seg [guo2019seg] | 55.46 | 35.86 | 26.48 | 10.52 | 32.08 | - | - | - | - | - | - | - | - | - | -
HRNetV2[wang2020deep] | 61.55 | 45.68 | 46.91 | 26.70 | 45.21 | 58.98 | 44.96 | 44.86 | 26.99 | 43.95 | 41.82 | 29.01 | 28.94 | 15.60 | 28.84
Swin-base[liu2021swin] | 62.95 | 53.46 | 50.56 | 23.46 | 47.61 | 60.12 | 51.10 | 50.85 | 23.38 | 46.36 | 42.98 | 34.42 | 34.15 | 13.24 | 31.20
Twins-SVT-B [chu2021Twins] | 59.71 | 49.96 | 52.72 | 22.03 | 46.11 | 56.83 | 45.04 | 53.19 | 21.54 | 44.15 | 39.70 | 29.08 | 36.24 | 12.07 | 29.28
M2MRF-A | 64.17 | 54.20 | 53.19 | 28.21 | 49.94 | 60.47 | 46.18 | 52.10 | 27.67 | 46.60 | 43.35 | 30.03 | 35.22 | 16.06 | 31.16
M2MRF-B | 63.88 | 55.47 | 50.01 | 28.33 | 49.42 | 60.20 | 46.81 | 48.58 | 27.51 | 45.77 | 43.06 | 30.56 | 32.08 | 15.95 | 30.41
M2MRF-C | 63.59 | 54.43 | 49.35 | 28.38 | 48.94 | 60.62 | 45.16 | 47.78 | 28.04 | 45.40 | 43.49 | 29.17 | 31.39 | 16.31 | 30.09
M2MRF-D | 64.17 | 54.72 | 49.64 | 28.46 | 49.25 | 61.15 | 45.29 | 48.02 | 27.81 | 45.57 | 44.04 | 29.28 | 31.60 | 16.15 | 30.27
Table 4: Segmentation results on DDR test set [li2019diagnostic]. M2MRF-A:
(one-step/one-step), M2MRF-B:(one-step/cascade), M2MRF-C:(cascade/one-step),
M2MRF-D: (cascade/cascade). The Best and Second Best results are highlighted
in boldface and underlined respectively. All results are averaged over three
repetitions.

To further illustrate the superiority of our M2MRFs as downsampling operators,
we visualise segmentation results on MAs and HEs in Fig. 5. The reason of
choosing these two types of lesions is that MAs and tiny HEs belong to red
lesions and exhibit as dark red dots with similar appearances. Correctly
identifying them requires that segmentation models are able to maintain subtle
activations about them and exploit long-range dependencies to boost the
discriminativeness of activations during downsampling. For clarity, we mark
challenge MAs with solid magenta boxes and HEs with solid yellow boxes in GT
map (see Fig. 5(b)) and miss and wrong identifications by variants of HRNetv2
[wang2020deep] utilising different downsampling operators in Fig. 5 (c-h) with
dotted boxes. Obviously, our proposed cascade M2MRF and one-step M2MRF (see
Fig. 5(c-d)) encounter less miss and wrong identifications than compared four
downsampling operators as shown in Fig. 5(e-h). Particularly, both of our
cascade M2MRF and one-step M2MRF are able to correctly distinguish between HEs
and MAs while variants of HRNetv2 [wang2020deep] with rest downsampling
operators are prone to be deceived by their confusion appearances and wrongly
identify HEs as MAs. Although CARAFE++ [wang2020carafeplus] is able to
maintain subtle activations about tiny lesions, it seems scarifies the feature
discrimination and tends to cause wrong identifications.

For upsampling, we utilise strided convolution for downsampling and replace
bilinear interpolation in HRNetV2 [wang2020deep] with four optional upsampling
operators, i.e., deconvolution and CARAFE++ [wang2020carafeplus], our proposed
cascade M2MRF and one-step M2MRF. Among them, bilinear interpolation is rule-
based and parameter-free while the rest are learning-based. We compare their
performance on DDR [li2019diagnostic] test set. As reported in Table 2,
comparing with bilinear interpolation, our two variants of M2MRF achieve
slightly better mAUPR, mF and mIoU but require extra parameters and inference
time. From Table 2 we also can see that, different to CARAFE++
[wang2020carafeplus] which works better as upsampling operator than downsample
operator, our M2MRFs work better as downsampling operators. Fig. 6 shows
visualised segmentation results. As exampled, it is challenge to well segment
tiny lesions clustering together. Comparatively speaking, our one-step M2MRF,
cascade M2MRF and CARAFE++ [wang2020carafeplus] make less mistakes than
bilinear interpolation and deconvolution.

Comparison to paired RF operators. We have four different RF pairs when
combining cascade M2MRF and one-step M2MRF for both downsampling and
upsampling. We denote them by M2MRF-A (one-step M2MRF for both downsampling
and upsampling), M2MRF-B (one-step M2MRF for downsampling and cascade M2MRF
for upsampling), M2MRF-C (cascade M2MRF for downsampling and one-step M2MRF
for upsampling), M2MRF-D (cascade M2MRF for both downsampling and upsampling).
We compare them with five pairs of RF operators, i.e., StrideConv/Bilinear,
CARAFE++ [wang2020carafeplus], MaxPool/Unpooling, IndexNet [lu2020index] and
A2U [dai2021learning] on DDR [li2019diagnostic] test set. Among those RF
operators, MaxPool/Unpooling and IndexNet [lu2020index] have to utilise the
indices generated during downsampling for upsampling. Table 3 reports
quantitative results. As listed, among four variants of our M2MRF, M2MRF-A
achieves best in mAUPR, mF and mIoU. Comparing with vanilla HRNetV2
[wang2020deep], our M2MRF-A improves the mAUPR by a large margin from 45.21%
to 49.94% and the mIoU from 28.84% to 31.16% with few extra parameters
(65.85M$\rightarrow$69.93M) and slightly slower inference speed
(11.31fps$\rightarrow$9.21fps). Comparing with existing RF operators, our
M2MRF-A achieves best in both mAUPR and mIoU and the rest three variants
achieve better mAUPR and competitive mIoU. Specifically, comparing with the
recent learning-based RF operators IndexNet [lu2020index] and CARAFE++
[wang2020carafeplus], our M2MRF-A and M2MRF-B achieve better performance in
mAUPR and mIoU with less parameters and competitive inference speed. Comparing
with A2U [dai2021learning], our four variants of M2MRF outperform it on both
mAPUR and mIoU with faster inference speed. Fig. 7 shows visualised
segmentation results, from which we can see that our four variants of M2MRF
are able to preserve very subtle lesion activations and make less mistakes
than those compared paired RFs.

####  4.2.3 Comparison to state-of-the-art segmentation methods

We compare our M2MRF-HRNetV2 with nine state-of-the-art segmentation
approaches as listed in Table 4. The first seven methods are CNN-based and the
rest two are transformer-based. Results by Swin-base [liu2021swin] are
obtained by fine-tuning the pre-trained model on ImageNet-22K with DDR
training set [li2019diagnostic] and the rest are obtained by fine-tuning the
pre-trained models on ImageNet-1K [krizhevsky2012imagenet]. As we can see
that: (1) among previous CNN-based segmentation methods, the vanilla HRNetV2
[wang2020deep] achieves best performance with 45.21% on mAUPR, 43.95% on mF
and 28.84% on mIoU; (2) comparing with vanilla HRNetV2 [wang2020deep], the
four variants of our M2MRF get better performance on the four lesion classes
consistently in terms of mAUPR, mF and mIoU; (3) comparing with one of the
most recent transformer-based methods, Swin-base [liu2021swin], our four
variants of M2MRF achieve better on mAUPR by a large margin while our M2MRF-A
achieves competitive performance on mF and mIoU; (4) comparing with another
transform-based method Twins-SVT-B [chu2021Twins], the four variants of our
M2MRF surpass it significantly and consistently on mAUPR, mF and mIoU.

RF operators | mAUPR | mF | mIoU
---|---|---|---
StrideConv/Bilinear | 65.01 | 63.14 | 47.52
MaxPool/Bilinear | 66.24 | 65.01 | 49.28
LIP [gao2019lip]/Bilinear | 63.04 | 61.50 | 45.78
StrideConv/Deconv | 64.64 | 62.86 | 47.23
MaxPool/Unpooling | 66.17 | 64.98 | 49.22
CARAFE++ [wang2020carafeplus] | 66.35 | 64.67 | 48.91
IndexNet [lu2020index] | 65.77 | 64.30 | 48.74
A2U [dai2021learning] | 64.93 | 63.23 | 47.52
M2MRF-A | 66.48 | 64.89 | 49.07
M2MRF-B | 66.00 | 64.45 | 48.56
M2MRF-C | 67.24 | 65.71 | 49.94
M2MRF-D | 66.66 | 65.15 | 49.36
Table 5: Comparing with different RF operators on IDRiD [porwal2020idrid] testing set. The first row is the baseline i.e. vanilla HRNetV2 [wang2020deep]. A: (one-step/one-step), B:(one-step/cascade), C:(cascade/one-step), D: (cascade/cascade). All results are averaged over three repetitions. Methods | AUPR | F | IoU
---|---|---|---
EX | HE | SE | MA | mAUPR | EX | HE | SE | MA | mF | EX | HE | SE | MA | mIoU
VRT(1st)[porwal2020idrid] | 71.27 | 68.04 | 69.95 | 49.51 | 64.69 | - | - | - | - | - | - | - | - | - | -
PATech(2nd)[porwal2020idrid] | 88.50 | 64.90 | - | 47.70 | - | - | - | - | - | - | - | - | - | - | -
iFLYTEK(3rd)[porwal2020idrid] | 87.41 | 55.88 | 65.88 | 50.17 | 64.84 | - | - | - | - | - | - | - | - | - | -
L-seg[guo2019seg] | 79.45 | 63.74 | 71.13 | 46.27 | 65.15 | - | - | - | - | - | - | - | - | - | -
HED[xie2015holistically] | 80.81 | 66.41 | 68.09 | 40.45 | 63.94 | 78.60 | 64.33 | 67.00 | 38.81 | 62.18 | 64.74 | 47.43 | 50.38 | 24.07 | 46.66
DNL[yin2020disentangled] | 75.12 | 64.04 | 64.73 | 32.48 | 59.09 | 73.15 | 61.87 | 63.96 | 32.78 | 57.94 | 57.67 | 44.80 | 47.03 | 19.61 | 42.28
Deeplabv3+[chen2018encoder] | 81.93 | 64.66 | 63.04 | 43.14 | 63.19 | 79.60 | 61.96 | 61.48 | 40.57 | 60.90 | 66.10 | 44.90 | 44.39 | 25.45 | 45.21
PSPNet[zhao2017pyramid] | 75.21 | 63.36 | 63.65 | 32.71 | 58.73 | 73.24 | 60.81 | 62.83 | 32.63 | 57.38 | 57.78 | 43.71 | 45.81 | 19.50 | 41.70
SPNet[hou2020strip] | 64.61 | 54.11 | 52.14 | 30.14 | 50.25 | 61.45 | 44.33 | 49.59 | 25.66 | 45.26 | 44.40 | 28.50 | 33.58 | 14.72 | 30.30
HRNetV2[wang2020deep] | 82.09 | 65.50 | 68.68 | 43.76 | 65.01 | 79.93 | 62.58 | 67.53 | 42.49 | 63.14 | 66.57 | 45.56 | 50.99 | 26.98 | 47.52
Swin-base[liu2021swin] | 81.30 | 67.70 | 66.46 | 44.19 | 64.91 | 79.64 | 66.42 | 66.00 | 44.09 | 64.04 | 66.17 | 49.72 | 49.27 | 28.28 | 48.36
Twins-SVT-B [chu2021Twins] | 80.09 | 63.12 | 68.86 | 43.27 | 63.84 | 78.56 | 61.98 | 68.19 | 42.42 | 62.79 | 64.68 | 44.91 | 51.76 | 26.92 | 47.07
M2MRF-A | 82.04 | 67.80 | 68.23 | 47.86 | 66.48 | 79.64 | 65.66 | 66.68 | 47.59 | 64.89 | 66.16 | 48.87 | 50.03 | 31.23 | 49.07
M2MRF-B | 81.98 | 67.41 | 66.68 | 47.91 | 66.00 | 79.57 | 65.39 | 65.01 | 47.81 | 64.45 | 66.07 | 48.58 | 48.16 | 31.42 | 48.56
M2MRF-C | 82.16 | 68.69 | 69.32 | 48.80 | 67.24 | 79.85 | 66.42 | 67.92 | 48.63 | 65.71 | 66.46 | 49.72 | 51.43 | 32.13 | 49.94
M2MRF-D | 82.29 | 66.94 | 69.00 | 48.43 | 66.66 | 79.97 | 64.88 | 67.50 | 48.26 | 65.15 | 66.62 | 48.04 | 50.98 | 31.81 | 49.36
Table 6: Results on IDRiD [porwal2020idrid]. M2MRF-A: (one-step/one-step),
M2MRF-B:(one-step/cascade), M2MRF-C:(cascade/one-step), M2MRF-D:
(cascade/cascade). The Best and Second Best results are highlighted in
boldface and underlined respectively.

###  4.3 Results on IDRiD

####  4.3.1 Comparison to state-of-the-art RF operators

To further validate the effectiveness of our proposed M2MRF, we conduct
experiments on IDRiD [porwal2020idrid] and make comparisons with existing
eight pairs of RF operators. Table 5 reports the results. It shows that
M2MRF-C, i.e., cascade M2MRF for downsampling and one-step M2MRF for
upsampling ranks first while our M2MRF-D, i.e., cascade M2MRF for both
downsampling and upsampling ranks second among listed competitors.
Particularly, our M2MRF-C surpasses the baseline consistently in terms of
mAUPR (67.24% vs. 65.01%), mF (65.71% vs. 63.14%) and mIoU (49.94% vs. 47.52%)
by a large margin. Interestingly, two pairs of rule-based RF operators, i.e.,
MaxPool/Bilinear interpolation and MaxPool/Unpooling achieve better than
previous learning-based RF operators. The possible reason is that it may be
difficult for learning-based RF operators to preserve activations caused by
tiny lesions via learning from a small local region with limited size of
receptive field.

Figure 8: Segmentation results on IDRiD test set [porwal2020idrid]. From (a)
to (n) are original image, patch with lesion annotations by experts, results
by our M2MRF-A/B/C/D, HED [xie2015holistically], DNL [yin2020disentangled],
Deeplabv3+ [chen2018encoder], PSPNet [zhao2017pyramid], SPNet [hou2020strip],
HRNetV2 [wang2020deep], Swin-base [liu2021swin] and Twins-SVT-B
[chu2021Twins]. Regions delineated in green, blue and red are HEs, MAs and EXs
by experts or segmentation approaches. Lesions marked with solid golden boxes
in (b) are challenge lesions. Four variants of HRNetV2 [wang2020deep] equipped
with our M2MRFs are almost able to correctly segment those challenge lesions
while compared methods are prone to encounter either miss or wrong
identifications which are marked with dotted golden boxes.

####  4.3.2 Comparison to state-of-the-art segmentation methods

We compare our M2MRF with four tiny lesion segmentation methods (VRT, PATech,
iFLYTEK [porwal2020idrid] and L-seg [guo2019seg]), six state-of-the-art CNN-
based methods and two recent transformer-based methods (Swin-base
[liu2021swin] and Twins-SVT-B [chu2021Twins]). Among four tiny lesion
segmentation methods, performance of L-Seg [guo2019seg] is directly borrowed
from original paper and the rest are top three methods borrowed from the 2018
ISBI grand challenge. Results of six CNN-based segmentation methods and Twins-
SVT-B [chu2021Twins] are obtained via fine-tuning pre-trained model on
ImageNet1K [krizhevsky2012imagenet] with IDRiD [porwal2020idrid] training set
while Swin-base [liu2021swin] is ImageNet-22K.

Table 6 reports comparison results, from which we have following observations:
(1) Our four variants of M2MRF surpass all the competitors consistently in
terms of mAUPR, mF and mIoU; (2) They contribute significantly to MA
segmentation and outperform vanilla HRNetV2 [wang2020deep] by more than 4% on
AUPR, 5% on F-score and 4% on IoU, which indicates that our M2MRF is able to
maintain more subtle activations about tiny lesions; (3) Among them, our
M2MRF-C achieves best performance, which surpasses the most recent
transformer-based segmentation method Swin-base [liu2021swin] by 2.33%, 1.67%
and 1.58% in mAUPR, mF and mIoU respectively; (4) Methods specially designed
for lesion segmentation achieve better in mAUPR than the compared CNN-based
methods except for HRNetV2 [wang2020deep]; (5) HRNetV2 [wang2020deep] is a
strong baseline on tiny lesion segmentation task and achieves competitive
mAUPR to methods participating grand challenge as it learns high-resolution
representations with detailed spatial information. We also make qualitative
comparison to compared state-of-the-art methods and visualise their results in
Fig. 8.

##  5 Conclusion

We present a simple RF operator named M2MRF for feature reassembly which
unifies feature downsampling and upsampling in one framework. Our M2MRF
considers contributions of long range spatial dependencies and simultaneously
reassembles multiple features in a large region to multiple target features
once. Thus, it is able to maintain activations caused by tiny lesions during
feature reassembly. It shows significant improvements on two public tiny
lesion segmentation datasets, i.e. DDR [li2019diagnostic] and IDRiD
[porwal2020idrid]. Moreover, our M2MRF only introduces marginal extra
parameters and could be used to replace arbitrary RF operators in existing CNN
architectures.