# WSSS4LUAD: Grand Challenge on Weakly-supervised Tissue Semantic Segmentation
for Lung Adenocarcinoma

Chu Han†, Xipeng Pan†, Lixu Yan†, Huan Lin†, Bingbing Li, Su Yao, Shanshan Lv,
Zhenwei Shi, Jinhai Mai, Jiatai Lin, Bingchao Zhao, Zeyan Xu, Zhizhen Wang,
Yumeng Wang, Yuan Zhang, Huihui Wang, Chao Zhu, Chunhui Lin, Lijian Mao, Min
Wu, Luwen Duan, Jingsong Zhu, Dong Hu, Zijie Fang, Yang Chen, Yongbing Zhang,
Yi Li, Yiwen Zou, Yiduo Yu, Xiaomeng Li, Haiming Li, Yanfen Cui, Guoqiang Han,
Yan Xu, Jun Xu, Huihua Yang, Chunming Li, Zhenbing Liu, Cheng Lu, Xin Chen∗,
Changhong Liang∗, Qingling Zhang∗, Zaiyi Liu∗ Chu Han, Xipeng Pan, Huan Lin,
Zhenwei Shi, Jinhai Mai, Bingchao Zhao, Zeyan Xu, Chao Zhu, Huihui Wang, Yuan
Zhang, Haiming Li, Yanfen Cui, Cheng Lu, Changhong Liang and Zaiyi Liu are
with the Department of Radiology, Guangdong Provincial People’s Hospital,
Guangdong Academy of Medical Sciences, Guangzhou, 510080 China; Guangdong
Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis
and Application, Guangdong Provincial People’s Hospital, Guangdong Academy of
Medical Sciences, Guangzhou, 510080 ChinaQingling Zhang, Lixu Yan, Su Yao and
Shanshan Lv are with the Department of Pathology, Guangdong Provincial
People’s Hospital, Guangdong Academy of Medical Sciences, Guangzhou, 510080
ChinaZhenbing Liu, Zhizhen Wang and Yumeng Wang are with the School of
Computer Science and Information Security, Guilin University of Electronic
Technology, Guangxi, 541004 ChinaBingbing Li is with the Department of
Pathology, Guangdong Provicial People’s Hospital Ganzhou Hospital, Ganzhou,
ChinaHuihua Yang is with the College of Artificial Intelligence, Beijing
University of Posts and Telecommunications, Beijing, 100876 ChinaJiatai Lin
and Guoqiang Han are with the School of Computer Science and Engineering,
South China University of Technology, Guangzhou, 510640 ChinaJun Xu is with
the Institute for AI in Medicine, School of Artificial Intelligence, Nanjing
University of Information Science and Technology, Nanjing, 210044 ChinaYan Xu
is with the School of Biological Science and Medical Engineering, State Key
Laboratory of Software Development Environment, Key Laboratory of Biomechanics
and Mechanobiology of Ministry of Education, Research Institute of Beihang
University in Shenzhen, Beijing Advanced Innovation Center for Biomedical
Engineering, Beihang University, Beijing, 100191 ChinaXin Chen is with the
Department of Radiology, Guangzhou First People’s Hospital, the Second
Affiliated Hospital of South China University of Technology, Guangzhou, 510180
ChinaChunhui Lin, Lijian Mao and Min Wu are with the Zhejiang Dahua Technology
Co. Ltd, Zhejiang, 310053 ChinaLuwen Duan is with the School of Biomedical
Engineering, University of Science and Technology of China, Hefei, 230022
ChinaZijie Fang, Yang Chen and Yongbing Zhang are with the Tsinghua Shenzhen
International Graduate School, Tsinghua University, Shenzhen, ChinaYi Li,
Yiwen Zou, Yiduo Yu and Xiaomeng Li are with Electronic and Computer
Engineering & Computer Science and Engineering, The Hong Kong University of
Science and Technology, Hong KongJingsong Zhu is the School of Automation
Science and Engineering, Xi’an Jiaotong University, Xian, 710049 ChinaDong Hu
is with Department of Computer Science, Huaqiao University, Xiamen, 361021
China†Equal contribution∗ Corresponding authors

###### Abstract

Lung cancer is the leading cause of cancer death worldwide, and adenocarcinoma
(LUAD) is the most common subtype. Exploiting the potential value of the
histopathology images can promote precision medicine in oncology. Tissue
segmentation is the basic upstream task of histopathology image analysis.
Existing deep learning models have achieved superior segmentation performance
but require sufficient pixel-level annotations, which is time-consuming and
expensive. To enrich the label resources of LUAD and to alleviate the
annotation efforts, we organize this challenge WSSS4LUAD to call for the
outstanding weakly-supervised semantic segmentation (WSSS) techniques for
histopathology images of LUAD. Participants have to design the algorithm to
segment tumor epithelial, tumor-associated stroma and normal tissue with only
patch-level labels. This challenge includes 10,091 patch-level annotations
(the training set) and over 130 million labeled pixels (the validation and
test sets), from 87 WSIs (67 from GDPH, 20 from TCGA). All the labels were
generated by a pathologist-in-the-loop pipeline with the help of AI models and
checked by the label review board. Among 532 registrations, 28 teams submitted
the results in the test phase with over 1,000 submissions. Finally, the first
place team achieved mIoU of 0.8413 (tumor: 0.8389, stroma: 0.7931, normal:
0.8919). According to the technical reports of the top-tier teams, CAM is
still the most popular approach in WSSS. Cutmix data augmentation has been
widely adopted to generate more reliable samples. With the success of this
challenge, we believe that WSSS approaches with patch-level annotations can be
a complement to the traditional pixel annotations while reducing the
annotation efforts. The entire dataset has been released to encourage more
researches on computational pathology in LUAD and more novel WSSS techniques.

###### Index Terms:

Lung adenocarcinoma, Grand challenge, Computational pathology, Weakly-
supervised semantic segmentation

##  I Introduction

Lung cancer is the leading cause of cancer death worldwide [1], and lung
adenocarcinoma (LUAD) is the most common subtype [2]. During the diagnosis
process of LUAD, histopathology examination of the tissue section, stained
with hematoxylin and eosin (H&E), remains the gold standard. It delivers
massive information on tumor microenvironment from different perspectives,
such as the morphology of the tissues, spatial arrange at tissue-level or
cell-level, nuclear atypia and etc [3]. Experienced pathologists are able to
evaluate the tumor progression by reading the patterns of the interaction
between the tumor and its microenvironment [4]. But it is experience-dependent
with limited interobserver agreement [5]. For an objective, quantitative and
precise analysis of histopathology sections, it is urged to translate the
visual language to the computer language.

Figure 1: Tumor heterogeneity in different cancers.

Tissue semantic segmentation is one of the most important upstream tasks in
Digital Pathology. It defines the tissue type for every single pixel and
allows quantitative analysis for the downstream cancer outcome predictions
[6]. The evolution of deep learning algorithms [7] is leading the image
segmentation and recognition tasks to a computational and data-driven manner.
These learning-based models achieved outstanding performance compared with
traditional approaches by strong capacity neural network models [8, 9] under
the supervision of a large among of annotated data [10]. However, for the
image segmentation task, drawing pixel-level annotation is extremely difficult
and may spend a tremendous amount of time and resources [11]. For
histopathology images, such difficulties have been even amplified due to the
specialty of the whole slide images (WSI). First of all, WSIs are scanned by
microscopic slide scanners. The image resolution of WSIs can be up to over
100000$\times$100000\. Drawing pixel-level annotations in such huge images are
much harder than in natural images. Second, malignant tumors are
heterogeneous. Tumor epithelial has complex morphological appearances, which
leads to ambiguities between two tissue types. For lung adenocarcinoma, the
labeling process is even harder than for other malignant tumors. As can be
seen in Fig. 1, the interactions between tumor and tumor-associated stroma in
LUAD are twisted and more complex than other two cancers (from TCGA
111https://www.cancer.gov/tcga). Last but not least, because of the image
specialty, crowdsourcing by normal people with no medical background is
incapable of the labeling job. It may spend a long time for pathologists to
collect sufficient pixel-level training samples for tissue semantic
segmentation models.

To tackle these problems, we set up the WSSS4LUAD challenge with two goals. 1)
Introducing more public labeling resources of WSIs in lung adenocarcinoma, to
promote the research on lung cancer histopathology image analysis. 2)
Discovering a more efficient and easier way to accelerate the labeling process
and alleviate the annotation efforts of pathologists for tissue semantic
segmentation. In computer vision, some researchers are focusing on using
image-level labels to achieve semantic segmentation, called weakly-supervised
semantic segmentation (WSSS). In this challenge, participants have to build
innovative WSSS techniques to achieve tissue semantic segmentation for LUng
ADenocarcinoma. That is the reason why this challenge is called WSSS4LUAD. To
the best of our knowledge, this is the first challenge and dataset for WSSS in
lung adenocarcinoma. We defined three majority tissue types for lung
adenocarcinoma, including tumor epithelial, tumor-associated stroma and normal
tissue. WSSS4LUAD was organized as a part of the ISICDM 2021 (the 5th
International Symposium on Image Computing and Digital Medicine). With the
outstanding WSSS algorithms tailored for histopathology images, we hope that
the conventional way of drawing pixel-level annotations can be replaced by a
simpler way of clicking and selecting the classes of the patch.

##  II Related Works

In this paper, we briefly review the recent works on histopathology image
analysis for lung cancer, histopathology image segmentation and annotation
efficient models.

###  II-A Lung Cancer - Histopathology Image Analysis

Histopathology slides are the golden standard for lung cancer diagnosis [12].
However, only relying on the visual inspection of the pathologists cannot
fully discover the potential value of the histopathology slides. The invention
of whole slide image scanners and the development of artificial intelligence
techniques bring computational pathology to lung cancer [13]. Comprehensive
analysis in different levels of the whole slide images is now available with
the computing power of GPU (graphics processing unit), including image-level
feature analysis, tissue-level quantitative analysis and cell-level spatial
distribution analysis [14].

Yu et al. [15] extracted a large set of image features and proposed a machine
learning pipeline for risk-stratification in patients with non-small cell lung
cancer (NSCLC). Coudray et al. [16] applied an end-to-end CNN model for lung
cancer diagnosis. The model classified lung cancer into three categories,
adenocarcinoma (LUAD), squamous cell carcinoma (LUSC) and normal lung tissues.
The diagnostic performance of the AI model was equivalent to that of
pathologists with the area under the curve (AUC) of 0.97. The authors also
demonstrated that deep learning models could be used to predict molecular
status, like STK11, EGFR, FAT1, SETBP1, KRAS and TP53 mutant genes.
Abduljabbar et al. [17] analyzed the spatial distribution of different types
of cells and found heterogeneity of immune status, defined as two immune
phenotypes (immune hot and immune cold). The immune phenotype can predict the
risk of cancer recurrence of lung adenocarcinoma. Zheng et al. [18] discussed
the interactions between tumor cells and tumor-associated macrophages (TAM) in
multiple immunofluorescence staining images and proved the density and spatial
distribution of TAM phenotypes to be the prognostic factors for lung cancer
survival. Lu et al. [19] constructed interpretable features by analyzing tumor
cellular diversity (CellDiv), which was proven to be a strong prognostic
factor of 5-year overall survival in NSCLC patients validated by the multi-
center data. Recently, an AI-powered spatial analysis model [20] has been
proposed to analyze tumor-infiltrating lymphocytes in the tumor
microenvironment, in order to find the correlation with immune checkpoint
inhibition in advanced NSCLC. This research team has made around $2.8\times
10^{9}\mu\rm{m}^{2}$ tissue-level annotated area in over 3,000 tumor slides,
which is a tremendous annotation workload.

Even a lot of excellent researches have been proposed for histopathology image
analysis for lung cancer, researchers still have to spend much time collecting
a large among of annotations for the segmentation model. Therefore, we
organize this challenge to enrich the annotation resources and propose a
simpler labeling way to relieve the annotation efforts.

###  II-B Fully-supervised Histopathology Image Segmentation

The main contribution of computational pathology [6] is to automatically,
precisely, objectively and quantitatively assess the whole slide image.
Histopathology image segmentation is the most basic task in this field. It can
help differentiate the constituents of the tumor in different levels, such as
tissue-level [21], gland-level [22] and cell-level [23, 24], translating the
visual language to the computer language.

Recently, more and more tissue segmentation challenges have been held such as,
BCSS [25] for breast cancer, Gleason 2019 [26] for prostate cancer, DigestPath
2019 [27] for colon cancer and LungHP Challenge 2019 [28] for lung cancer.
LungHP Challenge 2019 is the only one we found in this field. It aims to
segment cancer tissue in WSI in three different lung cancer subtypes, SCC,
SCLC and ADC. By exploring the digital pathology communities, we found that
tissue-level label resources are still limited, especially in lung cancer.
That is one of the reasons why we organize this challenge to promote and
complement the histopathology image analysis in lung cancer.

However, the conventional manual annotation way is slow, inefficient and
annoying. And this work cannot be done in a crowdsourcing way like natural
images, because people with no medical background are not qualified for this.
Therefore, we want to introduce computer algorithms in this process, pushing
the labeling job to an annotation-efficient manner.

###  II-C Annotation Efficient Approaches

With the great success of fully-supervised deep learning models, researchers
now focus on solving medical image segmentation tasks in an annotation
efficient manner. It can be roughly categorized into several directions,
active learning, self-supervised learning and not-so-supervised learning.

####  II-C1 Active Learning

Active learning (AL) [11] is a popular technique in medical image segmentation
due to the expertise requirements of the labelers. The concept of AL is to let
AI models decide what should be labeled, which are commonly the most valuable
and informative samples [29, 30].

Doyle et al. [31] used the AL approach to select the most informative regions
in prostate histopathology images. Such a strategy can alleviate the class-
imbalance problem of the minority classes. Zhou et al. [32] associated active
learning with transfer learning to select the worthy samples while
continuously improving the model by continual fine-tuning. Belharbi et al.
[33] introduced a self-training model to the active learning process. They
used the segmentation pseudo masks to guide the classification tasks.
Greenwald et al. [34] proposed a pathologist-in-the-loop TissueNet to create
the largest cell dataset for fluorescent multiplex immunohistochemistry with
over a million labeled cells.

####  II-C2 Self-supervised Learning

Self-supervised learning [35] aims to learn a good initialized feature
representation for the upcoming task by a pretext task without the necessity
of human annotations, including contrastive, predictive and generative tasks.

Contrastive learning [36] trains the model by measuring the similarity and
difference between the images. Following this idea, various famous works have
been proposed soon, including MoCo [37], SimCLR [38] and BYOL [39, 40].
Predictive learning sets up some prediction tasks with the generated labels by
the prior knowledge of the images, such as image reconstruction ([41]), jigsaw
puzzle solving [42] and Rubik cube solving [43, 44]. Generative models learn
the domain knowledge by restoring the manually corrupted images like image
inpainting [45].

In histopathology image analysis, Srinidhi et al. [46] proposed a “resolution
sequence prediction task” to mimic how pathologists read the WSIs at different
magnifications. Boyd et al. [47] introduced a visual field expansion to use
the input image to generate the pseudo contents of the surrounding area. Cheng
et al. [48] proposed a curriculum self-supervised learning strategy by
training several tasks with progressively increasing difficulties. It includes
the image reconstruction task, the image inpainting task and the stain
deconvolution task. Luo et al. [49] proposed an SD-MAE by associating self-
distillation augmented SSL with masked autoencoder and improved the
classification performance of histopathology images. Numerous approaches have
already proven that the self-supervised pre-training can effectively learn the
domain-specific feature representations and the inherent image characteristics
to alleviate the training burden and the data requirement for the upcoming
tasks.

####  II-C3 Not-so-supervised Learning

Besides training a well-initialized model by self-supervised pretraining
approaches, researchers also try their best to reduce the annotation efforts
in a not-so-supervised manner ([50]), such as to simplify the problem by
patch-level annotations (patch-level classification), to maximize the value of
the scarce labels (semi-supervised learning) or to replace pixel-level labels
by weaker labels (weakly-supervised learning).

Patch-level classification: To alleviate dense pixel-level annotation for
tissue segmentation, Kather et al. [51, 52] simplified the semantic
segmentation problem to a patch-level classification problem, which scarified
the boundaries precision. Following this idea, Lin et al. [53] proposed a
deep-broad plug-and-play module (Pyramidal Deep-Broad Learning, PDBL) that can
be easily adapted to any well-trained CNN classification backbone. The
proposed PDBL can greatly reduce the training burden and annotations while
maintaining a promising classification accuracy.

Semi-supervised learning: Semi-supervised learning explores the semantic
connections between the labeled images and the unlabeled images. Mainstream
models are usually formulated according to the following two spirits,
generating pseudo labels [54] or maintaining the semantic consistency [55,
56]. They are based on the assumption that the distribution between labeled
and unlabeled images should be close enough, which is an obstacle in
histopathology images. Due to the heterogeneous tumor microenvironment, the
same tissue class may show totally different morphological appearances. For
histopathology image segmentation, weaker labels with larger sample varieties
in a weakly-supervised manner may be more effective than dense annotations
with limited sample varieties in a semi-supervised manner.

Weakly-supervised learning: Researchers also tried different kinds of weak
labels to replace dense pixel ones. Multiple instance learning models [57, 58]
were proposed for cancerous region segmentation given the slide-level labels
only. But it can only segment cancerous or non-cancerous regions. Scribble-
based models were introduced to perform a fast user interaction and label
adjustment in both cellular-level [59] and tissue-level [60]. Various point-
based models [61, 62, 63, 64] have proven the effectiveness on the nuclei
segmentation task. Some recent works [65, 66] utilized the proportion of the
tissue as the labels.

CAM-based models [67] have been widely used in weakly-supervised semantic
segmentation (WSSS), including histopathology images [68]. The organizers of
WSSS4LUAD challenge also proposed a CAM-based model [69] to achieve WSSS for
tissue segmentation. They introduced a progressive dropout attention mechanism
to gradually deactivate the most distinguishable regions. A multi-layer
pseudo-supervision was proposed to enrich the information. In this challenge,
we apply this model to generate the initial pixel-level pseudo masks for the
validation and test sets, which greatly saves the annotation times of the
pathologists.

##  III Challenge Description

Figure 2: Brief introduction of WSSS4LUAD challenge. (a) We scanned 67
histopathology slides (one slide per patient) from Guangdong Provincial
People’s Hospital and downloaded 20 WSIs from TCGA. (b) one expert, two senior
and 8 junior pathologists were invited to annotate image-level training
samples and pixel-level validation samples. (c) and (d) shows some examples of
the training, validation and test sets.

In this challenge, we aim to use patch-level classification labels to predict
pixel-level semantic segmentation maps with three tissue categories, including
tumor epithelial (tumor), tumor-associated stroma (stroma) and normal tissue
(normal). So we prepare patch-level classification labels for the model
training and dense pixel-level annotations to evaluate the model performance.
A brief description of this challenge is shown in Fig. 2.

Figure 3: The pipeline of the pathologist-in-the-loop labeling strategy. (a)
Patch-level labeling for the training set. Pathologists first annotate 500
initial patch labels to train a preliminary model (ResNet38) and generate a
set of pseudo labels. Then they use a selection tool to check and correct the
pseudo labels. Pseudo labels with high confidence are verified by junior
pathologists. The ones with high uncertainty are corrected by senior
pathologists. Verified labels are then fed to fine-tune the classification
model. (b) Pixel-level labeling for the validation and test sets. The well-
trained ResNet38 in (a) is inherited in this phase to generate pixel-level
semantic segmentation maps by our previous work MLPS [69]. Pathologists refine
the pseudo masks using the pencil tool of Adobe Photoshop. The verified ground
truths are added to the training set to fine-tune the model. *For both tasks,
with more corrected labels, the model improves and the manual intervention
decreases.

###  III-A Whole Slide Images Preparation

The whole slide images are from two sites, including the Department of
Pathology, Guangdong Provincial People’s Hospital and TCGA (multiple centers).
Since LUAD is a highly heterogeneous tumor, in order to ensure the diversity
of the tissue-level morphological appearance, we scan 67 H&E stained
histopathology sections from 67 LUAD patients with surgical resection and
download 20 WSIs from TCGA. All the slides are scanned by a microscopic
digital slide scanner (Leica GT 450) at a 40X magnification with 0.2517
µm/pixel resolution and are under quality control by filtering out the blurry,
over-stained, under-stained and dirty slides. We do not perform stain
normalization to keep the unharmed source images for the participants. Because
of the ethical restrictions, the WSIs in our center cannot be released so far.
We are applying for approval from the ethical committee. Hopefully, the source
images can also be released in the future.

###  III-B Annotators Recruitment and Label Review Board

As demonstrated in Fig. 2 (b), we invite 11 annotators with medical
backgrounds to label the data and to verify the annotations, including one
expert pathologist with over 10 years of working experience in lung cancer
diagnosis, two senior pathologists with 5 years of working experience, and 8
junior pathologists with less than 5 years working experience or medical
students. All the labeling jobs are assigned to junior pathologists. The
expert pathologist and two senior pathologists form a label review board to
verify the labels. We introduce three mechanisms to guarantee consistency
among different labelers. 1) Before we start the labeling job, the expert
pathologist is asked to give a talk about the labeling criteria. 2) At the
beginning of the labeling process, we ask all the labelers to annotate a few
samples. And then all the labels are reviewed by the label review board and
form documentation of the label suggestions, which will be returned to each
labeler to adjust their labeling criteria. 3) We introduce a certain degree
(around 50%) of overlapping samples among adjacent labeling assignments.

Figure 4: More details of the dataset. (a) Data statistics include the number
of the patch-level and pixel-level labels for each tissue category in the
training, validation and test sets, respectively. And the distribution of the
multi-class labels in the training set. (b) demonstrates the image resolution
of the dataset. (c) shows the file naming convention of the sample in the
training set.

###  III-C Pathologist-in-the-loop Labeling Process

There are two kinds of labels we prepare for this challenge, patch-level and
pixel-level. To accelerate the annotation process and alleviate the annotation
efforts, we introduce a pathologist-in-the-loop labeling manner for both
annotation levels, demonstrated in Fig. 3. Such a process can greatly
accelerate the annotation process and reduce the annotation efforts.

Patch-level annotations: The way we construct the training set is demonstrated
in Fig. 3. First, pathologists have to annotate a set of classification labels
(around 500 patches) as the initial training set. Then we train a preliminary
classification model of ResNet38 to generate pseudo labels. The reason why we
choose ResNet38 instead of more popular classification backbones (like
ResNet50 or VGG16) is that ResNet38 is one of the most common backbones for
weakly-supervised semantic segmentation [69]. And the trained ResNet38 model
will be used to generate pixel-level labels in the next step. Next,
pathologists use a selection tool to check and correct the pseudo labels.
During the label correction process, we stratify the pseudo labels by the
model predictive confidence. Pseudo labels with high confidence are verified
by junior pathologists. The ones with low confidence and high uncertainty are
corrected by senior pathologists. All the patch labels are verified by at
least two pathologists. The inconsistent labels are further checked by the
label review board. Finally, the verified labels are then fed to fine-tune the
classification model.

Pixel-level annotations: We introduce our previous proposed approach MLPS
[69], a weakly-supervised semantic segmentation model tailored for the
histopathology images. We use the patch-level labels collected in the previous
phase to train MLPS. This model generates the heatmap by the class activation
maps technique for each tissue category. Then we obtain the pseudo masks by
combining the heatmaps. Pathologists refine the pseudo masks using the pencil
tool of Adobe Photoshop. The verified ground truths are added to the training
set to fine-tune the model. Note that, the large patches (over
1000$\times$1000 resolution) are first cut into several image tiles, and then
stitched back to the original size. All the pixel-level labels are finally
reviewed and confirmed by the label review board.

###  III-D Dataset Details

Here, we show the details of the dataset, including the statistics, image
resolution and file naming convention, shown in Fig. 4. Note that, the labels
of the training samples are three digit classification labels, [Tumor, Stroma,
Normal]. All the patches are under 10$\times$ magnification.

####  III-D1 Training set

In the training set, we randomly sample 49 WSIs from GDPH and 14 WSIs from
TCGA. A total of 15,000 patches are randomly cropped from the WSIs with
arbitrary resolution (around 150-300*150-300). During the histopathology
slide-making procedure, dust may exist in the slides inevitably, which will
cause out-of-focus blurs in some local regions when scanning the slides.
Therefore, all the patches are under quality control to filter out the blurry
and dirty regions. Furthermore, some highly ambiguous patches are also removed
to avoid introducing uncertain information to the model. Finally, 10,091
patches are left in the training set.

The statistics of the training data are shown in Fig. 4 (a). As can be seen,
the largest number of labels are [1,1,0], which is tumor and stroma without
normal. The second most labels are [1,0,0], [0,0,1] and [0,1,0] with only one
tissue categories. There are some label combinations are very few even is
nonexistent. Because of the invasiveness of LUAD, it is hard to observe a
clear boundary of the tumor and normal tissue. Therefore, it is almost
impossible to cover normal and tumor/stroma within such a small patch.

We put all the useful information of the training samples into the filename,
including the slide ID, $(x,y)$ coordinates and classification labels, shown
in Fig. 4 (c).

####  III-D2 Validation and test sets

We select 18 WSIs from GDPH and 6 WSIs from TCGA. Then the WSIs are randomly
and equally separated to the validation set (GDPH: 9, TCGA: 3) and the test
set (GDPH: 9, TCGA: 3). Total 40 patches are manually cropped by the label
review board for the validation set, including 9 large patches (around
1500-5000*1500-5000) and 31 small patches (around 200-500*200-500). Total 80
patches are manually cropped by the label review board for the test set,
including 14 large patches (around 1500-5000*1500-5000) and 66 small patches
(around 200-500*200-500). Fig. 4 (a) shows the total number of labeled pixels
in the validation and test sets. The ground truth pixel-level labels of each
tissue category are defined as follows.

  * •

Tumor epithelial: (0, 64, 128), (label: 0)

  * •

Tumor-associated stroma: (64, 128, 0), (label: 1)

  * •

Normal: (243, 152, 0), (label: 2)

  * •

White background: (255, 255, 255), (label: 3)

Since lungs are the main organ of the respiratory system. There are a lot of
alveoli (some air sacs) serving for exchanging the oxygen and carbon dioxide,
which forms some white background in WSIs. In this challenge, we pre-define
the white background by a simply threshold and exclude it when evaluating the
model.

###  III-E Evaluation Metrics

In this challenge, we use mIoU to evaluate the models. The white backgrounds
inside the alveoli are excluded when calculating mIoU. We have also provided a
background mask for each patch in the validation and test sets.

Participants have to only use the provided training data to train their own
models. External histopathological data is not allowed in this challenge.
ImageNet pre-trained model is allowed as the initialization.

##  IV Challenge Summary

Figure 5: The locations of the registrations. China 299, United States of
America 36, India 23, Canada 15, South Korea 14, Germany 12, France 11, Hong
Kong 10, United Kingdom 9, Taiwan 8, Netherlands 7, Japan 7, Iran 5, Spain 5,
Singapore 4, Russia 3, Romania 3, Italy 3, Pakistan 3, Turkey 3, Thailand 3,
Ukraine 2, Belgium 2, Australia 2, Austria 2, Malaysia 2, Egypt 1, Finland 1,
Sweden 1, Brazil 1, Norway 1, Indonesia 1, Israel 1, Bangladesh 1, Switzerland
1, Senegal 1, Hungary 1, Macao 1, Slovakia 1, Nepal 1, Vietnam 1, Kazakhstan
1, Afghanistan 1, Lithuania 1, unknown 20.

In this section, we summarize the challenge in the following aspects,
including the registrations and challenge results (quantitative and
qualitative).

###  IV-A Brief Summary

Until now, there are 532 participants have registered for this challenge to
access the data. Fig. 5 demonstrates the locations of the registration. The
top five countries or areas are China, USA, India, Canada and South Korea. 28
teams submit the results in the test phase with over 1000 submissions. Among
them, 14 teams achieved over 0.7 mIoU in the test phase, as shown in Table I.
Top-3 teams are required to submit the code, model and the introduction of the
methodology. For the other teams, they are optional. Next, we demonstrate the
methodology of the top-tier teams.

###  IV-B Top-tier Methods

####  IV-B1 Lin et al. (1st place, Team: ChunhuiLin)

Lin et al. propose a Siamese multi-task learning model for this challenge with
a classification task and a segmentation task. CAM generated by the
classification task is regarded as the pseudo-mask of the segmentation task.
ASPP is applied to obtain a larger receptive field. They introduce a cross-
supervision strategy between two identical siamese models. Here are the
details of the proposed model.

Pre-processing: Since the histopathology images are different from natural
images, Lin et al. first calculate the mean and variance of the dataset to
replace the parameters of ImageNet. Then, they upsample the number of normal
tissue to re-balance the class by data augmentation. For the patches with only
one class, they generate fake images by cutmix data augmentation to provide
the pixel-level ground truth for the segmentation task.

Model: The same as the most conventional weakly-supervised semantic
segmentation method, Lin et al. generate pseudo masks by calculating the class
activation maps via a multi-label image classification model. They find that
half of the training patches are single-class and the background of the
histopathology image is relatively simple. Thus, Lin et al. use traditional
image processing methods to generate more patch-level pseudo labels to enhance
the classification model. After that, Lin et al. introduce an additional
semantic segmentation branch and form a multi-task learning model. Two tasks
share the same encoder and use different decoders. At the same time, they
construct two datasets for each task. One is more than 4000 images with pixel-
level pseudo labels, and the other one contains image-level labels. Based on
the multi-task learning model, they construct a siamese multi-task network.
Two multi-task learning models with identical structures. A cross-supervision
strategy is proposed to use the pseudo masks generated by siamese models to
supervise each other.

Training & inference: Lin et al. improve the loss function through joint
Lovasz loss and cross-entropy loss. Online hard negative mining is applied
during training. During the inference, they ensemble the outputs of the
Siamese model. In addition, model ensemble has been done in this challenge,
including ResNet101 and ResNeSt269 based on Deeplabv3 and HRNetw48.

Post-processing: Lin et al. applied the BFS algorithm to fill the false
background pixel by searching the nearest neighbor non-background pixel. In
cancer pathological sections, Tumor and Stroma are often intertwined, but
normal tissues hardly overlap with these areas. Based on this prior knowledge,
they calculate the distribution of each category in the neighborhood (100x100)
of each Tumor and Stroma pixel. After that, they modify the Tumor or Stroma
pixels that accounted for more than 30% of the normal category in the
neighborhood to normal pixels.

####  IV-B2 Zhu et al. (2nd place, Team: baseline0412)

The task is the pixel-level prediction for tumor epithelial tissue, tumor
stromal tissue and normal tissue. Since only image-level annotations (3-digit
labels) are available in the training set, Zhu et al. employed weakly-
supervised learning model.

Dataset Enhancement: The concept of image splicing is to use parts of two or
more images to create a new fake image. In this challenge, Zhu et al. apply
the image splicing method to enhance the dataset in several aspects. First,
they select the training samples with a single class, which only contains
tumor epithelial, tumor-associated stroma or normal tissue, and extract the
foreground region by using threshold segmentation. These single-class patches
are used as ground truth samples for the segmentation task. Then, for the
above single-class patches, they further apply the image splicing method to
generate more fake image and ground truth pairs. They achieve this by randomly
cutting a small piece from one patch and pasting it on the patch with
different classes. And the patches with multiple classes are generated. Next,
Zhu et al. also apply the image splicing method to the pseudo masks generated
by CAM.

Model: Weakly supervised models often use deep neural networks to generate the
Class Activation Mapping (CAM), and to serve as pseudo-labels for the
segmentation network. However, only relying on CAMs cannot obtain precise
region boundaries. In order to enhance the localization accuracy of class
activation mapping, they propose a multi-task learning model with the image
splicing method. The proposed model consists of two modules, the first module
is the classification model with CAM which is commonly used in weakly
supervised semantic segmentation. The second module is a segmentation network
guided by pseudo masks.

TABLE I: Qualitative results of the leaderboard (Test phase). Team name | mIoU | Tumor | Normal | Stroma
---|---|---|---|---
ChunhuiLin | 0.8413 | 0.8389 | 0.8919 | 0.7931
baseline0412 | 0.8222 | 0.8402 | 0.8343 | 0.7921
Vison307 | 0.8058 | 0.8165 | 0.8554 | 0.7456
BinghongWu | 0.8057 | 0.8045 | 0.8654 | 0.7471
adbertyoungdalu | 0.8025 | 0.7967 | 0.8668 | 0.7440
DPPD | 0.7815 | 0.7895 | 0.8397 | 0.7153
chenxl | 0.7714 | 0.7897 | 0.8159 | 0.7088
sibet0222 | 0.7609 | 0.8121 | 0.7107 | 0.7599
guoxutao | 0.7552 | 0.8179 | 0.6840 | 0.7636
shichuanyexi | 0.7411 | 0.8192 | 0.6714 | 0.7325
zyw19990916 | 0.7382 | 0.8080 | 0.6868 | 0.7196
York | 0.7239 | 0.8023 | 0.6710 | 0.6985
<EMAIL_ADDRESS>| 0.7199 | 0.7557 | 0.7248 | 0.6791
Zlin3000 | 0.7064 | 0.7493 | 0.6863 | 0.6837

####  IV-B3 Fang et al. (3rd place, Team: Vison307)

Pseudo Images and Masks Generation: The same as the top two teams, Fang et al.
also perform a pseudo image generation by the patches with the single class.
They randomly select patches with labels [1, 0, 0], [0, 1, 0] and [0, 0, 1]
and randomly crop the patches and concatenate them to a new patch. Such a
process can generate a set of multi-label patches with pixel-level pseudo
masks to pre-train a segmentation model. Then, they augment the data by
introducing rotations and flips. Cutmix strategy is also applied to further
increase the training samples.

Figure 6: Qualitative results of top-tier approaches. Masks are shown at the
top-left corner of the images. For better visualization, we also overlay the
semantic segmentation masks on the images.

Model: Fang et al. use DeepLab V3+ as the segmentation backbone. The model is
first pre-trained by the generated samples and the corresponding pixel-level
pseudo labels. Then they infer the rest patches with multiple categories, and
the generated masks are regarded as pseudo masks. Next, the model is trained
by the entire dataset. They train multiple models and use an ensemble strategy
for the inference.

Figure 7: Top-1 model v.s. Ground truth. (a) are two images cropped from the
large patches in the test set. (b) are the ground truth masks overlaid on the
original images. The yellow arrows highlight some disagreement local areas
between GT and prediction results. (c) are the results generated by the Top-1
model overlaid on the original images.

####  IV-B4 Li et al. (Team: shichuanyexi)

Model: Li et al. proposed an online example mining strategy in the
segmentation part to emphasize the credible supervision signals of pseudo-
masks from the classification part. And thus, boost the performance via
mitigating the adverse impact of noise in pseudo-masks. Specifically, the loss
map $\boldsymbol{L}$ of the segmentation model is normalized by
$\frac{\rm{softmax}(-\boldsymbol{L})}{\rm{mean}(\rm{softmax}(-\boldsymbol{L}))}$
to generate the loss weight for weighted cross-entropy loss. From this
sampling strategy, easy examples at lower loss values are assigned higher loss
weights in optimization dynamically.

Besides this core innovation, this team implemented a two-stage framework for
fine-grained predictions based only on the image-level label. It starts from
the classification stage, whose backbone is ResNet38 [70]. The downsample
stride is 8, and the feature maps of the last three stages are fused with
interpolations before the classifier for multi-scale representations. All the
images are resized into 224 during training. Then the synthesized pseudo-masks
are obtained from CAM [67]. In the second stage, the segmentation framework is
PSPNet [71] with backbone ResNet38 at training resolution 224 and testing
resolution 320. Data augmentations include random flip, random resized crop,
and all patches are normalized by the calculated mean and variance of the
competition dataset. Besides, multi-scale testing is applied to both stages.

###  IV-C Quantitative and Qualitative Results

Table I demonstrates the quantitative results of the participants with
$mIoU>0.7$. As we can see, the top five teams have achieved over 0.80 mIoU.
According to the technical reports in Sec. IV-B, all the top three teams used
the cutmix-like data augmentation way to generate pixel-level labels from
single-class patches to provide fully supervision for the segmentation model,
which achieved better performance compared with the team that only rely on the
patch-level annotations (shichuanyexi). The top-1 team (ChunhuiLin) leverages
the prior knowledge that the normal tissue commonly has less crosstalk with
tumor epithelial and tumor-associated stroma and achieves the best normal
tissue segmentation performance.

In Fig. 6, we select three patches (two small patches and a large patch) with
different tissues categories, large intra-class varieties and different
staining styles to evaluate the robustness of the models. The top-1 team
maximizes the values of the entire training labels, including generating
pixel-level labels from single-class patches by cutmix and pixel-level pseudo
labels from multi-labels patches by CAM. The design of the cross-supervision
strategy with the Siamese network can also help suppress the noises in the
pseudo masks. Thus they achieve the best qualitative performance. The top-3
team only uses the generated labels from single-class patches and trains a
fully-supervised semantic segmentation model. They totally abundant the multi-
class patches, which make the model predict less effective and inaccurate
boundaries for the patches with tumor epithelial and tumor-associated stroma.

Considering the inherent difficulties of manual annotations on histopathology
images, the ground truth is inevitably imperfect with some noises, no matter
for pixel-level or for patch-level patches. Therefore, models proposed by the
top-tier teams may even perform better than we expected. Fig. 7 demonstrates
the comparison between the Top-1 model results and ground truth. Some obvious
disagreement areas are pointed out by the yellow arrows. We can find that most
human experts define the regions from a global perspective. Some very small
stroma regions that interact with tumor epithelial are ignored by
pathologists. For such a huge patch or even a whole slide image, it is also
very difficult to precisely define every single pixel. According to the
results generated by the deep learning model shown in Fig. 7 (b), the AI model
is more sensitive than pathologists to the small regions as long as they have
similar features representations.

##  V Conclusion

In this paper, we summarized the WSSS4LUAD challenge, a weakly-supervised
semantic segmentation challenge for WSIs in lung adenocarcinoma. The original
intention of this challenge actually comes from the daily research in
computational pathology. For each project, pathologists have to draw a large
among of pixel-level labels again and again outside of the working hours. Due
to the image specialty, the labeling job cannot be replaced by people with no
medical background. Such huge labeling efforts make them hard to focus on the
real insight of the research itself. Therefore, we want to leverage novel
computer vision techniques to save the annotation efforts.

We invited 11 pathologists to prepare labels for this challenge. A total of 87
WSIs were collected from 87 patients diagnosed with lung adenocarcinoma from
both GDPH and TCGA. We introduced a pathologist-in-the-loop strategy to
accelerate the labeling process. The final dataset contains 10091 patch-level
labels in the training set and over 131 million pixels annotations in the
validation and test sets. The entire dataset is released on the challenge
website.

There are 532 registrations from over 40 countries or regions. 28 teams
submitted their results with over 1,000 submissions. The best performance team
achieves 0.8413 mIoU. According to the technical reports from the top-tier
teams, CAM is still the first choice for the WSSS problem. We can see that
participants tried their best to minimize the information gap between the
patch-level labels and the pixel-level labels in several aspects. In the
technical aspect, some teams proposed a multi-task learning strategy with a
classification task and a segmentation task to learn more generalizable
features. The top-1 team proposed a cross-supervision with Siamese network
structure. They used the pseudo masks generated by the Siamese branches to
guide each other, which is like an ensemble or voting strategy. In the data
augmentation aspect, the cutmix (image splicing) approach has been widely used
to introduce more reliable labels by generating fake images from single-class
images. Class balancing is also applied to avoid the model being dominated by
the majority classes. After the challenge, we also invited the pathologists
who were involved in the labeling process to manually assess the segmentation
results, which have a high agreement.

From our experience, we believe that annotating patch-level weak labels is a
feasible solution to be a complement to the conventional drawing pixel
boundaries way for tissue semantic segmentation. It has the great potential to
alleviate the annotation efforts. In this challenge, we spent over two months
preparing the entire dataset. Most of the time has been spent on preparing the
pixel-level validation and test sets, around one and a half months. Preparing
the patch-level training set only spent less than a week. Based on the
labeling experience in this challenge, we are now refining our pathologist-in-
the-loop process with more superior WSSS models to further accelerate the
labeling process and improve the segmentation precision.

There are still two limitations to this challenge. First, even we have invited
11 pathologists to prepare the labels coupling with the help of AI models.
Imperfect labels are still inevitable, especially pixel-level labels. As shown
in Fig. 7, some local results generated by the WSSS model even outperform the
ground truth labels. We also observe that the noises in the patch-level
dataset are far less than that in the pixel-level dataset. Because the
difficulties of these two label ways are totally not on the same level. The
above observations lead to another thinking that whether pixel-level labels
with more noises are better than patch-level labels with fewer noises. We may
follow this thinking and discuss it in our future works. Second, in this
challenge, we only selected three majority tissue classes. There are still a
lot of minority tissue classes in the WSIs with LUAD, such as necrosis,
vessels, bronchus and etc. Some might be also important but only account for
an extremely small part of the whole slide image. Further evaluations and
discussions should also be included in the future.

## Acknowledgments

This work was supported by the Key R&D Program of Guangdong Province, China
(No. 2021B0101420006), the National Key R&D Program of China (No.
2021YFF1201003), the National Science Fund for Distinguished Young Scholars
(No.81925023), the National Natural Science Foundation of China (No. 62102103,
81771912, 82071892, 82001789, 81901704, 81702322, 81472712, 81772918, 81972277
and 62002082), High-level Hospital Construction Project (No. DFJH201805 and
DFJH201914).

We would like to thank all everyone who contribute the success of WSSS4LUAD
challenge. Challenge program committee: Zaiyi Liu, Qingling Zhang, Huihua
Yang, Chunming Li, Jun Xu, Zhenbing Liu, Yan Xu, Xipeng Pan, Zhenwei Shi and
Chu Han. Challenge preparation and technical supports: Jiatai Lin, Bingchao
Zhao, Zhizhen Wang and Yumeng Wang.

We would like to express our sincere thanks to all the pathologists, including
the expert pathologist (Lixu Yan), two senior pathologists (Su Yao and
Shanshan Lv) and 8 junior labelers (Bingbing Li, Huan Lin, Zeyan Xu, Chao Zhu,
Yuan Zhang, Huihui Wang, Huasheng Yao, Jinhai Mai).

## References

  * [1] H. Sung, J. Ferlay, R. L. Siegel, M. Laversanne, I. Soerjomataram, A. Jemal, and F. Bray, “Global cancer statistics 2020: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” _CA: a cancer journal for clinicians_ , vol. 71, no. 3, pp. 209–249, 2021.
  * [2] A. A. Thai, B. J. Solomon, L. V. Sequist, J. F. Gainor, and R. S. Heist, “Lung cancer,” _the Lancet_ , vol. 398, no. 10299, pp. 535–554, 2021.
  * [3] D. Hanahan, “Hallmarks of cancer: New dimensions,” _Cancer Discovery_ , vol. 12, no. 1, pp. 31–46, 2022.
  * [4] R. M. Bremnes, T. Dønnem, S. Al-Saad, K. Al-Shibli, S. Andersen, R. Sirera, C. Camps, I. Marinez, and L.-T. Busund, “The role of tumor stroma in cancer progression and prognosis: emphasis on carcinoma-associated fibroblasts and non-small cell lung cancer,” _Journal of thoracic oncology_ , vol. 6, no. 1, pp. 209–217, 2011.
  * [5] K.-H. Yu, A. L. Beam, and I. S. Kohane, “Artificial intelligence in healthcare,” _Nature biomedical engineering_ , vol. 2, no. 10, pp. 719–731, 2018.
  * [6] C. L. Srinidhi, O. Ciga, and A. L. Martel, “Deep neural network models for computational histopathology: A survey,” _Medical Image Analysis_ , vol. 67, p. 101813, 2021.
  * [7] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [8] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [9] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” _arXiv preprint arXiv:1706.05587_ , 2017.
  * [10] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, pp. 1097–1105, 2012.
  * [11] S. Budd, E. C. Robinson, and B. Kainz, “A survey on active learning and human-in-the-loop deep learning for medical image analysis,” _Medical Image Analysis_ , vol. 71, p. 102062, 2021.
  * [12] A. A. Thai, B. J. Solomon, L. V. Sequist, J. F. Gainor, and R. S. Heist, “Lung cancer,” _The Lancet_ , vol. 398, no. 10299, pp. 535–554, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0140673621003123
  * [13] S. Wang, D. M. Yang, R. Rong, X. Zhan, J. Fujimoto, H. Liu, J. Minna, I. I. Wistuba, Y. Xie, and G. Xiao, “Artificial intelligence in lung cancer pathology image analysis,” _Cancers_ , vol. 11, no. 11, p. 1673, 2019.
  * [14] T. Sakamoto, T. Furukawa, K. Lami, H. H. N. Pham, W. Uegami, K. Kuroda, M. Kawai, H. Sakanashi, L. A. D. Cooper, A. Bychkov _et al._ , “A narrative review of digital pathology and artificial intelligence: focusing on lung cancer,” _Translational Lung Cancer Research_ , vol. 9, no. 5, p. 2255, 2020.
  * [15] K.-H. Yu, C. Zhang, G. J. Berry, R. B. Altman, C. Ré, D. L. Rubin, and M. Snyder, “Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features,” _Nature communications_ , vol. 7, no. 1, pp. 1–10, 2016.
  * [16] N. Coudray, P. S. Ocampo, T. Sakellaropoulos, N. Narula, M. Snuderl, D. Fenyö, A. L. Moreira, N. Razavian, and A. Tsirigos, “Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning,” _Nature medicine_ , vol. 24, no. 10, pp. 1559–1567, 2018.
  * [17] K. AbdulJabbar, S. E. A. Raza, R. Rosenthal, M. Jamal-Hanjani, S. Veeriah, A. Akarca, T. Lund, D. A. Moore, R. Salgado, M. Al Bakir _et al._ , “Geospatial immune variability illuminates differential evolution of lung adenocarcinoma,” _Nature Medicine_ , vol. 26, no. 7, pp. 1054–1062, 2020\.
  * [18] X. Zheng, A. Weigert, S. Reu, S. Guenther, S. Mansouri, B. Bassaly, S. Gattenlöhner, F. Grimminger, S. S. Pullamsetti, W. Seeger _et al._ , “Spatial density and distribution of tumor-associated macrophages predict survival in non-small cell lung carcinoma,” _Cancer research_ , vol. 80, no. 20, pp. 4414–4425, 2020.
  * [19] C. Lu, K. Bera, X. Wang, P. Prasanna, J. Xu, A. Janowczyk, N. Beig, M. Yang, P. Fu, J. Lewis _et al._ , “A prognostic model for overall survival of patients with early-stage non-small cell lung cancer: a multicentre, retrospective study,” _The Lancet Digital Health_ , vol. 2, no. 11, pp. e594–e606, 2020.
  * [20] S. Park, C.-Y. Ock, H. Kim, S. Pereira, S. Park, M. Ma, S. Choi, S. Kim, S. Shin, B. J. Aum, K. Paeng, D. Yoo, H. Cha, S. Park, K. J. Suh, H. A. Jung, S. H. Kim, Y. J. Kim, J.-M. Sun, J.-H. Chung, J. S. Ahn, M.-J. Ahn, J. S. Lee, K. Park, S. Y. Song, Y.-J. Bang, Y.-L. Choi, T. S. Mok, and S.-H. Lee, “Artificial intelligence–powered spatial analysis of tumor-infiltrating lymphocytes as complementary biomarker for immune checkpoint inhibition in non–small-cell lung cancer,” _Journal of Clinical Oncology_ , vol. 0, no. 0, p. JCO.21.02010, 2022.
  * [21] M. van Rijthoven, M. Balkenhol, K. Siliņa, J. van der Laak, and F. Ciompi, “Hooknet: Multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images,” _Medical Image Analysis_ , vol. 68, p. 101890, 2021.
  * [22] H. Chen, X. Qi, L. Yu, and P.-A. Heng, “Dcan: deep contour-aware networks for accurate gland segmentation,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2487–2496.
  * [23] B. Zhao, X. Chen, Z. Li, Z. Yu, S. Yao, L. Yan, Y. Wang, Z. Liu, C. Liang, and C. Han, “Triple u-net: Hematoxylin-aware nuclei segmentation with progressive dense feature aggregation,” _Medical Image Analysis_ , vol. 65, p. 101786, 2020.
  * [24] S. Graham, Q. D. Vu, S. E. A. Raza, A. Azam, Y. W. Tsang, J. T. Kwak, and N. Rajpoot, “Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images,” _Medical Image Analysis_ , vol. 58, p. 101563, 2019.
  * [25] M. Amgad, H. Elfandy, H. Hussein, L. A. Atteya, M. A. Elsebaie, L. S. Abo Elnasr, R. A. Sakr, H. S. Salem, A. F. Ismail, A. M. Saad _et al._ , “Structured crowdsourcing enables convolutional segmentation of histology images,” _Bioinformatics_ , vol. 35, no. 18, pp. 3461–3467, 2019.
  * [26] G. Nir, S. Hor, D. Karimi, L. Fazli, B. F. Skinnider, P. Tavassoli, D. Turbin, C. F. Villamil, G. Wang, R. S. Wilson _et al._ , “Automatic grading of prostate cancer in digitized histopathology images: Learning from multiple experts,” _Medical image analysis_ , vol. 50, pp. 167–180, 2018.
  * [27] J. Li, S. Yang, X. Huang, Q. Da, X. Yang, Z. Hu, Q. Duan, C. Wang, and H. Li, “Signet ring cell detection with a semi-supervised learning framework,” in _International conference on information processing in medical imaging_. Springer, 2019, pp. 842–854.
  * [28] Z. Li, J. Zhang, T. Tan, X. Teng, X. Sun, H. Zhao, L. Liu, Y. Xiao, B. Lee, Y. Li _et al._ , “Deep learning methods for lung cancer segmentation in whole-slide histopathology images—the acdc@ lunghp challenge 2019,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 25, no. 2, pp. 429–440, 2020.
  * [29] S. Wen, T. M. Kurc, L. Hou, J. H. Saltz, R. R. Gupta, R. Batiste, T. Zhao, V. Nguyen, D. Samaras, and W. Zhu, “Comparison of different classifiers with active learning to support quality control in nucleus segmentation in pathology images,” _AMIA Summits on Translational Science Proceedings_ , vol. 2018, p. 227, 2018.
  * [30] H. Shen, K. Tian, P. Dong, J. Zhang, K. Yan, S. Che, J. Yao, P. Luo, and X. Han, “Deep active learning for breast cancer segmentation on immunohistochemistry images,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 509–518.
  * [31] S. Doyle, J. Monaco, M. Feldman, J. Tomaszewski, and A. Madabhushi, “An active learning based classification strategy for the minority class problem: application to histopathology annotation,” _BMC Bioinformatics_ , vol. 12, no. 1, p. 424, Oct 2011.
  * [32] Z. Zhou, J. Y. Shin, S. R. Gurudu, M. B. Gotway, and J. Liang, “Active, continual fine tuning of convolutional neural networks for reducing annotation efforts,” _Medical Image Analysis_ , vol. 71, p. 101997, 2021\.
  * [33] S. Belharbi, I. Ben Ayed, L. McCaffrey, and E. Granger, “Deep active learning for joint classification & segmentation with weak annotator,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021, pp. 3338–3347.
  * [34] N. F. Greenwald, G. Miller, E. Moen, A. Kong, A. Kagel, T. Dougherty, C. C. Fullaway, B. J. McIntosh, K. X. Leow, M. S. Schwartz _et al._ , “Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning,” _Nature biotechnology_ , pp. 1–11, 2021.
  * [35] S. Shurrab and R. Duwairi, “Self-supervised learning methods and applications in medical imaging analysis: A survey,” _arXiv preprint arXiv:2109.08685_ , 2021.
  * [36] A. Van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv e-prints_ , pp. arXiv–1807, 2018\.
  * [37] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 9729–9738.
  * [38] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [39] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar _et al._ , “Bootstrap your own latent-a new approach to self-supervised learning,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 21 271–21 284, 2020.
  * [40] X. Wang, S. Yang, J. Zhang, M. Wang, J. Zhang, J. Huang, W. Yang, and X. Han, “Transpath: Transformer-based self-supervised learning for histopathological image classification,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2021, pp. 186–195.
  * [41] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” _arXiv preprint arXiv:2111.06377_ , 2021.
  * [42] Y. Li, J. Chen, X. Xie, K. Ma, and Y. Zheng, “Self-loop uncertainty: A novel pseudo-label for semi-supervised medical image segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 614–623.
  * [43] X. Zhuang, Y. Li, Y. Hu, K. Ma, Y. Yang, and Y. Zheng, “Self-supervised feature learning for 3d medical images by playing a rubik’s cube,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2019, pp. 420–428.
  * [44] J. Zhu, Y. Li, Y. Hu, K. Ma, S. K. Zhou, and Y. Zheng, “Rubik’s cube+: A self-supervised feature learning framework for 3d medical image analysis,” _Medical image analysis_ , vol. 64, p. 101746, 2020.
  * [45] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, “Context encoders: Feature learning by inpainting,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2536–2544.
  * [46] C. L. Srinidhi, S. W. Kim, F.-D. Chen, and A. L. Martel, “Self-supervised driven consistency training for annotation efficient histopathology image analysis,” _Medical Image Analysis_ , vol. 75, p. 102256, 2022.
  * [47] J. Boyd, M. Liashuha, E. Deutsch, N. Paragios, S. Christodoulidis, and M. Vakalopoulou, “Self-supervised representation learning using visual field expansion on digital pathology,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 639–647.
  * [48] J. Cheng, J. Mai, X. Pan, X. Chen, C. Han, Z. Liu, and C. Liang, “Curriculum self-supervised learning for weakly-supervised histopathological image segmentation,” in _2021 10th International Conference on Internet Computing for Science and Engineering_ , 2021, pp. 116–123.
  * [49] Y. Luo, Z. Chen, and X. Gao, “Self-distillation augmented masked autoencoders for histopathological image classification,” 2022.
  * [50] V. Cheplygina, M. de Bruijne, and J. P. Pluim, “Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis,” _Medical image analysis_ , vol. 54, pp. 280–296, 2019.
  * [51] J. N. Kather, A. T. Pearson, N. Halama, D. Jäger, J. Krause, S. H. Loosen, A. Marx, P. Boor, F. Tacke, U. P. Neumann, H. I. Grabsch, T. Yoshikawa, H. Brenner, J. Chang-Claude, M. Hoffmeister, C. Trautwein, and T. Luedde, “Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer,” _Nature Medicine_ , vol. 25, no. 7, pp. 1054–1056, Jul 2019.
  * [52] J. N. Kather, J. Krisam, P. Charoentong, T. Luedde, E. Herpel, C.-A. Weis, T. Gaiser, A. Marx, N. A. Valous, D. Ferber _et al._ , “Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study,” _PLoS medicine_ , vol. 16, no. 1, p. e1002730, 2019.
  * [53] J. Lin, G. Han, X. Pan, Z. Liu, H. Chen, D. Li, X. Jia, Z. Shi, Z. Wang, Y. Cui, H. Li, C. Liang, L. Liang, Y. Wang, and C. Han, “Pdbl: Improving histopathological tissue classification with plug-and-play pyramidal deep-broad learning,” _IEEE Transactions on Medical Imaging_ , pp. 1–1, 2022\.
  * [54] Z. Chai, L. Luo, H. Lin, H. Chen, and P.-A. Heng, “Deep semi-supervised metric learning with dual alignment for cervical cancer cell detection,” _arXiv preprint arXiv:2104.03265_ , 2021.
  * [55] Y. Xie, J. Zhang, Z. Liao, J. Verjans, C. Shen, and Y. Xia, “Pairwise relation learning for semi-supervised gland segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 417–427.
  * [56] N. Marini, S. Otálora, H. Müller, and M. Atzori, “Semi-supervised training of deep convolutional neural networks with heterogeneous data and few local annotations: An experiment on prostate histopathology image classification,” _Medical image analysis_ , vol. 73, p. 102165, 2021.
  * [57] Z. Jia, X. Huang, I. Eric, C. Chang, and Y. Xu, “Constrained deep weak supervision for histopathology image segmentation,” _IEEE transactions on medical imaging_ , vol. 36, no. 11, pp. 2376–2388, 2017.
  * [58] M. Lerousseau, M. Vakalopoulou, M. Classe, J. Adam, E. Battistella, A. Carré, T. Estienne, T. Henry, E. Deutsch, and N. Paragios, “Weakly supervised multiple instance learning histopathological tumor segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 470–479.
  * [59] H. Lee and W.-K. Jeong, “Scribble2label: Scribble-supervised cell segmentation via self-generating pseudo-labels with consistency,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 14–23.
  * [60] M. Jahanifar, N. Z. Tajeddin, N. A. Koohbanani, and N. M. Rajpoot, “Robust interactive semantic segmentation of pathology images with minimal user input,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 674–683.
  * [61] H. Qu, P. Wu, Q. Huang, J. Yi, G. M. Riedlinger, S. De, and D. N. Metaxas, “Weakly supervised deep nuclei segmentation using points annotation in histopathology images,” in _International Conference on Medical Imaging with Deep Learning_. PMLR, 2019, pp. 390–400.
  * [62] H. Qu, P. Wu, Q. Huang, J. Yi, Z. Yan, K. Li, G. M. Riedlinger, S. De, S. Zhang, and D. N. Metaxas, “Weakly supervised deep nuclei segmentation using partial points annotation in histopathology images,” _IEEE Transactions on Medical Imaging_ , 2020.
  * [63] N. A. Koohbanani, M. Jahanifar, N. Z. Tajadin, and N. Rajpoot, “Nuclick: a deep learning framework for interactive segmentation of microscopic images,” _Medical Image Analysis_ , vol. 65, p. 101771, 2020.
  * [64] Y. Lin, Z. Qu, H. Chen, Z. Gao, Y. Li, L. Xia, K. Ma, Y. Zheng, and K.-T. Cheng, “Label propagation for annotation-efficient nuclei segmentation from pathology images,” _arXiv preprint arXiv:2202.08195_ , 2022.
  * [65] H. Tokunaga, B. K. Iwana, Y. Teramoto, A. Yoshizawa, and R. Bise, “Negative pseudo labeling using class proportion for semantic segmentation in pathology,” in _European Conference on Computer Vision_. Springer, 2020, pp. 430–446.
  * [66] J. Zhang, Z. Hua, K. Yan, K. Tian, J. Yao, E. Liu, M. Liu, and X. Han, “Joint fully convolutional and graph convolutional networks for weakly-supervised segmentation of pathology images,” _Medical Image Analysis_ , vol. 73, p. 102183, 2021.
  * [67] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2921–2929.
  * [68] L. Chan, M. S. Hosseini, C. Rowsell, K. N. Plataniotis, and S. Damaskinos, “Histosegnet: Semantic segmentation of histological tissue type in whole slide images,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 10 662–10 671.
  * [69] C. Han, J. Lin, J. Mai, Y. Wang, Q. Zhang, B. Zhao, X. Chen, X. Pan, Z. Shi, X. Xu _et al._ , “Multi-layer pseudo-supervision for histopathology tissue semantic segmentation using patch-level classification labels,” _arXiv preprint arXiv:2110.08048_ , 2021.
  * [70] Z. Wu, C. Shen, and A. Van Den Hengel, “Wider or deeper: Revisiting the resnet model for visual recognition,” _Pattern Recognition_ , vol. 90, pp. 119–133, 2019.
  * [71] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2881–2890.