ROI short=ROI, long=region of interest, IOU short=IOU, long=intersection over
union, cIOU short=cIOU, long=circle intersection over union, DoF short=DoF,
long=degrees of freedom, CPL short=CPL, long=Center Point Localization,

# Glo-In-One: Holistic Glomerular Detection, Segmentation, and Lesion
Characterization with Large-scale Web Image Mining

Tianyuan Yao, Yuzhe Lu, Jun Long, Aadarsh Jha, Zheyu Zhu, Zuhayr Asad, Haichun
Yang, Agnes B. Fogo, Yuankai Huo T. Yao, Y. Lu, Z. Zhu, A. Jha, Z. Asad, Y.
Huo were with the Department of Computer Science, Vanderbilt University,
Nashville, TN 37235 USA.J. Long was with Big Data Institute, Central South
University, Changsha, 410083, China.A. B. Fogo and H. Yang were with the
Department of Pathology, Vanderbilt University Medical Center, Nashville, TN,
37215, USA.

###### Abstract

The quantitative detection, segmentation, and characterization of glomeruli
from high-resolution whole slide imaging (WSI) play essential roles in the
computer-assisted diagnosis and scientific research in digital renal
pathology. Historically, such comprehensive quantification requires extensive
programming skills in order to be able to handle heterogeneous and customized
computational tools. To bridge the gap of performing glomerular quantification
for non-technical users, we develop the Glo-In-One toolkit to achieve holistic
glomerular detection, segmentation, and characterization via a single line of
command. Additionally, we release a large-scale collection of 30,000 unlabeled
glomerular images to further facilitate the algorithmic development of self-
supervised deep learning. The inputs of the Glo-In-One toolkit are WSIs, while
the outputs are (1) WSI-level multi-class circle glomerular detection results
(which can be directly manipulated with ImageScope), (2) glomerular image
patches with segmentation masks, and (3) different lesion types. In the
current version, the fine-grained global glomerulosclerosis (GGS)
characterization is provided, including assessed-solidified (S-GGS, associated
with hypertension-related injury), disappearing (D-GGS, a further end result
of the SGGS becoming contiguous with fibrotic interstitium), and obsolescent
(O-GGS, nonspecific GGS increasing with aging) glomeruli. To leverage the
performance of the Glo-In-One toolkit, we introduce self-supervised deep
learning to glomerular quantification via large-scale web image mining. The
GGS fine-grained classification model achieved a decent performance compared
with baseline supervised methods while only using 10% of the annotated data.
The glomerular detection achieved an average precision of 0.627 with circle
representations, while the glomerular segmentation achieved a 0.955 patch-wise
Dice Similarity Coefficient (DSC). In this paper, we develop and release an
open-source Glo-In-One toolkit, a software with holistic glomerular detection,
segmentation, and lesion characterization. This toolkit is user-friendly to
non-technical users via a single line of command.

The toolbox and the 30,000 web mined glomerular images have been made publicly
available at https://github.com/hrlblab/Glo-In-One.

###### Index Terms:

open-source, renal pathology, glomeular detection, lesion characterization,
self-supervised learning

*Corresponding Author: Yuankai Huo<EMAIL_ADDRESS>
##  I Introduction

Figure 1: This figure presents the overview of the Glo-In-One toolkit. The
inputs are raw WSIs, while the outputs are quantitative detection,
classification, and segmentation results. Moreover, the results can be
manipulated using the prevalent Aperio ImageScope software. The domain impact
is to enable comprehensive glomerular quantification via one single command
for non-technical users. The technical innovation is to introduce large-scale
self-supervised deep learning via web image mining.

Glomeruli are central tufts of looped capillaries located in the center of the
renal corpuscle [1, 2], which delivers blood and creates a large surface area
optimized for renal filtration [3, 4]. Identiﬁcation and characterization of
glomeruli are essential for computer-assisted pathological diagnosis in
digital renal pathology [5, 6]. However, such procedures are typically labor
and resource intensive. The recent advances in whole slide imaging (WSI) and
deep learning have led to an unprecedented opportunity to characterize
glomerular lesions in a large-scale and fully automatic manner [7, 8, 9, 10].
Typical high-resolution WSI images are in the order of gigapixels and are
usually stored in multi-resolution pyramidal format [11].

Historically, comprehensive computer-assisted glomerular quantification
requires extensive programming skills to handle the heterogeneous and custom
computational tools [12, 13, 14, 15, 16] . To bridge the gap of performing
glomerular quantification for non-technical users, we develop the Glo-In-One
toolkit to achieve holistic glomerular detection, segmentation, and
characterization via a single line of command (Fig. 1).

For the glomerular detection [17, 18, 19, 20], we apply an anchor-free
detection method with a circle representation [21]. The simple circle
representation is optimized for ball-shaped biomedical object detection with
smaller degrees of freedom (DoF) in fitting and superior detection performance
which is optimized for glomerular detection in renal pathology. Then, the Glo-
In-One toolkit further segments and classifies each detected glomerulus with a
pixel-level mask and fine-grained Global glomerulosclerosis (GGS)
classification. GGS is a prevalent pathological phenotype for patients with
both normal aging and kidney disease [22, 23, 24]. Fine-grained
characterization of glomerular lesions [25], is essential in nephropathology
in order to support both scientific research and clinical decision making[26].

Lesion characterization involves the development of quantitative machine
learning approaches (e.g., self-supervised deep learning) that characterize
glomerular lesions typically require large-scale heterogeneous images [16, 27,
28], which is resource extensive for individual labs. We first assess the
feasibility of leveraging fine-grained GGS characterization via large-scale
web image mining (e.g., from journals, search engines, websites) and self-
supervised deep learning. Three types of GGS are assessed-solidified (S-GGS,
associated with hypertension-related injury), disappearing (D-GGS, a further
end result of the SGGS becoming contiguous with fibrotic interstitium), and
obsolescent (O-GGS, nonspecific GGS increasing with aging). We employ a
SimSiam [29] network as the baseline method of self-supervised contrastive
learning. By deploying our previously developed compound figure separation
approach [30], we provide 30,000 unannotated glomerular images via web image
mining to train the SimSiam network. From the results, the GGS fine-grained
classification model achieved superior performance as compared with the
baseline methods. We release such large-scale collection of unlabeled
glomerular images to further facilitate the methodological development of
self-supervised deep learning for the engineering community.

Our contribution is four fold:

  * •

We develop and release an open-source Glo-In-One toolkit: a containerized
software with holistic glomerular detection, segmentation, and lesion
characterization. The toolkit is user friendly to non-technical users via a
single line of command, with standard WSI files as inputs.

  * •

We assess the feasibility of mining large-scale web images for fine-grained
GGS classification via contrastive learning.

  * •

We release a large-scale web-mined glomerular dataset with 30,000 unannotated
glomeruli to facilitate the methodological development of self-supervised deep
learning.

  * •

The toolbox and the 30,000 web mined glomerular images have been made publicly
available at https://github.com/hrlblab/Glo-In-One

##  II Related Works

The success of deep convolutional neural networks (CNNs) in recent years has
spurred extensive research on their application in renal pathology [31, 32,
33]. The innovations can be summarized as glomerular detection, segmentation,
and characterization, which have further advanced digital diagnostic imaging,
especially in WSIs technology [34, 35, 36].

###  II-A Glomerulosclerosis Classification

Many studies have been conducted to classify different glomerular lesions
using computer-aided approaches [37, 8, 38, 39]. Specifically, Uchino et al.
[40] trained deep learning algorithms to classify several renal pathological
findings and achieved an AUC score of 0.983 for global sclerosis
characterization. However, most algorithms are tested using in-house data and
thus, their performances can be highly volatile in the face of a distribution
shift [41]. Moreover, there are few, if any, studies that have developed deep
learning approaches to classify globally sclerotic glomeruli into the
following three fine-grained categories: obsolescent, solidified, and
disappearing glomerulosclerosis. Such fine-grained characterization is
challenging as the available data is rare and their distribution is highly
imbalanced. For instance, the presence of obsolescent glomerulosclerosis is
naturally much higher than solidified or disappearing glomerulosclerosis [42,
43], leading to the technical difficulty that is well known as the imbalanced
classification problem [44, 45, 46].

####  II-A1 Self-supervised learning method

Self-supervised learning represents a family of learning algorithms that could
learn the hidden regularities from using data without manual annotations
(e.g., unannotated renal pathological images) [47]. Such technology might
alleviate the bottleneck of the scale of human annotation in healthcare as
large-scale unannotated data could be available from clinical database.
Moreover, the generalizability of the AI models on unseen testing data would
be further improved by learning from a larger scale of training data, even
without human annotations [48].

Recently, a new family of self-supervised representation learning, known as
contrastive learning, has shown its superior performance in various computer
vision tasks [49, 50, 51, 52]. Using an approach where learning is achieved
from large-scale unlabeled data, contrastive learning can learn discriminative
features for downstream tasks. SimCLR [53] maximized the similarity between
images in the same category and repelled the representations of different
category images. The ‘similarity’ is typically defined as the cosine
similarity between features [54] in the previous prior arts. Wu et al. [49]
used an offline dictionary to store all data representation and randomly
selected training data to maximize negative pairs. MoCo [55] introduced a
momentum design to maintain a negative sample pool instead of an offline
dictionary. Such works demanded a large batch size in order to include
sufficient negative samples. To eliminate the needs of negative samples, BYOL
[56] was proposed to train a model with an asynchronous momentum encoder.
Recently, SimSiam [29] was proposed to further eliminate the momentum encoder
in BYOL, allowing for less GPU memory consumption.

###  II-B Glomerular Detection

The introduction of WSIs demonstrates a paradigm shift of computer-aided
diagnosis (CAD) from visual inspection to a more accurate quantitative
assessment. CAD, especially with recent advances in deep learning, offers an
unparalleled ability to efficiently manage patients, accelerate diagnosis, and
guide treatments in general health care. In renal pathology, the promising
results of integrating CAD in chronic kidney diseases (CKD) and kidney
transplantation have been shown from prior arts [57, 16], especially with
automatic glomerular characterization [37, 58]. In renal pathology, properly
distinguishing and characterizing different glomeruli from renal tissues is a
key task in both corresponding scientific research and clinical practice. In
recent years, such historically labor-intensive procedures have been advanced
by modern deep learning techniques, via classification, detection, and
segmentation approaches [59]. Several studies have shown the great accuracy by
which CNNs are able to properly detect and localize glomeruli within WSIs.
Similarly, CNNs have also been able to accurately segment glomeruli. Beyond
identifying the regional and pixel location of glomeruli, deep learning
algorithms have also been applied in CAD (e.g., pathological glomerular
characterization), such as quantifying and characterizing lesion sub-types of
glomeruli. For example, Gallego et al. [9] used CNN-based classification,
while Gloria et al. [60] utilized semantic segmentation to achieve glomeruli
detection in WSIs. More recently, Yang et al. proposed an anchor-free
detection strategy using circle representation [21] that is optimized for
round glomeruli and demonstrates superior performance.

###  II-C Glomerular Segmentation

An initial step for renal tissue assessment is the differentiation and
segmentation of relevant tissue structures in kidney specimens as it is
performing pixel-level quantification on the image [61, 32, 62]. However,
Segmentation on WSIs is challenging since the pixel-level annotation is
computational extensive on the gigapixel high-resolution images [63, 64].
Bouteldja et al investigated the concept of active learning for accurate
segmentation accuracy [65] by performing a large number of 72,722 expert-based
annotations, while Gadermayr et al has proposed a weakly supervised pipeline
for segmenting renal glomeruli [66]. Other method, to achieve instance object
level segmentation, has integrated fully convolutional networks with detection
methods on WSIs(e.g., Mask-RCNN [67]), but there is still room to improve [68,
69].Recently, Aadarsh et al. [70] shows that the ”detect-then-segment” two-
stage segmentation approach yields more accurate results than the conventional
single-stage instance segmentation tactics. Following this study, we propose a
detect-classify-segment pipeline to achieve even more accurate glomerular
segmentation results.

##  III Method

Generally, the glomerular quantification in this study can be summarized into
three major steps: (1) detection, (2) characterization, and (3) segmentation
(Fig. 2 and 3). First, CircleNet [21, 71] is employed to perform automatic
glomerular detection. After the detection, we extract image patches that
contain the detected glomeruli and add padding (50 pixels) around the patches.
Then, all the patches are cropped and resized to 256$\times$256-pixel for
downstream tasks using bilinear interpolation, while the information of the
bounding box and actual size (0.25 $\mu$m per pixel) is saved in an XML file.
For the task of classification, our self-supervised contrastive learning
approach uses minimal human efforts (e.g., self-supervised learning) to
achieve fine-grained GGS classification via large-scale web image mining. The
procedure consists of two parts: (1) web data mining from biomedical databases
via compound figure separation, and (2) self-supervised learning with
downstream classification tasks. For the step of segmentation, we apply the
DeepLab_v3 model to perform the segmentation and generate binary masks.

Figure 2: The self-supervised glomerular characterization pipeline is
presented. Our approach utilizes the large-scale unannotated web-mined
glomerular images via compound image separation to pretrain a contrastive
learning model. Then, a smaller amount of labeled data, especially for rare
GGS samples, are employed to finetune the pretrained model. The goal is to
improve the glomerular classification via large-scale unannotated web-mined
images and small-scale annotated in-house data. Figure 3: (a) The detection
framework with circle representation [21] is presented, which is an optimized
object detection method for glomeruli. The heatmap and local offset head
determines the center point of the circle while the circle radius head
determines the radius of the circle. (b) The DeepLab_v3 backbone is employed
in the ”detect-then-segment” pipeline. Figure 4: This figure shows fine-
grained classes of glomerular images that are stained with PAS. (a) Normal
glomeruli, (b) non-glomerular and (c)-(e) sclerosed glomeruli images. Three
types of sclerosed glomeruli are (c) assessed-solidified (S-GGS, associated
with hypertension-related injury), (d) disappearing (D-GGS, a further end
result of the SGGS becoming contiguous with fibrotic interstitium), and (e)
obsolescent (O-GGS, nonspecific GGS increasing with aging) glomeruli.

###  III-A Glomerular Classification

In the Glo-In-One toolkit, we performed a fine-grained GGS classification as
the application (Fig. 2 and 4). First, all glomeruli images that were detected
by Glo-In-One (details are provided the in the next section) were further
refined as glomerular vs. non-glomerular images. Second, all confirmed
glomeruli images were further classified into GGS vs. non-GGS images. Third,
GGS images were characterized into fine-grained categories: (1) solidified
(S-GGS, associated with hypertension-related injury), (2) disappearing (D-GGS,
an end result of the S-GGS becoming contiguous with fibrotic interstitium),
and (3) obsolescent (O-GGS, nonspecific GGS increasing with aging) glomeruli.
The technical innovation of our method was introducing the contrastive
learning as a self-supervised pretraining stage to enhance the glomerular
classification performance.

####  III-A1 Web Image Mining

We propose to achieve large-scale unannotated glomerular images from online
resources (e.g., open-access journals, NIH Open-i® [72] database, and search
engines) in order to facilitate self-supervised contrastive learning (Fig. 2).
Briefly, we collect 10,000 compound figures with the keywords “glomerular OR
glomeruli OR glomerulus” through the NIH Open-i® search engine. The details of
web image mining are provided in [30].

However, the images from online resources are typically in compound figures
(with multiple subplots), which cannot be directly used for self-supervised
learning. Thus, we employ our previously developed compound image separation
approach [30] to detect, separate, and curate subplots to individual images
for downstream learning tasks. Using the compound figure separation approach,
we acquired over 30,000 unannotated glomerular images via large web image
mining.

####  III-A2 Self-supervised Learning

As opposed to traditional supervised learning methods, self-supervised
learning [48] refers to inferring underlying patterns from an unlabeled
dataset without any reference to labeled outcomes or predictions (Fig. 2). We
employ the SimSiam network [53] according to our previous pathological image
analytical study using multiple contrastive learning methods [73]. The SimSiam
network can be interpreted as an iterative process of two steps: (1)
unsupervised clustering and (2) feature updates based on clustering (similar
to K-means or EM algorithms) [29]. After training the SimSiam with large-scale
mined images, a pretrained ResNet backbone is available as a so called feature
encoder. Then, the output of the encoder is passed through a predictor which
is again a shallow fully-connected network for the final classification
results.

###  III-B Glomerular Detection

The glomerular detection function is implemented based on our previously
proposed CircleNet method [21, 71], which introduces a new bounding circle
representations for glomerular detection (Fig. 3). Since CircleNet achieved
superior performance for glomerular detection compared with current benchmarks
[21, 71], we directly migrate CircleNet as the detection method in this study.
The detection outcomes are saved in one XML file that contains the circle
location, type, and the detection score for each detected object. The
detection score is a score within 0 to 1, where a larger score indicates the
stronger confidence to believe the detected object is a glomerulus. The XML
format file can be loaded with pathology slide viewing software such as Aperio
ImageScope.

###  III-C Glomerular Segmentation

The glomerular segmentation function is implemented based on our previously
proposed “Detect-then-Segment” method [70] (Fig. 3). In segmentation, instead
of directly apply segmentation approaches on lower resolution image patches,
our “Detect-then-Segment” two-stage design [70] applies DeepLab_v3 [74] on the
detected high-resolution glomerular patches with minized information loss.
Such design has been demonstrated to outperform the prevelent single-stage
approaches [70]. The DeepLab_v3 [74] is employed as the segmentation backbone.
DeepLab_v3 utilizes atrous convolution which allows it to enlarge the field of
view of filters to incorporate larger context. It thus offers an efficient
mechanism to balance accurate localization (small field-of-view) and context
assimilation (large field-of-view).

##  IV Experiments and Results

###  IV-A Glomerular Classification

####  IV-A1 Data

Model Pretrain.In this study, we collected over 10,000 compound figures (each
figure might contain multiple subplots) through the NIH Open-I® search engine
with the keywords “glomerular OR glomeruli OR glomerulus”. Then, our compound
figure separation method [30] was employed to separate compound images into
individual images, which were further categorized to different modalities
(e.g., light microscopy, florescent microscopy, and electron microscopy) as
well as different stain types within the light microscopy. To curate all
images, an automatic deep learning-based curator (detector) was trained using
only a smaller scale annotated images dataset [75]. Then, the curator was run
on all web mined images in order to drop the predicted glomerular figures
under a detection score of 0.7. The 0.7 threshold was determined empirically
by visual inspection. Eventually, more than 30,000 glomerular images
(subplots) passed the curation. The number and size of the image patches used
for different learning strategies are described in Experimental Design.

Table I: Distribution of subclasses in glomerular classifier

Type | Quantity | Percentage | Total
---|---|---|---
Normal | 3,617 | 16% |
obsolescent glomeruli | 6,647 | 30% |
solidified glomerul | 735 | 3% |
disappearing glomerul | 459 | 2% |
Non-glomerular | 10,619 | 49% | 22,077

*As mentioned, the collected dataset is naturally unbalanced.

Cross Validation. The classification models were trained on 22,077 images
extracted from WSIs using the EasierPath semi-manual annotation software [76].
The WSIs were acquired from 157 patients, whose tissues were routinely
processed and paraffin-embedded, with 3 $\mu$m thickness sections cut and
stained with Periodic acid–Schiff (PAS). The WSIs were acquired at 40× with
0.25 $\mu$m per pixel.

All glomeruli were manually annotated with 5 classes, including 3,617 normal
glomeruli, 6,647 globally obsolescent glomeruli, 735 global solidified
glomeruli, and 459 global disappearing glomeruli, and 10,619 non-glomerular
images (Table I). The images were resized to 224$\times$224-pixel for
training, validation, and testing. The data were de-identified, and studies
were approved by the Institutional Review Board (IRB) at VUMC.

Additionally, 120 glomerular patches were derived from five WSIs of the kidney
tissue which respectively from different patients and were utilized as
detection test set in our automatic detection experimentation. The details of
the data selection, acquisition and clinical background were provided in [71].

For the segmentation experiment, we formed a cohort with 704 training, 98
validation and 147 internal testing images images which respectively extracted
from 42, 7 and 7 WSIs using our detection method. Meanwhile, a group of 385
glomerulus pathologies from 5 external biopsy samples was used as external
testing data. Both GGS and none-GGS glomerulus patches were manually annotated
and the segmentation masks were traced by a experienced pathologist.

External Validation. In addition to our in-house datasets, a publicly
available glomeruli dataset [77] was also employed as an external validation
cohort. The dataset consisted of 2,340 images containing a single glomerulus.
Among them, 1,170 glomeruli were normal, while 1,170 glomeruli were sclerosed.

To further evaluate our method, the model was trained on both 5-classes and
2-classes (binary) designs. In the binary setting, (1) three types of
sclerosed glomeruli (S-GGS, D-GGS, O-GGS) were merged as a single positive
class; (2) the normal type was kept as the negative class; (3) 10,619 non-
glomerular images were discarded to ensure consistency with the definition in
[77].

####  IV-A2 Experimental Design

Fig 5 shows a precise presentation for separate experiments.

All the models were trained and tested with five-fold cross-validation [78],
where each fold was withheld as the testing data once [79]. The remaining data
for each fold was split as 75% training data and 25% validation data.
Furthermore, to avoid data contamination, all glomeruli from the same patient
were used either for training, validation, or testing. We employed a canonical
design to prevent the data from leaking since the testing data were withheld
from the training and validation data.

In our study, we used the same hyperparameters (initial learning rate, batch
size, and maximum number of epoch) and data augmentation strategies across all
folds [80, 81]. The batch size was determined as the maximum GPU memory
limitation, while the initial learning rate (lr = 0.0001) was determined
empirically by all validation data. Since the adaptive moment estimation
(Adam) optimizer was used, even with the same initial learning rate, the
underlying real learning rate was adjusted “fold-by-fold” as an adaptive
optimization strategy. Via the five-fold cross-validation [82], the optimal
epoch (model selection) of each fold was determined based on the validation
performance as a “fold-by-fold” manner.

Figure 5: Classification experiment setting: (a) We evaluate different methods
via the five-fold cross-validation using the in-house GGS dataset (multi-
class). (b) An independent public dataset (binary) is employed to evaluate the
trained models directly as an external validation. (c) The external validation
is performed by retraining a binary classifier using different methods via the
in-house GSS dataset (binary).

We adapted the SimSiam network [53] as the baseline method of contrastive
learning. Two random augmentations from the same image were used as training
data. In self-supervised pretraining, 30,000 glomerulus pathologies from web
mining were resized to $224\times 224$ pixels for model training. For our web-
mined glomerular dataset, 70% of the images’ length-width ratio are within 4:3
or 3:4. Moreover, the random affine transformation was employed in the data
augmentation, so that the images were resized to different aspect ratios,
rotated with random degrees, and even with shearing before the downstream
learning procedures. Therefore, we simply resized all images to $224\times
224$ resolution images. This strategy has been widely used for large-scale
image analysis in prior arts [83, 84, 85, 86]. Two particular augmentation
transforms-random resized crops and color distortions-are applied. In the
classifier evaluation, only resizing to $224\times 224$-pixel is applied in
the augmentation. We used the momentum SGD as the optimizer. The Weight decay
was set to 0.0001. The base learning rate was $lr=0.05$ and the batch size was
64. The learning rate was $lr\times$BatchSize$/256$, which followed a cosine
decay schedule [87].

We used ResNet-50 as the backbone in supervised training. The model was then
trained using the Adam optimizer with a base learning rate of 1e-4. The
optimizer learning rate followed (linear scaling [88])
$lr\times$BatchSize$/256$. We used the focal loss [89] with $\gamma=2.5$ for
unbalanced classes. Each fold was trained for 70 epochs, and the epoch with
the best balanced accuracy score on the validation set was used for testing.

To fully estimate the skill of the machine learning model on unseen data, all
the models were trained and tested with five-fold cross-validation, where each
fold was withheld as testing data once. The remaining data for each fold was
split as 75% training data and 25% validation data. Furthermore, to avoid data
contamination, all glomeruli from the same patient were used either for
training, validation, or testing.

To apply the self-supervised pre-training networks, we froze our pretrained
ResNet-50 model by adding one extra linear layer which followed the global
average pooling layer. When finetuning with the manually annotated glomeruli
data, only the extra linear layer was trained. We used focal loss and the Adam
optimizer to train the linear classifier with based (initial) learning rate of
$lr$=30, weight decay=0, momentum=0.9, and batch size=64 (follows [29]). To
evaluate the impact of training data size, we fine-tuned the classifier of our
self-supervised learning model and the ImageNet pretrained model on 1, 5, 10,
25 and 100 percentages of annotated training data.

All experiments are performed on the same workstation, with a NVIDIA Quadro
P5000 GPU of 16GB memory.

####  IV-A3 Results

Cross Validation on In-house Dataset. As shown in Table II, fine-tuning our
pretrained SimSiam (Backbone:ResNet-50) on the same size of dataset achieved
superior results as compared to training from scratch. Interestingly, our
model also outperformed ResNet-50 models that were pretrained on ImageNet.
Even Using 10% of all available labeled data, our self-supervised ResNet50
model achieved the comparable performance with the fully supervised ResNet-50
models that used the entire labeled cohort. Moreover, the computational
efficiency was also evaluated. Since contrastive learning only requires
finetuning on the linear layers, it yielded less GPU memory (larger batch
size) and higher computational efficiency (Table II). Moreover, our pretrained
model also boosted accuracy upon the ImageNet pretrained ResNet-50.

The confusion matrices are shown in Fig. 6. Our model achieved comparable
performance as compared to the fully supervised models, while only using 10%
labeled data. Note that our approach had superior performance on the two
categories (solidified and disappearing) which have smaller numbers of cases.

*Supervised method refers to training a ResNet-50 from scratch

Figure 6: Comparison between supervised method and our method  Table II:
Performance of fine-grained GGS classification

Model | Unlabeled data | Labeled data | Balance acc | F1 | Time (s)
---|---|---|---|---|---
ResNet-50 (train from scratch) | 0 | 100% | 65.0 | 61.7 | 159.8
ImageNet (finetune linear layers) | 0 | 100% | 62.0 | 51.9 | 43.7
25% | 59.6 | 50.5 | 36.6
10% | 59.2 | 48.4 | 29.1
5% | 58.0 | 46.1 | 28.7
1% | 52.3 | 42.1 | 28.2
SimSiam (finetune linear layers) | 30k | 100% | 68.0 | 62.5 | 42.1
25% | 66.8 | 60.8 | 35.2
10% | 65.1 | 59.7 | 33.2
5% | 64.4 | 54.8 | 29.8
1% | 59.2 | 52.8 | 27.8

*ResNet-50: we trained the entire ResNet-50 from scratch with fully supervised learning.
*ImageNet: we only finetuned the linear layers of an ImageNet pretrained ResNet-50 model.
*SimSiam: we only finetuned the linear layers of a SimSiam pretrained ResNet-50 model.
*Time: indicates the average computational time per epoch.

External Validation on Public Dataset. For external validation, as shown in
Table III, trained with 5-classes labeled data, our models showed superior
performance with a domain shift. The contrastive learning model achieved an
AUC score of 94.5 when testing directly on the external dataset without any
finetuning. Even the SimSiam model that previously finetuned on 10$\%$ of all
available labeled data yielded better accuracy than the fully supervised
ResNet-50 models.

In the binary setting (Table. LABEL:table:performance3), the results yielded a
similar trend when compared to the results from the 5-classes setting.
Briefly, using only 10% the labeled images, our model outperformed supervised
training models that used all labeled images.

Note that finetuning the ImageNet pretrained model outperformed the training-
from-scratch model in external validation, which is contrary to the results in
cross validation. Such phenomenon might be caused by the domain shift in
external validation. However, our method consistently outperformed those
methods in different scenarios.

Table III: Performance on external validation

(a) Train on 5-classes dataset

Methods | Labeled data | AUC
---|---|---
ResNet-50 (train from scratch) | 100% | 91.3
ImageNet (finetune linear layers) | 100% | 92.4
SimSiam (finetune linear layers) | 100% | 94.5
25% | 93.3
10% | 92.6
5% | 92.2
1% | 89.9

|  |

(b) Train on binary dataset

Methods | Labeled data | AUC
---|---|---
ResNet-50 (train from scratch) | 100% | 93.3
ImageNet (finetune linear layers) | 100% | 93.6
SimSiam (finetune linear layers) | 100% | 95.2
25% | 94.8
10% | 94.2
5% | 92.2
1% | 90.7

*Definitions please see Table II

###  IV-B Glomerular Detection

Figure 7: This figure presents the WSI-level multi-class circle glomerular
detection results, which are visualized by ImageScope.

####  IV-B1 Experimental Design

We employed the CircleNet pretrained model [21, 76] as the baseline model. In
this study, we integrated the glomerular detection and classification as a
unified framework to enhance the performance of object detection. Briefly, we
introduced the “non-glomerlar” class in the self-supervised
$\mathsection$Glomerular Classification as a post-processing module to further
filter out the false positive detection results. We report AP (average
precision), AP50 and AP75 (average precision evaluated at 0.5 and 0.75 IoU
threshold respectively), AP over small (APS), and medium (APM) objects, which
shows the result of CircleNet detection [90].

####  IV-B2 Results

The results of glomerular detection was presented in Table. IV and Fig. 7. By
introducing the classifier as a false-positive filter, the detection
performance in our Glo-In-One method achieved superior detection performance
compared with original CircleNet method.

###  IV-C Glomerular Segmentation

####  IV-C1 Experimental Design

We conducted analyses on two of the widely used segmentation backbones (U-Net
and DeepLab_v3) on two unique resolutions (512$\times$512, 256$\times$256).
Since the images from the previous stage are 256$\times$256 in the pipeline,
we did experiments respectively on bilinear interpolation resizing and
constant padding strategies. Mean and standard deviation of Dice similarity
coefficients (DSC) were the primary statistics used to evaluate segmentation
performance. The hyperparameters for training the U-Net and DeepLab_v3
pipelines were 150 epochs, a batch size of 4 and a learning rate of 1e-4. An
Adam Optimizer was used to adaptively alter the learning rate, with beta
values ranging from 0.9 to 0.999.

####  IV-C2 Results

From the results (Table. IV), our pipeline yielded decent DSC values across
two backbone methods at different training scenarios, which is comparable to
the prior arts [70]. Among the benchmark methods, the DeepLab_v3 with 512×512
image resolution is preferred.

Fig. 8 presents the segmentation performance at the individual WSIs level. The
results show that the model achieves consistent performance across different
slides.

(c) Detection Results

Method | $AP$ | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$
---|---|---|---|---|---
CircleNet | 0.621 | 0.920 | 0.588 | 0.554 | 0.749
Ours | 0.627 | 0.951 | 0.604 | 0.573 | 0.758

*Ours: we used the CircleNet_v2 model
with our second-stage classifier.

(d) Segmentation Results

Model | 512$\times 512^{a}$ | 512$\times 512^{b}$ | 256$\times$256
---|---|---|---
U-Net | 0.934 ± 0.062 | 0.940 ± 0.029 | 0.947 ± 0.037
DeepLab_v3 | 0.955 ± 0.045 | 0.950 ± 0.044 | 0.945 ± 0.047

*512$\times 512^{a}$: constant padding from 256$\times$256
*512$\times 512^{b}$: bilinear resized from 256$\times$256
*256$\times 256$: original resolution

Table IV: (a) For Detection, we report AP (average precision), AP50 and AP75
(average precision evaluated at 0.5 and 0.75 IoU threshold respectively), AP
over small (APS), and medium (APM) objects (b) For segmentation, we report
mean and standard deviation of Dice similarity coefficients (DSC) to evaluate
the two backbone methods. Figure 8: This figure presents the segmentation
performance for glomerulus on external WSIs respectively. The bar plots show
the mean values of Dice Similarity Coefficients (DSC) while the error bars
represent the standard deviation.

##  V Discussion

In this study, we introduced Glo-In-One, an integrated glomerular
quantification open-source software toolkit for glomerular detection,
segmentation, and lesion characterization. Only a single command line was
required to perform all quantification tasks as a operating system agnostic
Docker implementation. In this multitask system, we combined several well
validated deep learning models and new self-supervised learning algorithms to
enhance the reliability of the AI system. The proposed method would
potentially be applied to the kidney transplant patients to achieve a more
comprehensive pre-/post-operational evaluation with glomerulosclerosis
characterization across all glomeruli. Moreover, the web image mining-based
learning strategy might improve the diagnosis of rare pathological phenotypes
by integrating the constantly updated knowledge from the community (e.g., from
shared case studies on social media, new scientific publications, and new
books).

Beyond releasing a holistic glomerular quantification tool, the core technical
innovation was the self-supervised glomerular classification section, where we
introduced the contrastive learning paradigm of fine grained GSS
characterization via large-scale web image mining.The major focus of our
method is to assess the feasibility and efficacy of incorporating the large-
scale web mining data into computer-assisted nephropathological diagnosis to
(1) improve the accuracy of the learning-based glomerulosclerosis
characterization on unbalanced categories, and (2) enhance the
generalizability of the AI models on unseen testing data.

This approach allows neural networks to learn more with fewer labels, smaller
samples, or fewer trials. By taking advantages of these points, we come up
with the idea of combining self-supervised learning and web image mining.
Advanced detection algorithms and professional biomedical image database have
disabused our worries about quality control when massively mining web images
as training data. One promising future direction is to integrate self-
supervised learning methods with web image mining in a poor annotation
situation (in which only WSI-wise weak labels are provided) since our method
does not require exhaustive tuning of hyperparameters and needs a smaller
number of annotations.

In the classification experiment, the hyper-parameter tunning strategy in a
cross-validation study is typically a “variance-bias” tradeoff. For example,
as shown in the comprehensive study [82], cross-validation without hyper-
parameter tunning yielded a lower variance but a higher bias compared with
cross-validation with fold-by-fold hyper-parameter tunning. Therefore, we
further conducted external validations (using an independent external cohort)
to alleviate such concerns in cross-validation based evaluation, as a more
rigor assessment.

From the results, the proposed strategy achieved superior performance in the
task of GGS fine-grained classification, with fewer human efforts. By
deploying our previously developed compound figure separation approach, we
provided 30,000 web-mined unannotated glomerular images to pretrain the ResNet
classification network via SimSiam contrastive learning. From the results, the
GGS fine-grained classification model achieved superior performance when
compared with baseline methods on both cross validation and external
validation. However, the proposed method is still limited in its performance
gain (mostly around 1%). The recent advances in self-supervised learning have
drawn significant attention towards the promising data efficiency and
generalization ability. This approach assesses the feasibility of adapting
such learning strategies to digital pathology with smaller scale annotated
data and larger scale web mined data. One promising future direction is to
further extend the proposed learning framework to the weakly supervised
learning scenarios (in which only WSI-wise weak labels are provided).

Several limitations and potential improvements for our work are as follows.
First, integrating with a 3D registration algorithm would be an avenue for
improvement for renal pathology as sequential cutting is typically involved
[91]. With 3D quantification, we can potentially get more precise
quantification of glomeruli, such as the average volume that represents the
patients’ degree of kidney sclerosis. Second, integrating interactive user
quality assurance (QA) interface might enable more precise results with a
“human-in-the-loop” design. Third, from the results, the methods are limited
by around 50% of accuracy on GGS sub-classes classification, another potential
improvement in terms of the performance could be to build up a more class-
balanced fine grained GGS dataset. The binary classification cohort was used
as our external validation data. Another external validation cohort with
multiple GSS types would lead to a more comprehensive external validation.
However, to our best, such a dataset is not publicly available to further exam
the generalizability of our multi-class model.

##  VI Conclusions

In this paper, we develop and release a holistic Glo-In-One open-source
toolkit to provide holistic glomerular detection, segmentation, and lesion
characterization. The containerized machine learning system process gigapixel
renal WSIs in a fully automated manner with a single command line, which is
user-friendly for non-technical users. Our technical contribution focuses on
combining cost-efficient and large-scale web image mining with self-supervised
algorithms to achieve superior performance in glomerular detection and fine
grained GSS classification. The results demonstrate that our self-supervised
ResNet50 achieves superior performance as compared with a standard supervised
ResNet-50 using all labeled images.

## Acknowledgment

This work was supported by NIH NIDDK DK56942(ABF).

## References

  * [1] W. E. Hoy, R. N. Douglas-Denton, M. D. Hughson, A. Cass, K. Johnson, and J. F. Bertram, “A stereological study of glomerular number and volume: preliminary findings in a multiracial study of kidneys at autopsy,” _Kidney International_ , vol. 63, pp. S31–S37, 2003.
  * [2] P. Kimmelstiel and C. Wilson, “Intercapillary lesions in the glomeruli of the kidney,” _The American journal of pathology_ , vol. 12, no. 1, p. 83, 1936\.
  * [3] I. Murray and M. A. Paolini, “Histology, kidney and glomerulus,” 2020.
  * [4] R. P. Scott and S. E. Quaggin, “The cell biology of renal filtration,” _Journal of cell biology_ , vol. 209, no. 2, pp. 199–210, 2015.
  * [5] B. A. Santo, A. Z. Rosenberg, and P. Sarder, “Artificial intelligence driven next-generation renal histomorphometry,” _Current opinion in nephrology and hypertension_ , vol. 29, no. 3, pp. 265–272, 2020.
  * [6] G. K. Rangan and G. H. Tesch, “Quantification of renal pathology by image analysis (methods in renal research),” _Nephrology_ , vol. 12, no. 6, pp. 553–558, 2007.
  * [7] L. Barisoni, C. C. Nast, J. C. Jennette, J. B. Hodgin, A. M. Herzenberg, K. V. Lemley, C. M. Conway, J. B. Kopp, M. Kretzler, C. Lienczewski _et al._ , “Digital pathology evaluation in the multicenter nephrotic syndrome study network (neptune),” _Clinical Journal of the American Society of Nephrology_ , vol. 8, no. 8, pp. 1449–1459, 2013.
  * [8] C. Zeng, Y. Nan, F. Xu, Q. Lei, F. Li, T. Chen, S. Liang, X. Hou, B. Lv, D. Liang _et al._ , “Identification of glomerular lesions and intrinsic glomerular cell types in kidney diseases via deep learning,” _The Journal of pathology_ , vol. 252, no. 1, pp. 53–64, 2020.
  * [9] J. Gallego, A. Pedraza, S. Lopez, G. Steiner, L. Gonzalez, A. Laurinavicius, and G. Bueno, “Glomerulus classification and detection based on convolutional neural networks,” _Journal of Imaging_ , vol. 4, no. 1, p. 20, 2018.
  * [10] C. C. Nast, K. V. Lemley, J. B. Hodgin, S. Bagnasco, C. Avila-Casado, S. M. Hewitt, and L. Barisoni, “Morphology in the digital age: integrating high-resolution description of structural alterations with phenotypes and genotypes,” in _Seminars in nephrology_ , vol. 35, no. 3. Elsevier, 2015, pp. 266–278.
  * [11] M. Khened, A. Kori, H. Rajkumar, G. Krishnamurthi, and B. Srinivasan, “A generalized deep learning framework for whole-slide image segmentation and analysis,” _Scientific reports_ , vol. 11, no. 1, pp. 1–14, 2021.
  * [12] J. N. Kather, A. T. Pearson, N. Halama, D. Jäger, J. Krause, S. H. Loosen, A. Marx, P. Boor, F. Tacke, U. P. Neumann _et al._ , “Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer,” _Nature medicine_ , vol. 25, no. 7, pp. 1054–1056, 2019.
  * [13] E. Moen, D. Bannon, T. Kudo, W. Graf, M. Covert, and D. Van Valen, “Deep learning for cellular image analysis,” _Nature methods_ , vol. 16, no. 12, pp. 1233–1246, 2019.
  * [14] P. Mobadersany, S. Yousefi, M. Amgad, D. A. Gutman, J. S. Barnholtz-Sloan, J. E. V. Vega, D. J. Brat, and L. A. Cooper, “Predicting cancer outcomes from histology and genomics using convolutional networks,” _Proceedings of the National Academy of Sciences_ , vol. 115, no. 13, pp. E2970–E2979, 2018\.
  * [15] A. H. Beck, A. R. Sangoi, S. Leung, R. J. Marinelli, T. O. Nielsen, M. J. Van De Vijver, R. B. West, M. Van De Rijn, and D. Koller, “Systematic analysis of breast cancer morphology uncovers stromal features associated with survival,” _Science translational medicine_ , vol. 3, no. 108, pp. 108ra113–108ra113, 2011.
  * [16] Y. Huo, R. Deng, Q. Liu, A. B. Fogo, and H. Yang, “Ai applications in renal pathology,” _Kidney International_ , 2021.
  * [17] Y. Kawazoe, K. Shimamoto, R. Yamaguchi, Y. Shintani-Domoto, H. Uozaki, M. Fukayama, and K. Ohe, “Faster r-cnn-based glomerular detection in multistained human whole slide images,” _Journal of Imaging_ , vol. 4, no. 7, p. 91, 2018.
  * [18] M. Temerinac-Ott, G. Forestier, J. Schmitz, M. Hermsen, J. Bräsen, F. Feuerhake, and C. Wemmert, “Detection of glomeruli in renal pathology by mutual comparison of multiple staining modalities,” in _Proceedings of the 10th International Symposium on Image and Signal Processing and Analysis_. IEEE, 2017, pp. 19–24.
  * [19] J. D. Bukowy, A. Dayton, D. Cloutier, A. D. Manis, A. Staruschenko, J. H. Lombard, L. C. S. Woods, D. A. Beard, and A. W. Cowley, “Region-based convolutional neural nets for localization of glomeruli in trichrome-stained whole kidney sections,” _Journal of the American Society of Nephrology_ , vol. 29, no. 8, pp. 2081–2088, 2018.
  * [20] R. Marée, S. Dallongeville, J.-C. Olivo-Marin, and V. Meas-Yedid, “An approach for detection of glomeruli in multisite digital pathology,” in _2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI)_. IEEE, 2016, pp. 1033–1036.
  * [21] H. Yang, R. Deng, Y. Lu, Z. Zhu, Y. Chen, J. T. Roland, L. Lu, B. A. Landman, A. B. Fogo, and Y. Huo, “Circlenet: Anchor-free detection with circle representation,” _arXiv preprint arXiv:2006.02474_ , 2020.
  * [22] A. H. Abdelhafiz, S. H. Brown, A. Bello, and M. El Nahas, “Chronic kidney disease in older people: physiology, pathology or both?” _Nephron Clinical Practice_ , vol. 116, no. 1, pp. c19–c24, 2010.
  * [23] A. Greenberg, S. I. Bastacky, A. Iqbal, D. Borochovitz, and J. P. Johnson, “Focal segmental glomerulosclerosis associated with nephrotic syndrome in cholesterol atheroembolism: clinicopathological correlations,” _American journal of kidney diseases_ , vol. 29, no. 3, pp. 334–344, 1997\.
  * [24] S. K. Lorbach, J. A. Hokamp, J. M. Quimby, and R. E. Cianciolo, “Clinicopathologic characteristics, pathology, and prognosis of 77 dogs with focal segmental glomerulosclerosis,” _Journal of Veterinary Internal Medicine_ , vol. 34, no. 5, pp. 1948–1956, 2020.
  * [25] J. Tan, Y. Xu, Z. Jiang, G. Pei, Y. Tang, L. Tan, Z. Zhong, P. Tarun, and W. Qin, “Global glomerulosclerosis and segmental glomerulosclerosis could serve as effective markers for prognosis and treatment of iga vasculitis with nephritis,” _Frontiers in medicine_ , vol. 7, p. 704, 2020.
  * [26] C. Benchimol, “Focal segmental glomerulosclerosis: pathogenesis and treatment,” _Current opinion in pediatrics_ , vol. 15, no. 2, pp. 171–180, 2003.
  * [27] B. Ginley, J. E. Tomaszewski, R. Yacoub, F. Chen, and P. Sarder, “Unsupervised labeling of glomerular boundaries using gabor filters and statistical testing in renal histology,” _Journal of Medical Imaging_ , vol. 4, no. 2, p. 021102, 2017.
  * [28] L. David, J. Arús-Pous, J. Karlsson, O. Engkvist, E. J. Bjerrum, T. Kogej, J. M. Kriegl, B. Beck, and H. Chen, “Applications of deep-learning in exploiting large-scale and heterogeneous compound data in industrial pharmaceutical research,” _Frontiers in pharmacology_ , vol. 10, p. 1303, 2019.
  * [29] X. Chen and K. He, “Exploring simple siamese representation learning,” _arXiv preprint arXiv:2011.10566_ , 2020.
  * [30] T. Yao, C. Qu, Q. Liu, R. Deng, Y. Tian, J. Xu, A. Jha, S. Bao, M. Zhao, A. B. Fogo, B. A. Landman, C. Chang, H. Yang, and Y. Huo, “Compound figure separation of biomedical images with side loss,” 2021.
  * [31] V. B. Kolachalama, P. Singh, C. Q. Lin, D. Mun, M. E. Belghasem, J. M. Henderson, J. M. Francis, D. J. Salant, and V. C. Chitalia, “Association of pathological fibrosis with renal survival using deep neural networks,” _Kidney international reports_ , vol. 3, no. 2, pp. 464–475, 2018.
  * [32] M. Hermsen, T. de Bel, M. Den Boer, E. J. Steenbergen, J. Kers, S. Florquin, J. J. Roelofs, M. D. Stegall, M. P. Alexander, B. H. Smith _et al._ , “Deep learning–based histopathologic assessment of kidney tissue,” _Journal of the American Society of Nephrology_ , vol. 30, no. 10, pp. 1968–1979, 2019.
  * [33] L. Lu, Y. Zheng, G. Carneiro, and L. Yang, “Deep learning and convolutional neural networks for medical image computing,” _Advances in computer vision and pattern recognition_ , vol. 10, pp. 978–3, 2017.
  * [34] P. Sarder, B. Ginley, and J. E. Tomaszewski, “Automated renal histopathology: Digital extraction and quantification of renal pathology,” in _Medical Imaging 2016: Digital Pathology_ , vol. 9791. International Society for Optics and Photonics, 2016, p. 97910F.
  * [35] G. Bueno, M. M. Fernandez-Carrobles, L. Gonzalez-Lopez, and O. Deniz, “Glomerulosclerosis identification in whole slide images using semantic segmentation,” _Computer Methods and Programs in Biomedicine_ , vol. 184, p. 105273, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0169260719311381
  * [36] N. Kumar, R. Gupta, and S. Gupta, “Whole slide imaging (wsi) in pathology: current perspectives and future directions,” _Journal of Digital Imaging_ , vol. 33, pp. 1034–1040, 2020.
  * [37] J. N. Marsh, M. K. Matlock, S. Kudose, T.-C. Liu, T. S. Stappenbeck, J. P. Gaut, and S. J. Swamidass, “Deep learning global glomerulosclerosis in transplant kidney frozen sections,” _IEEE transactions on medical imaging_ , vol. 37, no. 12, pp. 2718–2728, 2018.
  * [38] B. Ginley, K.-Y. Jen, A. Rosenberg, G. M. Rossi, S. Jain, and P. Sarder, “Fully automated classification of glomerular lesions in lupus nephritis,” in _Medical Imaging 2020: Digital Pathology_ , vol. 11320. International Society for Optics and Photonics, 2020, p. 113200Y.
  * [39] P. Chagas, L. Souza, I. Araújo, N. Aldeman, A. Duarte, M. Angelo, W. L. Dos-Santos, and L. Oliveira, “Classification of glomerular hypercellularity using convolutional features and support vector machine,” _Artificial intelligence in medicine_ , vol. 103, p. 101808, 2020.
  * [40] E. Uchino, K. Suzuki, N. Sato, R. Kojima, Y. Tamada, S. Hiragi, H. Yokoi, N. Yugami, S. Minamiguchi, H. Haga _et al._ , “Classification of glomerular pathological findings using deep learning and nephrologist-ai collective intelligence approach,” _medRxiv_ , pp. 2019–12, 2020.
  * [41] A. Fernández, S. García, and F. Herrera, “Addressing the classification with imbalanced data: open problems and new challenges on class distribution,” in _International conference on hybrid artificial intelligence systems_. Springer, 2011, pp. 1–10.
  * [42] G. D. Cascarano, F. S. Debitonto, R. Lemma, A. Brunetti, D. Buongiorno, I. De Feudis, A. Guerriero, M. Rossini, F. Pesce, L. Gesualdo _et al._ , “An innovative neural network framework for glomerulus classification based on morphological and texture features evaluated in histological images of kidney biopsy,” in _International Conference on Intelligent Computing_. Springer, 2019, pp. 727–738.
  * [43] M. E. M. Pierpont, A. S. Hentges, L. J. Gears, B. Hirsch, and A. Sinaiko, “Unbalanced 4; 6 translocation and progressive renal disease,” _American journal of medical genetics_ , vol. 95, no. 3, pp. 275–280, 2000\.
  * [44] C. Huang, Y. Li, C. C. Loy, and X. Tang, “Learning deep representation for imbalanced classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 5375–5384.
  * [45] Q. Zou, S. Xie, Z. Lin, M. Wu, and Y. Ju, “Finding the best classification threshold in imbalanced classification,” _Big Data Research_ , vol. 5, pp. 2–8, 2016.
  * [46] V. López, A. Fernández, J. G. Moreno-Torres, and F. Herrera, “Analysis of preprocessing vs. cost-sensitive learning for imbalanced classification. open problems on intrinsic data characteristics,” _Expert Systems with Applications_ , vol. 39, no. 7, pp. 6585–6608, 2012.
  * [47] T. Hastie, R. Tibshirani, and J. Friedman, “Overview of supervised learning,” in _The elements of statistical learning_. Springer, 2009, pp. 9–41.
  * [48] M. E. Celebi and K. Aydin, _Unsupervised learning algorithms_. Springer, 2016.
  * [49] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3733–3742.
  * [50] M. Noroozi and P. Favaro, “Unsupervised learning of visual representations by solving jigsaw puzzles,” in _European conference on computer vision_. Springer, 2016, pp. 69–84.
  * [51] C. Zhuang, A. L. Zhai, and D. Yamins, “Local aggregation for unsupervised learning of visual embeddings,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 6002–6012.
  * [52] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio, “Learning deep representations by mutual information estimation and maximization,” _arXiv preprint arXiv:1808.06670_ , 2018.
  * [53] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [54] H. V. Nguyen and L. Bai, “Cosine similarity metric learning for face verification,” in _Asian conference on computer vision_. Springer, 2010, pp. 709–720.
  * [55] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9729–9738.
  * [56] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar _et al._ , “Bootstrap your own latent: A new approach to self-supervised learning,” _arXiv preprint arXiv:2006.07733_ , 2020.
  * [57] J. U. Becker, D. Mayerich, M. Padmanabhan, J. Barratt, A. Ernst, P. Boor, P. A. Cicalese, C. Mohan, H. V. Nguyen, and B. Roysam, “Artificial intelligence and machine learning in nephropathology,” _Kidney International_ , vol. 98, no. 1, pp. 65–75, 2020.
  * [58] G. O. Barros, B. Navarro, A. Duarte, and W. L. Dos-Santos, “Pathospotter-k: A computational tool for the automatic identification of glomerular lesions in histological images of kidneys,” _Scientific reports_ , vol. 7, no. 1, pp. 1–8, 2017.
  * [59] M. Y. Lu, D. F. Williamson, T. Y. Chen, R. J. Chen, M. Barbieri, and F. Mahmood, “Data-efficient and weakly supervised computational pathology on whole-slide images,” _Nature Biomedical Engineering_ , vol. 5, no. 6, pp. 555–570, 2021.
  * [60] G. Bueno, M. M. Fernandez-Carrobles, L. Gonzalez-Lopez, and O. Deniz, “Glomerulosclerosis identification in whole slide images using semantic segmentation,” _Computer methods and programs in biomedicine_ , vol. 184, p. 105273, 2020.
  * [61] T. de Bel, M. Hermsen, B. Smeets, L. Hilbrands, J. van der Laak, and G. Litjens, “Automatic segmentation of histopathological slides of renal tissue using deep learning,” in _Medical Imaging 2018: Digital Pathology_ , vol. 10581. International Society for Optics and Photonics, 2018, p. 1058112.
  * [62] N. Altini, G. D. Cascarano, A. Brunetti, I. De Feudis, D. Buongiorno, M. Rossini, F. Pesce, L. Gesualdo, and V. Bevilacqua, “A deep learning instance segmentation approach for global glomerulosclerosis assessment in donor kidney biopsies,” _Electronics_ , vol. 9, no. 11, p. 1768, 2020.
  * [63] T. Qaiser, Y.-W. Tsang, D. Taniyama, N. Sakamoto, K. Nakane, D. Epstein, and N. Rajpoot, “Fast and accurate tumor segmentation of histology images using persistent homology and deep convolutional features,” _Medical image analysis_ , vol. 55, pp. 1–14, 2019.
  * [64] M. Gadermayr, A. Dombrowski, B. Klinkhammer, P. Boor, and D. Merhof, “Cnn cascades for segmenting whole slide images of the kidney. arxiv 2017,” _arXiv preprint arXiv:1708.00251_.
  * [65] N. Bouteldja, B. M. Klinkhammer, R. D. Bülow, P. Droste, S. W. Otten, S. F. von Stillfried, J. Moellmann, S. M. Sheehan, R. Korstanje, S. Menzel _et al._ , “Deep learning–based segmentation and quantification in experimental kidney histopathology,” _Journal of the American Society of Nephrology_ , vol. 32, no. 1, pp. 52–68, 2021.
  * [66] M. Gadermayr, D. Eschweiler, A. Jeevanesan, B. M. Klinkhammer, P. Boor, and D. Merhof, “Segmenting renal whole slide images virtually without training data,” _Computers in biology and medicine_ , vol. 90, pp. 88–97, 2017.
  * [67] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [68] G. Cao, W. Song, and Z. Zhao, “Gastric cancer diagnosis with mask r-cnn,” in _2019 11th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC)_ , vol. 1. IEEE, 2019, pp. 60–63.
  * [69] G. Lv, K. Wen, Z. Wu, X. Jin, H. An, and J. He, “Nuclei r-cnn: Improve mask r-cnn for nuclei segmentation,” in _2019 IEEE 2nd International Conference on Information Communication and Signal Processing (ICICSP)_. IEEE, 2019, pp. 357–362.
  * [70] A. Jha, H. Yang, R. Deng, M. E. Kapp, A. B. Fogo, and Y. Huo, “Instance segmentation for whole slide imaging: end-to-end or detect-then-segment,” _Journal of Medical Imaging_ , vol. 8, no. 1, p. 014001, 2021.
  * [71] E. H. Nguyen, H. Yang, R. Deng, Y. Lu, Z. Zhu, J. T. Roland, L. Lu, B. A. Landman, A. B. Fogo, and Y. Huo, “Circle Representation for Medical Object Detection,” _IEEE transactions on medical imaging_ , 2021.
  * [72] D. Demner-Fushman, S. Antani, M. Simpson, and G. R. Thoma, “Design and development of a multimodal biomedical information retrieval system,” _Journal of Computing Science and Engineering_ , vol. 6, no. 2, pp. 168–177, 2012.
  * [73] Q. Liu, P. C. Louis, Y. Lu, A. Jha, M. Zhao, R. Deng, T. Yao, J. T. Roland, H. Yang, S. Zhao _et al._ , “Simtriplet: Simple triplet representation learning with a single gpu,” _arXiv preprint arXiv:2103.05585_ , 2021.
  * [74] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” _arXiv preprint arXiv:1706.05587_ , 2017.
  * [75] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” _arXiv preprint arXiv:2004.10934_ , 2020.
  * [76] Z. Zhu, Y. Lu, R. Deng, H. Yang, A. B. Fogo, and Y. Huo, “Easierpath: an open-source tool for human-in-the-loop deep learning of renal pathology,” in _Interpretable and Annotation-Efficient Learning for Medical Image Computing_. Springer, 2020, pp. 214–222.
  * [77] G. Bueno, L. Gonzalez-Lopez, M. Garcia-Rojo, A. Laurinavicius, and O. Deniz, “Data for glomeruli characterization in histopathological images,” _Data in brief_ , vol. 29, p. 105314, 2020.
  * [78] P. Refaeilzadeh, L. Tang, and H. Liu, “Cross-validation.” _Encyclopedia of database systems_ , vol. 5, pp. 532–538, 2009.
  * [79] Y. Tang, “Deep learning using linear support vector machines,” _arXiv preprint arXiv:1306.0239_ , 2013.
  * [80] M. Wodzinski, I. Ciepiela, T. Kuszewski, P. Kedzierawski, and A. Skalski, “Semi-supervised deep learning-based image registration method with volume penalty for real-time breast tumor bed localization,” _Sensors_ , vol. 21, no. 12, p. 4085, 2021.
  * [81] V. Perminov, V. Ermakov, and D. Korzun, “Edge analytics for bearing fault diagnosis based on convolution neural network,” in _Fuzzy Systems and Data Mining VII_. IOS Press, 2021, pp. 94–103.
  * [82] I. Tsamardinos, A. Rakhshani, and V. Lagani, “Performance-estimation properties of cross-validation-based protocols with simultaneous hyper-parameter optimization,” _International Journal on Artificial Intelligence Tools_ , vol. 24, no. 05, p. 1540023, 2015.
  * [83] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, pp. 1097–1105, 2012.
  * [84] C. Jayakumari, V. Lavanya, and E. Sumesh, “Automated diabetic retinopathy detection and classification using imagenet convolution neural network using fundus images,” in _2020 International Conference on Smart Electronics and Communication (ICOSEC)_. IEEE, 2020, pp. 577–582.
  * [85] R. EL SALEH, S. BAKHSHI, and N.-A. Amine, “Deep convolutional neural network for face skin diseases identification,” in _2019 Fifth International Conference on Advances in Biomedical Engineering (ICABME)_. IEEE, 2019, pp. 1–4.
  * [86] R. Ali, R. C. Hardie, B. N. Narayanan, and S. De Silva, “Deep learning ensemble methods for skin lesion analysis towards melanoma detection,” in _2019 IEEE National Aerospace and electronics conference (NAECON)_. IEEE, 2019, pp. 311–316.
  * [87] I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” 2017.
  * [88] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, “Accurate, large minibatch sgd: Training imagenet in 1 hour,” _arXiv preprint arXiv:1706.02677_ , 2017.
  * [89] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [90] M. Everingham and J. Winn, “The pascal visual object classes challenge 2012 (voc2012) development kit,” _Pattern Analysis, Statistical Modelling and Computational Learning, Tech. Rep_ , vol. 8, p. 5, 2011.
  * [91] R. Deng, H. Yang, A. Jha, Y. Lu, P. Chu, A. B. Fogo, and Y. Huo, “Map3d: registration based multi-object tracking on 3d serial whole slide images,” _IEEE Transactions on Medical Imaging_ , 2021.