11institutetext:  The Hong Kong University of Science and Technology, Hong
Kong, China. 22institutetext: Shanghai Jiao Tong University, China.
22email: 22email<EMAIL_ADDRESS>22email<EMAIL_ADDRESS>
# Exploring Visual Prompts for Whole Slide Image Classification with Multiple
Instance Learning

Yi Lin† 11 Zhongchen Zhao† 22 Zhengjie ZHU 11 Lisheng Wang🖂 22 Kwang-Ting
Cheng 11 Hao Chen🖂 11

###### Abstract

Multiple instance learning (MIL) has emerged as a popular method for
classifying histopathology whole slide images (WSIs). However, existing
approaches typically rely on pre-trained models from large natural image
datasets, such as ImageNet, to generate instance features, which can be sub-
optimal due to the significant differences between natural images and
histopathology images that lead to a domain shift. In this paper, we present a
novel, simple yet effective method for learning domain-specific knowledge
transformation from pre-trained models to histopathology images. Our approach
entails using a prompt component to assist the pre-trained model in discerning
differences between the pre-trained dataset and the target histopathology
dataset, resulting in improved performance of MIL models. We validate our
method on two publicly available datasets, Camelyon16 and TCGA-NSCLC.
Extensive experimental results demonstrate the significant performance
improvement of our method for different MIL models and backbones. Upon
publication of this paper, we will release the source code for our method.

###### Keywords:

Visual prompt Multiple instance learning Whole slide image Deep learning.

${\dagger}$${\dagger}$footnotetext: Equal contribution; 🖂 corresponding
author.

##  1 Introduction

Whole slide images (WSI) play a vital role in histopathology image analysis
and clinical disease diagnosis [6, 23, 26]. With the advent of deep-learning-
based techniques, histopathology image analysis has undergone a significant
transformation [9, 24]. However, there are still challenges when it comes to
classifying WSI. Due to their massive size, WSIs cannot be directly fed into
typical deep-learning models. Therefore, WSIs are often divided into patches
for processing [11]. Unfortunately, annotating patch-level labels is labor-
intensive and time-consuming, which limits the applicability of conventional
supervised learning methods [16, 18]. To address this issue, multiple instance
learning (MIL) has emerged as the dominant technique for WSI analysis. In this
approach, each WSI is considered as a bag containing multiple patches
(instances), and a WSI bag is labeled negative only if all patches (instances)
of this bag are negative. Conversely, the bag’s label is positive if at least
one of its instances is positive.

Downsampling and feature extraction are necessary due to the large number of
patches in a WSI. The quality of the extracted patch features greatly
influences the performance of the subsequent MIL classification. Most existing
methods [13, 17, 29, 19] extract patch features by a frozen feature extractor
pre-trained on the large natural image datasets, such as ImageNet [7], and
then train the MIL classifier for the WSI prediction, as shown in Fig. 1
$(a)$. However, such a MIL training scheme overlooks the domain shift issue
between natural and pathological images. To narrow the domain shift, some
researchers [28] propose to use self-supervised pre-training methods such as
SimCLR [5] to train the feature extractor. However, these self-supervised
learning methods do not take full advantage of the bag labels, resulting in
limited performance. Another naive solution is to use partial patches instead
of all patches to fine-tune the pre-trained feature extractor, as illustrated
in Fig. 1 $(b)$. However, fine-tuning all the parameters of the feature
extractor using limited patches without patch-level labels may impair the
benefits from pre-training on large-scale datasets like ImageNet, which
increases the risk of overfitting in downstream tasks.

Figure 1: An illustration of three adapting schemes for WSI classification
with MIL: (a) freezing the pretrained feature extractor while only training
the MIL classifier; (b) fine-tuning the feature extractor and training the MIL
classifier; (c) only training the prompt blocks and the MIL classifier.

Inspired by the breakthrough of prompt learning in natural language processing
(NLP), we introduce visual prompts to adapt the pre-trained feature extractor
to pathological images, addressing the aforementioned issue. Our prompt
learning framework for MIL-based WSI classification is shown in Fig. 1 $(c)$.
Limited by memory capacity, we propose a feasible solution that selects
representative pathological images, instead of all patches, to fine-tune the
pre-trained feature extractor. Based on the selected images, we design a
prompt component added to the feature extractor to learn visual prompts, and
freeze the backbone while only training the prompt component with the
lightweight MIL classifier. In this way, our method can improve the
performance of the ImageNet pre-trained feature extractor and achieve domain
transformation to pathological image data. Our method also makes the entire
training process highly efficient and lightweight. To the best of our
knowledge, this is the first work to explore prompt learning for WSI
classification. In summary, our contributions are three-fold:

  * $\bullet$

We, for the first time, introduce visual prompts into WSI classification,
which enables data domain transformation by learning prompt components.

  * $\bullet$

We propose an intuitive but effective method for end-to-end prompt training,
which involves representative patch selection to reduce the number of
instances in a WSI bag.

  * $\bullet$

We conduct extensive experiments to validate the effectiveness of the proposed
method on two public datasets, i.e., Camelyon16 and TCGA-NSCLC. Experimental
results demonstrate consistent improvements across different MIL methods and
backbones.

##  2 Related Work

MIL for WSI classification. MIL-based methods [13, 17, 29, 19] have gained
popularity in WSI classification due to their high effectiveness. These
methods typically involve using a feature extractor to extract features from
image patches of WSI, followed by an aggregation step to obtain a feature
representation at the WSI level. A lightweight classifier is then employed to
predict the WSI category [13]. In WSI classification, an effective feature
extractor that generates representative feature is crucial for accurate
classification results. However, existing MIL-based methods for WSI
classification mainly adopt a pre-trained feature extractor without fine-
tuning, which results in sub-optimal performance [12]. This is due to the
domain shift and task discrepancies between the pre-training task (e.g.,
ImageNet) and the downstream task (e.g., histopathology) [27]. For this issue,
we introduce visual prompts for WSI classification, enabling smooth feature
modulation from the upstream dataset to the downstream WSI classification.

Prompt Learning. Prompt learning has recently emerged as a lightweight and
efficient transfer learning paradigm in NLP and has achieved remarkable
success [14]. The fundamental idea behind prompt learning is to freeze large-
scale NLP models, such as BERT [8] and GPT-3 [3], that have been pre-trained
on vast datasets and use task-specific prompts to adapt them to diverse
downstream tasks without updating any parameters [1]. Building on the NLP
prompt learning paradigm, several studies [20, 22, 4] have proposed to extend
prompt learning to natural images in computer vision. For example, Lüddecke et
al. [20] used text and image prompts to adapt the frozen pre-trained CLIP
model [25] to new image segmentation tasks. However, the effectiveness of
prompt learning in the field of histopathology analysis is under-investigated.

##  3 Method

Fig. 2 illustrates the proposed prompt learning framework, which consists of
three primary steps: (i@) MIL classifier training, (ii@) representative patch
selection, and (iii@) prompt fine-tuning. First, an ImageNet pre-trained
ResNet [10] is used to extract patch features, which are then used to train
the MIL classifier for the attention score of each patch. Second,
representative patches in each WSI bag are chosen based on their attention
scores to form a new bag. Third, the representative patches are used to fine-
tune the prompt blocks plugged into the feature extractor and MIL classifier
in an end-to-end manner. we will elaborate on each step in the following
sections.

Figure 2: Overview of our method. (i@) MIL classifier training with all patch
features. (ii@) Representative patch selection. (iii@) Prompt learning with
representative patches.

###  3.1 Attention-based MIL Classifier with Frozen Feature Extractor

In attention-based MIL for WSI classification, the standard training process
first uses a frozen feature extractor $f(\cdot)$ pre-trained on ImageNet to
extract all patch features. Then all patch features in a WSI bag are
aggregated to form the WSI feature using the attention mechanism, which learns
an attention score $\alpha_{k}$ for each patch $k$ through the MIL classifier.
The WSI feature $\bm{F}$ is obtained by computing the attention-weighted
average of all patch features in a WSI as [13]:

${\bm{F}}=\sum\limits_{k=1}^{{K}}{{\alpha_{k}}{f(\bm{x}_{k})}},$ (1)

where

${\alpha_{k}}=\dfrac{{\exp\left\\{{{{\bm{w}}^{\rm{T}}}\left({\tanh\left({{{\bm{V}}_{1}}{f(\bm{x}_{k})}}\right)\odot{\rm{sigmoid}}\left({{{\bm{V}}_{2}}{f(\bm{x}_{k})}}\right)}\right)}\right\\}}}{{\sum\limits_{j=1}^{K}{\exp}\left\\{{{{\bm{w}}^{\rm{T}}}\left({\tanh\left({{{\bm{V}}_{1}}{f(\bm{x}_{k})}}\right)\odot{\rm{sigmoid}}\left({{{\bm{V}}_{2}}{f(\bm{x}_{k})}}\right)}\right)}\right\\}}},$
(2)

where $\bm{w}$, $\bm{V}_{1}$ and $\bm{V}_{2}$ are learnable parameters in the
MIL classifier, $\odot$ is the element-wise multiplication, and $\tanh(\cdot)$
and $\rm{sigmoid(\cdot)}$ denote the tanh and sigmoid activation function,
respectively.
Finally, the MIL classifier head $h(\cdot)$ predicts the label of WSI from the
WSI feature $\bm{F}$, represented as:

$\tilde{\bm{y}}=h(\bm{F}),$ (3)

where $\tilde{\bm{y}}$ denotes the prediction of the WSI label. During
training, we minimize the prediction error using the cross-entropy (CE) loss.

###  3.2 Representative Patch Selection.

In WSI classification, it’s common for only a small number of patches within a
WSI to be associated with the disease of interest. For example, in the
positive slides of Camelyon16 [2], on average, less than 10% of the patches in
a WSI are tumor patches. Thus, only a few patches are sufficient to represent
the entire WSI bag. Based on this observation, we propose a feasible solution
that selects representative pathological images to fine-tune the pre-trained
feature extractor for WSI classification. In attention-based MIL, the patch
with a higher attention score $\alpha_{k}$ in a WSI bag is more likely to have
the same category semantics as the WSI. Thus, based on the patches’ attention
score $\alpha_{k}$ calculated by the MIL classifier, we select the top-$K$
patches with the highest attention scores in each WSI as a new bag. Here, $K$
is set to 200, which will be discussed in Section 4.2. The new bag label is
assigned as the original WSI bag label for prompt fine-tuning. In this way, we
reduce vast quantities of patches in each WSI to a small subset, enabling an
end-to-end training for both feature extractor and MIL classifier.

###  3.3 Prompt Fine-tuning.

With the selected representative patches, we design a prompt learning
framework to adapt the feature extractor from the natural image domain to the
pathological image domain, while retaining the advantage of pre-training on
the large and diverse ImageNet dataset. Specifically, we design a prompt block
that sequentially consists of a global average pooling (GAP), a multi-layer
perceptron (MLP) with two layers, and a sigmoid activation, as shown in Fig.
2. Given the intermediate feature map $\bm{f}_{i}$ from the $i$-th block of
the feature extractor, the prompt block is added in parallel to the basic
ResNet block $g_{i}(\cdot)$ to generate the visual prompt
$\bm{p}_{i}\in{\mathbb{R}^{D}}$, where $D$ denotes the dimension of the prompt
vector. Subsequently, the generated prompts $\bm{p}_{i}$ are channel-wise
multiplied with feature maps $\bm{f}_{i+1}$ in the next block, represented as:

$\bm{p}_{i}={\rm{sigmoid}}(\bm{W}_{i2}{\rm{ReLU}}(\bm{W}_{i1}{\rm{GAP}}(\bm{f}_{i}))),$
(4) $\bm{f}_{i+1}=g_{i}(\bm{f}_{i})\odot\bm{p}_{i},$ (5)

where $\text{ReLU}(\cdot)$ denotes rectified linear unit, $\bm{W}_{i1}$ and
$\bm{W}_{i2}$ are the learnable parameters to be fine-tuned, and the
parameters of $g_{i}(\cdot)$ remain frozen during training. During the
training process, only the parameters of prompt blocks and the lightweight MIL
classifier are updated in an end-to-end manner, while the original pre-trained
feature extractor is frozen.

##  4 Experiments

###  4.1 Datasets

CAMELYON16. The Camelyon16 [2] dataset consists of 399 H&E stained slides from
breast cancer screening, with two classes: normal and tumor. We employ the
official 129 testing set, and the official 270 training set is further divided
randomly into training and validation sets, with a ratio of $9\colon 1$. After
preprocessing, a total of 4,610,687 patches with the size of $256\times 256$
are obtained at $20\times$ magnification, with an average of 11,556 patches
per slide.
TCGA-NSCLC. The Cancer Genome
Atlas{\dagger}{\dagger}${\dagger}$https://www.cancer.gov/tcga (TCGA) non-small
cell lung cancer (NSCLC) [21] dataset is comprised of two subtypes of lung
cancer: Lung Adenocarcinoma (TCGA-LUAD) and Lung Squamous Cell Carcinoma
(TGCA-LUSC). It contains a total of 1,053 WSIs, with 541 LUAD slides and 512
LUSC slides. We split the dataset into training, validation, and testing sets
at the slide level, with a distribution ratio of 65:10:25. In our study, we
extract $256\times 256$ patches at $20\times$ magnification from each WSI.
After preprocessing, a total of 3,252,431 patches are obtained, with an
average of 3,089 patches per WSI.

###  4.2 Implementation Details

In our approach, we adopt the ResNet model as the feature extractor, which is
pre-trained on the ImageNet dataset. We remove the last layer of the ResNet
following [19] and add prompt blocks to the third layer. For ResNet-18 and
ResNet-50, we set the number of prompt blocks as 2 and 6, respectively. For
representative patch selection, we select the top 200 patches for each WSI bag
by default. For the prompt fine-tuning, we freeze the backbone of ResNet and
only train the prompt block and MIL classifier using the Adam optimizer [15]
with a learning rate of 1e-4 and weight decay of 1e-4 for 100 epochs. All the
experiments are conducted with an NVIDIA GeForce RTX 3090 GPU.

###  4.3 Comparison Results

Table 1: Results (%) of comparative experiments on Camelyon-16 and TCGA-NSCLC dataset. In the table, the best results are in bold. “RPS”: representative patch selection, “FT”: fine-tuning, “PT”: prompt fine-tuning. Dataset | Method | ResNet-18 | ResNet-50
---|---|---|---
AUC | F1 | Acc | AUC | F1 | Acc
Camelyon16 | DTFD [29] | 87.11 | 78.05 | 86.05 | 90.07 | 81.63 | 86.05
DTFD-RPS-FT | 81.40 | 69.23 | 81.58 | 89.34 | 81.32 | 86.82
DTFD-RPS-PT | 90.94 | 78.72 | 84.50 | 91.40 | 83.52 | 88.37
ABMIL [13] | 84.08 | 74.07 | 83.72 | 85.96 | 77.11 | 85.27
ABMIL-RPS-FT | 83.98 | 76.19 | 82.95 | 83.78 | 80.90 | 86.82
ABMIL-RPS-PT | 87.45 | 76.40 | 83.72 | 86.73 | 78.65 | 85.27
TCGA NSCLC | DTFD [29] | 92.91 | 87.22 | 87.12 | 93.47 | 89.38 | 89.01
DTFD-RPS-FT | 92.97 | 85.28 | 85.22 | 92.27 | 86.55 | 85.99
DTFD-RPS-PT | 94.19 | 87.16 | 87.50 | 93.80 | 89.71 | 89.39
ABMIL [13] | 93.56 | 86.05 | 86.36 | 93.34 | 88.39 | 88.26
ABMIL-RPS-FT | 89.76 | 83.22 | 81.06 | 90.85 | 85.29 | 84.85
ABMIL-RPS-PT | 93.84 | 87.46 | 86.74 | 94.21 | 89.55 | 89.39

In this study, we comprehensively evaluate the effectiveness of our proposed
framework with various experiment settings, including two datasets: Cameylon16
[2] and TCGA-NSCLC [21]; two backbone networks: ResNet-18 and ResNet-50; and
two MIL classifiers: a state-of-the-art model DTFD [29] and a common model
ABMIL [13]. We report the area under curve (AUC), accuracy (Acc), and F1-score
as evaluation metrics for WSI classification task. Table 1 shows the eight
different settings of experiments. The first row in each setting represents
the baseline approach with the frozen feature extractor. The second row (RPS-
FT) represents using the representative patches to fine-tune both the feature
extractor and MIL classifier. The third row (RPS-PT) represents our prompt
fine-tuning method.

In Table 1, we can observe that our method achieves a consistent improvement
compared to both the baseline approach and the fine-tuning approach. Notably,
when using ResNet-18 as the feature extractor, our method achieves a higher
AUC than the DTFD baseline, with an improvement of 3.83% on the Camelyon16
dataset and 1.28% on the TCGA-NSCLC dataset. Notably, our method with prompt
blocks significantly outperforms the RPS-FT scheme, demonstrating the
advantage of our visual prompts. Besides, our method shows particularly
superior results on the Camelyon16 dataset compared to the results on the
TCGA-NSCLC dataset. This can be attributed to the fact that the Camelyon16
dataset has fewer WSIs and a smaller proportion of tumor regions. This further
confirms the advantage of our approach in handling challenging datasets.
Overall, the experiment results indicate that our method effectively improves
the performance of attention-based MIL methods on the WSI classification
tasks.

###  4.4 Ablation Study

Figure 3: Results of ablation studies on CAMELYON16 dataset: (a) Effectiveness
of the number of prompt blocks (b) Influence of the number of representative
patches.

Effectiveness of the prompt block number. To study the impact of the prompt
component number, we conduct experiments on the Camelyon16 dataset using DTFD
as the MIL classifier with different numbers of prompt blocks added to
ResNet-50. The results presented in Fig. 3 $(a)$ show that the AUC values with
different prompt quantities are consistently improved by approximately 1%
compared to the baseline approach when using more than one prompt block. These
results suggest that our prompt block component is robust and effectively
improves the performance of the feature extractor.

Influence of the number of representative patches. We investigate the effect
of different top-$K$ values in the RPS procedure on the performance of our
method using ResNet-50 and DTFD on the Camelyon16 dataset. Fig. 3 $(b)$ shows
the AUC values under two different settings: fine-tuning scheme (RPS-FT) and
our prompt fine-tuning scheme (RPS-PT). Our method consistently outperforms
the fine-tuning scheme across all $K$ values using less than 50% GPU
resources, demonstrating the efficiency and effectiveness of our method.

###  4.5 Analysis on the Effectiveness of Visual Prompts

During the experiment, we found that fine-tuning produced unsatisfactory
performance, which was even inferior to the baselines where the feature
extractor was frozen. This subpar performance can be attributed to the
specific nature of the WSI classification task. It requires processing all the
patches in a given WSI during each iteration of model updating, which is
computationally expensive. As a result, only a small portion of the instances
are used for training, leading to a high risk of overfitting. In contrast, our
proposed method of using visual prompts allows for the quick learning of the
policy of the current task based on the previously learned representation,
enabling an efficient task and domain adaption, thereby overcoming the
limitations of fine-tuning.

##  5 Conclusion

In this paper, we propose a novel prompt learning method to learn domain-
specific knowledge transformation from ImageNet pre-trained models to
pathological images. Our innovation is based on the observation that there is
a large domain shift and task discrepancy between the upstream datasets and
pathological tasks, resulting in sub-optimal feature representation. To
relieve this issue, we introduce a prompt component and representative patches
selection strategy to fine-tune the prompt blocks while freezing the feature
extractor backbone. In this way, the extracted patch features can be adapted
for pathological images and boost the WSI classification with MIL models.
Experiments on two public datasets (i.e., Cameylon16 and TCGA-NSCLC) with two
MIL classifiers (i.e., DTFD and ABMIL) demonstrate the effectiveness and
efficiency of our method.

## References

  * [1] Bahng, H., Jahanian, A., Sankaranarayanan, S., Isola, P.: Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274 1(3), 4 (2022)
  * [2] Bejnordi, B.E., Veta, M., Van Diest, P.J., Van Ginneken, B., Karssemeijer, N., Litjens, G., Van Der Laak, J.A., Hermsen, M., Manson, Q.F., Balkenhol, M., et al.: Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama 318(22), 2199–2210 (2017)
  * [3] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
  * [4] Chen, H., Tao, R., Zhang, H., Wang, Y., Ye, W., Wang, J., Hu, G., Savvides, M.: Conv-adapter: Exploring parameter efficient transfer learning for convnets. arXiv preprint arXiv:2208.07463 (2022)
  * [5] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning. pp. 1597–1607. PMLR (2020)
  * [6] Cornish, T.C., Swapp, R.E., Kaplan, K.J.: Whole-slide imaging: routine pathologic diagnosis. Advances in Anatomic Pathology 19(3), 152–159 (2012)
  * [7] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–255. Ieee (2009)
  * [8] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [9] Hashimoto, N., Fukushima, D., Koga, R., Takagi, Y., Ko, K., Kohno, K., Nakaguro, M., Nakamura, S., Hontani, H., Takeuchi, I.: Multi-scale domain-adversarial multiple-instance cnn for cancer subtype classification with unannotated histopathological images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3852–3861 (2020)
  * [10] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
  * [11] Hou, L., Samaras, D., Kurc, T.M., Gao, Y., Davis, J.E., Saltz, J.H.: Patch-based convolutional neural network for whole slide tissue image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2424–2433 (2016)
  * [12] Hu, Y., Sirinukunwattana, K., Gaitskell, K., Wood, R., Verrill, C., Rittscher, J.: Predicting molecular traits from tissue morphology through self-interactive multi-instance learning. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part II. pp. 130–139. Springer (2022)
  * [13] Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: International Conference on Machine Learning. pp. 2127–2136. PMLR (2018)
  * [14] Jia, M., Tang, L., Chen, B.C., Cardie, C., Belongie, S., Hariharan, B., Lim, S.N.: Visual prompt tuning. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII. pp. 709–727. Springer (2022)
  * [15] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [16] Lerousseau, M., Vakalopoulou, M., Classe, M., Adam, J., Battistella, E., Carré, A., Estienne, T., Henry, T., Deutsch, E., Paragios, N.: Weakly supervised multiple instance learning histopathological tumor segmentation. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part V 23. pp. 470–479. Springer (2020)
  * [17] Li, B., Li, Y., Eliceiri, K.W.: Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14318–14328 (2021)
  * [18] Li, S., Liu, Y., Sui, X., Chen, C., Tjio, G., Ting, D.S.W., Goh, R.S.M.: Multi-instance multi-scale cnn for medical image classification. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part IV 22. pp. 531–539. Springer (2019)
  * [19] Lu, M.Y., Williamson, D.F., Chen, T.Y., Chen, R.J., Barbieri, M., Mahmood, F.: Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering 5(6), 555–570 (2021)
  * [20] Lüddecke, T., Ecker, A.: Image segmentation using text and image prompts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7086–7096 (2022)
  * [21] Newman, A.M., Liu, C.L., Green, M.R., Gentles, A.J., Feng, W., Xu, Y., Hoang, C.D., Diehn, M., Alizadeh, A.A.: Robust enumeration of cell subsets from tissue expression profiles. Nature Methods 12(5), 453–457 (2015)
  * [22] Nie, X., Ni, B., Chang, J., Meng, G., Huo, C., Zhang, Z., Xiang, S., Tian, Q., Pan, C.: Pro-tuning: Unified prompt tuning for vision tasks. arXiv preprint arXiv:2207.14381 (2022)
  * [23] Pantanowitz, L., Valenstein, P.N., Evans, A.J., Kaplan, K.J., Pfeifer, J.D., Wilbur, D.C., Collins, L.C., Colgan, T.J.: Review of the current state of whole slide imaging in pathology. Journal of Pathology Informatics 2(1), 36 (2011)
  * [24] Pinckaers, H., Van Ginneken, B., Litjens, G.: Streaming convolutional neural networks for end-to-end learning with multi-megapixel images. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(3), 1581–1590 (2020)
  * [25] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 8748–8763. PMLR (2021)
  * [26] Srinidhi, C.L., Ciga, O., Martel, A.L.: Deep neural network models for computational histopathology: A survey. Medical Image Analysis 67, 101813 (2021)
  * [27] Stacke, K., Eilertsen, G., Unger, J., Lundström, C.: Measuring domain shift for deep learning in histopathology. IEEE Journal of Biomedical and Health Informatics 25(2), 325–336 (2020)
  * [28] Yang, P., Hong, Z., Yin, X., Zhu, C., Jiang, R.: Self-supervised visual representation learning for histopathological images. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24. pp. 47–57. Springer (2021)
  * [29] Zhang, H., Meng, Y., Zhao, Y., Qiao, Y., Yang, X., Coupland, S.E., Zheng, Y.: DTFD-MIL: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18802–18812 (2022)


$\displaystyle-\frac{1}{2N-S^{(N)}(\xi^{(N)}_{2i+1})-23N^{\frac{1}{2}-\varepsilon+\delta}}-\frac{1}{S^{(N)}(\xi^{(N)}_{2i+1})-N+23N^{\frac{1}{2}-\varepsilon+\delta}}$
$\displaystyle\leq$ $\displaystyle\frac{1}{S^{(N)}(\xi^{(N)}_{2i+1})-N-\beta
N^{\frac{1}{2}-\varepsilon}-13N^{\frac{1}{2}-\varepsilon+\delta}}-\frac{1}{S^{(N)}(\xi^{(N)}_{2i+1})-N+23N^{\frac{1}{2}-\varepsilon+\delta}}$
$\displaystyle\leq$ $\displaystyle CN^{-\frac{1}{2}-3\varepsilon+\delta}$

and from (E),

$\mathbb{E}\left[\left(\sup_{j\in\\{0,...,i^{*}_{N}-1\\}}\sum_{i=0}^{j}\int_{\xi^{(N)}_{2i+3}}^{\xi^{(N)}_{u,i}}\frac{\lambda^{(N)}_{u,i}}{\mu^{(N)}_{u,i}-\lambda^{(N)}_{u,i}}ds\right)\mathds{1}\left(\\{i^{*}_{N}\leq\lceil
12TN^{\frac{1}{2}+3\varepsilon+\delta}\rceil\\}\cap\bar{E}^{(N)}\right)\right]\leq
C^{\prime}N^{\frac{1}{2}-\varepsilon+2\delta}.$

By Markov’s inequality and Lemma E.2, we get that

$\mathbb{P}\bigg{(}\bigg{\\{}\sup_{j\in\\{0,...,i^{*}_{N}-1\\}}\frac{1}{N^{\frac{1}{2}+\varepsilon}}\sum_{i=1}^{j}\int_{\xi^{(N)}_{2i+3}}^{\xi^{(N)}_{u,i}}\frac{\lambda^{(N)}_{u,i}}{\mu^{(N)}_{u,i}-\lambda^{(N)}_{u,i}}ds\geq
C^{\prime}N^{-2\varepsilon+4\delta}\bigg{\\}}\cap\bar{E}^{(N)}\bigg{)}\leq
N^{-\delta}.$ (E.38)

From (E), (E.31) and (E.38), we conclude that
$\sup_{j\in\\{0,...,i^{*}_{N}-1\\}}\frac{1}{N^{\frac{1}{2}+\varepsilon}}\sum_{i=0}^{j}\int_{\xi^{(N)}_{2i+1}}^{\xi^{(N)}_{2i+3}}\Big{(}I^{(N)}_{u,i}(s)-\frac{\lambda^{(N)}_{u,i}}{\mu^{(N)}_{u,i}-\lambda^{(N)}_{u,i}}\Big{)}ds\xrightarrow{\
\mathbb{P}\ }0.$ ∎

###### Proof.

Proof of Lemma 6.6. We will prove (6.6). The proof of (6.7) is similar. Note
that due to Lemma 6.3, it suffices to prove that the left hand side converges
to zero under the event $\bar{E}^{(N)}$. Note that, for any
$i\in\\{0,...,i^{*}_{N}-1\\}$,
$\frac{\lambda^{(N)}_{l,i}}{\mu^{(N)}_{l,i}-\lambda^{(N)}_{l,i}}=\frac{2N-S^{(N)}(\xi^{(N)}_{2i+1})-23N^{\frac{1}{2}-\varepsilon+\delta}}{S^{(N)}(\xi^{(N)}_{2i+1})-N+23N^{\frac{1}{2}-\varepsilon+\delta}}.$
Now, $x\mapsto\frac{2N-x}{x-N}$ is Lipschitz continuous on the interval
$[N+\frac{1}{2}BN^{\frac{1}{2}+\varepsilon},\infty)$ with Lipschitz constant
$\frac{4}{B^{2}}N^{-2\varepsilon}$. Therefore, for any
$s\in[\xi^{(N)}_{2i+1},\xi^{(N)}_{2i+3})$, under the event $\bar{E}^{(N)}$,

$\displaystyle\Big{|}\frac{2N-S^{(N)}(\xi^{(N)}_{2i+1})-23N^{\frac{1}{2}-\varepsilon+\delta}}{S^{(N)}(\xi^{(N)}_{2i+1})-N+23N^{\frac{1}{2}-\varepsilon+\delta}}-\frac{2N-S^{(N)}(s)}{S^{(N)}(s)-N}\Big{|}$
$\displaystyle\leq$
$\displaystyle\frac{4}{B^{2}}N^{-2\varepsilon}\bigg{(}\sup_{i\in\\{0,...,i^{*}_{N}-1\\}}\sup_{\xi^{(N)}_{2i+1}\leq
s\leq\xi^{(N)}_{2i+3}}\left|S^{(N)}(\xi^{(N)}_{2i+1})-S^{(N)}(s)\right|+23N^{\frac{1}{2}-\varepsilon+\delta}\bigg{)}$
$\displaystyle\leq$
$\displaystyle\frac{144N^{\frac{1}{2}-3\varepsilon+\delta}}{B^{2}}.$

Thus,

$\displaystyle\sup_{j\in\\{0,...,i^{*}_{N}-1\\}}\frac{1}{N^{\frac{1}{2}+\varepsilon}}\Big{|}\sum_{i=0}^{j}\int_{\xi^{(N)}_{2i+1}}^{\xi^{(N)}_{2i+3}}\Big{(}\frac{\lambda^{(N)}_{l,i}}{\mu^{(N)}_{l,i}-\lambda^{(N)}_{l,i}}-\frac{2N-S^{(N)}(s)}{S^{(N)}(s)-N}\Big{)}ds\Big{|}$
$\displaystyle\leq$
$\displaystyle\frac{1}{N^{\frac{1}{2}+\varepsilon}}\int_{0}^{N^{2\varepsilon}T}\frac{144N^{\frac{1}{2}-3\varepsilon+\delta}}{B^{2}}ds=\frac{144T}{B^{2}}N^{-2\varepsilon+\delta},$

which tends to $0$ as $n\to\infty$, as $\delta<\varepsilon/2$. ∎

###### Proof.

Proof of Lemma 6.7. Fix any
$0<\delta<(2\varepsilon)\wedge\left(\frac{1}{2}-\varepsilon\right)$. By Lemma
6.1, for all $N\geq N_{1}$,

$\sup_{\begin{subarray}{c}x\leq K_{1}N^{\frac{1}{2}-\varepsilon},\\\
BN^{\frac{1}{2}+\varepsilon}<y\leq
K_{2}N^{\frac{1}{2}+\varepsilon}\end{subarray}}\mathbb{P}_{(x,y,\underline{0})}\Big{(}\sup_{i\in\\{0,...,i^{*}_{N}-1\\}}\xi^{(N)}_{2i+3}-\xi^{(N)}_{2i+1}\geq
2N^{\delta-2\varepsilon}\Big{)}\leq a_{1}e^{-b_{1}N^{\frac{\delta}{5}}}.$
(E.39)

By Proposition 5.5,

$\mathbb{P}\big{(}\sup_{0\leq t\leq T(N,B)}I^{(N)}(t)\geq
5N^{\frac{1}{2}-\varepsilon+\delta}\big{)}\leq ae^{-bN^{\frac{\delta}{2}}}.$
(E.40)

and similarly as in (E), for all $N\geq N_{0}^{\prime}$,

$\sup_{\begin{subarray}{c}x\leq K_{1}N^{\frac{1}{2}-\varepsilon},\\\
BN^{\frac{1}{2}+\varepsilon}<y\leq
K_{2}N^{\frac{1}{2}+\varepsilon}\end{subarray}}\mathbb{P}_{(x,y,\underline{0})}\Big{(}\sup_{0\leq
t\leq N^{2\varepsilon}T}Q^{(N)}_{3}(t)>0\Big{)}\leq
c_{1}e^{-c_{2}N^{(\frac{1}{2}-\varepsilon)/5}}.$ (E.41)

Define the event

$\displaystyle E^{(N)}\coloneqq\Big{\\{}\sup_{0\leq t\leq
T(N,B)}I^{(N)}(t)<5N^{\frac{1}{2}-\varepsilon+\delta},$
$\displaystyle\sup_{0\leq t\leq T(N,B)}Q^{(N)}_{3}(t)=0,\text{ and }$
$\displaystyle\xi^{(N)}_{2i+3}-\xi^{(N)}_{2i+1}<2N^{\delta-2\varepsilon},\forall
i\in\\{0,...,i^{*}_{N}-1\\}\Big{\\}}.$

Since $\delta<2\varepsilon$, there exists $N^{\prime}_{B}>0$ such that for
$N\geq N^{\prime}_{B}$, on the event $E^{(N)}$, for $s\in[0,T(N,B)]$,

$\displaystyle S^{(N)}(s)$ $\displaystyle=Q^{(N)}_{2}(s)+N-I^{(N)}(s)$
$\displaystyle\geq
BN^{\frac{1}{2}+\varepsilon}+N-5N^{\frac{1}{2}-\varepsilon+\delta}\geq\frac{1}{2}BN^{\frac{1}{2}+\varepsilon}+N,$

implying that for $N\geq N^{\prime}_{B}$ and $s\in[0,T(N,B)]$,
$\frac{2N-S^{(N)}(s)}{S^{(N)}(s)-N}\leq\frac{2N^{\frac{1}{2}-\varepsilon}}{B}.$
Hence, on the event $E^{(N)}$,

$\displaystyle\sup_{\xi^{(N)}_{2i+1}\leq
t\leq\xi^{(N)}_{2i+3}}\left[\frac{1}{N^{\frac{1}{2}+\varepsilon}}\int_{\xi^{(N)}_{2i+1}}^{t}I^{(N)}(s)ds+\frac{1}{N^{\frac{1}{2}+\varepsilon}}\int_{\xi^{(N)}_{2i+1}}^{t}\frac{2N-S^{(N)}(s)}{S^{(N)}(s)-N}ds\right]$
$\displaystyle\leq
10N^{2\delta-4\varepsilon}+\frac{4}{B}N^{\delta-4\varepsilon}$
$\displaystyle\rightarrow 0\text{ as }N\rightarrow\infty,$

since $\delta<(2\varepsilon)\wedge(\frac{1}{2}-\varepsilon)$. By
(E.39)–(E.41), we have $\lim_{N\rightarrow\infty}\mathbb{P}(E^{(N)})=1$. Thus,
the desired result holds. ∎

##  Appendix F Upper bound on $I^{(N)}$

######  Proposition F.1.

For any fixed $K_{1},K_{2},T>0$ and $0<\delta<\varepsilon$, there exist
constants $a,b,N_{1}>0$, such that for all $N\geq N_{1},x\leq
K_{1}N^{\frac{1}{2}-\varepsilon},y\leq K_{2}N^{\frac{1}{2}+\varepsilon}$,

$\sup_{\underline{z}}\mathbb{P}_{(x,y,\underline{z})}\Big{(}\sup_{0\leq t\leq
N^{2\varepsilon}T}I^{(N)}(t)\geq 2N^{\frac{1}{2}+\delta}\Big{)}\leq
ae^{-bN^{2\delta}},$

and consequently, for all $\tilde{\varepsilon}>0$,

$\lim_{N\rightarrow\infty}\sup_{\underline{z}}\mathbb{P}_{(x,y,\underline{z})}\Big{(}\frac{1}{N^{1+2\varepsilon}}\int_{0}^{N^{2\varepsilon}T}I^{(N)}(s)ds\geq\tilde{\varepsilon}\Big{)}=0.$
(F.1)

###### Proof.

Proof. Let $\tilde{I}^{(N)}$ denote a birth-death process with increase rate
$N-\tilde{I}^{(N)}$ and decrease rate $N-\beta N^{\frac{1}{2}-\varepsilon}$.
By Lemma [16, Lemma EC.17], starting from $1$, the probability $h_{1}^{x}$ of
$\tilde{I}^{(N)}$ hitting $x$ before returning to $0$ is given by

$\begin{split}h_{1}^{x}=\frac{1}{\sum_{n=1}^{x}\prod_{m=1}^{n-1}\frac{N-\beta
N^{\frac{1}{2}-\varepsilon}}{N-m}}&\leq\prod_{m\leq x-1}\frac{N-m}{N-\beta
N^{\frac{1}{2}-\varepsilon}}\\\ &\leq\exp\Big{\\{}-\sum_{m\leq
x-1}\frac{m-\beta
N^{\frac{1}{2}-\varepsilon}}{N}\Big{\\}}=\exp\Big{\\{}-\frac{x(x-1)}{2N}+\frac{\beta
N^{\frac{1}{2}-\varepsilon}(x-1)}{N}\Big{\\}},\end{split}$ (F.2)

where the second inequality is due to $1-z\leq e^{-z}$, for $z\geq 0$. Now,
recall $\xi_{i}^{(N)}$ for $i\geq 1$, from (6.1). Since rate of increase of
$I^{(N)}$ is $Q^{(N)}_{1}-Q^{(N)}_{2}=N-I^{(N)}-Q^{(N)}_{2}\leq N-I^{(N)}$, in
the interval $[\xi^{(N)}_{2i},\xi^{(N)}_{2i+1}]$, we can naturally couple the
processes $I^{(N)}(t)$ and $\tilde{I}^{(N)}(t)$ with
$I^{(N)}(\xi^{(N)}_{2i})=\tilde{I}^{(N)}(\xi^{(N)}_{2i})=1$, such that sample
path-wise, $I^{(N)}$ is dominated by the process $\tilde{I}^{(N)}$. Thus,
taking $x=N^{\frac{1}{2}+\delta}$ in (F.2), we have that for each $i\geq 1$
and large enough $N$,

$\sup_{i\geq
1}\sup_{x,y,\underline{z}}\mathbb{P}_{(x,y,\underline{z})}\Big{(}\sup_{\xi^{(N)}_{2i}\leq
t\leq\xi^{(N)}_{2i+1}}I^{(N)}(t)\geq N^{\frac{1}{2}+\delta}\Big{)}\leq
h_{1}^{N^{\frac{1}{2}+\delta}}\leq e^{-cN^{2\delta}},$ (F.3)

where $c$ is a positive constant. This bounds $I^{(N)}$ on each of its
excursions. Now we will consider the interval $[0,\xi^{(N)}_{1}]$. Take
$N_{1}\in\mathbb{N}$ such that $N^{\frac{1}{2}+\delta}\geq
K_{1}N^{\frac{1}{2}-\varepsilon}$ and (F.3) holds for all $N\geq N_{1}$. Note
that for all $N\geq N_{1}$ and $x\leq K_{1}N^{\frac{1}{2}-\varepsilon}$,
$y\geq 0$,

$\displaystyle\mathbb{P}_{(x,y,\underline{z})}\Big{(}\sup_{0\leq
t\leq\xi^{(N)}_{1}}I^{(N)}(t)\geq
2N^{\frac{1}{2}+\delta}\Big{)}\leq\mathbb{P}\Big{(}\tilde{I}^{(N)}(t)\text{
hits }2N^{\frac{1}{2}+\delta}\text{ before hitting
}0\,|\,\tilde{I}^{(N)}(0)=x\Big{)}$
$\displaystyle\leq\mathbb{P}\Big{(}\tilde{I}^{(N)}(t)\text{ hits
}2N^{\frac{1}{2}+\delta}\text{ before hitting
}0\,|\,\tilde{I}^{(N)}(0)=\lfloor
K_{1}N^{\frac{1}{2}-\varepsilon}\rfloor\Big{)}$
$\displaystyle\leq\mathbb{P}\Big{(}\tilde{I}^{(N)}(t)\text{ hits
}N^{\frac{1}{2}+\delta}\text{ before hitting
}0\,|\,\tilde{I}^{(N)}(0)=1\Big{)}\leq e^{-cN^{2\delta}},$

where for the third inequality, we consider two copies of the process
$\tilde{I}^{(N)}$:
$\big{\\{}\tilde{I}^{(N)}_{1}(t),t>0|\tilde{I}^{(N)}_{1}(0)=1\big{\\}}$ and
$\big{\\{}\tilde{I}^{(N)}_{2}(t),t>0|\tilde{I}^{(N)}_{2}(0)=\lfloor
K_{1}N^{\frac{1}{2}-\varepsilon}\big{\rfloor}\\}$ which are coupled as
follows: if there is a new arrival in $\tilde{I}^{(N)}_{1}$ (that is,
$\tilde{I}^{(N)}_{1}$ increases by $1$), then there is an arrival in
$\tilde{I}^{(N)}_{2}$ with probability
$\frac{N-\tilde{I}^{(N)}_{2}}{N-\tilde{I}^{(N)}_{1}}$; if there is a departure
in $\tilde{I}^{(N)}_{2}$, then there is a departure in $\tilde{I}^{(N)}_{1}$
(or if $\tilde{I}^{(N)}_{1}$ was zero before the departure, it stays zero
after it). In this way, $|\tilde{I}^{(N)}_{1}(t)-\tilde{I}^{(N)}_{2}(t)|\leq
K_{1}N^{\frac{1}{2}-\varepsilon}$ for all $t$ which implies that the third
inequality holds. Also, writing $i^{**}_{N}:=\inf\\{i\geq
0:\xi^{(N)}_{2i+3}\geq N^{2\varepsilon}T\\}$, we have by the same argument
used to derive (E.8),

$\sup_{\begin{subarray}{c}x\leq K_{1}N^{\frac{1}{2}-\varepsilon},\\\ y\leq
K_{2}N^{\frac{1}{2}+\varepsilon}\end{subarray}}\mathbb{P}\Big{(}i^{**}_{N}\geq\frac{3}{2}N^{1+2\varepsilon}T\Big{)}\leq
e^{-\frac{3}{32}N^{1+2\varepsilon}T}.$

Hence, uniformly over all $x\leq K_{1}N^{\frac{1}{2}-\varepsilon}$, $y\leq
K_{2}N^{\frac{1}{2}+\varepsilon}$, and feasible $\underline{z}$,

$\begin{split}&\mathbb{P}_{(x,y,\underline{z})}\Big{(}\sup_{0\leq t\leq
N^{2\varepsilon}T}I^{(N)}\geq 2N^{\frac{1}{2}+\delta}\Big{)}\\\
&\leq\mathbb{P}_{(x,y,\underline{z})}\Big{(}i^{**}_{N}\geq\frac{3}{2}N^{1+2\varepsilon}T\Big{)}+\mathbb{P}_{(x,y,\underline{z})}\Big{(}\sup_{0\leq
t\leq\xi^{(N)}_{1}}I^{(N)}(t)\geq 2N^{\frac{1}{2}+\delta}\Big{)}\\\ &\hskip
128.0374pt+\frac{3}{2}N^{1+2\varepsilon}T\sup_{i\geq
1}\mathbb{P}_{(x,y,\underline{z})}\Big{(}\sup_{\xi^{(N)}_{2i}\leq
t\leq\xi^{(N)}_{2i+1}}I^{(N)}\geq N^{\frac{1}{2}+\delta}\Big{)}\leq
ae^{-bN^{2\delta}}.\end{split}$

Moreover, for all $N\geq N_{1}$, $x\leq K_{1}N^{\frac{1}{2}-\varepsilon}$,
$y\leq K_{2}N^{\frac{1}{2}+\varepsilon}$,

$\displaystyle\sup_{\underline{z}}\mathbb{P}_{(x,y,\underline{z})}\Big{(}\frac{1}{N^{1+2\varepsilon}}\int_{0}^{N^{2\varepsilon}T}I^{(N)}(s)ds\geq
2N^{-\frac{1}{2}+\delta}\Big{)}$ $\displaystyle\hskip
85.35826pt\quad\leq\sup_{\underline{z}}\mathbb{P}_{(x,y,\underline{z})}\Big{(}\sup_{0\leq
s\leq N^{2\varepsilon}T}I^{(N)}(s)\geq 2N^{\frac{1}{2}+\delta}\Big{)}\leq
ae^{-bN^{2\delta}}.$

This completes the proof. ∎

## Acknowledgments.

Banerjee was supported in part by the NSF CAREER award DMS-2141621 and the NSF
RTG grant DMS-2134107. Mukherjee and Zhao were partially supported by NSF
grant CIF-2113027.

## References

  * [1] Ivo Adan and Jacques Resing.  Queueing Systems.  Eindhoven University of Technology, 2015.
  * [2] Rami Atar.  A diffusion regime with nondegenerate slowdown.  Oper. Res., 60(2):490–500, 2012.
  * [3] Sayan Banerjee and Debankur Mukherjee.  Join-the-shortest queue diffusion limit in Halfin-Whitt regime: Tail asymptotics and scaling of extrema.  Ann. Appl. Probab., 29(2):1262–1309, 2019.
  * [4] Sayan Banerjee and Debankur Mukherjee.  Join-the-shortest queue diffusion limit in Halfin-Whitt regime: Sensitivity on the heavy traffic parameter.  Ann. Appl. Probab., 30(1):80–144, 2020.
  * [5] P. Billingsley.  Probability and Measure.  Wiley Series in Probability and Statistics. Wiley, 2012.
  * [6] Maury Bramson.  Stability of join the shortest queue networks.  Ann. Appl. Probab., 21(4):1568–1625, 2011.
  * [7] Anton Braverman.  Steady-state analysis of the join-the-shortest-queue model in the Halfin-Whitt regime.  Math. Oper. Res., 45(3):1069–1103, 2020.
  * [8] Anton Braverman.  Steady-state analysis of the join-the-shortest-queue model in the halfin–whitt regime.  Mathematics of Operations Research, 45(3):1069–1103, 2020.
  * [9] Anton Braverman.  The join-the-shortest-queue system in the Halfin-Whitt regime: rates of convergence to the diffusion limit.  arXiv:2202.02889, 2022.
  * [10] Amarjit Budhiraja, Eric Friedlander, and Ruoyu Wu.  Many-server asymptotics for Join-the-Shortest-Queue: Large deviations and rare events.  arXiv:1904.04938, 2019.
  * [11] Amarjit Budhiraja and Chihoon Lee.  Stationary distribution convergence for generalized jackson networks in heavy traffic.  Mathematics of Operations Research, 34(1):45–56, 2009.
  * [12] Anthony Ephremides, P Varaiya, and Jean Walrand.  A simple dynamic routing problem.  IEEE Trans. Autom. Control, 25(4):690–693, 1980.
  * [13] Patrick Eschenfeldt and David Gamarnik.  Join the shortest queue with many servers. The heavy traffic-asymptotics.  Math. Oper. Res., 43(3):867–886, 2018.
  * [14] G. Foschini and J. Salz.  A basic dynamic routing problem and diffusion.  IEEE Trans. Commun., 26(3):320–327, 1978.
  * [15] G J Foschini.  On heavy traffic diffusion analysis and dynamic routing in packet switched networks.  Comp. Perf., pages 499–513, 1977.
  * [16] Varun Gupta and Neil Walton.  Load balancing in the nondegenerate slowdown regime.  Oper. Res., 67(1):281–294, 2019.
  * [17] I. Gurvich.  Design and control of the m/m/n queue with multi-class customers and many servers.  M.Sc. Thesis, 2004.
  * [18] P.J. Hunt and T.G. Kurtz.  Large loss networks.  Stoch. Proc. Appl., 53(2):363–378, 1994.
  * [19] Daniela Hurtado-Lange and Siva Theja Maguluri.  Load balancing system under Join the Shortest Queue: Many-server-heavy-traffic asymptotics.  arXiv:2004.04826, 2020.
  * [20] Daniela Hurtado-Lange and Siva Theja Maguluri.  Load balancing system in the many-server heavy-traffic asymptotics.  Preprint, 2021.
  * [21] I. Karatzas and S. Shreve.  Brownian Motion and Stochastic Calculus.  Graduate Texts in Mathematics. Springer New York, 2014.
  * [22] Leonard Kleinrock.  Theory, Volume 1, Queueing Systems.  Wiley-Interscience, USA, 1975.
  * [23] László Lakatos, Laszlo Szeidl, and Miklos Telek.  Introduction to queueing systems with telecommunication applications.  Springer, 2019.
  * [24] Robert Liptser and Albert Nikolaevich Shiryayev.  Theory of martingales, volume 49.  Springer Science & Business Media, 2012.
  * [25] Xin Liu and Lei Ying.  A simple steady-state analysis of load balancing algorithms in the sub-Halfin-Whitt regime.  ACM SIGMETRICS Perform. Eval. Rev., 46(2):15–17, 2019.
  * [26] Xin Liu and Lei Ying.  Universal scaling of distributed queues under load balancing in the super-Halfin-Whitt regime.  IEEE/ACM Trans. Netw., 30(1):190–201, 2022.
  * [27] Costis Maglaras, John Yao, and Assaf Zeevi.  Optimal price and delay differentiation in large-scale queueing systems.  Manage. Sci., 64(5):2427–2444, 2018.
  * [28] A. Mandelbaum.  QED Q’s: Quality- and efficiency-driven queues.  Lecture Notes, 2008.
  * [29] Colin McDiarmid.  Concentration.  Springer Berlin Heidelberg, Berlin, Heidelberg, 1998.
  * [30] Sean P Meyn and Richard L Tweedie.  Markov chains and stochastic stability.  Springer Science & Business Media, 2012.
  * [31] Debankur Mukherjee, Sem C. Borst, Johan S. H. Van Leeuwaarden, and Philip A. Whiting.  Universality of load balancing schemes on the diffusion scale.  J. Appl. Probab., 53(4), 2016.
  * [32] Debankur Mukherjee, Sem C Borst, Johan S H van Leeuwaarden, and Philip A Whiting.  Universality of power-of-d load balancing in many-server systems.  Stoch. Syst., 8(4):265–292, 2018.
  * [33] Guodong Pang, Rishi Talreja, and Ward Whitt.  Martingale proofs of many-server heavy-traffic limits for markovian queues.  Probab. Surveys, 4:193–267, 2007.
  * [34] Martin I. Reiman.  Some diffusion approximations with state space collapse.  In Modelling and performance evaluation methodology, pages 207–240. Springer, Berlin, Heidelberg, 1984.
  * [35] Gareth O. Roberts and Richard L. Tweedie.  Exponential convergence of Langevin distributions and their discrete approximations.  Bernoulli, 2(4):341 – 363, 1996.
  * [36] L. C. G. Rogers and David Williams.  Diffusions, Markov Processes, and Martingales, volume 1 of Cambridge Mathematical Library.  Cambridge University Press, 2 edition, 2000.
  * [37] M. van der Boor, S.C. Borst, J.S.H. van Leeuwaarden, and D. Mukherjee.  Scalable load balancing in networked systems: A survey of recent advances.  SIAM Rev. (forthcoming), 2022.
  * [38] Ward Whitt.  How multiserver queues scale with growing congestion-dependent demand.  Oper. Res., 51(4):531–542, 2003.
  * [39] Ward Whitt.  Proofs of the martingale fclt.  Probab. Surveys, 4:268–302, 2007.
  * [40] Wayne Winston.  Optimality of the shortest line discipline.  J. Appl. Probab., 14(1):181–189, 1977.
  * [41] Hanqin Zhang, Guang-Hui Hsu, and Rongxin Wang.  Heavy traffic limit theorems for a sequence of shortest queueing systems.  Queueing Syst., 21(1):217–238, 1995.


# ST-RAP: A Spatio-Temporal Framework for Real Estate Appraisal

Hojoon Lee<EMAIL_ADDRESS>KAIST AI ,  Hawon Jeong<EMAIL_ADDRESS>KAIST AI ,  Byungkun Lee<EMAIL_ADDRESS>KAIST AI ,  Kyungyup Lee
<EMAIL_ADDRESS>Spacewalk and  Jaegul Choo<EMAIL_ADDRESS>KAIST AI

(2023)

###### Abstract.

In this paper, we introduce ST-RAP, a novel Spatio-Temporal framework for Real
estate APpraisal. ST-RAP employs a hierarchical architecture with a
heterogeneous graph neural network to encapsulate temporal dynamics and
spatial relationships simultaneously. Through comprehensive experiments on a
large-scale real estate dataset, ST-RAP outperforms previous methods,
demonstrating the significant benefits of integrating spatial and temporal
aspects in real estate appraisal. Our code and dataset are available at
https://github.com/dojeon-ai/STRAP.

Real Estate Appraisal; Spatio-Temporal Network; Graph Network

††copyright: acmcopyright††journalyear: 2023††doi:
10.1145/1122445.1122456††ccs: Information systems Spatial-temporal systems

##  1\. Introduction

Real estate appraisal, as known as property value estimation, profoundly
impacts the financial decisions of individuals and corporations and steers the
strategic direction of government policies. Inaccurate appraisals can have
severe consequences, leading to substantial financial losses for stakeholders
or contributing to government budget deficits. The 2008 subprime mortgage
crisis is a representative example of the potential economic repercussions
caused by erroneous real estate appraisal (Pavlov and Wachter, 2011; Demyanyk
and Van Hemert, 2011).

To mitigate such risks, data-driven approaches have been adopted to enhance
the precision of real estate price predictions. Traditional methods utilize
inherent attributes such as size, age, and location, coupled with simple
machine learning (ML) models such as linear regression (Şipoş et al., 2008;
Ahn et al., 2012), support vector machine (Smola and Schölkopf, 2004; Lin and
Chen, 2011), or multi-layer neural network (Peterson and Flanagan, 2009; Law
et al., 2019; Poursaeed et al., 2018; Wang et al., 2021; Chanasit et al.,
2021). However, these traditional approaches often fail to reflect complex
spatial relationships, such as the interactions between properties within
residential communities.

Recent studies have attempted to address this limitation by adopting graph
neural networks to model spatial relationships between properties (Bin et al.,
2019; Zhang et al., 2021; Li et al., 2022). These models represent spatial
relationships as a graph, with each node denoting a property. For example, in
MugRep (Zhang et al., 2021), nodes are connected based on geographical
proximity, incorporating neighborhood properties in their predictions. Despite
these advancements in spatial modeling, these studies often overlooked the
importance of temporal aspects in real estate appraisal. Property prices are
dynamic, influenced by market trends, economic policies, and community
developments over time. This temporal component is often insufficiently
represented (Zhang et al., 2021; Li et al., 2022) or entirely disregarded
(Pagourtzi et al., 2003; Peterson and Flanagan, 2009) in previous studies,
which could potentially lead to decreased prediction accuracy.

In response, we introduce ST-RAP, a spatio-temporal framework designed for
Real estate APpraisal. ST-RAP constitutes a hierarchical model with a temporal
and a spatial model (Lee et al., 2022). First, the temporal model captures the
dynamic changes in property values. Subsequently, the spatial model aggregates
the temporal model’s output, integrating the temporal information from
neighboring properties. Furthermore, different from previous studies, we
include amenities (e.g., schools, hospitals) and transportation stations as
nodes and employ a heterogeneous graph network to encapsulate the spatial
relationship between properties and these entities.

To verify the effectiveness of ST-RAP, we have manually collected 3.6 million
real estate transactions in the Republic of Korea from 2016 to 2020, along
with local amenities. Through comprehensive experiments, we reveal that ST-RAP
outperforms previous methods, with the incorporation of temporal trends in ST-
RAP notably contributing to enhanced performance.

Figure 1. The overall architecture of ST-RAP consists of 3 different models:
(a) temporal; (b) spatial, and (c) prediction models.

##  2\. Related Work

The existing literature on data-driven real estate appraisal largely falls
into two categories: (1) traditional automated valuation models (AVMs) and (2)
recent graph-based models. Traditional AVMs focus on property features such as
size, age, and location, and employ simple ML models such as linear regression
(Şipoş et al., 2008; Ahn et al., 2012), support vector machines (Smola and
Schölkopf, 2004; Lin and Chen, 2011), and neural networks (Peterson and
Flanagan, 2009; Law et al., 2019; Poursaeed et al., 2018; Wang et al., 2021;
Chanasit et al., 2021). Compared to methods that rely on domain expertise
(Baum et al., 2017; Guo et al., 2014; Gonzalez and Laureano-Ortiz, 1992; Betts
and Ely, 2020; Shiller and Weiss, 1999), AVMs are efficient and scalable,
improving their performance as more data becomes available. Despite their
advantages, they struggle to capture the spatial relationships between real
estate transactions, which can significantly enhance real estate appraisal.

To address this, recent studies have utilized graph neural networks (GNN) to
model spatial relationships between properties (Bin et al., 2019; Zhang et
al., 2021; Li et al., 2022; Xiao et al., 2023a; Xiao et al., 2023b). MugRep
(Zhang et al., 2021) constructs multiple graphs to capture the relationships
between individual transactions and residential communities (Veličković et
al., 2018). ReGram (Li et al., 2022) further employs an attention module to
account for regional differences.

Despite these advancements, prior GNN-based methods face challenges in
capturing the temporal trends (e.g., price inflation) in real estate values.
They attempt to integrate temporal trends by incorporating features such as
mean and variance of past transaction values as additional inputs.
Nevertheless, such an approach is insufficient for fully capturing the
complex, dynamic temporal pattern of real estate values. Furthermore, these
models fail to sufficiently leverage amenities and transportation data, which
are critical factors in real estate appraisal.

In response, we introduce ST-RAP, a spatiotemporal framework for real estate
appraisal. ST-RAP not only captures temporal dynamics but also includes a
heterogeneous GNN to encapsulate the relationships between transactions, and
local amenities.

##  3\. Preliminaries

###  3.1. Dataset

We construct a real estate dataset from the Republic of Korea (2016-2020),
which is categorized into four categories: Transaction, Resident, Amenity, and
Transportation, as shown in Table 1. We generate connections between entities
where the entities within a 5-kilometer radius are considered to be connected.

Table 1. Statistics of real estate dataset.

Category | # of events | # of features | # of connections to Resident
---|---|---|---
Transaction | 3,632,022 | 13 | -
Resident | 41,871 | 25 | 3,449,406
Amenity | 14,851 | 10 | 562,521
Transportation | 13,549 | 7 | 1,412,919

The Transaction category contains 3.6 million records, each defined by 13
attributes including transaction type, date, land area, and price. The
Resident category is composed of 25 features (e.g., location, number of
buildings, and the year of construction), with a total of 41,871 records. Each
resident is connected based on geographical proximity (total 3.4 million
edges). Additionally, our dataset comprises 15,000 amenities (schools,
hospitals, and department stores) and 13,500 transportation stations (train,
subway, and bus), generating 1.9 million connections to the residents.

In contrast to previous studies using 0.3 million transactions, our dataset
contains 3.6 million transactions that are planned to be publicized. We
believe that our dataset will serve as a valuable resource to promote research
in real estate appraisal.

###  3.2. Problem Formation

We represent a residential community as $r_{i}$, where $1\leq i\leq N_{r}$,
with $N_{r}$ being the total number of residents in the dataset. A transaction
event within resident $r_{i}$ is defined as $e_{i,t}$, with $1\leq t\leq
T_{i}$ and $T_{i}$ denotes the total number of transaction events within
resident $r_{i}$.

Each transaction event, $e_{i,t}$, is described by a feature vector
$ef_{i,t}\in\mathbb{R}^{d_{e}}$ and its price $p_{i,t}\in\mathbb{R}$, with
$d_{e}$ representing the dimension of the transaction feature. Hence, we
express $e_{i,t}$ as

(1) $e_{i,t}=\\{ef_{i,t},p_{i,t}\\}.$

For each residential community $r_{i}$, we define a resident feature vector as
$rf_{i}\in\mathbb{R}^{d_{r}}$ and a sequence of transaction events as
$h_{i,1:T_{i}}$, where $d_{r}$ denotes the dimension of the resident feature.
Resident $r_{i}$ is therefore represented as

(2) $\displaystyle r_{i,1:T_{i}}$ $\displaystyle=\\{rf_{i},h_{i,1:T_{i}}\\}$
(3) $\displaystyle h_{i,1:T_{i}}$
$\displaystyle=[e_{i,1},e_{i,2},...,e_{i,T_{i}}].$

In the sequence of transactions $h_{i,1:T_{i}}$, $e_{i,T_{i}}$ refers to the
latest transaction, while $e_{i,1}$ refers to the earliest transaction in
$r_{i}$.

Residents are connected based on geographical proximity, and we define the set
of connected residents to $r_{i}$ as:

(4) $R_{i}=\\{r_{v}\;|\;r_{v}\text{ is connected to }r_{i}\\}.$

Resident $r_{i}$ is also connected to nearby amenities and transportation
stations. Each amenity and station is represented as
$a_{j}\in\mathbb{R}^{d_{a}}$ and $s_{j}\in\mathbb{R}^{d_{s}}$, respectively,
where $d_{a}$ and $d_{s}$ denote the dimensions of the corresponding feature
vectors. The sets of connected amenities and transportation stations to
$r_{i}$ are defined as:

(5) $\displaystyle A_{i}=\\{a_{v}\;|\;a_{v}\text{ is connected to }r_{i}\\}$
(6) $\displaystyle S_{i}=\\{s_{v}\;|\;s_{v}\text{ is connected to }r_{i}\\}.$

Our goal is to predict the price $p_{i,t+1}$ of the next transaction event
$e_{i,t+1}$, given the transaction features $ef_{i,t+1}$. We seek to
incorporate spatiotemporal factors by utilizing the history of past
transaction events $r_{i,1:t}$ and connected entities, including residents
$R_{i}$, amenities $A_{i}$, and transportation stations $S_{i}$.

We train a function $f$ parameterized by $\theta$ to perform prediction:

(7) $\hat{p}_{i,t+1}=f_{\theta}(ef_{i,t+1},r_{i,1:t},R_{i},A_{i},S_{i}).$

##  4\. Method

This section presents ST-RAP, a hierarchical architecture with temporal and
spatial models. The temporal model captures the temporal trend of transaction
prices based on transaction history. Afterward, the spatial model aggregates
the output of the temporal model with nearby entities. The overall
architecture is illustrated in Figure 1.

###  4.1. Temporal model

Here, we describe the temporal model as illustrated in Figure.1(a).

Embedding. For each transaction event $e_{i,t}$, we linearly embed the price
and feature ($p_{i,t}$, $ef_{i,t}$) as $\tilde{p_{i,t}}\in\mathbb{R}^{d_{h}}$
and $\tilde{ef_{i,t}}\in\mathbb{R}^{d_{h}}$, where $d_{h}$ is a hidden
dimension of the embedding matrix. Then, we obtain the transaction embedding,
$\tilde{e}_{i,t}$, by summing up these vectors as

(8) $\tilde{e}_{i,t}=\tilde{p}_{i,t}+\tilde{ef}_{i,t}.$

Subsequently, for each transaction history $h_{i,1:t}$, the transaction
history embedding $\tilde{h}_{i,1:t}$ is denoted as

(9) $\tilde{h}_{i,1:t}=[\tilde{e}_{i,1},\tilde{e}_{i,2},...,\tilde{e}_{i,t}].$

Temporal modelling. After the embedding stage, the transaction history matrix,
$\tilde{h}_{i,1:t}$, is processed by a transition model to effectively capture
the temporal dynamics of the transaction events. Here, we employ a Gated
Recurrent Unit (GRU (Chung et al., 2014)) as the transition model. The output
of GRU is defined as

(10) $\tilde{h}^{\prime}_{i,1:t}=GRU(\tilde{h}_{i,1:t}).$

By adding the linearly embedded resident feature,
$\tilde{rf}_{i}\in\mathbb{R}^{d_{h}}$, to the last sequence of the GRU-
processed history embedding, $\tilde{h}^{\prime}_{i,t}$, we derive the
resident embedding that reflects temporal information.

(11) $\tilde{r}_{i}=\tilde{h}^{\prime}_{i,t}+\tilde{rf}_{i}.$

###  4.2. Spatial model

Here, we describe the spatial model as demonstrated in Figure.1(b).

Embedding. As described in the equations 4, 5, 6, the resident $r_{i}$ is
connected to the nearby residents $R_{i}$, amenities $A_{i}$, and
transportations $S_{i}$. Each resident $r_{v}\in R_{i}$ is embedded by a
temporal model as $\tilde{r}_{v}\in\mathbb{R}^{d_{h}}$. Then, each amenity
$a_{v}\in A_{i}$ and transportation station in $s_{v}\in S_{i}$ are linearly
embedded as $\tilde{a}_{v}\in\mathbb{R}^{d_{h}}$ and
$\tilde{s}_{v}\in\mathbb{R}^{d_{h}}$.

Spatial modelling. After embedding, we aggregate the features of nearby
residents, amenities, and transportation stations to capture the diverse
spatial relationships between these entities. Given the heterogeneity of the
entity types, we utilize a Heterogeneous Graph Neural Network (HGNN (Zhang et
al., 2019)) to construct a comprehensive representation of spatial features.

Given the resident vector $\tilde{r}_{i}$ and its neighbor entities
$\tilde{R}_{i},\tilde{A}_{i}$, and $\tilde{S}_{i}$, the HGNN is constructed
as:

(12) $\displaystyle\bar{r}_{i}$
$\displaystyle=W_{rr}(\sigma(W_{rv}\tilde{r}_{i}+W_{vr}\cdot\text{mean}_{\tilde{r}_{v}\in\tilde{R}_{i}}\tilde{r}_{v}))+b_{rr}$
(13) $\displaystyle\bar{a}_{i}$
$\displaystyle=W_{ar}(\sigma(W_{av}\tilde{r}_{i}+W_{va}\cdot\text{mean}_{\tilde{a}_{v}\in\tilde{A}_{i}}\tilde{a}_{v}))+b_{ar}$
(14) $\displaystyle\bar{s}_{i}$
$\displaystyle=W_{sr}(\sigma(W_{sv}\tilde{r}_{i}+W_{vs}\cdot\text{mean}_{\tilde{s}_{v}\in\tilde{S}_{i}}\tilde{s}_{v}))+b_{sr}$

where $\sigma$ denotes the ReLU activation function. The shape of all weight
matrices and bias vectors are identical to $W\in\mathbb{R}^{d_{h}\times
d_{h}}$ and $b\in\mathbb{R}^{d_{h}}$. Here, the output is defined as
$\bar{r}_{i}\in\mathbb{R}^{d_{h}},\bar{a}_{i}\in\mathbb{R}^{d_{h}},\bar{s}_{i}\in\mathbb{R}^{d_{h}}$.

Prediction. Finally, to predict the price of the transaction event
$e_{i,{t+1}}$, we construct the state $x_{i,t+1}$ by aggregating the
transaction embedding $\tilde{ef}_{i,t+1}$ with the spatiotemporal features
$\bar{r}_{i},\bar{a}_{i},\bar{s}_{i}$ as

(15) ${x}_{i,t+1}=\tilde{ef}_{i,t+1}+\bar{r}_{i}+\bar{a}_{i}+\bar{s}_{i}.$

Then, we predict the price using a feed-forward network as

(16) $\hat{p}_{i,t+1}=W_{hp}(\sigma(W_{xh}x_{i,{t+1}}+b_{xh}))+b_{hp}$

where $\sigma$ denotes the ReLU activation, $W_{xh}\in\mathbb{R}^{d_{h}\times
d_{h}},W_{hp}\in\mathbb{R}^{d_{h}\times 1}$ are weight matrices, and
$b_{xh}\in\mathbb{R}^{d_{h}},b_{hp}\in\mathbb{R}$ are bias terms.

Training objective. The training objective is to minimize the mean absolute
error (MAE) between the predicted price $\hat{p}_{i,t}$, and the actual price
$p_{i,t}$. We define the loss function $\mathcal{L}$ as the average error
across all residents and all transaction events, written as

(17)
$\mathcal{L}=\frac{1}{N_{r}T_{i}}\sum_{i=1}^{N_{r}}\sum_{t=1}^{T_{i}}|p_{i,t}-\hat{p}_{i,t}|.$

By optimizing this objective, the model learns to make accurate predictions
given the real estate dataset.

##  5\. Experiments

###  5.1. Experimental Setup

We split the dataset into training, validation, and test sets in chronological
order with a ratio of 90:5:5. We trained the model in the training set for 10
epochs, and used the validation set to select the best models. We use Mean
Absolute Error (MAE), Root Mean Squared Error (RMSE), and Mean Absolute
Percentage Error (MAPE) for evaluation. For MAE and RMSE, the unit price is
1,000,000 KRW. For all the metrics, the lower score indicates a better
performance.

Baselines. We consider the following baselines: (i) two classical machine
learning (ML) baselines (Linear Regression and Support Vector Regression)
(Pedregosa et al., 2011), (ii) a rule-based model Repeat which simply outputs
the previous transaction price in the same residential community, (iii) three
graph-based models (GCN (Kipf and Welling, 2017), MugRep (Zhang et al., 2021)
and ReGram (Li et al., 2022)) which captures the spatial dependency between
real estate transactions and residential communities.

Implementation Details. For all the methods, we tuned the hyperparameters
using the k-fold cross-validation $(K=5)$. We used the AdamW (Loshchilov and
Hutter, 2017) with varying learning rates of $\\{0.1,0.01,0.001\\}$ and
$L_{2}$ weight regularization strength from $\\{0.01,0.001,0.0001\\}$. We set
the batch size to 128, weight decay to 1e-5, and gradient clipping to 0.5 and
trained for 100 epochs. For MugRep and ReGram, we search the number of
attention heads from {1,2,4,8}.

Table 2. Performance comparison for real estate appraisal. The results are
averaged over 5 random seeds.

Method | MAE (M) $\downarrow$ | RMSE (M) $\downarrow$ | MAPE $\downarrow$
---|---|---|---
LR | 105.09 $\pm$1.52 | 194.59 $\pm$1.18 | 65.62% $\pm$1.05%
SVR | 90.31 $\pm$3.30 | 137.96 $\pm$1.99 | 53.22% $\pm$2.21%
Repeat | 46.01 $\pm$0.00 | 97.55 $\pm$0.00 | 28.67% $\pm$0.00%
GCN (Kipf and Welling, 2017) | 50.17 $\pm$2.19 | 94.13 $\pm$3.45 | 30.87% $\pm$0.02%
MugRep (Zhang et al., 2021) | 35.46 $\pm$2.43 | 74.08 $\pm$3.64 | 22.29% $\pm$0.01%
ReGram (Li et al., 2022) | 34.61 $\pm$4.61 | 64.48 $\pm$5.03 | 21.81% $\pm$0.03%
ST-RAP | 21.77 $\pm$0.78 | 48.03 $\pm$1.70 | 13.72% $\pm$0.05%

###  5.2. Main Results

Table 2 provides the comparative analysis. Comparing regression models (LR,
SVR) to the rule-based model (Repeat), Repeat significantly outperformed
regression models. Repeat can be seen as the simplest version of a temporal
model, as it outputs the price of the most recent transaction within a
residential community. This notable gap implies that the integration of
temporal information is crucial in accurately appraising real estate prices.

In addition, we observed that GNN-based models (GCN, MugRep, and Regram),
outperform non-spatial models (LR and SVR) across all the evaluation metrics.
These findings emphasize the significance of incorporating spatial information
for real estate appraisal.

Most importantly, our proposed model, ST-RAP, demonstrates a remarkable
improvement over all the comparative models. ST-RAP integrates both temporal
and spatial dependencies among residents, amenities, and transportation
stations. This improvement affirms the effectiveness of reflecting
spatiotemporal factors and the use of heterogeneous graphs for an accurate
real estate appraisal.

###  5.3. Ablation Studies

Architecture. We conducted an architecture ablation study by separating the
spatial (S-RAP) and temporal (T-RAP) components of ST-RAP. The transactions
are classified into three groups for this analysis: ALL, which includes all
transactions, the COLD group, representing transactions without any historical
records within the residents, and the WARM group, representing the remaining
transactions with historical records.

Table 3. MAE of S-RAP, T-RAP, and ST-RAP for all transactions (ALL),
transactions without any historical records within residents (COLD), and the
others (WARM).

Mod | ALL $\downarrow$ | COLD $\downarrow$ | WARM $\downarrow$
---|---|---|---
S-RAP | 26.98 $\pm$0.39 | 212.20 $\pm$36.86 | 26.23 $\pm$0.37
T-RAP | 22.62 $\pm$0.54 | 230.52 $\pm$12.22 | 21.99 $\pm$0.54
ST-RAP | 21.77 $\pm$0.78 | 189.42 $\pm$12.68 | 21.21 $\pm$0.76

Table 3 shows that ST-RAP outperforms both S-RAP and T-RAP across all metrics,
emphasizing the importance of integrating both temporal trends and spatial
factors in the model.

Figure 2. Error distribution of S-RAP, T-RAP, and ST-RAP. The diameter
indicates the number of transaction histories.

ST-RAP’s improved performance over S-RAP underscores the importance of
temporal dynamics. In contrast, the sharp performance drop for T-RAP in the
COLD group highlights the importance of spatial modeling when temporal data is
scarce. This is further illustrated in Figure 2, where S-RAP fails to provide
accurate predictions even with an abundance of historical transaction records,
and T-RAP’s performance diminishes in the absence of such records.

These results reiterate the necessity of incorporating both spatial and
temporal aspects for precise real estate appraisal.

Table 4. Ablation of the maximum history length.

History Len | 1 | 2 | 5 | 10 | 20 | 30 | 50
---|---|---|---|---|---|---|---
MAE (M) | 26.98 | 24.73 | 22.59 | 22.24 | 21.93 | 21.77 | 21.89

History Length. To evaluate the importance of temporal information on
transaction value prediction, we conduct an ablation study varying the
sequence length (i.e., the number of previous transactions in the residential
community). Table 4 displays the performance based on the sequence length for
GRU. For both ST-RAP and T-RAP, which consider temporal information,
performance improves as the sequence length increases.

Table 5. Ablation of the temporal model.

Architecture | MAE (M) $\downarrow$ | RMSE (M) $\downarrow$ | MAPE $\downarrow$
---|---|---|---
Transformer | 24.35 $\pm$0.71 | 54.73 $\pm$0.43 | 15.32% $\pm$0.44%
GRU | 21.77 $\pm$0.78 | 48.03 $\pm$1.70 | 13.72% $\pm$0.05%

Temporal Model. While transformer-based models have shown impressive
capabilities in handling unstructured time-series data such as language, their
performance has been found to be suboptimal when applied to structured time-
series data (Xu et al., 2020; Cirstea et al., 2022; Li et al., 2019; Zhou et
al., 2021; Wen et al., 2023; Wu et al., 2021; Zeng et al., 2022). Replacing
our GRU-based temporal model with a transformer led to a drop in performance,
as depicted in Table 5. This observation underscores the need for an inductive
bias that transformers may lack when dealing with structured temporal trends.

##  6\. Conclusion

We introduced ST-RAP, a novel spatial-temporal framework for real estate
appraisal. By integrating the temporal dynamics via a recurrent model and
spatial relationships through a heterogeneous graph model, ST-RAP
significantly outperforms previous baselines which underscore the importance
of reflecting the spatiotemporal factors in real estate appraisal. In
addition, we plan to publicize both our code and dataset to foster future
research in this field.

##  7\. Acknowledgements

This work was supported by Woomi Construction Co. Ltd. (G01210242), Institute
for Information & communications Technology Planning & Evaluation(IITP) grant
funded by the Korea government(MSIT) (No. 2020-0-00368, A Neural-Symbolic
Model for Knowledge Acquisition and Inference Techniques), and the National
Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)
(No. NRF-2022R1A2B5B02001913).

## References

  * (1)
  * Ahn et al. (2012) Jae Joon Ahn, Hyun Woo Byun, Kyong Joo Oh, and Tae Yoon Kim. 2012\.  Using ridge regression with genetic algorithm to enhance real estate appraisal forecasting.  _Expert Systems with Applications_ 39, 9 (2012), 8369–8379.
  * Baum et al. (2017) Andrew Baum, David Mackmin, and Nick Nunnington. 2017\.  _The Income Approach to Property Valuation_ (7th. ed.).  Routledge.
  * Betts and Ely (2020) R Betts and Silas Ely. 2020\.  _Basic Real Estate Appraisal_ (5th. ed.).  South-Western.
  * Bin et al. (2019) Junchi Bin, Bryan Gardiner, Eric Li, and Zheng Liu. 2019\.  Peer-dependence valuation model for real estate appraisal.  _Data-Enabled Discovery and Applications_ 3 (2019), 1–11.
  * Chanasit et al. (2021) Kankawee Chanasit, Ekapol Chuangsuwanich, Atiwong Suchato, and Proadpran Punyabukkana. 2021.  A real estate valuation model using boosted feature selection.  _IEEE Access_ 9 (2021), 86938–86953.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ (2014).
  * Cirstea et al. (2022) Razvan-Gabriel Cirstea, Chenjuan Guo, Bin Yang, Tung Kieu, Xuanyi Dong, and Shirui Pan. 2022.  Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting–Full Version.  _Proc. the International Joint Conference on Artificial Intelligence (IJCAI)_ (2022).
  * Demyanyk and Van Hemert (2011) Yuliya Demyanyk and Otto Van Hemert. 2011.  Understanding the subprime mortgage crisis.  _The review of financial studies_ 24, 6 (2011), 1848–1880.
  * Gonzalez and Laureano-Ortiz (1992) Avelino J. Gonzalez and Raymond Laureano-Ortiz. 1992.  A case-based reasoning approach to real estate property appraisal.  _Expert Systems with Applications_ 4, 2 (1992), 229–246.
  * Guo et al. (2014) Jingjuan Guo, Shoubo Xu, and Zhuming Bi. 2014.  An integrated cost-based approach for real estate appraisals.  _Information Technology and Management_ 15 (2014), 131–139.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-Supervised Classification with Graph Convolutional Networks.  _Proc. the International Conference on Learning Representations (ICLR)_ (2017).
  * Law et al. (2019) Stephen Law, Brooks Paige, and Chris Russell. 2019\.  Take a look around: using street view and satellite images to estimate house prices.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ 10, 5 (2019), 1–19.
  * Lee et al. (2022) Hojoon Lee, Dongyoon Hwang, Hyunseung Kim, Byungkun Lee, and Jaegul Choo. 2022.  Draftrec: personalized draft recommendation for winning in multi-player online battle arena games. In _Proceedings of the ACM Web Conference 2022_. 3428–3439.
  * Li et al. (2022) Chih-Chia Li, Wei-Yao Wang, Wei-Wei Du, and Wen-Chih Peng. 2022\.  Look Around! A Neighbor Relation Graph Learning Framework for Real Estate Appraisal.  _Proc. the AAAI Conference on Artificial Intelligence (AAAI)_ (2022).
  * Li et al. (2019) Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2019.  Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting.
  * Lin and Chen (2011) Hongyu Lin and Kuentai Chen. 2011.  Predicting price of Taiwan real estates by neural networks and support vector regression. In _Proc. of the 15th WSEAS Int. Conf. on Syst_. 220–225.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ (2017).
  * Pagourtzi et al. (2003) Elli Pagourtzi, Vassilis Assimakopoulos, Thomas Hatzichristos, and Nick French. 2003.  Real estate appraisal: a review of valuation methods.  _Journal of property investment & finance_ 21, 4 (2003), 383–401.
  * Pavlov and Wachter (2011) Andrey Pavlov and Susan Wachter. 2011.  Subprime lending and real estate prices.  _Real Estate Economics_ 39, 1 (2011), 1–17.
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake VanderPlas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. 2011.  Scikit-learn: Machine Learning in Python.  _Journal of Machine Learning Research_ 12 (2011).
  * Peterson and Flanagan (2009) Steven Peterson and Albert Flanagan. 2009.  Neural network hedonic pricing models in mass real estate appraisal.  _Journal of real estate research_ 31, 2 (2009), 147–164.
  * Poursaeed et al. (2018) Omid Poursaeed, Tomáš Matera, and Serge Belongie. 2018.  Vision-based real estate price estimation.  _Machine Vision and Applications_ 29, 4 (2018), 667–676.
  * Shiller and Weiss (1999) Robert J. Shiller and Allan N. Weiss. 1999.  Evaluating Real Estate Valuation Systems.  _The Journal of Real Estate Finance and Economics_ 18 (1999), 147–161.
  * Şipoş et al. (2008) Ciprian Şipoş, Eng Adrian Crivii, and MBA FRICS. 2008\.  A linear regression model for real estate appraisal. In _WAVO Valuation Congress Valuation in Diversified and Emerging Economies_. 17–18.
  * Smola and Schölkopf (2004) Alex J Smola and Bernhard Schölkopf. 2004.  A tutorial on support vector regression.  _Statistics and computing_ 14 (2004), 199–222.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph Attention Networks.  _Proc. the International Conference on Learning Representations (ICLR)_ (2018).
  * Wang et al. (2021) Pei-Ying Wang, Chiao-Ting Chen, Jain-Wun Su, Ting-Yun Wang, and Szu-Hao Huang. 2021.  Deep learning model for house price prediction using heterogeneous data analysis along with joint self-attention mechanism.  _IEEE Access_ 9 (2021), 55244–55259.
  * Wen et al. (2023) Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. 2023.  Transformers in Time Series: A Survey.  _Proc. the International Joint Conference on Artificial Intelligence (IJCAI)_ (2023).
  * Wu et al. (2021) Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021\.  Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting.  _Proc. the Advances in Neural Information Processing Systems (NeurIPS)_.
  * Xiao et al. (2023a) Congxi Xiao, Jingbo Zhou, Jizhou Huang, Tong Xu, and Hui Xiong. 2023a.  Spatial Heterophily Aware Graph Neural Networks.  _arXiv preprint arXiv:2306.12139_ (2023).
  * Xiao et al. (2023b) Congxi Xiao, Jingbo Zhou, Jizhou Huang, Hengshu Zhu, Tong Xu, Dejing Dou, and Hui Xiong. 2023b.  A contextual master-slave framework on urban region graph for urban village detection. In _2023 IEEE 39th International Conference on Data Engineering (ICDE)_. IEEE, 736–748.
  * Xu et al. (2020) Mingxing Xu, Wenrui Dai, Chunmiao Liu, Xing Gao, Weiyao Lin, Guo-Jun Qi, and Hongkai Xiong. 2020.  Spatial-temporal transformer networks for traffic flow forecasting.  _arXiv preprint arXiv:2001.02908_ (2020).
  * Zeng et al. (2022) Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2022\.  Are Transformers Effective for Time Series Forecasting?  _Proc. the AAAI Conference on Artificial Intelligence (AAAI)_.
  * Zhang et al. (2019) Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. 2019.  Heterogeneous graph neural network. In _Proc. the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)_. 793–803.
  * Zhang et al. (2021) Weijia Zhang, Hao Liu, Lijun Zha, Hengshu Zhu, Ji Liu, Dejing Dou, and Hui Xiong. 2021.  MugRep: A multi-task hierarchical graph representation learning framework for real estate appraisal. In _Proc. the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)_. 3937–3947.
  * Zhou et al. (2021) Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021.  Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting.  _Proc. the AAAI Conference on Artificial Intelligence (AAAI)_ (2021).