# Bridging the Domain Gap: A Simple Domain Matching Method for Reference-based
Image Super-Resolution in Remote Sensing

Jeongho Min, Yejun Lee, Dongyoung Kim, Jaejun Yoo Jeongho Min, Yejun Lee,
Dongyoung Kim and Jaejun Yoo† are with the Graduate School of Artificial
Intelligence, Ulsan National Institute of Science and Technology, Ulsan,
44919, Republic of Korea e-mail: {jeongho.min, yejun.lee, dykim94,
<EMAIL_ADDRESS>: Corresponding author

###### Abstract

Recently, reference-based image super-resolution (RefSR) has shown excellent
performance in image super-resolution (SR) tasks. The main idea of RefSR is to
utilize additional information from the reference (Ref) image to recover the
high-frequency components in low-resolution (LR) images. By transferring
relevant textures through feature matching, RefSR models outperform existing
single image super-resolution (SISR) models. However, their performance
significantly declines when a domain gap between Ref and LR images exists,
which often occurs in real-world scenarios, such as satellite imaging. In this
letter, we introduce a Domain Matching (DM) module that can be seamlessly
integrated with existing RefSR models to enhance their performance in a plug-
and-play manner. To the best of our knowledge, we are the first to explore
Domain Matching-based RefSR in remote sensing image processing. Our analysis
reveals that their domain gaps often occur in different satellites, and our
model effectively addresses these challenges, whereas existing models
struggle. Our experiments demonstrate that the proposed DM module improves SR
performance both qualitatively and quantitatively for remote sensing super-
resolution tasks.

###### Index Terms:

Reference-based Image Super Resolution, Domain Adaptation, Remote Sensing

##  I Introduction

Improving the resolution of satellite imagery can enhance the performance of
sub-tasks such as semantic segmentation [1], object detection [2], change
detection [3], and data fusion [4]. It can also find active utilization in
military applications or the assessment of damage extent resulting from
natural disasters. However, obtaining high-resolution satellite data
inherently comes with significant costs, particularly in terms of hardware
investments. As a result, the software-based improvements achieved by applying
super-resolution techniques on satellite data hold immense potential for
generating substantial economic value.

Single image super-resolution (SISR) methods that reconstruct high-resolution
(HR) images from a single low-resolution (LR) image have demonstrated
excellent performance in remote sensing tasks [5, 6, 7, 8]. However, SISR
algorithms face challenges when applied to images outside of the training
data, since these algorithms are often trained on specific types of image
distributions. To address these challenges, numerous works [9, 10] have been
proposed to enhance the efficacy of SISR methodologies. However, SISR models
still have a fundamental limitation in that they only rely on a single LR
image to estimate the HR image.

Unlike SISR models, reference-based super-resolution (RefSR) models
incorporate an additional high-resolution reference (Ref) image. A typical
RefSR framework consists of two major parts: correspondence matching and
texture transfer. Correspondence matching involves finding the most similar
correspondence between the input and Ref images, while texture transfer
entails transferring features from the reference image to the input feature
based on their similarity. By exploiting the HR Ref images that are rich in
texture and high-frequency information, RefSR models generally have shown
superior performance than the SISR models [11, 12, 13, 14, 15, 16, 17] on the
benchmark dataset such as CUFED5 dataset [12].

However, the CUFED5 dataset is constructed by pairing closely matched images
together, where the input and reference images can be considered as aligned
paired data. It is important to note that such closely matched data scenarios
are not commonly encountered in real-world situations. Moreover, this real-
world data, acquired from different devices, presents additional challenges
for the RefSR task compared to the idealized scenarios. We aim to perform
super-resolution tasks using a real-world dataset, specifically a remote
sensing dataset. One advantage of the remote sensing dataset is that we have
access to paired images captured by different satellites on the same
geographical region [17]. This allows us to utilize the reference images from
the satellites to enhance the resolution of the input images through super-
resolution techniques. However, there exists a large domain gap caused by
variations in shooting angles, capture times, and other factors due to
acquisition from different satellites. Consequently, there is no assurance
that RefSR models will accurately match and effectively transfer genuinely
informative textures in such situations.

To address this challenge, we propose a simple but effective method to reduce
the domain gap between LR and Ref images. Specifically, we propose to perform
a grayscale transformation on the input and Ref images to enhance the matching
performance while preserving structural information. We then employ a style
transfer-based approach to generate a reference image that is adapted to the
distribution of the input image, enabling more effective texture transfer.
Moreover, we leverage the renowned Whitening and Coloring Transform (WCT), a
well-established method in the field of style transfer, to facilitate seamless
texture transfer. However, since WCT is developed for artistic style transfer,
it is not designed to preserve structural information. To avoid such
distortion, we incorporate the Phase Replacement (PR) technique into the
process, ensuring the preservation of structural information, which is crucial
for our SR task.

As demonstrated by both quantitative and qualitative evaluations, our method
brings remarkable enhancements over existing state-of-the-art (SOTA) RefSR
models. Our results highlight the efficacy of our proposed method in
mitigating the domain gap problem and improving the overall performance of
RefSR in real-world scenarios, specifically for electro-optical imagery. Our
contributions are summarized as follows:

  * •

We provide a simple but effective method that can super-resolve an image well
even when the Ref image exhibits a different distribution from the input
image.

  * •

Our method has significantly improved existing models with 0.03 to 0.16 in
PSNR for SR performance and 0.1 to 0.15 in SSIM-S for matching performance.

  * •

Our method requires no extra training, making it easy to integrate into
existing RefSR models.

Figure 1: Overview We applied a grayscale transform to the input images before
feeding it into the matching encoder during the correspondence matching
process. In the texture transfer process, $F^{t}_{Ref}$ is the domain adapted
by $F^{t}_{LR}$ to enhance the information of transferring textures. In this
experiment, the domain adaptation module indicates WCT with phase replacement
technique. $EO_{A}$ and $EO_{B}$ refer to electro-optical images captured by
different satellites.

##  II Analysis

In this section, we undertake a visual examination to assess the effectiveness
of the correspondence matching process in the existing RefSR models and
investigate the impact of the DM module on the matching process. Surprisingly,
contrary to common belief, our findings reveal that the existing RefSR models
demonstrate poor matching performance when applied to remote sensing datasets,
which have a large domain gap.

Investigation of the matching quality. In RefSR, the correspondence matching
process plays a crucial role in determining its final super-resolution (SR)
performance. However, to the best of our knowledge, no previous study has
conducted a comprehensive analysis of the matching results. Previous studies
[12, 14, 15, 16] have typically evaluated the impact of the correspondence
matching process indirectly, through simple ablation studies tied to the final
SR performance. However, improvements in the final SR performance may not
necessarily stem from a superior matching module; other factors such as
increased model capacity could also contribute.

To directly assess the quality of matching and its influence on the final SR
performance, we propose to undertake a detailed examination of the matching
process and visualize the results for clarity (Figure 2). The correspondence
matching process is based on cosine similarity, which involves extracting the
index and score of the Ref image that corresponds to each pixel in the input
image. If the matched feature from the Ref image gives useful information, SR
performance would ideally increase. In the case of EO data, the acquired data
originates from the same location. Assuming minimal terrain variations and an
ideally performed matching process, it is expected that patches from the same
location would be successfully matched. Consequently, the matching result
should ideally reflect the structural characteristics of the LR image.
Therefore, evaluating the structural similarity between the matching result
and the LR image becomes a means to assess the performance of correspondence
matching and determine how well it preserves the structural properties.

To evaluate the quality of the correspondence matching, we utilize the
structure component $s$ of $\text{SSIM}(x,y)=l(x,y)c(x,y)s(x,y)$, which we
denote as SSIM-s:

$\displaystyle\text{SSIM-s}:s(x,y)=\frac{\sigma_{xy}+C_{3}}{\sigma_{x}\sigma_{y}+C_{3}},~{}~{}C_{3}=\frac{(0.03L)^{2}}{2}$
(1)

where $x$ and $y$ indicates LR image and matching result respectively, $l$ is
luminance, $c$ is contrast, $\sigma_{x}^{2}$ and $\sigma_{y}^{2}$ are the
variances of $x$ and $y$, and $\sigma_{xy}$ is the covariance between $x$ and
$y$. Additionally, $L$ denotes the data range. A higher $s(x,y)$ value
indicates that the two images have more similar structures or spatial
dependencies. To the best of our knowledge, we are the first to explicitly
evaluate the actual matching quality.

Figure 2:  Visualization of matching results of DATSR with and without
grayscale transformation in the corresponding matching process. The matching
result indicates the outcome after identifying the most relevant reference
feature to the input, implying that a better matching is achieved as it
becomes closer to the input. Our approach demonstrates a much cleaner and more
effective retrieval of detailed information compared to the existing baseline.

Visual interpretation of the matching procedure. The result of the matching
process in Figure 2 is generated by reorganizing patches from the reference
image based on these indices. Contrary to the widely held notion that
correspondence matching modules should locate similar information, the
structure of the matched results is fragmented, while clean matching results
are obtained when grayscale transformation is used (Figure 2). We hypothesize
that this result may be due to the visually disparate distributions (e.g.,
color, brightness, contrast) introduced by different acquisition devices,
despite them capturing the same scene.

To validate our hypothesis, we develop a simple adjustment, a grayscale
transformation, aimed at reducing basic visual disparity. As shown in Figure
2, the application of this grayscale transformation, which we term as Gray
Matching, yields visually less noisy results compared to the baseline and more
accurately reflects the structure information inherent in the LR image. This
outcome indicates the performance improvement in the matching process achieved
by implementing this simple adjustment. Later, we show that this enhancement
in correspondence matching indeed translates into the performance enhancement
of the final SR outcome (Table III).

##  III Method

Inspired by our prior analysis, we propose Domain Matching (DM) modules that
aim to minimize the domain gap between the source and reference images, while
preserving crucial structural information: Gray Matching, Whitening and
Coloring Transform (WCT), and Phase Replacement (PR).

Gray Matching. To reduce the domain gap in the correspondence matching step,
we propose to introduce grayscale transformation for input images. The
grayscale transformation involves averaging three channels to create a single-
channel image before it is fed into the encoder as input, prior to the
Matching encoder process. Applying grayscale transformation to an image
effectively eliminates color information, leaving the structural information
of the image. By this simple correction, we find that the impact of the
distribution gap (e.g., hue, brightness) between two images is reduced
effectively, and the matching performance is improved.

Whitening and Coloring Transform (WCT). To reduce the domain gap in the
texture transfer step, we use Whitening and Coloring Transform (WCT) [18]. WCT
matches the covariance matrix of the content feature map to the style feature
map from the pre-trained VGG network:

$f_{c}\leftarrow
f_{c}-m_{c}~{},~{}~{}\hat{f}_{c}=E_{c}D_{c}^{-\frac{1}{2}}E_{c}^{\top}f_{c}$
(2)
$\hat{f_{cs}}=E_{s}D_{s}^{\frac{1}{2}}E_{s}^{\top}\hat{f_{c}}~{},~{}~{}\hat{f}_{cs}\leftarrow\hat{f}_{cs}+m_{s}$
(3)

$f_{c}$ and $f_{s}$ are feature maps of content images and style images from a
certain layer of the VGG network. 1) Whitening : Whitening Transform is a
linear transformation to make the covariance matrix of a feature map to the
identity matrix. $D$ is a diagonal matrix for scaling, and $E$ is the
corresponding orthogonal matrix of eigenvectors for rotation. $f_{c}$ is
centered data by subtracting the sample mean $m_{c}$. 2) Coloring : Coloring
Transform is a reverse of a whitening process. It transforms the whitened data
to have a specific covariance matrix. $E_{s}$ and $D_{s}$ are an orthogonal
eigenvector matrix and diagonal matrix of $f_{s}f_{s}\top$. Then, by adding
the mean of a style feature $m_{s}$ to $f_{cs}$, the output gets stylized of
$f_{s}$.

Stylized feature map modification via Fourier perspective. Directly applying
style transfer techniques may damage the structural information of the HR Ref
image. To address this issue, we propose the Phase Replacement (PR) technique.
Following [19], we preserve the phase information of the content feature map
while keeping the amplitude of the stylized feature map during the WCT
process. This approach allows us to retain the structural information of the
image [19]:

${\cal F}^{cs}_{u,v}=|{\cal F}|^{cs}_{u,v}\odot\cos\angle{\cal
F}^{c}_{u,v}+j|{\cal F}|^{cs}_{u,v}\odot\sin\angle{\cal F}^{c}_{u,v}.$ (4)

Here, $\odot$ denotes element-wise multiplication, and ${\cal F}^{cs}$ and
${\cal F}^{c}$ denote discrete Fourier transformed stylized feature map and
content feature map, respectively. By utilizing this technique, we can
effectively narrow down the domain gap between input and reference features
while preserving the structural information, which eventually allows the RefSR
model to achieve an improved SR performance.

Figure 3: Qualitative comparison of baseline RefSR models and proposed
methods. Our proposed method shows better performance compared to previous
SOTA models and recovers more details and shows good visual quality.

##  IV Experiments

###  IV-A Experimental settings

Baseline models. We use C2-Matching, AMSA, and DATSR as our baseline. We
examine the SR performance on $\times 4$ scale when adding our domain matching
modules into these models. For training specifics and hyperparameter
configurations, we follow the setting of C2-Matching [14].

Evaluation Metrics. We measured the performance of the models using PSNR and
SSIM, specifically in the Y channel of the YCbCr color space. Higher PSNR and
SSIM are interpreted as better model performance.

Datasets. We utilize the RRSSRD dataset to train our RefSR models. The RRSSRD
dataset comprises electro-optical data acquired from satellites and includes
paired data captured from different satellites of the same scene. The training
set consists of 4,047 paired 480×480 sized images. HR images in the dataset
consist of Gaofen-2, WorldView, and Microsoft Virtual Earth 2018, while Ref
images are sourced from Google Earth.

###  IV-B Results on the dataset with domain gap

We experiment with our baseline RefSR models (C2-Matching, AMSA, DATSR) under
the existence of domain gaps. In an ideal scenario, the domain matching module
should exhibit comparable performance to the baseline model in the absence of
a domain gap, while demonstrating greater robustness than the baseline model
in situations where a domain gap exists (i.e., exhibiting lower performance
degradation). To verify this, we conduct experiments in remote sensing
datasets. For the models, we trained them separately and evaluated them on
their own test sets.

Qualitative results. In Figure 3, we apply our proposed domain matching module
to the three baseline models and compared their visual results. We test them
on real-world images from the RRSSRD dataset, which has a significant domain
gap between input and reference images. As we can see, the baseline models
fail to reduce the domain gap, resulting in blurry outputs or the inability to
successfully reconstruct textures. However, by plugging in our proposed domain
matching module, we successfully reduce the domain gap and generate visually
appealing results for all models.

Quantitative Results. Table I presents the quantitative results, showing that
our method significantly improves the performance of all state-of-the-art
models on the RRSSRD dataset across all testing sets. Our method shows good
generalization and easy applicability to different domains with a large domain
gap, as evidenced by its strong performance on the 2nd and 4th testing sets.
Furthermore, the comparison between C2-matching and DATSR, reveals that the
latest SOTA model falls short of the RRSSRD dataset. This highlights the
importance of addressing the domain gap in real-world datasets. Table II shows
the comparison of SSIM-s when performing the corresponding matching with and
without our method. This measures how well the reference matches the input in
terms of similarity. This again demonstrates that improved matching
performance translates into the performance enhancement of final SR outputs.

TABLE I:  Quantitative comparison on RRSSRD Testset

Methods | Params | 1st test set | 2nd test set | 3rd test set | 4th test set
---|---|---|---|---|---
PSNR$\uparrow$ |  SSIM$\uparrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$
C2-Matching | 8.9M | 34.05 | 0.891 | 33.13 | 0.860 | 31.19 | 0.827 | 33.37 | 0.847
\+ Proposed | +0.03 | +0.000 | +0.10 | +0.001 | +0.14 | +0.003 | +0.12 | +0.001
AMSA | 9.7M | 34.12 | 0.891 | 33.16 | 0.860 | 31.24 | 0.828 | 33.43 | 0.847
\+ Proposed | +0.04 | +0.001 | +0.05 | +0.001 | +0.07 | +0.002 | +0.07 | +0.002
DATSR | 18.9M | 33.98 | 0.890 | 33.03 | 0.858 | 31.13 | 0.826 | 32.28 | 0.845
\+ Proposed | +0.12 | +0.001 | +0.16 | +0.003 | +0.18 | +0.003 | +0.16 | +0.002

TABLE II: SSIM and SSIM-s comparison on RRSSRD

Methods | Metric | 1st test set | 2nd test set | 3rd test set | 4th test set
---|---|---|---|---|---
DATSR |  SSIM $\uparrow$ | 0.2306 | 0.2372 | 0.2131 | 0.2195
SSIM-s $\uparrow$ | 0.3665 | 0.3771 | 0.3347 | 0.3170
Ours |  SSIM $\uparrow$ | +0.1256 | +0.1056 | +0.1523 | +0.1468
SSIM-s $\uparrow$ | +0.1123 | +0.0981 | +0.1400 | +0.1374

###  IV-C Ablation study

In this section, we demonstrate the effectiveness of incorporating our module
into the baseline model. As shown in Table III and Figure 2, by utilizing Gray
Matching in the corresponding matching stage, matching performance is enhanced
compared to DATSR both qualitatively and quantitatively. Applying both methods
including WCT and PR demonstrates a significant improvement in the results.
Experimental results show that our method improves the performance of SOTA
RefSR models.

TABLE III:  Ablation study on our module on RRSSRD Testset

Methods | Metric | 1st test set | 2nd test set | 3rd test set | 4th test set
---|---|---|---|---|---
DATSR (Baseline) |  PSNR$\uparrow$ | 33.98 | 33.03 | 31.13 | 33.28
SSIM$\uparrow$ | 0.890 | 0.858 | 0.826 | 0.845
\+ Gray Matching |  PSNR$\uparrow$ | +0.08 | +0.1 | +0.15 | +0.13
SSIM$\uparrow$ | +0.000 | +0.002 | +0.003 | +0.002
\+ $WCT_{PR}$ |  PSNR$\uparrow$ | +0.10 | +0.09 | +0.06 | +0.10
SSIM$\uparrow$ | +0.001 | +0.001 | +0.000 | +0.001
\+ Gray Mat. $AdaIN_{PR}$ |  PSNR$\uparrow$ | +0.09 | +0.11 | +0.16 | +0.10
SSIM$\uparrow$ | +0.000 | +0.002 | +0.003 | +0.001
Ours |  PSNR$\uparrow$ | +0.12 | +0.16 | +0.18 | +0.16
SSIM$\uparrow$ | +0.001 | +0.003 | +0.003 | +0.002

##  V Conclusion

We proposed a domain-matching module for the reference-based image super-
resolution task, which can be implemented in a plug-and-play manner. We
investigated the matching process of RefSR models and investigated the
matching results. Based on our analysis of the matching process, we proposed
the domain-matching processes in RefSR models. By aligning the domains in the
correspondence matching and texture transfer, we showed that our method
improves performance of each part when plugged into the existing RefSR
framework.

## Acknowledgements

This work was supported by the Korea Research Institute for Defence Technology
Planning and Advancement (KRIT) grant funded by the Korea government (DAPA) in
2022 (KRIT-CT-22-037, SAR Image Super-Resolution Technology for Improving of
Target Identification Performance 50%), National Research Foundation of Korea
(NRF) grant funded by the Korea government (MSIT) (2022R1C1C100849612 20%),
Institute of Information & communications Technology Planning & Evaluation
(IITP) grant funded by the Korea government (MSIT) (No.2020-0-01336,
Artificial Intelligence Graduate School Program (UNIST), 5%, No.2021-0-02068,
Artificial Intelligence Innovation Hub, 5%, No.2022-0-00959, (Part 2) Few-Shot
Learning of Causal Inference in Vision and Language for Decision Making, 10%,
No.2022-0-00264, Comprehensive Video Understanding and Generation with
Knowledge-based Deep Logic Neural Network, 10%).

## References

  * [1] F. I. Diakogiannis, F. Waldner, P. Caccetta, and C. Wu, “Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 162, pp. 94–114, 2020.
  * [2] J. Han, D. Zhang, G. Cheng, L. Guo, and J. Ren, “Object detection in optical remote sensing images based on weakly supervised learning and high-level feature learning,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 53, no. 6, pp. 3325–3337, 2014.
  * [3] H. Chen, Z. Qi, and Z. Shi, “Remote sensing image change detection with transformers,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–14, 2021.
  * [4] Y. Gao, M. Zhang, J. Wang, and W. Li, “Cross-scale mixing attention for multisource remote sensing data fusion and classification,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 61, pp. 1–15, 2023.
  * [5] Z. Pan, W. Ma, J. Guo, and B. Lei, “Super-resolution of single remote sensing image based on residual dense backprojection networks,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 57, no. 10, pp. 7918–7933, 2019.
  * [6] S. Zhang, Q. Yuan, J. Li, J. Sun, and X. Zhang, “Scene-adaptive remote sensing image super-resolution using a multiscale attention network,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 58, no. 7, pp. 4764–4779, 2020.
  * [7] L. Zhang, D. Chen, J. Ma, and J. Zhang, “Remote-sensing image superresolution based on visual saliency analysis and unequal reconstruction networks,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 58, no. 6, pp. 4099–4115, 2020.
  * [8] Y. Wang, Z. Shao, T. Lu, C. Wu, and J. Wang, “Remote sensing image super-resolution via multiscale enhancement network,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 20, pp. 1–5, 2023.
  * [9] X. Kang, J. Li, P. Duan, F. Ma, and S. Li, “Multilayer degradation representation-guided blind super-resolution for remote sensing images,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–12, 2022.
  * [10] Z. Luo, Y. Huang, S. Li, L. Wang, and T. Tan, “Learning the degradation distribution for blind image super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 6063–6072.
  * [11] H. Zheng, M. Ji, H. Wang, Y. Liu, and L. Fang, “Crossnet: An end-to-end reference-based super resolution network using cross-scale warping,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 88–104.
  * [12] Z. Zhang, Z. Wang, Z. Lin, and H. Qi, “Image super-resolution by neural texture transfer,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 7982–7991.
  * [13] G. Shim, J. Park, and I. S. Kweon, “Robust reference-based super-resolution with similarity-aware deformable convolution,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 8425–8434.
  * [14] Y. Jiang, K. C. Chan, X. Wang, C. C. Loy, and Z. Liu, “Robust reference-based super-resolution via c2-matching,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2103–2112.
  * [15] L. Lu, W. Li, X. Tao, J. Lu, and J. Jia, “Masa-sr: Matching acceleration and spatial adaptation for reference-based image super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 6368–6377.
  * [16] J. Cao, J. Liang, K. Zhang, Y. Li, Y. Zhang, W. Wang, and L. V. Gool, “Reference-based image super-resolution with deformable attention transformer,” in _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVIII_. Springer, 2022, pp. 325–342.
  * [17] R. Dong, L. Zhang, and H. Fu, “Rrsgan: Reference-based super-resolution for remote sensing image,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–17, 2021.
  * [18] Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M.-H. Yang, “Universal style transfer via feature transforms,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [19] Z. Jin, X. Shen, B. Li, and X. Xue, “Style spectroscope: Improve interpretability and controllability through fourier analysis,” _arXiv preprint arXiv:2208.06140_ , 2022.


# From Local to Global: A Graph RAG Approach to Query-Focused Summarization

Darren Edge1† &Ha Trinh1† &Newman Cheng2 &Joshua Bradley2 &Alex Chao3 &Apurva
Mody3 &Steven Truitt2 &Jonathan Larson1

1Microsoft Research
2Microsoft Strategic Missions and Technologies
3Microsoft Office of the CTO

{daedge,trinhha,newmancheng,joshbradley,achao,moapurva,steventruitt,jolarso}
@microsoft.com

†These authors contributed equally to this work

###### Abstract

The use of retrieval-augmented generation (RAG) to retrieve relevant
information from an external knowledge source enables large language models
(LLMs) to answer questions over private and/or previously unseen document
collections. However, RAG fails on global questions directed at an entire text
corpus, such as “What are the main themes in the dataset?”, since this is
inherently a query-focused summarization (QFS) task, rather than an explicit
retrieval task. Prior QFS methods, meanwhile, fail to scale to the quantities
of text indexed by typical RAG systems. To combine the strengths of these
contrasting methods, we propose a Graph RAG approach to question answering
over private text corpora that scales with both the generality of user
questions and the quantity of source text to be indexed. Our approach uses an
LLM to build a graph-based text index in two stages: first to derive an entity
knowledge graph from the source documents, then to pregenerate community
summaries for all groups of closely-related entities. Given a question, each
community summary is used to generate a partial response, before all partial
responses are again summarized in a final response to the user. For a class of
global sensemaking questions over datasets in the 1 million token range, we
show that Graph RAG leads to substantial improvements over a naïve RAG
baseline for both the comprehensiveness and diversity of generated answers. An
open-source, Python-based implementation of both global and local Graph RAG
approaches is forthcoming at https://aka.ms/graphrag.

##  1 Introduction

Human endeavors across a range of domains rely on our ability to read and
reason about large collections of documents, often reaching conclusions that
go beyond anything stated in the source texts themselves. With the emergence
of large language models (LLMs), we are already witnessing attempts to
automate human-like sensemaking in complex domains like scientific discovery
(Microsoft,, 2023) and intelligence analysis (Ranade and Joshi,, 2023), where
sensemaking is defined as “ _a motivated, continuous effort to understand
connections (which can be among people, places, and events) in order to
anticipate their trajectories and act effectively_ ” (Klein et al., 2006a, ).
Supporting human-led sensemaking over entire text corpora, however, needs a
way for people to both apply and refine their mental model of the data (Klein
et al., 2006b, ) by asking questions of a global nature.

Retrieval-augmented generation (RAG, Lewis et al.,, 2020) is an established
approach to answering user questions over entire datasets, but it is designed
for situations where these answers are contained locally within regions of
text whose retrieval provides sufficient grounding for the generation task.
Instead, a more appropriate task framing is query-focused summarization (QFS,
Dang,, 2006), and in particular, query-focused _abstractive_ summarization
that generates natural language summaries and not just concatenated excerpts
(Yao et al.,, 2017; Baumel et al.,, 2018; Laskar et al.,, 2020) . In recent
years, however, such distinctions between summarization tasks that are
abstractive versus extractive, generic versus query-focused, and single-
document versus multi-document, have become less relevant. While early
applications of the transformer architecture showed substantial improvements
on the state-of-the-art for all such summarization tasks (Liu and Lapata,,
2019; Laskar et al.,, 2022; Goodwin et al.,, 2020), these tasks are now
trivialized by modern LLMs, including the GPT (Brown et al.,, 2020; Achiam et
al.,, 2023), Llama (Touvron et al.,, 2023), and Gemini (Anil et al.,, 2023)
series, all of which can use in-context learning to summarize any content
provided in their context window.

Source Documents Text Chunks text extraction and chunking Element Instances
domain-tailored summarization Element Summaries domain-tailored summarization
Graph Communities community detection Community Summaries domain-tailored
summarization Community Answers query-focused summarization Global Answer
query-focused summarization Indexing Time Query Time Pipeline Stage Figure 1:
Graph RAG pipeline using an LLM-derived graph index of source document text.
This index spans nodes (e.g., entities), edges (e.g., relationships), and
covariates (e.g., claims) that have been detected, extracted, and summarized
by LLM prompts tailored to the domain of the dataset. Community detection
(e.g., Leiden, Traag et al.,, 2019) is used to partition the graph index into
groups of elements (nodes, edges, covariates) that the LLM can summarize in
parallel at both indexing time and query time. The “global answer” to a given
query is produced using a final round of query-focused summarization over all
community summaries reporting relevance to that query.

The challenge remains, however, for query-focused abstractive summarization
over an entire corpus. Such volumes of text can greatly exceed the limits of
LLM context windows, and the expansion of such windows may not be enough given
that information can be “lost in the middle” of longer contexts (Liu et al.,,
2023; Kuratov et al.,, 2024). In addition, although the direct retrieval of
text chunks in naïve RAG is likely inadequate for QFS tasks, it is possible
that an alternative form of pre-indexing could support a new RAG approach
specifically targeting global summarization.

In this paper, we present a _Graph RAG_ approach based on global summarization
of an LLM-derived knowledge graph (Figure 1). In contrast with related work
that exploits the structured retrieval and traversal affordances of graph
indexes (subsection 4.2), we focus on a previously unexplored quality of
graphs in this context: their inherent _modularity_ (Newman,, 2006) and the
ability of community detection algorithms to partition graphs into modular
communities of closely-related nodes (e.g., Louvain, Blondel et al.,, 2008;
Leiden, Traag et al.,, 2019). LLM-generated summaries of these community
descriptions provide complete coverage of the underlying graph index and the
input documents it represents. Query-focused summarization of an entire corpus
is then made possible using a map-reduce approach: first using each community
summary to answer the query independently and in parallel, then summarizing
all relevant partial answers into a final global answer.

To evaluate this approach, we used an LLM to generate a diverse set of
activity-centered sensemaking questions from short descriptions of two
representative real-world datasets, containing podcast transcripts and news
articles respectively. For the target qualities of comprehensiveness,
diversity, and empowerment (defined in subsection 3.4) that develop
understanding of broad issues and themes, we both explore the impact of
varying the the hierarchical level of community summaries used to answer
queries, as well as compare to naïve RAG and global map-reduce summarization
of source texts. We show that all global approaches outperform naïve RAG on
comprehensiveness and diversity, and that Graph RAG with intermediate- and
low-level community summaries shows favorable performance over source text
summarization on these same metrics, at lower token costs.

##  2 Graph RAG Approach & Pipeline

We now unpack the high-level data flow of the Graph RAG approach (Figure 1)
and pipeline, describing key design parameters, techniques, and implementation
details for each step.

###  2.1 Source Documents $\rightarrow$ Text Chunks

A fundamental design decision is the granularity with which input texts
extracted from source documents should be split into text chunks for
processing. In the following step, each of these chunks will be passed to a
set of LLM prompts designed to extract the various elements of a graph index.
Longer text chunks require fewer LLM calls for such extraction, but suffer
from the recall degradation of longer LLM context windows (Liu et al.,, 2023;
Kuratov et al.,, 2024). This behavior can be observed in Figure 2 in the case
of a single extraction round (i.e., with zero gleanings): on a sample dataset
(HotPotQA, Yang et al.,, 2018), using a chunk size of 600 token extracted
almost twice as many entity references as when using a chunk size of 2400.
While more references are generally better, any extraction process needs to
balance recall and precision for the target activity.

$0$$1$$2$$3$0100002000030000Number of gleanings performedEntity references
detected600 chunk size1200 chunk size2400 chunk size

Figure 2: How the entity references detected in the HotPotQA dataset (Yang et
al.,, 2018)

varies with chunk size and gleanings for our generic entity extraction prompt
with gpt-4-turbo.

###  2.2 Text Chunks $\rightarrow$ Element Instances

The baseline requirement for this step is to identify and extract instances of
graph nodes and edges from each chunk of source text. We do this using a
multipart LLM prompt that first identifies all _entities_ in the text,
including their name, type, and description, before identifying all
_relationships_ between clearly-related entities, including the source and
target entities and a description of their relationship. Both kinds of element
instance are output in a single list of delimited tuples.

The primary opportunity to tailor this prompt to the domain of the document
corpus lies in the choice of few-shot examples provided to the LLM for in-
context learning (Brown et al.,, 2020). For example, while our default prompt
extracting the broad class of “named entities” like people, places, and
organizations is generally applicable, domains with specialized knowledge
(e.g., science, medicine, law) will benefit from few-shot examples specialized
to those domains. We also support a secondary extraction prompt for any
additional covariates we would like to associate with the extracted node
instances. Our default covariate prompt aims to extract _claims_ linked to
detected entities, including the subject, object, type, description, source
text span, and start and end dates.

To balance the needs of efficiency and quality, we use multiple rounds of
“gleanings”, up to a specified maximum, to encourage the LLM to detect any
additional entities it may have missed on prior extraction rounds. This is a
multi-stage process in which we first ask the LLM to assess whether all
entities were extracted, using a logit bias of 100 to force a yes/no decision.
If the LLM responds that entities were missed, then a continuation indicating
that “MANY entities were missed in the last extraction” encourages the LLM to
glean these missing entities. This approach allows us to use larger chunk
sizes without a drop in quality (Figure 2) or the forced introduction of
noise.

###  2.3 Element Instances $\rightarrow$ Element Summaries

The use of an LLM to “extract” descriptions of entities, relationships, and
claims represented in source texts is already a form of abstractive
summarization, relying on the LLM to create independently meaningful summaries
of concepts that may be implied but not stated by the text itself (e.g., the
presence of implied relationships). To convert all such instance-level
summaries into single blocks of descriptive text for each graph element (i.e.,
entity node, relationship edge, and claim covariate) requires a further round
of LLM summarization over matching groups of instances.

A potential concern at this stage is that the LLM may not consistently extract
references to the same entity in the same text format, resulting in duplicate
entity elements and thus duplicate nodes in the entity graph. However, since
all closely-related “communities” of entities will be detected and summarized
in the following step, and given that LLMs can understand the common entity
behind multiple name variations, our overall approach is resilient to such
variations provided there is sufficient connectivity from all variations to a
shared set of closely-related entities.

Overall, our use of rich descriptive text for homogeneous nodes in a
potentially noisy graph structure is aligned with both the capabilities of
LLMs and the needs of global, query-focused summarization. These qualities
also differentiate our graph index from typical knowledge graphs, which rely
on concise and consistent knowledge triples (subject, predicate, object) for
downstream reasoning tasks.

|
---|---
(a) Root communities at level 0 | (b) Sub-communities at level 1
Figure 3: Graph communities detected using the Leiden algorithm (Traag et
al.,, 2019) over the MultiHop-RAG (Tang and Yang,, 2024) dataset as indexed.
Circles represent entity nodes with size proportional to their degree. Node
layout was performed via OpenORD (Martin et al.,, 2011) and Force Atlas 2
(Jacomy et al.,, 2014). Node colors represent entity communities, shown at two
levels of hierarchical clustering: (a) Level 0, corresponding to the
hierarchical partition with maximum modularity, and (b) Level 1, which reveals
internal structure within these root-level communities.

###  2.4 Element Summaries $\rightarrow$ Graph Communities

The index created in the previous step can be modelled as an homogeneous
undirected weighted graph in which entity nodes are connected by relationship
edges, with edge weights representing the normalized counts of detected
relationship instances. Given such a graph, a variety of community detection
algorithms may be used to partition the graph into communities of nodes with
stronger connections to one another than to the other nodes in the graph
(e.g., see the surveys by Fortunato,, 2010 and Jin et al.,, 2021). In our
pipeline, we use Leiden (Traag et al.,, 2019) on account of its ability to
recover hierarchical community structure of large-scale graphs efficiently
(Figure 3). Each level of this hierarchy provides a community partition that
covers the nodes of the graph in a mutually-exclusive, collective-exhaustive
way, enabling divide-and-conquer global summarization.

###  2.5 Graph Communities $\rightarrow$ Community Summaries

The next step is to create report-like summaries of each community in the
Leiden hierarchy, using a method designed to scale to very large datasets.
These summaries are independently useful in their own right as a way to
understand the global structure and semantics of the dataset, and may
themselves be used to make sense of a corpus in the absence of a question. For
example, a user may scan through community summaries at one level looking for
general themes of interest, then follow links to the reports at the lower
level that provide more details for each of the subtopics. Here, however, we
focus on their utility as part of a graph-based index used for answering
global queries.

Community summaries are generated in the following way:

  * •

_Leaf-level communities_. The element summaries of a leaf-level community
(nodes, edges, covariates) are prioritized and then iteratively added to the
LLM context window until the token limit is reached. The prioritization is as
follows: for each community edge in decreasing order of combined source and
target node degree (i.e., overall prominance), add descriptions of the source
node, target node, linked covariates, and the edge itself.

  * •

_Higher-level communities_. If all element summaries fit within the token
limit of the context window, proceed as for leaf-level communities and
summarize all element summaries within the community. Otherwise, rank sub-
communities in decreasing order of element summary tokens and iteratively
substitute sub-community summaries (shorter) for their associated element
summaries (longer) until fit within the context window is achieved.

###  2.6 Community Summaries $\rightarrow$ Community Answers $\rightarrow$
Global Answer

Given a user query, the community summaries generated in the previous step can
be used to generate a final answer in a multi-stage process. The hierarchical
nature of the community structure also means that questions can be answered
using the community summaries from different levels, raising the question of
whether a particular level in the hierarchical community structure offers the
best balance of summary detail and scope for general sensemaking questions
(evaluated in section 3).

For a given community level, the global answer to any user query is generated
as follows:

  * •

_Prepare community summaries_. Community summaries are randomly shuffled and
divided into chunks of pre-specified token size. This ensures relevant
information is distributed across chunks, rather than concentrated (and
potentially lost) in a single context window.

  * •

_Map community answers_. Generate intermediate answers in parallel, one for
each chunk. The LLM is also asked to generate a score between 0-100 indicating
how helpful the generated answer is in answering the target question. Answers
with score 0 are filtered out.

  * •

_Reduce to global answer_. Intermediate community answers are sorted in
descending order of helpfulness score and iteratively added into a new context
window until the token limit is reached. This final context is used to
generate the global answer returned to the user.

##  3 Evaluation

###  3.1 Datasets

We selected two datasets in the one million token range, each equivalent to
about 10 novels of text and representative of the kind of corpora that users
may encounter in their real world activities:

  * •

_Podcast transcripts_. Compiled transcripts of podcast conversations between
Kevin Scott, Microsoft CTO, and other technology leaders (Behind the Tech,
Scott,, 2024). Size: 1669 $\times$ 600-token text chunks, with 100-token
overlaps between chunks ($\sim$1 million tokens).

  * •

_News articles_. Benchmark dataset comprising news articles published from
September 2013 to December 2023 in a range of categories, including
entertainment, business, sports, technology, health, and science (MultiHop-
RAG; Tang and Yang,, 2024). Size: 3197 $\times$ 600-token text chunks, with
100-token overlaps between chunks ($\sim$1.7 million tokens).

###  3.2 Queries

Many benchmark datasets for open-domain question answering exist, including
HotPotQA (Yang et al.,, 2018), MultiHop-RAG (Tang and Yang,, 2024), and MT-
Bench (Zheng et al.,, 2024). However, the associated question sets target
explicit fact retrieval rather than summarization for the purpose of data
sensemaking, i.e., the process though which people inspect, engage with, and
contextualize data within the broader scope of real-world activities (Koesten
et al.,, 2021). Similarly, methods for extracting latent summarization queries
from source texts also exist (Xu and Lapata,, 2021), but such extracted
questions can target details that betray prior knowledge of the texts.

To evaluate the effectiveness of RAG systems for more global sensemaking
tasks, we need questions that convey only a high-level understanding of
dataset contents, and not the details of specific texts. We used an activity-
centered approach to automate the generation of such questions: given a short
description of a dataset, we asked the LLM to identify $N$ potential users and
$N$ tasks per user, then for each (user, task) combination, we asked the LLM
to generate $N$ questions that require understanding of the entire corpus. For
our evaluation, a value of $N$ = 5 resulted in 125 test questions per dataset.
Table 1 shows example questions for each of the two evaluation datasets.

Dataset |  Example activity framing and generation of global sensemaking questions
---|---
Podcast transcripts |  _User_ : A tech journalist looking for insights and trends in the tech industry
_Task_ : Understanding how tech leaders view the role of policy and regulation
_Questions_ :
1\. Which episodes deal primarily with tech policy and government regulation?
2\. How do guests perceive the impact of privacy laws on technology
development?
3\. Do any guests discuss the balance between innovation and ethical
considerations?
4\. What are the suggested changes to current policies mentioned by the
guests?
5\. Are collaborations between tech companies and governments discussed and
how?
News articles |  _User_ : Educator incorporating current affairs into curricula
_Task_ : Teaching about health and wellness
_Questions_ :
1\. What current topics in health can be integrated into health education
curricula?
2\. How do news articles address the concepts of preventive medicine and
wellness?
3\. Are there examples of health articles that contradict each other, and if
so, why?
4\. What insights can be gleaned about public health priorities based on news
coverage?
5\. How can educators use the dataset to highlight the importance of health
literacy?
Table 1: Examples of potential users, tasks, and questions generated by the
LLM based on short descriptions of the target datasets. Questions target
global understanding rather than specific details.

###  3.3 Conditions

We compare six different conditions in our analysis, including Graph RAG using
four levels of graph communities (C0, C1, C2, C3), a text summarization method
applying our map-reduce approach directly to source texts (TS), and a naïve
“semantic search” RAG approach (SS):

  * •

CO. Uses root-level community summaries (fewest in number) to answer user
queries.

  * •

C1. Uses high-level community summaries to answer queries. These are sub-
communities of C0, if present, otherwise C0 communities projected down.

  * •

C2. Uses intermediate-level community summaries to answer queries. These are
sub-communities of C1, if present, otherwise C1 communities projected down.

  * •

C3. Uses low-level community summaries (greatest in number) to answer queries.
These are sub-communities of C2, if present, otherwise C2 communities
projected down.

  * •

TS. The same method as in subsection 2.6, except source texts (rather than
community summaries) are shuffled and chunked for the map-reduce summarization
stages.

  * •

SS. An implementation of naïve RAG in which text chunks are retrieved and
added to the available context window until the specified token limit is
reached.

The size of the context window and the prompts used for answer generation are
the same across all six conditions (except for minor modifications to
reference styles to match the types of context information used). Conditions
only differ in how the contents of the context window are created.

The graph index supporting conditions C0-C3 was created using our generic
prompts for entity and relationship extraction only, with entity types and
few-shot examples tailored to the domain of the data. The graph indexing
process used a context window size of 600 tokens with 1 gleaning for the
Podcast dataset and 0 gleanings for the News dataset.

Question |  Which public figures are repeatedly mentioned across various entertainment articles?
---|---
Graph RAG |  Overview of Prominent Public Figures in Entertainment The entertainment industry is vast and diverse, encompassing film, television, music, sports, and digital media. Certain public figures stand out due to their significant contributions and influence across these sectors. The following summary highlights key individuals who are repeatedly mentioned in various entertainment articles, reflecting their impact and presence within the industry. Actors and Directors […] Public Figures in Controversy […] Musicians and Executives […] Athletes and Coaches […] Influencers and Entrepreneurs […] The repeated mention of these figures in entertainment articles signifies their ongoing relevance and the public’s interest in their work. Their influence spans across various aspects of entertainment, from shaping cultural narratives in film and television to driving trends in music and digital media. These individuals not only contribute to their respective fields but also influence the broader cultural landscape, often becoming central figures in social discussions and public discourse.
Naïve RAG |  Public figures who are repeatedly mentioned across various entertainment articles include Taylor Swift, Travis Kelce, Britney Spears, and Justin Timberlake. These individuals have been highlighted for various reasons, ranging from their professional achievements to their personal lives. Taylor Swift […] Travis Kelce […] Britney Spears […] Justin Timberlake […] These figures are frequently covered due to their high-profile status and the public’s interest in their careers and personal lives. Their activities, whether in music, sports, or personal relationships, have significant cultural and economic impacts, as evidenced by the media coverage and public reactions.
LLM Decision |  Comprehensiveness: Winner=1 (Graph RAG)
Answer 1 is better because it provides a more comprehensive and detailed list
of public figures from a wider range of entertainment sectors, including film,
television, music, sports, gaming, and digital media. It also includes
specific examples of their contributions and the impact they have on their
respective fields, as well as mentions of controversies and their
implications. Answer 2, while detailed in its coverage of a few individuals,
is limited to a smaller number of public figures and focuses primarily on
their personal lives and relationships rather than a broad spectrum of their
professional influence across the entertainment industry.
Diversity: Winner=1 (Graph RAG)
Answer 1 is better because it provides a more varied and rich response by
covering a wide range of public figures from different sectors of the
entertainment industry, including film, television, music, sports, gaming, and
digital media. It offers insights into the contributions and influence of
these figures, as well as controversies and their impact on public discourse.
The answer also cites specific data sources for each mentioned figure,
indicating a diverse range of evidence to support the claims. In contrast,
Answer 2 focuses on a smaller group of public figures, primarily from the
music industry and sports, and relies heavily on a single source for data,
which makes it less diverse in perspectives and insights.
Empowerment: Winner=1 (Graph RAG)
Answer 1 is better because it provides a comprehensive and structured overview
of public figures across various sectors of the entertainment industry,
including film, television, music, sports, and digital media. It lists
multiple individuals, providing specific examples of their contributions and
the context in which they are mentioned in entertainment articles, along with
references to data reports for each claim. This approach helps the reader
understand the breadth of the topic and make informed judgments without being
misled. In contrast, Answer 2 focuses on a smaller group of public figures and
primarily discusses their personal lives and relationships, which may not
provide as broad an understanding of the topic. While Answer 2 also cites
sources, it does not match the depth and variety of Answer 1.
Directness: Winner=2 (Naïve RAG)
Answer 2 is better because it directly lists specific public figures who are
repeatedly mentioned across various entertainment articles, such as Taylor
Swift, Travis Kelce, Britney Spears, and Justin Timberlake, and provides
concise explanations for their frequent mentions. Answer 1, while
comprehensive, includes a lot of detailed information about various figures in
different sectors of entertainment, which, while informative, does not
directly answer the question with the same level of conciseness and
specificity as Answer 2.
Table 2: Example question for the News article dataset, with generated answers
from Graph RAG (C2) and Naïve RAG, as well as LLM-generated assessments.

###  3.4 Metrics

LLMs have been shown to be good evaluators of natural language generation,
achieving state-of-the-art or competitive results compared against human
judgements (Wang et al., 2023a, ; Zheng et al.,, 2024). While this approach
can generate reference-based metrics when gold standard answers are known, it
is also capable of measuring the qualities of generated texts (e.g., fluency)
in a reference-free style (Wang et al., 2023a, ) as well as in head-to-head
comparison of competing outputs (LLM-as-a-judge, Zheng et al.,, 2024). LLMs
have also shown promise at evaluating the performance of conventional RAG
systems, automatically evaluating qualities like context relevance,
faithfulness, and answer relevance (RAGAS, Es et al.,, 2023).

Given the multi-stage nature of our Graph RAG mechanism, the multiple
conditions we wanted to compare, and the lack of gold standard answers to our
activity-based sensemaking questions, we decided to adopt a head-to-head
comparison approach using an LLM evaluator. We selected three target metrics
capturing qualities that are desirable for sensemaking activities, as well as
a control metric (directness) used as a indicator of validity. Since
directness is effectively in opposition to comprehensiveness and diversity, we
would not expect any method to win across all four metrics.

Our head-to-head measures computed using an LLM evaluator are as follows:

  * •

_Comprehensiveness_. How much detail does the answer provide to cover all
aspects and details of the question?

  * •

_Diversity_. How varied and rich is the answer in providing different
perspectives and insights on the question?

  * •

_Empowerment_. How well does the answer help the reader understand and make
informed judgements about the topic?

  * •

_Directness_. How specifically and clearly does the answer address the
question?

For our evaluation, the LLM is provided with the question, target metric, and
a pair of answers, and asked to assess which answer is better according to the
metric, as well as why. It returns the winner if one exists, otherwise a tie
if they are fundamentally similar and the differences are negligible. To
account for the stochasticity of LLMs, we run each comparison five times and
use mean scores. Table 2 shows an example of LLM-generated assessment.

###  3.5 Configuration

The effect of context window size on any particular task is unclear,
especially for models like gpt-4-turbo with a large context size of 128k
tokens. Given the potential for information to be “lost in the middle” of
longer contexts (Liu et al.,, 2023; Kuratov et al.,, 2024), we wanted to
explore the effects of varying the context window size for our combinations of
datasets, questions, and metrics. In particular, our goal was to determine the
optimum context size for our baseline condition (SS) and then use this
uniformly for all query-time LLM use. To that end, we tested four context
window sizes: 8k, 16k, 32k and 64k. Surprisingly, the smallest context window
size tested (8k) was universally better for all comparisons on
comprehensiveness (average win rate of 58.1%), while performing comparably
with larger context sizes on diversity (average win rate = 52.4%), and
empowerment (average win rate = 51.3%). Given our preference for more
comprehensive and diverse answers, we therefore used a fixed context window
size of 8k tokens for the final evaluation.

###  3.6 Results

The indexing process resulted in a graph consisting of 8564 nodes and 20691
edges for the Podcast dataset, and a larger graph of 15754 nodes and 19520
edges for the News dataset. Table 3 shows the number of community summaries at
different levels of each graph community hierarchy.

Podcast transcripts
501728252221835050484344725050535049755247505250785750485052795651504850000000000000Comprehensiveness
501823251919825050504346775050504644755050504445815754565048815456555250000000000000Diversity
504257524951585059555251434150494748484551504950514853515051494952504950000000000000Empowerment
505665606060445055525152354550474848404853505050404952505050404852505050000000000000Directness

News articles
502028252121805044413836725650525452755948505855796246425059796448454150000000000000Comprehensiveness
503338352931675053454440624750404141655560505050715659505051696059504950000000000000Diversity
504757495050535058505048434250424544515058505251505055485050505256495050000000000000Empowerment
5054595555544650555352
52414550484847454752504949454852515049464853515150000000000000Directness

Figure 4: Head-to-head win rate percentages of (row condition) over (column condition) across two datasets, four metrics, and 125 questions per comparison (each repeated five times and averaged). The overall winner per dataset and metric is shown in bold. Self-win rates were not computed but are shown as the expected 50% for reference. All Graph RAG conditions outperformed naïve RAG on comprehensiveness and diversity. Conditions C1-C3 also showed slight improvements in answer comprehensiveness and diversity over TS (global text summarization without a graph index). | Podcast Transcripts | News Articles
---|---|---
|  C0 |  C1 |  C2 |  C3 |  TS |  C0 |  C1 |  C2 |  C3 |  TS
Units |  34 |  367 |  969 |  1310 |  1669 |  55 |  555 |  1797 |  2142 |  3197
Tokens |  26657 |  225756 |  565720 |  746100 |  1014611 |  39770 |  352641 |  980898 |  1140266 |  1707694
% Max |  2.6 |  22.2 |  55.8 |  73.5 |  100 |  2.3 |  20.7 |  57.4 |  66.8 |  100
Table 3: Number of context units (community summaries for C0-C3 and text
chunks for TS), corresponding token counts, and percentage of the maximum
token count. Map-reduce summarization of source texts is the most resource-
intensive approach requiring the highest number of context tokens. Root-level
community summaries (C0) require dramatically fewer tokens per query (9x-43x).

Global approaches vs. naïve RAG. As shown in Figure 4, global approaches
consistently outperformed the naïve RAG (SS) approach in both
comprehensiveness and diversity metrics across datasets. Specifically, global
approaches achieved comprehensiveness win rates between 72-83% for Podcast
transcripts and 72-80% for News articles, while diversity win rates ranged
from 75-82% and 62-71% respectively. Our use of directness as a validity test
also achieved the expected results, i.e., that naïve RAG produces the most
direct responses across all comparisons.

Community summaries vs. source texts. When comparing community summaries to
source texts using Graph RAG, community summaries generally provided a small
but consistent improvement in answer comprehensiveness and diversity, except
for root-level summaries. Intermediate-level summaries in the Podcast dataset
and low-level community summaries in the News dataset achieved
comprehensiveness win rates of 57% and 64%, respectively. Diversity win rates
were 57% for Podcast intermediate-level summaries and 60% for News low-level
community summaries. Table 3 also illustrates the scalability advantages of
Graph RAG compared to source text summarization: for low-level community
summaries (C3), Graph RAG required 26-33% fewer context tokens, while for
root-level community summaries (C0), it required over 97% fewer tokens. For a
modest drop in performance compared with other global methods, root-level
Graph RAG offers a highly efficient method for the iterative question
answering that characterizes sensemaking activity, while retaining advantages
in comprehensiveness (72% win rate) and diversity (62% win rate) over naïve
RAG.

Empowerment. Empowerment comparisons showed mixed results for both global
approaches versus naïve RAG (SS) and Graph RAG approaches versus source text
summarization (TS). Ad-hoc LLM use to analyze LLM reasoning for this measure
indicated that the ability to provide specific examples, quotes, and citations
was judged to be key to helping users reach an informed understanding. Tuning
element extraction prompts may help to retain more of these details in the
Graph RAG index.

##  4 Related Work

###  4.1 RAG Approaches and Systems

When using LLMs, RAG involves first retrieving relevant information from
external data sources, then adding this information to the context window of
the LLM along with the original query (Ram et al.,, 2023). Naïve RAG
approaches (Gao et al.,, 2023) do this by converting documents to text,
splitting text into chunks, and embedding these chunks into a vector space in
which similar positions represent similar semantics. Queries are then embedded
into the same vector space, with the text chunks of the nearest _k_ vectors
used as context. More advanced variations exist, but all solve the problem of
what to do when an external dataset of interest exceeds the LLM’s context
window.

Advanced RAG systems include pre-retrieval, retrieval, post-retrieval
strategies designed to overcome the drawbacks of Naïve RAG, while Modular RAG
systems include patterns for iterative and dynamic cycles of interleaved
retrieval and generation (Gao et al.,, 2023). Our implementation of Graph RAG
incorporates multiple concepts related to other systems. For example, our
community summaries are a kind of self-memory (Selfmem, Cheng et al.,, 2024)
for generation-augmented retrieval (GAR, Mao et al.,, 2020) that facilitates
future generation cycles, while our parallel generation of community answers
from these summaries is a kind of iterative (Iter-RetGen, Shao et al.,, 2023)
or federated (FeB4RAG, Wang et al.,, 2024) retrieval-generation strategy.
Other systems have also combined these concepts for multi-document
summarization (CAiRE-COVID, Su et al.,, 2020) and multi-hop question answering
(ITRG, Feng et al.,, 2023; IR-CoT, Trivedi et al.,, 2022; DSP, Khattab et
al.,, 2022). Our use of a hierarchical index and summarization also bears
resemblance to further approaches, such as generating a hierarchical index of
text chunks by clustering the vectors of text embeddings (RAPTOR, Sarthi et
al.,, 2024) or generating a “tree of clarifications” to answer multiple
interpretations of ambiguous questions (Kim et al.,, 2023). However, none of
these iterative or hierarchical approaches use the kind of self-generated
graph index that enables Graph RAG.

###  4.2 Graphs and LLMs

Use of graphs in connection with LLMs and RAG is a developing research area,
with multiple directions already established. These include using LLMs for
knowledge graph creation (Trajanoska et al.,, 2023) and completion (Yao et
al.,, 2023), as well as for the extraction of causal graphs (Ban et al.,,
2023; Zhang et al.,, 2024) from source texts. They also include forms of
advanced RAG (Gao et al.,, 2023) where the index is a knowledge graph (KAPING,
Baek et al.,, 2023), where subsets of the graph structure (G-Retriever, He et
al.,, 2024) or derived graph metrics (Graph-ToolFormer, Zhang,, 2023) are the
objects of enquiry, where narrative outputs are strongly grounded in the facts
of retrieved subgraphs (SURGE, Kang et al.,, 2023), where retrieved event-plot
subgraphs are serialized using narrative templates (FABULA, Ranade and Joshi,,
2023), and where the system supports both creation and traversal of text-
relationship graphs for multi-hop question answering (Wang et al., 2023b, ).
In terms of open-source software, a variety a graph databases are supported by
both the LangChain (LangChain,, 2024) and LlamaIndex (LlamaIndex,, 2024)
libraries, while a more general class of graph-based RAG applications is also
emerging, including systems that can create and reason over knowledge graphs
in both Neo4J (NaLLM, Neo4J,, 2024) and NebulaGraph (GraphRAG, NebulaGraph,,
2024) formats. Unlike our Graph RAG approach, however, none of these systems
use the natural modularity of graphs to partition data for global
summarization.

##  5 Discussion

_Limitations of evaluation approach_. Our evaluation to date has only examined
a certain class of sensemaking questions for two corpora in the region of 1
million tokens. More work is needed to understand how performance varies
across different ranges of question types, data types, and dataset sizes, as
well as to validate our sensemaking questions and target metrics with end
users. Comparison of fabrication rates, e.g., using approaches like
SelfCheckGPT (Manakul et al.,, 2023), would also improve on the current
analysis.

_Trade-offs of building a graph index_. We consistently observed Graph RAG
achieve the best head-to-head results against other methods, but in many cases
the graph-free approach to global summarization of source texts performed
competitively. The real-world decision about whether to invest in building a
graph index depends on multiple factors, including the compute budget,
expected number of lifetime queries per dataset, and value obtained from other
aspects of the graph index (including the generic community summaries and the
use of other graph-related RAG approaches).

_Future work_. The graph index, rich text annotations, and hierarchical
community structure supporting the current Graph RAG approach offer many
possibilities for refinement and adaptation. This includes RAG approaches that
operate in a more local manner, via embedding-based matching of user queries
and graph annotations, as well as the possibility of hybrid RAG schemes that
combine embedding-based matching against community reports before employing
our map-reduce summarization mechanisms. This “roll-up” operation could also
be extended across more levels of the community hierarchy, as well as
implemented as a more exploratory “drill down” mechanism that follows the
information scent contained in higher-level community summaries.

##  6 Conclusion

We have presented a global approach to Graph RAG, combining knowledge graph
generation, retrieval-augmented generation (RAG), and query-focused
summarization (QFS) to support human sensemaking over entire text corpora.
Initial evaluations show substantial improvements over a naïve RAG baseline
for both the comprehensiveness and diversity of answers, as well as favorable
comparisons to a global but graph-free approach using map-reduce source text
summarization. For situations requiring many global queries over the same
dataset, summaries of root-level communities in the entity-based graph index
provide a data index that is both superior to naïve RAG and achieves
competitive performance to other global methods at a fraction of the token
cost.

An open-source, Python-based implementation of both global and local Graph RAG
approaches is forthcoming at https://aka.ms/graphrag.

## Acknowledgements

We would also like to thank the following people who contributed to the work:
Alonso Guevara Fernández, Amber Hoak, Andrés Morales Esquivel, Ben Cutler,
Billie Rinaldi, Chris Sanchez, Chris Trevino, Christine Caggiano, David
Tittsworth, Dayenne de Souza, Douglas Orbaker, Ed Clark, Gabriel Nieves-Ponce,
Gaudy Blanco Meneses, Kate Lytvynets, Katy Smith, Mónica Carvajal, Nathan
Evans, Richard Ortega, Rodrigo Racanicci, Sarah Smith, and Shane Solomon.

## References

  * Achiam et al., (2023) Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023).  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774.
  * Anil et al., (2023) Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. (2023).  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805.
  * Baek et al., (2023) Baek, J., Aji, A. F., and Saffari, A. (2023).  Knowledge-augmented language model prompting for zero-shot knowledge graph question answering.  arXiv preprint arXiv:2306.04136.
  * Ban et al., (2023) Ban, T., Chen, L., Wang, X., and Chen, H. (2023).  From query tools to causal architects: Harnessing large language models for advanced causal discovery from data.
  * Baumel et al., (2018) Baumel, T., Eyal, M., and Elhadad, M. (2018).  Query focused abstractive summarization: Incorporating query relevance, multi-document coverage, and summary length constraints into seq2seq models.  arXiv preprint arXiv:1801.07704.
  * Blondel et al., (2008) Blondel, V. D., Guillaume, J.-L., Lambiotte, R., and Lefebvre, E. (2008).  Fast unfolding of communities in large networks.  Journal of statistical mechanics: theory and experiment, 2008(10):P10008.
  * Brown et al., (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020).  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901.
  * Cheng et al., (2024) Cheng, X., Luo, D., Chen, X., Liu, L., Zhao, D., and Yan, R. (2024).  Lift yourself up: Retrieval-augmented text generation with self-memory.  Advances in Neural Information Processing Systems, 36.
  * Dang, (2006) Dang, H. T. (2006).  Duc 2005: Evaluation of question-focused summarization systems.  In Proceedings of the Workshop on Task-Focused Summarization and Question Answering, pages 48–55.
  * Es et al., (2023) Es, S., James, J., Espinosa-Anke, L., and Schockaert, S. (2023).  Ragas: Automated evaluation of retrieval augmented generation.  arXiv preprint arXiv:2309.15217.
  * Feng et al., (2023) Feng, Z., Feng, X., Zhao, D., Yang, M., and Qin, B. (2023).  Retrieval-generation synergy augmented large language models.  arXiv preprint arXiv:2310.05149.
  * Fortunato, (2010) Fortunato, S. (2010).  Community detection in graphs.  Physics reports, 486(3-5):75–174.
  * Gao et al., (2023) Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. (2023).  Retrieval-augmented generation for large language models: A survey.  arXiv preprint arXiv:2312.10997.
  * Goodwin et al., (2020) Goodwin, T. R., Savery, M. E., and Demner-Fushman, D. (2020).  Flight of the pegasus? comparing transformers on few-shot and zero-shot multi-document abstractive summarization.  In Proceedings of COLING. International Conference on Computational Linguistics, volume 2020, page 5640. NIH Public Access.
  * He et al., (2024) He, X., Tian, Y., Sun, Y., Chawla, N. V., Laurent, T., LeCun, Y., Bresson, X., and Hooi, B. (2024).  G-retriever: Retrieval-augmented generation for textual graph understanding and question answering.  arXiv preprint arXiv:2402.07630.
  * Jacomy et al., (2014) Jacomy, M., Venturini, T., Heymann, S., and Bastian, M. (2014).  Forceatlas2, a continuous graph layout algorithm for handy network visualization designed for the gephi software.  PLoS ONE 9(6): e98679. https://doi.org/10.1371/journal.pone.0098679.
  * Jin et al., (2021) Jin, D., Yu, Z., Jiao, P., Pan, S., He, D., Wu, J., Philip, S. Y., and Zhang, W. (2021).  A survey of community detection approaches: From statistical modeling to deep learning.  IEEE Transactions on Knowledge and Data Engineering, 35(2):1149–1170.
  * Kang et al., (2023) Kang, M., Kwak, J. M., Baek, J., and Hwang, S. J. (2023).  Knowledge graph-augmented language models for knowledge-grounded dialogue generation.  arXiv preprint arXiv:2305.18846.
  * Khattab et al., (2022) Khattab, O., Santhanam, K., Li, X. L., Hall, D., Liang, P., Potts, C., and Zaharia, M. (2022).  Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp.  arXiv preprint arXiv:2212.14024.
  * Kim et al., (2023) Kim, G., Kim, S., Jeon, B., Park, J., and Kang, J. (2023).  Tree of clarifications: Answering ambiguous questions with retrieval-augmented large language models.  arXiv preprint arXiv:2310.14696.
  * (21) Klein, G., Moon, B., and Hoffman, R. R. (2006a).  Making sense of sensemaking 1: Alternative perspectives.  IEEE intelligent systems, 21(4):70–73.
  * (22) Klein, G., Moon, B., and Hoffman, R. R. (2006b).  Making sense of sensemaking 2: A macrocognitive model.  IEEE Intelligent systems, 21(5):88–92.
  * Koesten et al., (2021) Koesten, L., Gregory, K., Groth, P., and Simperl, E. (2021).  Talking datasets–understanding data sensemaking behaviours.  International journal of human-computer studies, 146:102562.
  * Kuratov et al., (2024) Kuratov, Y., Bulatov, A., Anokhin, P., Sorokin, D., Sorokin, A., and Burtsev, M. (2024).  In search of needles in a 11m haystack: Recurrent memory finds what llms miss.
  * LangChain, (2024) LangChain (2024).  Langchain graphs.  https://python.langchain.com/docs/use_cases/graph/.
  * Laskar et al., (2020) Laskar, M. T. R., Hoque, E., and Huang, J. (2020).  Query focused abstractive summarization via incorporating query relevance and transfer learning with transformer models.  In Advances in Artificial Intelligence: 33rd Canadian Conference on Artificial Intelligence, Canadian AI 2020, Ottawa, ON, Canada, May 13–15, 2020, Proceedings 33, pages 342–348. Springer.
  * Laskar et al., (2022) Laskar, M. T. R., Hoque, E., and Huang, J. X. (2022).  Domain adaptation with pre-trained transformers for query-focused abstractive text summarization.  Computational Linguistics, 48(2):279–320.
  * Lewis et al., (2020) Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. (2020).  Retrieval-augmented generation for knowledge-intensive nlp tasks.  Advances in Neural Information Processing Systems, 33:9459–9474.
  * Liu et al., (2023) Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2023).  Lost in the middle: How language models use long contexts.  arXiv:2307.03172.
  * Liu and Lapata, (2019) Liu, Y. and Lapata, M. (2019).  Hierarchical transformers for multi-document summarization.  arXiv preprint arXiv:1905.13164.
  * LlamaIndex, (2024) LlamaIndex (2024).  LlamaIndex Knowledge Graph Index.  https://docs.llamaindex.ai/en/stable/examples/index_structs/knowledge_graph/KnowledgeGraphDemo.html.
  * Manakul et al., (2023) Manakul, P., Liusie, A., and Gales, M. J. (2023).  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  arXiv preprint arXiv:2303.08896.
  * Mao et al., (2020) Mao, Y., He, P., Liu, X., Shen, Y., Gao, J., Han, J., and Chen, W. (2020).  Generation-augmented retrieval for open-domain question answering.  arXiv preprint arXiv:2009.08553.
  * Martin et al., (2011) Martin, S., Brown, W. M., Klavans, R., and Boyack, K. (2011).  Openord: An open-source toolbox for large graph layout.  SPIE Conference on Visualization and Data Analysis (VDA).
  * Microsoft, (2023) Microsoft (2023).  The impact of large language models on scientific discovery: a preliminary study using gpt-4.
  * NebulaGraph, (2024) NebulaGraph (2024).  Nebulagraph launches industry-first graph rag: Retrieval-augmented generation with llm based on knowledge graphs.  https://www.nebula-graph.io/posts/graph-RAG.
  * Neo4J, (2024) Neo4J (2024).  Project NaLLM.  https://github.com/neo4j/NaLLM.
  * Newman, (2006) Newman, M. E. (2006).  Modularity and community structure in networks.  Proceedings of the national academy of sciences, 103(23):8577–8582.
  * Ram et al., (2023) Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., and Shoham, Y. (2023).  In-context retrieval-augmented language models.  Transactions of the Association for Computational Linguistics, 11:1316–1331.
  * Ranade and Joshi, (2023) Ranade, P. and Joshi, A. (2023).  Fabula: Intelligence report generation using retrieval-augmented narrative construction.  arXiv preprint arXiv:2310.13848.
  * Sarthi et al., (2024) Sarthi, P., Abdullah, S., Tuli, A., Khanna, S., Goldie, A., and Manning, C. D. (2024).  Raptor: Recursive abstractive processing for tree-organized retrieval.  arXiv preprint arXiv:2401.18059.
  * Scott, (2024) Scott, K. (2024).  Behind the Tech.  https://www.microsoft.com/en-us/behind-the-tech.
  * Shao et al., (2023) Shao, Z., Gong, Y., Shen, Y., Huang, M., Duan, N., and Chen, W. (2023).  Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy.  arXiv preprint arXiv:2305.15294.
  * Su et al., (2020) Su, D., Xu, Y., Yu, T., Siddique, F. B., Barezi, E. J., and Fung, P. (2020).  Caire-covid: A question answering and query-focused multi-document summarization system for covid-19 scholarly information management.  arXiv preprint arXiv:2005.03975.
  * Tang and Yang, (2024) Tang, Y. and Yang, Y. (2024).  MultiHop-RAG: Benchmarking retrieval-augmented generation for multi-hop queries.  arXiv preprint arXiv:2401.15391.
  * Touvron et al., (2023) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023).  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288.
  * Traag et al., (2019) Traag, V. A., Waltman, L., and Van Eck, N. J. (2019).  From Louvain to Leiden: guaranteeing well-connected communities.  Scientific Reports, 9(1).
  * Trajanoska et al., (2023) Trajanoska, M., Stojanov, R., and Trajanov, D. (2023).  Enhancing knowledge graph construction using large language models.  ArXiv, abs/2305.04676.
  * Trivedi et al., (2022) Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. (2022).  Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions.  arXiv preprint arXiv:2212.10509.
  * (50) Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., and Zhou, J. (2023a).  Is chatgpt a good nlg evaluator? a preliminary study.  arXiv preprint arXiv:2303.04048.
  * Wang et al., (2024) Wang, S., Khramtsova, E., Zhuang, S., and Zuccon, G. (2024).  Feb4rag: Evaluating federated search in the context of retrieval augmented generation.  arXiv preprint arXiv:2402.11891.
  * (52) Wang, Y., Lipka, N., Rossi, R. A., Siu, A., Zhang, R., and Derr, T. (2023b).  Knowledge graph prompting for multi-document question answering.
  * Xu and Lapata, (2021) Xu, Y. and Lapata, M. (2021).  Text summarization with latent queries.  arXiv preprint arXiv:2106.00104.
  * Yang et al., (2018) Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. (2018).  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In Conference on Empirical Methods in Natural Language Processing (EMNLP).
  * Yao et al., (2017) Yao, J.-g., Wan, X., and Xiao, J. (2017).  Recent advances in document summarization.  Knowledge and Information Systems, 53:297–336.
  * Yao et al., (2023) Yao, L., Peng, J., Mao, C., and Luo, Y. (2023).  Exploring large language models for knowledge graph completion.
  * Zhang, (2023) Zhang, J. (2023).  Graph-toolformer: To empower llms with graph reasoning ability via prompt augmented by chatgpt.  arXiv preprint arXiv:2304.11116.
  * Zhang et al., (2024) Zhang, Y., Zhang, Y., Gan, Y., Yao, L., and Wang, C. (2024).  Causal graph discovery with retrieval-augmented generation based large language models.  arXiv preprint arXiv:2402.15301.
  * Zheng et al., (2024) Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. (2024).  Judging llm-as-a-judge with mt-bench and chatbot arena.  Advances in Neural Information Processing Systems, 36.