11institutetext: 1ZIP Lab, Monash University, Australia 2ReLER, AAII, UTS
3MBZUAI

# LongVLM: Efficient Long Video Understanding via Large Language Models

Yuetian Weng 11 Mingfei Han 22 Haoyu He 11
Xiaojun Chang 2233 Bohan Zhuang
Corresponding author. Email: $\tt<EMAIL_ADDRESS>
###### Abstract

Empowered by Large Language Models (LLMs), recent advancements in VideoLLMs
have driven progress in various video understanding tasks. These models encode
video representations through pooling or query aggregation over a vast number
of visual tokens, making computational and memory costs affordable. Despite
successfully providing an overall comprehension of video content, existing
VideoLLMs still face challenges in achieving detailed understanding in videos
due to overlooking local information in long-term videos. To tackle this
challenge, we introduce LongVLM, a straightforward yet powerful VideoLLM for
long video understanding, building upon the observation that long videos often
consist of sequential key events, complex actions, and camera movements. Our
approach proposes to decompose long videos into multiple short-term segments
and encode local features for each local segment via a hierarchical token
merging module. These features are concatenated in temporal order to maintain
the storyline across sequential short-term segments. Additionally, we propose
to integrate global semantics into each local feature to enhance context
understanding. In this way, we encode video representations that incorporate
both local and global information, enabling the LLM to generate comprehensive
responses for long-term videos. Experimental results on the VideoChatGPT
benchmark and zero-shot video question-answering datasets demonstrate the
superior capabilities of our model over the previous state-of-the-art methods.
Qualitative examples demonstrate that our model produces more precise
responses for long videos understanding. Code will be available at
https://github.com/ziplab/LongVLM.

##  1 Introduction

(a) Architectures.

(b) An example from Video-ChatGPT benchmark [35].

Figure 1: Previous approaches employ pooling operation or query aggregation
for visual representations, focusing primarily on modeling global semantics
and potentially overlooking detailed information in long-term videos. In
contrast, our LongVLM proposes to aggregate local features for short-term
segments and integrate global semantics into segment-level local feature
sequence. Text highlighted in bold green denotes correct content, while text
in red indicates errors. The video duration is 1 minute and 20 seconds.

Large language models (LLMs) [36, 37, 45, 46, 43, 7] have revolutionized
natural language understanding tasks and have demonstrated a remarkable
capability to follow human instructions and intentions, emerging as a
universal agent for general-purpose assistants. Drawing from the development
of LLMs, Multi-modal Large Language Models (MLLMs) [30, 65, 8, 15] have driven
the advance in vision-language learning by integrating visual encoders with
LLMs and finetuning on language-image instruction-following data. However,
developing Video-based Large Language Models (VideoLLMs) still poses a
significant challenge due to the necessity of processing a large number of
tokens for jointly modeling spatial-temporal dependencies across consecutive
video frames. For instance, employing OpenAI CLIP-ViT-L/14 [40] as a visual
encoder for a 100-frame video clip necessitates handling 25.6K visual tokens,
leading to impractical computational costs with existing LLMs. To address
this, recent approaches propose to extract video representation via
precompression over visual tokens, utilizing pooling operation [35, 34] or
query aggregation [24, 61, 42] over the video token sequence before feeding
them into the LLM, as shown in Fig. 1(a). While these models showcase
impressive capabilities in providing a meaningful understanding of video
content, they still face challenges in achieving a significant advantage in
fine-grained understanding of long-term videos. For example, as shown in Fig.
1(b), while all models recognize the overall environment (workshop), the
object (bike), and the action (fixing), previous methods may fail to correctly
identify details such as the color of the clothes (brown), or the specific
component being fixed (bicycle chain).

The main reason is that long-term videos typically involve numerous key
actions, complex activities, and camera movements. Consequently, a long video
can be divided into a sequence of short-term segments. For instance, in the
example depicted in Fig. 1(b), various short-term actions occur, _e.g_.
speaking, displaying spare components, grabbing the bicycle chain, along with
the camera moving from the human to the bicycle wheel, and eventually focusing
on the broken chain. Similarly, prior methods in video recognition task
suggest to decompose complex activities into sequences of sub-activities [47,
19, 14]. These approaches treat the features of each short-range activity as
the local information within the videos and emphasize the importance of
reasoning over local features to develop a temporal-structural understanding
[47, 14, 48, 49, 19, 60] within long-term videos for comprehending fine-
grained information. From this perspective, existing VideoLLMs treat all
visual tokens equally and aggregate them into compact representations through
pooling operations [35, 34] and query aggregation [24, 61, 42]. While they
successfully capture the global semantic context spanning the entire long-term
videos, they often overlook preserving the local information for the short-
term segments and the temporal structure of different short-term components,
_e.g_., the order of events or sub-actions. However, exclusively modeling the
temporal structure through the sequence of local features may still lead to
inconsistent recognition across different segments and impede the overall
understanding of the videos. To comprehend the content in long videos, the
human visual system relies on a blend of local and global information [44].
Building on this insight, earlier approaches in video object detection [52,
54] suggest integrating global semantics into local localization descriptors,
motivating us to include global semantic information into the sequence of
local features for enriching the context understanding for each short-term
segment.

In this paper, we present LongVLM, a simple yet effective VideoLLM for
efficient long video understanding, as illustrated in Fig. 2. We propose to
extract video representations as sequences of short-term local features, and
integrate global semantics into each short-term segment feature. Specifically,
we begin by uniformly sampling a sequence of video frames from long-term
videos and utilize a pretrained visual encoder, _e.g_., CLIP-ViT-L/14 [40], to
extract visual features for each individual video frame. These frame-level
features include the [CLS] tokens from a range of encoder layers and the patch
features from the last second layer of the visual encoder. Then, we divide the
sequence of patch features along the temporal dimension, resulting in a
sequence of short-term segments. Each segment is considered as a local unit in
the videos and includes patch features of the video frames within that
segment. To reduce computational costs and obtain the compact features for
each segment, a token merging module is employed to aggregate these patch
features for the specific segment into a condensed set of tokens. In this way,
we obtain the local features for each segment. We next concatenate these
features sequentially to explicitly preserve the temporal order of the short-
term segments in long-term videos. Moreover, we average the [CLS] tokens from
each video frame along the temporal dimension to represent the global semantic
information of the entire video. To integrate the global information, we
sequentially arrange the averaged [CLS] tokens and the segment-level features,
and then feed them into the LLM after passing through a projection layer.
Benefiting from the causal attention mechanism in the LLM, we simultaneously
achieve temporal structure modeling over the sequence of short-term segments
and inject global semantics into the local features. Finally, the LLM
generates responses based on the input sequence, which is composed of the
obtained video representation and the designed system command with the
specific user queries.

Overall, our main contributions is threefold:

  * •

We propose LongVLM, a simple yet effective VideoLLM for efficient long-term
video understanding at a fine-grained level while maintaining affordable
computational cost.

  * •

We propose decomposing long videos into short segments and extracting local
features for each segment to preserve their temporal order. To compactly
represent each segment, we propose a hierarchical token merging module to
aggregate visual tokens. Additionally, we integrate global semantics into each
segment to enhance context understanding.

  * •

Extensive experiments on VideoChatGPT benchmark [35] and zero-shot video
question-answering datasets [59, 55] demonstrate that our LongVLM surpasses
the previous state-of-the-art methods by a significant margin while generating
more precise and accurate response at fine-grained level for long-term videos.

##  2 Related Work

###  2.1 Large Language Models

Large Language Models (LLMs) have revolutionized natural language processing
in recent years. Pretrained on large text corpora, LLMs like GPT [2], OPT
[62], and LLaMA [45, 46] utilize auto-regressive Transformer models to predict
subsequent tokens, showcasing remarkable adaptability and generalization.
Models such as InstructGPT [38], ChatGPT [36], and GPT-4 [37] benefit from
instruction-tuning techeque [51] on instructional datasets, leveraging the
knowledge of pretrained LLMs and demonstrating improvements in diverse
conversational interaction capabilities. This strategy is widely adopted in
open-source models like Alpaca [43] and Vicuna [7], which build upon the
advancements made by LLaMA [45] using specially designed instruction pairs.
Drawing from the advancement of LLMs, recent Multi-modal Large Language Models
(MLLMs), _e.g_., BLIP-2 [23], Mini-GPT4 [65], LLaVA [30], LLama Adapter v2
[15], have demonstrated the feasibility of enabling visual conversation
capabilities of LLMs over input images through instruction tuning on image-
text instruction datasets. Our model aims to utilize existing MLLMs to develop
efficient video dialogue model for long-term video understanding.

###  2.2 Video-based Large Language Models

Building upon pretrained image-language models, traditional approaches [33,
13, 11, 22, 50, 28, 6, 26] are proposed to address various video-language
learning tasks.With the advent of LLMs, recent approaches are exploring the
potential of developing Video-based Large Language Models (VideoLLMs) to unify
various video-language understanding scenarios through human-video dialogue
interactions. Existing VideoLLMs typically follow a common paradigm, which
involves using a pretrained visual encoder to encode visual features, a
projection layer to convert visual representations into the text latent space
of LLMs, and a pretrained LLM for response generation. VideoChatGPT [35] and
Valley [34] rely on pooling over visual tokens to obtain compact visual
representations. VideoChat [24] utilizes pretrained video foundation models
[25, 50] and Q-Former from BLIP-2 [23] to aggregate video representations.
Video-LLaMA [61] proposes a Video Q-Former and an Audio Q-Former, enabling
multiple modalities for video comprehension, while Video-ChatCaptioner [5]
employs ChatGPT [36] to summarize video descriptions in multiple rounds of
interactive question-and-answer conversation. Recently, MovieChat [42]
proposes an effective memory management mechanism to enable LLMs to reason
over hour-long videos. Multiple video-centric instruction datasets [34, 24,
35, 42] have also been proposed to finetune VideoLLMs for better video
understanding capacity. Moreover, BT-Adapter [31] proposes a temporal adapter
alongside the visual encoder for post-pretraining, while Video-Teller [29]
highlights the importance of modality alignment in pretraining. Overall, these
VideoLLMs rely on pooling and query aggregation on the whole long videos to
extract visual representation for developing VideoLLMs, which lack of local
information for fine-grained understanding in long videos. In contrast to
existing VideoLLMs, we propose a simple yet effective framework that is
feasible for aggregating both local and global information in long-term videos
and preserves fine-grained content understanding.

###  2.3 Long-term Video Processing

Long-term video understanding poses several challenges due to the need to
exploit complicated spatial-temporal dependencies while removing temporal
redundancy over extended time duration. Previous studies propose efficient
architectures [9, 20], temporal pooling/aggregation [12, 41, 64, 53], dynamic
clip selection [21, 17, 16] to aggregate video representation while removing
redundant information in videos. Other methods in video-language understanding
tasks suggest to capture event temporality, causality, and dynamics in long-
term videos by designing temporal alignment modules [18, 3]. Memory mechanism
is also widely adopted in video dense prediction tasks [32, 52, 58, 52, 54,
63] to capture historical information and maintain temporal coherence, which
results in more accurate and consistent prediction over time in long-term
videos. In contrast to previous methods, we investigate the techniques for
long-term video understanding in VideoLLMs. We propose to aggregate both local
segment-level information and global semantic information, empowering MLLMs
enhanced fine-grained understanding for long-term videos.

##  3 Method

Figure 2: Overall architecture of the proposed LongVLM. We start by uniformly
sampling $T$ frames from a video and employing a visual encoder to extract
frame-level features. We divide the input video into $S$ segments, each with
$K$ frames. To obtain compact local features, we apply a hierarchical token
merging module within each segment. These segment-level features are
concatenated sequentially to explicitly preserve the temporal order of
multiple short-term segments in long videos. Additionally, we incorporate
[CLS] tokens to aggregate global semantic features. The global features and
the sequence of local features are concatenated to form the video
representations. Finally, the projected visual features are combined with the
tokenized system command and user queries and inputted into the LLM to
generate the responses.

In Sec. 3.1, we introduce the overall architecture and generation pipeline of
the proposed LongVLM. In Sec. 3.2, we introduce the process of constructing
local representation via short-term feature aggregation. In Sec. 3.3, we
discuss the integration of both local segment-level feature and global
semantic feature for the video representation.

###  3.1 Overall Architecture

The overall architecture consists of three components: a visual encoder, a
projection layer, and a large language model, as illustrated in Fig. 2.

Given an input video ${\mathcal{V}}\in{\mathbb{R}}^{T\times H\times W\times
3}$, we employ a visual encoder to extract frame-level features $\\{{\bf
X}^{t},{\bf P}^{t}\\}_{t=1}^{T}$ for each video frame independently. Following
previous methods [30, 35], we utilize the patch feature ${\bf
P}^{t}\in{\mathbb{R}}^{N\times d}$ from the penultimate encoder layer for each
video frame, where $N,d$ are the number of patch tokens and the channel
dimension of the visual encoder, respectively. Additionally, we gather the
[CLS] tokens ${\bf X}^{t}\in{\mathbb{R}}^{E\times d}$ from $E$ selected
encoder layers for each individual video frame.

To enable fine-grained understanding in long videos, we propose to divide long
videos into a sequence of short-term segments, where each segment corresponds
to the local features in the long videos. Without loss of generality, the
input video ${\mathcal{V}}$ is divided into $S$ segments, where each segment
includes $K$ frames, _i.e_., $K=\frac{T}{S}$. We collect patch features within
the $s^{th}$ segment, _i.e_., ${\bf V}^{s}=\\{{\bf
P}^{t}\\}_{t=(s-1)K}^{t=sK}$, and apply a token merging module
${\mathcal{G}}(\cdot)$ to aggregate ${\bf V}^{s}$ into the compact segment-
level feature ${\bf{Z}}^{s}={\mathcal{G}}({\bf V}^{s})$. These segment-level
features are sequentially concatenated as the sequence of local representation
${\bf L}$, explicitly preserving temporal order of short-term segments in long
videos. Furthermore, to integrate global semantic information, we propose to
collect the [CLS] tokens for each frame from $E$ encoder layers and average
them in the time dimension, resulting in our global feature ${\bf G}$.

We forward the concatenated global features and the sequence of local features
into a projection layer ${\mathcal{F}}_{V}(\cdot;{\bf W})$ to obtain the
projected visual feature, _i.e_., ${\bf H}_{V}=[\hat{{\bf G}},\hat{{\bf
L}}]=[{\mathcal{F}}_{V}({\bf G};{\bf W}),{\mathcal{F}}_{V}({\bf L};{\bf W})]$.
The projected visual feature ${\bf H}_{V}$ are concatenated with the tokenized
system command and user queries ${\bf H}_{q}$, which are inputted into LLM for
response generation ${\bf H}_{a}$, _i.e_., ${\bf
H}_{a}={\mathcal{F}}_{LLM}([{\bf H}_{V},{\bf H}_{q}])$, where
${\mathcal{F}}_{LLM}(\cdot)$ denotes an LLM model.

###  3.2 Local Feature Aggregation

After obtaining the frame-level patch feature $\\{{\bf P}^{t}\\}_{t=1}^{T}$
from the last second layer of visual encoder, previous methods either apply
factorized spatio-temporal pooling [35, 34], or utilize query aggregation [24,
61, 42] over all visual tokens, which may miss local information referring to
the short-term events or actions. Nevertheless, videos have heavy spatio-
temporal redundancy, which results in redundant computational costs by
directly considering all the patch features as the local representation for
each segment. Therefore, we propose to aggregate compact visual features
within each short-term segment. Specifically, we collect the patch feature for
the $s^{th}$ segment ${\bf V}^{s}=\\{{\bf
P}^{t}\\}_{t=(s-1)K}^{sK}\in{\mathbb{R}}^{KN\times d}$ and apply a
hierarchical token merging module to aggregate the local feature while
reducing the number of visual tokens. Inspired by ToMe [1], we resort to the
bipartite soft matching method and gradually merge the visual tokens for each
short-term segment. At the $i^{th}$ step, we randomly partition the $R_{i}$
tokens into two non-overlap token sets ${\mathbb{P}}_{i}$ with $r_{i}$ tokens
and ${\mathbb{Q}}_{i}$ with $R_{i}-r_{i}$ tokens, where initial $R_{0}=KN$.
Then we calculate the similarity scores between the tokens in set
${\mathbb{P}}_{i}$ and ${\mathbb{Q}}_{i}$ based on the patch features. To
obtain the similarity scores, each visual token is divided into $C$ heads
along channel dimension, each with $\frac{d}{C}$ channels. The similarity
score for each token pair is obtained by averaging the cosine similarity
scores over all heads following

$a^{p_{i}q_{i}}=\frac{1}{C}[\sum_{c=1}^{C}{cos({\bf p}_{c}^{(p_{i})},{\bf
p}_{c}^{(q_{i})})}],$

where $p_{i}\in\\{1,...,r_{i}\\}$ and $q_{i}\in\\{1,...,(R_{i}-r_{i})\\}$ are
the indexes of patch feature ${\bf p}$ in set ${\mathbb{P}}_{i}$ and set
${\mathbb{Q}}_{i}$, respectively. We select the top-$r_{i}$ token pairs with
the highest similarity scores and merge the paired tokens by average pooling.
Finally, the remaining tokens in the two sets are concatenated back together,
resulting in $R_{i}-r_{i}$ tokens after the $i^{th}$ merging step. We
iteratively merge the tokens within each short-term segment, until the number
of visual tokens reaches $M$, where $M<<K\times N$. The compact local feature
for the $s^{th}$ segment is denoted as ${\bf{Z}}^{s}=\\{{\bf
z}_{m}\\}_{m=1}^{M}\in{\mathbb{R}}^{M\times d}$. These segment-level features
are concatenated sequentially as the sequence of local features ${\bf
L}=\\{{\bf{Z}}^{s}\\}_{s=1}^{S}=[{\bf z}_{1}^{1},...,{\bf z}_{M}^{1},...,{\bf
z}_{1}^{S},...,{\bf z}_{M}^{S}]\in{\mathbb{R}}^{MS\times d}$. Thanks to the
positional encoding in the LLM, the sequence of local representation ${\bf L}$
explicitly preserves the order of short-term segments in long-term videos,
enabling LLMs to be aware of the temporal structure of multiple event
occurrences in long videos. By utilizing the token merging module, we
efficiently encode compact local features for each segment while eliminating
redundancy in the visual token sequence.

###  3.3 Global Semantics Integration

The local feature provides fine-grained information about different events or
actions in the generation process of VideoLLMs, enhancing the detailed
understanding capabilities of the LongVLM. However, the local feature for each
segment may be insufficient for the model to reason the relationship between
different segments and generate reasonable response over the entire videos.
Therefore, we additionally introduce global semantic features to enrich the
local feature sequence with contextual information. Specifically, we collect
the [CLS] tokens from $E$ encoder layers from each video frame, _i.e_.,
$\\{{\bf x}^{t}_{e}\\}_{e=1,t=1}^{e=E,t=T}$, and then average the [CLS] tokens
along temporal dimension, resulting in $\bar{{\bf
X}}_{e}=\operatorname{AvgPool}(\\{{\bf
x}_{e}^{t}\\}_{t=1}^{T})\in{\mathbb{R}}^{d},e\in[1,...,E]$. By default, $E$
can be the number of layers in the visual encoder. However, previous studies
demonstrate different properties between intermediate features from the
shallow layers and deeper layers in ViT models [27, 39], and showcase the
deeper layers tend to aggregate global semantics. Thus, we concatenate the
$E$-scale features along sequentially, resulting in the global semantic
feature for the entire video, _i.e_., ${\bf G}=\\{\bar{{\bf
X}}_{e}\\}_{e=1}^{E}\in{\mathbb{R}}^{E\times d}$.

Following the previous studies, a projection layer converts the visual
features into the language space, and then the visual features are
concatenated with the instruction as the input of LLM. By utilizing the
attention mechanism in the LLM, we can easily enable each token in the local
feature to attend to the global semantic feature, thereby achieving
straightforward injection of global semantics into the local feature.

Remark. To address the risk of overlooking detailed understanding in long-term
videos, we propose to divide long videos into multiple short-term segments and
aggregate local spatial-temporal representation for each segment and
preserving the temporal structure over the sequence of local feature vectors.
Moreover, we enrich the local features with context information for better
response generation by integrating global semantic information into short-term
features. Different from previous approaches that rely solely on global
semantics for long video understanding, we present a straightforward yet
effective VideoLLM for achieving fine-grained understanding in long-term
videos.

##  4 Experiments

###  4.1 Experimental Settings

Datasets and evaluation metrics. We conduct quantitative evaluations of our
model using the VideoChatGPT benchmark [35] to assess its performance in
generating text from videos. The benchmark comprises 500 videos sampled from
ActivityNet-v1.3 dataset [10], with 2000, 2000, 2000, 500, and 1000 questions
in terms of five evaluation aspects: Correctness Information(CI), Detail
Orientation(DO), Contextual Understanding(CU), Temporal Understanding(TU) and
Consistency(C). Additionally, we evaluate the model on the zero-shot question-
answering task using the ANET-QA [59] dataset, which contains 8000 QA pairs
for 800 videos sampled from ActivityNet-v1.3 dataset [10]. The videos range
from several seconds to minutes long and cover a wide range of daily human
activities. We also utilize MSRVTT-QA [56, 55](72821 QA pairs for 2990 videos)
and MSVD-QA [4, 55](13157 QA pairs for 520 videos) to evaluate the model
performance, derived from publicly available video captioning, MSRVTT [56],
and MRVDC [4], respectively. Following the evaluation protocol outlined in
Video-ChatGPT [35], we employ ChatGPT [36] for response evaluation and report
the generation quality scores on VideoChatGPT benchmark and the answer
accuracy and quality scores of models on zero-shot video QA tasks.

Table 1: Comparison with state-of-the-art methods on video conversation benchmark [35] in terms of five evaluation aspects including Correctness of Information (CI), Detail Orientation (DO), Contextual Understanding (CU), Temporal Understanding (TU) and Consistency (C), along with the average scores across all evaluation metrics (Mean). We also report the dataset scale used for finetuning the model. Method | Data Source | CI | DO | CU | TU | C | Mean
---|---|---|---|---|---|---|---
VideoChat [24] | 10M | 2.25 | 2.50 | 2.54 | 1.98 | 1.84 | 2.22
LLaMA Adapter v2 [15] | 700K | 2.03 | 2.32 | 2.30 | 1.98 | 2.15 | 2.16
Video LLaMA [61] | 10M | 1.96 | 2.18 | 2.16 | 1.82 | 1.79 | 1.98
Video-ChatGPT [35] | 100K | 2.50 | 2.57 | 2.69 | 2.16 | 2.20 | 2.42
Valley [34] | 234k | 2.43 | 2.13 | 2.86 | 2.04 | 2.45 | 2.38
BT-Adapter [31] | 10M | 2.16 | 2.46 | 2.89 | 2.13 | 2.20 | 2.37
BT-Adapter [31] | 10M+100K | 2.68 | 2.69 | 3.27 | 2.34 | 2.46 | 2.69
LongVLM | 100K | 2.76 | 2.86 | 3.34 | 2.39 | 3.11 | 2.89
Table 2: Comparison with state-of-the-art methods on three zero-shot question answering datasets. We report the Accuracy (Acc.) and Score for the generated answer for each question. We also report the dataset scale used for finetuning the model. Method | Data source | ANET-QA | MSRVTT-QA | MSVD-QA
---|---|---|---|---
Acc. | Score | Acc. | Score | Acc. | Score
FrozenBiLM [57] | 10M | 24.7 | - | 16.8 | - | 32.2 | -
VideoChat [24] | 10M | 26.5 | 2.2 | 45.0 | 2.5 | 56.3 | 2.8
LLaMA Adapter v2 [15] | 700K | 34.2 | 2.7 | 43.8 | 2.7 | 54.9 | 3.1
Video LLaMA [61] | 10M | 12.4 | 1.1 | 29.6 | 1.8 | 51.6 | 2.5
Video-ChatGPT [35] | 100K | 35.2 | 2.7 | 49.3 | 2.8 | 64.9 | 3.3
Valley [34] | 234K | 45.1 | 3.2 | 51.1 | 2.9 | 60.5 | 3.3
BT-Adapter [31] | 10M+100K | 45.7 | 3.2 | 57.0 | 3.2 | 67.5 | 3.7
LongVLM | 100K | 47.6 | 3.3 | 59.8 | 3.3 | 70.0 | 3.8

Implementation details. We employ CLIP-ViT-L/14 [40] as the visual encoder and
Vicuna-7B-v1.1 [7] as the LLM. We initialize them with the pretrained weights
in LLaVA-7B-v1.1 [30]. We finetune the model on the Video-ChatGPT-100K
instruction dataset [35] for 3 epochs, with a learning rate of $2\times
e^{-5}$ and a batch size of 32. We only finetune the linear projection layer
to align the visual features into the input space of the LLM, keeping both the
visual encoder and LLM frozen. It takes three hours to train three epochs on 4
A100 80GB GPUs. During training and inference, we sample $T=100$ video frames
for each video, and resize the frames to $224\times 224$ resolutions. We set
$S=10$ for each video, and the number of tokens in each segment-level feature
is $M=30$. We collect the $[CLS]$ tokens from the last five encoder layers and
average them along the temporal axis, resulting in $E=5$ tokens as the global
semantic features. Therefore, the length of visual tokens for a video sequence
is $M\times S+E=305$.

###  4.2 Main Results

Results on the video-based generation benchmark. In Tab. 1, we present a
comprehensive evaluation of our LongVLM against state-of-the-art models on the
video-based generation benchmark [35]. Our LongVLM outperforms all other
models across all the evaluation aspects. Particularly noteworthy is its
significant advantage in Detail Orientation (DO) and Consistency (C), showing
improvements of +0.17 and +0.65, respectively, over BT-Adapter [31]. These
results underscore superior capability of LongVLM in fine-grained video
understanding and robust generation performance.

Results on zero-shot video question-answering. In Tab. 2, we compare the
performance of LongVLM against various existing methods on three zero-shot
video QA datasets: ANET-QA [59, 10], MSRVTT-QA [56, 55] and MSVD-QA [4, 55].
Our model achieves the highest accuracy of 47.6%, 59.8%, and 70.0% on the
three QA datasets, surpassing the previous SOTA approach BT-Adapter [31] by
1.9%, 2.8% and 2.5%, respectively. Furthermore, we achieve the highest score
in terms of generation quality over the three datasets.

###  4.3 Ablation Study

Effects of local feature aggregation. As discussed in Sec. 1, pooling
operations or query aggregation might overlook local information in achieving
fine-grained understanding in long-term videos. To this end, we introduce
short-term segment-level features to retain local information and temporal
structure within long-term videos. The first two rows in Tab. 3 present the
effects for the design of using local features as the visual representations
for videos. We compare the token merging module with local pooling operation
using a pooling kernel and stride of [5,4,4] within each short-term segment.
The proposed hierarchical merging module achieves higher scores over spatial-
temporal pooling operation. This could be attributed to the dynamic
aggregation mechanism via the token similarity in the merging module, while
averaging pooling statically aggregates visual tokens within each small 3D
window. Additionally, we observe that aggregating local features for short-
term segments either improves or maintains comparable performance across all
evaluation metrics compared to the SOTA models which extract global semantics
only, highlighting the significance of preserving local features for short-
term segments in long-term video understanding.

Table 3: Ablation of the local and global aggregation design. Pooling: Using local 3D pooling kernel to obtain local features; Merging: Using the proposed hierarchical token merging to obtain local features; $[{\bf L},{\bf G}]$: Concatenating local feature then global feature; $[{\bf G},{\bf L}]$: Concatenating global feature then local feature. Variants | Local | Global | CI | DO | CU | TU | C | Mean
---|---|---|---|---|---|---|---|---
Pooling | ✓ | ✗ | 2.53 | 2.64 | 3.13 | 2.29 | 2.61 | 2.64
Merging | ✓ | ✗ | 2.62 | 2.74 | 3.15 | 2.23 | 2.86 | 2.72
$[{\bf L},{\bf G}]$ | ✓ | ✓ | 2.69 | 2.81 | 3.31 | 2.31 | 2.99 | 2.82
$[{\bf G},{\bf L}]$ | ✓ | ✓ | 2.76 | 2.86 | 3.34 | 2.39 | 3.11 | 2.89
Table 4: Effect of $M$. We evaluate the model performance by varying $M$ from
$\\{10,20,30,40\\}$, while keeping the token length of global semantic
features at $E=5$.

M | CI | DO | CU | TU | C | Mean
---|---|---|---|---|---|---
10 | 2.61 | 2.72 | 3.22 | 2.26 | 2.78 | 2.72
20 | 2.72 | 2.86 | 3.34 | 2.34 | 2.96 | 2.84
30 | 2.76 | 2.86 | 3.34 | 2.39 | 3.11 | 2.89
40 | 2.74 | 2.84 | 3.39 | 2.34 | 3.06 | 2.87
(a) Video-ChatGPT Benchmark.

M | Accuracy | Score | Memory(G)
---|---|---|---
10 | 44.6 | 3.2 | 14.65
20 | 45.7 | 3.4 | 14.74
30 | 47.6 | 3.3 | 14.86
40 | 46.0 | 3.3 | 14.96
(b) ANET-QA.

Table 5: Effect of the number of selected encoder layers. We evaluate the performance of model varying $E$ in {1, 5, 10, 15, 20, 24}, keeping $M=30,K=10$. E | 1 | 5 | 10 | 15 | 20 | 24
---|---|---|---|---|---|---
CI | 2.74 | 2.76 | 2.78 | 2.72 | 2.62 | 2.65
TU | 2.32 | 2.39 | 2.30 | 2.16 | 2.28 | 2.22
C | 3.04 | 3.11 | 3.04 | 2.97 | 2.93 | 2.81

Effects of global semantics integration. Inspired by the human visual system
that using a combination of local and global information for recognizing video
content [44], we propose to enhance visual representation by injecting global
semantic features into local features. The last two rows in Tab. 3 demonstrate
the effects of integrating global semantics. Compared to the first two rows,
introducing global semantic features significantly enhances performance
compared to models using local features only across all evaluation aspects.
The notable improvements in Contextual Understanding (CU) and Consistency (C)
underscore the significance of integrating global semantic information with
local short-term features. Moreover, concatenating global features before
local features yields better results than the opposite concatenation order.
This arrangement allows each the local feature to access the global semantic
information across the entire video by leveraging the causal attention
mechanism in the LLM. Consequently, this enriches the contextual information
of the local features and enhances the response consistency of the model.

(a) Quantitative results on Video-ChatGPT-100K [35] benchmark and the task of
zero-shot question answering on ANet-QA [59], MSRVTT-QA [56, 55] and MSVD-QA
[4, 55]. Our model delivers the best performance on multiple evaluation
aspects, compared with the state-of-the-art video dialogue models: Video Chat
[24], LLaMA Adapter [15], Video LLaMA [61], Video-ChatGPT [35], and BT-Adapter
[31]. Evaluation metrics and comparison details are given in Section 4.2.

(b) An example of zero-shot question answering. Video duration is 3 minutes
and 46 seconds.

Figure 3: Quantitative results and qualitative examples of our LongVLM. Figure
4: Two examples from Video-ChatGPT benchmark [35]. Text highlighted in bold
green denotes correct content, while text in red indicates errors. The video
durations are 42 seconds and 1 minute 39 seconds, respectively.  Figure 5:
More generative examples from the Video-ChatGPT benchmark [35] of the proposed
LongVLM. Text in bold denotes the correct content. The LongVLM is able to
capture the detailed information videos.

Effects of $M$. We report the model performance on VideoChatGPT benchmark and
ANET-QA task on the selection of $M$, _i.e_., $M=\\{10,20,30,40\\}$, keeping
the same number of global semantic tokens in Tab. 4. For ANET-QA task, we also
report the averaging GPU memory usage for generating each response. In
general, the token length involves a trade-off between memory costs and
performance. A shorter token sequence reduces computational costs for
generating a single new token using LLM, thereby lowering memory costs for
generating responses to individual user queries. However, it may also lead to
insufficient visual information for generating accurate responses. The
performance of our model is beneficial from the suitable length of visual
tokens. Specifically, using $M=10$ leads to the lowest performance in both
tasks, while it still shows a comparable performance compared to the SOTA
models, indicating the effectiveness of our model design. Increasing $M$ from
10 to 40 results in a significant improvement in terms of most evaluation
aspects, while the setting of $M=40$ leads to neglecting improvement but
requires more memory cost compared to $M=30$. Therefore, we choose $M=30$ for
our model.

Effects of $E$. We evaluate the model performance on the VideoChatGPT
benchmark with varying $E$ by selecting the [CLS] tokens from the last
$1,5,10,15,20,24$ visual encoder layers, while maintaining the same $M$ for
local features. As depicted in Tab. 5, increasing the number of global
semantic tokens from 1 to 5 improves the generation quality scores in terms of
all evaluation aspects. However, increasing $E$ from 5 to 24 leads to degraded
performance, possibly because the [CLS] tokens from earlier layers carry less
semantic information for the model. Therefore, we choose $E=5$ in our model.

###  4.4 Qualitative Results

As illustrated in Sec. 1, Fig. 1(b) demonstrates the advancement of our model
in terms of fine-grained understanding in long-term videos. Despite taking the
same number of video frames as input, our model excels in capturing detailed
information within the videos, discerning nuances like fixing chain rather
than fixing wheel. In comparison, Video-ChatGPT [35] can describe the overall
video content but may inaccurately recognize detailed information. For
instance, it might identify objects such as helmets and gloves in the scene
but erroneously recognize the location for these objects. This emphasize the
importance of decomposing long videos into multiple short-term segments and
aggregate local features to achieve fine-grained understanding in videos. The
examples depicted in Fig. 4 ablate the effectiveness of integrating global
semantic information into local short-term features. With global semantics
integration, the model is able to recognize the actions (long jump) and
objects (axe) compared to the variant that using local features only. We
provide more generated examples in Fig. 3(b) and Fig. 5 from ANET-QA and
Video-ChatGPT benchmark, respectively, which showcase the precise description
of the video content generated by our LongVLM.

##  5 Conclusion

In this work, we have introduced LongVLM, an effective and efficient
multimodal LLM designed for long-term video understanding. By extracting local
features for short-term segments, we efficiently model local dependencies
while preserving the temporal structure of sequential events in long-term
video sequences. Through the integration of local and global information,
LongVLM captures detailed information and provides consistent and accurate
responses for long-term videos.

Limitations and Further work. While we introduce a novel video conversation
model for fine-grained long-term video comprehension, our framework is
specifically designed for video-to-text generation scenarios. Future work may
include extending our framework into video-centric multimodal generation tasks
and training the model on large-scale, extended-duration videos for long-
context understanding.

Figure 6: Additional response examples from the Video-ChatGPT benchmark [35]
generated by our LongVLM, focusing on temporal understanding (TU) evaluation
perspective. Text in bold indicates the correct content. Figure 7: Additional
examples from Zero-shot ANET-QA [59] generated by the proposed LongVLM. Text
in bold indicates the correct content.  Figure 8: Response examples from Zero-
shot MSVD-QA and MSRVTT-QA [55] generated by the proposed LongVLM. Text in
bold indicates the correct content.

## References

  * [1] Bolya, D., Fu, C.Y., Dai, X., Zhang, P., Feichtenhofer, C., Hoffman, J.: Token merging: Your vit but faster. In: ICLR (2022)
  * [2] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. NIPS 33, 1877–1901 (2020)
  * [3] Buch, S., Eyzaguirre, C., Gaidon, A., Wu, J., Fei-Fei, L., Niebles, J.C.: Revisiting the" video" in video-language understanding. In: CVPR. pp. 2917–2927 (2022)
  * [4] Chen, D.L., Dolan, W.B.: Collecting highly parallel data for paraphrase evaluation. In: ACL. Portland, OR (June 2011)
  * [5] Chen, J., Zhu, D., Haydarov, K., Li, X., Elhoseiny, M.: Video chatcaptioner: Towards the enriched spatiotemporal descriptions. arXiv preprint arXiv:2304.04227 (2023)
  * [6] Cheng, F., Wang, X., Lei, J., Crandall, D., Bansal, M., Bertasius, G.: Vindlu: A recipe for effective video-and-language pretraining. In: CVPR. pp. 10739–10750 (2023)
  * [7] Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., et al.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. https://vicuna.lmsys.org (2023)
  * [8] Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning (2023)
  * [9] Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual recognition and description. In: CVPR. pp. 2625–2634 (2015)
  * [10] Fabian Caba Heilbron, Victor Escorcia, B.G., Niebles, J.C.: Activitynet: A large-scale video benchmark for human activity understanding. In: CVPR. pp. 961–970 (2015)
  * [11] Fang, H., Xiong, P., Xu, L., Chen, Y.: Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097 (2021)
  * [12] Fernando, B., Gavves, E., Oramas, J., Ghodrati, A., Tuytelaars, T.: Rank pooling for action recognition. PAMI 39(4), 773–787 (2016)
  * [13] Fu, T.J., Li, L., Gan, Z., Lin, K., Wang, W.Y., Wang, L., Liu, Z.: Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681 (2021)
  * [14] Gaidon, A., Harchaoui, Z., Schmid, C.: Temporal localization of actions with actoms. TPAMI 35(11), 2782–2795 (2013)
  * [15] Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., et al.: Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010 (2023)
  * [16] Ghodrati, A., Bejnordi, B.E., Habibian, A.: Frameexit: Conditional early exiting for efficient video recognition. In: CVPR. pp. 15608–15618 (2021)
  * [17] Gowda, S.N., Rohrbach, M., Sevilla-Lara, L.: Smart frame selection for action recognition. In: AAAI. vol. 35, pp. 1451–1459 (2021)
  * [18] Han, T., Xie, W., Zisserman, A.: Temporal alignment networks for long-term video. In: CVPR. pp. 2906–2916 (2022)
  * [19] Hussein, N., Gavves, E., Smeulders, A.W.: Timeception for complex action recognition. In: CVPR. pp. 254–263 (2019)
  * [20] Kondratyuk, D., Yuan, L., Li, Y., Zhang, L., Tan, M., Brown, M., Gong, B.: Movinets: Mobile video networks for efficient video recognition. In: CVPR. pp. 16020–16030 (2021)
  * [21] Korbar, B., Tran, D., Torresani, L.: Scsampler: Sampling salient clips from video for efficient action recognition. In: ICCV. pp. 6232–6242 (2019)
  * [22] Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T.L., Bansal, M., Liu, J.: Less is more: Clipbert for video-and-language learning via sparse sampling. In: CVPR. pp. 7331–7341 (2021)
  * [23] Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)
  * [24] Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., Qiao, Y.: Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 (2023)
  * [25] Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Wang, L., Qiao, Y.: Uniformerv2: Unlocking the potential of image vits for video understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1632–1643 (2023)
  * [26] Li, K., Wang, Y., Li, Y., Wang, Y., He, Y., Wang, L., Qiao, Y.: Unmasked teacher: Towards training-efficient video foundation models. arXiv preprint arXiv:2303.16058 (2023)
  * [27] Li, K., Wang, Y., Peng, G., Song, G., Liu, Y., Li, H., Qiao, Y.: Uniformer: Unified transformer for efficient spatial-temporal representation learning. In: ICLR (2021)
  * [28] Li, L., Gan, Z., Lin, K., Lin, C.C., Liu, Z., Liu, C., Wang, L.: Lavender: Unifying video-language understanding as masked language modeling. In: CVPR. pp. 23119–23129 (2023)
  * [29] Liu, H., Fan, Q., Liu, T., Yang, L., Tao, Y., Huang, H., He, R., Yang, H.: Video-teller: Enhancing cross-modal generation with fusion and decoupling. arXiv preprint arXiv:2310.04991 (2023)
  * [30] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. arXiv preprint arXiv:2304.08485 (2023)
  * [31] Liu, R., Li, C., Ge, Y., Shan, Y., Li, T.H., Li, G.: One for all: Video conversation is feasible without video instruction tuning. arXiv preprint arXiv:2309.15785 (2023)
  * [32] Lu, Y., Lu, C., Tang, C.K.: Online video object detection using association lstm. In: ICCV. pp. 2344–2352 (2017)
  * [33] Luo, H., Ji, L., Shi, B., Huang, H., Duan, N., Li, T., Li, J., Bharti, T., Zhou, M.: Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353 (2020)
  * [34] Luo, R., Zhao, Z., Yang, M., Dong, J., Qiu, M., Lu, P., Wang, T., Wei, Z.: Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207 (2023)
  * [35] Maaz, M., Rasheed, H., Khan, S., Khan, F.S.: Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424 (2023)
  * [36] OpenAI: Chatgpt. https://openai.com/blog/chatgpt/ (2023)
  * [37] OpenAI: GPT-4 technical report (2023)
  * [38] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. NIPS 35, 27730–27744 (2022)
  * [39] Pan, Z., Zhuang, B., He, H., Liu, J., Cai, J.: Less is more: Pay less attention in vision transformers. In: AAAI. vol. 36, pp. 2035–2043 (2022)
  * [40] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763. PMLR (2021)
  * [41] Sener, F., Singhania, D., Yao, A.: Temporal aggregate representations for long-range video understanding. In: ECCV. pp. 154–171. Springer (2020)
  * [42] Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Guo, X., Ye, T., Lu, Y., Hwang, J.N., et al.: Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449 (2023)
  * [43] Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., Hashimoto, T.B.: Stanford alpaca: An instruction-following llama model (2023)
  * [44] Tian, Y., Yang, M., Zhang, L., Zhang, Z., Liu, Y., Xie, X., Que, X., Wang, W.: View while moving: Efficient video recognition in long-untrimmed videos. In: ACMMM. pp. 173–183 (2023)
  * [45] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
  * [46] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
  * [47] Wang, L., Qiao, Y., Tang, X.: Latent hierarchical model of temporal structure for complex activity classification. TIP 23(2), 810–822 (2013)
  * [48] Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., Van Gool, L.: Temporal segment networks: Towards good practices for deep action recognition. In: ECCV. pp. 20–36. Springer (2016)
  * [49] Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., Van Gool, L.: Temporal segment networks for action recognition in videos. TPAMI 41(11), 2740–2755 (2018)
  * [50] Wang, Y., Li, K., Li, Y., He, Y., Huang, B., Zhao, Z., Zhang, H., Xu, J., Liu, Y., Wang, Z., et al.: Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191 (2022)
  * [51] Wei, J., Bosma, M., Zhao, V.Y., Guu, K., Yu, A.W., Lester, B., Du, N., Dai, A.M., Le, Q.V.: Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021)
  * [52] Wu, C.Y., Feichtenhofer, C., Fan, H., He, K., Krahenbuhl, P., Girshick, R.: Long-term feature banks for detailed video understanding. In: CVPR. pp. 284–293 (2019)
  * [53] Wu, C.Y., Krahenbuhl, P.: Towards long-form video understanding. In: CVPR. pp. 1884–1894 (2021)
  * [54] Wu, C.Y., Li, Y., Mangalam, K., Fan, H., Xiong, B., Malik, J., Feichtenhofer, C.: Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. In: CVPR. pp. 13587–13597 (2022)
  * [55] Xu, D., Zhao, Z., Xiao, J., Wu, F., Zhang, H., He, X., Zhuang, Y.: Video question answering via gradually refined attention over appearance and motion. In: ACMMM. pp. 1645–1653 (2017)
  * [56] Xu, J., Mei, T., Yao, T., Rui, Y.: Msr-vtt: A large video description dataset for bridging video and language. In: CVPR. pp. 5288–5296 (2016)
  * [57] Yang, A., Miech, A., Sivic, J., Laptev, I., Schmid, C.: Zero-shot video question answering via frozen bidirectional language models. NeurIPS 35, 124–141 (2022)
  * [58] Yang, T., Chan, A.B.: Learning dynamic memory networks for object tracking. In: ECCV. pp. 152–167 (2018)
  * [59] Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., Tao, D.: Activitynet-qa: A dataset for understanding complex web videos via question answering. In: AAAI. vol. 33, pp. 9127–9134 (2019)
  * [60] Zhang, C., Gupta, A., Zisserman, A.: Temporal query networks for fine-grained video understanding. In: CVPR. pp. 4486–4496 (2021)
  * [61] Zhang, H., Li, X., Bing, L.: Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858 (2023)
  * [62] Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X.V., et al.: Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022)
  * [63] Zhao, Y., Luo, C., Tang, C., Chen, D., Codella, N., Zha, Z.J.: Streaming video model. In: CVPR. pp. 14602–14612 (2023)
  * [64] Zhou, B., Andonian, A., Oliva, A., Torralba, A.: Temporal relational reasoning in videos. In: ECCV. pp. 803–818 (2018)
  * [65] Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023)


# Enhanced quantum metric due to vacancies in graphene

Quentin Marsal  Department of Physics and Astronomy, Uppsala University, Box
516, 751 20 Uppsala, Sweden  Annica M. Black-Schaffer  Department of Physics
and Astronomy, Uppsala University, Box 516, 751 20 Uppsala, Sweden

(August 28, 2024)

###### Abstract

Random vacancies in a graphene monolayer induce defect states are known to
from a narrow impurity band centered around zero energy at half-filling. We
use a space-resolved formulation of the quantum metric and establish a strong
enhancement of the electronic correlations in this impurity band. The
enhancement is primarily due to strong correlations between pairs of vacancies
situated on different sublattices at anomalously large spatial distances. We
trace the strong enhancement to both the multifractal vacancy wave functions,
which ties the system exactly at the Anderson insulator transition for all
defect concentrations, and preserving the chiral symmetry.

The two-dimensionality and linear Dirac dispersion of graphene generate
remarkable physical properties Novoselov  _et al._ (2004, 2005); Castro Neto
_et al._ (2009); Wehling  _et al._ (2014). However, the vanishing density of
states at half-filling, at the Dirac point, causes a natural insensitivity to
electron interactions and its generated many-body instabilities, otherwise key
concepts in condensed matter. But, it makes graphene sensitive to the presence
of structural defects Castro Neto  _et al._ (2009); Banhart  _et al._ (2011);
Gao  _et al._ (2018). Defects often create localized states that affects the
conductivity, including in graphene Stauber  _et al._ (2007); Wehling  _et
al._ (2010), and can even turn a metal into an insulator Evers and Mirlin
(2008), while defect states that are not fully localized have been shown to
favor some long-range correlations Feigel’man  _et al._ (2007); Burmistrov
_et al._ (2012, 2021).

In this work, we consider the archetype defect in graphene: vacancies Ugeda
_et al._ (2010, 2011). Vacancies, as well as hydrogenation Banhart  _et al._
(2011); Gao  _et al._ (2018) which induce the same type of lattice defects,
are both common and controllable. Graphene vacancies create critical defect
states pinned at the zero-energy Dirac point by the chiral symmetry Evers and
Mirlin (2008); Gade (1993); Altland and Simons (1999); Fukui (2000); Ostrovsky
_et al._ (2006, 2007); Pereira  _et al._ (2006); Ramires and Lado (2019).
However, the disorder-origin of the resulting zero-energy flat band raises a
strong initial hesitation about its ability to generate physics beyond the
localized disorder regime, including many-body instabilities or transport
properties.

We use a space-resolved version of the Fubini-Study quantum metric Marzari and
Vanderbilt (1997) and show that vacancies in graphene introduce strong long-
range correlations. In flat bands in crystalline systems, such a finite
quantum metric has recently been shown to generate superconducting phase
coherence Peotta and Törmä (2015); Peotta  _et al._ ; Julku  _et al._ (2016);
Kopnin  _et al._ (2011), as well as being needed to assess the existence of
fractional excitations Varjas  _et al._ (2022). This includes the plethora of
phenomena recently discovered in magic-angle twisted or strongly biased bi-
and trilayer graphene due to its narrow crystalline flat bands Cao  _et al._
(2018); Hao  _et al._ (2021); Park  _et al._ (2021); Zhou  _et al._ (2021,
2022); Kim  _et al._ (2022); Andrei and MacDonald (2020). A finite quantum
metric has also recently been shown to give a nonlinear anomalous Hall effect
Gao  _et al._ (2023); Wang  _et al._ (2023). Our results establish that these
discovered phenomena in crystalline systems can also be accessible by simply
using localized disorder.

Space-resolved quantum metric.— In crystalline systems, due to translational
invariance, Bloch’s states $u_{k}^{\alpha}$ form an eigenbasis for the
Hamiltonian, with $\alpha$ labeling bands and $k$ the momentum of the quantum
state. For a given filling fraction, the quantum metric then reads Peotta and
Törmä (2015); Marzari and Vanderbilt (1997):

$\Omega_{\mu\nu}=\int_{BZ}\mathrm{d}\mathbf{k}\sum_{\alpha\
\mathrm{filled}}\left(\partial_{k_{\mu}}u_{k}^{\alpha}\right)^{\dagger}(\mathbb{I}-u_{k}^{\alpha}u_{k}^{\alpha\dagger})\left(\partial_{k_{\nu}}u_{k}^{\alpha}\right),$
(1)

with the indices $\mu,\nu$ enumerating the spatial dimensions. The quantum
metric measures the distance in Hilbert space between states corresponding to
neighboring momenta $k$ and $k+\mathrm{d}k$. With the integration over the
Brillouin zone, the quantum metric is the total volume spanned by the filled
band states in the Hilbert space.

With disorder present, a real space formulation of the quantum metric is
instead required. The origin of such can be traced back to work defining
maximally localized Wannier functions Marzari and Vanderbilt (1997); Resta and
Sorella (1999); Peotta  _et al._ ; Tam and Peotta . From there the quantum
metric can be identified as the reverse Fourier transform of Eq. (1) Marzari
and Vanderbilt (1997):

$\Omega_{\mu\nu}=\mathrm{Tr}\left[P\mathbf{r}_{\mu}(1-P)\mathbf{r}_{\nu}P\right],$
(2)

with multiplications by the position operator $\mathbf{r}$ replacing momenta
derivatives and $P$ being the projector on the occupied states. Due to the
trace, $\Omega_{\mu\nu}$ can be computed over any basis of the Hilbert space.
In crystalline systems, the usual basis is the Bloch’s states, as in Eq. (1).
In non-crystalline systems we find it useful to compute the trace in the site
(atomic) basis, $|i\rangle$, to be able to define a space-resolved quantum
metric marker: $\langle i|P\mathbf{r}_{\mu}(1-P)\mathbf{r}_{\nu}P|i\rangle$.
Together with the recently developed local Chern marker Bianco and Resta
(2011); Marsal  _et al._ (2020), this provide a fully space-resolved notion of
the quantum geometric tensor Resta (2002).

A physical interpretation of Eq. (2) is that it characterizes the spreading of
the occupied states, as $\Omega=\Omega_{xx}+\Omega_{yy}$ can be rewritten as
the expectation value
$\left<\mathbf{r}^{2}\right>-\left<\mathbf{r}\right>^{2}$ Marzari and
Vanderbilt (1997); Resta and Sorella (1999). In particular, focusing on a
single band, the functional $\Omega$ is measuring the quadratic spreading of
the maximally localized Wannier orbitals that can be built for that band. As a
consequence, if the Wannier functions can be fully localized around single
atomic sites, their spreading is practically zero and the quantum metric
vanishes, defining a trivial atomic insulator. In contrast, in topologically
non-trivial bands, the Wannier basis cannot be fully localized Bradlyn  _et
al._ (2017), and the quantum metric is bounded by below by the topological
invariant, e.g. in two-dimensional Chern insulators
$\Omega\geq\frac{A}{2\pi}|C|$, with $C$ the Chern number and A the sample
area. Peotta and Törmä (2015).

Graphene vacancies.—With a space-resolved quantum metric now at hand we apply
it to show how systems with localized disorder can host a non-trivial quantum
metric. In particular, we focus on a random distribution of vacancies in a
graphene monolayer. We model graphene using the usual tight-binding
Hamiltonian on honeycomb lattice, encoding the conducting $p_{z}$ electron on
each site interacting through nearest-neighbor hopping and with unit cell
length $a$ Castro Neto  _et al._ (2009). We then introduce vacancies by
removing a number $N_{vac}$ of sites $(v_{i})$ from the lattice, setting all
hopping terms to the vacancy site to zero, resulting in the Hamiltonian

$\mathcal{H}=\sum_{\left<i,j\right>}tc_{i}^{\dagger}c_{j}-\sum_{\left<v_{i},j\right>}t(c_{v_{i}}^{\dagger}c_{j}+H.c.).$
(3)

Finally, we remove the trivially isolated orbitals $c_{v_{i}}$ from the
Hilbert space. We typically consider large systems with open boundary
conditions to avoid periodic images, using up to $N=5400$ sites and check our
key results for spurious finite size effects. As a vacancy defect only acts on
bonds (by removing them), it preserves the chiral symmetry of graphene and is
thus a type of chiral disorder, which has been studied both generically and in
the honeycomb lattice Evers and Mirlin (2008); Gade (1993); Altland and Simons
(1999); Fukui (2000); Ostrovsky  _et al._ (2006, 2007). Each vacancy removes
one state from the spectrum, thereby leaving one state that has to be its own
chiral partner, thus pinned to zero energy. This defect state is localized on
the opposite sublattice to that of the vacancy (see Fig. 1a) and, often
missed, it is multifractal, meaning its wave function amplitude decreases as a
power-law with distance from the vacancy, in contrast to both fully localized
defect states and extended Bloch states. While carbon vacancies may also
induce lattice reconstruction Meyer  _et al._ (2008), hydrogenation also
effectively remove one $p_{z}$ electron and occur primarily without lattice
reconstruction and is experimentally controllable Banhart  _et al._ (2011);
Gao  _et al._ (2018). Technically, adatoms create an onsite perturbation,
breaking chiral symmetry, but hydrogen absorbs strongly and still closely
resembles a vacancy Casolo  _et al._ (2009); González-Herrero  _et al._ (2016)
and can thus be modeled by Eq. (3).

Vacancy pair.— We first illustrate the behavior of a single pair of vacancies.
The situation is particularly intriguing when the two vacancies belong to
different sublattices. The two defect states then hybridize, forming bonding
and anti-bonding states that are chiral symmetry partners, see also
Supplementary Material (SM) SM . We illustrate this in Fig. 1(b), using
$\psi_{i}(\mathbf{r})=\psi(\mathbf{r}-\mathbf{r}_{i})$ for the wave function
of a vacancy at position $\mathbf{r}_{i}$ and $|\psi_{\pm}\rangle$ for the
bonding/anti-bonding states. The hybridized vacancy states
$|\psi_{\pm}\rangle$ are delocalized over the two vacancies and we numerically
extract their energy splitting
$\delta\propto|\psi_{1}(\mathbf{r}_{2})|^{2}+|\psi_{2}(\mathbf{r}_{1})|^{2}$.
Being delocalized, $|\psi_{\pm}\rangle$ must contribute to the quantum metric
at half filling. Indeed, with the bonding state $|\psi_{+}\rangle$ occupied
and the anti-bonding $|\psi_{-}\rangle$ not, their contribution to quantum
metric is (for derivation see SM SM )

$\Omega_{pair}=\sum_{\mu}\left|\langle\psi_{+}|\mathbf{r}_{\mu}|\psi_{-}\rangle\right|^{2}=\frac{1}{4}\sum_{\mu}(\mathbf{r}_{1\mu}-\mathbf{r}_{2\mu})^{2},$
(4)

as long as the vacancy states are fully rotationally symmetric, i.e. ignoring
lattice effects. This quadratic defect contribution adds to the intrinsic
quantum metric from the bulk states at half-filling, which is independent the
vacancy positions. We illustrate this in Fig. 1(c) showing the total quantum
metric density as a function of distance separating a pair of vacancies. The
spread in $\Omega$ is due to the lattice inducing anisotropy into Eq. (4) and
not due to finite size effects (see SM SM ).

Figure 1: (a) Local density of states at zero energy for a single vacancy. The
defect wave function is localized on the opposite sublattice to that of the
vacancy. (b) Two vacancies $|\psi_{1,2}\rangle$ on different sublattices
hybridize, forming bonding $|\psi_{+}\rangle$ and anti-bonding
$|\psi_{-}\rangle$ states delocalized over the two vacancies. Quantum metric
(per unit cell) as a function of vacancy separation, for random vacancies
pairs on different (c) and same (d) sublattice. Here $N=1174$ for a hexagonal-
shaped lattice with armchair edges, with one vacancy at the center and the
other successively occupying all the other sites.

By contrast, no hybridization occurs when the two vacancies are on the same
sublattice, since the chiral symmetry forces them to remain at zero energy.
Any superposition of these two original vacancy states thus remains a pair of
eigenstates, each localized around its own vacancy, and we expect no
significant contribution to the quantum metric. We verify this in Fig. 1(d),
which displays only a weak enhancement of the quantum metric and, importantly,
no distance dependence. In fact, the quantum metric technically becomes ill-
defined for same sublattice vacancies, as it is impossible to separate between
occupied and unoccupied zero-energy states. Due to finite size effects we can
however still often find a numerical energy separation and are thus able to
produce Fig. 1(d). We attribute the occasional larger quantum metric values to
artifacts related to this numerical issue, which occasionally generates
eigenstates that are superpositions of both vacancy states, giving a spurious
contribution to the quantum metric.

Vacancy band.— Having understood vacancy pairs, we tackle a finite
distribution of vacancies. The spectrum of this disordered system contains a
vacancy band, centered at energy $E=0$, due to the retained chiral symmetry.
If the vacancies were infinitely far from one another, chiral symmetry would
pin the vacancy band exactly at zero energy, but at any finite concentration,
vacancies in different sublattices hybridize in (at least) a pair-wise fashion
according to the mechanism above, broadening the impurity band.

Since eigenstates may now involve more than two vacancies, it is a priori
impossible to isolate the contribution of single vacancy pairs to the quantum
metric. Still, we can proceed because the quantum metric involves paired
crossed correlations between all occupied states and all unoccupied states,
see SM SM

$\Omega=\sum_{\mu}\sum_{\alpha\ \mathrm{filled}}\sum_{\beta\
\mathrm{empty}}\mathrm{Tr}\left[P_{\alpha}\mathbf{r}_{\mu}P_{\beta}\mathbf{r}_{\mu}P_{\alpha}\right],$
(5)

where
$P_{\alpha,\beta}=|\psi_{\alpha,\beta}\rangle\langle\psi_{\alpha,\beta}|$.
Every term of this sum is positive, no matter if the states $\alpha$ and
$\beta$ are chiral symmetry partners. To isolate the contribution of chiral
symmetric states, we denote for any occupied state $\alpha$,
$\overline{\alpha}$ the (unoccupied) chiral symmetric state, i.e. the
equivalent of a vacancy pair, and use the decomposition

$\Omega=\sum_{\alpha\ \mathrm{filled}}\Omega_{\alpha}+\delta\Omega_{\alpha},$
(6)

with
$\Omega_{\alpha}=\mathrm{Tr}[P_{\alpha}\mathbf{r}_{\mu}P_{\overline{\alpha}}\mathbf{r}_{\mu}]$,
and
$\delta\Omega_{\alpha}=\mathrm{Tr}[P_{\alpha}\mathbf{r}_{\mu}(Q-P_{\overline{\alpha}})\mathbf{r}_{\mu}]$,
with $Q=1-P$. We illustrate this decomposition in Fig. 2(a), where blue
crosses illustrate a chiral symmetry pair of states. Fig. 2(b) shows
$\Omega_{\alpha}$ and $\delta\Omega_{\alpha}$ as a function of the energy
$E_{\alpha}$ of the eigenstate for a random distribution of vacancies.
Clearly, the strongest contribution to the quantum metric comes from the
chiral symmetry partners. In addition, we find that such partners contribute
most if their energy splitting is small, corresponding to large spatial
separations, in agreement with vacancy pairs in Fig. 1(c). This demonstrates
that for a finite distribution of vacancies, there is still strong pairing
between vacancies on different sublattices, with an associated large
contribution to the quantum metric, especially for long-distance pairs. We
also note that bulk contributions, characterized by $|E_{\alpha}|$ larger than
the typical impurity bandwidth $\delta$ (black dotted line in Fig. 2(b)), are
small compared to the vacancy contributions. Moreover, if one sublattice host
more vacancies than the other, they result in unpaired vacancy states at
$E=0$, which, due to finite size effects, are at energies below $10^{-10}t$
and do not contribute. If instead computing the quantum metric for finite
Fermi energy $E_{F}$, we sum contributions from right to left, until
$|E_{\alpha}|=|E_{F}|$. Then, pairs of chiral symmetric states can be both
either filled or empty, which always decreases the quantum metric.

Figure 2: (a) Decomposition of the contribution to the quantum metric at half-
filling from each occupied state into two parts: $\Omega_{\alpha}$ accounting
for the correlation between the chiral symmetric states $\alpha$ and
$\overline{\alpha}$ and $\delta\Omega_{\alpha}$ accounting for correlations
with all remaining unoccupied states. (b) Contributions $\Omega_{\alpha}$ and
$\delta\Omega_{\alpha}$ as a function of eigenstate energy $E_{\alpha}$,
obtained from $20$ disorder realizations, with each configuration
$N_{vac}=120$ in a $N=2400$ lattice. Black dotted line separates the $N_{vac}$
closest states to zero energy, considered as vacancy states, from bulk states
and defines $\delta\approx 0.34t$. Black dashed line indicates the fit
described in text and the fitting range.

The results in Fig. 2 could in hindsight be anticipated. First, for any
states, both $\Omega_{\alpha}$ and $\delta\Omega_{\alpha}$ bear a positive
contribution to the quantum metric, such that there is never internal
cancellations. Second, states that are not related by chiral symmetry are
generally expected to be localized around different sets of vacancies, while
we know that different subblatice vacancy pairs instead delocalize and
generate large quantum metric contributions. Thus, quantum metric
contributions of the form
$P_{\alpha}\mathbf{r}_{\mu}P_{\beta}\mathbf{r}_{\mu}$ where
$\beta\neq\overline{\alpha}$ are expected to remain smaller than contributions
from chiral partners,
$P_{\alpha}\mathbf{r}_{\mu}P_{\overline{\alpha}}\mathbf{r}_{\mu}$. As a
consequence, we find $\Omega\approx\sum_{\alpha\
\mathrm{filled}}\Omega_{\alpha}$.

We can further understand our results by noting that chiral disorder (disorder
preserving chiral symmetry) is known to create multifractal states, meaning
that they are neither exponentially localized nor fully delocalized Gade
(1993); Altland and Simons (1999); Fukui (2000); Ostrovsky  _et al._ (2006,
2007); Häfner  _et al._ (2014). This implies a close relationship with the
critical regime of the Anderson metal-insulator localization transition Evers
and Mirlin (2008). Indeed, vacancies in graphene, irrespective of the disorder
concentration, can be expected to behave as a multifractal insulator Cuevas
and Kravtsov (2007), with enhanced correlations between states with small
energy separation. This is consistent with the enhancement of
$\Omega_{\alpha}$ at small energies in Fig. 2(b). To deepen the analogy with
the Anderson localization transition we write, for a pair of chiral symmetric
states $\alpha$ and $\overline{\alpha}$ with energy splitting $\omega$:

$\displaystyle\Omega_{\alpha}$
$\displaystyle=\mathrm{Tr}[P_{\alpha}\mathbf{r}P_{\overline{\alpha}}\mathbf{r}]=\left|\langle\psi_{\alpha}|\mathbf{r}|\psi_{\overline{\alpha}}\rangle\right|^{2}$
(7)
$\displaystyle=\iint\mathrm{d}\mathbf{r}\mathrm{d}\mathbf{r}^{\prime}(\mathbf{r}\cdot\mathbf{r}^{\prime})\psi_{\alpha}(\mathbf{r})^{*}\psi_{\overline{\alpha}}(\mathbf{r})\psi_{\overline{\alpha}}(\mathbf{r}^{\prime})^{*}\psi_{\alpha}(\mathbf{r}^{\prime}).$
(8)

$\Omega_{\alpha}$ is now a two-point and two-state correlator whose exact
value depends on the exact position of the vacancies. By utilizing a
variational approach developed for Anderson localization Houghton  _et al._
(1980), we find that, on average, the strongest contribution to the integral
in Eq. (8) comes from pairs of vacancies and only depends on the energy
splitting $\omega$ such that
$\Omega_{\alpha}\propto\log^{d+1}\left(\frac{\delta}{\omega}\right)$, where
$d=2$ is the system dimension. This variational approach is valid for small
energy separation $\omega\ll\delta$ Houghton  _et al._ (1980); Feigel’man  _et
al._ (2010), which is the case for a vacancy band. In Fig. 2(b) we also
include a fit to the $\log^{p}$ functional form, finding $p=d+1\approx 2.6$,
which gives good agreement. For this fit we exclude vacancies with
$\Omega_{\alpha}>\frac{\sqrt{3}}{4}Na^{2}$, the area of the sample, which sit
close to the sample edge and thus result in strong fluctuations in
$\Omega_{\alpha}$ due to finite size effects. These arguments establish that
the strong quantum metric enhancement we observe is a consequence of the
multifractality of hybridized vacancy states. Notably, the quantum metric
enhancement is as strongest for vacancies far apart, a seemingly non-intuitive
consequence of the multifractalness.

Concentration dependence.— Above we established that vacancy pairs strongly
enhance the quantum metric. A vacancy preferentially hybridize with its
closest different sublattice vacancy neighbor. In a disordered system, more
complex hybridization schemes involving more than two vacancies are in
principle possible but we find only limited contributions from such to the
quantum metric, as $\delta\Omega_{\alpha}$ is small. On the other hand, we
find larger quantum metric contributions from vacancies with larger spatial
separations. This dichotomy in the spatial dependence may result in an
interesting concentration dependence.

In Fig. 3(a) we plot the enhancement of the quantum metric (beyond the
pristine bulk value $\Omega_{0}$) at half-filling as a function of the vacancy
concentration $n^{*}=N_{vac}/N$. The quantum metric (blue points) initially
grows with defect concentration, but then saturates and eventually decreases.
To understand the small to moderate concentration regime, we imagine a
situation where all vacancies are separated by the average distance
$d^{*}=1/\sqrt{n^{*}}$. An increase in concentration then results in more
pairs contributing to the quantum metric, but with smaller contributions each,
with the quantum metric enhancement given by
$\Omega^{*}\propto\sum_{\mathrm{pairs\
i,j}}(\mathbf{r}_{i}-\mathbf{r}_{j})^{2}=\frac{Sn^{*}}{2}\times
d^{*2}=\frac{S}{2}$, with $S$ the surface area of the sample times a constant.
We plot $\Omega^{*}$ in Fig. 3(a) (dashed horizontal line) and note that the
quantum metric is always substantially larger than this averaged value. Hence
we conclude that “rare” long-distance pairs must be responsible for the
quantum metric increase with concentration. At a concentration $\sim 5\%$, the
effect of these “rare” pairs saturates and they are instead superseded by
closer neighboring pairs, eventually leading to neighboring vacancies, which
explains the diminishing quantum metric at large concentrations.

Figure 3: (a) Disorder induced quantum metric (per unit cell) as a function of
defect concentration (beyond the pristine bulk value $\Omega_{0}$) for random
distributions of vacancies (blue), achiral vacancies (orange), and localized
defect states (red). Spread marks a standard deviation, extracted from
averaging over 40 disorder configurations. Black dashed line shows
$\Omega-\Omega_{0}=S/2=\sqrt{3}Na^{2}/8$, the expected enhancement for
periodically spaced defects. (b) Squared amplitude of the electronic wave
function as a function of the distance to the defect position for the three
different types defects: the chiral vacancy as described earlier, the achiral
vacancy and localized defect state obtained by adding an onsite potential
$V=4t$ to a single site.

Other defects.— Finally, in order to further highlight the importance of
chiral disorder and its related multifractal wave functions, we compare the
quantum metric contributions from vacancies with related defects, obtained by
instead adding an onsite potential $V$ to the defect site. This creates two
defect states with different characteristics. One state has energy around
$E=V$ and is exponentially localized to the defect site. We call this the
localized defect state. The other state remains close to zero energy, at
$E\propto\frac{t}{1+V/t}$. It shares the same localization properties as the
vacancy, except for the fact that it breaks chiral symmetry and then also has
a finite weight on both sublattices. We therefore call this an achiral
vacancy. In the limit $V\rightarrow\infty$ the achiral vacancy becomes
equivalent to the chiral vacancy state at $E=0$, as expected for
hydrogenation, while the other, the localized defect state, is effectively
removed from the spectrum.

In Fig. 3(b) we plot the localization profile as the square amplitude of the
wave function for all three defect states: chiral vacancy, achiral vacancy,
and localized defect state obtained for onsite defect potential $V=4t$. We
find a notable difference between the chiral and achiral vacancies when it
comes to the sublattice occupation, directly tied to the chiral symmetry.
Still, both are displaying the same long-range decay, tied to the multifractal
nature of the chiral vacancy and clearly inherited by the achiral vacancy.
This is in contrast to the localized defect which displays exponential
localization. Finally we compare also the quantum metric for all three defects
in Fig. 3(a). We find no enhancement of the quantum metric for localized
defects. This is due to their exponentially decaying tail effectively
introducing a cut-off distance, thus preventing hybridization over long
distance and giving a vanishing quantum metric. Achiral vacancies have a more
interesting behavior. Despite their multifractal localization profile in Fig.
3(b), we find a notably smaller quantum metric enhancement than for chiral
vacancies. Thus, it is actually not only the multifractal wave nature that
drives the enhanced quantum metric for vacancies, but preserving chiral
symmetry is also crucial for an impurity band to generate a strongly enhanced
quantum metric. This is consistent with the fact that hybridization does not
occur when vacancies occupy the same sublattice, as established in Fig. 1(d).
The chiral symmetry also pins the system precisely at the critical point of
the metal-insulator Anderson localization transition for all concentrations,
with the quantum metric the key property to describe the special properties.
Breaking the chiral symmetry, the impurity band starts to localize and the
system eventually becomes a trivial insulator with a quantum metric not
influenced by the defects.

In summary, by introducing a space-resolved formulation of the quantum metric
we show that random vacancies in a graphene monolayer generates strong long-
range correlations at half filling, primarily from pairs of chiral symmetric
vacancy states situated on different sublattices and with a small energy
splitting, or equivalently, large spatial separation. Such “rare” long-
distance pairs provide an anomalous contribution, particularly at finite
concentrations. By utilizing similarities with the Anderson localization
transition, we show that the multifractal vacancy wave functions is a key
component of the quantum metric enhancement, but we also find that chiral
symmetry is an additional important ingredient. A large quantum metric
establishes the highly nontrivial nature of the flat vacancy band in graphene.
Possible extensions include studying the space-resolved quantum metric for
other non-crystalline systems hosting critical states, including
quasicrystals, fractals Iliasov  _et al._ , or amorphous systems with
vacancies. The spatial repartition of the quantum metric may also provide
supplementary information about the electronic properties of the system. We
also note that some similar phenomenology was very recently also demonstrated
in a Kitaev spin liquid on a honeycomb lattice with vacancies Kao  _et al._ .

Acknowledgements.— We thank T. Löthman, R. Arouca, L. Baldo, and A.
Bhattacharya for interesting and fruitful discussions related to this work. We
acknowledge funding from the Knut and Alice Wallenberg Foundation through the
Wallenberg Academy Fellows program and the European Research Council (ERC)
under the European Union’s Horizon 2020 research and innovation programme
(ERC-2022-CoG, Grant agreement No. 101087096). Our calculations were performed
using the Python package kwant Groth  _et al._ (2014) and our plots using
matplotlib Hunter (2007). The code used for the numerical calculations and the
data shown in the manuscript is available at Ref. Marsal and Black-Schaffer
(2023).

## References

  * Novoselov  _et al._ (2004) K. S. Novoselov, A. K. Geim, S. V. Morozov, D. Jiang, Y. Zhang, S. V. Dubonos, I. V. Grigorieva, and A. A. Firsov, Science 306, 666 (2004).
  * Novoselov  _et al._ (2005) K. S. Novoselov, A. K. Geim, S. V. Morozov, D. Jiang, M. I. Katsnelson, I. V. Grigorieva, S. V. Dubonos, and A. A. Firsov, Nature 438, 197 (2005).
  * Castro Neto  _et al._ (2009) A. H. Castro Neto, F. Guinea, N. M. R. Peres, K. S. Novoselov, and A. K. Geim, Rev. Mod. Phys. 81, 109 (2009).
  * Wehling  _et al._ (2014) T. O. Wehling, A. M. Black-Schaffer, and A. V. Balatsky, Adv. Phys. 63, 1 (2014).
  * Banhart  _et al._ (2011) F. Banhart, J. Kotakoski, and A. V. Krasheninnikov, ACS Nano 5, 26 (2011).
  * Gao  _et al._ (2018) Y. Gao, L. Lihua, B. L. Wing, and C. N. Man, Science and Technology of Advanced Materials 19, 613 (2018), pMID: 30181789.
  * Stauber  _et al._ (2007) T. Stauber, N. M. R. Peres, and F. Guinea, Phys. Rev. B 76, 205423 (2007).
  * Wehling  _et al._ (2010) T. O. Wehling, S. Yuan, A. I. Lichtenstein, A. K. Geim, and M. I. Katsnelson, Phys. Rev. Lett. 105, 056802 (2010).
  * Evers and Mirlin (2008) F. Evers and A. D. Mirlin, Rev. Mod. Phys. 80, 1355 (2008).
  * Feigel’man  _et al._ (2007) M. V. Feigel’man, L. B. Ioffe, V. E. Kravtsov, and E. A. Yuzbashyan, Phys. Rev. Lett. 98, 027001 (2007).
  * Burmistrov  _et al._ (2012) I. S. Burmistrov, I. V. Gornyi, and A. D. Mirlin, Phys. Rev. Lett. 108, 017002 (2012).
  * Burmistrov  _et al._ (2021) I. Burmistrov, I. Gornyi, and A. Mirlin, Ann. Phys. 435, 168499 (2021).
  * Ugeda  _et al._ (2011) M. M. Ugeda, D. Fernández-Torre, I. Brihuega, P. Pou, A. J. Martínez-Galera, R. Pérez, and J. M. Gómez-Rodríguez, Phys. Rev. Lett. 107, 116803 (2011).
  * Ugeda  _et al._ (2010) M. M. Ugeda, I. Brihuega, F. Guinea, and J. M. Gómez-Rodríguez, Phys. Rev. Lett. 104, 096804 (2010).
  * Gade (1993) R. Gade, Nucl. Phys. B 398, 499 (1993).
  * Altland and Simons (1999) A. Altland and B. Simons, Nucl. Phys. B 562, 445 (1999).
  * Fukui (2000) T. Fukui, Nucl. Phys. B 575, 673 (2000).
  * Ostrovsky  _et al._ (2006) P. M. Ostrovsky, I. V. Gornyi, and A. D. Mirlin, Phys. Rev. B 74, 235443 (2006).
  * Ostrovsky  _et al._ (2007) P. M. Ostrovsky, I. V. Gornyi, and A. D. Mirlin, Phys. Rev. Lett. 98, 256801 (2007).
  * Pereira  _et al._ (2006) V. M. Pereira, F. Guinea, J. M. B. Lopes dos Santos, N. M. R. Peres, and A. H. Castro Neto, Phys. Rev. Lett. 96, 036801 (2006).
  * Ramires and Lado (2019) A. Ramires and J. L. Lado, Phys. Rev. B 99, 245118 (2019).
  * Marzari and Vanderbilt (1997) N. Marzari and D. Vanderbilt, Phys. Rev. B 56, 12847 (1997).
  * Peotta and Törmä (2015) S. Peotta and P. Törmä, Nat. Commun. 6, 8944 (2015).
  * (24) S. Peotta, K.-E. Huhtinen, and P. Törmä, arXiv:2308.08248 .
  * Julku  _et al._ (2016) A. Julku, S. Peotta, T. I. Vanhala, D.-H. Kim, and P. Törmä, Phys. Rev. Lett. 117, 045303 (2016).
  * Kopnin  _et al._ (2011) N. B. Kopnin, T. T. Heikkilä, and G. E. Volovik, Phys. Rev. B 83, 220503 (2011).
  * Gao  _et al._ (2023) A. Gao, Y.-F. Liu, J.-X. Qiu, B. Ghosh, T. V. Trevisan, Y. Onishi, C. Hu, T. Qian, H.-J. Tien, S.-W. Chen, M. Huang, D. Bérubé, H. Li, C. Tzschaschel, T. Dinh, Z. Sun, S.-C. Ho, S.-W. Lien, B. Singh, K. Watanabe, T. Taniguchi, D. C. Bell, H. Lin, T.-R. Chang, C. R. Du, A. Bansil, L. Fu, N. Ni, P. P. Orth, Q. Ma, and S.-Y. Xu, Science 381, 181 (2023).
  * Wang  _et al._ (2023) N. Wang, D. Kaplan, Z. Zhang, T. Holder, N. Cao, A. Wang, X. Zhou, F. Zhou, Z. Jiang, C. Zhang, S. Ru, H. Cai, K. Watanabe, T. Tanigushi, B. Yan, and W. Gao, Nature 621, 487 (2023).
  * Varjas  _et al._ (2022) D. Varjas, A. Abouelkomsan, K. Yang, and E. J. Bergholtz, SciPost Phys. 12, 118 (2022).
  * Cao  _et al._ (2018) Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Nature 556, 43 (2018).
  * Hao  _et al._ (2021) Z. Hao, A. M. Zimmerman, P. Ledwith, E. Khalaf, D. H. Najafabadi, K. Watanabe, T. Taniguchi, A. Vishwanath, and P. Kim, Science 371, 1133 (2021).
  * Park  _et al._ (2021) J. M. Park, Y. Cao, K. Watanabe, T. Taniguchi, and P. Jarillo-Herrero, Nature 590, 249 (2021).
  * Zhou  _et al._ (2021) H. Zhou, T. Xie, T. Taniguchi, K. Watanabe, and A. F. Young, Nature 598, 434 (2021).
  * Zhou  _et al._ (2022) H. Zhou, L. Holleis, Y. Saito, L. Cohen, W. Huynh, C. L. Patterson, F. Yang, T. Taniguchi, K. Watanabe, and A. F. Young, Science 375, 774 (2022).
  * Kim  _et al._ (2022) H. Kim, Y. Choi, C. Lewandowski, A. Thomson, Y. Zhang, R. Polski, K. Watanabe, T. Taniguchi, J. Alicea, and S. Nadj-Perge, Nature 606, 494 (2022).
  * Andrei and MacDonald (2020) E. Y. Andrei and A. H. MacDonald, Nat. Mater. 19, 1265 (2020).
  * Resta and Sorella (1999) R. Resta and S. Sorella, Phys. Rev. Lett. 82, 370 (1999).
  * (38) M. Tam and S. Peotta, arXiv:2308.10780 .
  * Bianco and Resta (2011) R. Bianco and R. Resta, Phys. Rev. B 84, 241106 (2011).
  * Marsal  _et al._ (2020) Q. Marsal, D. Varjas, and A. G. Grushin, Proc. Nat. Acad. Sci. 117, 30260 (2020).
  * Resta (2002) R. Resta, Journal of Physics: Condensed Matter 14, R625 (2002).
  * Bradlyn  _et al._ (2017) B. Bradlyn, L. Elcoro, J. Cano, M. G. Vergniory, Z. Wang, C. Felser, M. I. Aroyo, and B. A. Bernevig, Nature 547, 298 (2017).
  * Häfner  _et al._ (2014) V. Häfner, J. Schindler, N. Weik, T. Mayer, S. Balakrishnan, R. Narayanan, S. Bera, and F. Evers, Phys. Rev. Lett. 113, 186802 (2014).
  * Meyer  _et al._ (2008) J. C. Meyer, C. Kisielowski, R. Erni, M. D. Rossell, M. Crommie, and A. Zettl, Nano Lett. 8, 3582 (2008).
  * Casolo  _et al._ (2009) S. Casolo, O. M. Løvvik, R. Martinazzo, and G. F. Tantardini, The Journal of Chemical Physics 130, 054704 (2009).
  * González-Herrero  _et al._ (2016) H. González-Herrero, J. M. Gómez-Rodríguez, P. Mallet, M. Moaied, J. J. Palacios, C. Salgado, M. M. Ugeda, J.-Y. Veuillen, F. Yndurain, and I. Brihuega, Science 352, 437 (2016).
  * (47) See Supplementary Material (SM) where we derive the expressions for the quantum metric for vacancy pairs and bands .
  * Cuevas and Kravtsov (2007) E. Cuevas and V. E. Kravtsov, Phys. Rev. B 76, 235119 (2007).
  * Houghton  _et al._ (1980) A. Houghton, L. Schäfer, and F. J. Wegner, Phys. Rev. B 22, 3598 (1980).
  * Feigel’man  _et al._ (2010) M. Feigel’man, L. Ioffe, V. Kravtsov, and E. Cuevas, Ann. Phys. 325, 1390 (2010), july 2010 Special Issue.
  * (51) A. A. Iliasov, M. I. Katsnelson, and A. A. Bagrov,  arXiv:2310.11497 .
  * (52) W.-H. Kao, G. B. Halász, and N. B. Perkins,  arXiv:2310.06891 .
  * Groth  _et al._ (2014) C. W. Groth, M. Wimmer, A. R. Akhmerov, and X. Waintal, New Journal of Physics 16, 063065 (2014).
  * Hunter (2007) J. D. Hunter, Computing in Science & Engineering 9, 90 (2007).
  * Marsal and Black-Schaffer (2023) Q. Marsal and A. Black-Schaffer,  (2023), 10.5281/zenodo.10284813.