#  LongEmbed: Extending Embedding Models for Long Context Retrieval

Dawei Zhu ${}^{\text{\texteta}}$ Liang Wang ${}^{\text{\textpi}}$ Nan Yang
${}^{\text{\textpi}}$ Yifan Song ${}^{\text{\texteta}}$ Wenhao Wu
${}^{\text{\texteta}}$
Furu Wei ${}^{\text{\textpi}}$ Sujian Li ${}^{\text{\texteta}}$
${}^{\text{\texteta}}$ Peking University ${}^{\text{\textpi}}$ Microsoft
Corporation Work done during Dawei’s internship at MSR Asia. Prof. Sujian Li
is the corresponding author.

###### Abstract

Embedding models play a pivot role in modern NLP applications such as IR and
RAG. While the context limit of LLMs has been pushed beyond 1 million tokens,
embedding models are still confined to a narrow context window not exceeding
8k tokens, refrained from application scenarios requiring long inputs such as
legal contracts. This paper explores context window extension of existing
embedding models, pushing the limit to 32k without requiring additional
training. First, we examine the performance of current embedding models for
long context retrieval on our newly constructed LongEmbed benchmark. LongEmbed
comprises two synthetic tasks and four carefully chosen real-world tasks,
featuring documents of varying length and dispersed target information.
Benchmarking results underscore huge room for improvement in these models.
Based on this, comprehensive experiments show that training-free context
window extension strategies like position interpolation can effectively extend
the context window of existing embedding models by several folds, regardless
of their original context being 512 or beyond 4k. Furthermore, for models
employing absolute position encoding (APE), we show the possibility of further
fine-tuning to harvest notable performance gains while strictly preserving
original behavior for short inputs. For models using rotary position embedding
(RoPE), significant enhancements are observed when employing RoPE-specific
methods, such as NTK and SelfExtend, indicating RoPE’s superiority over APE
for context window extension. To facilitate future research, we release
E5Base-4k and E5-RoPEBase, along with the LongEmbed benchmark.

https://github.com/dwzhu-pku/LongEmbed

(a)

(b)

(c)

Figure 1: (a) Overview of the LongEmbed benchmark. (b) Performance of current
embedding models on passkey retrieval, with evaluation length ranging from 256
to 32,768.222For simplicity, we report results from the base versions of the
included models by default.  $\blacktriangle$ / $\blacklozenge$ denotes
embedding models with 512 / $\geq$ 4k context. The greener  a cell is, the
higher retrieval accuracy this model achieves on the corresponding evaluation
length. (c) Effects of context window extension methods on E5, E5-RoPE,
E5-Mistral, measured by improvements of Avg. Scores on LongEmbed. SE / NTK is
short for SelfExtend / NTK-Aware Interpolation.

##  1 Introduction

Text embeddings are vector representations of natural language that encode its
semantic information. They play a pivotal role in various natural language
processing (NLP) tasks, including information retrieval (IR) and retrieval-
augmented generation (RAG). However, embedding models for producing these
vector representations still operates within a very narrow context window,
typically 512 input tokens (Wang et al., 2022; Xiao et al., 2023; Ni et al.,
2022), This narrow context window has greatly hindered their application in
scenarios requiring long inputs, such as long wikipedias and meeting scripts
(Saad-Falcon et al., 2024).

Previous efforts that train a long context embedding model from scratch suffer
significant computational overhead, due to the combined demand for large batch
sizes and long sequences. For example, Chen et al. (2024) utilized 96 A100
GPUs to train BGE-M3 which supports 8k context. Meanwhile, there have been
many successes in extending context window of existing LLMs in a plug-and-play
way or via efficient fine-tuning, pushing their context from 4k to 128k (Xiong
et al., 2023) and even 2 million tokens (Ding et al., 2024). Motivated by
this, instead of training long context embedding models from scratch, this
paper explores context window extension of existing embedding models.

First, we examine the capability of existing embedding models in processing
long context. Retrieval is selected as the proxy task, as it closely mirrors
real-world application scenarios. While there have been some retrieval
benchmarks such as BEIR (Thakur et al., 2021) and LoCo (Saad-Falcon et al.,
2024), we identify two major limitations with these existing benchmarks: 1)
limited document length, 2) biased distribution of target information. To
overcome this, we introduce the LongEmbed benchmark that integrates two
synthetic tasks to enable flexible control over document length, and four real
tasks featuring dispersed target information. Results on LongEmbed indicates
huge room for improvement in current embedding models.

Based on this, we explore plug-and-play strategies to extend embedding models,
including parallel context windows, reorganizing position ids, and position
interpolation. Comprehensive experiments show that these strategies can
effectively extend the context window of existing embedding models by several
folds, regardless of their original context being 512 or beyond 4k.
Furthermore, for models employing absolute position encoding (APE), we show
the possibility of harvesting further improvements via fine-tuning while
strictly preserving original behavior within the short context. In this way,
we have extended E5Base (Wang et al., 2022) from 512 to 4k (See Figure 1(c)).

For models utilizing RoPE (Su et al., 2021), substantial enhancements on
LongEmbed are observed when employing methods that fully leverage RoPE’s
advantages, such as NTK (Peng & Quesnelle, 2023) and SelfExtend (Jin et al.,
2024). As illustrated in Figure 1(b) and 1(c), leveraging NTK extends the
context window of E5-Mistral to 32k, achieving close-to-perfect accuracy on
passkey retrieval and state-of-the-art performance on LongEmbed. Further, for
fair comparison of APE / RoPE-based embedding models, we pre-train E5-RoPE
following the training procedure and data of E5. Thorough comparison of E5 and
E5-RoPE reveals the superiority of RoPE-based embedding models in context
window extension.

To facilitate future research in long context embedding models, we release
E5Base-4k, E5-RoPEBase, and the LongEmbed benchmarks. E5Base-4k is further
fine-tuned on E5Base to support 4k context, while strictly preserving original
behavior for inputs not exceeding 512 tokens. E5-RoPEBase follows the same
training procedure as E5Base, except for the substitution of APE with RoPE. It
is released to facilitate comparison between APE & RoPE-Based embedding
models. Furthermore, we have integrated LongEmbed into MTEB (Muennighoff et
al., 2023) to make evaluation more convenient.

##  2 Related Work

Text Embedding Models. Text embeddings are continuous, low-dimensional vector
representations of text that encode semantic information, laying the
foundation of numerous NLP applications. Early attempts on text embeddings
includes latent semantic indexing (Deerwester et al., 1990) and weighted
average of word embeddings (Mikolov et al., 2013). Modern embedding models
(Wang et al., 2022; Xiao et al., 2023; Neelakantan et al., 2022) exploit
supervision from labeled query-document pairs, adopting a multi-stage training
paradigm, where they are first pre-trained on large-scale weakly-supervised
text pairs using contrastive loss, then fine-tuned on small scale but high-
quality datasets. More recently, Muennighoff et al. (2024) explores the
combination of generative and embedding tasks on LLMs, introducing GritLM that
harvests improvements in both aspects.

Existing efforts in developing long-context embedding models typically involve
first obtaining a long-context backbone model, either by pre-training with
long inputs from scratch (Günther et al., 2023; Nussbaum et al., 2024; Chen et
al., 2024) or using existing ones (Wang et al., 2023b), followed by training
the backbone model to produce embeddings. Instead, this paper endows existing
embedding models with the ability to handle long context through context
window extension.

Context Window Extension for Large Language Models. Due to the high cost of
pre-training an LLM from scratch, there have been many efforts towards
extending the context window of existing LLMs in a plug-and-play manner. We
categorize these efforts as follows: 1) Divide-and-conquer, which involves
segmenting long inputs into short chunks, processing each chunk with the
model, and aggregating the results, as demonstrated by PCW (Ratner et al.,
2023); 2) Position reorganization, which reorganizes position ids to boost
length extrapolation, as exemplified by SelfExtend (Jin et al., 2024), DCA (An
et al., 2024), and others; 3) Position interpolation, which introduces new
position embeddings by interpolating existing ones, includes PI (Chen et al.,
2023), NTK (Peng & Quesnelle, 2023), YaRN (Peng et al., 2023), and Resonance
RoPE (Wang et al., 2024a). Our paper thoroughly investigates these three lines
of methods on embedding models. We also acknowledge other efforts for
extending the context window, such as prompt & KV compression (Jiang et al.,
2023; Ge et al., 2023; Zhang et al., 2024a) and memory-based transformers
(Wang et al., 2024b; Xiao et al., 2024). However, the former is not applicable
for bidirectional attention, and the latter requires complex mechanisms for
accessing encoded content, hence we do not experiment with these two
categories.

In addition to their plug-and-play usability, further fine-tuning on top of
these methods with long training samples has been proven to yield better
performance (Xiong et al., 2023; Fu et al., 2024; Zhang et al., 2024b; Yen et
al., 2024). Addressing the overhead of training on long inputs and the
scarcity of extremely long training data, a line of research investigates
simulating long inputs within short context, including Randomized Positions
(Ruoss et al., 2023), Positional Skip-wise (PoSE) training (Zhu et al., 2023).
This paper also leverage these efforts to synthesize long training samples
from the original training data, facilitating further fine-tuning on top of
plug-and-play methods.

##  3 The LongEmbed benchmark

In this section, we first identify two limitations of existing retrieval
benchmarks for evaluating long-context capabilities (Section 3.1). Then, we
introduce the retrieval tasks adopted in our LongEmbed, including both
synthetic ones (Section 3.2) and real ones (Section 3.3).

###  3.1 Examination of Existing Retrieval Benchmarks

There are mainly two desiderata for curating a benchmark for long context
retrieval. First, the candidate documents should be long enough. Second, the
target information to answer user query should be as uniformly distributed
across the document as possible. This prevents embedding models from solely
focusing on specific parts, such as the beginning (Coelho et al., 2024), to
achieve unreasonably high scores. Based on these criteria, we evaluate
existing benchmarks for text retrieval as follows:

Figure 2: Results of E5Base on 8 LoCo tasks.

BEIR Benchmark (Thakur et al., 2021) is a collection of 18 information
retrieval datasets, ranging across ad-hoc web search, question answering, fact
verification and duplicate question retrieval, etc. However, documents in this
benchmark contains fewer than 300 words on average (See Table 5 in Appendix),
making it unsuitable for measuring long context retrieval that usually
involves documents of thousands or tens of thousands of words.

LoCo Benchmark (Saad-Falcon et al., 2024) consists 12 retrieval tasks that
requires long context reasoning, spanning diverse domains such as law,
science, finance, etc. However, we show that it still suffers from biased
distribution of key information. Figure 2 presents results of E5Base on 8 LoCo
tasks that are publicly available. With only 512 context length, E5Base
achieves >85% nDCG scores on 3 out of 8 retrieval tasks. This severely biased
distribution of target information undermines its ability to reflect model
performance as context length increases.

###  3.2 Synthetic Tasks in LongEmbed

First, we tailor the passkey retrieval and needle-in-a-haystack retrieval task
designed for LLMs to measure context length of embedding models as follows:

Personalized Passkey Retrieval. Passkey retrieval (Mohtashami & Jaggi, 2023)
requires LLMs to recover a random passkey hidden within a long document
comprising garbage information. For embedding models, we adopt the
personalized passkey retrieval task proposed by Wang et al. (2023b), where
each document contains a unique person name and his/her passkey at random
position. The goal is to retrieve the document containing the given person’s
passkey from all candidates documents.

Figure 3: Example for the passkey and needle test.

Needle-in-a-haystack Retrieval. While passkey retrieval surrounds key
information with garbage sentences, needle-in-a-haystack retrieval (Kamradt,
2023) randomly inserts key information into an arbitrary position of a long
essay, making the task more challenging. To tailor this task for embedding
models, we instruct GPT-4 to generate 100 facts covering a variety of domains
including physics, history, geometry, art, etc, and 100 queries
correspondingly. The facts are treated as needles and randomly inserted into
the PaulGrahamEssay to form 100 candidate documents. Our task is to correctly
retrieve the document that contains corresponding needle given the query.

The advantage of synthetic data is that we can flexibly control context length
and distribution of target information. For both tasks, we evaluate a broad
context range of $\\{0.25,0.5,1,2,4,8,16,32\\}\times 1024$ tokens 333Since
token numbers vary w.r.t. tokenizers, we use a rough estimation that 1 token =
0.75 word, and constraint the word numbers to not exceed
$\\{0.25,0.5,1,2,4,8,16,32\\}\times 1024\times 0.75$.. For each context
length, we include 50 test samples, each comprising 1 query and 100 candidate
documents. 444The original version of personalized passkey retrieval uses
different candidate documents for each query, resulting in 50 queries and
5,000 documents to encode for each context length. To speed up evaluation, we
share the candidate documents for different queries within each context
length. In this way, we can measure the effective context size of embedding
models for up to 32k tokens. Examples for both synthetic tasks are presented
in Figure 3. For the passkey test, the <prefix / suffix> are repeats of "The
grass is green. The sky is blue. The sun is yellow. Here we go. There and back
again." For the needle test, the <prefix> and <suffix> form a long essay.

###  3.3 Real Tasks in LongEmbed

While synthetic tasks offer flexibility in manipulating context length and
distributing key information, they still differ from real-world scenarios. To
conduct a comprehensive evaluation, we have tailored following long-form QA
and summarization tasks for long context retrieval. Note that for QA and
summarization datasets, we use the questions and summaries as queries,
respectively.

NarrativeQA (Kočiský et al., 2018) is a QA dataset comprising long stories
averaging 50,474 words and corresponding questions about specific content such
as characters, events. As these details are dispersed throughout the story,
models must process the entire long context to get correct answers.

2WikiMultihopQA (Ho et al., 2020) is a multi-hop QA dataset featuring
questions with up to 5 hops, synthesized through manually designed templates
to prevent shortcut solutions. This necessitates the ability to process and
reason over long context, ensuring that answers cannot be obtained by merely
focusing on a short span within the document.

QMSum (Zhong et al., 2021) is a query-based meeting summarization dataset that
requires selecting and summarizing relevant segments of meetings in response
to queries. Due to the involvement of multiple participants and topics in the
meeting, summarization regarding specific queries naturally requires
aggregating information dispersed throughout the entire text.

SummScreenFD (Chen et al., 2022) is a screenplay summarization dataset
comprising pairs of TV series transcripts and human-written summaries. Similar
to QMSum, its plot details are scattered throughout the transcript and must be
integrated to form succinct descriptions in the summary.

Table LABEL:tab:statistics presents the overall statistics of LongEmbed.
Considering the computational complexity that increases quadratically with
input length, we intentionally restrict the number of candidate documents in
each task to to not exceed $10^{3}$. In this way, we can efficiently evaluate
the basic long context capabilities of embedding models. For further
elaboration on the source and examples for each dataset, please refer to
Appendix C.

Table 1: Overview of the LongEmbed benchmark. Average word number is rounded to the nearest integer. † For needle and passkey test, we have 8 groups of queries and candidate documents, with the documents averaging $\\{0.25,0.5,1,2,4,8,16,32\\}\times 0.75k$ words, respectively. Dataset | Domain | # Queries | # Docs | Avg. Query | Avg. Doc
---|---|---|---|---|---
Words | Words
Real Tasks
NarrativeQA | Literature, Film | 10,449 | 355 | 9 | 50,474
QMSum | Meeting | 1,527 | 197 | 71 | 10,058
2WikimQA | Wikipedia | 300 | 300 | 12 | 6,132
SummScreenFD | ScreenWriting | 336 | 336 | 102 | 5,582
Synthetic Tasks
Passkey | Synthetic | 400 | 800 | 11 | †
Needle | Synthetic | 400 | 800 | 7 | †

##  4 Methodology

###  4.1 Absolute Position Embedding (APE) & Rotary Position Embedding (RoPE)

Absolute Position Embedding (APE) stands as the predominant positional
encoding strategy for embedding models, as majority of them follows the BERT
architecture (Devlin et al., 2019). APE-based models first embed absolute
position ids into position vectors and add token embeddings to their
corresponding position vectors, before feeding them to a stack of transformer
layers.

Rotary Position Embedding (RoPE) is the most pervasive position embedding
strategy in the era of LLMs, including LLaMA (Touvron et al., 2023), Gemma
(Team et al., 2024), QWen (Bai et al., 2023a), etc. It encodes position
information of tokens with a rotation matrix that naturally incorporates
explicit relative position dependency. To elucidate, given a hidden vector
${\bm{h}}=[h_{0},h_{1},...,h_{d-1}]$ of dimension $d$, and a position index
$m$, RoPE operates as follows:

$f({\bm{h}},m)=[(h_{0}+\mathrm{i}h_{1})e^{\mathrm{i}m\theta_{0}},(h_{2}+\mathrm{i}h_{3})e^{\mathrm{i}m\theta_{1}},...,(h_{d-2}+\mathrm{i}h_{d-1})e^{\mathrm{i}m\theta_{d/2-1}}]$
(1)

where $\theta_{j}=10000^{-2j/d},j\in\\{0,1,...,d/2-1\\}$,
$\mathrm{i}=\sqrt{-1}$ is the imaginary unit. Unlike APE that is directly
applied to the input vector ${\bm{x}}$, RoPE is employed on the query and key
vectors at each layer. The attention score $a({\bm{q}},{\bm{k}})$ between a
query ${\bm{q}}$ at position $m$ and a key ${\bm{k}}$ at position $n$ is
defined as:

$\displaystyle a({\bm{q}},{\bm{k}})$ $\displaystyle=\mathrm{Re}\langle
f({\bm{q}},m),f({\bm{k}},n)\rangle=\mathrm{Re}\left[\sum_{j=0}^{d/2-1}(q_{2j}+\mathrm{i}q_{2j+1})(k_{2}j-\mathrm{i}k_{2j+1})e^{\mathrm{i}(m-n)\theta_{j}}\right]$
$\displaystyle:=g({\bm{q}},{\bm{k}},(m-n){\bm{\theta}})$ (2)

where g(·) is an abstract mapping function exclusively dependent on
${\bm{q}},{\bm{k}}$ and $(m-n){\bm{\theta}}$.

###  4.2 Context Window Extension for APE-based Models

As delineated in Section 2, training-free context extension strategies
applicable to embedding models can be classified into 3 categories: 1) Divide-
and-conquer; 2) Position reorganization; 3) Position interpolation. In this
section, we introduce methods from each of these categories to assess their
applicability to embedding models. Further fine-tuning on top of these methods
is also included. Let $L_{o}$ represent the original context length,
$\mathcal{D}=\\{x_{1},x_{2},...,x_{L_{t}}\\}$ denote a long document of target
context length $L_{t}$, and $s=\lceil L_{t}/L_{o}\rceil$ indicate the context
scaling factor. The context extension methods we investigated are described
below:

Figure 4: (Left) Arrangement of pids for extending APE-based models from 512
to 1024. (Right) Illustration of learnable () and frozen () position vectors
when further tuning on RP / PI.

Parallel Context Windows (PCW). To process a long document with a short-
context model, PCW divides the long document into multiple short chunks,
processes each chunk in parallel, and aggregates their results (Ratner et al.,
2023; Yen et al., 2024). In practice, we first segment $\mathcal{D}$ into
chunks of $L_{o}$ tokens, then average over each chunk’s embeddings to get the
embedding of $\mathcal{D}$. For simplicity, we set the overlap between
adjacent chunks to 0, except for the last chunk, which conditionally overlaps
with the preceding chunk to ensure it contains $L_{o}$ tokens.

Grouped Positions (GP) & Recurrent Positions (RP). Dividing inputs into chunks
and processing them separately sacrifices their interaction in between. By
contrast, position reorganization accommodates longer context by reusing the
original position ids. To be specific, we experiment with two simple
strategies: Grouped Positions and Recurrent Positions. The former groups the
original position ids as such: $f_{gp}(pid)\rightarrow\lfloor pid/s\rfloor$,
while the latter assigns the position ids recurrently within the range
$\\{0,1,...,L_{o}-1\\}$, formulated as: $f_{rp}(pid)\rightarrow pid\bmod
L_{o}$.

Linear Position Interpolation (PI). Instead of reusing position ids, Chen et
al. (2023) introduces new position embeddings via linear interpolation of
existing ones. To apply PI on APE-based models, we map the positions ids as
such: $f_{pi}(pid)\rightarrow pid/s$, and assign embeddings for non-integers
as linear interpolation of that of neighboring integers. In practice, we first
extend the original position embedding matrix
${E}_{o}\in{\mathbb{R}}^{L_{o}\times d}$ into
${E}_{t}\in{\mathbb{R}}^{L_{t}\times d}$, where $d$ stands for hidden size.
Next, we assign ${E}_{t}[i\cdot s]={E}_{o}[i],i\in\\{0,1,...,L_{o}-1\\}$. For
non-integer position id $j$ between $i$ and $i+1$, we determine their
embeddings as follows: ${E}_{t}[s\cdot j]=((i+1-j){E}_{t}[i\cdot
s]+(j-i){E}_{t}[(i+1)\cdot s])$.

Further Tuning. Except for PCW, which divides long texts into smaller blocks
and processes separately, GP, RP, and PI can all be seen as extending the
position embedding matrix. Since APE-based models assign an independent vector
to each position, we can freeze the original model parameters while updating
only the newly added position embeddings. In this way, we can strictly
maintain model ability within 512 context, while harvesting further
performance gains in handling long context as free lunch. Specifically,
further fine-tuning on top of RP and PI is explored in this paper, as
illustrated in Figure 4 (Right). Since the traditional training data for
embedding models are short queries and passages not exceeding 512 tokens, we
manipulate position ids to simulate long training samples, as proposed in Zhu
et al. (2023). See Appendix B for details of further fine-tuning.

###  4.3 Context Window Extension for RoPE-based Models

For RoPE-based models, we further explore Self Extend and NTK, which
respectively advances over GP and PI, harnessing the inherent advantages of
RoPE. Since there is no simple strategy for further training while exactly
maintaining original performance like APE, we leave comprehensive exploration
of training-based context window extension for RoPE-based models for future
work.

Self Extend (SE). Compared with APE, RoPE operates on the query and key
vectors at each layer to encode relative positions, offering enhanced
flexibility for position reorganization. For each token, instead of assigning
grouped relative positions to all other tokens, SelfExtend (Jin et al., 2024)
re-introduces normal relative positions within the nearest neighbor window
$w$, achieving improved performance. For example, consider a document of 10
tokens $\\{x_{0},x_{1},...,x_{9}\\}$ with a neighbor window size $w=4$ and a
group size $g=2$. The relative positions for $x_{0}$ are
$\\{0,1,2,3,4,4,5,5,6,6\\}$. For $x_{4}$, the relative positions of the other
tokens are $\\{-4,-3,-2,-1,0,1,2,3,4,4\\}$.

NTK-Aware Interpolation (NTK). Given a scaling factor $s$, PI proportionally
down-scales position index $m$ to $m/s$. In this way, the attention score
$a({\bm{q}},{\bm{k}})$ defined in Equation 2 becomes
$g({\bm{q}},{\bm{k}},(m-n){\bm{\theta}}/s)$. This is also equivalent to
reducing the frequencies ${\bm{\theta}}$ uniformly, which may prevent the
model from learning high-frequency features, as shown by the Neural Tangent
Kernel (NTK) theory (Jacot et al., 2018). To remedy this, NTK-Aware
interpolation (Peng & Quesnelle, 2023) scales high frequencies less and low
frequencies more to spread out the interpolation pressure across multiple
dimensions. This is achieved by directly altering the original
$\theta_{j}=10000^{-2j/d}$ into $\theta^{\prime}_{j}=(10000\lambda)^{-2j/d}$,
where $\lambda$ is conventionally chosen to be slightly greater than $s$.

##  5 Experiments

###  5.1 Experimental Setup

Benchmarked Models. We evaluate both open-sourced and proprietary models on
LongEmbed, including E5Base (Wang et al., 2022), GTEBase (Li et al., 2023),
BGE-Base (Xiao et al., 2023), Contriever (Izacard et al., 2021), GTR-Base (Ni
et al., 2022), E5-Mistral (Wang et al., 2023b), Jina-V2 (Günther et al.,
2023), Nomic-V1 (Nussbaum et al., 2024), BGE-M3 (Chen et al., 2024), OpenAI-
ada-002. For BGE-M3, we utilize dense vectors. M2 (Saad-Falcon et al., 2024)
is not included in our evaluation, given its training data partly overlaps
with test samples in LongEmbed.

Candidate Models for Extension. From each of the APE-based and RoPE-based
category, we select 2 candidate models for comprehensive study. The former
includes E5Base and GTEBase. The latter includes the 4,096-context E5-Mistral,
and a newly trained E5-RoPEBase, which supports 512 context (See Appendix A
for its training details and BEIR results). Note that E5-RoPEBase employs the
same training procedure and training data as E5Base, only with APE substituted
with RoPE. This facilitates fair comparison of APE / RoPE-based models in
context window extension, as presented in Section 5.4. For implementation
details of each context window extension strategies on each model, please
refer to Appendix B.

###  5.2 Main Results

Table 2: Results (%) of existing and extended embedding models on LongEmbed. NQA, SFD, 2WmQA is short for NarrativeQA, SummScreenFD, 2WikiMultihopQA, respectively. We show that context window extension can effectively improve existing embedding models in handling long context. Model | Param. | Synthetic (Acc@1) | Real (nDCG@10) | Avg.
---|---|---|---|---
Passkey | Needle | NQA | QMSum | SFD | 2WmQA
512 Context Models
E5Base (Wang et al., 2022) | 110M | 38.0 | 28.5 | 25.3 | 23.8 | 74.7 | 55.8 | 41.0
E5-RoPEBase | 110M | 38.5 | 31.5 | 24.6 | 23.2 | 66.6 | 58.8 | 40.5
GTEBase (Li et al., 2023) | 110M | 31.0 | 24.5 | 28.6 | 21.8 | 55.8 | 47.3 | 34.8
BGE-Base (Xiao et al., 2023) | 110M | 18.0 | 25.3 | 25.6 | 22.4 | 60.3 | 51.7 | 33.9
Contriever (Izacard et al., 2021) | 110M | 38.5 | 29.0 | 26.7 | 25.5 | 73.5 | 47.3 | 40.1
GTR-Base (Ni et al., 2022) | 110M | 38.5 | 26.3 | 26.5 | 18.3 | 63.7 | 52.2 | 36.5
$\geq$ 4k Context Models
E5-Mistral (Wang et al., 2023b) | 7B | 71.0 | 48.3 | 44.6 | 43.6 | 96.8 | 82.0 | 64.4
Jina-V2 (Günther et al., 2023) | 137M | 50.3 | 54.5 | 37.9 | 38.9 | 93.5 | 74.0 | 58.2
Nomic-V1(Nussbaum et al., 2024) | 137M | 60.7 | 39.5 | 41.2 | 36.7 | 93.0 | 73.8 | 57.5
BGE-M3 (Chen et al., 2024) | 568M | 59.3 | 40.5 | 45.8 | 35.5 | 94.0 | 78.0 | 58.9
OpenAI-Ada-002 | - | 50.8 | 36.8 | 41.1 | 40.0 | 91.8 | 80.1 | 56.8
Our Extended Models
E5Base \+ Tuning (4k) | 110M | 67.3 | 41.5 | 30.4 | 35.7 | 95.2 | 69.2 | 56.6
E5-RoPEBase \+ SelfExtend (4k) | 110M | 73.5 | 53.5 | 32.3 | 39.1 | 91.9 | 74.6 | 60.8
E5-Mistral + NTK (32k) | 7B | 93.8 | 66.8 | 49.8 | 49.2 | 97.1 | 95.2 | 75.3

Table 2 demonstrates the performance of existing embedding models on our
LongEmbed benchmark. Among the 512-context models, E5Base achieves the highest
average score of 41.0 points, closely followed by E5-RoPEBase and Contriever.
As the supported context length increases beyond 4k, exemplified by E5-Mistral
and Jina-V2, a discernible increase in scores is observed. This verifies both
the efficacy of these long-context models and the validity of LongEmbed to
assess long-context retrieval. Note that even the best performing model
attains only 64.4 pts on average, indicating huge room for improvement in
current models.

In the last row block of Table 2, we further include the best results achieved
by E5Base, E5-RoPEBase and E5-Mistral after context window extension. For
E5Base and E5-RoPEBase, we extend their contexts from 512 to 4,096. For
E5-Mistral, we extend its context from 4,096 to 32,768. Compared to the
original versions, the extended models achieve an average score increase of
+15.6 / +20.3 / +10.9 points. This indicates the efficacy of these context
extension strategies on embedding models, enabling them to handle inputs of
several folds longer. Detailed performance comparison of different extension
strategies on APE & RoPE-based embedding models is presented in Section 5.3.

###  5.3 Performance Comparison of Context Extension Methods

(a)

(b)

Figure 5: (a) Effects of different context window extension methods on E5Base
and GTEBase. We found that plug-and-play methods obtain similar scores, while
further tuning yields the best results. Note that for Tuning, we report
results based on PI by default, as evidenced by: (b) Further ablation of
tuning setups. Tuning on PI consistently outperforms RP across both models.

APE-Based Models. Figure 5(a) illustrates the impact of various context
extension strategies on E5Base and GTEBase across different target context
lengths. We observe that plug-and-play methods including GP, RP, LPI and PCW
strategies yield comparable results with no significant disparities. On the
other hand, further tuning consistently yields additional performance gains
for both models, across all target context lengths. Particularly noteworthy is
GTEBase, which showcases a substantial average score increase of approximately
5 points after further tuning. This suggests that freezing the original model
weights and fine-tuning exclusively the added position embeddings can
effectively extend the model’s context window while strictly maintaining
model’s original ability.

Model | Synthetic (Acc@1) | Real (nDCG@10) | Avg.
---|---|---|---
Passkey | Needle | NQA | QMSum | SFD | 2WmQA
E5-RoPEBase | 38.5 | 31.5 | 24.6 | 23.2 | 66.6 | 58.8 | 40.5
\+ PCW (4k) | 42.5 | 50.8 | 25.1 | 34.9 | 94.9 | 69.3 | 52.9
\+ GP (4k) | 68.0 | 38.8 | 25.9 | 30.9 | 85.8 | 65.8 | 52.5
\+ PI (4k) | 68.3 | 36.0 | 25.9 | 30.8 | 84.9 | 65.3 | 51.9
\+ SE (4k) | 73.5 | 53.5 | 32.3 | 39.1 | 91.9 | 74.6 | 60.8
\+ NTK (4k) | 66.3 | 46.5 | 25.5 | 35.8 | 90.8 | 71.7 | 56.1
E5-Mistral | 71.0 | 48.3 | 44.6 | 43.6 | 96.8 | 82.0 | 64.4
\+ PCW (32k) | 63.5 | 49.5 | 59.3 | 51.3 | 97.3 | 91.2 | 68.7
\+ GP (32k) | 81.0 | 48.8 | 37.0 | 42.9 | 90.6 | 88.1 | 64.7
\+ PI (32k) | 89.8 | 48.5 | 37.8 | 40.4 | 76.8 | 63.0 | 59.4
\+ SE (32k) | 90.8 | 52 | 49.3 | 48.7 | 97.2 | 96.4 | 72.4
\+ NTK (32k) | 93.8 | 66.8 | 49.8 | 49.2 | 97.1 | 95.2 | 75.3
Table 3: Results (%) of context window extension methods on E5-RoPEBase and
E5-Mistral. NQA, SFD, 2WmQA is short for NarrativeQA, SummScreenFD,
2WikiMultihopQA, respectively.

RoPE-Based Models. Table 3 depicts the outcomes of E5-RoPEBase and E5-Mistral
on each dataset of LongEmbed after context window extension via PCW, GP, PI,
SE and NTK. It is observed that RoPE-specific methods including NTK and SE
yield significant improvements for both models across all datasets, surpassing
PCW, PI and GP by a large margin.

###  5.4 Analysis

Tuning on PI vs. RP. Figure 5(b) compares further tuning on top of RP vs. PI.
In the former approach, the initial 512 position embeddings are frozen while
the remaining embeddings are tuned, whereas for the latter, the frozen /
learnable embedding vectors are arranged in an interleaved manner. Our
observations indicate that tuning applied to PI consistently produces superior
results across both models. This superiority may be attributed to the fixed
vectors acting as anchors, thereby preventing the learnable vectors from
converging to suboptimal values.

Figure 6: Best results achieved by extended versions of E5 / E5-RoPE.

RoPE vs. APE. We further discuss the potential of APE / RoPE-based models for
context window extension. E5Base and E5-RoPEBase are selected as the
comparison subjects thanks to their shared training process, training data,
and comparable performance on BEIR and LongEmbed benchmarks. At each target
context length ($\\{1k,2k,4k\\}$), we report the best scores achieved by each
model on LongEmbed, as illustrated in Figure 6. Without requiring further
training, E5-RoPEBase consistently demonstrates superior performance compared
to E5Base across all target lengths. Furthermore, as the target window length
increases, this superiority becomes more pronounced, even surpassing the fine-
tuned version of E5Base by a large margin. This suggests that RoPE-based
models can better extrapolate to to longer context. Consequently, we advocate
for the use of RoPE in future embedding models.

##  6 Conclusion

This paper explores context window extension of existing embedding models.
Through extensive experiments on our LongEmbed benchmark, we show that
training-free context window extension strategies can effectively increase the
input length of these models by several folds. Further, our analysis reveals
the superiority of RoPE-based embedding models over APE-based ones in context
window extension. Hence, we advocate for the use of RoPE for future embedding
models.

## Limitations

As a pioneering work in applying context window extension on embedding models,
this paper is still limited in several aspects, particularly in that most of
the context extension strategies explored in this paper are training-free. As
evidenced by previous findings (Xiong et al., 2023; Fu et al., 2024; Zhang et
al., 2024b; Yen et al., 2024), and the additional performance gain achieved
via tuning on E5Base and GTEBase, we believe further fine-tuning on top of
plug-and-play methods can bring even better extension results. In the future,
we will make comprehensive exploration of training-based context window
extension for embedding models, especially for RoPE-based ones.

## References

  * An et al. (2024) Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong.  Training-free long-context scaling of large language models.  _arXiv preprint arXiv:2402.17463_ , 2024.
  * Bai et al. (2023a) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_ , 2023a.
  * Bai et al. (2023b) Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al.  Longbench: A bilingual, multitask benchmark for long context understanding.  _arXiv preprint arXiv:2308.14508_ , 2023b.
  * Chen et al. (2024) Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu.  Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.  _arXiv preprint arXiv:2402.03216_ , 2024.
  * Chen et al. (2022) Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel.  Summscreen: A dataset for abstractive screenplay summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 8602–8615, 2022.
  * Chen et al. (2023) Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian.  Extending context window of large language models via positional interpolation.  _arXiv preprint arXiv:2306.15595_ , 2023.
  * Chiang & Cholak (2022) David Chiang and Peter Cholak.  Overcoming a theoretical limitation of self-attention.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 7654–7664, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.527.  URL https://aclanthology.org/2022.acl-long.527.
  * Coelho et al. (2024) João Coelho, Bruno Martins, João Magalhães, Jamie Callan, and Chenyan Xiong.  Dwell in the beginning: How language models embed long documents for dense retrieval.  _arXiv preprint arXiv:2404.04163_ , 2024.
  * Deerwester et al. (1990) Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman.  Indexing by latent semantic analysis.  _Journal of the American society for information science_ , 41(6):391–407, 1990.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Ding et al. (2024) Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang.  Longrope: Extending llm context window beyond 2 million tokens.  _arXiv preprint arXiv:2402.13753_ , 2024.
  * Fu et al. (2024) Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng.  Data engineering for scaling language models to 128k context.  _arXiv preprint arXiv:2402.10171_ , 2024.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen.  Simcse: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 6894–6910, 2021.
  * Ge et al. (2023) Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei.  In-context autoencoder for context compression in a large language model.  _arXiv preprint arXiv:2307.06945_ , 2023.
  * Günther et al. (2023) Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, et al.  Jina embeddings 2: 8192-token general-purpose text embeddings for long documents.  _arXiv preprint arXiv:2310.19923_ , 2023.
  * Ho et al. (2020) Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa.  Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pp. 6609–6625, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.  URL https://www.aclweb.org/anthology/2020.coling-main.580.
  * Izacard et al. (2021) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave.  Towards unsupervised dense information retrieval with contrastive learning.  _arXiv preprint arXiv:2112.09118_ , 2(3), 2021.
  * Jacot et al. (2018) Arthur Jacot, Franck Gabriel, and Clément Hongler.  Neural tangent kernel: Convergence and generalization in neural networks.  _Advances in neural information processing systems_ , 31, 2018.
  * Jiang et al. (2023) Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu.  Llmlingua: Compressing prompts for accelerated inference of large language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 13358–13376, 2023.
  * Jin et al. (2024) Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu.  Llm maybe longlm: Self-extend llm context window without tuning.  _arXiv preprint arXiv:2401.01325_ , 2024.
  * Kamradt (2023) Greg Kamradt.  Needle in a haystack - pressure testing llms.  https://github.com/gkamradt/LLMTest_NeedleInAHaystack, 2023.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 6769–6781, 2020.
  * Kočiský et al. (2018) Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette.  The NarrativeQA reading comprehension challenge.  _Transactions of the Association for Computational Linguistics_ , 6:317–328, 2018.  doi: 10.1162/tacl_a_00023.  URL https://aclanthology.org/Q18-1023.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466, 2019.
  * Lefaudeux et al. (2022) Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov.  xformers: A modular and hackable transformer modelling library.  https://github.com/facebookresearch/xformers, 2022.
  * Li et al. (2023) Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang.  Towards general text embeddings with multi-stage contrastive learning.  _arXiv preprint arXiv:2308.03281_ , 2023.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ , 2013.
  * Mohtashami & Jaggi (2023) Amirkeivan Mohtashami and Martin Jaggi.  Landmark attention: Random-access infinite context length for transformers.  _arXiv preprint arXiv:2305.16300_ , 2023.
  * Muennighoff et al. (2023) Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers.  Mteb: Massive text embedding benchmark.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pp. 2014–2037, 2023.
  * Muennighoff et al. (2024) Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela.  Generative representational instruction tuning.  _arXiv preprint arXiv:2402.09906_ , 2024.
  * Neelakantan et al. (2022) Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al.  Text and code embeddings by contrastive pre-training.  _arXiv preprint arXiv:2201.10005_ , 2022.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng.  Ms marco: A human-generated machine reading comprehension dataset.  2016\.
  * Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, et al.  Large dual encoders are generalizable retrievers.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 9844–9855, 2022.
  * Nussbaum et al. (2024) Zach Nussbaum, John X Morris, Brandon Duderstadt, and Andriy Mulyar.  Nomic embed: Training a reproducible long context text embedder.  _arXiv preprint arXiv:2402.01613_ , 2024.
  * Peng & Quesnelle (2023) Bowen Peng and Jeffrey Quesnelle.  Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.  https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have, 2023.
  * Peng et al. (2023) Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.  Yarn: Efficient context window extension of large language models.  _arXiv preprint arXiv:2309.00071_ , 2023.
  * Ratner et al. (2023) Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham.  Parallel context windows for large language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 6383–6402, 2023.
  * Ruoss et al. (2023) Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness.  Randomized positional encodings boost length generalization of transformers.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pp. 1889–1903, 2023.
  * Saad-Falcon et al. (2024) Jon Saad-Falcon, Daniel Y Fu, Simran Arora, Neel Guha, and Christopher Ré.  Benchmarking and building long-context retrieval models with loco and m2-bert.  _arXiv preprint arXiv:2402.07440_ , 2024.
  * Shaham et al. (2022) Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy.  SCROLLS: Standardized CompaRison over long language sequences.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 12007–12021, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.823.  URL https://aclanthology.org/2022.emnlp-main.823.
  * Su (2021) Jianlin Su.  Understanding attention scaling from the perspective of entropy invariance.  https://spaces.ac.cn/archives/8823, Dec 2021.
  * Su et al. (2021) Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.  Roformer: Enhanced transformer with rotary position embedding.  _arXiv preprint arXiv:2104.09864_ , 2021.
  * Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al.  Gemma: Open models based on gemini research and technology.  _arXiv preprint arXiv:2403.08295_ , 2024.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych.  BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.  URL https://openreview.net/forum?id=wCu6T5xFjeJ.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wang et al. (2022) Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei.  Text embeddings by weakly-supervised contrastive pre-training.  _arXiv preprint arXiv:2212.03533_ , 2022.
  * Wang et al. (2023a) Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei.  Simlm: Pre-training with representation bottleneck for dense passage retrieval.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 2244–2258, 2023a.
  * Wang et al. (2023b) Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei.  Improving text embeddings with large language models.  _arXiv preprint arXiv:2401.00368_ , 2023b.
  * Wang et al. (2024a) Suyuchen Wang, Ivan Kobyzev, Peng Lu, Mehdi Rezagholizadeh, and Bang Liu.  Resonance rope: Improving context length generalization of large language models.  _arXiv preprint arXiv:2403.00071_ , 2024a.
  * Wang et al. (2024b) Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei.  Augmenting language models with long-term memory.  _Advances in Neural Information Processing Systems_ , 36, 2024b.
  * Xiao et al. (2024) Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, and Maosong Sun.  Infllm: Unveiling the intrinsic capacity of llms for understanding extremely long sequences with training-free memory.  _arXiv preprint arXiv:2402.04617_ , 2024.
  * Xiao et al. (2023) Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof.  C-pack: Packaged resources to advance general chinese embedding.  _arXiv preprint arXiv:2309.07597_ , 2023.
  * Xiong et al. (2023) Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al.  Effective long-context scaling of foundation models.  _arXiv preprint arXiv:2309.16039_ , 2023.
  * Yen et al. (2024) Howard Yen, Tianyu Gao, and Danqi Chen.  Long-context language modeling with parallel context encoding, 2024.
  * Zhang et al. (2024a) Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou.  Soaring from 4k to 400k: Extending llm’s context with activation beacon.  _arXiv preprint arXiv:2401.03462_ , 2024a.
  * Zhang et al. (2024b) Yikai Zhang, Junlong Li, and Pengfei Liu.  Extending llms’ context window with 100 samples.  _arXiv preprint arXiv:2401.07004_ , 2024b.
  * Zhong et al. (2021) Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev.  QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization.  In _North American Association for Computational Linguistics (NAACL)_ , 2021.
  * Zhu et al. (2023) Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li.  Pose: Efficient context window extension of llms via positional skip-wise training.  In _The Twelfth International Conference on Learning Representations_ , 2023.

##  Appendix A Training Details for E5-RoPEBase

Params |  Pre-training |  Fine-tuning
---|---|---
E5Base | E5-RoPEBase | E5Base | E5-RoPEBase
learning rate | 2$\times 10^{-4}$ | 2$\times 10^{-4}$ | 2$\times 10^{-5}$ | 2$\times 10^{-5}$
GPUs (V100) | 32 | 32 | 8 | 8
warmup steps | 1000 | 1000 | 400 | 400
max length | 128 | 512 | 192 | 192
batch size | 32k | 16k | 256 | 256
max steps | 20k | 20k | n.a. | n.a.
epochs | n.a. | n.a. | 3 | 3
$\tau$ | 0.01 | 0.01 | 0.01 | 0.01
$\alpha$ | n.a. | n.a. | 0.2 | 0.2
weight decay | 0.01 | 0.01 | 0.01 | 0.01
hard negatives | 0 | 0 | 7 | 7
pos embedding | APE | RoPE | APE | RoPE
Table 4: Hyperparameters for contrastive pre-training and fine-tuning of
E5Base and E5-RoPEBase.

In this section, we describe the training details of E5-RoPEBase. Our training
procedure and data exactly follows that of E5 (Wang et al., 2022), where we
first perform contrastive pre-training on their collected CCPairs, then
perform fine-tuning on the concatenation of 3 datasets: MS-MARCO passage
ranking (Nguyen et al., 2016), NQ (Karpukhin et al., 2020; Kwiatkowski et al.,
2019), and NLI (Gao et al., 2021). Each example is paired with 7 hard
negatives. We leverage the mined hard negatives and re-ranker scores from
SimLM (Wang et al., 2023a) for the first two datasets. As the NLI dataset only
provides 1 hard negative per example, we randomly sample 6 sentences from the
entire corpus. xFormers (Lefaudeux et al., 2022) is used for memory efficient
training. As presented in Table 4, training hyperparameters for E5Base and
E5-RoPEBase are identical, except in two aspects:

  * •

Initialization. Before contrastive pre-training, E5Base is initialized on
BERTBase (Devlin et al., 2019), which employs absolute position embeddings
(APE). For the initialization of E5-RoPEBase, we simply replace the APE part
of BERTBase with RoPE. It’s worth noting that the BERTBase model after this
replacement cannot function properly. We count on the subsequent pre-training
phase to adapt the model to RoPE.

  * •

Pre-training length and batch size. E5Base does not update its position
embedding matrix during the training phase, i.e., it utilizes the same
position embedding matrix as BERTBase. This allows it to generalize to input
sequences of up to 512 tokens, while being trained with a max training length
of 192. As for E5-RoPE, replacing APE with RoPE during initialization prevents
us from directly inheriting the original model’s capability in handling 512
tokens. Consequently, in the pre-training phase of E5-RoPE, we set the maximum
training length to 512, and reduce the batch size to 16k according to memory
constraints.

Tasks |  # W/Q. |  # W/D. |  E5Base |  E5-RoPEBase
---|---|---|---|---
MS MARCO |  6.0 |  56.0 |  41.8 |  42.4
Trec-Covid |  10.6 |  160.8 |  69.6 |  73.3
NFCorpus |  3.3 |  232.3 |  35.4 |  34.9
NQ |  9.2 |  78.9 |  58.2 |  60.1
HotpotQA |  17.6 |  46.3 |  69.1 |  61.0
FiQA |  10.8 |  132.3 |  39.8 |  36.4
ArguAna |  193.0 |  166.8 |  44.6 |  54.2
Touche-2020 |  6.6 |  292.4 |  26.4 |  26.6
CQADupStack |  8.6 |  129.1 |  37.4 |  36.5
Quora |  9.5 |  11.4 |  86.6 |  87.7
DBPedia |  5.4 |  49.7 |  42.2 |  40.0
Scidocs |  9.4 |  176.2 |  18.7 |  18.1
Fever |  8.1 |  84.8 |  85.0 |  68.0
Climate-Fever |  20.1 |  84.8 |  26.6 |  19.0
Scifact |  12.4 |  213.6 |  72.0 |  71.0
Average |  < 200 |  < 300 |  50.23 |  48.61
Table 5: Statistics and performance comparison of E5Base and E5-RoPEBase on 15
publicly available BEIR tasks. # W/Q. and # W/D. stands for word number per
query and per document, respectively.

Table 5 demonstrates results of E5Base and E5-RoPEBase on 15 publicly
available BEIR tasks. We observe comparable overall scores between both
models. This comparable performance, along with their shared training process
and training data, facilitates fair comparison of APE and RoPE-based models’s
capabilities in length extrapolation. Note that the slight performance loss of
E5-RoPEBase could possibly be attributed to the replacement of position
embedding in the initialization phase, or the reduced batch size in the pre-
training phase, as mentioned before.

Extension | PCW & GP & RP & PI | NTK | SE
---|---|---|---
GTEBase & E5Base
512 -> 1,024 | $L_{o}=512,L_{t}=1,024,s=2$ | - | -
512 -> 2,048 | $L_{o}=512,L_{t}=2,048,s=4$ | - | -
512 -> 4,096 | $L_{o}=512,L_{t}=4,096,s=8$ | - | -
E5-RoPEBase
512 -> 1,024 | $L_{o}=512,L_{t}=1,024,s=2$ |  $\lambda=3$ (10,000 -> 30,000) | $g=3,w=256$
512 -> 2,048 | $L_{o}=512,L_{t}=2,048,s=4$ |  $\lambda=5$ (10,000 -> 50,000) | $g=5,w=128$
512 -> 4,096 | $L_{o}=512,L_{t}=4,096,s=8$ |  $\lambda=10$ (10,000 -> 100,000) | $g=9,w=64$
E5-Mistral
4,096 -> 8,192 | $L_{o}=4,096,L_{t}=8,192,s=2$ |  $\lambda=3$ (10,000 -> 30,000) | $g=3,w=2,048$
4,096 -> 16,384 | $L_{o}=4,096,L_{t}=16,384,s=4$ |  $\lambda=5$ (10,000 -> 50,000) | $g=5,w=1,024$
4,096 -> 32,768 | $L_{o}=4,096,L_{t}=32,768,s=8$ |  $\lambda=10$ (10,000 -> 100,000) | $g=9,w=512$
Table 6: Hyperparameters for plug-and-play context extension strategies.

##  Appendix B Implementation Details for Context Extension Strategies

This section describes implementation details for the explored context
extension stratgies. For plug-and-play methods including PCW, RP, GP, PI, NTK
and SE, Table 6 summarizes their hyperparameters under each condition.

Further Tuning. On top of PI and RP, we perform further tuning on both E5Base
and GTEBase, utilizing the fine-tuning dataset mentioned in Appendix A.
Following the practice of PoSE (Zhu et al., 2023), we manipulate position ids
to simulate long training samples. Concretely, given an input document
$\mathcal{D}=\\{x_{0},x_{1},...,x_{L_{o}-1}\\}$ of original context length
$L_{o}$, we introduce a skipping bias term $u$ at the beginning of
$\mathcal{D}$, transferring the original position ids $\mathcal{D}$ into
$\\{0,1,...,L_{o}-1\\}$ into $\\{u,u+1,...,u+L_{o}-1\\}$. 555The original
practice of PoSE focuses on relative position, hence introduces bias terms at
the middle of document $\mathcal{D}$. For APE-based models, we simply skips
from the beginning. For every piece of training data, $u$ is re-sampled from
the discrete uniform distribution $\mathcal{U}(\\{0,1,...,L_{t}-L_{o}\\})$. In
this way, we ensure comprehensive coverage of target context window. The
training procedure spans 3 epochs on 2 A100 GPUs, with a learning rate of
$5e^{-4}$, a batch size of 512, and 100 steps for warmup. Other
hyperparameters are same as Table 4.

Inference. In inference time, attention scaling (Su, 2021; Chiang & Cholak,
2022) is used by default for all tested models for better length extrapolation
ability. Especially for GTEBase and E5Base tuned on PI, we use the original
position ids when input length not exceeds 512. This is achived by mapping the
position ids $\\{0,1,...,l\\}$ into $\\{0,s,...,l\times s\\}$, where $s$ is
the scaling factor, $l<512$.

##  Appendix C Further details on LongEmbed

Figure 7: Source and examples for each dataset in LongEmbed.

Figure 7 presents source and examples for each dataset included in LongEmbed.
For QA datasets including NarrativeQA and 2WikiMultihopQA, we adopt their test
splits. Note that for 2WikiMultihopQA, we adopt the length-uniformly sampled
version from Bai et al. (2023b) to better assess the model’s capabilities
across various context lengths. For summarization datasets including QMSum and
SummScreenFD, we adopt the version processed by SCROLLS (Shaham et al., 2022).
Since SCROLLS does not include ground truth summarization in its test sets, we
switch to validation set for these two datasets. Particularly for QMSum, as
its validation set only have 60 documents, which is too small for document
retrieval, we included the train set as well.


0 Research please specify Bryce Morrow is with the University of North
Carolina at Chapel Hill. E-mail<EMAIL_ADDRESS>Trevor Manz is with
Harvard Medical School. Email<EMAIL_ADDRESS>Arlene E. Chung is
with the University of North Carolina School of Medicine. E-mail:
<EMAIL_ADDRESS>Nils Gehlenborg is with Harvard Medical School.
Email<EMAIL_ADDRESS>David Gotz is with the University of North
Carolina at Chapel Hill. Email<EMAIL_ADDRESS>Morrow et al.: Periphery Plots

# Periphery Plots for Contextualizing Heterogeneous Time-Based Charts

Bryce Morrow  Trevor Manz  Arlene E. Chung  Nils Gehlenborg  David Gotz

###### Abstract

Patterns in temporal data can often be found across different scales, such as
days, weeks, and months, making effective visualization of time-based data
challenging. Here we propose a new approach for providing focus and context in
time-based charts to enable interpretation of patterns across time scales. Our
approach employs a focus zone with a time and a second axis, that can either
represent quantities or categories, as well as a set of adjacent periphery
plots that can aggregate data along the time, value, or both dimensions. We
present a framework for periphery plots and describe two use cases that
demonstrate the utility of our approach.

###### keywords:

Time-based data, Focus + context techniques, Health informatics, mHealth,
Patient-generated health data.

Periphery plots used within a visualization of multi-variate climate data. The
(a) control timeline includes interactive controls to define (b1, b2) context
and (c) the focus zone. This example includes four tracks (f), each with
(d1-d8) two periphery plots. (e) Annotations support quick comparisons, such
as the seasonal variation between average maximum temperatures in summer
months (the focus region) and the preceding and subsequent winter periods (d1,
d5). Introduction

Temporal data is found frequently in various application domains and comes in
many forms. Data can range from sparse point-based and interval events (e.g.,
procedures and medication changes) to densely sampled time series with
repeated scalar measurements (e.g., longitudinal climate measurements). Often,
data have combinations of variables and data types (e.g., daily stock prices
and trading volumes along with the dates of key news events).

Based on these temporal data sets, users must often explore data and answer
complex temporal questions. For example, consider a clinician exploring data
gathered from mobile health (mHealth) devices (e.g., fitness trackers and
mobile phones) along with patient-reported data (e.g., diet and patient-
reported outcomes about health status) and clinical data (e.g., medications
and diagnoses). To make sense out of this kind of data, one must: (1) view
data across a range of time scales (e.g., daily exercise fluctuations vs.
monthly patient-reported outcomes vs. semi-annual doctor visits), (2)
correlate across variables (e.g., does a change in medication correlate with
changes in patient-reported outcomes?), and (3) compare across time intervals
(e.g., has a patient’s current condition improved compared to historical
data?).

Given the ubiquity of temporal data, a wide variety of visual analytics
techniques have been proposed for representing time-based events. This
includes multi-variate techniques as well as a range of different focus +
context methods. However, these techniques often focus on homogeneous types of
data (e.g., scalar valued time series) or have other constraints that limit
their use (see Section 1). This paper presents _periphery plots_ , a technique
for augmenting time-aligned temporal charts with a set of horizontally aligned
peripheral views that provide multi-resolution contexts before and after a
focal temporal period. This approach is applicable to heterogeneous data types
and varied time samplings (regular vs. irregular; hourly vs. annually).
Moreover, it is capable of representing a variety of previously proposed focus
+ context methods, while simultaneously supporting new contextualization
capabilities. More specifically, this research has three key contributions:

  * •

A Framework for Periphery Plots. The framework includes key concepts of focus
and context zones. It also includes an axis preservation design space for
plots that allow visualizations with varying levels of summarization.

  * •

A Periphery Plot Toolkit. The framework is implemented within an open-source
toolkit that includes a collection of specific view types within a general
purpose architecture.

  * •

Example Applications. Two use cases (longitudinal climate data analysis and
pattern finding within patient-generated health data) demonstrate the utility
of periphery plots.

##  1 Related Work

This section provides a brief review of the most relevant themes from prior
research on temporal data visualization.

###  1.1 Visual Representations of Temporal Data

Time series data sets are ubiquitous and are most commonly represented as line
charts. Prior research has mainly focused on developing alternative visual
encodings to convey equal information while using less space, such as horizon
charts [8] and braided graphs [12]. For horizon charts, interpretability can
be preserved and improved while reducing space usage by optimizing chart
parameters (number of bands, height of chart) [10]. Another approach is to
create complex visual encodings with high information density. For example,
both ThemeRiver[9] and RankExplorer [18] utilize stacked graphs as a base
chart to display categorical partitions of multi-variate time series data,
while the latter layers additional glyphs to represent changes in information
density between categories and time points. Alternatively, Chronodes condenses
multiple features into a single event timeline to highlight frequent event
sub-sequences, where glyphs encode unique health related features [16]. The
periphery plots framework is compatible with a wide range of these
aforementioned techniques because it supports the use of both common and
bespoke chart types. Moreover, it can dynamically transition between types in
response to changes in data density.

###  1.2 Detail and Context Interfaces

Exploratory analysis often requires users to interact with data at a level of
detail that cannot be displayed on a single screen. While interactive methods
such as zooming & panning offer the ability to view details otherwise hidden
due to summarization, these interactions alone may obscure context and hinder
interpretation due to narrow fields-of-view [4]. A variety of techniques have
been developed to address this challenge, including overview + detail and
focus + context approaches. The former employs spatial separation to partition
global and detailed information (e.g., Stack Zooming [11]) while the latter
combines these displays in a single continuous view (e.g., PerspectiveWall
[15] or fish-eye lenses [6, 19]).

Complementary approaches which incorporate these schemes with interactive
techniques have yielded powerful dynamic tools for visual exploration,
particularly for temporal data. SignalLens utilizes focus + context with
panning to enable rapid navigation of time series data, but users are unable
to define the initial focus from a global overview [13]. In contrast,
MultiStream [7] combines overview + detail and focus + context, linking
interactive brush selections from a global overview to contextualize the focus
of a fisheye distortion in a more detailed multi-resolution view. Our proposed
periphery plot framework uses a similar combined overview + detail and focus +
context approach. However, it focuses on views of context zones (rather than
the focus) and includes a design space for various levels of contextual
summarization.

###  1.3 Design Space for Multi-variate Time-Based Charts

KronoMiner [20] presents a taxonomy for the design space of layouts for
visualizing multi-variate, time-based data. It includes stacking (i.e.,
multiple views are stacked on an axis to create global time alignment) and
overplotting (i.e., multiple visual encodings are layered in a single view).
Our framework supports both techniques and adds the ability to utilize
multiple non-overlapping encodings (one for each zone) in a single stacked
track.

##  2 Periphery Plots Framework

The periphery plots framework provides multi-scale contextualization
capabilities for viewing heterogeneous time-based data, and includes several
core concepts for how periphery plots connect with each other and other
visualization elements. The framework also defines a two-dimensional design
space for periphery plots themselves and a set of common user interactions.

###  2.1 Concepts

Periphery plots are defined by the relationship between temporally linked
_focus_ and _context zones_ , which represent user-specified intervals along a
dataset’s time axis. These zones are used to determine the intervals of data
to be visualized within time-based _tracks_. Each track includes a set of
_periphery plots_ and a _focus plot_ with one plot for each zone. The zones
can be manipulated via interactions with the plots on individual tracks or
through a global _control timeline_.

Focus and Context Zones. Zones are the primary concept that underpins the
framework for periphery plots. For a given temporal data visualization,
periphery plots define a single focus zone and a set of context zones. Each
zone is an interval of time defined by start and end points that are linked
together in a non-overlapping, contiguous sequential ordering. Figure
Periphery Plots for Contextualizing Heterogeneous Time-Based Charts(b1,c,b2)
shows an example that has two context zones, one to each side of the focus
zone. The framework allows multiple context zones on either side of the focus
as demonstrated in the accompanying video figure.

The focus zone corresponds to the time period of primary interest. The
visualization used for the focus period is designed to show data at high
resolution for detailed analysis (e.g., a line chart, a horizon graph, or an
event timeline) with the visual design dependent, in part, on the specific
data type and application. The focus zone is equivalent to the interval of
temporal data displayed in a more typical visualization that does not include
periphery plots.

In contrast, context zones are defined relative to the focus zone as either
pre- or post-focus intervals of time. Context zones are not generally the
user’s primary focus of attention. Instead, they represent neighboring periods
of time to the focus which provide context to users during interpretation of
data found within the focus zone.

While it is common to define two context zones (one pre- and one post-focus),
periphery plots allow multiple context zones to enable multi-scale
contextualization (e.g, six context zones can be used to contextualize a focus
time interval with a week, month, or year before and afterwards). Thus, an
arbitrary set of $n$ context zones are defined relative to a single focus
zone.

Each context zone is visualized using periphery plots. As the screen space
allocated to a periphery plot is typically more constrained than the focus,
data summarization is a key element in designing effective periphery plots.
This motivates the periphery plot design space outlined in Section 2.2.

Tracks. A track is a container that horizontally aligns a set of
visualizations that correspond, typically, to a single variable. As shown in
Figure Periphery Plots for Contextualizing Heterogeneous Time-Based Charts(f),
each track contains a periphery plot for each context zone and a single focus
visualization corresponding to the focus zone. The visualizations within a
track are arranged contiguously from left to right, ordered by the sequence of
time intervals for the corresponding context and focus zones.

Multiple tracks can be used to visualize multiple variables simultaneously. In
this case, all tracks are defined using the same set of context and focus
zones. In addition, the tracks are stacked vertically with boundaries between
zones aligned as shown in Figures Periphery Plots for Contextualizing
Heterogeneous Time-Based Charts and 2. This facilitates comparison across
tracks by lining up data from the same time interval in vertical columns. The
visual encodings for each track can be configured independently, supporting
the visualization of heterogeneous data types.

Control timeline. The specification of context and focus zones is supported
via a single control timeline. This mechanism includes a set of multiple-
linked brushes that map one-to-one to the individual zones. Any changes to the
focus or context zones made through the control timeline are tightly
coordinated with the tracks to support a range of interactions as described in
Section 2.3.

###  2.2 Periphery Plot Design Space

Each context zone is visualized using one periphery plot per track. As
previously described, periphery plots are typically space constrained which
makes summarization a key design goal. We characterize the design space for
periphery plots in terms of an Axis Preservation Model that includes a two-
dimensional taxonomy for visual encodings of temporal data. The model
categorizes encodings based on whether or not they map the _time domain_
and/or _value domain_ from data space to the $x$ and $y$ coordinates of screen
space, respectively. This model is used for both continuous and discrete
values (e.g., ordinal values or nominal categories).

This model leads to four types of periphery plots as illustrated in Figure 1:
time-value-axis preserving (TVAP), time-axis preserving (TAP), value-axis
preserving (VAP), and no-axis preserving (NAP).

TVAP plots, such as the bar charts in Figure Periphery Plots for
Contextualizing Heterogeneous Time-Based Charts(d2) or dot plots in Figure
Periphery Plots for Contextualizing Heterogeneous Time-Based Charts(d3),
present the most detailed information. As data gets dense, aggregation by
value or time can lead to more summarized views. For example, VAP plots
aggregate over time but preserve values as in the histograms in Figure
Periphery Plots for Contextualizing Heterogeneous Time-Based Charts(d1,d4).
TAP plots, in contrast, aggregate by value but preserve time. For example, a
vertical band chart that shows the counts of observations per time period
without distinguishing between values of observations as depicted symbolically
in Figure 1(2). NAP plots adopt designs that preserve neither time nor value,
such as grid-based or pixel-based visualizations.

We note that the axis preservation model applies to a variety of time- and
value-based transformations. For example, a moving average envelope (e.g.,
Figure 1(1)) that aggregates values within a sliding time window would be
classified as TVAP because it maintains time as a dimension despite the
windowing transformation.

Summarization. Axis preservation can be viewed as an approach to summarization
through relaxed constraints on visual encoding. Preserving both time and value
places constraints on visual design that can make it difficult for periphery
plots to effectively display the large scale volumes of data that often fall
within a context zone. This problem is exacerbated by the limited screen space
typically allocated to periphery plots.

Sacrificing an axis allows for more flexibility in visual encoding. This can
make visualizations less informative in some ways (the omitted axis), but
enables new types of more space-efficient visual representations. This
observation imposes an ordering on plot types with increasing levels of
summarization: TVAP $\rightarrow$ (VAP,TAP) $\rightarrow$ NAP. Transitions
between periphery plot types are a natural approach to handling changes in
data volume within a context zone, and two summarization paths fit naturally
within the axis preservation model as shown by the arrows in Figure 1(a,b).
Moreover, such transitions can be automated as discussed in Section 3.2.

Annotations. To facilitate comparisons between zones within a single track,
our design introduces _annotations_. These are layered graphics that allow for
comparisons of derived measures between plots, including plots that implement
different visual encodings. Many common types of derived measurements can be
compared via annotations, including averages, confidence intervals, and
quantiles.

The temp max track in Figure Periphery Plots for Contextualizing Heterogeneous
Time-Based Charts demonstrates the utility of average line annotations for
identifying differences in values in the focus area and the context zones. As
the orange dotted line annotations in Figure Periphery Plots for
Contextualizing Heterogeneous Time-Based Charts(e) show, the value within the
focus zone is higher than in the subsequent time period. As demonstrated in
the accompanying video, further insights can be obtained from dynamic changes
in annotations in response to user interactions. This is because those
interactions can result in different intervals of data being displayed in both
focus and context zones.

Figure 1: Periphery plot types can be characterized by their preservation of
the time and value axes. Summarization can prioritize (a) value preservation
or (b) time preservation.

###  2.3 Interactions

Periphery plots can be informative even for static visualizations. However,
our framework for periphery plots includes a set of user interactions that
build upon the core concepts defined in this section to help support
interactive data exploration.

Zooming and Panning. Users can zoom and pan through the overall time scale to
explore data. Users can execute these interactions from the focus zone on any
track, using the scroll-wheel to zoom or dragging the mouse to pan. These
interactions are not supported in periphery plots, which may have a
drastically different time scale (or none at all). Similar zoom and pan mouse
gestures can be performed via the control timeline. Regardless of where a pan
or zoom originates, all tracks are kept synchronized to maintain temporal
alignment of the various focus and context zones.

Additionally, mouse hovers within any focus zone will display a vertical time
indicator across all tracks. This helps facilitate navigation of the time
axis, identification of specific date and time values, and visual alignment of
data in the time axis.

Multi-Brush Zone Control. Beyond the aforementioned pan and zoom interactions,
all other manipulations of the focus and context zones are performed via the
control timeline. A pan or zoom event may cause the extent of a context zone
to extend beyond the limits of the time axis. This allows users to move the
focus zone to the very start or end of the overall time axis without having to
remove or adjust context zones.

Users can change the extent of a single zone by manipulating the zone’s
corresponding brush on the control timeline. Each brush includes two _handles_
representing the start and end of the corresponding zone. The handles are
visible as green squares in Figure Periphery Plots for Contextualizing
Heterogeneous Time-Based Charts(b1,c,b2). Users can drag any handle to resize
the corresponding focus or control zone. Other zones will retain their size,
but will be “pushed” or “pulled” to maintain the requirement that all zones
are contiguous without gaps between them.

The “push” and “pull” behavior is not always desired. For this reason, users
can enable _locks_ to anchor a zone boundary point in time. The locks, visible
in Figure Periphery Plots for Contextualizing Heterogeneous Time-Based
Charts(b1,c,b2) as black squares, can be enabled or disabled by clicking. When
a lock is enabled, the handles above it are disabled since the corresponding
boundary cannot be modified until the lock is removed. Locks also impact
panning behaviors. This allows, for example, a user to lock context zones at
the very start and end of the control timeline to create periphery plots that
summarize “all data before” and “all data after” the focus zone.

Figure 2: The Precision VISSTA App for data exploration uses periphery plots
to help users understand how the values in the current focus area relate to
data recorded prior to or after the focus time period. This example shows
sparse and discrete values shown in two formats: (a,b) time and value
preserving charts that condense the time axis, or (c) value-preserving
histogram charts that eliminate the time axis in order to display data about a
higher number of observations. Periphery plots can be configured to
automatically transition between these modes.

##  3 Toolkit and Example Use Cases

Periphery plots were developed in the context of a broader project (Precision
VISSTA [2]) focused on patient-generated health data (PGHD). The techniques
were then generalized within a reusable toolkit for easy application to
support the analysis of other datasets and domains. This section describes the
reusable periphery plot toolkit and two example applications.

###  3.1 Toolkit

The core components of the periphery plot framework were first developed in
JavaScript as part of the patient-generated health data platform described in
Section 3.2. That application leverages the React framework and D3.js [5] to
implement the control timeline, individual data tracks, and the corresponding
periphery plots. To facilitate reuse, we extracted these components and
bundled them as a generic periphery plot toolkit. The toolkit provides all
core capabilities within a React-based component that can be: (1) easily
integrated into React-based web applications; and (2) easily customized and
extended with new graphic styles, chart types, annotations, or interactions.
The source code has been released as open source software [1].

###  3.2 Use Case 1: Patient-Generated Health Data

Periphery plots were first developed to support visualizations of PGHD for
clinical interpretation [2]. This includes mHealth data gathered from various
sensors (e.g., total steps or time awake during sleep) and manually entered
logs (e.g., nutrition), as well as patient-reported outcomes data gathered via
validated electronic surveys. PGHD contains multiple longitudinal variables
with heterogeneous data types. Moreover, these variables are recorded at
different rates with varied sparseness, and exhibit different forms of
missingness. To make this complex combination of longitudinal records
clinically useful, a visualization must allow users to quickly look for long
term trends, outliers, patterns, and correlations across variables.

We applied the periphery plot framework to this problem as shown in Figure 2.
The vertical alignment across tracks allows comparisons across variables. The
most widely used periphery plot chart type was a value-preserving histogram
showing the distribution of values in the pre- and post-focus context periods.
This helps to show, for example, that the spikes in the _time awake during
sleep_ track are unusually high compared to values in the context zones (which
are set to show all data prior to or after the focus).

When the data in a context region is sufficiently sparse, a value-and-time
preserving periphery plot chart type is used. This was motivated by the
relatively sparse nature of many of the PGHD variables which are collected
manually (rather than via sensors). Figure 2(a) exploits this feature to show
not only that earlier anxiety scores were higher, but that they have been
decreasing leading up to the focus time period. This feature is very useful,
but does not scale well as the number of values in the context increases. For
this reason, periphery plots are designed to automatically transition from (a)
the value-and-time preserving view to (c) a value-preserving summarization, if
a user interaction results in a higher number of samples in the periphery that
exceeds a defined threshold.

###  3.3 Use Case 2: Climate Data

To demonstrate the versatility of periphery plots, we applied the toolkit to a
second data set: a publicly available collection of Seattle weather
observations [3]. This use case is highlighted in Figure Periphery Plots for
Contextualizing Heterogeneous Time-Based Charts.

Compared to the health application above, weather data is more homogeneous
(mostly scalars over time) and is more uniformly sampled over time. Moreover,
it exhibits less sparsity and missingness. Yet despite these differences, the
periphery plot framework can be easily applied to this use case. This
application was developed with minimal effort by connecting CSV data with
built-in chart types.

The interactive application allows for the discovery of several interesting
observations. The screenshot in Figure Periphery Plots for Contextualizing
Heterogeneous Time-Based Charts shows examples, including the seasonal
fluctuations in temperatures highlighted in the _temp max_ track. Here, the
focus and context zones were each set to approximately six months in duration.
The annotations in that track, Figure Periphery Plots for Contextualizing
Heterogeneous Time-Based Charts(e), show that the focus period is much hotter
than the period before and after. Panning the control timeline forward or back
shows this pattern oscillate as the focus moves from winter to summer an back
again. A second finding is shown in Figure Periphery Plots for Contextualizing
Heterogeneous Time-Based Charts(d4) and highlights some drizzling
precipitation. This is quite different from the current focus, which is more
similar to the distribution of observations shown in the post-focus context
region in Figure Periphery Plots for Contextualizing Heterogeneous Time-Based
Charts(d8).

##  4 Discussion and Conclusion

The use cases in Section 3 illustrate the strengths of periphery plots in
contextualizing detailed time-based data within a larger time frame. The
examples demonstrate, in particular, how the periphery plot framework provides
a flexible approach for visual encoding in the context zone. This includes the
automated transition between plot types to adjust the level of information
preservation based on available space. There are, however, some limitations in
our current definition of the framework. For example, the framework does not
explicitly cover a scenario in which multiple variables are shown within the
same track. Furthermore, the framework does not ensure that periphery plots
are easy to compare within or between tracks. In future work, we hope to
explore solutions to this challenge by building upon prior research into the
design space and best practices for multi-view comparability and consistency
[17, 14].

In summary, we demonstrate that the periphery plot framework for providing
contextual information for heterogeneous time-based data is generalizable and
delivers insights that would not be possible without the context provided by
our approach. Despite the relatively limited design space, periphery plots are
an effective approach that is easy to implement and use. Furthermore, the
approach does not require any complex visual representations or interactions
to convey information critical to the interpretation of the data. Therefore,
we anticipate that periphery plots have the potential to be effective for a
wide range of users across multiple application domains.

###### Acknowledgements.

This research was supported by the National Institutes of Health under award
number 5R01EB025024. Additional support was provided by the National Science
Foundation via Grant #1704018.

## References

  * [1] Periphery Plots Open Source Software.  https://github.com/PrecisionVISSTA/PeripheryPlots/.  Accessed: 2019-06-10.
  * [2] Precision VISSTA: Enabling Precision Medicine through the Development of Quantitative and Visualization Methods.  http://precisionvissta.web.unc.edu/.  Accessed: 2019-06-10.
  * [3] Seattle Weather Exploration.  https://vega.github.io/vega-lite/examples/interactive_seattle_weather.html.  Accessed: 2019-06-09.
  * [4] D. Borland, W. Wang, and D. Gotz.  Contextual Visualization.  IEEE Computer Graphics and Applications, 38:17–23, 2018. doi: 10 . 1109/MCG . 2018 . 2874782
  * [5] M. Bostock, V. Ogievetsky, and J. Heer.  D3: Data-Driven documents.  IEEE Trans. Vis. Comput. Graph., 17(12):2301–2309, Dec. 2011.
  * [6] A. Cockburn, A. Karlson, and B. B. Bederson.  A review of overview+detail, zooming, and focus+context interfaces.  ACM Computing Surveys, 41(1):1–31, Dec. 2008. doi: 10 . 1145/1456650 . 1456652
  * [7] E. Cuenca, A. Sallaberry, F. Y. Wang, and P. Poncelet.  MultiStream: A Multiresolution Streamgraph Approach to Explore Hierarchical Time Series.  IEEE Transactions on Visualization and Computer Graphics, 24(12):3160–3173, Dec. 2018. doi: 10 . 1109/TVCG . 2018 . 2796591
  * [8] S. Few.  Time on the Horizon.  Visual Business Intelligence Newsletter, p. 7, July 2008.
  * [9] S. Havre, B. Hetzler, and L. Nowell.  ThemeRiver: visualizing theme changes over time.  In IEEE Symposium on Information Visualization 2000. INFOVIS 2000. Proceedings, pp. 115–123, Oct. 2000. doi: 10 . 1109/INFVIS . 2000 . 885098
  * [10] J. Heer, N. Kong, and M. Agrawala.  Sizing the Horizon: The Effects of Chart Size and Layering on the Graphical Perception of Time Series Visualizations.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’09, pp. 1303–1312. ACM, New York, NY, USA, 2009\.  event-place: Boston, MA, USA. doi: 10 . 1145/1518701 . 1518897
  * [11] W. Javed and N. Elmqvist.  Stack zooming for multifocus interaction in skewed-aspect visual spaces.  IEEE transactions on visualization and computer graphics, 19(8):1362–1374, Aug. 2013. doi: 10 . 1109/TVCG . 2012 . 323
  * [12] W. Javed, B. McDonnel, and N. Elmqvist.  Graphical Perception of Multiple Time Series.  IEEE Transactions on Visualization and Computer Graphics, 16(6):927–934, Nov. 2010. doi: 10 . 1109/TVCG . 2010 . 162
  * [13] R. Kincaid.  SignalLens: Focus+Context Applied to Electronic Time Series.  IEEE Transactions on Visualization and Computer Graphics, 16(6):900–907, Nov. 2010. doi: 10 . 1109/TVCG . 2010 . 193
  * [14] S. Knudsen and S. Carpendale.  View Relations: An Exploratory Study on Between-View Meta-Visualizations.  In Proceedings of the 9th Nordic Conference on Human-Computer Interaction, NordiCHI ’16, pp. 15:1–15:10. ACM, New York, NY, USA, 2016.  event-place: Gothenburg, Sweden. doi: 10 . 1145/2971485 . 2971566
  * [15] J. D. Mackinlay, G. G. Robertson, and S. K. Card.  The perspective wall: detail and context smoothly integrated Mackinlay color plates.  In Proceedings of the SIGCHI conference on Human factors in computing systems Reaching through technology - CHI ’91, pp. 173–176. ACM Press, New Orleans, Louisiana, United States, 1991. doi: 10 . 1145/108844 . 108870
  * [16] P. J. Polack, S.-T. Chen, M. Kahng, K. DE Barbaro, R. Basole, M. Sharmin, and D. H. Chau.  Chronodes: Interactive Multifocus Exploration of Event Sequences.  ACM transactions on interactive intelligent systems, 8(1), Feb. 2018\. doi: 10 . 1145/3152888
  * [17] Z. Qu and J. Hullman.  Keeping Multiple Views Consistent: Constraints, Validations, and Exceptions in Visualization Authoring.  IEEE transactions on visualization and computer graphics, 24(1):468–477, 2018. doi: 10 . 1109/TVCG . 2017 . 2744198
  * [18] C. Shi, W. Cui, S. Liu, P. Xu, W. Chen, and H. Qu.  RankExplorer: Visualization of Ranking Changes in Large Time Series Data.  IEEE Transactions on Visualization and Computer Graphics, 18(12):2669–2678, Dec. 2012. doi: 10 . 1109/TVCG . 2012 . 253
  * [19] M. Tory and T. Moller.  Human factors in visualization research.  IEEE Transactions on Visualization and Computer Graphics, 10(1):72–84, Jan. 2004. doi: 10 . 1109/TVCG . 2004 . 1260759
  * [20] J. Zhao, F. Chevalier, and R. Balakrishnan.  KronoMiner: Using Multi-foci Navigation for the Visual Exploration of Time-series Data.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’11, pp. 1737–1746. ACM, New York, NY, USA, 2011\.  event-place: Vancouver, BC, Canada. doi: 10 . 1145/1978942 . 1979195