In this paper, we propose a new dense retrieval model which learns diverse document representations with deep query interactions.
Our model encodes each document with a set of generated pseudo-queries to get query-informed, multi-view document representations.
It not only enjoys high inference efficiency like the vanilla dual-encoder models, but also enables deep query-document interactions in document encoding and provides multi-faceted representations to better match different queries.
Experiments on several benchmarks demonstrate the effectiveness of the proposed method, out-performing strong dual encoder baselines.[The code is available at <https://github.com/jordane95/dual-cross-encoder>.]

§ INTRODUCTION

Document retrieval plays an important role in information retrieval (IR) tasks such as web search and open domain question answering <cit.>.
Early works such as BM25-based retriever <cit.> rely on lexical term matching to calculate the relevance of a pair of texts.
Recently, neural network based dense retrieval <cit.> has gained traction in research community. Dense retrieval learns a neural encoder to map queries and documents into a dense, low-dimensional vector space, and is less vulnerable to term mismatch problem compared to lexical match-based approaches.

There are two architectures to model the relevance between queries and documents.
Dual encoder architecture encodes query and document separately into fixed-dimensional vectors <cit.>, where
the similarity between query and document is usually instantiated as a dot product or cosine similarity between their vectors. As there are no interactions between query and document, dual encoder approach permits efficient inference with vector space search on pre-computed document vectors.
Cross encoder architecture feeds the concatenation of a query and document pair into one encoder to calculate its relevance score <cit.>.
Compared to dual encoder, cross encoder is more accurate due to the deep interaction between query and document, but comes with computation costs infeasible for first-stage retrieval. It is highly desirable to design a retrieval model which can match the performance of the cross encoder approach while maintaining the inference efficiency of the dual encoder approach.

To this end, previous works mainly focus on two directions: late-interaction and distillation.
The first solution is to design a hybrid architecture, where the early layers act as a dual encoder while the late layers work like a cross encoder <cit.>.
Its effectiveness comes with the cost of retrieval latency due to the computation involved with late layers.
Another solution is knowledge distillation <cit.>, using the cross encoder to augment the training data <cit.>,
or distilling the ranking scores or attention matrices of a more powerful cross encoder reranker to a dual encoder retriever <cit.>.

In this paper, we propose to achieve this goal by pre-computing the interaction-based representations.
As depicted in Figure <ref>, the document representations are obtained by feeding the concatenation of query and document through a cross encoder while the query representation is obtained in the same way as in the vanilla dual encoder.
For every document, we use a query generation model to generate several queries which will each concatenate with the document to obtain a separate document representation.

Our model has the following advantages.
Firstly, we can obtain document representation with deep query interactions without much additional inference cost.
Additionally, we can naturally get multi-view document representations <cit.> by treating the query as explicit view extractor.

de/.style =
rounded corners = 5pt,

minimum height = 40pt,
inner sep=5pt,
fill = lightgray!10,
align = center,
ce/.style =
rounded corners =5pt,
minimum width = 110pt,
minimum height =40pt,
inner sep=5pt,
draw = gray,
fill = gray!20,
align = center,
query/.style =
rounded corners = 3pt,

minimum height = 20pt,

draw = gray,
fill = pink!15,
align = center,
doc/.style =
rounded corners = 3pt,

minimum height = 20pt,

draw = gray,
fill = red!15,
align = center,
cls/.style =
minimum width = 3pt,
minimum height = 1pt,
draw = gray,
fill = yellow!50,
align = center,

[query] (q) at (0, 0.5) Query;
[doc] (d) at (2, 0.5) Document;
[de] (qe) at (0,2) Query
[de] (de) at (2,2) Document
[cls] (qcls) at (0, 3.5) CLS;
[cls] (dcls) at (2, 3.5) CLS;
(score) at (1, 4.5) score;
[-latex] (q) – (qe);
[-latex] (d) – (de);
[-latex] (qe) – (qcls);
[-latex] (de) – (dcls);
[-latex] (qcls) – (score);
[-latex] (dcls) – (score);

Dual Encoder

[query] (q) at (0, 0.5) Query;
[doc] (d) at (2, 0.5) Document;
[ce] (ce) at (1, 2) Cross Encoder;
[de, minimum width = 5pt, minimum height = 5pt] (mlp) at (1, 3.5) MLP;
(score) at (1, 4.5) score;
[-latex] (q.north) – (q.north|-ce.south);
[-latex] (d.north) – (d.north|-ce.south);
[-latex] (ce) – (mlp) – (score);

Cross Encoder

[query] (q) at (0, 0.5) Query;
[query] (qq) at (2, 0.5) Query;
[doc] (d) at (4, 0.5) Document;
[de] (qe) at (0, 2) Query
[ce] (ce) at (3, 2) Document
[cls] (qcls) at (0, 3.5) CLS;
[cls] (dcls) at (3, 3.5) CLS;
(score) at (1.5, 4.5) score;
[-latex] (q) – (qe);
[-latex] (qq.north) – (qq.north|-ce.south);
[-latex] (d.north) – (d.north|-ce.south);
[-latex] (qe) – (qcls);
[-latex] (ce) – (dcls);
[-latex] (qcls) – (score);
[-latex] (dcls) – (score);

Dual Cross Encoder

Illustration of different matching paradigms with different architectures.

We follow the popular contrastive learning paradigm for learning such representations.
Experiments on several retrieval benchmarks demonstrate the effectiveness of the proposed approach.

To summarize, our main contributions are as follows:

* We propose a new model architecture for dense retrieval, which can benefit from deep query-document interaction with low inference latency and learn multi-view document representations to better match different queries.

* We show the effectiveness of this model over various baselines by experiments on several large-scale retrieval benchmarks.

§ RELATED WORK

§.§ Dense Retrieval

Dense passage retrieval (DPR) <cit.> learns a two-tower BERT encoder to represent question and passage as vectors and takes their dot product as relevance score.
The training of such dense retrievers can be optimized with more sophisticated negative sampling strategy <cit.>,
or knowledge distillation from a more powerful cross-encoder teacher <cit.>.

Recently, some work have been devoted to trading off the efficiency and effectiveness with a late-interaction architecture.
<cit.> compress the query context into multiple dense vectors with a Poly-Encoder architecture. The relevance score is modeled by a attention-weighted sum of individual matching scores.
<cit.> further improve the multi-encoding scheme through $k$-means clustering over all document tokens' embeddings.
ColBERT <cit.> learns word level representations for both query and document and calculates the relevance score with a MaxSim operation followed by a sum pooling aggregator.
Although powerful, they cannot fully utilize the maximum inner product search (MIPS).
In contrast, we employ a pre-interaction mechanism combined with a max pooler which is compatible with MIPS.

Multi-vector encoding is essential in these late-interaction models, but is also gradually borrowed to learn effective dense retrieval models.
<cit.> represent each document with its first $k$ token embeddings.
To learn multi-view document representations, <cit.> substitute the  token with $k$ special  tokens as view extractors and propose a local contrastive loss with annealing temperature between different views.
In comparison, our model learns diverse document representations through interactions with different queries.

§.§ Query Generation

Query generation (QG) is originally introduced to the IR community as a document expansion technique <cit.>.
<cit.> show that appending the T5-generated queries to the document before building the inverted index can bring substantial improvements over BM25.
More recently, <cit.> use generated queries as term expansion to learn better sparse representations for documents.

In the context of dense retrieval, query generation is usually used for domain adaptation in data scarcity scenarios.
For example, <cit.> use QG model trained on general domain to generate synthetic queries on target domain for model training.
To reduce noise in generated data, <cit.> further introduce a cross encoder for pseudo labeling.
Different from the previous work, we mainly use the generated queries to learn query-informed document representations.

§ METHOD

§.§ Task Definition

Given a query $q$ and a collection of $N$ documents $\mathcal{D}=\{d_1,d_2,...,d_i,...,d_N\}$, a retriever aims to find a set of $K$ relevant documents $\mathcal{D}_+=\{d_{i_1},d_{i_2},...,d_{i_j},...,d_{i_K}\}$,[Usually $K\ll N$.]
by ranking the document in the corpus $\mathcal{D}$ according to its relevance score with respect to the query $q$, for next stage re-ranking or downstream applications.

§.§ Dual Encoder

We first introduce the dual encoder (DE) architecture for dense retrieval.
In this framework, a query encoder $DE_q$ and a document encoder $DE_d$ are used to encode the query and document into low-dimensional vectors, respectively.
To measure their relevance, a lightweight dot product between the two vectors is usually adopted to enable fast search,

\begin{equation}
s(q,d)=DE_q(q)\cdot DE_d(d).
\end{equation}

The common design choice for the encoders is using multi-layer Transformers <cit.> initialized from pre-trained language models (PLMs), such as BERT <cit.>.
How to get the representation from BERT is also an interesting question but beyond the scope of this paper.
For simplicity, we directly take the  vector at the final layer as the text representation.
The two encoders can share or use separate parameters.
We tie the encoder parameters in main experiments but also provide results of untied parameters in ablation study.

§.§ Cross Encoder

The cross encoder (CE) takes the concatenation of query and document as input and uses deep neural network to model their deep interactions.
Given a pair of query and document consisting of multiple tokens, we feed their concatenation through a cross encoder to get the interaction-aware representation,

\begin{equation}
\mathbf{r}=CE(q+d).
\end{equation}

Then a multi-layer perceptron (MLP) is applied on top of the interaction-aware representation to predict the relevance score,

\begin{equation}
s(q,d) = MLP(\mathbf{r}).
\end{equation}

The cross encoder is also usually instanced as a multi-layer Transformer network initialized from BERT.
It can model term-level interactions between query and document, providing more fine-grained relevance estimation.

§.§ Dual Cross Encoder

We present our dual cross encoder where the document encoder acts as a cross encoder whereas the query encoder works like a dual encoder.
Specifically, the query representation and document representation with query interaction are calculated as

\begin{equation}\label{eq:qe}
\textbf{q}=DE_q(q),
\end{equation}

\begin{equation}\label{eq:qd}
\textbf{d}=CE_d(q+d).
\end{equation}

Their similarity is measured by a dot product like in the vanilla dual encoder,

\begin{equation}
s(q,d)=\textbf{q}\cdot \textbf{d}.
\end{equation}

Query Generation.
Note that the query from the query encoder side and document encoder side do not necessarily have to be the same since we only have access to the gold query for documents appearing in the training set.
It is impractical to manually write potential queries for each document in the whole corpus.
Hence, we use a T5 model <cit.> fine-tuned on the doc-to-query task to generate queries for each document.
We empirically adopt 10 queries decoded with top-$k$ sampling strategy <cit.> to encourage the query generation diversity.

The advantages of this architecture are two-fold.
On the one hand, we can model the query-document interaction in the pre-computed document representations.
On the other hand, we can enjoy the retrieval efficiency of the vanilla dual encoder by pre-computing the interaction-aware document representations.

§.§ Training

The conventional way to train a dense retriever requires a set of $(q,d_+,d_-)$ pairs.
The model is trained by optimizing the contrastive loss,

\begin{equation}
L(q,d_+,\mathcal{D}_-) = - \log\frac{e^{s(q,d_+)}}{\sum\limits_{d\in \{d_+\}\cup\mathcal{D}_-}e^{s(q,d)}},
\end{equation}

where $ \mathcal{D}_-$ contains a set of negative documents $d_-$ for query $q$.
Following <cit.>, we include both BM25 hard negatives and in-batch negatives in $ \mathcal{D}_-$.

Constructing Positives and Negatives.
Fusing query information into document representation requires redefining the positive and negative pairs.
For a given query $q$, our framework potentially permits four types of positive and negatives, namely, $(q_++d_+),(q_++d_-),(q_-+d_+)$ and $(q_-+d_-)$.
To train our model, we convert the traditional positive and negative pair from the training set into that in our framework with the mapping function

\begin{equation}
f: (q,d_+,d_-) \mapsto (q,q+d_+,q+d_-),
\end{equation}

where the $+$ is the concatenation operation used in cross encoder.
This mapping leads to the positive of type $(q_++d_+)$ and the following two types of negatives.
We leave the exploration of other types of negatives to future work.

Hard Negatives.
The negative documents $d_-$ are usually randomly sampled from BM25 top-ranked documents.
After the mapping function defined above, these negatives fall in the type of negative $(q_++d_-)$, which serve as hard negatives in our framework.
This type of negatives can teach the model to learn more fine-grained information, as $d_-$ is usually topically related to the gold query but cannot exactly answer the query.
It also prevents our model from learning the shortcut, i.e., only learning matching signals from the query side and ignoring the document side information.

In-Batch Negatives.
To improve the training efficiency, we also adopt in-batch negatives to train our model.
In our framework, the in-batch negatives belong to the negative type $(q_-+d_-)$.
This type of negatives is simple and can enable the model to learn topic-level discrimination ability.

Data Augmentation.
Regarding the generated queries as weakly annotated data, we can first pre-train our model on these noisy data as a warm-up stage and then fine-tune it on the human-annotated high-quality training set.

§.§ Inference

Index We encode the corpus following the same format as Equation <ref>, to get multi-view document representations with deep query interactions.

Denoting $\mathbf{d}_j^i$ as the $i$-th view of the $j$-th document $d_j\in\mathcal{D}$, we have

\begin{equation}
q_j^i\sim P_{QG}(q|d_j),
\end{equation}

\begin{equation}
\mathbf{d}_j^i = CE_d(q_j^i+d_j),
\end{equation}

where $P_{QG}(q|d)$ denotes the query generation model, $i\in\{1,...,k\}$ and $j\in\{1,...,N\}$.

Retrieval When a query comes, we encode it with the query encoder to get its contextualized representation $\mathbf{q}$ as in Equation <ref>.
We adopted multi-vector encodings for a document $d_j$, the relevance score between the query $q$ and the document $d_j$ is taken as the max pooling of its different views' scores,

\begin{equation}
s(q,d_j) = \max_i \mathbf{q}^T\mathbf{d}_j^i.
\end{equation}

This operation is compatible with MIPS for efficiency optimization,[Note that to get top-$K$ documents, we first retrieve $10K$ documents to ensure that we have at least $K$ documents after pooling.]

\begin{equation}
p = \arg \max_d \mathbf{q}^T \mathbf{d}.
\end{equation}

§ EXPERIMENT

In this section, we evaluate our model on different retrieval benchmarks and compare it with various baselines.

§.§ Datasets

We conduct experiments on the following retrieval benchmarks.

MS MARCO is a retrieval benchmark that originates from a machine reading comprehension dataset containing real user queries collected from Bing search and passages from web collection <cit.>.
We evaluate our model on the passage retrieval task.
The corpus contains about 8.8M passages.
The training set consists of about 500k annotated query-document pairs.
The dev set has 6980 annotated queries.
Since the test set is not publicly available, we evaluate on the dev set following previous work.

TREC Deep Learning (DL) tracks provide test sets with more elaborate annotations to evaluate the real capacity of ranking models.
We evaluate on the 2019 and 2020 test set <cit.>.
The 2019 test set contains 43 annotated queries and the 2020 test set contains 54 annotated queries.
Both of them share the same corpus with the MS MARCO passage retrieval benchmark.

§.§ Evaluation Metrics

Following previous work, we mainly evaluate the retrieval performance on MS MARCO passage retrieval benchmark with MRR@10 but also report the score of Recall@1000.
For datasets from TREC DL tracks, we evaluate with nDCG@10.

§.§ Baselines

We mainly compare our model against the DPR <cit.> baseline with a dual encoder architecture, but also report results of the following models most related to ours.

* BM25 <cit.> is the traditional lexical retriever.

* DocT5Query <cit.> appends generated queries to the document before building the inverted index of BM25.

* DeepImpact <cit.> learns sparse representation for documents using generated queries as expanded terms.

* ANCE <cit.> trains the DPR model with iterative hard negative mining strategy. We include this baseline since this technique is used in ME-BERT and DRPQ.

* ME-BERT <cit.> utilizes the first $k$ token embeddings as multi-vector encodings for documents and adopts max pooling for score aggregation.

* DRPQ <cit.> improves over ME-BERT by performing a $k$-means over all tokens' embeddings and utilizing a attention-based score aggregator.

* ColBERT <cit.> represents query and document at token-level and uses a MaxSim pooler followed by a sum aggregator to calculate the relevance score.

§.§ Implementation

We implement our model based on the  toolkit <cit.>.
For a fair comparison with our model, we re-implement the DPR baseline using the same set of hyperparameters.

We train all the models on 8 NVIDIA Telsa V100 GPUs with 32GB memory.
We initialize all the encoders with .
The max sequence length is 16 for query and 128 for passage.
The number of positive and negative passages follows a ratio of 1:7 for each sample.
We set the batch size to 32.
We use both officially provided BM25 negatives and in-batch negatives to train the models.
We use Adam optimizer with the learning rate of $5\times 10^{-6}$, linear decay with 10% warmup steps.

In the preliminary study without data augmentation, we train both models for 10 epochs.
To make full use of generated queries, we first pre-train the models for 10 epochs on the corpus with a batch size of 256 and only in-batch negatives, and then fine-tune the models for 20 epochs till convergence on the training set.
We haven't tuned other hyperparameters.
The pre-training stage takes about 15 hours and the fine-tuning stage takes about 8 hours.

During inference, we use  of the  library <cit.> to index the corpus and perform an exact search.

§.§ Results

Table <ref> illustrates the evaluation results of our model and the baselines.

2*Model     2*PLM     2c|MS MARCO     TREC DL 19     TREC DL 20

MRR@10     Recall@1k     nDCG@10     nDCG@10

BM25     -     18.4     85.3     50.6     48.0

DocT5Query     -     27.7     94.7     64.8     61.9

DeepImpact     BERT$_{base}$     32.6     94.8     69.5     65.1

DPR     BERT$_{base}$     31.4     95.3     59.0     62.1

ANCE     RoBERTa$_{base}$     33.0     95.9     64.8     -

ME-BERT     BERT$_{large}$     33.4     -     68.7     -

DRPQ     BERT$_{base}$     34.5     96.4     -     -

ColBERT     BERT$_{base}$     36.0     96.8     69.4     67.6

Ours     BERT$_{base}$     36.0     96.4     68.3     68.9

Evaluation results on MS MARCO passage retrieval benchmark and TREC DL track.
DocT5Query and DeepImpact can be seen as the sparse counterparts of our model.
Both ME-BERT and DRPQ learn multi-vector encodings for documents, and have used the hard negative mining technique proposed in ANCE.
ColBERT learns term-level representations of both query and document for late interaction.
Results not available are marked as `-'.

We first compare our model against the DPR dual encoder baseline.
We can observe substantial improvements in terms of MRR@10 and nDCG@10 across all these datasets, which demonstrate the effectiveness of our approach.
The Recall@1k also exhibits a slight improvement.

Our approach is also competitive with other baselines.
On MS MARCO, it surpasses other baselines and is comparable to ColBERT, while being more efficient.
On TREC DL 19, the results are comparable to ME-BERT, which used a more powerful large-size model as backbone and the hard negative mining technique of ANCE.
On TREC DL 20, our model even outperforms the ColBERT model.

§.§ Ablation Study

We conduct ablation studies on our model design choice.

§.§.§ Effect of Data Augmentation

We used the generated queries as data augmentation for pre-training.
We ablate on the effect of pre-training in this section.
The results of different training stages on MS MARCO dev set are shown in Table <ref>.

MRR@10     Pretrain     Finetune     Full

DPR     25.6     31.4     34.2

Ours     26.1     33.2     36.0

Ablation of different training stages on MS MARCO dev set. Pretrain: only use generated data for training; Finetune: only use data from training set for training; Full: Pretrain + Finetune.

We can see that using generated data for pre-training gives a MRR@10 score comparable to DocT5Query but lower than directly fine-tuning using data from the training set.
The top-$k$ sampling decoding strategy in query generation may introduce some noise, which explains why the pre-training underperforms directly fine-tuning with high-quality data.
However, the pre-training stage is still beneficial for the fine-tuning stage.

The results on TREC DL track are shown in Table <ref>.
Our model still consistently outperforms the dual encoder baseline under different settings.
The improvements are more significant on this benchmark since the annotation is more complete.
Notably, our model without data augmentation is comparable to the DPR baseline with data augmentation on this benchmark.

nDCG@10     DL 19     DL 20

3cw/o Data Augmentation

DPR     59.0     62.1

Ours     63.0     67.6

3cw/ Data Augmentation

DPR     63.1     66.5

Ours     68.3     68.9

Results on TREC DL track under different settings.

§.§.§ Effect of Sharing Parameters

Sharing the encoder parameters can reduce the number of model parameters to half.
We tie our encoder parameters in main experiments but also provide ablation of untied encoder parameters in Table <ref> to study its effect.

MRR@10     tie     untie

DPR     31.4     31.7

Ours     33.2     33.8

Results of tie / untie encoder parameters on MS MARCO dev set.

We can observe that using two sets of encoder parameters gives slightly better performance but not so significantly.
Using separate encoders brings more improvements to our model, which is normal since the nature of two encoders in our model is more asymmetric than that in the vanilla dual encoder.

§ ANALYSIS

Our experimental results in the previous section demonstrate that it is indeed beneficial to incorporate query interactions into the document representations.
The generated queries are crucial to the success of our model.
In this section, we analyse the influence of query quality and query diversity to the retrieval performance.

§.§ On the Query Quality

The number of queries is an important factor in our framework.
Too few queries have low diversity while too many queries will sacrifice efficiency.
Thus we provide an analysis here to study its effect.
We evaluate the query generation performance on the dev set of MS MARCO and reveal its relation with the retrieval performance.

To measure the generation quality, we calculate the maximum ROUGE-L score between generated queries and the gold query on the dev set.
For retrieval performance, we report the MRR@10.
Figure <ref> illustrates the evolution of the two metrics with different number of queries.[
Please refer to Table <ref> in Appendix <ref> for exact numbers.

xmin = 1, xmax = 10,
ymin = 40, ymax = 80,
axis y line*=right,
xlabel = Number of queries $k$,
ylabel = Rouge-L$_{max}$,
ylabel near ticks,
minor tick num = 1,
major grid style = lightgray,
minor grid style = lightgray!25,
width = 0.4,
legend cell align = left,
legend pos = south east,

[blue, mark = *] table [x = k, y = rouge-l] ;

xmin = 1, xmax = 10,
ymin = 20, ymax = 35,
hide x axis,
axis y line*=left,
ylabel = MRR@10,
ylabel near ticks,
minor tick num = 1,
major grid style = lightgray,
minor grid style = lightgray!25,
width = 0.4,
legend cell align = left,
legend pos = north west,
[green, mark = triangle*] table [x = k, y = mrr] ;

The evolution of ROUGE-L and MRR@10 on MS MARCO dev set when varying number of queries from 1 to 10.

xmin = 40, xmax = 70,
ymin = 25, ymax = 35,
xlabel = Rouge-L$_{max}$,
ylabel = MRR@10,
width = 0.4,
xtick distance = 5,
ytick distance = 5,
grid = both,
minor tick num = 5,
major grid style = lightgray,
minor grid style = lightgray!25,
legend cell align = left,
legend pos = north west,
[scatter, only marks, mark = x, mark size = 3pt] table[x=rouge-l, y=mrr] rouge-mrr.dat;
[thick, black!30] table[
x = rouge-l,
y = create col/linear regression = y = mrr
] rouge-mrr.dat;

R = 0.9958,

Correlation between generation and retrieval performance on MS MARCO dev set.

We can see that as the number of queries grows, the retrieval performance becomes better because of the improved generation quality.
The correlation between the two metrics is shown in Figure <ref>.
The Pearson coefficient is 0.9958, indicating a strong positive correlation.
Keep increasing the number of queries will consistently improve the retrieval performance but more marginally.

§.§ On the Query Diversity

Intuitively, more diverse queries can potentially hit more types of queries.
We used top-$k$ sampling strategy to encourage the query generation diversity.
However, whether and to what extent the generated queries are diverse remains unclear.
To this end, we adopt the self-BLEU <cit.> to measure the query generation diversity for a document.

Diversity distribution.

Retrieval v.s. Diversity.

Quality v.s. Diversity.

Statistics of query diversity on MS MARCO dev set. We divided the diversity into 5 levels based on an average division of the self-BLEU-4 score.

We partition the documents of MS MARCO dev set to subsets of different query diversity according to their self-BLEU-4 score and measure the retrieval performance on these subsets.
The statistics are shown in Figure <ref>.
First, we observe that most of the documents have high query generation diversity thanks to the top-$k$ sampling strategy (see Figure <ref>).
Second, the retrieval performance drops with higher diversity (see Figure <ref>).
One possible reason is that the QG model will generate more diverse queries when it doesn't know the right one.
As such, higher diversity indicates lower quality (see Figure <ref>) and the retrieval performance drops with lower generation quality (see Figure <ref>).
It would be desirable to design a diversity metric that takes into account the generation quality.

§.§ Case Study

Query     how old is canada

Ours Rank 1     Canada was finally established as a country in 1867. It is 148 years old as of July 1 2015. Canada has been a country for 147 years. The first attempt at colonization occurred in 1000 A.D. by the Norsemen. There was no further European exploration until 1497 A.D. when the Italian sailor John Cabot came along. It then started being inhabited by more Europeans.

Generated Queries     l

when was canada established

when was canada discovered

what year was canada founded

how long has canada been a country

how old is canada

DPR Rank 1     it depends where you live but in Canada you have to be at least 16 years old.

Generated Queries     l

what is the legal age to be in canada

how old do you have to be to live in canada

how old do you have to be to enter canada as a citizen

at what age can i go to canada to study in canada

what is the minimum age to join the military

Case Study on MS MARCO dev set.

We conduct a case study on the dev set to intuitively compare our model and the dual encoder baseline, as well as to illustrate the QG performance.

Table <ref> shows an example drawn from the MS MARCO dev set.
Our model can retrieve the correct passage by generating the right query.
DPR retrieves a hard negative passage where the content is corresponding to the query keywords but can not correctly answer the query.
By generating queries, our model can better distinguish the difference among document meanings.

§ DISCUSSION

The ranking task is usually approached with a two-stage pipeline: retrieve-then-rerank.
The two stages usually use different architectures due to the effectiveness and efficiency trade-off.
Dual encoder is more efficient for retrieval, while cross encoder is more powerful for reranking.
How to take advantage of each other's strengths for mutual improvements is a hot topic of research.
We propose a new dual cross encoder architecture to benefit from both with a pre-interaction mechanism.

One limitation of our framework is that there exists a discrepancy between training and inference.
We used the gold query to train the model but do not have access to the gold query during inference.
Generating more queries would bridge this gap, but at the cost of efficiency.
We wish to close this gap with improved training strategy or improved query generation quality in the future.

§ CONCLUSION

We proposed a novel dense retrieval model to bridge the gap between dual encoder and cross encoder.
In our framework, the document representations are obtained by pre-interacting with a set of generated pseudo-queries through a cross encoder.
Our approach enables multi-view document representation with deep query interaction while maintaining the inference efficiency of the dual encoder approach.
We demonstrated its effectiveness compared to dual encoder baseline via experiments on various retrieval benchmarks.
In the future work, we would like to explore how to better incorporate generated queries for model training and how to improve the query generation quality for better retrieval performance.

§ APPENDIX

$k$     ROUGE-L     MRR@10

1     42.49     27.74

2     50.93     30.09

3     55.67     31.15

4     58.45     31.66

5     60.63     31.92

6     62.28     32.38

7     63.57     32.67

8     64.62     32.88

9     65.46     32.96

10     66.22     33.23

Results of generation and retrieval performance on MS MARCO dev set when varying number of queries (correspond to Figure <ref> and Figure <ref>).


We investigate the spatial correlations of microscopic stresses in soft
particulate gels, using 2D and 3D numerical simulations. We use a recently
developed theoretical framework predicting the analytical form of stress-
stress correlations in amorphous assemblies of athermal grains that acquire
rigidity under an external load. These correlations exhibit a pinch-point
singularity in Fourier space leading to long-range correlations and strong
anisotropy in real space, which are at the origin of force-chains in granular
solids. Our analysis for the model particulate gels at low particle volume
fractions demonstrates that stress-stress correlations in these soft materials
have characteristics very similar to those in granular solids and can be used
to identify force chains. We show that the stress-stress correlations can
distinguish floppy from rigid gel networks and the intensity patterns reflect
changes in shear moduli and network topology, due to the emergence of rigid
structures during solidification.

# Stress-stress Correlations Reveal Force Chains in Gels

H. A. Vinutha  Fabiola Diaz Ruiz  Department of Physics, Institute for Soft
Matter Synthesis and Metrology, Georgetown University, Washington, DC, USA
Xiaoming Mao  Department of Physics, University of Michigan, Ann Arbor MI, USA
Bulbul Chakraborty  Martin Fisher School of Physics, Brandeis University,
Waltham MA, USA  Emanuela Del Gado  Department of Physics, Institute for Soft
Matter Synthesis and Metrology, Georgetown University, Washington, DC, USA

##  I Introduction

Soft particulate gels consist of particulate matter (polymers, colloids,
proteins etc) aggregated in a solid matrix, which is embedded in a fluid and
typically sparse and porous Del Gado  _et al._ (2016). Gels of this type are
found in the body tissues, in food, drugs, personal care products, and even in
cement used in concrete based construction Petekidis and Wagner (2021);
Ioannidou  _et al._ (2016). Their structures can be very soft and re-
configurable and their stress response is determined by a complex interplay
between molecular cohesion or surface interactions, microstructural
reorganization and external driving. As a consequence, understanding,
predicting and designing the mechanics and rheology of these materials remains
very challenging.

While particulate gels can be extremely soft because of the overall low solid
content, the strength of the interactions that drive the aggregation of the
colloidal units into the gel network must be large enough with respect to
$k_{B}T$ to overcome thermal fluctuations and stabilize the gel structural
elements and their connections Anderson and Lekkerkerker (2002); Royall  _et
al._ (2021). The implication is that the elasticity is largely enthalpic in
nature in these gels, contrarily to polymer networks, and that the interaction
strength, together with the total volume fraction of the solid content,
controls elastic moduli, viscoelastic spectra, and nonlinear response.
Remarkably, the interaction strength alone does not allow to predict the gel
properties: the morphology of the gel networks, the structural elements and
their connectivity, may change a lot due to different kinetics at play during
the gel self-assembly, leading to huge variations of the mechanical response,
sometimes even for identical compositions of the initial particle solution.

The solidification processes through which particulate gels form, initiated,
for example, via irreversible fractal aggregation or equilibrium phase
separation, are typically sources of frozen-in stresses Bouzid  _et al._
(2017) and help build a memory in their history dependent response Zhang  _et
al._ (2021): they determine how local stresses and mechanical heterogeneities
get embedded and remain frozen-in, in the gel structure, during gelation. The
rigidity attained ultimately depends on the geometry of the microstructure
that self-assembles during solidification to accommodate the local and global
constraints of mechanical equilibrium. Consequently, significant structural
correlations, induced for example by attractive interactions, are naturally
incorporated in the way stresses are transmitted through the rigid backbone of
the final solid. This process of self-organization can provide an explanation
for the possible emergence of rigidity at very low densities in colloidal
gels, since the structural correlations are precisely those that satisfy the
constraints of mechanical equilibrium, as indeed shown in Ref.Zhang  _et al._
(2019). The fact that this self-organization occurs in overall amorphous and
spatially heterogeneous structures also naturally leads to stress localization
which manifests itself clearly under deformation Leocmach  _et al._ (2014);
Colombo and Del Gado (2014a); Filiberti, Piazza, and Buzzaccaro (2019); Aime,
Ramos, and Cipelletti (2018).

The tendency to localize strain and stresses and the signature of mechanical
heterogeneities, shown under deformation, may suggest that stress transmission
could be very anisotropic and strongly localized even in the material at rest
Alexander (1998), just as a result of the microstructural development during
gelation. As a consequence, phenomena such as force chains in granular solid,
where forces are transmitted across the sample along lines of grains forming a
sparse network Cates  _et al._ (1998); Radjai  _et al._ (1998); Gendelman  _et
al._ (2016); Vinutha and Sastry (2019); Wang, Wang, and Zhang (2020), may be
relevant to the physics of soft particulate gels as well. Detecting force
chains and identifying the stress bearing part of gel structures is, however,
incredibly difficult in experiments and elusive even in numerical simulations
of model materials, since it requires disentangling mechanical heterogeneities
from structural heterogeneities and relies upon threshold values for local
stresses or forces which are not easily justified. For amorphous solids,
recent theoretical studies have clarified that just the constraints of
mechanical equilibrium and of structural isotropy guarantee that the spatial
correlations of the stresses are long-ranged Henkes and Chakraborty (2009);
Mao  _et al._ (2009); DeGiuli (2018); Lemaître (2018). The emergence of
elasticity can then be obtained, even in granular solids where the grain
contacts cannot be regarded as elastic springs, within a new theoretical
framework that can be mapped onto a tensorial electromagnetism with vector
charges (VCT) Pretko (2017); Nampoothiri  _et al._ (2020, 2022). In
particular, this VCT framework provides analytical predictions for the stress-
stress correlations, demonstrating that they are not only long-ranged but also
anisotropic, and clearly identifying the presence of force chains in
disordered assemblies of grains which are jammed under an external load.

Here we use the predictions of the VCT framework to compute stress-stress
correlations in model particulate gels, obtained in 2D and 3D numerical
simulations. We find that the intensity patterns of these correlations in
Fourier and real space share several common features with those detected in
experiment and simulations of granular solids, and predicted by the VCT
framework Nampoothiri  _et al._ (2020, 2022). In particular, we recover the
anisotropy of the stress-stress correlations, which therefore becomes an
indicator of the presence of force chains in particulate gels. Moreover, we
show that the stress-stress correlations distinguish floppy from rigid gels
and are sensitive to the distance of the model gels from the rigidity
threshold. These results pave the way to develop further understanding of
stress transmission in soft materials and provide the first basis to develop a
VCT framework for soft gel materials.

The paper is organized as follows. In section II.A we briefly review the basic
ingredients of the VCT framework and its predictions for the stress-stress
correlations in granular solids, while section II.B contains the information
on the numerical models and simulations used, and in section II.C the
calculations of the stress correlations are described. We then discuss the
results obtained in section III and provide a summary and outlook in section
IV.

##  II Theory predictions and numerical simulations

###  II.1 Elasticity theory of athermal amorphous solids and VCT framework.

At the heart of the mechanical response of jammed amorphous solids is the
athermal nature of the force-bearing networks that obey the constraints of
mechanical equilibrium, implemented locally: each particle satisfies the
constraints of force and torque balance. In a continuum formulation the force-
balance constraints are expressed as :

$\partial_{i}\sigma_{ij}=f_{j}$ (1)

and torque balance leads to the symmetry of the stress tensor. However, in $D$
dimensions, $d$ conditions of mechanical equilibrium are not enough to solve
for the $D(D+1)/2$ components of the stress tensor. Canonical continuum
elasticity solves for the stress components by defining the strain in terms of
the displacement field from the crystal reference structure, which is uniquely
defined, and uses the constitutive relation of linear elasticity to obtain a
complete set of equations Timoshenko and Goodier (1951); Landau  _et al._
(2012). In the case of amorphous solids, there is no unique reference
structure to define strain and the continuum elasticity description does not
work. The VCT framework described in Ref. Nampoothiri  _et al._ (2020)
provides a stress-only framework to describe elasticity in such solids. This
stress-only theory of elasticity is defined by equations that bear a
remarkable similarity to that of classical elasticity theoryNampoothiri  _et
al._ (2022):

$\displaystyle\partial_{i}\sigma_{ij}$ $\displaystyle=f_{j}^{\rm~{}external},$
$\displaystyle
E_{ij}=\frac{1}{2}(\partial_{i}\varphi_{j}+\partial_{j}\varphi_{i})$
$\displaystyle\implies\epsilon_{iab}\epsilon_{jcd}\partial_{a}\partial_{c}E_{bd}=0,$
$\displaystyle\sigma_{ij}=(\delta_{ijkl}+\chi_{ijkl})$ $\displaystyle
E_{kl}\equiv{\Lambda^{-1}_{ijkl}E_{kl}.}$ (2)

Here, $\hat{\sigma}$ is the stress-tensor field, which is related to $\hat{E}$
via an emergent elastic modulus tensor, $\hat{\Lambda}^{-1}$. $\hat{E}$ plays
a role analogous to the strain tensor in classical elasticity theory, and
external, body forces such as gravity are represented by $\vec{f}$. The two
crucial differences from classical elasticity theory are that (a) the physical
displacement field defining strain is replaced by the $\varphi$ field, which
is a gauge potential since there is no unique reference structure, and (b) the
$\Lambda^{-}1$ tensor is an emergent elastic modulus tensor that reflects the
coarse-grained properties of the self-assembled, force-bearing network and
specifically the force-dipoles that represent the frustration in the
interactions between particles.

The VCT framework makes explicit predictions about stress-stress correlations
and stress response in terms of the emergent elastic modulus tensor,
$\Lambda^{-1}$. Here we summarize crucial features of the stress-stress
correlations. [i] A hallmark of the theory is the appearance of a
characteristic pinch-point in the stress-stress correlations,
$C_{ijkl}(q)=\langle\sigma_{ij}(q)\sigma_{kl}(-q)\rangle$. These predictions
have been tested against experimental measurements in frictional granular
materials, and in 2$D$ and in 3$D$ model assemblies of frictionless soft
grains. [ii] The chain-like structures, commonly referred to as force-chains,
are a visual representation of the the highly anisotropic nature of
$C_{ijkl}$. [iii] The $q-$space correlations predicted by the theory, and
observed in experiments imply a power-law decay at large lengthscales:
$\langle\sigma_{ij}(r)\sigma_{ij}(0)\rangle\propto 1/r^{D}$, where the plus
sign appears for longitudinal correlations and the minus sign for transverse
correlations, and $D$ is the spatial dimension. In this paper we analyze
stress-correlations in numerically simulated 2$D$ and 3$D$ particle gels,
using the VCT framework.

###  II.2 Simulation Details

Gel configurations in two and three dimensions are obtained using models of
$N$ interacting colloidal particles of diameter $d$ that undergo gelation as
described in Refs.Colombo and Del Gado (2014b, a); Bouzid and Del Gado (2018);
Bantawa  _et al._ (2021); Zhang  _et al._ (2019). We perform Molecular
Dynamics (MD) simulations in a cubic (square in 2D) box of size $L$ with
periodic boundary conditions. We first reach thermal equilibrium at high
temperature and then slowly quench the particle configurations to different
target temperatures $T$, i.e., with different cooling rates $C_{r}$, using a
Nosé–Hoover thermostat and allowing the particles to aggregate and form gel
networks Noro and Frenkel (2000); Colombo and Del Gado (2014b). The mechanical
equilibrium in the final gel states is obtained by slowly drawing all kinetic
energies with an overdamped dynamics, following the protocol described in
Refs. Zhang  _et al._ (2019); Colombo and Del Gado (2014a); Bantawa  _et al._
(2021). All the simulations are performed using LAMMPS Plimpton (1995)
modified to include specific particle interactions. We define an approximate
volume fraction $\phi=\frac{\pi d^{2}N}{4L^{2}}$ and $\phi=\frac{\pi
d^{3}N}{6L^{3}}$ in two and three dimensions, respectively.

For 2$D$ gels, we consider $N=10000$ monodisperse particles, interacting via a
short range attractive potential
$U(r)=A\epsilon\left(a\left(d/r\right)^{18}-\left(d/r\right)^{16}\right)$
where $r$ is the interparticle distance, $A$ and $a$ are dimensionless
constants. The values of $A=6.27$ and $a=0.85$ are chosen to obtain a short-
ranged attractive well of depth $\epsilon$ and range $r_{c}\approx 0.3d$ (the
potential is cut and shifted to 0 at large distances). In 3D, we consider
$N\approx 16000$ monodisperse particles that interact via the same short range
attractive interactions and with an angular term that introduces bending
rigidity to inter-particle bonds Colombo and Del Gado (2014a); Bouzid and Del
Gado (2018); Bantawa  _et al._ (2021).

All the simulations quantities described in the following are expressed in the
reduced units: $d$ for the unit of length, the unit of energy is $\epsilon$,
the particle mass $m$ is the unit of mass. The reduced temperature is
expressed in units $\epsilon/k_{B}$, where $k_{B}$ is the Boltzmann’s
constant. The unit of stress is therefore $\epsilon/d^{3}$. When performing
the quenches into aggregation from high temperature, the cooling rate
$C_{\text{r}}$ is defined as $\Delta T/\Delta t$, where $\Delta T$ is the
distance between the initial and target temperatures in the gelation protocol
described above and $\Delta t$ is the duration of the quench, with the unit
time $\tau_{0}$ being $d\sqrt{m/\epsilon}$. In 2D, we show data for $\phi=0.5$
at $T=0.3$ (floppy gels) and $T=0.18$ (rigid gels). In 3D, we analyze gels
obtained using the cooling rates $C_{r}=9\times
10^{-5}\epsilon/(\tau_{0}k_{B})$, and $9\times
10^{-2}\epsilon/(\tau_{0}k_{B})$ at $\phi=0.125,0.075$ and $0.05$.

To infer rigidity, we use the pebble game algorithm on the 2D gel networks to
identify rigid clusters and floppy regions Jacobs and Hendrickson (1997);
Zhang  _et al._ (2019): a rigid gel is characterized by a spanning rigid
cluster. In 3D, we have instead computed the linear viscoelastic response of
the gels and obtained their low frequency moduli Colombo and Del Gado (2014b);
Bouzid and Del Gado (2018); Bouzid  _et al._ (2018). For rigid gels, the low
frequency storage modulus is larger than the loss component.

All the data for the stress correlations have been averaged over $10$
independently generated samples.

###  II.3 Computing stress correlation functions

To compute the stress correlations in the gel samples, we coarse-grain the
stress fluctuations in Fourier space by imposing a cutoff at large $q$,
corresponding to $q_{max}=2\pi/d$, i.e. we do not consider any stress
fluctuations occurring at length scales shorter than $d$. The lower $q$ cutoff
in Fourier space is set by the simulation box size $q_{min}=2\pi/L$ and the
periodic boundary conditions. For every particle $i$ we compute the force-
moment tensor defined as

$\hat{\sigma}_{i}=\sum_{j=1}^{N_{c}}\vec{r}_{ij}\otimes\vec{f}_{ij},$ (3)

where $\vec{r}_{ij}$ is the vector connecting the center of particles $i$ and
$j$, and $\vec{f}_{ij}$ is the force between them. $N_{c}$ is the number of
neighbors $j$ of particle $i$ within the interaction range $r_{ij}\leq r_{c}$.
The force-moment tensor in Fourier space is given by

$\hat{\sigma}(\vec{q})=\sum_{i=1}^{N}\hat{\sigma}_{i}\exp(i\vec{q}\cdot\vec{r}_{i})$
(4)

To obtain the stress tensor in Fourier space we need to divide Eq. 4 by the
volume (area) $V$ of the simulation box, however, for simplicity, we ignore
the constant factor of $1/V$ in our calculations. The stress correlations in
Fourier space is computed using:

$\displaystyle
C_{klmn}(\vec{q})=\langle\Delta\sigma_{kl}(\vec{q})\Delta\sigma_{mn}(-\vec{q})\rangle$
(5)
$\displaystyle\Delta\sigma_{kl}(\vec{q})=\sum_{i=1}^{N}(\sigma_{kl,i}-\langle\sigma_{kl}\rangle)\exp(i\vec{q}\cdot\vec{r}_{i})$
(6)

where $\langle\sigma_{kl}\rangle$ is the average value of the stress
component. The angular brackets $\langle...\rangle$ denotes an ensemble
average performed over all the particles and the average over all
statistically independent samples prepared in the same conditions. For 3$D$
gels, the computation of $\hat{\sigma}_{i}$ also includes the contribution
from the three body or angular term of the interaction potential Thompson,
Plimpton, and Mattson (2009).

From the stress correlation calculations, we construct maps of the correlation
intensity in Fourier space in polar $(q,\theta)$ representation for 2$D$ gels,
for a range of $q$ from $2\pi/L$ to $56\pi/L$. For the angular variation of
the correlations, along with the radial averaging, the data are also averaged
over a small angular bin width of $\approx 6^{\circ}$ to reduce noise. For
comparison, we also compute the correlations in real space along both $x$ and
$y$ direction with the coarse-graining length $d$. In 3D, the range of $q$
used is from $2\pi/L$ to $42\pi/L$, the stress correlations are averaged over
$|q|$ and projected in 2D using the Hammer projection coordinate system
$(H_{x},H_{y})$Nampoothiri  _et al._ (2020, 2022). $H_{x}$ and $H_{y}$ are
computed using

$H_{x}=\frac{2\sqrt{2}\cos{\alpha}\sin{\beta/2}}{\sqrt{1+\cos{\alpha}\cos{\beta/2}}}$
$H_{y}=\frac{\sqrt{2}\sin{\alpha}}{\sqrt{1+\cos{\alpha}\cos{\beta/2}}}$

where $\alpha$ and $\beta$ are the latitude and longitude, respectively.
$\alpha=\theta-\pi/2$ and $\beta=\varphi$, where $(\theta,\varphi)$ are the
spherical polar coordinate angles.

All stress correlations are scaled by the maximum value of the $C_{xxxx}$
correlation function.

Figure 1:  Snapshots of 2D gel showing $xx$ and $xy$ components of the
particle stresses for a rigid (top row) and floppy gel (bottom row). Figure 2:
Stress correlations $C_{xxxx}$ and $C_{xyxy}$ in Fourier space for the rigid
and floppy gel in the top and bottom row, respectively. Figure 3:  Stress
correlations $C_{yyyy}$ (top row) and $C_{xxyy}$ (bottom row) in $q$\- space
(left panel) and real space (right panel). In the middle panel the angular
dependence of the $q$\- space correlations is shown.  Figure 4:  (Top panel)
Snapshot of the gel network at $\phi=0.125$, only the bonds are shown. The
particle bonds are red if the particle stress is positive and blue if the
particle stress is negative. The bond width is proportional to the magnitude
of the stress. (Bottom panel) Hammer projection of $C_{xxxx}$ in Fourier space
of the 3D gels at $\phi=0.125$. Figure 5:  Comparison of $C_{zzzz}$ of gels at
$\phi=0.125$ (top figure) with the isotropic jammed solids (bottom figure).
The jammed solids data is from Ref. Nampoothiri  _et al._ (2022) Figure 6:
Hammer projection of $C_{xzzz}$ for the gel configurations at packing fraction
$0.125$ (left), $0.075$ (middle) and $0.05$ (right). The gel configurations
are prepared with cooling rate $C_{r}=9\times
10^{-5}\epsilon/(\tau_{0}k_{B})$.

##  III Results

We now discuss the result obtained for 2$D$ and 3$D$ model particulate gels in
mechanical equilibrium. As discussed in the Introduction, soft particulate
gels have strong structural heterogeneities which depend on the aggregation
kinetics and the path to gelation. These structural heterogeneities are
associated to stress heterogeneities, however the microstructural origin of
the stress heterogeneities is not easily identified. Fig. 1 shows snapshots of
a rigid gel close to the rigidity threshold Zhang  _et al._ (2019) (top
panels) and a floppy gel (bottom panel) from our $2D$ simulations. The
particles are colored according to their magnitude of $xx$ and $xy$ component
of $\hat{\sigma}_{i}$ computed using Eq. 3. For these numerical samples, these
components of the stress tensor have a magnitude varying between
$-10\epsilon/d^{3}$ and $8\epsilon/d^{3}$. If we use the whole range for
$\sigma_{xx}$ and $\sigma_{xy}$ in the color maps the gels appear mechanically
homogeneous, as they are overall soft. By zooming in the stress variation,
down to $\Delta\sigma_{ij}\simeq 1\epsilon/d^{3}$, chain-like patterns emerge
in the stress maps of rigid gels. Floppy gels feature relatively large stress
fluctuations, even at this large stress magnification, which however appear
randomly distributed and uncorrelated. For rigid gels, instead, the stress
fluctuations display an anisotropic pattern, which strongly suggests
anisotropic and localized stress correlation across the samples and is
reminiscent of force chains in granular packings Cates  _et al._ (1998);
Radjai  _et al._ (1998); Gendelman  _et al._ (2016); Behringer and Chakraborty
(2018); Vinutha and Sastry (2019); Wang, Wang, and Zhang (2020).

These qualitative observations can be quantitatively confirmed by analyzing
the intensity maps of the stress correlations, for the corresponding component
of the stress tensor, as shown in Figure 2 (left). While for floppy gels the
stress-stress correlation maps do not feature any distinctive pattern, for
rigid gels they do. The patterns indicate both the presence of correlations
that span the whole system, and the fact that they do so in a very directional
and anisotropic fashion, in spite of the particle interactions and the overall
gels being isotropic. The $C_{xxxx}$ map obtained from the rigid gels shows a
large $q$ cutoff in the correlations, which simply reflects the fact that well
inside the gel branches the material is spatially and mechanically
homogeneous. However the correlation intensity is significant (along the
$q_{x}$ axis) even for the smallest $q$, i.e., for distances up to the whole
system size. The value of $C_{xxxx}$ being instead zero along the line
$q_{y}=0$ indicates that the forces can propagate approximately only along
their own direction and very unlikely perpendicular to it. This is a direct
consequence of force balance, as discussed in Section II.1 and provides a
quantitative evidence to the presence of force chains. The same
characteristics, i.e., the long range correlations and the strong anisotropy
indicated by the angular dependence, are also present in the four-lobe pattern
of the $C_{xyxy}$ map for the rigid gels (Fig. 2 right). We note that for
$C_{xyxy}$ the signal at low $q$ is much weaker, probably due to system size
effects and the proximity to the rigidity threshold of the rigid gels
considered here.

The two-lobe pattern in $C_{xxxx}$ and four-lobe pattern in $C_{xyxy}$ for the
rigid gels are identical to the correlation patterns already obtained in
experiments and simulations of granular solids, confirming the pinch-point
structure of stress correlations in Fourier space predicted by the elasticity
theory of the VCT framework Nampoothiri  _et al._ (2020). In Fig. 3, we also
show the $C_{yyyy}$ (top) and $C_{xxyy}$ (bottom) maps for the 2D rigid gels
in Fourier space (left) and the corresponding radially averaged angular plots
(center). The stress correlations maps in the real space (right) complement,
and confirm, the insight gained with the maps of the correlations in Fourier
space. The remaining stress correlations for 2D gels, considering all stress
tensor components, are shown in Appendix A. These maps quantitatively
establish the similarity of stress transmission in soft gels and in granular
solids. They support the idea that the rigidity and elasticity of soft
particulate gels, when particle interactions are sufficiently large with
respect to $k_{B}T$, can be fundamentally understood as an emerging property
which is the result of the local and global constraints imposed by mechanical
equilibrium, rather than just a consequence of their microscopic interactions.
We note that, as briefly discussed in Section II A (see eq.2), in the VCT
framework the angular dependence of the correlation intensity (Fig. 3 center)
provides direct information on the properties of the emerging elastic tensor
$\Lambda$. When comparing our data to the results in Ref.Nampoothiri  _et al._
(2020), we notice interesting quantitative differences in the corresponding
angular plots, therefore suggesting that the obvious significant differences
between the overall elastic behavior of gels and granular solids, can also be
further investigated with this approach.

Figure 7:  Hammer projection of $C_{xzzz}$ for the gel configurations at
$\phi=0.125$ and $C_{r}=9\times 10^{-2}\epsilon/(\tau_{0}k_{B})$.

In the case of 3$D$ gels, we have computed all $21$ stress-stress
correlations. We discuss some of them here in detail, while the remaining ones
are shown in Appendix B. A snapshot of one of the gel networks (at volume
fraction $\phi=0.125$ and obtained with cooling rate $C_{r}=9\times
10^{-5}\epsilon/(\tau_{0}k_{B})$ is shown in Fig. 4 (top panel). We have
colored the particle bonds in red if the $xx$ component of the particle stress
is positive (tension) and blue if negative (compression), to highlight how,
even with this information, the presence of force chains or stress
localization can not be easily detected. However, the maps of the
corresponding stress correlation function clearly reveals their presence: also
in this case all stress-stress correlations are long-ranged, as determined by
the mechanical equilibrium constraints Henkes and Chakraborty (2009); Lemaître
(2018), and the $2D$ Hammer projection of the correlation intensity for the
fluctuations of the $xx$ component of the stress tensor (Fig. 4, bottom panel)
are strongly anisotropic, with the angular pattern indicating that stress
transmit across the sample along specific and localized directions.

In Figure 5, we compare the intensity pattern of $C_{zzzz}$ in rigid gels
(top) and granular solids (bottom, from Ref. Nampoothiri  _et al._ (2022)) to
point out the striking difference in the angular dependence. The map obtained
for a model granular solid shows that the stress-stress correlations are
basically independent on the azimuthal angle $\phi$ and only depend on
$\theta$, whereas the pattern for the gels indicates a strong dependence on
both $\theta$ and $\phi$, confirming that this approach is able not just to
reveal similarities and common traits, but also to identify differences. With
this respect, we think that this striking difference in the angular dependence
may be related to the fact that all structures in a granular solid can only
transmit compression or shear stresses, whereas in gels also tension can be
transmitted. All the remaining stress correlations from $3D$ gels at
$\phi=0.125$ are shown in Appendix B.

The pattern of the stress correlation in gels are sensitive to the distance
from the rigidity threshold, as shown in Fig. 6, where the $C_{xzzz}$ maps are
displayed for gels at different $\phi=0.125,0.075$ and $0.05$ obtained for
cooling rate $C_{r}=9\times 10^{-5}\epsilon/(\tau_{0}k_{B})$. As the volume
fraction increases the local connectivity and gel moduli increase. At
$\phi=0.05$, the networks are sparse, spatially heterogeneous and barely
rigid, as also discussed in Refs. Bantawa  _et al._ (2022); Bouzid and Del
Gado (2018), and their elasticity is extremely week. With increasing the
particle volume fraction, the gels obtained with the same cooling rate become
stiffer, more locally connected and move away from the rigidity threshold. The
stress correlations at different densities, interestingly, have the same
general pattern, which however becomes increasingly blurred as the gels
approach the rigidity threshold (Fig. 6 from left to right). A similar trend
is observed for the correlations of all stress components. These findings
support the idea that the stress correlation intensity variation is directly
related to the elastic moduli, suggesting that they may be used to detect the
distance from the rigidity threshold and provide information on the marginal
stability of the gels. We confirm these findings also by varying, for the same
particle volume fraction, the cooling rate at which the gel is formed, since
increaseing this rate leads to gels that are weaker and closer to the rigidity
threshold. By comparing, for example, the stress correlation map in Fig. 7,
which shows $C_{xzzz}$ for $\phi=0.125$ at $C_{r}=9\times
10^{-2}\epsilon/(\tau_{0}k_{B})$, with the first map on the left of Figure 6
(corresponding to the same volume fraction and $C_{r}=9\times
10^{-5}\epsilon/(\tau_{0}k_{B})$) we can notice how again the correlation
pattern becomes more blurred for weaker and more marginal gels. Clearly,
therefore, the changes seen in the stress correlations are due to the
different elasticity of the network and not simply the gel density.

##  IV Summary

In summary, we have demonstrated the presence of force chains in soft
particulate gels by computing the stress-stress correlation functions in 2D
and 3D model gels. The stress correlation patterns distinguish the rigid gels
from the floppy gels and display the same general characteristics as granular
solids, i.e., the strong angular dependence, and the pinch point singularity
predicted by a stress-only theoretical framework (the VCT framework in Ref.
Nampoothiri  _et al._ (2020)), where the elastic response is a property
emerging only from the constraints of mechanical equilibrium, without
resorting to the constitutive relation of linear elasticity. We also
demonstrate that the stress-stress correlation patterns nevertheless highlight
distinctive differences between gels and granular solids, and that they are
sensitive to variation of the gel moduli, due to changes in the gel topology
and marginal stability, not simply to the gel density. These results are
consistent with the idea that the stress-stress correlation patterns can
provide further information on the tensorial properties of the emerging
elasticity in these materials. Stress-stress correlations patterns have been
measured in experiments on granular solids using photoelastic disks (see Ref.
Nampoothiri  _et al._ (2020)), however this is clearly much more challenging
for experiments on particulate gels, where measurement techniques to extract
local stresses and their spatial distributions are still being developed
Arevalo  _et al._ (2015); Lin and Cohen (2016); Gauthier  _et al._ (2021);
Dong  _et al._ (2022). Using the stress-stress correlation functions obtained
in simulations to extract, from their angular dependence, the properties of
the elastic tensor, instead, would provide predictions for the overall
mechanical response to different deformation modes (e.g. shear, compression,
extension, etc…) which are indeed accessible to mechanical and rheological
experiments. This information, testable on a broad range of gels and
experimental conditions, could then be fed back to the theory to obtain the
stress-stress correlation patterns from the experiments. Hence developing
further the analysis proposed here seems a promising route to gain novel
insight into the emerging elasticity of soft particulate gels and its
connection to stress localization.

###### Acknowledgements.

We thank Shang Zhang, Jishnu N Nampoothiri and Minaspi Bantawa for valuable
discussions and configurations. This work was supported by the National
Science Foundation (NSF-DMR-2026842; NSF-DMR-2026825 and NSF-DMR-2026834) and
the Clare Boothe Luce Program. Acknowledgement is made to the donors of the
American Chemical Society Petroleum Research Fund for partial support of this
research.

## References

  * Del Gado  _et al._ (2016) E. Del Gado, D. Fiocco, G. Foffi, S. Manley, V. Trappe, and A. Zaccone, “Colloidal gelation,” in  _Fluids, Colloids and Soft Materials_ (John Wiley & Sons, Ltd, 2016) Chap. 14, pp. 279–291.
  * Petekidis and Wagner (2021) G. Petekidis and N. J. Wagner, “Rheology of colloidal glasses and gels,” in  _Theory and Applications of Colloidal Suspension Rheology_ (Cambridge University Press, 2021) p. 173–226.
  * Ioannidou  _et al._ (2016) K. Ioannidou, M. Kanduč, L. Li, D. Frenkel, J. Dobnikar, and E. Del Gado, “The crucial effect of early-stage gelation on the mechanical properties of cement hydrates,” Nature Communications 7, 12106 (2016).
  * Anderson and Lekkerkerker (2002) V. J. Anderson and H. N. Lekkerkerker, “Insights into phase transition kinetics from colloid science,” Nature 416, 811–815 (2002).
  * Royall  _et al._ (2021) C. P. Royall, M. A. Faers, S. L. Fussell, and J. E. Hallett, “Real space analysis of colloidal gels: triumphs, challenges and future directions,” J. Condens. Matter Phys. 33, 453002 (2021).
  * Bouzid  _et al._ (2017) M. Bouzid, J. Colombo, L. V. Barbosa, and E. Del Gado, “Elastically driven intermittent microscopic dynamics in soft solids,” Nature communications 8, 1–8 (2017).
  * Zhang  _et al._ (2021) S. Zhang, E. Stanifer, V. Vasisht, L. Zhang, E. Del Gado, and X. Mao, “Prestressed elasticity of amorphous solids,”  (2021).
  * Zhang  _et al._ (2019) S. Zhang, L. Zhang, M. Bouzid, D. Z. Rocklin, E. Del Gado, and X. Mao, “Correlated rigidity percolation and colloidal gels,” Physical review letters 123, 058001 (2019).
  * Leocmach  _et al._ (2014) M. Leocmach, C. Perge, T. Divoux, and S. Manneville, “Creep and fracture of a protein gel under stress,” Physical review letters 113, 038303 (2014).
  * Colombo and Del Gado (2014a) J. Colombo and E. Del Gado, “Stress localization, stiffening, and yielding in a model colloidal gel,” Journal of rheology 58, 1089–1116 (2014a).
  * Filiberti, Piazza, and Buzzaccaro (2019) Z. Filiberti, R. Piazza, and S. Buzzaccaro, “Multiscale relaxation in aging colloidal gels: From localized plastic events to system-spanning quakes,” Phys. Rev. E 100, 042607 (2019).
  * Aime, Ramos, and Cipelletti (2018) S. Aime, L. Ramos, and L. Cipelletti, “Microscopic dynamics and failure precursors of a gel under mechanical load,” Proceedings of the National Academy of Sciences 115, 3587–3592 (2018), https://www.pnas.org/doi/pdf/10.1073/pnas.1717403115 .
  * Alexander (1998) S. Alexander, “Amorphous solids: their structure, lattice dynamics and elasticity,” Physics reports 296, 65–236 (1998).
  * Cates  _et al._ (1998) M. Cates, J. Wittmer, J.-P. Bouchaud, and P. Claudin, “Jamming, force chains, and fragile matter,” Physical review letters 81, 1841 (1998).
  * Radjai  _et al._ (1998) F. Radjai, D. E. Wolf, M. Jean, and J.-J. Moreau, “Bimodal character of stress transmission in granular packings,” Physical review letters 80, 61 (1998).
  * Gendelman  _et al._ (2016) O. Gendelman, Y. G. Pollack, I. Procaccia, S. Sengupta, and J. Zylberg, “What determines the static force chains in stressed granular media?” Physical review letters 116, 078001 (2016).
  * Vinutha and Sastry (2019) H. A. Vinutha and S. Sastry, “Force networks and jamming in shear-deformed sphere packings,” Physical Review E 99, 012123 (2019).
  * Wang, Wang, and Zhang (2020) Y. Wang, Y. Wang, and J. Zhang, “Connecting shear localization with the long-range correlated polarized stress fields in granular materials,” Nature communications 11, 1–7 (2020).
  * Henkes and Chakraborty (2009) S. Henkes and B. Chakraborty, “Statistical mechanics framework for static granular matter,” Phys. Rev. E 79, 061301 (2009).
  * Mao  _et al._ (2009) X. Mao, P. M. Goldbart, X. Xing, and A. Zippelius, “Soft random solids and their heterogeneous elasticity,” Phys. Rev. E 80, 031140 (2009).
  * DeGiuli (2018) E. DeGiuli, “Field theory for amorphous solids,” Physical review letters 121, 118001 (2018).
  * Lemaître (2018) A. Lemaître, “Stress correlations in glasses,” The Journal of Chemical Physics 149, 104107 (2018).
  * Pretko (2017) M. Pretko, “Generalized electromagnetism of subdimensional particles: A spin liquid story,” Physical Review B 96, 035119 (2017).
  * Nampoothiri  _et al._ (2020) J. N. Nampoothiri, Y. Wang, K. Ramola, J. Zhang, S. Bhattacharjee, and B. Chakraborty, “Emergent elasticity in amorphous solids,” Physical review letters 125, 118002 (2020).
  * Nampoothiri  _et al._ (2022) J. N. Nampoothiri, M. D’Eon, K. Ramola, B. Chakraborty, and S. Bhattacharjee, “Tensor electromagnetism and emergent elasticity in jammed solids,” arXiv preprint arXiv:2204.11811 (2022).
  * Timoshenko and Goodier (1951) S. Timoshenko and J. N. Goodier,  _Theory of Elasticity: by S. Timoshenko and JN Goodier_ (McGraw-Hill, 1951).
  * Landau  _et al._ (2012) L. Landau, L. Pitaevskii, A. Kosevich, and E. Lifshitz,  _Theory of Elasticity: Volume 7_, v. 7 (Elsevier Science, 2012).
  * Colombo and Del Gado (2014b) J. Colombo and E. Del Gado, “Self-assembly and cooperative dynamics of a model colloidal gel network,” Soft Matter 10, 4003–4015 (2014b).
  * Bouzid and Del Gado (2018) M. Bouzid and E. Del Gado, “Network topology in soft gels: Hardening and softening materials,” Langmuir 34, 773–781 (2018).
  * Bantawa  _et al._ (2021) M. Bantawa, W. A. Fontaine-Seiler, P. D. Olmsted, and E. Del Gado, “Microscopic interactions and emerging elasticity in model soft particulate gels,” Journal of Physics: Condensed Matter 33, 414001 (2021).
  * Noro and Frenkel (2000) M. G. Noro and D. Frenkel, “Extended corresponding-states behavior for particles with variable range attractions,” The Journal of Chemical Physics 113, 2941–2944 (2000).
  * Plimpton (1995) S. Plimpton, “Fast parallel algorithms for short-range molecular dynamics,” Journal of computational physics 117, 1–19 (1995).
  * Jacobs and Hendrickson (1997) D. J. Jacobs and B. Hendrickson, “An algorithm for two-dimensional rigidity percolation: the pebble game,” Journal of Computational Physics 137, 346–365 (1997).
  * Bouzid  _et al._ (2018) M. Bouzid, B. Keshavarz, M. Geri, T. Divoux, E. Del Gado, and G. H. McKinley, “Computing the linear viscoelastic properties of soft gels using an optimally windowed chirp protocol,” Journal of Rheology 62, 1037–1050 (2018).
  * Thompson, Plimpton, and Mattson (2009) A. P. Thompson, S. J. Plimpton, and W. Mattson, “General formulation of pressure and stress tensor for arbitrary many-body interaction potentials under periodic boundary conditions,” The Journal of chemical physics 131, 154107 (2009).
  * Behringer and Chakraborty (2018) R. P. Behringer and B. Chakraborty, “The physics of jamming for granular materials: a review,” Reports on Progress in Physics 82, 012601 (2018).
  * Bantawa  _et al._ (2022) M. Bantawa, B. Keshavarz, M. Geri, M. Bouzid, T. Divoux, G. H. McKinley, and E. Del Gado, “The hidden hierarchical nature of soft particulate gels,” preprint (2022).
  * Arevalo  _et al._ (2015) R. C. Arevalo, P. Kumar, J. S. Urbach, and D. L. Blair, “Stress heterogeneities in sheared type-i collagen networks revealed by boundary stress microscopy,” PLOS ONE 10, 1–12 (2015).
  * Lin and Cohen (2016) N. Y. C. Lin and I. Cohen, “Relating microstructure and particle-level stress in colloidal crystals under increased confinement,” Soft Matter 12, 9058–9067 (2016).
  * Gauthier  _et al._ (2021) A. Gauthier, M. Pruvost, O. Gamache, and A. Colin, “A new pressure sensor array for normal stress measurement in complex fluids,” Journal of Rheology 65, 583–594 (2021).
  * Dong  _et al._ (2022) J. Dong, F. Turci, R. L. Jack, M. A. Faers, and C. P. Royall, “Direct imaging of contacts and forces in colloidal gels,” The Journal of Chemical Physics 156, 214907 (2022), https://doi.org/10.1063/5.0089276 .
  * Chaikin, Lubensky, and Witten (1995) P. M. Chaikin, T. C. Lubensky, and T. A. Witten,  _Principles of condensed matter physics_ , Vol. 10 (Cambridge university press Cambridge, 1995).

##  Appendix A 2D correlation functions

In Fig. 3, we show data for $C_{yyyy}$ and $C_{xxyy}$. Here, we show all the
remaining stress correlations for the rigid gels. The correlation functions
are computed using Eq. 5. The angular dependence of these correlation
functions matches with the granular case Nampoothiri  _et al._ (2020).

Figure 8:  The stress correlation functions for the 2D rigid gels. The three
columns show respectively: the stress correlation in $q-$ space, and the
corresponding radially averaged angular plots and the stress correlation in
real space.

##  Appendix B Stress correlations in Fourier space for 3D gels

In the manuscript, we show $C_{xxxx},C_{zzzz}$ and $C_{xzzz}$ correlation
functions of gels at $\phi=0.125$. In Fig. 9, the remaining $18$ correlation
functions are shown.

Figure 9:  The $q$-space stress correlations for 3D gels at $\phi=0.125$.