# Better RAG using Relevant Information Gain

Marc Pickett &Jeremy Hartman Ayan Kumar Bhowmick &Raquib-ul Alam &Aditya
Vempaty
Emergence AI
Contact<EMAIL_ADDRESS>
###### Abstract

A common way to extend the memory of large language models (LLMs) is by
retrieval augmented generation (RAG), which inserts text retrieved from a
larger memory into an LLM’s context window. However, the context window is
typically limited to several thousand tokens, which limits the number of
retrieved passages that can inform a model’s response. For this reason, it’s
important to avoid occupying context window space with redundant information
by ensuring a degree of _diversity_ among retrieved passages. At the same
time, the information should also be _relevant_ to the current task. Most
prior methods that encourage diversity among retrieved results, such as
Maximal Marginal Relevance (MMR), do so by incorporating an objective that
explicitly trades off diversity and relevance. We propose a novel simple
optimization metric based on _relevant information gain_ , a probabilistic
measure of the _total_ information relevant to a query for a set of retrieved
results. By optimizing this metric, diversity organically emerges from our
system. When used as a drop-in replacement for the retrieval component of a
RAG system, this method yields state-of-the-art performance on question
answering tasks from the Retrieval Augmented Generation Benchmark (RGB),
outperforming existing metrics that directly optimize for relevance and
diversity. 111Code is available at https://github.com/EmergenceAI/dartboard.

Better RAG using Relevant Information Gain

Marc Pickett Jeremy Hartman

Ayan Kumar Bhowmick Raquib-ul Alam Aditya Vempaty

Emergence AI Contact<EMAIL_ADDRESS>

##  1 Introduction

A limitation of transformer-based Large Language Models (LLMs) is that the
number of tokens is bounded by the transformer’s context window, which is
typically in the thousands. This is often insufficient for representing large
texts, such as novels and corporate documentation. A common way to mitigate
this constraint is via retrieval augmented generation (RAG), in which a
relatively small subset of relevant passages are retrieved from a larger
database and inserted into an LLM’s context window Gao et al. (2024).
Typically, this process involves applying a similarity metric, such as cosine
similarity, to (precomputed) embeddings of passages and the embedding of a
query. Using this metric, many systems then use K-nearest-neighbors or a fast
approximation with a vector database such as FAISS Douze et al. (2024).
Importantly, K-nearest-neighbors Bijalwan et al. (2014) and related methods
(such as a cross-encoder reranker Nogueira and Cho (2020)) simply return the
highest _individually_ relevant passages, without regard to whether the
information in the passages is redundant. Given the premium value on LLM
context-window real estate, it’s important to make best use of this limited
resource by minimizing redundancy, while maintaining relevance.

To appreciate the importance of minimizing redundancy in a RAG context,
consider a toy database of facts and the two possible sets of retrieval
results in Table 1, for the same query, “Tell me some facts about sharks.”
Both sets of retrieved results are highly relevant to the query, but only the
second set is diverse enough to support a satisfactory answer.

Shark dataset
---
Sharks are boneless.
Sharks do not have any bones.
Sharks have no bones.
Sharks have excellent vision.
Sharks are very fierce.
Sharks are apex predators.

Query: Tell me some facts about sharks
---
Retrieval results 1:
Sharks are boneless.
Sharks have no bones.
Sharks do not have any bones.
Retrieval results 2:
Sharks are boneless.
Sharks have excellent vision.
Sharks are apex predators.

Table 1: A toy database of shark facts (top) and two possible sets of
retrieval results for the same query (bottom).

A family of methods from the Information Retrieval literature attempts to
address the general issue of diversity in retrieved results by introducing a
measure that explicitly balances diversity and relevance Carbonell and
Goldstein (1998). In this paper, we propose a more principled method,
_Dartboard_ , that instead seeks to directly accomplish what previous methods
are indirectly aiming for - maximize the _total_ amount of information
relevant for a given query in a set of $k$ results. The intuition behind
_Dartboard_ is simple - we assume that one passage is the “correct” one for a
given query. Our system is allowed $k$ “guesses” and it aims to maximize the
relevance score of its _most relevant_ guess. Since the best guess is not
known ahead of time, this score is weighted by the probability of that guess
being the most relevant. This objective is sufficient to encourage diversity
in the guesses. This is because a redundant guess does little to increase the
relevance of the most relevant guess.

The main contributions of this paper are 3-fold:

  * •

We introduce the _Dartboard_ algorithm, a principled retrieval method based on
optimizing a simple metric of total information gain relevant to a given query
(§2).

  * •

We demonstrate the effectiveness of _Dartboard_ on Retrieval-Augmented
Generation Benchmark (RGB) Chen et al. (2023), a closed-domain question
answering task. This benchmark consists of a retrieval component, and an end-
to-end question-answering component. We show that the _Dartboard_ algorithm,
when used as the retrieval component, outperforms all existing baselines at
both the component level and at end-to-end level (§3.1).

  * •

We show that instead of directly encouraging diversity, diversity naturally
emerges by optimizing this metric (§A.5).

##  2 Dartboard

The _Dartboard_ algorithm is based on the following analogy illustrated in
Figure 1: Suppose that we have a cooperative two-player game where a dartboard
is covered with a random collection of points. Player 1 is given one of these
points arbitrarily as the _target_. Player 1 then throws her dart aiming for
the target, and it lands somewhere on the board. Where it lands is the
_query_. Player 2 sees where Player 1’s dart landed (the query), but doesn’t
know where the actual target is. Player 2 then picks $k$ of the points on the
board. The true target is revealed, and the score (which the players are
trying to minimize) is the distance from the target to the _closest_ guess.
Note that to minimize the score, Player 2 would not want to put all his
guesses right next to each other. Also, Player 2 should take into account how
accurate Player 1’s throws are in general. In our implementation, Player 1’s
accuracy is modeled by a Gaussian distribution with standard deviation
$\sigma$.

Figure 1: A visualization of _Dartboard_. The _query_ is represented by the
red star. All points are represented by blue dots. The five dots highlighted
by grey background are the query’s 5 nearest neighbors, while the dots circled
in green are the five points selected by the _Dartboard_ algorithm (numbered
in the order selected by the greedy algorithm). The concentric red circles are
spaced at multiples of $\sigma$, which represents the standard deviation of
our uncertainty for the query’s accuracy. Note the possible redundancy by
naive k-nearest-neighbors, which ignores points above or to the right of the
query.

More formally, Player 1 selects a target $T$ from a set of all points $A$ and
gives a query $q$. Then Player 2 makes a set of guesses $G\subseteq A$,
resulting in a score $s\left(G,q,A,\sigma\right)$ which is given as:

$s\left(G,q,A,\sigma\right)=\sum_{t\in A}P\left(T=t|q,\sigma\right)\min_{g\in
G}D\left(t|g\right)$ (1)

where $D$ is a distance function. For $d$ dimensional vectors,
$A\subseteq\mathbb{R}^{d}$; under some assumptions, we can use a Gaussian
kernel for the distance functions. For example, we can set
$P\left(T=t|q,\sigma\right)=\mathcal{N}\left(q,t,\sigma\right)$. Thus, our
equation becomes:

$s\left(G,q,A,\sigma\right)\propto-\sum_{t\in
A}\mathcal{N}\left(q,t,\sigma\right)\max_{g\in
G}\mathcal{N}\left(t,g,\sigma\right)$ (2)

###  2.1 The Dartboard Algorithm

The _Dartboard_ Algorithm aims to maximize Equation 2 given a distance metric.
In practice, we can greedily build our set $G$, which works well as it saves
us combinatorial search, and allows reuse of previous answers (since the
top-$k$ results are a subset of the top-$k+1$ results). We begin by ranking
top-$k$ passages $A^{\prime}$ from our initial dataset of passages $A$ using
$K$-nearest-neighbors based on cosine similarity. We use a linear search, but
sub-linear methods such as FAISS Douze et al. (2024) could also be used for
this initial ranking. Our search is a simple greedy optimization method with
two changes - (a) we stay in log space to avoid numerical underflow, and (b)
we reuse the results ($maxes$) from previous loops to avoid recomputing the
maximums. The detailed algorithm is given in Algorithm 1 in Appendix A.1. In
Appendix A.3, we also show how to adapt _Dartboard_ to use a cross-encoder
based reranker (resulting in two methods called _Dartboard crosscoder_ and
_Dartboard hybrid_), and Appendix A.4 shows that _Dartboard_ generalizes KNN
and MMR retrieval algorithms Onal et al. (2015).

##  3 Experiments

We tested _Dartboard_ on benchmark datasets from Chen et al. (2023), from
which we used two types of closed-domain question answering. In the _simple
question answering_ case, a query is answerable from a single passage
retrieved from the corpus. For example, consider the query When is the
premiere of ‘Carole King & James Taylor: Just Call Out My Name’?. On the other
hand, in the _information integration_ case, a query would require multiple
passages to be retrieved to answer the query. For example, consider the query
Who is the director of ‘Carole King & James Taylor: Just Call Out My Name’ and
when is its premiere?. We modified this benchmark for our setup in the
following way. The original benchmark contains “positive” and “negative”
labeled passages for each query. The positive passages are useful for
answering, while the negative ones are related but ineffective in answering
the query. Since we are interested in the retrieval component of this task, we
merged the positive and negative passages for all queries into a single
collection of $11,641$ passages for the $300$ _simple question answering_ test
cases and $5,701$ passages for the $100$ _information integration_ test cases.
The evaluation is otherwise identical apart from the retrieval component. Note
that the innovation of _Dartboard_ is solely on the retrieval component.
Therefore, we keep the rest of the RAG pipeline fixed. In particular, we do
not modify the prompting of LLMs or try to optimize passage embeddings.

Given a query and the full set of thousands of passage embeddings, we measured
both a direct retrieval score and the overall end-to-end performance of the
system with the only change being the retrieval algorithm. For the direct
retrieval score, we computed the Normalized Discounted Cumulative Gain (NDCG)
score Wang et al. (2013) on retrieving any one of the “positive” passages
relevant to a specific query. In the information integration case, the
positive passages were split into positive ones for each component of the
question. Therefore, in this case, we calculated the NDCG score for retrieving
at least one positive passage for _each_ component of the query. For the end-
to-end score, given an LLM’s response to the query (generated from retrieved
passages), we use the same evaluation as Chen et al. (2023), which does a
string match of the response on a set of correct answers, marking each
response as either correct or incorrect.

Some of the methods (described in Appendix A.2), including _Dartboard_ , have
tunable parameters. For instance, Maximal Marginal Relevance (MMR) has a
diversity parameter that varies from 0 to 1. We performed a grid search over
these parameters, reporting the best results for each method.

###  3.1 Results

From the results shown in Table 2, we observe that _Dartboard_ outperforms all
state-of-the-art methods in terms of all metrics across all the tasks.

| Simple | Integrated
---|---|---
| QA | NDCG | QA | NDCG
_Oracle_ | 89.3% | 1.000 | 36% | .826
D-H (ours) | 85.6% | 0.973 | 41% | .609
D-CC (ours) | 84.3% | 0.971 | 42% | .595
D-CS (ours) | 83.0% | 0.975 | 36% | .545
MMR Crosscoder | 84.3% | 0.971 | 40% | .598
MMR Cossim | 81.0% | 0.974 | 36% | .541
KNN Crosscoder | 84.3% | 0.968 | 36% | .580
KNN Cossim | 80.0% | 0.973 | 25% | .514
Empty | 3.3% | 0.000 | 3% | .000
Random | 3.3% | 0.044 | 2% | .028
Table 2: Results for the _Dartboard_ retrieval system on the QA benchmarks
using $k=5$. For methods with tunable parameters (_Dartboard_ and MMR), the
best score over a parameter sweep is reported.

Figure 2 shows the performance of different retrieval methods on the end-to-
end QA task (simple) as the parameters vary. Although _Dartboard Crosscoder
(D-CC)_ and _Dartboard hybrid (D-H)_ are fairly robust to a range of $\sigma$
values, the best performance is achieved for _Dartboard hybrid_ with
$\sigma=0.096$ (See Appendix A.2 for baselines).

Figure 2: Performance on end-to-end QA task (simple) as parameters vary. For
_Dartboard_ , we show its performance as $\sigma$ varies. For MMR, we show its
performance as the diversity parameter varies.

##  4 Related Work

MMR retrieves documents Carbonell and Goldstein (1998) that are both relevant
to the query and dissimilar to previously retrieved documents. It combines a
relevance score (e.g., from BM25) with a novelty score that penalizes
documents similar to those already retrieved. It have been used extensively
for building recommendation systems Xia et al. (2015); Wu et al. (2023) as
well as for summarization tasks Agarwal et al. (2022); Adams et al. (2022).
However, MMR suffers from few limitations. First is that MMR requires the
_diversity_ parameter to control the balance between relevance and novelty.
This parameter is often dataset-specific and requires careful tuning, making
it impractical for real-world applications. Second is that MMR can favor exact
duplicates of previously retrieved documents as they retain a high relevance
score while minimally impacting the average novelty score (See Appendix A.7).

KNN retrieves documents based on their similarity to a query embedding Dharani
and Aroquiaraj (2013); Bijalwan et al. (2014). While efficient, KNN often
suffers from redundancy as nearby documents in the embedding space tend to be
semantically similar Taunk et al. (2019). This can lead to a retrieved set
dominated by passages conveying the same information with slight variations.

Several recent works have explored incorporating diversity objectives into
retrieval models Angel and Koudas (2011); Li et al. (2015); Fromm et al.
(2021). These approaches often involve complex optimization functions or
require additional training data for diversity estimation. For example,
Learning-to-Rank with Diversity methods leverage learning-to-rank frameworks
that incorporate diversity objectives directly into the ranking function. This
allows for the optimization of both relevance and diversity during the ranking
process. However, these approaches often require large amounts of labeled
training data for diversity, which can be expensive and time-consuming to
obtain Wasilewski and Hurley (2016); Yan et al. (2021). Bandit-based
approaches model document selection as a multi-armed bandit problem Hofmann et
al. (2011); Wang et al. (2021). The model explores different retrieval
strategies and receives feedback based on the relevance and diversity of the
retrieved passages. These approaches can be effective but can be
computationally expensive for large-scale retrieval tasks.

RAG models have also been extended to incorporate diversity objectives. For
example, RAG with Dense Passage Retrieval retrieves a large number of
candidate passages Cuconasu et al. (2024); Reichman and Heck (2024);
Siriwardhana et al. (2023). It then employs a two-stage selection process:
first selecting a diverse subset based on novelty scores, then selecting the
most relevant passages from this subset. While effective, this approach
requires careful tuning of the selection thresholds.

##  5 Discussion

In this paper, we introduce _Dartboard_ , a principled retrieval algorithm
that implicitly encourages diversity of retrieved passages by optimizing for
relevant information gain. We demonstrate that _Dartboard_ outperforms
existing state-of-the-art retrieval algorithms on both retrieval and end-to-
end QA tasks. We view this work as an initial step for a more general line of
work that optimizes information gain during retrieval, especially in the
context of RAG systems. In future work, we plan to investigate _Dartboard_ for
other retrieval tasks, such as suggestion generation (see Appendix A.6).

##  6 Limitations

We have not done a systematic investigation of the run time of _Dartboard_. In
the worst case scenario, _Dartboard_ is quadratic in the number of ranked
passages. However, in practice, _Dartboard hybrid_ typically runs in a
fraction of a second for ranking (based on cosine-similarity with query) a set
of $100$ passages (note that a full cross-encoder based MMR/_Dartboard_ needs
to run the cross-encoder $10,000$ times, and can take several seconds). This
retrieval time is minimal compared to the time required for a LLM to process
the retrieved passages and generate an answer.

Our experimental results are limited to a single benchmark and a single LLM
i.e. ChatGLM Hou et al. (2024). It remains to be seen whether our results
would generalize to other benchmarks and LLMs. We plan to investigate this in
future work.

One shortcoming of our method (also shared by MMR) is that it requires a
hyperparameter that affects how much diversity is encouraged. While we show
that _Dartboard_ is robust to the choice of this hyperparameter, it would be
ideal to have a method that does not require manual tuning. As part of future
work, we plan to investigate methods that automatically adapt to the context
of the query. For example, the hyperparameter could be set based on a held-out
validation set.

Another topic for future work is to investigate if it is also possible for
$\sigma$ to vary depending on the type of query. For example, a query like
“Tell me facts about The Beatles” would warrant a broader range of passages
than a query like “Tell me facts about George Harrison”.

Another shortcoming of our approach is that our benchmarking criteria is
limited in terms of the evaluation protocol we are using. Our evaluation is
based on an exact string match of the output answer generated from the LLM
with a set of possible answers. For example, for one question, the generated
output answer is considered correct if it contains the exact string ‘January 2
2022’, ‘Jan 2, 2022’, etc., but would be considered incorrect if it only
contains ‘January 2nd, 2022’. However, we left the benchmark as is (modulo our
modifications mentioned above) so that our method is easily comparable to that
of others.

Finally, though the initial cosine similarity based proposed _Dartboard_
method is principled, the hybrid variation of _Dartboard_ is not that
principled. This is because it tries to compare logits from a cross-encoder
with the cosine similarity of a different embedding model, similar to
comparing apples with oranges, though it seems to work well as seen in our
presented empirical results.

## References

  * Adams et al. (2022) David Adams, Gandharv Suri, and Yllias Chali. 2022.  Combining state-of-the-art models with maximal marginal relevance for few-shot and zero-shot multi-document summarization.  _arXiv preprint arXiv:2211.10808_.
  * Agarwal et al. (2022) Abhishek Agarwal, Shanshan Xu, and Matthias Grabmair. 2022.  Extractive summarization of legal decisions using multi-task learning and maximal marginal relevance.  _arXiv preprint arXiv:2210.12437_.
  * Angel and Koudas (2011) Albert Angel and Nick Koudas. 2011.  Efficient diversity-aware search.  In _Proceedings of the 2011 ACM SIGMOD International Conference on Management of data_ , pages 781–792.
  * Bijalwan et al. (2014) Vishwanath Bijalwan, Vinay Kumar, Pinki Kumari, and Jordan Pascual. 2014.  Knn based machine learning approach for text and document mining.  _International Journal of Database Theory and Application_ , 7(1):61–70.
  * Carbonell and Goldstein (1998) Jaime Carbonell and Jade Goldstein. 1998.  The use of mmr, diversity-based reranking for reordering documents and producing summaries.  In _Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 335–336.
  * Chen et al. (2023) Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2023.  Benchmarking large language models in retrieval-augmented generation.  _Preprint_ , arXiv:2309.01431.
  * Cuconasu et al. (2024) Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024.  The power of noise: Redefining retrieval for rag systems.  _arXiv preprint arXiv:2401.14887_.
  * Dharani and Aroquiaraj (2013) T Dharani and I Laurence Aroquiaraj. 2013.  Content based image retrieval system using feature classification with modified knn algorithm.  _arXiv preprint arXiv:1307.4717_.
  * Douze et al. (2024) Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024.  The faiss library.  _Preprint_ , arXiv:2401.08281.
  * Fromm et al. (2021) Michael Fromm, Max Berrendorf, Sandra Obermeier, Thomas Seidl, and Evgeniy Faerman. 2021.  Diversity aware relevance learning for argument search.  In _Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28–April 1, 2021, Proceedings, Part II 43_ , pages 264–271. Springer.
  * Gao et al. (2024) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024.  Retrieval-augmented generation for large language models: A survey.  _Preprint_ , arXiv:2312.10997.
  * Hofmann et al. (2011) Katja Hofmann, Shimon Whiteson, Maarten de Rijke, et al. 2011.  Contextual bandits for information retrieval.  In _NIPS 2011 Workshop on Bayesian optimization, experimental design, and bandits, granada_ , volume 12, page 2011.
  * Hou et al. (2024) Zhenyu Hou, Yiin Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng, Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang, et al. 2024.  Chatglm-rlhf: Practices of aligning large language models with human feedback.  _arXiv preprint arXiv:2404.00934_.
  * Li et al. (2015) Jianqiang Li, Chunchen Liu, Bo Liu, Rui Mao, Yongcai Wang, Shi Chen, Ji-Jiang Yang, Hui Pan, and Qing Wang. 2015.  Diversity-aware retrieval of medical records.  _Computers in Industry_ , 69:81–91.
  * Nogueira and Cho (2020) Rodrigo Nogueira and Kyunghyun Cho. 2020.  Passage re-ranking with bert.  _Preprint_ , arXiv:1901.04085.
  * Onal et al. (2015) Kezban Dilek Onal, Ismail Sengor Altingovde, and Pinar Karagoz. 2015.  Utilizing word embeddings for result diversification in tweet search.  In _Information Retrieval Technology: 11th Asia Information Retrieval Societies Conference, AIRS 2015, Brisbane, QLD, Australia, December 2-4, 2015. Proceedings 11_ , pages 366–378. Springer.
  * Reichman and Heck (2024) Benjamin Reichman and Larry Heck. 2024.  Retrieval-augmented generation: Is dense passage retrieval retrieving?  _arXiv preprint arXiv:2402.11035_.
  * Siriwardhana et al. (2023) Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. 2023.  Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering.  _Transactions of the Association for Computational Linguistics_ , 11:1–17.
  * Taunk et al. (2019) Kashvi Taunk, Sanjukta De, Srishti Verma, and Aleena Swetapadma. 2019.  A brief review of nearest neighbor algorithm for learning and classification.  In _2019 international conference on intelligent computing and control systems (ICCS)_ , pages 1255–1260. IEEE.
  * Wang et al. (2021) Huazheng Wang, Yiling Jia, and Hongning Wang. 2021.  Interactive information retrieval with bandit feedback.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2658–2661.
  * Wang et al. (2013) Yining Wang, Liwei Wang, Yuanzhi Li, Di He, and Tie-Yan Liu. 2013.  A theoretical analysis of ndcg type ranking measures.  In _Conference on learning theory_ , pages 25–54. PMLR.
  * Wasilewski and Hurley (2016) Jacek Wasilewski and Neil Hurley. 2016.  Incorporating diversity in a learning to rank recommender system.  In _The twenty-ninth international flairs conference_.
  * Wu et al. (2023) Chun-Ho Wu, Yue Wang, and Jie Ma. 2023.  Maximal marginal relevance-based recommendation for product customisation.  _Enterprise Information Systems_ , 17(5):1992018.
  * Xia et al. (2015) Long Xia, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. 2015.  Learning maximal marginal relevance model via directly optimizing diversity evaluation measures.  In _Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval_ , pages 113–122.
  * Yan et al. (2021) Le Yan, Zhen Qin, Rama Kumar Pasumarthi, Xuanhui Wang, and Michael Bendersky. 2021.  Diversification-aware learning to rank using distributed representation.  In _Proceedings of the Web Conference 2021_ , pages 127–136.

##  Appendix A Appendix

###  A.1 Dartboard Algorithm Details

The full algorithm for Dartboard is described in Algorithm 1.

# Natural log of Gaussian pdf.

1 function LogNorm($\mu$, $\sigma$)

2 return
$\ln\left(\sigma\right)-\frac{1}{2}\ln\left(2\pi\right)-\frac{\mu^{2}}{2\sigma^{2}}$

3

# $q$: the query.

# $A$: set of all points.

# $K$: number of points to triage.

# $k$: number of points to return.

# $\sigma$: The standard deviation of the Gaussians. A measure of spread.

4 function Dartboard($q$, $A$, $K$, $k$, $\sigma$)

# Triage and get the distances.

$A^{\prime},ids\leftarrow\texttt{KNN($q$, $A$, $K$)}$ # Triage using KNN.

5

$D\leftarrow\texttt{dists($A^{\prime}$, $A^{\prime}$, $\sigma$)}$ # $K$x$K$
distance matrix.

6

$Q\leftarrow\texttt{dists($q$, $A^{\prime}$, $\sigma$)}$ # Distance from each
$a\in A^{\prime}$ to $q$.

7

# Work in log space for numerical stability.

# Note that $D$ and $Q$ are now log probabilities, not distances.

8 $D\leftarrow\texttt{LogNorm($D$, $\sigma$)}$

9 $Q\leftarrow\texttt{LogNorm($Q$, $\sigma$)}$

# Greedily seed and search.

# We only track the last addition’s contribution.

10 $m\leftarrow\operatorname*{arg\,max}_{i}\left(Q\right)$

11 $maxes\leftarrow D_{m}$

12 $ret\leftarrow\left[ids_{m}\right]$

# Incrementally add until we have $k$ elements.

13 while $\left|ret\right|<k$ do

14 $newmax\leftarrow\max\left(maxes,D\right)$

15 $scores\leftarrow\texttt{LogSumExp($newmax+Q$)}$

# Get the best candidate.

16 $m\leftarrow\operatorname*{arg\,max}_{i}\left(scores\right)$

17 $maxes\leftarrow newmax_{m}$

18 $ret\leftarrow\texttt{append($ret$, $ids_{m}$)}$

19

20 return $ret$

21

Algorithm 1 Dartboard

###  A.2 Baselines

In this section, we briefly describe the different variations of _Dartboard_
as well as the competing retrieval methods that we use to compare the
performance of _Dartboard_ in Table 2 in the main paper. All methods that rely
on using the cross-encoder first use KNN to retrieve the top $100$ passages.

  * •

Dartboard cossim (D-CS): This is the variation of the proposed _Dartboard_
method that relies on using cosine similarity for ranking passages.

  * •

Dartboard crosscoder (D-CC): This is the variation of the proposed _Dartboard_
method that relies on using cross-encoder based similarity.

  * •

Dartboard hybrid (D-H): This is the variation of the proposed _Dartboard_
method that relies on using cross-encoder for the Gaussian kernel
$\mathcal{N}\left(q,t,\sigma\right)$ and cosine similarity for the Gaussian
kernel $\mathcal{N}\left(t,g,\sigma\right)$.

  * •

KNN cossim: This is the variation of K-nearest neighbors algorithm that relies
on using using cosine similarity.

  * •

KNN crosscoder: This is the variation of K-nearest neighbors algorithm that
relies on using cross-encoder similarity.

  * •

MMR cossim: This is the variation of the Maximal Marginal Relevance method
that relies on using cosine similarity.

  * •

MMR crosscoder: This is the variation of the Maximal Marginal Relevance method
that relies on using cross-encoder similarity.

  * •

Empty: This is a method that involves no retrieval step but uses just the LLM
to generate the answer for a given query.

  * •

Oracle: This method retrieves only the “positive” labeled passages. For the
information integration case, we retrieve positive passages for each component
of the query up to $k$. If the number of positive passages is less than $k$,
we use the negative passages to fill in the rest.

  * •

Random: This method randomly retrieves $k$ passages from the full passage set.

###  A.3 Modification for cross-encoder based reranker

Cross-encoder-based reranking has been shown to outperform embedding-based
approaches such as cosine similarity Nogueira and Cho (2020), as it uses the
full computational power of a transformer model, rather than being limited to
simple vector operations. We have proposed two variations of _Dartboard_ ,
namely _Dartboard Crosscoder_ and _Dartboard Hybrid_ , based on how we compute
the cross-encoder scores for the Gaussian kernels in Equation 2 given in the
main paper. For the _Dartboard Crosscoder_ variation, we use the cross-encoder
score $C\left(q,t\right)$ before computing the Gaussian kernel for both
$\mathcal{N}\left(q,t,\sigma\right)$ and $\mathcal{N}\left(t,g,\sigma\right)$
in Equation 2. Note that the cross-encoder score is asymmetric, so we simply
average the two possible ways to compute the cross-encoder score for
$\mathcal{N}\left(t,g,\sigma\right)$, i.e.,
$\frac{1}{2}\left(C\left(t,g\right)+C\left(g,t\right)\right)$. For
$\mathcal{N}\left(q,t,\sigma\right)$, we are only interested in the likelihood
of $t$ given $q$, so we only use the cross-encoder score $C\left(q,t\right)$.

However, the cross-encoder is computationally expensive to run for $k^{2}$
pairs. Hence, we rely on the _Dartboard-Hybrid_ variation wherein we use the
cross-encoder score only for the Gaussian kernel
$\mathcal{N}\left(q,t,\sigma\right)$ whereas we use cosine similarity for the
Gaussian kernel $\mathcal{N}\left(t,g,\sigma\right)$.

###  A.4 Dartboard generalizes KNN and MMR

The _Dartboard_ algorithm can be viewed as a generalization of the traditional
retrieval algorithms, KNN and MMR. In order to verify this claim, let us look
at the score presented in Equation 1 in the main paper. When the Player 1 has
a perfect aim, or in other words, $\sigma\to 0$, $P(T=t|q,\sigma)$ tends to a
point mass distribution such that $t=q$, and hence the score becomes

$s\left(G,q,A,\sigma\right)\to\min_{g\in G}D\left(q|g\right)$ (3)

where $D$ is the distance function as before. If the chosen distance function
is proportional to the similarity measure, this is nothing but the KNN
algorithm. On the other hand, when the chosen distance function is the
weighted sum of the similarity between query and guess, and dissimilarity
between current guess and past guesses, it reduces to the MMR algorithm.

###  A.5 Dartboard inherently promotes diversity

In Figure 3, we show the diversity of the retrieved passages from RGB for both
_Dartboard_ and MMR, measured as one minus the average cosine similarity
between pairs of retrieved passages. While MMR explicitly encourages
diversity, Dartboard does not. However, we observe from the figure that as the
parameter $\sigma$ increases, the diversity of the retrieved passages also
increases. This implies that by optimizing the relevant information gain
metric, _Dartboard_ inherently ensures diversity in the set of retrieved
passages.

Figure 3: We show the diversity in the set of retrieved passages from RGB for
both _Dartboard_ and MMR (for $k=5$), where diversity is one minus the average
cosine similarity between pairs of retrieved passages. For both MMR and
_Dartboard_ , diversity increases as the value of the parameters ($\sigma$ and
_diversity_ for _Dartboard_ and MMR respectively) increases.

###  A.6 Example of a generative use of Dartboard

Below is an example of the set of retrieved passages for a query that shows
that the passages retrieved by _Dartboard_ are highly diverse compared to
those retrieved by KNN which has high redundancy, if we consider the cross-
encoder based variations:

    Query: Do you want to watch soccer?

    Candidates:
     1: Absolutely!
     2: Affirmative!
     3: I don’t know!
     4: I’d love to!
     5: Maybe later.
     6: Maybe!
     7: Maybe...
     8: No thanks.
     9: No way!
    10: No, I don’t wanna do dat.
    11: No, thank you!
    12: No, thank you.
    13: Not right now.
    14: Not today.
    15: Perhaps..
    16: Sure!
    17: Yeah!
    18: Yes!
    19: Yes, please can we?
    20: Yes, please!
    21: Yes, please.
    22: Yes, we ought to!
    23: Yes, we shall!
    24: Yes, we should!

    KNN crosscoder:
      18: Yes!
      21: Yes, please.
      20: Yes, please!

    Dartboard crosscoder:
      18: Yes!
       7: Maybe...
      12: No, thank you.

###  A.7 Dartboard does not allow for the possibility of exact duplicates

The “max” in Equation 2 given in the main paper ensures that the same vector
(passage) is not selected twice (unless all non-duplicate/unique passages have
been exhausted) in case of _Dartboard_. This is in contrast to MMR, which can
select the same vector (passage).

Here is an example where MMR produces exact duplicates. Consider the scenario
when our passage database consists of the vectors
$\\{(2,1),(2,1),(1,2),(0,1)\\}$ (with a duplicate $(2,1)$). Now if we use
cosine similarity based scoring, and set diversity to $.5$ for $k=3$ in case
of MMR, the bag that maximizes the score for probe $(2,1)$ for MMR is
$\\{(0,1),(2,1),(2,1)\\}$, which has an exact duplicate passage vector
$(2,1)$. This verifies that MMR can allow for exact duplicates, which can
increase the MMR score because it decreases the average distance to the query,
while (possibly) only marginally decreasing the diversity.

On the contrary, in case of _Dartboard_ , an exact duplicate passage vector
will add zero information i.e. it would not increase the chances of hitting
the target. So it will not be selected for retrieval until all other non-
duplicate options are exhausted.

###  A.8 More results

In Figure 4, we show the relation between NDCG score and final end-to-end
performance on the question answering (QA) task.

Figure 4: Scatter plot of NDCG score and final end-to-end performance on the
QA task. The best performing methods are in the upper right hand side of the
plot.


# Validation Diagnostics for SBI algorithms based on Normalizing Flows

Julia Linhart
Université Paris-Saclay, Inria, CEA
Palaiseau 91120, France
<EMAIL_ADDRESS>
&Alexandre Gramfort
Université Paris-Saclay, Inria, CEA
Palaiseau 91120, France
<EMAIL_ADDRESS>
Pedro L. C. Rodrigues
Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK
Grenoble 38000, France
<EMAIL_ADDRESS>

###### Abstract

Building on the recent trend of new deep generative models known as
Normalizing Flows (NF), simulation-based inference (SBI) algorithms can now
efficiently accommodate arbitrary complex and high-dimensional data
distributions. The development of appropriate validation methods however has
fallen behind. Indeed, most of the existing metrics either require access to
the true posterior distribution, or fail to provide theoretical guarantees on
the consistency of the inferred approximation beyond the one-dimensional
setting. This work proposes easy to interpret validation diagnostics for
multi-dimensional conditional (posterior) density estimators based on NF. It
also offers theoretical guarantees based on results of local consistency. The
proposed workflow can be used to check, analyse and guarantee consistent
behavior of the estimator. The method is illustrated with a challenging
example that involves tightly coupled parameters in the context of
computational neuroscience. This work should help the design of better
specified models or drive the development of novel SBI-algorithms, hence
allowing to build up trust on their ability to address important questions in
experimental science.

##  1 Introduction

Recent advances in computing have led to a new generation of expressive
simulators used to study complex systems in many scientific fields [4]. They
implicitly encode the intractable likelihood $p(x\mid\bm{\theta})$ of the
underlying mechanistic model which relates observed data $x\in\mathbb{R}^{d}$
to scientifically meaningful internal parameters
$\bm{\theta}\in\mathbb{R}^{m}$. To perform statistical inference in this
setting, one can recur to simulation based inference (SBI) [4] to approximate
the posterior distribution $p(\bm{\theta}\mid x)$ using samples from the joint
pdf $p(x,\bm{\theta})$. In this work, we consider SBI methods based on
normalizing flows [14], which are invertible neural networks that can be
trained via maximum likelihood. Once the flow is trained, and for any new
observation $x$, one can directly evaluate the estimated density over the
parameter space (i.e. $\bm{\theta}$-space) [6], draw samples to construct
confidence regions [11], etc.

Flow-based SBI has been used in many recent applied works [10, 1, 6], but it
lacks an important feature before becoming a technology for experimental
science: _validation_. Ideally, one would like to have a method that provides
finite-sample guarantees of nominal coverage (or calibration) of the estimated
posterior regions, but also ensures that the approximation fits the true
underlying posterior of the model when new data is observed. While solutions
have been proposed to provide such finite-sample guarantees [5, 11], assessing
the convergence and consistency of the underlying inference remains a
challenging task [10]. Simulation based calibration (SBC) [18] is arguably the
most popular metric for validating posterior approximations in the applied SBI
community [3], but it only provides necessary conditions for consistency and
fails to give any insight on the local behavior of the estimator. Moreover, it
is a univariate procedure, thus ignoring any information about the coupling
between parameters. Zhao et al. [19] recently proposed local coverage tests
(LCT). They leverage machine learning to evaluate their test quantities on
different locations of the feature space (i.e. $x$-space). In 1D, the
probability integral transform (PIT) provides _necessary and sufficient_
conditions for consistency, which is not the case for the proposed
multivariate extensions (e.g. HPD).

In this work, we present a multivariate version of LCT for density models
based on normalizing flows. Our method comes with the same theoretical
guarantees on local consistency and interpretable diagnostics as provided by
[19] in 1D. We also present a workflow that can be used as a practical user
guide. Lastly, we provide numerical illustrations on a well known model from
computational neuroscience [8], and demonstrate the importance of trustworthy
diagnostics for multivariate posterior distributions when correlations between
parameter variables can play an important role.

##  2 Methods

Our conditional density estimator $q_{\phi}$ is a normalizing flow defined for
samples $\bm{\theta}\in\mathbb{R}^{m}$ and is conditioned on observations
$x\in\mathbb{R}^{d}$. It uses a Gaussian base distribution
$p(\bm{z})=\mathcal{N}(0,\bm{I}_{m})$ and a bijective transformation defined
for every $x$, $T_{\phi}(.;x):=(T_{\phi,1}(.;x),\dots,T_{\phi,m}(.;x))$, with
Jacobian $J_{T_{\phi}}(.;x)$, such that $q_{\phi}(\bm{\theta}\mid
x)=p(T_{\phi}(\bm{z};x))=p(\bm{z})\left|\operatorname{det}J_{T_{\phi}}(\bm{z};x)\right|^{-1}~{}$.

Our goal is to evaluate the _local consistency_ [9] of $q_{\phi}$ with respect
to the true posterior density, i.e. whether for a given $x$ the following null
hypothesis holds:

$\mathcal{H}_{0}(x):q_{\phi}(\bm{\theta}\mid x)=p(\bm{\theta}\mid
x),\quad\forall\bm{\theta}\in\mathbb{R}^{m}~{}.$ (1)

We define the multivariate probability integral transform (PIT) of
$\bm{\theta}$ at $x$ and associated to $q_{\phi}$ as the vector of $m$ one-
dimensional projections:

$\text{PIT}_{m}(\bm{\theta},x,q_{\phi})=[P_{1}(\bm{\theta},x),\dots,P_{m}(\bm{\theta},x)],\quad
P_{i}(.,x)=F_{\mathcal{N}(0,1)}\circ T_{\phi,i}^{-1}(.;x),~{}\forall
i\in[1,m]$ (2)

where $F_{\mathcal{N}(0,1)}$ is the cumulative distribution function (c.d.f.)
of the univariate normal distribution.

Theorem 1: Local Consistency and multivariate PIT. For any
$x\in\mathbb{R}^{d}$, the null hypothesis $\mathcal{H}_{0}(x)$ holds if, and
only if, the covariates of $\text{PIT}_{m}(\bm{\theta},x,q_{\phi})$
conditioned on $x$ are mutually independent and uniformly distributed over
$(0,1)$. We refer to Appendix A for a detailed proof of this result.

$\displaystyle p(\bm{\theta}\mid x)=q_{\phi}(\bm{\theta}\mid x)$
$\displaystyle\iff p(T^{-1}_{\phi}(\bm{\theta},x)\mid x)=p(\bm{z})$
$\displaystyle\iff p(T^{-1}_{\phi,1}(\bm{\theta},x),\dots
T^{-1}_{\phi,m}(\bm{\theta},x)\mid x)=\mathcal{N}(0,\bm{I}_{m})$
$\displaystyle\iff\left\\{\begin{array}[]{ll}p(P_{i}(\bm{\bm{\theta}},x)\mid
x)=\mathcal{U}(0,1)\quad\forall i\in[1,m]\quad\text{and}\\\\[5.0pt]
\\{P_{i}(\bm{\theta},x)\mid x\\}_{i=1,\dots,m}\quad\text{are mutually
independent}\end{array}\right.$

We can now verify the null hypothesis of local consistency
$\mathcal{H}_{0}(x)$ (1) via $m$ statistical tests for the uniformity of the
1D local PIT covariates and an additional test for their mutual independence.

Multivariate Local Coverage Tests (LCT). Noting that for every $i=1,\dots,m$
we have

$P_{i}(\bm{\theta},x)\mid x\sim\mathcal{U}(0,1)\iff\forall\alpha\in[0,1]\quad
r_{i,\alpha}(x)=\mathbb{P}(P_{i}(\bm{\theta},x)\leq\alpha\mid x)=\alpha$ (3)

and following the same approach as in [19], we propose $m$ test statistics

$\mathcal{T}_{i}(x):=\frac{1}{|G|}\sum_{\alpha\in
G}\Big{(}\widehat{r}_{i,\alpha}(x)-\alpha\Big{)}^{2}\quad\forall
i\in[1,m]~{},$ (4)

where $G$ is a grid of $\alpha$-values and the estimators
$\widehat{r}_{i,\alpha}$ are obtained by regressing
$\mathbb{I}_{\\{P_{i}(\bm{\theta},x)\leq\alpha\\}}$ on $x$, which is optimal
for $\mathbb{E}[\mathbb{I}_{\\{P_{i}(\bm{\theta},x)\leq\alpha\\}}\mid
x]=r_{i,\alpha}(x)$ (when using appropriate loss-functions [12]). We refer to
Algorithm 1 (resp. 2) in [19] for computing the $p$-values (resp. confidence
bands) associated to each test. Since there are $m$ independent tests, we
recur to a Bonferroni correction of the $p$-values (resp. confidence levels)
[2]. These tests come with interpretable graphical diagnostics such as PP-
plots or histograms that depict distributional deviations in 1D, as shown in
Figure 1. If any of the uniformity tests is rejected, we reject
$\mathcal{H}_{0}(x)$. If not, we proceed to the mutual independence test.

We now assume that the covariates of $\text{PIT}_{m}$ conditioned on $x$ are
uniformly distributed over $(0,1)$, which also means that every coordinate of
the flow-transformation $T_{\phi}^{-1}(\bm{\theta},x)$ follows a normal
distribution given $x$ (cf. Theorem 1). Their mutual independence is thus
characterized by a covariance matrix equal to the identity $\mathbf{I}_{m}$.
We are currently working on how to perform this check in practice.

The workflow in practice. Let
$\mathcal{D}=\\{x_{n},\bm{\theta}_{n}\\}_{n=1}^{N}$ be a calibration dataset
with $(x_{n},\bm{\theta}_{n})\sim p(x,\bm{\theta})$ which were not used to
train $q_{\phi}$. We use $\mathcal{D}$ to calculate the PIT values
$\text{PIT}_{m}(\bm{\theta}_{n},x_{n},q_{\phi})$ (2) and estimate our local
test quantities as functions of $x$. We investigate the _consistency_ of
$q_{\phi}$ in two parts:

  1. (1)

_Global consistency check_ : first, we look at the global PIT-distribution,
i.e. on average over the entire $x$-space. More specifically, we directly
compute the empirical approximation of

$\textstyle
r_{i,\alpha}=\mathbb{P}(P_{i}(\bm{\theta},x)\leq\alpha)\approx\tfrac{1}{N}\sum_{n=1}^{N}\mathbb{I}_{\\{P_{i}(\bm{\theta}_{n},x_{n})\leq\alpha\\}},\quad\forall
i\in[1,m]$

with samples from $\mathcal{D}$ to test the global uniformity of each PIT-
covariate and check which one(s) might be responsible for making $q_{\phi}$
deviate from the true posterior distribution. Note that passing such global
test is only a neccessary condition for consistency of $q_{\phi}$, as it is
insensitive to covariate transformations in $x$-space [19] (and ignores the
condition on mutual independence).

  2. (2)

_Local consistency check_ : we construct $m\times|G|$ transformed datasets
from $\mathcal{D}$

$\mathcal{D}_{i,\alpha}=\\{(x_{n},W_{n}^{i,\alpha})\\}_{n=1}^{N},\quad\alpha\in
G,\quad\forall i\in[1,m]$

where
$W_{n}^{i,\alpha}=\mathbb{I}_{\\{P_{i}(\bm{\theta}_{n},x_{n})\leq\alpha\\}}$.
We can then compute the test statistics defined in (4) for any new observation
$x$ and check whether $\mathcal{H}_{0}(x)$ should be rejected or not. In the
latter case we proceed to the mutual independence test. Only then can we
conclude on the validity of $\mathcal{H}_{0}(x)$ according to Theorem 1. If
the check in (1) passes, this allows us to _guarantee_ (or reject) consistency
anywhere in $x$-space. Even if the check in (1) does not pass, analyzing local
consistency allows to ‘open the box’ and better understand why and where in
$x$-space the estimator fails. In such situations – as the goal is not to
guarantee local consistency – it can be enough to test for the covariate-wise
uniformity of $\text{PIT}_{m}$, putting the test for mutual independence
aside. This approach was adopted for our numerical illustrations in Section 3.

##  3 Numerical illustrations

We apply our method to check the validity of a posterior estimate $q_{\phi}$
of the Jansen and Rit neural mass model (JRNMM) [8]. This well known model
from computational neuroscience takes parameters
$\bm{\theta}=(C,\mu,\sigma,g)\in\mathbb{R}^{4}$ as input and generates time
series $x\in\mathbb{R}^{1024}$ with properties similar to brain signals
obtained in neurophysiology. Parameter $C$ influences the oscillatory behavior
of the signals and $(\mu,\sigma)$ characterize their amplitude. The gain
factor $g$ rescales the signals and models the effects of the amplifier used
for measuring them in practice. Note that the coupling-effect of $g$ and
$(\mu,\sigma)$ on the amplitude leads to intrinsic indeterminacies in the
inversion of the model [16]. Our approximation $q_{\phi}$ is a conditioned
masked autoregressive flow (MAF) [13] with 10 layers and implemented in the
sbi package [20]. Based on the setup in [16], we train $q_{\phi}$ on 50 000
simulations from the JRNMM with a uniform prior defined on physiologically
relevant values (cf. pair-plots in Figure 2).

Results and discussion. All test quantities are computed with a calibration
dataset $\mathcal{D}$ containing 10 000 simulations and a grid of $|G|=100$
$\alpha$-values in [0, 1]. Following the workflow described in Section 2, we
first check the global consistency of $q_{\phi}$. In the left part of Figure
1, our graphical diagnostics illustrate how the c.d.f. for every PIT-covariate
deviates from the identity function (black dashed line), outside of the
$95\%$-confidence region (in gray), thus rejecting the null hypothesis of
global consistency. We compare our method to SBC (right part of Figure 1) as
implemented in the sbi package. We observe that SBC is unable to detect any
inconsistencies in $q_{\phi}$.

Figure 1: Global Consistency of the JRNMM posterior estimator: PP-plots and
corresponding histograms obtained for every global PIT covariate
$P_{i}(\bm{\theta})$ (right) and for SBC applied to every element $\theta_{i}$
of $\bm{\theta}$ (left). The gray zone indicates the $95\%$-confidence region
of acceptance outside of which the uniformity test is rejected. SBC fails to
reject the null hypothesis of global consistency.

We proceed to investigate the local consistency of $q_{\phi}$ on different
locations of $x$-space. Note that since the global test for consistency has
not passed, we may focus on the uniformity of the local PIT-covariates (cf.
(2) of the workflow in Section 2). We consider a 1D subspace in
$\bm{\theta}$-space, where $(C_{0},\mu_{0},\sigma_{0})$ are fixed and the gain
$g_{0}$ varies in $[-20,20]$ to generate observations $x_{0}$. The upper-left
part of Figure 2 shows how the test statistics evolve with $g_{0}$. We observe
a strong deviation from uniformity for covariates $P_{2}$ and $P_{3}$: they
vary smoothly in a ‘U-shape’, with higher values as $g_{0}$ deviates from
zero. We also generate local PP-plots (lower-left part of Figure 2) and
observe positive (resp. negative) bias for small (resp. high) values of
$g_{0}$. Our procedure shows that there are certain locations in $x$-space
(here $g_{0}=0$) where $q_{\phi}$ performs well (i.e. test statistics are
close to zero and PP-plots show little deviations from the $95\%$-confidence
region in gray), even though global consistency does not hold. These
observations can be compared to the pair-plots (right part of Figure 2) which
shows the estimated posterior density over the entire $\bm{\theta}$-space. We
observe that gain values at the boundary of the prior support ($g_{0}=-20$ and
$g_{0}=20$) induce discrepancies (cf. marginals plotted in the diagonals of
the pair-plots in Figure 2) that are detected by our PIT-based diagnostics.

Figure 2: Local Consistency Analysis. (Left) Diagnostics obtained for every
PIT-covariate, when evaluated for different $x_{0}$, simulated via the JRNMM
with fixed parameters $(C_{0},\mu_{0},\sigma_{0})$ and variable $g_{0}$ in
$[-20,20]$. The test-statistics are plotted as a function of $g_{0}$ and
indicate inconsistent behavior (values are not constantly close to zero).
Below, the PP-plots report the nature (bias/dispersion) of these
inconsistencies for $g_{0}\in\\{-20,0,20\\}$. The gray zone indicates the
$95\%$-confidence region of acceptance outside of which the uniformity test is
rejected. (Right) Pair-plots of $q_{\phi}(\bm{\theta}\mid x_{0})$ with ground-
truth parameters $\bm{\theta}_{0}=(C_{0},\mu_{0},\sigma_{0},g_{0})$ (black
dots and dashed lines) for $g_{0}\in\\{-20,0,20\\}$.

The above results are obtained by directly applying the method from [19] using
their default regressor, MLPClassifier from scikit-learn [15]. Indeed, this
model is well suited for the binary target variables $W^{i,\alpha}$ and, based
on neural networks, is able to scale to high-dimensional $x$-spaces.
Furthermore, it has been shown [12] that for a large enough dataset
$\mathcal{D}$ (and number of training iterations), the cross-entropy loss
converges to the optimal solution of our regression problem. Note that
although such method allows to capture and visualize local discrepancies, it
does not cope well with our high-dimensional, complex data-distributions (high
variances represented by large confidence regions computed over $100$ trials).
Also, it requires the training of a large number ($m\times|G|$) of regressors
(one for each $\bm{\theta}$-dimension and $\alpha$-value). Finally, this
method is only applicable for the uniformity tests. We are currently
investigating algorithms that are computationally more efficient and more
accurate in estimating the _full_ multivariate local PIT-distribution.

Conclusions. Numerical illustrations demonstrate that our diagnostics capture
well the inconsistencies in $q_{\phi}$ with respect to the true JRNMM
posterior. To be precise, the proposed validation method is statistically more
powerful and computationally more efficient than SBC and, importantly, it
allows for a local analysis that reveals where in $x$-space the estimations
should be improved. Our method exploits useful properties of modern
normalizing flows. Indeed, contrarily to other SBI-strategies [7], they allow
for efficient density evaluation and use bijective transformations involving
the mixing between elements of $\bm{\theta}$. These transformations can be
interpreted as a set of 1D-projections of $\bm{\theta}$, including information
about its joint p.d.f. (i.e. interactions between its elements $\theta_{i}$).
The covariates of our multivariate PIT therefore define fast and easy to
compute 1D test-quantities on which we can perform univariate LCTs that do not
completely ignore correlations in the $\bm{\theta}$-space. Combined with a
test to check their mutual independence, this would provide theoretical
guarantees for consistency, which is not the case for HPD, the existing
multivariate version of LCT proposed in [19].

Ongoing work will investigate the link between PIT-covariates and their
deviations from uniformity with the actual parameters of the model. This could
reveal the true nature of their coupling which can be used for the development
of better specified models (e.g. hierarchical posterior estimation [16, 17]).
Ultimately, the goal is to use our diagnostics as a tool for model selection.

## Broader Impact

This work tackles an important open question in simulation based inference. We
introduce theoretically valid and interpretable validation diagnostics that
scale to both high-dimensional data and parameter spaces. Our contribution
should help to further improve SBI methods and drive the design of better
specified models, hence allowing to build up trust on their ability to address
important questions in experimental science. Moreover, our work is in line
with many other tentatives in the machine learning community of ensuring the
quality and calibration of complex models based on neural networks.

## Acknowledgments

Julia Linhart is recipient of the Pierre-Aguilar Scholarship and thankful for
the funding of the Capital Fund Management (CFM). Alexandre Gramfort thanks
the support of the ANR BrAIN (ANR-20-CHIA0016) grant.

## References

  * Bittner et al. [2021] Sean R. Bittner, Agostina Palmigiano, Alex T. Piet, Chunyu A. Duan, Carlos D. Brody, Kenneth D. Miller, and John Cunningham.  Interrogating theoretical models of neural computation with emergent property inference.  _eLife_ , 10, 7 2021.  ISSN 2050084X.  doi: 10.7554/eLife.56265.
  * Bland and Altman [1995] J. Martin Bland and Douglas G. Altman.  Multiple significance tests: the bonferroni method.  _BMJ (Clinical research ed.)_ , 310:170, 1 1995.  ISSN 0959-8138.  doi: 10.1136/BMJ.310.6973.170.
  * Boelts et al. [2022] Jan Boelts, Jan-Matthis Lueckmann, Richard Gao, and Jakob H. Macke.  Flexible and efficient simulation-based inference for models of decision-making.  _bioRxiv_ , page 2021.12.22.473472, 6 2022.  ISSN 2050084X.  doi: 10.1101/2021.12.22.473472.
  * Cranmer et al. [2020] Kyle Cranmer, Johann Brehmer, and Gilles Louppe.  The frontier of simulation-based inference.  _Proceedings of the National Academy of Sciences (PNAS)_ , 117:30055–30062, 2020.  ISSN 0027-8424.  doi: 10.1073/pnas.1912789117.
  * Dalmasso et al. [2021] Niccolò Dalmasso, David Zhao, Rafael Izbicki, and Ann B. Lee.  Likelihood-free frequentist inference: Confidence sets with correct conditional coverage.  7 2021.  doi: 10.48550/arxiv.2107.03920.
  * Gonçalves et al. [2020] Pedro J. Gonçalves, Jan Matthis Lueckmann, Michael Deistler, Marcel Nonnenmacher, Kaan Öcal, Giacomo Bassetto, Chaitanya Chintaluri, William F. Podlaski, Sara A. Haddad, Tim P. Vogels, David S. Greenberg, and Jakob H. Macke.  Training deep neural density estimators to identify mechanistic models of neural dynamics.  _eLife_ , 9:1–46, 9 2020.  ISSN 2050084X.  doi: 10.7554/ELIFE.56261.
  * Hermans et al. [2020] Joeri Hermans, Volodimir Begy, and Gilles Louppe.  Likelihood-free MCMC with amortized approximate ratio estimators.  In Hal Daumé III and Aarti Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 4239–4248. PMLR, 13–18 Jul 2020.  doi: 10.48550/arxiv.1903.04057.
  * Jansen and Rit [1995] Ben H. Jansen and Vincent G. Rit.  Electroencephalogram and visual evoked potential generation in a mathematical model of coupled cortical columns.  _Biological Cybernetics 1995 73:4_ , 73:357–366, 9 1995\.  ISSN 1432-0770.  doi: 10.1007/BF00199471.
  * Kim et al. [2018] Ilmun Kim, Ann B. Lee, and Jing Lei.  Global and local two-sample tests via regression.  _Electronic Journal of Statistics_ , 13:5253–5305, 12 2018\.  ISSN 19357524.  doi: 10.48550/arxiv.1812.08927.
  * Lueckmann et al. [2021] Jan-Matthis Lueckmann, Jan Boelts, David S Greenberg, Pedro J Gonçalves, and Jakob H Macke.  Benchmarking simulation-based inference.  _Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (PMLR)_ , 130:343–351, 4 2021.  doi: 10.48550/arxiv.2101.04653.
  * Masserano et al. [2022] Luca Masserano, Tommaso Dorigo, Rafael Izbicki, Mikael Kuusela, and Ann B. Lee.  Simulation-based inference with waldo: Perfectly calibrated confidence regions using any prediction or posterior estimation algorithm.  5 2022.  doi: 10.48550/arxiv.2205.15680.
  * Miller et al. [1993] John W. Miller, Rod Goodman, and Padhraic Smyth.  On loss functions which minimii to conditional expected values and posterior probabilities.  _IEEE TRANSACTIONS ON INFORMAITON THEORY_ , 39, 1993.
  * Papamakarios et al. [2017] George Papamakarios, Theo Pavlakou, and Iain Murray.  Masked autoregressive flow for density estimation.  _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 2339–2348, 12 2017.  ISSN 10495258.  doi: 10.48550/arxiv.1705.07057.
  * Papamakarios et al. [2021] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan.  Normalizing flows for probabilistic modeling and inference.  _Journal of Machine Learning Research_ , 22:1–64, 2021\.  ISSN 15337928.  doi: 10.48550/arxiv.1912.02762.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Rodrigues et al. [2021] Pedro L C Rodrigues, Thomas Moreau, and Gilles Louppe.  Hnpe: Leveraging global parameters for neural posterior estimation.  _Neural Information Proceedings (NeurIPS)_ , 12 2021.  ISSN 10495258.  doi: 10.48550/arxiv.2102.06477.
  * Rouillard and Wassermann [2022] Louis Rouillard and Demian Wassermann.  ADAVI: Automatic Dual Amortized Variational Inference Applied To Pyramidal Bayesian Models.  In _ICLR 2022_ , Virtual, France, April 2022.  doi: 10.48550/arXiv.2106.12248.
  * Talts et al. [2018] Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman.  Validating bayesian inference algorithms with simulation-based calibration.  4 2018.  doi: 10.48550/arxiv.1804.06788.
  * Zhao et al. [2021] David Zhao, Niccolò Dalmasso, Rafael Izbicki, and Ann B. Lee.  Diagnostics for conditional density models and bayesian inference algorithms.  _PMLR_ , 2 2021.  ISSN 2640-3498.
  * Álvaro Tejero-Cantero et al. [2020] Álvaro Tejero-Cantero, Jan Boelts, Michael Deistler, Jan-Matthis Lueckmann, Conor Durkan, Pedro J Gonçalves, David S Greenberg, Jakob H Macke, Computational Neuroengineering, and T U Munich.  Sbi – a toolkit for simulation-based inference.  _Journal of Open Source Software_ , 5:2505, 7 2020.  ISSN 2475-9066.  doi: 10.21105/joss.02505.

##  Appendix A Appendix

Proof of Theorem 1. Let $x\in\mathbb{R}^{d}$. We consider the random variables
defined on $\mathbb{R}^{m}$: $\Theta\sim p(\bm{\theta}\mid x)$ and
$\Theta_{\phi}\sim q_{\phi}(\bm{\theta}\mid x)$. Our local null hypothesis
(cf. Equation 1), can be rewritten as

$\mathcal{H}_{0}(x)~{}:~{}\text{Prob}\Big{(}\Theta\in{\Omega_{\theta}}\Big{)}=\text{Prob}\Big{(}\Theta_{\phi}\in{\Omega_{\theta}}\Big{)}\quad\forall\Omega_{\theta}\subset\mathbb{R}^{m}$
(5)

From the definition of our normalizing flow $q_{\phi}$ with bijective
transformation $T_{\phi}$ and normal base distribution, there exists $Z\sim
p(\mathbf{z})=\mathcal{N}(0,\mathbf{I}_{m})$, such that
$\Theta_{\phi}=T_{\phi}(Z,x)$.

Replacing $\Theta_{\phi}$ by $T_{\phi}(Z,x)$ in (5) and defining
$\Omega_{z}=T_{\phi}^{-1}(\Omega_{\theta};x)$, we have

$\begin{array}[]{rcl}\text{Prob}\Big{(}\Theta\in{\Omega_{\theta}}\Big{)}=\text{Prob}\Big{(}T_{\phi}(Z;x)\in{\Omega_{\theta}}\Big{)}&\iff&\text{Prob}\Big{(}T_{\phi}^{-1}(\Theta;x)\in\Omega_{z}\Big{)}=\text{Prob}\Big{(}Z\in\Omega_{z}\Big{)}\end{array}$

Considering $T^{-1}_{\phi,i}$, the $i$-th coordinate of $T^{-1}_{\phi}$, and
$Z_{1},\dots,Z_{m}$ the mutually independent and normally distributed
covariates of $Z\sim\mathcal{N}(0,\mathbf{I}_{m})$, we can rewrite the right
part of the above equivalence as:

$\text{Prob}\Big{(}T_{\phi,1}^{-1}(\Theta;x)\in\Omega_{z,1}\cap\dots\cap
T_{\phi,m}^{-1}(\Theta;x)\in\Omega_{z,m}\Big{)}=\displaystyle\prod_{i=1}^{m}\text{Prob}\Big{(}Z_{i}\in\Omega_{z,i}\Big{)}$
(6)

where the $\Omega_{z,i}$ are projections of the set $\Omega_{z}$ on each of
its dimensions.

Note that since the above equivalence is true for any choice of
$\Omega_{\theta}$, hence of $\Omega_{z}$, we can write what happens when we
fix a given $\Omega_{z,1}$ and let
$\Omega_{z,2}=\dots=\Omega_{z,m}=\mathbb{R}$:

$\text{Prob}\Big{(}T_{\phi,1}^{-1}(\Theta;x)\in\Omega_{z,1}\Big{)}=\text{Prob}\Big{(}Z_{1}\in\Omega_{z,1}\Big{)}$

Doing the same for all other coordinates and using these equalities in (6), we
end up with

$\displaystyle\forall
i\in[1,m],\quad\forall\Omega_{z,i}\subset\mathbb{R},\quad\text{Prob}\Big{(}T_{\phi,i}^{-1}(\Theta;x)\in\Omega_{z,i}\Big{)}$
$\displaystyle=\text{Prob}\Big{(}Z_{i}\in\Omega_{z,i}\Big{)}$ (7)
$\displaystyle\text{and}\quad\text{Prob}\Big{(}T_{\phi,1}^{-1}(\Theta;x)\in\Omega_{z,1}\cap\dots\cap
T_{\phi,m}^{-1}(\Theta;x)\in\Omega_{z,p}\Big{)}$
$\displaystyle=\displaystyle\prod_{i=1}^{m}\text{Prob}\Big{(}T_{\phi,i}^{-1}(\Theta;x)\in\Omega_{z,i}\Big{)}$
(8)

where (8) is the definition of mutual independence itself.

We now apply the c.d.f. $F_{\mathcal{N}(0,1)}$ to each random variable in (7)
and (8). Mutual independence stays true and the probability integral transform
theorem (in 1D) states that $F_{\mathcal{N}(0,1)}(Z_{i})\sim\mathcal{U}(0,1)$.
We therefore get that the null hypothesis $\mathcal{H}_{0}(x)$ holds if, and
only if,

$\left\\{\begin{array}[]{ll}P_{i}(\Theta,x)=F_{\mathcal{N}(0,1)}(T_{\phi,i}^{-1}(\Theta;x))\sim\mathcal{U}(0,1),\quad\forall
i=[1,m]\quad\textit{and}\\\\[5.0pt]
\\{P_{i}(\Theta,x)\\}_{i=1,\dots,m}\quad\textit{are mutually
independent}\end{array}\right.$ (9)

The result in Theorem 1 directly follows from rewriting (5) and (9) with
initial notations from Section 2: remember that $\Theta\sim p(\bm{\theta}\mid
x)$, so $P_{i}(\Theta,x)\sim p(P_{i}(\bm{\theta},x)\mid x)$ and we get:

$p(\bm{\theta}\mid x)=q_{\phi}(\bm{\theta}\mid
x)\iff\left\\{\begin{array}[]{ll}p(P_{i}(\bm{\bm{\theta}},x)\mid
x)=\mathcal{U}(0,1)\quad\forall i\in[1,m]\quad\textit{and}\\\\[5.0pt]
\\{P_{i}(\bm{\theta},x)\mid x\\}_{i=1,\dots,m}\quad\textit{are mutually
independent}\end{array}\right.$

where $P_{i}(\bm{\theta},x)$ is the $i^{th}$ covariate of
PIT${}_{m}(\bm{\theta},x,q_{\phi})$, the multivariate PIT of $\bm{\theta}$ at
$x$, associated to $q_{\phi}$.

Conclusion: The null hypothesis $\mathcal{H}_{0}(x)$ holds if, and only if,
the covariates of PITm conditioned on $x$ are mutually independent and
uniformly distributed over $(0,1)$.


# ON C3-LIKE FINSLER METRICS UNDER RICCI FLOW

Ranadip Gangopadhyay and Bankteshwar Tiwari
DST-CIMS, Institute of Science, Banaras Hindu University,
Varanasi-221005, India

###### Abstract

In this paper we have studied the class of Finsler metrics, called C3-like
metrics which satisfy the un-normal and normal Ricci flow equation and proved
that such metrics are Einstein.

This paper is accepted for publication in Bulletin of the Transilvania
University of Brasov Series III - Mathematics and Computer Science

##  1 Introduction

The Ricci flow theory, which was introduced by Richard Hamilton, became a very
powerful method in understanding the geometry and topology of Riemannian
manifolds. Finally in the first decade of twenty first century Perelman able
to classify the three dimensional smooth manifolds using Ricci flow equation
[5],[6],[7]. The Ricci flow equation is the evolution equation

$\frac{d}{dt}g_{ij}=-2Ric_{ij}$,

for a Riemannian metric $g_{ij}$, where $Ric_{ij}$ is the Ricci curvature
tensor. Hamilton showed that there is a unique solution to this equation for
an arbitrary smooth metric on a closed manifold over a sufficiently short time
[3],[4].
The concept of Einstein metric is also very important topic in differential
geometry as well as in physics. A Riemannian manifold $(M,g)$ is called
Einstein if it’s Ricci tensor $Ric$ can be given by $Ric=kg$, where $k$ is
some constant. It is named after Albert Einstein because this condition is
equivalent to saying that the metric is a solution of the vacuum Einstein
field equations. A Finsler manifold $(M,F)$ is called Einstein if there exist
a scalar function $K(x)$ on $M$ such that $Ric=(n-1)KF^{2}$.
Finsler metrics satisfying Ricci flow equation has been an important topic of
research. For instance Sadegzadeh and Razavi have studied C-reducible metrics
satisfying Ricci flow equation [9], where as Tayebi, Payghan and Najafi have
studied semi C-reducible Finsler metrics satisfying Ricci flow equation [10]
and proved that such metrics are Einstein when they satisfies the Ricci flow
equation.
In the present paper we have extended those results to a more general class of
Finsler metric, namely C3-like Finsler metric, introduced by Prasad and Singh
[8]. More precisely we have prove the following theorems:

###### Theorem 1.1

Let $(M,F)$ be a Finsler manifold of dimension $n\geq 3$. Suppose that $F$ be
a C3-like Finsler metric, then every deformation $F_{t}$ of the metric $F$
satisfying un-normal Ricci flow equation is an Einstein metric.

###### Theorem 1.2

Let $(M,F)$ be a Finsler manifold of dimension $n\geq 3$. Suppose that $F$ be
a C3-like Finsler metric, then every deformation $F_{t}$ of the metric $F$
satisfying normal Ricci flow equation is an Einstein metric.

##  2 Preliminaries

Let $M$ be an $n$-dimensional smooth manifold and $T_{x}M$ denotes the tangent
space of $M$ at $x$. The tangent bundle of $M$ is the union of tangent spaces
$TM:=\bigcup_{x\in M}T_{x}M$. We denote the elements of $TM$ by $(x,y)$ where
$y\in T_{x}M$. Let $TM_{0}:=TM\setminus\left\\{0\right\\}$.

###### Definition 2.1

[2] A Finsler metric on $M$ is a function $F:TM\rightarrow[0,\infty)$ with the
following properties:
(i) $F$ is $C^{\infty}$ on $TM_{0}$,
(ii) $F$ is a positively 1-homogeneous on the fibers of tangent bundle $TM$.
(iii) The Hessian of $\frac{F^{2}}{2}$ with element
$g_{ij}=\frac{1}{2}\frac{\partial^{2}F^{2}}{\partial y^{i}\partial y^{j}}$ is
positive definite on $TM_{0}$.
The pair $(M,F)$ is called a Finsler space, $F$ is called the fundamental
function and $g_{ij}$ is called the fundamental tensor.

Let $\left(M,F\right)$ be a Finsler space. For a vector $y\in
T_{x}M\setminus\left\\{0\right\\}$, let

$C_{y}(u,v,w):=\frac{1}{4}\frac{\partial^{3}}{\partial s\partial t\partial
r}[F^{2}(y+su+tv+rw)]_{s=t=r=0}$

where $u,v,w\in T_{x}M$. Each $C_{y}$ is a symmetric trilinear form on $V$. We
call the family $C:=\left\\{C_{y}:y\in
T_{x}M\setminus\left\\{0\right\\}\right\\}$ the Cartan torsion. Let
$\left\\{b_{i}\right\\}$ be a basis for $T_{x}M$ and define
$g_{ij}:=g_{y}(b_{i},b_{j})$, $C_{ijk}:=C_{y}(b_{i},b_{j},b_{k})$.
Then
$C_{ijk}=\frac{1}{4}[F^{2}]_{y^{i}y^{j}y^{k}}=\frac{1}{2}\frac{\partial}{\partial
y^{k}}(g_{ij})$.
Define the mean value of the Cartan torsion by

$I_{y}(u):=\Sigma_{i=1}^{n}g^{ij}(y)C_{y}(u,b_{i},b_{j})$ , $u\in T_{x}M$.

We call the family $I:=\left\\{I_{y}|y\in
T_{x}M\setminus\left\\{0\right\\}\right\\}$ the mean Cartan torsion at $x\in
M$

i.e. $I_{i}=g^{jk}C_{ijk}$.

###### Definition 2.2

[8] A Finsler metric $F$ is called C3-like if its Cartan torsion is given by

$C_{ijk}=\left\\{a_{i}h_{jk}+a_{j}h_{ki}+a_{k}h_{ij}\right\\}+\left\\{b_{i}I_{j}I_{k}+b_{j}I_{i}I_{k}+b_{k}I_{i}I_{j}\right\\},$
(1)

where $a_{i}=a_{i}(x,y)$ and $b_{i}=b_{i}(x,y)$ are covectors on $TM$ and of
degree $-1$ and $1$, respectively and $h_{ij}$ is angular metric tensor given
by $h_{ij}=F\frac{\partial^{2}F}{\partial y^{i}\partial y^{j}}$.

In particular :
(i) if $a_{i}=0$, then we have
$C_{ijk}={b_{i}I_{j}I_{k}+b_{j}I_{i}I_{k}+b_{k}I_{i}I_{j}}$, contracting it
with $g^{ij}$ implies that $b_{i}=\frac{1}{3I^{2}}I^{i}$, then $F$ is a
C2-like Finsler metric,
(ii) if $b_{i}=0$, then we have $C_{ijk}=a_{i}h_{jk}+a_{j}h_{ki}+a_{k}h_{ij}$,
contracting it with $g^{ij}$ implies that $a_{i}=\frac{1}{n+1}I_{i}$, then $F$
is a C-reducible Finsler metric.
(iii) if $a_{i}=\frac{p}{n+1}I_{i}$ and $b_{i}=\frac{q}{3I^{2}}I_{i}$, where
$p=p(x,y)$ and $q=q(x,y)$ are scalar functions on $TM$, then $F$ is a semi
C-reducible Finsler metric.
Therefore, a C3-like Finsler metric may be consider as a generalization of
C-reducible, semi C-reducible and C2-like Finsler metrics.

##  3 Un-normal Ricci flow equation on C3-like Finsler metrics

The geometric evolution equation

$\frac{d}{dt}g_{ij}=-2Ric_{ij},$ (2)

is known as the un-normalized Ricci flow in Riemannian geometry. In principle,
the same equation can be used in Finsler setting, because both $g_{ij}$ and
$Ric_{ij}$ have been generalised to the broader framework, albeit gaining a
$y$ dependence in the process. However, there are two reasons why we shall
refrain from doing so.
(i) Not every symmetric covariant 2-tensor $g_{ij}(x,y)$ arises from a Finsler
metric $F(x,y)$.
(ii) There is more than one geometrical context in which $g_{ij}$ makes sense.

A deformation of Finsler metrics means a $1$-parameter family of metrics
$g_{ij}(x,y,t)$, such that $t\in[-\epsilon,\epsilon]$ and $\epsilon>0$ is
sufficiently small. For such a metric $\omega=u_{i}dx^{i}$, the volume element
as well as the connections attached to it depend on $t$. The same equation can
be used in the Finsler setting. But instead of the above tensor evolution
equation, we will use a different form of the above equation. By contracting
$\frac{d}{dt}g_{ij}=-2Ric_{ij}$ with $y^{i}$ and $y^{j}$ respectively and
using Eulers theorem, we get

$\frac{\partial F^{2}}{\partial t}=-2F^{2}R$,

where $R=\frac{Ric}{F^{2}}$. That is,

$d\log F=-R,F(t=0)=F_{0}.$

This scalar equation directly addresses the evolution of the Finsler metric
$F$, and makes geometrical sense on both the manifold of nonzero tangent
vectors $TM_{0}$ and the manifold of rays. It is therefore suitable as an un-
normalized Ricci flow for Finsler geometry. In this section we will study
C3-like Finsler metrics satisfying un-normal Ricci flow equation.
Let us assume that $F_{t}$ be a deformation of C3-like Finsler metrics which
satisfies the Un-Normalised Ricci flow equation given by

$g_{ij}^{\prime}=-2Ric_{ij}\quad\textnormal{or}\quad\frac{F^{\prime}}{F}=-R,\quad\textnormal{where}\quad
g_{ij}^{\prime}=\frac{\partial g_{ij}}{\partial t}$

Before proving Theorem 1.1 we need the following lemmas:

###### Lemma 3.1

If $F_{t}$ be a deformation of a C3-like Finsler metrics $F$ on manifold $M$
of dimension $n\geq 3$, then the variation of Cartan torsion is given by

$C_{ijk}^{\prime}I^{i}I^{j}I^{k}=3(a_{i}+3b_{i})I^{i}\|I\|^{2}+\frac{1}{2}F^{2}R_{,i,j,k}I^{i}I^{j}I^{k}+3\
\|I\|^{2}I_{m}R_{,m},$ (3)

where $R_{,m}=\frac{\partial R}{\partial y^{m}}.$

### Proof:

Let us assume that $F_{t}$ be a deformation of a Finsler metric F on a
2-dimensional manifold $M$ satisfies Ricci flow equation given by (2).
By definition of Ricci tensor, we have

$\displaystyle Ric_{ij}$ $\displaystyle=$
$\displaystyle\frac{1}{2}[RF^{2}]_{y^{i}y^{j}}$ (4) $\displaystyle=$
$\displaystyle Rg_{ij}+\frac{1}{2}F^{2}R_{,i,j}+R_{,i}y_{j}+R_{,j}y_{i}$

where $R_{,i}=\frac{\partial R}{\partial y^{i}}$ and
$R_{,i,j}=\frac{\partial^{2}R}{\partial y^{i}\partial y^{j}}$.
Taking vertical derivative of (4) and from the fact $y_{i,j}=g_{ij}$ and
$FF_{k}=y_{k}$ we have

$\displaystyle
Ric_{ij,k}=2RC_{ijk}+\frac{1}{2}F^{2}R_{,i,j,k}+(g_{jk}R_{,i}+g_{ij}R_{,k}+g_{ki}R_{,j})$
$\displaystyle+\left\\{R_{,j,k}y_{i}+R_{,i,j}y_{k}+R_{,k,i}y_{j}\right\\}.$
(5)

Multiplying (3) with $I^{i}I^{j}I^{k}$ and using $y_{i}I^{i}=y^{i}I_{i}=0$ we
obtain

$Ric_{ij,k}I^{i}I^{j}I^{k}=2RC_{ijk}I^{i}I^{j}I^{k}+\frac{1}{2}F^{2}R_{,i,j,k}I^{i}I^{j}I^{k}+3\
\|I\|^{2}I_{m}R_{,m}$ (6)

Now multiplying (1) by $I^{i}I^{j}I^{k}$ we have

$C_{ijk}I^{i}I^{j}I^{k}=3(a_{i}+3b_{i})I^{i}\|I\|^{2}.$ (7)

Since $F_{t}$ satisfies the Ricci flow equation so using (7) we have

$C_{ijk}^{\prime}I^{i}I^{j}I^{k}=3(a_{i}+3b_{i})I^{i}\|I\|^{2}+\frac{1}{2}F^{2}R_{,i,j,k}I^{i}I^{j}I^{k}+3\
\|I\|^{2}I_{m}R_{,m}.$ (8)

$\square$

###### Lemma 3.2

If $F_{t}$ be a deformation of C3-like Finsler metrics $F$ on a manifold $M$
of dimension $n\geq 3$, then $C_{ijk}^{\prime}I^{i}I^{j}I^{k}$ is divisible by
$\|I\|^{2}$.

### Proof:

Since $g^{ij}g_{jk}=\delta_{k}^{i}$, then differentiating with respect to $t$
and using $g^{\prime}_{ij}=-2Ric_{ij}$ we have $g^{\prime il}=2Ric^{il}$.
At first we will calculate the value of $I_{i}^{\prime}$

$\displaystyle I_{i}^{\prime}$ $\displaystyle=$
$\displaystyle(g^{jk}C_{ijk})^{\prime}$ (9) $\displaystyle=$
$\displaystyle(g^{jk})^{\prime}C_{ijk}+g^{jk}C_{ijk}^{\prime}$
$\displaystyle=$ $\displaystyle 2Ric^{jk}C_{ijk}-g^{jk}Ric_{jk,i}$
$\displaystyle=$ $\displaystyle
Ric^{jk}g_{jk,i}-(g^{jk}Ric_{jk})_{,i}+g^{jk}_{,i}Ric_{jk}$ $\displaystyle=$
$\displaystyle-(g^{jk}Ric_{jk}),i$ $\displaystyle=$ $\displaystyle-\rho_{i}$

where $\rho:=g^{jk}Ric_{jk}$ and $\rho_{i}=\frac{\partial\rho}{\partial
y^{i}}$.
Since $y_{i}:=FF_{y^{i}}$, differentiating with respect to $t$ we have

$y_{i}^{\prime}=-2Ric_{im}y^{m}$.

Now we compute $h_{ij}^{\prime}$.

$\displaystyle h_{ij}^{\prime}$ $\displaystyle=$
$\displaystyle(g_{ij}-F^{-2}y_{i}y_{j})^{\prime}$ (10) $\displaystyle=$
$\displaystyle
g_{ij}^{\prime}-\left[-2F^{-3}F^{\prime}y_{i}y_{j}+F^{-2}y_{i}^{\prime}y_{j}+F^{-2}y_{i}y_{j}^{\prime}\right]$
$\displaystyle=$
$\displaystyle-2Ric_{ij}+2F^{-2}\frac{F^{\prime}}{F}y_{i}y_{j}-F^{-2}(y_{i}^{\prime}y_{j}+y_{i}y_{j}^{\prime})$
$\displaystyle=$
$\displaystyle-2Ric_{ij}+2F^{-2}\frac{F^{\prime}}{F}y_{i}y_{j}+2F^{-2}(Ric_{im}y_{j}+Ric_{jm}y_{i})y^{m}$
$\displaystyle=$
$\displaystyle-2Ric_{ij}-2Rl_{i}l_{j}+2(Ric_{im}l_{j}+Ric_{jm}l_{i})l^{m}$
$\displaystyle=$
$\displaystyle-2Ric_{ij}+2R(h_{ij}-g_{ij})+2(Ric_{im}l_{j}+Ric_{jm}l_{i})l^{m}$

where $l_{i}=\frac{y_{i}}{F}$.
Multiplying (1) by $g^{ij}$ we have

$a_{i}=\frac{1}{n+1}\left\\{\left(1-2I^{m}b_{m}\right)I_{i}-\|I^{2}\|b_{i}\right\\}.$
(11)

After differentiating (1) with respect to $t$ we have

$\displaystyle
C_{ijk}^{\prime}=(a_{i}h_{jk}^{\prime}+a_{j}h_{ki}^{\prime}+a_{k}h_{ij}^{\prime})+(a_{i}^{\prime}h_{jk}+a_{j}^{\prime}h_{ki}+a_{k}^{\prime}h_{ij})\hskip
56.9055pt$
$\displaystyle-(b_{i}(\rho_{j}I_{k}+I_{j}\rho_{k})+b_{j}(\rho_{i}I_{k}+I_{i}\rho_{k})+b_{k}(\rho_{i}I_{j}+I_{i}\rho_{j})).$
(12)

Now contracting (3) with $I^{i}I^{j}I^{k}$ we have

$C_{ijk}^{\prime}I^{i}I^{j}I^{k}=\left(a_{i}h^{\prime}_{jk}+a_{j}h^{\prime}_{ki}+a_{k}h^{\prime}_{ij}\right)I^{i}I^{j}I^{k}+3a^{\prime}_{i}I^{i}\|I\|^{2}-6b_{i}\rho_{j}\|I\|^{2}I^{i}I^{j}.$
(13)

Using (10) and (11) in (13) we obtain

$C_{ijk}^{\prime}I^{i}I^{j}I^{k}=\frac{-6}{n+1}Ric_{ij}I^{i}I^{j}\left(1-3b_{m}I^{m}\right)\|I^{2}\|+3a^{\prime}_{i}I^{i}\|I^{2}\|-6b_{i}\rho_{j}\|I^{2}\|I^{i}I^{j}.$
(14)

Therfore the result follows. $\square$

Proof of Theorem 1.1

In the view of (8) and (14) it follows that $R_{,i,j,k}I^{i}I^{j}I^{k}$ is
divisible by $\|I^{2}\|$. Then we can write

$\displaystyle R_{,i,j,k}=A_{ij}I_{k}+B_{i}g_{jk}.$ (15)

By contracting $R_{,i,j,k}$ with $y^{j}$ or $y^{k}$ we have $R_{,i}=0$. So
$F_{t}$ is Einstein.

##  4 Normal Ricci flow equation on C3-like Finsler metrics

If $M$ is compact, then so is $SM$, and we can normalize the above equation by
requiring that the flow keeps the volume of $SM$ constant. Recalling the
Hilbert form $\omega:=F_{y^{i}}dx^{i}$, that volume is

$Vol_{SM}:=\int_{SM}\frac{(-1)^{\frac{(n-1)(n-2)}{2}}}{(n-1)!}\omega\wedge(d\omega)^{(n-1)}:=\int_{SM}dV_{SM}.$

During the evolution, $F$, $\omega$ and consequently the volume form $dV_{SM}$
and the volume $Vol_{SM}$, all depend on $t$. On the other hand, the domain of
integration $SM$, being the quotient space of $TM_{0}$ under the equivalence
relation $z\equiv y$, $z=\lambda y$ for some $\lambda>0$, is totally
independent of any Finsler metric and hence does not depend on $t$. We have
from insights in [1]

$\frac{d}{dt}(dV_{SM})=\left[g_{ij}-g_{ij}^{\prime}-n\frac{d}{dt}\log
F\right]dV_{SM}$.

A normalized Ricci flow for Finsler metrics is proposed by Bao and it is given
by

$\frac{d}{dt}\log F=-R+\frac{1}{Vol(SM)}\int_{SM}RdV,\quad F(t=0)=F_{0},$ (16)

where the given manifold $M$ is compact. Now, we let $Vol(SM)=1$. Then all of
Ricci-constant metrics are exactly the fixed points of the above flow. Let

$Ric_{ij}=\frac{1}{2}(F^{2}R)_{.y^{i}.y^{j}}.$ (17)

Differentiating (16) with respect to $y^{i}$ and $y^{j}$ we have,

$\frac{d}{dt}(g_{ij})=-2Ric_{ij}+\frac{2}{Vol(SM)}\int_{SM}RdVg_{ij},\quad
g(t=0)=g_{0}.$ (18)

Starting with any familiar metric on M as the initial data $F_{0}$, we may
deform it using the proposed normalized Ricci flow, in the hope of arriving at
a Ricci constant metric.
Proof of Theorem 1.2
Here we consider Finsler manifolds that satisfies the normal Ricci flow
equation. Then ,

$\displaystyle\frac{dg_{ij}}{dt}=-2Ric_{ij}+2\int_{SM}RdVg_{ij},\quad d(\log
F)=\frac{F^{\prime}}{F}=-R+\int_{SM}RdV.$ (19)

In this case,

$y_{i}^{\prime}=\left(-2Ric_{im}+2\int_{SM}RdVg_{im}\right)y^{m}.$ (20)

Therefore,

$\displaystyle I_{i}^{\prime}$ $\displaystyle=$
$\displaystyle(g^{jk}C_{ijk})^{\prime}$ (21) $\displaystyle=$
$\displaystyle(g^{jk})^{\prime}C_{ijk}+g^{jk}C_{ijk}^{\prime}$
$\displaystyle=$ $\displaystyle
2\left[Ric^{jk}-\int_{SM}RdVg^{jk}]C_{ijk}+g^{jk}[Ric_{jk,i}+2\int_{SM}RdVC_{ijk}\right]$
$\displaystyle=$ $\displaystyle
2Ric^{jk}g_{jk,i}-(g^{jk}Ric_{jk})_{,i}+g^{jk}_{,}Ric_{jk}$ $\displaystyle=$
$\displaystyle-(g^{jk}Ric_{jk}),i$ $\displaystyle=$ $\displaystyle-\rho_{i}.$

And

$\displaystyle h_{ij}^{\prime}$ $\displaystyle=$
$\displaystyle(g_{ij}-F^{-2}y_{i}y_{j})^{\prime}$ (22) $\displaystyle=$
$\displaystyle
g_{ij}^{\prime}-\left[-2F^{-3}F^{\prime}y_{i}y_{j}+F^{-2}y_{i}^{\prime}y_{j}+F^{-2}y_{i}y_{j}^{\prime}\right]$
$\displaystyle=$ $\displaystyle
g_{ij}^{\prime}+2F^{-2}\frac{F^{\prime}}{F}y_{i}y_{j}-F^{-2}\left(y_{i}^{\prime}y_{j}+y_{i}y_{j}^{\prime}\right)$
$\displaystyle=$
$\displaystyle-2Ric_{ij}+2\int_{SM}RdVg_{ij}+2\left(-R-\int_{SM}RdV\right)l_{i}l_{j}$
$\displaystyle+2(Ric_{im}l_{j}+Ric_{jm}l_{i})l^{m}$ $\displaystyle=$
$\displaystyle-2Ric_{ij}+2\int_{SM}RdVg_{ij}+2(-R-\int_{SM}RdV)(g_{ij}-h_{ij})$
$\displaystyle+2(Ric_{im}l_{j}+Ric_{jm}l_{i})l^{m}$ $\displaystyle=$
$\displaystyle-2Ric_{ij}-2R(h_{ij}-g_{ij})+2\int_{SM}RdVh_{ij}$
$\displaystyle+2(Ric_{im}l_{j}+Ric_{jm}l_{i})l^{m}.$

Therefore,

$h_{ij}=-2Ric_{ij}-2R(h_{ij}-g_{ij})+2\int_{SM}RdVh_{ij}+2(Ric_{im}l_{j}+Ric_{jm}l_{i})l^{m}.$
(23)

Now from the un-normalised case,

$\displaystyle
C_{ijk}^{\prime}I^{i}I^{j}I^{k}=(a_{i}h_{jk}^{\prime}+a_{j}h_{ki}^{\prime}+a_{k}h_{ij}^{\prime})I^{i}I^{j}I^{k}+(a_{i}^{\prime}h_{jk}+a_{j}^{\prime}h_{ki}+a_{k}^{\prime}h_{ij})I^{i}I^{j}I^{k}$
$\displaystyle-\left\\{b_{i}(\rho_{j}I_{k}+I_{j}\rho_{k})+b_{j}(\rho_{i}I_{k}+I_{i}\rho_{k})+b_{k}(\rho_{i}I_{j}+I_{i}\rho_{j})\right\\}I^{i}I^{j}I^{k}.$
(24)

Now using (22) and (11) we get

$\displaystyle
C^{\prime}_{ijk}I^{i}I^{j}I^{k}=\frac{-6}{n+1}Ric_{ij}I^{i}I^{j}(1-3b_{m}I^{m})\|I^{2}\|+3a_{i}^{\prime}I_{i}\|I^{2}\|$
$\displaystyle-6b_{i}\rho_{j}I^{i}I^{j}\|I^{2}\|+6a_{k}\int_{SM}RdVI^{k}\|I^{2}\|.$
(25)

So by the same arguement of un-normalised case the result follows.

## References

  * [1] Akbar-Zadeh, H., Sur les espaces de Finsler à courbures sectionnelles constantes. Acad. Roy. Belg. Bull. Cl. Sci(5). 74 281-322 (1988)
  * [2] Chern.S.S., Shen.Z., Riemannian-Finsler geometry, World Scientific Publisher, Singapore, 2005.
  * [3] Hamilton, R.S., Three-manifolds with positive Ricci curvature. J. Differential Geom. 17, 255-306 (1982)
  * [4] Hamilton, R.S., Four-manifolds with positive curvature operator. J. Differential Geom. 24, 153-179 (1986)
  * [5] Perelman, G., The entropy formula for the Ricci flow and its geometric applications. arXiv: math.DG/ 0211159
  * [6] Perelman, G., Ricci flow with surgery on three manifolds. arXiv:math.DG/ 03109
  * [7] Perelman, G., Finite extinction time for the solutions to the Ricci flow on certain three manifolds. arXiv: math.DG/ 0307245
  * [8] Prasad, B.N., Singh, J.N., On C3-like Finsler Spaces. Indian J.pure appl. Math. 19(5), 423-428 (1988)
  * [9] Sadeghzadeh, N. and Razavi,A., Ricci flow equation on C-Reducible Metrics. Int. J. Geom. Methods Mod. Phys. 8(4) 773-781 (2011)
  * [10] Tayebi, A., Payghan, E., Najafi, B, Ricci Flow Equation on ($\alpha,\beta$)-metrics. arXiv:1108.0134v1 [math.DG] 31 Jul 2011