# Reconstructing Hand-Object Interactions in the Wild

Zhe Cao∗ Ilija Radosavovic∗ Angjoo Kanazawa Jitendra Malik
University of California  Berkeley

###### Abstract

In this work we explore reconstructing hand-object interactions in the wild.
The core challenge of this problem is the lack of appropriate 3D labeled data.
To overcome this issue, we propose an optimization-based procedure which does
not require direct 3D supervision. The general strategy we adopt is to exploit
all available related data (2D bounding boxes, 2D hand keypoints, 2D instance
masks, 3D object models, 3D in-the-lab MoCap) to provide constraints for the
3D reconstruction. Rather than optimizing the hand and object individually, we
optimize them jointly which allows us to impose additional constraints based
on hand-object contact, collision, and occlusion. Our method produces
compelling reconstructions on the challenging in-the-wild data from the EPIC
Kitchens and the 100 Days of Hands datasets, across a range of object
categories. Quantitatively, we demonstrate that our approach compares
favorably to existing approaches in the lab settings where ground truth 3D
annotations are available.

Figure 1: Reconstructions in the wild. For each row, we show the input image
(top), the reconstructed hand and object in two different viewpoints (bottom).
Our method can achieve compelling results for various object categories in
different interaction scenarios.

00footnotetext: Equal contribution. More information at this project page.

##  1 Introduction

Our hands are the primary way we interact with objects in the world. In turn,
we designed our world with hands in mind. Therefore, understanding hand-object
interactions is an important ingredient for building agents that perceive and
act in the world. For example, it can allow them to learn object affordances
[10], infer human intents [33], and learn manipulation skills from humans [43,
30].

What does it mean to understand hand-object interactions? We argue that
_fully_ capturing the richness of hand-object interactions requires 3D
understanding. In general, recovering 3D from a single RGB image is an
underconstrained problem. In the case of hand-object interactions, the problem
is even more challenging due to heavy occlusions and varied viewpoints that
arise naturally when hands are interacting with objects [39].

Figure 2: Availability of labeled data. Our community has made substantial
progress in hand-object reconstruction. However, the gap between the in-the-
lab datasets with 3D annotations (left) and the unlabeled in-the-wild data
(right) remains considerable. This setting motivates our work.

Overall, there has already been a considerable progress toward building better
models for hand-object reconstruction [18, 16, 17]. Due to the difficulty of
obtaining annotations for this problem, the focus has largely been on
synthetic and in-the-lab datasets. However, the gap between the in-the-lab and
in-the-wild settings remains large (Figure 2). Thus, methods trained on
existing datasets do not typically generalize to in-the-wild settings.

In this paper, we make a step toward the more challenging setting and explore
_reconstructing hand-object interactions in the wild_. We argue that manual 3D
annotation of in-the-wild data is challenging and may not even be necessary.
The general strategy we adopt is to exploit all available related data (2D
bounding boxes, 2D instance masks, 2D keypoints, 3D object models, and 3D in-
the-lab MoCap) to provide constraints for 3D reconstruction. This general
philosophy has proven successful for 2D recognition [42] and human 3D
reconstruction [20].

To tackle this setting, we employ a multi-step optimization procedure whose
components build on prior work [24, 54, 19, 66, 56]. Our method consists of
four steps: (1) optimizing the hand pose using 2D hand keypoints; (2)
optimizing the object pose using differentiable rendering; (3) jointly
optimizing the hand-object arrangement in 3D; and (4) refining the pose
estimation results using a refinement network that captures 3D contact priors.

Our method produces compelling reconstructions on the challenging in-the-wild
data from the EPIC Kitchens [8] and the 100 Days of Hands datasets [49],
across a range of object categories (Figure 1). Quantitatively, our approach
compares favorably to existing approaches in the lab settings where ground
truth 3D annotations are available.

In addition to the supervision already embodied in the collection of the
images we use, our pipeline has an additional step which requires human
supervision, namely, the selection of the 3D CAD model of the object being
manipulated. This extra labelling step is a practical constraint for creating
large datasets of hand-object interactions using our techniques, but we expect
advances in recognition of 3D CAD models from 2D images to help automate this
step.

In summary, our key contributions are: (1) We present an optimization-based
procedure that leverages all available related data for reconstructing hand-
object interactions in the wild; (2) We show promising results on challenging
real-world images; (3) We curate a set of images with corresponding instance
category and 3D CAD models.

##  2 Related Work

3D hand pose estimation. Many previous works on hand pose estimation directly
predict 3D joint locations from either depth [50, 53, 55, 59, 64, 65, 35] or
RGB [45, 36, 4, 63, 68] images. While this representation of sparse hand
joints is effective in modeling the hand action and motion, it does not model
the hand shape and hand-object contacts. In contrast, some previous works
predict 3D hand joint rotations and shape parameters for 3D parametric hand
model such as MANO [46]. Fitting-based approaches [41, 60, 24] fit such
parametric models to 2D keypoint detections to optimize 3D hand parameters.
Learning-based approaches [67, 47] utilize deep networks to directly predict
the MANO hand parameter from RGB image input. Recently, [25, 24] proposes to
use mesh convolution to directly predict 3d hand mesh reconstruction. We use a
learning-based method [47] to obtain the initial hand pose estimation and
further improve the result by imposing constraints on 2D hand keypoints and 3D
hand-object contact priors.

3D object pose estimation. There are many existing works on estimating 3D
object pose from a single image. Some approaches [58, 26, 12, 27] utilize
neural network to predict the object shape, translation, and global
orientation in the camera coordinate. These methods are trained with limited
object categories and have difficulty generalizing to new objects. On the
other hand, some approaches [28, 34, 61, 66, 54] assume known object CAD model
and focus on 6DOF object pose prediction. In this work, we take a fitting-
based approach similar to [21, 54, 48, 66] and we further refine object pose
based on its arrangement with the hand.

Figure 3: Reconstruction procedure. To reconstruct hand-object interactions in
the wild, we leverage all available related data (2D keypoints, 2D instance
masks, 3D object models, 3D in-the-lab MoCap) through an optimization-based
procedure that consists of four steps: (a) hand pose estimation by 2D
keypoints fitting, (b) object pose estimation via differentiable rendering,
(c) joint optimization for spatial arrangement, and (d) pose refinement using
3D contact priors.

3D hand and object pose estimation. Early approaches in modeling hand and
object require the input of multi-view image [40] or RGB-D sequence [59].
Recently, [18] proposes a deep model to reconstruct hand and object meshes
from a monocular RGB image. To train the model with 3D supervision, they
collect a synthetic dataset from simulation. [57] designs a neural network to
jointly predict 3D hand pose and 3D object bounding boxes with a focus on
egocentric scenarios. Given object CAD model as input, [17] proposes to
leverage photometric consistency from temporal frames as additional signal for
training the model with sparse set of annotated data. All these approaches
were trained and tested on in-the-lab or synthetic datasets. In contrast, our
method does not require direct 3D supervision and can be applied to in the
wild images.

3D hand-object datasets. Early datasets in hand grasping scenario requires
manual annotations [52] or depth tracking [59] to obtain the ground truth,
resulting in limited dataset size. To avoid the manual efforts, [9] uses
motion capture system with magnetic sensors to collect hand pose annotation in
RGB-D video sequence. [18] introduces a synthetic dataset of hands grasping
objects collected from simulation. [69, 16] introduces large-scale dataset
with 3D annotation optimized from multi-view setups. Some recent datasets [2,
3, 56] also provide annotation for hand-object contact area in addition to 3D
hand pose and object pose. [3] collects large-scale RGB-D image sequences of
hand-object interaction that are automatically annotated using multi-view
setups and a thermal sensor. [56] captures whole-body human grasping objects
with accurate 3D annotation using marker-based MoCap system. Due to the
requirement of specific camera setup, all the mentioned datasets are collected
in the lab setting. In this work, we utilize 3D hand-object contact priors
learnt from these datasets to further refine our reconstructions for in-the-
wild images.

Hand-object data in the wild. Datasets like EPIC [8], 100 Days of Hands [49],
Something-Something [13], Something-Else [32], MECCANO [44], contain data with
natural hand-object interactions in different scenarios. However, they do not
provide 3D annotations. The goal of our work is to reconstruct hand-object
interactions in such scenarios without requiring direct 3D supervision.

Human-object interactions. A number of previous works study different aspects
of human-object interactions in 2D. Examples include detection [11, 15], video
recognition [62], interaction hotspots [37], and affordances [38]. Recently,
Zhang et al. [66] propose a method to infer 3D spatial arrangements of humans
and objects. In this work, we focus on understanding _hand_ -object
interactions in 3D.

Figure 4: Intermediate results. _Top row:_ input images. _2nd row:_ results
from individually optimizing hand and object. _3rd row:_ results from joint
optimization (two viewpoints per example). _Bottom row:_ results after the
refinement.

##  3 Reconstructing Hand-Object Interactions

In this section, we describe our optimization-based procedure for
reconstruction hand-object interactions from a single RGB image
$\boldsymbol{I}$. Our pipeline, shown in Figure 3, involves 4 steps: we first
optimize the object pose and hand pose individually, then we optimize their 3D
arrangement jointly, and finally we refine the pose estimation result based on
3D contact priors. We show the intermediate results from each step in Figure
4.

We motivate and describe our modular design for each stage in the rest of the
section. We note that while our method can be applied to multiple hands, for
brevity, we describe the optimization procedure for one pair of hand object
interaction in the rest of section.

###  3.1 Hand Pose Estimation

The first step of our pipeline involves hand pose estimation (Figure 3a). We
represent the hand using a parametric model defined by MANO [46]:

$\boldsymbol{V}_{h}=H(\boldsymbol{\theta},\boldsymbol{\beta}),\\\ $ (1)

where $\boldsymbol{\theta}\in\mathbb{R}^{3\times 15}$ are hand pose parameters
and $\boldsymbol{\beta}\in\mathbb{R}^{10}$ are the shape parameters for hand
model.

Given a single RGB image, we use [49] to detect 2D hand bounding box. Taking
the cropped hand patch, we use FrankMocap [47] to estimate the weak-
perspective camera model ${\Pi}_{h}=(t_{x},t_{y},s_{h})$, and initial 3D hand
parameters $\boldsymbol{\theta}$ and $\boldsymbol{\beta}$. We further optimize
the hand pose by a fitting procedure which uses 2D hand keypoints from
OpenPose [6, 51]

The hand pose optimization objective is to minimize the difference between 2D
keypoints detection and 2D projection of 3D hand keypoints [24, 1]:

$\boldsymbol{\theta}^{*},\boldsymbol{\beta}^{*}=\arg\min_{\boldsymbol{\theta},\boldsymbol{\beta}}(L_{2D}+L_{reg}),$
(2)

consisting of a 2D keypoints distance term $L_{2D}$ and a regularization term
$L_{reg}$ for hand shape $\boldsymbol{\beta}$.

We convert the weak-perspective camera to perspective camera by assuming a
fixed focal length $f$. The depth of the hand is approximated by the the focal
length divided by the camera scale $s_{h}$. The final hand vertices are
obtained by:

$\boldsymbol{V}^{*}_{h}=H(\boldsymbol{\theta}^{*},\boldsymbol{\beta}^{*})+[t_{x},t_{y},f/s_{h}],\\\
$ (3)

###  3.2 Object Pose Estimation

In the next step of our method, Figure 3b, we recover the object pose using
differentiable neural mesh renderer [22], similar to [21, 48, 66]. Given an
input image, we first estimate the 2D instance masks $\boldsymbol{M}$ using
PointRend [23]. We further estimate the object scale $s_{o}\in\mathbb{R}$, 3D
rotation $\boldsymbol{R}\in SO(3)$, and translation
$\boldsymbol{T_{o}}\in\mathbb{R}^{3}$ by fitting the corresponding 3D object
model to the 2D object mask $\boldsymbol{M}_{o}$. The object parameters are
then optimized by minimizing the difference between the rendered object mask
and the estimated object mask $\boldsymbol{M}_{o}$.

$s_{o}^{*},\boldsymbol{R}^{*},\boldsymbol{T}^{*}_{o}=\operatorname*{arg\,min}_{s_{o},\boldsymbol{R},\boldsymbol{T_{o}}}\lVert\textrm{NR}(s_{o},\boldsymbol{R},\boldsymbol{T}_{o})-\boldsymbol{M}_{o}\rVert,\\\
$ (4)

where $\textrm{NR}(\cdot)$ denotes the differentiable operation which renders
3D mesh into 2D mask. The final object vertices can be obtained by:

$\boldsymbol{V}_{o}^{*}=s_{o}^{*}(\boldsymbol{R}^{*}\boldsymbol{V}_{o}+\boldsymbol{T}^{*}_{o}).\\\
$ (5)

The optimization is under-constrained. In other words, there are multiple
possible solutions for object pose which may have 2D masks similar to ground-
truth. We randomly initialize the object rotations for multiple times and
select the result with lowest loss after the optimization. We note that more
sophisticated objectives are possible. For example, [66] uses a silhouette
loss with distance transform for optimizing object pose. In our experiments,
we find that using a simple mask difference loss can achieve better results on
smaller objects, such as pen, spoon, and fork.

###  3.3 Joint Optimization

So far we described how we optimize hand pose and object pose individually. To
obtain the final reconstruction, we optimize their 3D arrangement jointly.

Naively putting the hand and object together may result in implausible results
(Figure 4, row 2), i.e., the hand and object are far away from each other in
3D or having interpenetration. Obtaining a coherent 3D arrangement of hand and
object in one shot is challenging due to the depth and scale ambiguity given
only 2D input. For example, a large object distant from the camera can have
the same rendering result in 2D as a smaller object closer to the camera.

To resolve the ambiguity issue, we perform one more step to optimize the hand
and object jointly (Figure 3c). The human hand scale varies much less than the
object scale and can thus be used as a reference to constrain the object
scale. Moreover, by optimizing hand and object together, we can impose
additional constraints based on their depth ordering, hand-object contact, and
interpenetration.

Here, we first define the overall objective for jointly optimizing the hand
and the object:

$L=\lambda_{d}L_{depth}+\lambda_{i}L_{interact}+\lambda_{p}L_{penetrate}.$ (6)

In the following, we describe each loss term in detail. We note that including
additional constraints to enforce more in-domain knowledge is possible.
However, we find these simple constraints to work reasonably well in practice.

Figure 5: Qualitative results, EPIC. Results of our reconstruction procedure
on images from the EPIC Kitchens [7] dataset. Overall, the reconstructions
have reasonably high quality, though as expected there are still imperfections
(bottle example).

Ordinal depth loss. One common error in the reconstruction is the wrong depth
ordering of hand and object, e.g., a bottle lying on the back side of human
hand resulting in an infeasible grasp. To encourage correct depth ordering, we
add the ordinal depth loss term introduced in Jiang et al [19]. With instance
masks prediction from [23], we first obtain a joint mask
$\boldsymbol{M}_{all}$ for hand and object, where $\boldsymbol{M}_{all}(p)$
stores the category label for the pixel $p$, it is $0$ if the pixel in
background, $1$ if the pixel in hand, $2$ if the pixel in object. We can also
obtain a rendered mask $\boldsymbol{M}_{rend}(p)$using the 3D hand and object
into 2D. In the meantime, we render depth maps for both object and hand,
denoted as $\boldsymbol{D}_{o}$ and $\boldsymbol{D}_{h}$ respectively.
Finally, we calculate the ordinal depth loss for all pixels with a wrong depth
ordering:

$L_{depth}=\sum_{p\in\mathcal{S}}\log\left(1+\exp\left(\boldsymbol{D}_{h}(p)-\boldsymbol{D}_{o}(p)\right)\right),$
(7)

where $\mathcal{S}=\\{p\in
I:\,\boldsymbol{M}_{all}(p)\neq\boldsymbol{M}_{rend}(p)\\}$ represents the set
of pixels for image $I$ which have a different category label in the
segmentation mask from PointRend and the rendered mask.

Interaction loss. The reconstructed hand and object with correct depth
ordering could still be distant in 3D space. However, when the hand is
interacting with objects, their distance should be small. To push the
interaction pair closer in 3D, we define an interaction loss based on their
distance:

$L_{interact}=\|\textrm{mean}(\boldsymbol{V}_{h}^{*})-\textrm{mean}(\boldsymbol{V}_{o}^{*})\|_{2},$
(8)

where $\textrm{mean}(\cdot)$ represents the operation to calculate the average
of all vertices, i.e., the 3D center location of the mesh, similar to [66]. We
find this simple loss term is helpful in correcting the object scale by moving
it closer to the hand.

Interpenetration loss. Using the interaction loss alone may result in
implausible artifacts, e.g., hand colliding with the object. To resolve the
issue, we add an interpenetration loss term to penalize the object vertices
that are inside the hand mesh. We use the Signed Distance Field (SDF) from the
hand mesh to check if any object vertex is inside the hand. We first calculate
a tight box around the hand and voxelize it as a 3D grid for storing the SDF
value. Following [19], we use a modified SDF function $\phi$ for the hand
mesh:

$\phi(\boldsymbol{c})=-\min(\textrm{SDF}(c_{x},c_{y},c_{z}),0).$ (9)

For each voxel cell $\boldsymbol{c}=(c_{x},c_{y},c_{z})$ in the 3D grid, if
the cell is inside the hand mesh, $\phi$ takes positive values, proportional
to the distance from the hand surface, while $\phi$ is 0 outside of the hand
mesh. Then, the interpenetration loss can be calculated as:

$L_{penetrate}=\sum_{\boldsymbol{v}\in\boldsymbol{V}_{o}^{*}}\phi(\boldsymbol{v}),$
(10)

where $\phi(\boldsymbol{v})$ samples the SDF value of each object vertex
$\boldsymbol{v}$ from the 3D hand grid in a differentiable way.

Figure 6: Object category mismatch. We show example EPIC Kitchens [8] images
with predictions from a PointRend [23] model trained on COCO [29]. We note
that while the category labels are incorrect the instance segmentation masks
are very reasonable. We leverage such masks for reconstructing object
categories that are not in COCO.

###  3.4 Pose Refinement

A physically plausible hand-object reconstruction should not only be
collision-free, but also have enough hand-object contact area to support the
action. However, the interaction loss described in Section 3.3 does not take
into account the fine-grained hand-object contact. To further refine the 3D
reconstruction, we impose constraints on the hand-object contact as the final
step of our procedure (Figure 3d).

Addressing this issue would be easy if we had per-vertex contact area
annotation for both hand and object as we can enforce the contact region to be
closer. However, obtaining such annotations for large collection of in-the-
wild images is challenging. As a more practical solution, we learn 3D contact
priors from a large-scale hand MoCap dataset [56]. The priors include the
region of an object that the person is likely to touch or grab. For example,
human is more likely to hold the mug by its handle.

Given the hand mesh and object mesh obtained from the joint optimization, we
want to update the hand pose so that it has more reasonable contact with the
object. Inspired by RefineNet from [56], we train a small network to perform
hand pose refinement.

The input to the network are the initial hand parameters $(\theta,\gamma)$ and
the distance field $\boldsymbol{F}$ from the hand vertices
$\boldsymbol{V}_{h}^{*}$ to the object vertices $\boldsymbol{V}_{o}^{*}$. For
each hand vertex $\boldsymbol{v}_{h}$, we compute the distance to its nearest
object vertex:

$\boldsymbol{F}(\boldsymbol{v}_{h})=\min_{\boldsymbol{v}_{o}\in\boldsymbol{V}_{o}^{*}}\lVert\boldsymbol{v}_{h}-\boldsymbol{v}_{o}{\rVert}^{2}_{2}$
(11)

Then, the network refines the hand parameters $(\theta,\gamma)$ in an
iterative fashion. After each iteration, the distance field between hand and
object is updated so that it can be used as input to the next step. The
training data is obtained by perturbing the ground-truth hand pose parameters
and translation to simulate noisy input estimates. As shown in Figure 4, we
can observe the results after refinement (4th row) can reconstruct more
realistic interaction between hand and object than the previous step (3rd
row).

###  3.5 Implementation Details

To make the optimization more efficient, we preprocess our object CAD models
to have $\scriptstyle\sim$800 faces. When optimizing the object pose, we use
$1000$ initializations for object rotation sampled uniformly from
$[-\pi,\pi]$. Object translation is zero initialized. We select the object
pose with the lowest optimization loss. When optimizing the hand pose, we
weight the 2D keypoint loss $L_{2D}$ from Eq. 2 using the confidence score of
each predicted hand joint.

For the joint optimization of hand and object, our default loss weights are as
follows: $\lambda_{d}=20,\lambda_{i}=1000,\lambda_{p}=5$. We note that in some
cases the weights need to be adjusted further. For example, increasing
$\lambda_{i}$ to bring the object closer to the hand or increasing
$\lambda_{p}$ to reduce the collision. When computing the interpenetration
loss using SDF, we adopt the CUDA implementation of SDF from [19]. We voxelize
the hand mesh in a $32\times 32\times 32$ grid.

For the refinement networks, we use $10,000$ object vertices to compute the
distance field $\boldsymbol{D}$ from hand to object and perform 3 refinement
iterations. In practice, it takes $\scriptstyle\sim$1 minute to run our method
on an image using a GTX 2080 Ti GPU, including $\scriptstyle\sim$18 seconds
for optimizing the object pose and $\scriptstyle\sim$30 seconds for the joint
optimization.

object | COCO categories
---|---
spatula | fork, toothbrush, scissors, spoon, chair
kitchen knife | knife, scissors, tennis racket, bird
wooden spoon | knife, spoon
plate | bowl, frisbee, toy, remote, mouse, toilet
cereal box | book, cake
frying pan | bowl, spoon
screwdriver | scissors, knife, cell phone
rubiks cube | book, keyboard

Table 1: Category correspondences. We show example mappings between the true
object categories (observed in an image) and the sets of predicted COCO
categories. Figure 7: Qualitative results, 100DOH. Results of our
reconstruction procedure on images from the 100 Days of Hands [49] dataset.
Our method produces reconstructions of reasonably high-quality across a range
of viewpoints, activities, and objects.

##  4 Data collection

In the previous section, we described our reconstruction procedure assuming
access to unlabeled in-the-wild images, 2D instance masks, 3D object models,
and object-mask correspondences. Here, we discuss our procedure for acquiring
each one in turn.

In-the-wild images. As a source of unlabeled in the wild data we use frames
from the EPIC Kitchens [8, 7] and the 100 Days of Hands [49] datasets, noting
that we do not exploit any temporal information. These datasets contain a
range of interesting hand-object interaction scenarios with varied objects and
viewpoints (both first- and third-person).

3D object models. Our procedure for collecting 3D object models is semi-
automatic and relies on the human in the loop. In particular, we started with
a set of 3D object models we expected to see in these datasets (, forks,
spoons, plates, ) and kept growing the set over time as we observed new
objects (, microphone, motor oil, wood block, ). Two primary object sources we
used are the YCB dataset [5] from the robotics community and the Free3D online
platform. We collected _78 object models_ with both within and across category
variation (and their corresponding COCO candidate categories; discussed next).
We note, however, that a number of them are still outside the scope of our
method (see Figure 8 for discussion).

2D instance masks. Modern 2D recognition models trained on large labeled
datasets are now accurate enough that it is possible to apply them to real-
world data and obtain reasonable predictions [42]. However, in our case we
require instance masks for a range of object categories that are not present
in the available labeled datasets (, COCO categories). Thus, we cannot expect
the available models to categorize the objects correctly in our setting.
However, we observe that even if the predicted categories are incorrect, the
instance masks are still quite reasonable for a variety of objects. For
example, the models do not know what a spatula is called but are able to
segment it (Figure 6). Consequently, we can still leverage the available
models to obtain 2D instance masks for our use-case. Specifically, we use a
PointRend model [23] trained on COCO [29].

Object-mask correspondences. Given an input image we first manually select the
appropriate 3D model for the object of interest. Next, we need to determine
the corresponding 2D instance mask to use for reconstruction. One way to do
this would be to perform exhaustive matching with all detected instances
(typically up to 100) and select the one that results in the lowest loss.
Alternatively, we could manually inspect and select the appropriate instance
mask. In practice, we observe that each object typically gets recognized under
a small set of COCO categories. Thus, we establish correspondences between the
3D object models and sets of candidate COCO categories (see Table 1 for
examples). At reconstruction time, we use the precomputed mappings to select
the candidate object instances to use for reconstruction. We note that in the
cases where the object of interest is not detected at all the reconstruction
fails. We expect that this step can be improved further as models trained on
larger vocabulary datasets [14] become available.

##  5 Experiments

In this section, we evaluate our method in two settings: in-the-wild and in-
the-lab. First, we show qualitative results in the wild. Next, we perform
quantitative comparisons to existing approaches in the lab settings where
ground truth 3D annotations are available. Finally, we present ablation
studies and discuss the failure modes of our approach.

###  5.1 Results in the wild

We consider two datasets as the source of in-the-wild data for hand-object
reconstruction: EPIC Kitchens [8] and 100 Days of Hands [49]. Due to the lack
of appropriate ground truth annotations, quantitative evaluation in this
setting is challenging. We present qualitative results here and supplement
them with quantitative evaluation in the lab settings where ground truth
annotations are available (next subsection). We try to showcase examples that
work fairly well, have minor imperfections, and represent clear failure cases.
We show reconstructions for images from the two aforementioned datasets in
Figure 5, Figure 7, and Figure 9. Overall, our approach produces promising
reconstructions across a variety of viewpoints, activities, and objects.

###  5.2 Results in the lab

#### Dataset.

For quantitative evaluation of our method, we use HO3D [16] dataset which
contains video sequences of hand interacting with daily objects. All images
have 3D annotations for both the hand and object. The dataset consists of
77,558 frames, 68 sequences, 10 persons, and 10 objects. The object CAD models
are from YCB dataset [5]. Although our method does not require extra training,
to have a fair comparison to existing work which requires 3D supervision, we
split the dataset into training and testing set based on object categories. We
use 3 categories, i.e., banana, cracker box, and cleanser bottle for
evaluation. We do not use the official testing set from HO3D [16] because it
does not contain evaluation for object pose. In all of the experiments, we use
the default optimization hyperparameters for our method (Section 3.5).

|  [17] |  [17]⋆ | Ours
---|---|---|---
Hand joint MAE ↓ | 8.9 | 18.8 | 9.5
Hand verts MAE ↓ | 9.0 | 18.3 | 9.8
$F_{1}$@15mm ↑ | 63.8 | 30.1 | 52.3
$F_{1}$@30mm ↑ | 83.6 | 59.7 | 78.6
Chamfer distance ↓ | 18.4 | 29.3 | 20.5

Table 2: Quantitative comparison in HO3D dataset. [17] denotes the author’s
trained model (some images from the testing set were used for training due to
different split). [17]⋆ denotes the model trained by us with the new split.
MAE is the mean average error in mm. $F_{1}$@15mm denotes the $F_{1}$ score
with the distance threshold of 15mm.

| Bottle | Box | Banana
---|---|---|---
Hand joint MAE ↓ | 9.0 | 9.8 | 10.1
Hand verts MAE ↓ | 9.1 | 9.8 | 10.9
$F_{1}$@15mm ↑ | 54.8 | 49.6 | 54.7
$F_{1}$@30mm ↑ | 79.1 | 82.9 | 74.1
Chamfer distance ↓ | 19.8 | 19.9 | 21.6

Table 3: Consistent performance across categories.

Evaluation metric. For evaluating hand pose, we report the mean average error
(MAE) over 21 hand joints and 778 hand mesh vertices respectively. The error
measures the Euclidean distance between predicted and ground truth hand
joints/vertices. Following [16], we calculate the error after aligning the
position of the hand root joint and global scale with the ground-truth.

For evaluating object pose, we calculate the Chamfer distance between ground-
truth object vertices and predicted object vertices (obtained by rotating the
input CAD model with the predicted object pose). We remove the global
translation of the object for this evaluation. We also report the F-score [31]
between the predicted object vertices and the ground truth at different
distance threshold $\tau$. For two point clouds, $F_{1}$ score is calculated
as the harmonic mean between recall (fraction of ground-truth points within
$\tau$ of a predicted point) and precision (fraction of predicted points
within $\tau$ of a ground-truth point).

Top $k$ poses |  CF Dist ↓ |  $F_{1}$@15mm ↑ |  $F_{1}$@30mm ↑
---|---|---|---
$k$=1 | 20.1 | 52.9 | 79.3
$k$=3 | 15.9 | 64.3 | 87.5
$k$=5 | 13.7 | 70.6 | 91.4
$k$=10 | 12.0 | 76.4 | 94.3

Table 4: Ablations on the number of object pose hypotheses. We report Chamfer
distance (CF Dist) and $F_{1}$ score between predicted object mesh and ground-
truth object mesh. With more hypotheses of object pose, we can obtain better
reconstruction result.

|  HO Distance ↓ |  Collision Score ↓
---|---|---
Individual results | 414.8 | 0
\+ Interaction loss | 71.5 | 39.8
\+ Ordinal depth loss | 75.2 | 17.6
\+ Penetration loss | 76.4 | 7.7
\+ Refinement | 75.8 | 6.5

Table 5: Ablations on loss terms and pose refinement. From top to bottom, we
add each component one by one (cumulative) and evaluate the prediction in
terms of the distance between hand and object, and the collision score.

Quantitative comparison. In Table 2, we perform quantitative evaluation of our
object and hand pose prediction in the HO3D dataset [16]. We compare against
the state-of-the-art approach [17] with the same input of monocular RGB image
and the known object CAD model. [17] uses a feed-forward neural network to
predict 3D hand pose and object pose assuming object CAD model is available.
With the new split based on object categories, we retrain their model using
their released code and report their evaluation result (denoted as [17]⋆ in
Table 2). For a reference, we also report the evaluation results (denoted as
[17] in Table 2) from their released model which split the data based on
images. Due to the split difference, some testing images from the new split
were used to train the original reference model. In comparison, we directly
test our optimization-based method on this dataset without extra tuning.

As shown in the table, our method achieves close performance to the reference
model, with a hand joint error of $9.5$ mm $8.9$ mm, and Chamfer distance of
$20.5$ $18.4$. The retrained model using the new split has worse performance
than the reference model, with a larger hand joints error ($18.8$ mm $8.9$ mm)
and smaller $F_{1}$ score ($30.1$ mm $29.3$ at 15 mm distance threshold). The
reasons are two folds: (1) the reference model was trained on part of the
images in the new split and thus performs better. (2) The model retrained with
the category-based split has difficulty in generalizing to new object
categories. In Table 3, we directly evaluate our method in different object
categories without further tuning on this dataset, and we do not observe large
performance difference across object categories.

Figure 8: Failure cases. We show representative failure cases of our
reconstruction procedure on the EPIC Kitchens and the 100 Days of Hands
datasets. We observe several failure modes due to the failure of the
individual steps in our procedure: hand pose estimation (first column), object
pose estimation (columns 2-4), and the joint optimization (last column).

###  5.3 Ablation Studies

In this section, we perform ablation studies to better understand different
components of our method. We sample a subset of images from the test set for
this analysis.

Number of object pose hypotheses. In Table 4, we did experiment of having
multiple hypotheses about object pose estimation result. When optimizing the
object pose, we start with $1000$ initializations for object rotation sampled
uniformly from $[-\pi,\pi]$. After the optimization, instead of using the top
$1$ result with lowest loss, we can use the top $k$ results as input to the
later stage. This will result in $k$ different final hand-object
reconstruction results after jointly optimizing hand-object arrangement. In
the table, we report the object pose error for the result which best matches
to the ground truth. When we increase the value of $k$, we can see the Chamfer
distance between the predicted object mesh and ground-truth object mesh
decreases substantially from $20.1$ $(k=1)$ to $12.0$ $(k=10)$. This indicates
having more object pose hypotheses could help resolve some ambiguity resulting
from 2D masks loss.

Optimization loss terms. In Table 5, we evaluate the influence from each loss
term for the joint optimization and RefineNet in the reconstruction result. We
report the distance between the estimated hand center and object center. The
collision score is calculated as the sum of all SDF values using Equation 9
and 10. The deeper the object intersects with the hand, the larger the
collision score will be. A good reconstruction should have small collision and
small hand-object distance in most cases. From the table, we can observe that
the initial reconstructed hand and object from individual optimization are far
away from each other, resulting in large center distance, i.e., $414.8$ mm and
zero collision. By adding the coarse interaction loss, their distance
decreases quickly to $71.5$ mm but end up with large collision, i.e, $39.8$.
Adding the ordinal depth and penetration loss could further reducing the
collision score to $7.7$ while keeping a similar hand-object distance, i.e.,
$76.4$ mm. The RefineNet makes small adjustment to the final result and can
slightly reduce the collision score ($6.5$ $7.7$) and hand-object distance
($75.8$ mm $76.4$ mm) in the same time. These findings align with our
visualization in Figure 4.

###  5.4 Failure Cases

While our method shows promising results, there is still a lot of room for
improvement. In Figure 8, we show the representative failure modes of our
method and hope that they may help inform future methods. In the first column,
we observe that the hand pose estimation fails when the hand is highly
occluded or cropped by the image. In the columns 2-4, we see incorrect object
pose predictions due to the ambiguity of the 2D mask loss when optimizing the
object pose. In the last column, we observe examples where the joint
optimization converges to an undesirable local minima. To resolve some
ambiguity in this task, we experiment with having multiple object pose
hypotheses and observe large improvement as shown in Table 4. We believe that
semi-automated procedures which utilize multiple hypotheses could be a
promising direction for obtaining large collections of hand-object
reconstructions in-the-wild.

##  6 Conclusion

In this work we explore reconstructing hand-object interactions in the wild.
We show that by leveraging all available related data, through an
optimization-based procedure, we can obtain promising reconstructions for the
challenging in-the-wild data. We hope that our work will attract more
attention to this exciting and practical setting.

Figure 9: Additional qualitative results. Our procedure produces promising
results across a range of scenarios and objects.

## References

  * [1] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black.  Keep it smpl: Automatic estimation of 3d human pose and shape from a single image.  In ECCV, 2016.
  * [2] Samarth Brahmbhatt, Cusuh Ham, Charles C. Kemp, and James Hays.  ContactDB: Analyzing and predicting grasp contact via thermal imaging.  In CVPR, 2019.
  * [3] Samarth Brahmbhatt, Chengcheng Tang, Christopher D. Twigg, Charles C. Kemp, and James Hays.  ContactPose: A dataset of grasps with object contact and hand pose.  In ECCV, 2020.
  * [4] Yujun Cai, Liuhao Ge, Jianfei Cai, and Junsong Yuan.  Weakly-supervised 3d hand pose estimation from monocular rgb images.  In ECCV, 2018.
  * [5] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar.  The ycb object and model set: Towards common benchmarks for manipulation research.  In ICRA, 2015.
  * [6] Zhe Cao, Gines Hidalgo Martinez, Tomas Simon, Shih-En. Wei, and Yaser Sheikh.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields.  TPAMI, 2019.
  * [7] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al.  Rescaling egocentric vision.  arXiv:2006.13256, 2020.
  * [8] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al.  Scaling egocentric vision: The epic-kitchens dataset.  In ECCV, 2018.
  * [9] Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim.  First-person hand action benchmark with RGB-D videos and 3D hand pose annotations.  In CVPR, 2018.
  * [10] James J Gibson.  The theory of affordances.  Hilldale, USA, 1977.
  * [11] Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He.  Detecting and recognizing human-object interactions.  In CVPR, 2018.
  * [12] Georgia Gkioxari, Jitendra Malik, and Justin Johnson.  Mesh r-cnn.  In ICCV, 2019.
  * [13] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al.  The ”something something” video database for learning and evaluating visual common sense.  In ICCV, 2017.
  * [14] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In CVPR, 2019.
  * [15] Tanmay Gupta, Alexander Schwing, and Derek Hoiem.  No-frills human-object interaction detection: Factorization, layout encodings, and training techniques.  In ICCV, 2019.
  * [16] Shreyas Hampali, Mahdi Rad Markus Oberweger, and Vincent Lepetit.  Honnotate: A method for 3d annotation of hand and object poses.  In CVPR, 2020.
  * [17] Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Marc Pollefeys, and Cordelia Schmid.  Leveraging photometric consistency over time for sparsely supervised hand-object reconstruction.  In CVPR, 2020.
  * [18] Yana Hasson, Gül Varol, Dimitris Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid.  Learning joint reconstruction of hands and manipulated objects.  In CVPR, 2019.
  * [19] Wen Jiang, Nikos Kolotouros, Georgios Pavlakos, Xiaowei Zhou, and Kostas Daniilidis.  Coherent reconstruction of multiple humans from a single image.  In CVPR, 2020.
  * [20] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In CVPR, 2018.
  * [21] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik.  Learning category-specific mesh reconstruction from image collections.  In ECCV, 2018.
  * [22] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada.  Neural 3d mesh renderer.  In CVPR, 2018.
  * [23] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick.  Pointrend: Image segmentation as rendering.  In CVPR, 2020.
  * [24] Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M. Bronstein, and Stefanos Zafeiriou.  Weakly-supervised mesh-convolutional hand reconstruction in the wild.  In CVPR, 2020.
  * [25] Dominik Kulon, Haoyang Wang, Riza Alp Güler, Michael Bronstein, and Stefanos Zafeiriou.  Single image 3d hand reconstruction with mesh convolutions.  BMVC, 2019.
  * [26] Abhijit Kundu, Yin Li, and James M Rehg.  3d-rcnn: Instance-level 3d object reconstruction via render-and-compare.  In CVPR, 2018.
  * [27] Weicheng Kuo, Anelia Angelova, Tsung-Yi Lin, and Angela Dai.  Mask2cad: 3d shape prediction by learning to segment and retrieve.  ECCV, 2020.
  * [28] Joseph J Lim, Hamed Pirsiavash, and Antonio Torralba.  Parsing ikea objects: Fine pose estimation.  In ICCV, 2013.
  * [29] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [30] Priyanka Mandikal and Kristen Grauman.  Dexterous robotic grasping with object-centric visual affordances.  arXiv:2009.01439, 2020.
  * [31] Christopher D Manning, Hinrich Schütze, and Prabhakar Raghavan.  Introduction to information retrieval.  Cambridge university press, 2008.
  * [32] Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell.  Something-else: Compositional action recognition with spatial-temporal interaction networks.  In CVPR, 2020.
  * [33] Andrew N Meltzoff.  Understanding the intentions of others: re-enactment of intended acts by 18-month-old children.  Developmental psychology, 1995.
  * [34] Frank Michel, Alexander Kirillov, Eric Brachmann, Alexander Krull, Stefan Gumhold, Bogdan Savchynskyy, and Carsten Rother.  Global hypothesis generation for 6d object pose estimation.  In CVPR, 2017.
  * [35] Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee.  V2v-posenet: Voxel-to-voxel prediction network for accurate 3d hand and human pose estimation from a single depth map.  In CVPR, 2018.
  * [36] Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt.  Ganerated hands for real-time 3d hand tracking from monocular rgb.  In CVPR, 2018.
  * [37] Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman.  Grounded human-object interaction hotspots from video.  In ICCV, 2019.
  * [38] Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman.  Ego-topo: Environment affordances from egocentric video.  In CVPR, 2020.
  * [39] Yuzuko C Nakamura, Daniel M Troniak, Alberto Rodriguez, Matthew T Mason, and Nancy S Pollard.  The complexities of grasping in the wild.  In Humanoids, 2017.
  * [40] Iason Oikonomidis, Nikolaos Kyriazis, and Antonis A Argyros.  Full dof tracking of a hand interacting with an object by modeling occlusions and physical constraints.  In ICCV, 2011.
  * [41] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  In CVPR, 2019.
  * [42] Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He.  Data distillation: Towards omni-supervised learning.  In CVPR, 2018.
  * [43] Ilija Radosavovic, Xiaolong Wang, Lerrel Pinto, and Jitendra Malik.  State-only imitation learning for dexterous manipulation.  arXiv:2004.04650, 2020.
  * [44] Francesco Ragusa, Antonino Furnari, Salvatore Livatino, and Giovanni Maria Farinella.  The meccano dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain.  arXiv:2010.05654, 2020.
  * [45] Javier Romero, Hedvig Kjellström, and Danica Kragic.  Hands in action: real-time 3d reconstruction of hands in interaction with objects.  In ICRA, 2010.
  * [46] Javier Romero, Dimitrios Tzionas, and Michael J Black.  Embodied hands: Modeling and capturing hands and bodies together.  TOG, 2017.
  * [47] Yu Rong, Takaaki Shiratori, and Hanbyul Joo.  Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration.  arXiv:2008.08324, 2020.
  * [48] Mihir Sahasrabudhe, Zhixin Shu, Edward Bartrum, Riza Alp Guler, Dimitris Samaras, and Iasonas Kokkinos.  Lifting autoencoders: Unsupervised learning of a fully-disentangled 3d morphable model using deep non-rigid structure from motion.  In ICCV, 2019.
  * [49] Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey.  Understanding human hands in contact at internet scale.  In CVPR, 2020.
  * [50] Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim, Christoph Rhemann, Ido Leichter, Alon Vinnikov, Yichen Wei, et al.  Accurate, robust, and flexible real-time hand tracking.  In CHI, 2015.
  * [51] Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh.  Hand keypoint detection in single images using multiview bootstrapping.  In CVPR, 2017.
  * [52] Srinath Sridhar, Franziska Mueller, Michael Zollhöfer, Dan Casas, Antti Oulasvirta, and Christian Theobalt.  Real-time joint tracking of a hand manipulating an object from rgb-d input.  In ECCV, 2016.
  * [53] Srinath Sridhar, Antti Oulasvirta, and Christian Theobalt.  Interactive markerless articulated hand motion tracking using rgb and depth data.  In ICCV, 2013.
  * [54] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman.  Pix3d: Dataset and methods for single-image 3d shape modeling.  In CVPR, 2018.
  * [55] Andrea Tagliasacchi, Matthias Schröder, Anastasia Tkach, Sofien Bouaziz, Mario Botsch, and Mark Pauly.  Robust articulated-icp for real-time hand tracking.  In Computer Graphics Forum, 2015.
  * [56] Omid Taheri, Nima Ghorbani, Michael J. Black, and Dimitrios Tzionas.  GRAB: A dataset of whole-body human grasping of objects.  In ECCV, 2020.
  * [57] Bugra Tekin, Federica Bogo, and Marc Pollefeys.  H+o: Unified egocentric recognition of 3d hand-object poses and interactions.  In CVPR, 2019.
  * [58] Shubham Tulsiani, Saurabh Gupta, David Fouhey, Alexei A. Efros, and Jitendra Malik.  Factoring shape, pose, and layout from the 2d image of a 3d scene.  In CVPR, 2018.
  * [59] Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall.  Capturing hands in action using discriminative salient points and physics simulation.  IJCV, 2016.
  * [60] Donglai Xiang, Hanbyul Joo, and Yaser Sheikh.  Monocular total capture: Posing face, body, and hands in the wild.  In CVPR, 2019.
  * [61] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox.  Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes.  In RSS, 2018.
  * [62] Tete Xiao, Quanfu Fan, Dan Gutfreund, Mathew Monfort, Aude Oliva, and Bolei Zhou.  Reasoning about human-object interactions through dual attention networks.  In ICCV, 2019.
  * [63] Linlin Yang and Angela Yao.  Disentangling latent hands for image synthesis and pose estimation.  In CVPR, 2019.
  * [64] Qi Ye, Shanxin Yuan, and Tae-Kyun Kim.  Spatial attention deep net with partial pso for hierarchical hybrid hand pose estimation.  In ECCV, 2016.
  * [65] Shanxin Yuan, Guillermo Garcia-Hernando, Björn Stenger, Gyeongsik Moon, Ju Yong Chang, Kyoung Mu Lee, Pavlo Molchanov, Jan Kautz, Sina Honari, Liuhao Ge, et al.  Depth-based 3d hand pose estimation: From current achievements to future goals.  In CVPR, 2018.
  * [66] Jason Y. Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa.  Perceiving 3d human-object spatial arrangements from a single image in the wild.  In ECCV, 2020.
  * [67] Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu.  Monocular real-time hand shape and motion capture using multi-modal data.  In CVPR, 2020.
  * [68] Christian Zimmermann and Thomas Brox.  Learning to estimate 3d hand pose from single rgb images.  In ICCV, 2017.
  * [69] Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox.  Freihand: A dataset for markerless capture of hand pose and shape from single rgb images.  In ICCV, 2019.


#  Science Checker Reloaded: A Bidirectional Paradigm for Transparency and
Logical Reasoning

Loïc Rakotoson Opscidia
Paris, France
email<EMAIL_ADDRESS>Sylvain Massip Opscidia
Paris, France
email<EMAIL_ADDRESS>Fréjus A. A. Laleye Opscidia
Paris, France
email<EMAIL_ADDRESS>
###### Abstract

Information retrieval is a rapidly evolving field. However it still faces
significant limitations in the scientific and industrial vast amounts of
information, such as semantic divergence and vocabulary gaps in sparse
retrieval, low precision and lack of interpretability in semantic search, or
hallucination and outdated information in generative models. In this paper, we
introduce a two-block approach to tackle these hurdles for long documents. The
first block enhances language understanding in sparse retrieval by query
expansion to retrieve relevant documents. The second block deepens the result
by providing comprehensive and informative answers to the complex question
using only the information spread in the long document, enabling bidirectional
engagement. At various stages of the pipeline, intermediate results are
presented to users to facilitate understanding of the system’s reasoning. We
believe this bidirectional approach brings significant advancements in terms
of transparency, logical thinking, and comprehensive understanding in the
field of scientific information retrieval.

###### Index Terms:

Information Retrieval; Question Answering; Fact Checking.

##  I Introduction

In recent years, advancements in Natural Language Processing have
significantly reshaped the information retrieval landscape with the
introduction of semantic vector search and large language models. The
integration of dense retrieval into vector databases, which leverages low-
dimensional contextual information, goes beyond sparse methods by tapping into
nuanced semantic relationships, enriching the understanding of textual data.

While Best Match 25 (BM25) has limitations in addressing semantic divergence
and vocabulary gaps, hindering its ability to capture the nuances of language
effectively, it continues to dominate industry usage [1]. This prevalence can
be ascribed to its simplicity and efficiency, coupled with the challenges
posed by the lack of generalization, interpretability issues, and the black-
box nature of Transformer-based dense methods. Despite the industry’s ongoing
reliance on BM25, dense retrieval methods have yet to reach maturity for
widespread adoption, making BM25 the go-to method for its efficiency in
industrial applications.

Moreover, the computational demands of Large Language Models (LLMs) in
Retrieval Augmented Generation (Section II-B) pose scalability concerns,
making their seamless integration into real-world applications challenging.
Additionally, these models inability to provide explanations for their answers
and occasional tendency to hallucinate information [2][3] create significant
challenges, particularly in domains where accountability and reliability are
paramount. In addition, the cost of computing embeddings and the storage in a
dedicated database required to save the embeddings of documents in a knowledge
management context makes it difficult to adopt these methods and industrialize
them in real-world applications [4][5].

To address these limitations, our approach aims to go beyond technological
innovation. We aim to address the fundamental issues of transparency and
logical reasoning in answer generation and retrieval. By providing a clearer
understanding of how the system arrives at its responses, we empower users
with the ability to fact-check and exert control over the reasoning process.
Aiming for this goal, our approach seeks to bridge the existing gaps and help
in transparency and logical reasoning in scientific information retrieval. The
construction of the solution must diminish slow and heavy processes with
relatively small improvements to simpler or lighter versions to obtain a
better trade-off between performance and cost. We are addressing in this work
the context of open-domain query with scientific and technical documents,
which are generally long and contain dense information.

This paper is organized as follows. Section II provides an overview of the
background works in scientific information retrieval and retrieval augmented
generation. Section III presents our approach, which consists of two blocks:
document retrieval and answer generation. Section IV discusses the evaluations
of our approach. Section V presents the discussion of our approach, concludes
the paper and outlines future work.

##  II Background Works

###  II-A Scientific Information Retrieval

Science based fact checking and question answering is a complex task due to
the complexity of scientific language [6] in comparison to general language.
It is essential for combating the spread of misinformation and assisting
researchers in knowledge discovery. Various methods have been proposed,
including scientific claim generation, boolean question answering, and semi-
automatic discovery of relevant expert opinions [6][7][8]. In [9], authors
proposed an extractive-boolean system with justification and contradiction
resolution in yes/no/neutral questions related to biomedical studies. However,
the deployment and industrialization of these methods are not done since they
do not integrate with any real usage [10]. These systems do not bring
improvements in the selection of relevant documents, do not update their
knowledge and are difficult to interact with. The introduction of chatbot-like
systems with large language models in the industry has made it possible to
interact with the system and to have a better understanding of the system’s
reasoning.

###  II-B Retrieval Augmented Generation (RAG)

The usage of LLMs in fact checking has become increasingly important for
accurate and credible information retrieval in complex knowledge-intensive
tasks [11][12], but it still faces challenges such as generating fictitious
responses and hallucinations.

Figure 1: Retrieval Augmented Generation Architecture

To overcome these challenges, researchers have proposed various approaches
including the incorporation of information retrieval (IR) systems to provide
external knowledge to LLMs. These approaches aim to improve the accuracy,
credibility, and traceability of LLMs by verifying answers, correcting
inaccuracies, and providing missing knowledge [13][14]. Additionally, domain-
specific adaptations of LLMs have been explored to optimize performance in
specialized domains [15]. Furthermore, studies have examined the impact of
retrieval augmentation on long-form question answering, including analysis of
answer attribution and errors [16]. Another perspective suggests replacing
document retrievers with LLM generators for knowledge-intensive tasks,
resulting in improved recall of acceptable answers [17].

####  II-B1 Dense Retrieval

To resolve the limitations of LLMs in knowledge update, vector databases have
been proposed as a solution to the scalability and interpretability issues.
These databases store dense representations of documents, enabling efficient
retrieval of relevant documents [12][18][19]. The classic architecture of a
RAG includes an ingestion module that divides each document into several
chunks, with the document being stored in the form of embeddings of its
chunks. A search module retrieves the embeddings of the most relevant chunks
for a given question. A generation module takes the embeddings of the chunks
as input and generates a response to the question (Figure 1).

Aside from the computational costs of embeddings and inferences, which can be
optimized through caching [20], the storage of document embeddings in a
dedicated database has a significant cost. In a knowledge management context
with long documents and content-aware chunking, the storage space required for
a document can be approximated by the following formula, without taking
compression into account:

$\text{Storage (bytes)}=\text{Chunks}\times\text{Embedding
Size}+\text{Tokens}$

The latest top 10 most performant embedding models in massive text embedding
benchmark [21] have a maximum embedding size of 4096 dimensions and a minimum
of 768 dimensions. The number of chunks in content-aware chunking is averaged
at 4 to 12 sections per article. Which gives, for a small index of 10 million
documents, a range of $0.1$ to $0.6$ TB of storage in a dense database. The
same database in sparse representation will cost only $32$ GB.
In addition to their high cost, dense databases are less deployed in the
industry due to their relatively low advantage compared to sparse databases.
On top of this low efficiency, this immaturity is explained by the difficulty
of adopting dense methods due to their lack of generalization, their low
token-level performance, and especially the difficulty of interpreting the
results [22][23] for end users.

####  II-B2 Knowledge Graphs

In the field of knowledge-intensive language tasks, innovative approaches have
been developed to leverage external knowledge and enhance the capabilities of
LLMs with knowledge graphs [24]. One such approach is the Knowledge Graph
Induction framework [25], which combines outputs from different models trained
on tasks like slot filling, open domain question answering, dialogue, and
fact-checking. This approach improves accuracy by cross-examining the outputs
of various models, particularly enhancing dialogue using a question answering
model. Another approach called Knowledge-Augmented language model Prompting
(KAPING) [26] focuses on zero-shot knowledge graph question answering. This
approach augments LLMs by retrieving relevant facts from a knowledge graph and
incorporating them into the prompt without requiring additional model
training. This method outperforms relevant zero-shot baselines by up to $48\%$
in average across multiple LLMs of varying sizes.
These approaches allow to increase the precision of the answers with easily
verifiable knowledge and to make them more understandable for users.

Figure 3:  Answer Generation with Iterative Deepening. Yellow: User access
points.

##  III Our Approach

We aim to build an approach that can be industrialized, by eliminating the
least efficient processes, allowing several pipeline stages to provide
intermediate results to facilitate understanding of the system’s reasoning,
and enabling two-way interaction with the end user. The approach must follow
the evolution of the user’s funnel-like journey from information search to
answer comprehension. The field of application is the retrieval of scientific
and technical information, usually contained in long, dense documents.
We propose a two-block system. The first block focuses on retrieving relevant
documents, while the second block iterates over each document to deepen the
answer.

###  III-A Document Retrieval

Figure 2:  Sparse Retrieval with Ontology-Oriented Query Expansion. Yellow:
User access points.

To retrieve relevant documents from a large collection based on a query, we
apply an ontology-oriented sparse query expansion (Figure 2). We use the most
efficient approach by supercharching BM25. To address the limitations of
semantic divergence and vocabulary gaps, this approach leverages the semantic
relationships between words and concepts to enrich the query with synonyms,
hypernyms, and hyponyms.
The system extracts entities and performs expansion from the ontology. We use
SciBERT [27] as a model. Each entity constitutes a MUST clause with its
expansions as variations. Verbs are augmented with their Wordnet synonyms and
will constitute SHOULD clauses. The database indexes the title, abstract, and
sections of the article in different fields. The results are ranked by
relevance and presented to the user. Depending on the scope of knowledge
management, the specificity of the ontology can be of different levels. We
used Wikidata [28] as a generalist ontology, and some domain-specific like
MeSH [29][30] or internal.

###  III-B Answer Generation

Once a relevant document has been identified by the user, they can query the
long document more precisely to obtain an answer to their question. To achieve
this, we implement a hybrid search system by creating an in-memory index of
all the chunks of the document. The system includes a retrieval block
consisting of sparse retrieval with a BM25 model and a dense retrieval in
multihop [31] with 3 iterations using multi-lingual semantic textual
similarity sentence transformers [32]. The main output orders these results
using a cross-encoder [33] and applies an extractive QA head to return the
passages in the sections that answer the question.

Alternative outputs allow for a generative response. The first uses the
iterations of the multihop embedding retrieval to generate a sequence of
logical reasoning from the selected chunks. The second is placed at the output
of the hybrid retrieval by scoring the sparse and dense results with a
reciprocal rank fusion. Before feeding the generation model with this long
context, a reranking in ”lost in the middle” [34] is performed to place the
most relevant parts of the context at the beginning and end, using the primacy
and recency biases of generative models. The two outputs are under
construction and their comparative evaluation between each other and with the
primary output should validate the best approach for generating responses for
end users according to different contexts of use and document nature.
In both cases, the generation model is used for its editorial and information
synthesis functions from external sources of knowledge. As the model’s
internal knowledge should not be mobilized, the specialized language model is
simpler, more efficient [35][36], and less prone to hallucination [37].

At various stages of the system, intermediate results are presented to the
user to facilitate understanding of the system’s reasoning. These access
points are checkpoints for the user, who can thus verify the consistency of
the results. Those access points are presented in yellow in the figures 2 and
3. They are the entry points for the user to interact with the system and to
check the relevance of the sources and the reasoning of the system.

##  IV Evaluations

The two blocks of our approach are systems that can be evaluated
independently. To evaluate the usability of the whole, it is necessary to
carry out user tests [10], especially to evaluate the understanding of the
response, the transparency of the system, and the ease of use. However, at the
early stage of our work, we conducted quantitative performance evaluations of
document retrieval. There is no benchmark sufficiently provided in long
document retrieval on scientific data, however the Multilingual Long-Document
Retrieval dataset built on Wikipedia and the associated work on BGE
M3-Embedding [38] are references to evaluate the performance of our document
retrieval system. The dataset consists of text pairs of questions generated by
GPT-3.5 based on paragraphs sampled from lengthy articles in 13 languages, and
the corresponding articles as the retrieval candidates. We focused on the
English language.

We evaluated 3 versions of our system on an Elasticsearch base. The first is
the optimized use of the clauses of the base with different types of sub-
fields of text, analyzers, and representations in n-grams. The clauses are
boosted according to their degree of precision. As there is only one main
textual field, the multi-match clauses are evaluated in most_fields. The
second is the use of query expansion with an external knowledge graph. We used
the Wikidata Ontology [28] to extend the entities detected in the query. These
must respond in MUST clauses and boosted. The third is the same version with
more flexible clauses on the recognized entities. We report the normalized
discounted cumulative gain among the top 10 results.

TABLE I:  ndcg@10 of the different versions of our system and report of the last benchmark on the multilingual long-document retrieval dataset.  Model | Max Length | nDCG@10
---|---|---
BM25 | 8192 | 57.0
mDPR | 512 | 23.9
mContriveer | 512 | 28.7
mE5-large | 512 | 33.0
E5-mistral-7b | 8192 | 43.3
openai-ada-002 | 8191 | 38.7
jina-embeddings-v2-base-en | 8192 | 37.0
BGE M3 |  |
Dense | 8192 | 48.9
Sparse | 8192 | 62.1
Dense+Sparse | 8192 | 64.2
Ours |  |
Optimized most_fields | - | 62.4
Must entity Wiki expansion | - | 59.6
Should entity Wiki expansion | - | 64.8

While the first version gives 1,000 results for each query like the evaluation
dataset, it turns out that the two versions with expansion give fewer search
results. The absence of results is significant in the version with the
expansion of entities executed in MUST clauses, with 18% of queries without
relevant documents. This is due to the strong precision constraint on the
entities and results in a lower nDCG score than the version without expansion.
By relaxing this constraint, with the SHOULD attribution clauses version, we
improve the recall on the returned documents. This version, less complex and
heavy than the hybrid version of BGE M3, gives performances that are roughly
equivalent. In general, our approach, simpler and lighter, gives better
performance than dense approaches based on more complex LLMs. As with the
second block, its results will need to be refined as the complete system is
developed, with additional measures needed to capture the quality of the
response, usability, and performance of the complete system in scientific and
technical domains.

##  V Discussion

Our approach is designed to address the limitations of existing systems in
scientific information retrieval on long documents. It aims to provide a
transparent and logical reasoning process, enabling users to fact-check and
understand the system’s reasoning. The system is designed to be
industrialized, with a focus on eliminating the least efficient processes and
being cost-aware. These early stage results are presented to highlight the
principle of efficiency of accessible models in opposition to increasingly
complex models, much more black-boxes. The results of the document retrieval
evaluation show that our approach, simpler and lighter, gives better
performance than dense approaches based on heavier LLMs. This is a promising
result for the industrialization of our approach. However, the evaluation of
the complete system will require additional measures to capture the quality of
the response, usability, and performance in scientific and technical domains.

In a knowledge management context on a specific domain, our retrieval system
performs well. However, we observe the limitations of our current system due
to the absence of a more general knowledge graph. In these specific cases, the
use of the hybrid system proposed by BGE M3 gives better results, despite
these results being frozen by the model’s weights. The development of top-
level ontologies that allow the interoperability of more specific graphs, such
as the WikiProject Ontology, is a method to have a system that improves with
the advancement of knowledge. This is a development path for our system to
have better knowledge coverage, in addition to regularly updated article
databases.

Finally, as the goal of our system is to make exchanges between the pipeline
and the user, we plan to include user tests in its development to evaluate the
transparency of the system, ease of use, and understanding of the responses.
These tests will validate the quality of our system and improve it. The
ultimate goal is to propose a system that is easily industrializable and can
be shaped according to deployment contexts. We conduct continuous tests on the
system we are building during our research.

##  VI Conclusion and Future Work

In this paper, we presented a two-block approach for scientific information
retrieval on long documents. Our approach combines sparse retrieval with
ontology-oriented query expansion and hybrid retrieval with iterative
deepening to provide comprehensive and informative answers to complex
questions. We also designed our system to be transparent and logical, enabling
oriented users evaluation with the system and understand its reasoning
process. We evaluated our document retrieval block on the MLDR dataset and
showed that our approach outperforms dense retrieval methods based on LLMs.
The answer generation block is under evaluation. We plan to conduct user tests
to evaluate the usability and quality of our complete system in scientific and
technical domains. We also aim to improve our knowledge coverage by
integrating top-level ontologies that allow the interoperability of more
specific graphs. Moreover, we intend to explore the use of knowledge graphs
and generative models to enhance the answer generation block and provide more
credible and traceable responses. Finally, we hope to develop a system that is
easily industrializable and adaptable to different deployment contexts.

## References

  * [1] M. Luo, A. Mitra, T. Gokhale, and C. Baral, “Improving biomedical information retrieval with neural retrievers,” in _AAAI Conference on Artificial Intelligence_ , 2022.
  * [2] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” _ACM Comput. Surv._ , vol. 55, no. 12, mar 2023.
  * [3] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” _ArXiv_ , vol. abs/2311.05232, 2023.
  * [4] R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green ai,” _Commun. ACM_ , vol. 63, no. 12, p. 54–63, nov 2020. [Online]. Available: https://doi.org/10.1145/3381831
  * [5] M. Musser, “A cost analysis of generative language models and influence operations,” _arXiv preprint arXiv:2308.03740_ , 2023.
  * [6] D. Wright, D. Wadden, K. Lo, B. Kuehl, A. Cohan, I. Augenstein, and L. Wang, “Generating scientific claims for zero-shot scientific fact checking,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2022.
  * [7] E. Altuncu, J. R. C. Nurse, M. Bagriacik, S. Kaleba, H. Yuan, L. Bonheme, and S. Li, “aedfact: Scientific fact-checking made easier via semi-automatic discovery of relevant expert opinions,” in _Proceedings of the 17th International AAAI Conference on Web and Social Media_ , 2023.
  * [8] J. Vladika and F. Matthes, “Scientific fact-checking: A survey of resources and approaches,” in _Findings of the Association for Computational Linguistics: ACL 2023_ , 2023.
  * [9] L. Rakotoson, C. Letaillieur, S. Massip, and F. A. A. Laleye, “Extractive-boolean question answering for scientific fact checking,” in _Proceedings of the 1st International Workshop on Multimedia AI against Disinformation_ , ser. MAD ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 27–34.
  * [10] G. Kell, I. Marshall, B. Wallace, and A. Jaun, “What would it take to get biomedical QA systems into practice?” in _Proceedings of the 3rd Workshop on Machine Reading for Question Answering_ , A. Fisch, A. Talmor, D. Chen, E. Choi, M. Seo, P. Lewis, R. Jia, and S. Min, Eds. Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 28–41. [Online]. Available: https://aclanthology.org/2021.mrqa-1.3
  * [11] H.-T. Chen, F. Xu, S. Arora, and E. Choi, “Understanding retrieval augmentation for long-form question answering,” _arXiv preprint arXiv:2310.12150_ , 2023.
  * [12] W. Yu, D. Iter, S. Wang, Y. Xu, M. Ju, S. Sanyal, C. Zhu, M. Zeng, and M. Jiang, “Generate rather than retrieve: Large language models are strong context generators,” _arXiv preprint arXiv:2209.10063_ , 9 2022. [Online]. Available: https://arxiv.org/pdf/2209.10063.pdf
  * [13] B. Galitsky, “Truth-o-meter: Collaborating with llm in fighting its hallucinations,” _Preprints_ , 7 2023. [Online]. Available: http://dx.doi.org/10.20944/preprints202307.1723.v1
  * [14] S. Xu, L. Pang, H. Shen, X. Cheng, and T.-S. Chua, “Search-in-the-chain: Towards accurate, credible and traceable large language models for knowledge-intensive tasks,” _arXiv preprint arXiv:2304.14732_ , 2023.
  * [15] J. Liu, J. S. Jin, Z. Wang, J. Cheng, Z. Dou, and J.-R. Wen, “Reta-llm: A retrieval-augmented large language model toolkit,” _arXiv preprint arXiv:2306.05212_ , 6 2023. [Online]. Available: https://arxiv.org/pdf/2306.05212.pdf
  * [16] S. Bhatia, J. H. Lau, and T. Baldwin, “Automatic claim review for climate science via explanation generation,” _arXiv preprint arXiv:2107.14740_ , 8 2021. [Online]. Available: https://arxiv.org/pdf/2107.14740.pdf
  * [17] S. Siriwardhana, R. Weerasekera, E. Wen, T. Kaluarachchi, R. Rana, and S. Nanayakkara, “Improving the domain adaptation of retrieval augmented generation (RAG) models for open domain question answering,” _Transactions of the Association for Computational Linguistics_ , vol. 11, pp. 1–17, jan 2023. [Online]. Available: https://aclanthology.org/2023.tacl-1.1
  * [18] V. Sanca and A. Ailamaki, “E-scan: Consuming contextual data with model plugins,” in _Joint Workshops at 49th International Conference on Very Large Data Bases (VLDBW’23)_ , 2023.
  * [19] Y. Han, C. Liu, and P. Wang, “A comprehensive survey on vector database: Storage and retrieval technique, challenge,” _arXiv preprint arXiv:2310.11703_ , 2023.
  * [20] F. Bang, “GPTCache: An open-source semantic cache for LLM applications enabling faster answers and cost savings,” in _Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)_ , L. Tan, D. Milajevs, G. Chauhan, J. Gwinnup, and E. Rippeth, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 212–218. [Online]. Available: https://aclanthology.org/2023.nlposs-1.24
  * [21] N. Muennighoff, N. Tazi, L. Magne, and N. Reimers, “Mteb: Massive text embedding benchmark,” _arXiv preprint arXiv:2210.07316_ , 2022.
  * [22] N. Arabzadeh, X. Yan, and C. L. A. Clarke, “Predicting efficiency/effectiveness trade-offs for dense vs. sparse retrieval strategy selection,” _arXiv preprint arXiv:2109.10739_ , 2021.
  * [23] C. Sciavolino, Z. Zhong, J. Lee, and D. Chen, “Simple entity-centric questions challenge dense retrievers,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 6138–6148. [Online]. Available: https://aclanthology.org/2021.emnlp-main.496
  * [24] N. Mihindukulasooriya, S. Tiwari, C. F. Enguix, and K. Lata, “Text2kgbench: A benchmark for ontology-driven knowledge graph generation from text,” _arXiv preprint arXiv:2308.02357_ , 2023.
  * [25] M. F. M. Chowdhury, M. Glass, G. Rossiello, A. Gliozzo, and N. Mihindukulasooriya, “Kgi: An integrated framework for knowledge intensive language tasks,” _arXiv preprint arXiv:2204.03985_ , 9 2022. [Online]. Available: https://arxiv.org/pdf/2204.03985.pdf
  * [26] J. Baek, A. F. Aji, and A. Saffari, “Knowledge-augmented language model prompting for zero-shot knowledge graph question answering,” in _Proceedings of the First Workshop on Matching From Unstructured and Structured Data (MATCHING 2023)_. Toronto, ON, Canada: Association for Computational Linguistics, Jul. 2023, pp. 70–98. [Online]. Available: http://dx.doi.org/10.18653/v1/2023.matching-1.7
  * [27] I. Beltagy, K. Lo, and A. Cohan, “SciBERT: A pretrained language model for scientific text,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , K. Inui, J. Jiang, V. Ng, and X. Wan, Eds. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 3615–3620. [Online]. Available: https://aclanthology.org/D19-1371
  * [28] F. Brasileiro, J. a. P. A. Almeida, V. A. Carvalho, and G. Guizzardi, “Applying a multi-level modeling theory to assess taxonomic hierarchies in wikidata,” in _Proceedings of the 25th International Conference Companion on World Wide Web_ , ser. WWW ’16 Companion. Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steering Committee, 2016, p. 975–980.
  * [29] F. B. Rogers, “Medical subject headings,” _Bull. Med. Libr. Assoc._ , vol. 51, pp. 114–116, Jan. 1963.
  * [30] D. Altinok, “An ontology-based dialogue management system for banking and finance dialogue systems,” _arXiv preprint arXiv:1804.04838_ , 2018.
  * [31] W. Xiong, X. L. Li, S. Iyer, J. Du, P. Lewis, W. Y. Wang, Y. Mehdad, W. tau Yih, S. Riedel, D. Kiela, and B. Oğuz, “Answering complex open-domain questions with multi-hop dense retrieval,” _arXiv preprint arXiv:2009.12756_ , 2021.
  * [32] N. Reimers and I. Gurevych, “Making monolingual sentence embeddings multilingual using knowledge distillation,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 11 2020. [Online]. Available: https://arxiv.org/abs/2004.09813
  * [33] ——, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 11 2019. [Online]. Available: https://arxiv.org/abs/1908.10084
  * [34] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” _arXiv preprint arXiv:2307.03172_ , 2023.
  * [35] W. Shen, C. Li, H. Chen, M. Yan, X. Quan, H. Chen, J. Zhang, and F. Huang, “Small llms are weak tool learners: A multi-llm agent,” _arXiv preprint arXiv:2401.07324_ , 2024.
  * [36] G. Juneja, S. Dutta, S. Chakrabarti, S. Manchanda, and T. Chakraborty, “Small language models fine-tuned to coordinate larger language models improve complex reasoning,” _arXiv preprint arXiv:2310.18338_ , 2023.
  * [37] S. Verma, K. Tran, Y. Ali, and G. Min, “Reducing llm hallucinations using epistemic neural networks,” _arXiv preprint arXiv:2312.15576_ , 2023.
  * [38] J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu, “Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,” _arXiv preprint arXiv:2402.03216_ , 2024.