[1]Qi Dou

1]Department of Computer Science and Engineering, The Chinese University of
Hong Kong, HKSAR, China

2]Department of Biomedical Engineering, The Chinese University of Hong Kong,
HKSAR, China

# Efficient EndoNeRF Reconstruction and Its Application for Data-driven
Surgical Simulation

Yuehao Wang  Bingchen Gong  Yonghao Long  Siu Hin Fan  [  [

###### Abstract

Purpose The healthcare industry has a growing need for realistic modeling and
efficient simulation of surgical scenes. With effective models of deformable
surgical scenes, clinicians are able to conduct surgical planning and surgery
training on scenarios close to real-world cases. However, a significant
challenge in achieving such a goal is the scarcity of high-quality soft tissue
models with accurate shapes and textures. To address this gap, we present a
data-driven framework that leverages emerging neural radiance field technology
to enable high-quality surgical reconstruction and explore its application for
surgical simulations.

Method We first focus on developing a fast NeRF-based surgical scene 3D
reconstruction approach that achieves state-of-the-art performance. This
method can significantly outperform traditional 3D reconstruction methods,
which have failed to capture large deformations and produce fine-grained
shapes and textures. We then propose an automated creation pipeline of
interactive surgical simulation environments through a closed mesh extraction
algorithm.

Results Our experiments have validated the superior performance and efficiency
of our proposed approach in surgical scene 3D reconstruction. We further
utilize our reconstructed soft tissues to conduct FEM and MPM simulations,
showcasing the practical application of our method in data-driven surgical
simulations.

Conclusion We have proposed a novel NeRF-based reconstruction framework with
an emphasis on simulation purposes. Our reconstruction framework facilitates
the efficient creation of high-quality surgical soft tissue 3D models. With
multiple soft tissue simulations demonstrated, we show that our work has the
potential to benefit downstream clinical tasks, such as surgical education.

###### keywords:

3D Reconstruction, NeRF, Robotic Surgery, Surgery Simulation

##  1 Introduction

The development of realistic robotic surgery scenes is important for VR-based
surgical training. The conventional method for creating these surgery scenes
involves manual creation of soft tissue models with in-vivo textures by
skilled artists. However, this approach is highly time-consuming and restricts
the level of detail and variety achievable in surgical simulation. To overcome
these limitations, we propose an automated approach to reconstruct interactive
surgical environments using captured real data.

Surgical reconstruction [1, 2, 3, 4, 5, 6, 7], as an emerging task, aims to
recover the 3D shapes and appearance of soft tissues from in-vivo surgery
videos. As pointed out by previous literature [6, 7], surgical reconstruction
is cursed with three typical challenges over natural scene reconstruction: 1)
Soft tissues will undergo large and drastic deformations. Many surgical
operations, e.g., cutting and tearing, can even damage the topologies of soft
tissues. 2) Surgical tools usually appear on the surgery videos and partially
occlude underlying soft tissues from observation. 3) Endoscopic surgery videos
are captured in confined in-vivo spaces, resulting in limited multi-view
geometric clues of the 3D shapes. Our recent work EndoNeRF [7] exploits the
strong capacity of NeRF [8] for scene representations and incorporates
tailored modules for handling tool occlusion and single-viewpoint input,
achieving significant improvements in surgical reconstruction, particularly
for scenes with large deformations. However, EndoNeRF encounters new practical
challenges when constructing surgical simulation environments. First, the
process of reconstructing a surgical scene from endoscopic videos using
EndoNeRF is inefficient, requiring over 10 hours for per-scene optimization.
Second, the optimized geometry of EndoNeRF is represented in a purely implicit
field, i.e., the whole scene is encoded by network parameters. However, many
physically-based methods in soft-body simulation [9, 10, 11, 12] require
explicit geometry model, e.g., meshes, particles, or tetrahedrons, rather than
implicit fields. It is also worth noting that the realistic interaction of
soft tissues is reliant on the underlying content beneath the tissue surface.
While the geometry in the EndoNeRF only represents the surfaces of soft
tissues. Hence, apart from surface reconstruction, another significant
challenge lies in recovering topologically closed counterparts of soft tissues
for simulation purposes.

To fill this gap, this work is the first attempt to create surgical simulation
environments with soft tissue surfaces automatically reconstructed from
endoscopic surgery videos. Technically, we propose a novel framework for
dynamic surgical reconstruction, which can yield realistic and simulator-
friendly counterparts of the soft tissues in the input robotic surgery videos.
We summarize our main contributions as follows:

  * •

We adopt a novel voxel grids-based scene representation for faster dynamic
surgical scene reconstruction.

  * •

We build a pipeline for converting radiance fields into a closed mesh, which
enables physically-based simulation of the reconstructed surgical scenes.

  * •

We exhibit multiple robotic surgery simulations with our reconstructed soft
tissues on multiple simulation engines, including Taichi MPM [13, 14] and
NVIDIA Isaac Sim [15].

This work builds upon a preliminary version presented at MICCAI 2022 [7]. In
this paper, we have made significant revisions and extensions to the original
conference version. The major improvements include:

  * –

We designed a new deformable scene representation with grid-based radiance
fields and 4D tensor-decomposed motion fields for faster training convergence.

  * –

We proposed a novel pipeline for extracting closed meshes from radiance
fields, in order to generate simulatable soft tissues.

  * –

We conduct multiple surgical scene simulations with our reconstructed soft
tissues.

Our code is available at https://github.com/med-air/EndoNeRF.

##  2 Method

Figure 1: Pipeline of our proposed FastEndoNeRF framework, consisting of a
4D-decomposed motion field and dense 3D voxel grids.

We first aim to propose a dynamic scene representation to model soft tissue’s
3D shapes and textures from a stereo video clip of a dynamic surgical scene.
Then we devise a particular de-occlusion rendering and stereo depth-supervised
loss for optimizing the scene representation. Finally, we fill the
reconstructed mesh surfaces into closed meshes and perform soft-body
simulations on the filled meshes. The detailed descriptions are as follows.

###  2.1 Efficient EndoNeRF Scene Representations

In order to enable high-fidelity reconstruction of the surgical simulation
environments, we resort to neural radiance fields. The fundamental neural
radiance fields [8] for 3D scene representations are modeled in a coordinate-
based MLP. Optimizing such scene representation to convergence is slow.
Alternatively, we adopt an implicit-explicit voxel grids-based scene
representation, which is shown to achieve much faster optimization [16, 17,
18, 19]. Specifically, we model the shape and appearance of the scene in
density volume grids $\mathbf{V}_{\sigma}\in\mathbb{R}^{H\times W\times D}$
and feature volume grids $\mathbf{V}_{a}\in\mathbb{R}^{C\times H\times W\times
D}$, where $H$, $W$, and $D$ are the resolutions for the x, y and z dimensions
and $C$ is the channel number of the appearance features. For the density
volume grids $\mathbf{V}_{\sigma}$, each grid vertex maintains its occupancy
probability. For the feature volume grids $\mathbf{V}_{a}$, each grid vertex
holds an appearance code. To map the appearance code into RGB color, we
introduce a shallow MLP $S_{\Theta}:\mathbb{R}^{C}\rightarrow\mathbb{R}^{3}$
as a learnable implicit shading module. The geometry and appearance of any
point $\boldsymbol{x}$ in the 3D space can be retrieved via tri-linear
interpolation (denoted as $\operatorname{interp}(\cdot)$) of the 8 surrounding
vertices’ densities and features, i.e., the density
$\sigma(\boldsymbol{x})=\operatorname{interp}(\boldsymbol{x},\mathbf{V}_{\sigma})$
and the color
$\mathbf{c}(\boldsymbol{x})=S_{\Theta}(\operatorname{interp}(\boldsymbol{x},\mathbf{V}_{a}))$.

Next, we consider surgical scene deformations. A dynamic surgical scene can be
decomposed into a canonical radiance field and a time-dependent deformation
field [20, 21]. Thereby the dynamic scene at time $t$ can be viewed as the
canonical field warped by the deformation field at $t$. In our proposed
method, the canonical radiance field is represented by the
$\mathbf{V}_{\sigma}$ and $\mathbf{V}_{a}$. To support large and topology-
varying deformations, we adopt decomposed 4D motion fields and a 3-layer MLP
to model the deformation field, which maps a spatial-temporal coordinate
$(\boldsymbol{x},t)$ into its corresponding displacement
$\Delta\boldsymbol{x}$. In specific, we define a motion feature field as a
$H\times W\times D\times T\times C_{T}$ tensor $\mathcal{T}$ [22], where $T$
is the resolution of the time dimension and $C_{T}$ is the temporal feature
channel number. Direct dense 5D modeling of the motion feature field is costly
in storage and over-high-dimensional for optimization on sparsely captured
frames. Thus, we need to seek another compact representation. Since
deformations can be locally continuous and low-rank, as observed in [23, 19],
we can decompose this tensor via outer product (Eq. 1):

$\mathcal{T}=\sum_{r=1}^{R_{1}}\tau_{r}^{4}\circ\mathcal{V}_{r}^{1,2,3}\circ
b_{r}^{4}+\sum_{r=1}^{R_{2}}\tau_{r}^{3}\circ\mathcal{V}_{r}^{1,2,4}\circ
b_{r}^{3}+\sum_{r=1}^{R_{3}}\tau_{r}^{2}\circ\mathcal{V}_{r}^{1,3,4}\circ
b_{r}^{2}+\sum_{r=1}^{R_{4}}\tau_{r}^{1}\circ\mathcal{V}_{r}^{2,3,4}\circ
b_{r}^{1},$ (1)

where $R_{1},R_{2},R_{3}$ and $R_{4}$ are expected rank for each dimension,
$\tau_{r}^{l}$ is a 1-D vector of the $l$-th dimension, $b_{r}^{l}$ is a
feature basis of the $l$-th dimension, and $\mathcal{V}_{r}^{i,j,k}$ is a 3-D
volume encompassing $i,j,k$-th dimensions. For each continuously queried point
$(\boldsymbol{x},t)$, we tri-linearly interpolate component tensors
$\tau_{r}^{l}$ and $\mathcal{V}_{r}^{i,j,k}$ to obtain a motion feature
vector. Then we feed the motion feature vector into a 3-layer MLP $G_{\phi}$
to compute the output displacement vector. In this way, the corresponding
coordinates in the canonical field can be obtained by
$\boldsymbol{x}^{\prime}=\boldsymbol{x}+\Delta\boldsymbol{x}(\boldsymbol{x},t)$
with
$\Delta\boldsymbol{x}(\boldsymbol{x},t)=G_{\phi}(\operatorname{interp}(\boldsymbol{x},t,\mathcal{T}))$.

###  2.2 Rendering and Optimization

Volume rendering. With this scene representation, we can reconstruct the
deformable surgical scene by optimizing the loss between rendered color
$\hat{\mathbf{C}}$ and ground truth color $\mathbf{C}$. Specifically, the
rendered color of the ray $\mathbf{r}(z)=\mathbf{o}+z\mathbf{d}$ at time $t$
can be evaluated by volume rendering as shown in Eq. 2:

$\hat{\mathbf{C}}(\mathbf{r}(z),t)=\sum^{M}_{j=1}w_{j}\mathbf{c}_{j},\quad
w_{j}=\exp\bigg{(}-\sum_{i=1}^{j-1}\sigma_{i}\delta_{i}\bigg{)}\bigg{(}1-\exp\big{(}-\sigma_{j}\delta_{j}\big{)}\bigg{)},$
(2)

where $M$ is the number of sampled points along $\mathbf{r}(z)$, $\delta_{i}$
is the sampling step length, $\sigma_{j}$ and $\mathbf{c}_{j}$ are the density
and color of the j-th sample evaluated by
$\sigma(\boldsymbol{x}_{j}+\Delta\boldsymbol{x}(\boldsymbol{x}_{j},t))$ and
$\mathbf{c}(\boldsymbol{x}_{j}+\Delta\boldsymbol{x}(\boldsymbol{x}_{j},t))$,
respectively. The attenuation term $w_{j}$ can be regarded as the probability
that the ray is transmitted to the j-th sample.

De-occlusion of surgical tools. According to the literature [6, 7], soft
tissues in surgical videos can often be occluded by surgical tools in the
foreground. To address this issue and accurately reconstruct the soft tissues,
our approach focuses on training the rays corresponding to tool pixels.
Following the methodology proposed in EndoNeRF [7], we generate binary tool
masks for the left view of each frame. Instead of the mask-guided ray sampling
proposed in EndoNeRF [7], which bypasses rays per training iteration, we pre-
compute all possible camera rays and check for intersections between these
rays and the tool masks prior to training. This saves the computational costs
during the scene optimization procedure, resulting in faster training. Any
rays that pass through the tool masks are excluded from the training process.
During training, the training batch $\mathcal{R}$ is randomly sampled from the
pre-computed rays that have been screened in this manner. By doing so, we
ensure that the optimization of the scene representation bypasses the tool
pixels. Leveraging the auto-interpolation property of radiance fields, we can
patch the occluded soft tissue areas using information from adjacent frames
throughout the training procedures.

Distillation of stereo correspondence. To exploit stereo geometry in confined
in-vivo input, we propose to leverage stereo geometry to enrich 3D clues over
the optimization of the scene representation. Very recent work unimatch [24]
learns dense correspondence on general vision datasets in a unified
formulation for optical flow, stereo matching, and depth estimation tasks. Due
to its superior performance over the previous method [25], we propose to
distill stereo correspondence learned on general data into the surgical data
along with the optimization of the surgical scene. To measure the learned
stereo correspondence of the scene representation, we render depth from the
radiance fields via
$\hat{\mathbf{D}}(\mathbf{r}(z),t)=\sum^{M}_{j=1}w_{j}z_{j}$, where $z_{j}$ is
the distance of the j-th sample along the ray $\mathbf{r}(z)$. The rendered
depth is expected to converge to the estimated stereo depth once well-matched
stereo correspondence is attained by optimizing the scene representation.
Thus, we estimate stereo depth $\mathbf{D}(\mathbf{r}(z),t)$ by stereo-
matching the feature correspondence of the robotic surgery videos from
unimatch [24]. Lastly, we add a depth-supervised loss to the objective
function, resulting in the final loss function:

$\begin{split}\mathcal{L}&=\sum_{\mathbf{r}(z)\in\mathcal{R},t\in[0,1]}\left\|\hat{\mathbf{C}}(\mathbf{r}(z),t)-\mathbf{C}(\mathbf{r}(z),t)\right\|_{2}^{2}+\lambda_{d}\operatorname{Huber}\left(\hat{\mathbf{D}}(\mathbf{r}(z),t),\mathbf{D}(\mathbf{r}(z),t)\right),\end{split}$
(3)

where $\mathbf{C}(\mathbf{r}(z),t)$ and $\mathbf{D}(\mathbf{r}(z),t)$ is the
corresponding ground truth pixel color and unimatch [24] stereo depth of
camera ray $\mathbf{r}(z)$ at the time $t$. Here we adopt Huber loss [26]
which is more stable to outliers. Compared with the stereo depth maps
predicted by STTR [25], supervising better depth maps via unimatch can further
decrease the training time since the depth refinement module proposed in
EndoNeRF [7], which requires depth rendering of all training images, is no
longer needed.

Data: Open mesh surface $\mathcal{M}=(\mathcal{V},\mathcal{F})$ which has one
hole, soft tissue thickness $\zeta$

Result: Closed mesh surface with updated $\mathcal{V}$ and $\mathcal{F}$

$\mathcal{A}\leftarrow\text{non-manifold edges of $\mathcal{F}$}$;

for  _each vertex $v\in\mathcal{V}$_ do

$adj_{v}\leftarrow\\{\\}$;

end for

for  _each edge $\\{u,v\\}\in\mathcal{A}$_ do

$adj_{u}\leftarrow adj_{u}\cup\\{v\\}$;

$adj_{v}\leftarrow adj_{v}\cup\\{u\\}$;

end for

/* Conduct BSF to sort boundary vertices into an ordered list $\mathcal{C}$ */

$\\{u,v\\}\leftarrow\operatorname{Head(\mathcal{A})}$ ;

$\mathcal{C}\leftarrow\\{u\\}$ ;

while  _$v\neq u$_ do

$\operatorname{Enqueue(\mathcal{C},v)}$ ;

$u\leftarrow v$;

for  _each $k\in adj_{v}$_ do

if  _$k\notin\mathcal{C}$_ then

$v\leftarrow k$;

break;

end if

end for

end while

$\operatorname{Enqueue(\mathcal{C},\operatorname{Head(\mathcal{C})})}$;

$center\leftarrow(\frac{1}{\left|\mathcal{V}\right|}\sum_{v\in\mathcal{V}}X_{v},\frac{1}{\left|\mathcal{V}\right|}\sum_{v\in\mathcal{V}}Y_{v},\zeta)$;

$\mathcal{V}\leftarrow\mathcal{V}\cup\\{center\\}$;

$first\leftarrow\operatorname{True}$;

/* Iteratively build a base plane of the soft tissue surface and connect the
base with the boundary vertices along $\mathcal{C}$ */

while  _$!\operatorname{isEmpty(\mathcal{C})}$_ do

$v\leftarrow\operatorname{Dequeue(\mathcal{C})}$;

if  _$!first$_ then

/* Projection of the last $v$ */

$p\leftarrow v^{\prime}$;

end if

/* Projection of $v$ in the appended base of the soft tissue */

$v^{\prime}\leftarrow(X_{v},Y_{v},\zeta)$;

$\mathcal{V}\leftarrow\mathcal{V}\cup\\{v^{\prime}\\}$;

if  _$!\operatorname{isEmpty(\mathcal{C})}$_ then

$u\leftarrow\operatorname{Head(\mathcal{C})}$;

$\mathcal{F}\leftarrow\mathcal{F}\cup\\{(v,u,v^{\prime})\\}$;

end if

if  _$!first$_ then

$\mathcal{F}\leftarrow\mathcal{F}\cup\\{(p,v,v^{\prime})\\}$;

$\mathcal{F}\leftarrow\mathcal{F}\cup\\{(p,center,v^{\prime})\\}$;

end if

$first\leftarrow\operatorname{False}$;

end while

Algorithm 1 mesh-open2closed (Sec. 2.3)

###  2.3 Extraction of Closed Meshes for Soft-Body Simulations

Figure 2: Reconsruction results. The first column gives the reference input
image, the second column exhibits reconstructed point clouds, the third column
shows the meshes obtained by Poisson surface reconstruction on the point
clouds, and the last column displays the closed meshes appended with a base.

After we obtain an optimized dynamic radiance field, we aim to perform
physically-based simulations on the reconstruction. Numerically solving
physically-based simulation systems requires dividing the object material
domain into a number of geometry primitives. Since our reconstructed scene
representation only encodes the seen soft tissue surface in an implicit
geometry, we need to first obtain its explicit form and convert it into a
simulatable object. To do this, we propose the following procedure. We first
render the reconstructed canonical radiance fields to color and depth maps.
Then, we back-project RGB-D maps into point clouds. Namely, each 3D point
$(x,y,z)$ can be computed from a corresponding pixel $(P_{u},P_{v})$ with
depth value $\hat{\mathbf{D}}$ as
$(x,y,z)=(\hat{\mathbf{D}}(P_{u}-C_{x})/f,\hat{\mathbf{D}}(P_{v}-C_{y})/f,\hat{\mathbf{D}})$,
where $(C_{x},C_{y})$ is the principal point and $f$ is the focal length.
Bilateral filtering is also applied to smooth the point clouds. After
conversion to point clouds, we perform Poisson surface reconstruction to
extract the mesh surface from the simplified point clouds. Subsequently, we
need to construct supporting structures underneath the surface for deformable
object simulations. Material Point Method (MPM) and Finite Element Method
(FEM) both require a closed mesh surface as the input for discretization. For
the MPM solvers, dense particles are sampled to fill the soft tissue surface
[27]. As for the FEM solver, robust tetrahedral meshing algorithms [28, 29,
30] are proposed to convert surface objects into tetrahedrons. Thus, we tailor
an efficient mesh-open2closed algorithm that can universally enclose the
reconstructed mesh surfaces. The pseudocode of the algorithm is given in
Algorithm 1, where the input mesh vertices $\mathcal{V}$ and triangles
$\mathcal{F}$ are structured in 2D arrays, and $X_{v}$, $Y_{v}$ denote the x
and y dimensions of vertex $v$. The algorithm begins with constructing the
boundary edges of the reconstructed surface and organizing them in a list.
Those boundary edges can be classified by non-manifold test, i.e., manifold
edges should be simultaneously included in 2 triangles. After finding non-
manifold edges, we can conduct a breadth-first search (BFS) to sort boundary
vertices into an ordered list. Then, we iteratively build a base plane of the
soft tissue surface in the shape of the open mesh boundary and connect the
base with the boundary vertices along the ordered list. During this procedure,
we loop for each vertex $v$ in the ordered edge list and find the projection
of $v$ in the appended base of the soft tissue. Then we connect the projection
of the last $v$, the projection of the current $v$, and the center of the base
plane to create new faces. It is noteworthy that our algorithm is designed for
the input mesh with a single “hole”, i.e., there is only one connected edge.
This assumption usually holds since the incisions on soft tissues are
relatively shallow in in-vivo surgical scenes. If there are two disjoint
surfaces represented in the reconstructed field, a solution is to run the
algorithm separately for each surface.

Table 1: Quantitative evaluation and comparison of our method and baselines. We evaluate photometric errors and training time of the dynamic reconstruction. Methods |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  Time $\downarrow$
---|---|---|---|---
E-DSSR [6] |  13.398 $\pm$ 1.387 |  0.630 $\pm$ 0.057 |  0.423 $\pm$ 0.047 |  0
EndoNeRF [7] |  29.831 $\pm$ 2.208 |  0.925 $\pm$ 0.020 |  0.081 $\pm$ 0.022 |  > 10h
FastEndoNeRF |  29.586 $\pm$ 3.48 |  0.892 $\pm$ 0.040 |  0.145 $\pm$ 0.050 |  27min

##  3 Experiments

###  3.1 Evaluation of Efficient EndoNeRF

Figure 3: Comparisons of reconstruction quality between EndoNeRF and our
method within the first 3 minutes of training.

We conducted an evaluation of our proposed method on a set of typical clips of
robotic surgery videos, captured from 10 cases of our in-house DaVinci robotic
prostatectomy dataset. In addition to the cases used in EndoNeRF [7], the new
cases contain suturing, bleeding, and cutting on soft tissues. Each clip
lasted for 4 to 8 seconds and was sampled into 45 $\sim$ 180 frames. These
clips were captured from stereo cameras, and they encompassed challenging
scenes with non-rigid deformation and tool occlusion. To establish the
effectiveness of our new method, we compared it with two strong baselines: the
recent NeRF-based method EndoNeRF [7] and the traditional DynamicFusion-based
approach E-DSSR [6]. For qualitative evaluation, we exhibit the reconstruction
objects produced by our method, including reconstructed point clouds, surface
meshes, and closed meshes. Due to clinical regulations, it is infeasible to
collect ground truth depth for numerical evaluation on 3D structures. To
perform quantitative comparisons, we instead used photometric errors, such as
PSNR, SSIM, LPIPS, and training time, as evaluation metrics. This evaluation
methodology is consistent with that used in previous work on surgical scene
reconstruction, such as [6, 7], and is widely used in the field of neural
rendering.

Figure 2 showcases our reconstruction outcomes, including extracted point
clouds, soft tissue mesh surfaces, and closed meshes. Our FastEndoNeRF
algorithm excels at reconstructing watertight surfaces of soft tissues from
videos, faithfully capturing the intricate in-vivo textures. Despite the
presence of large deformations, our method tracks the dynamics of the soft
tissues using our proposed 4D decomposed motion field. For tool occlusion in
the input videos, our method manages to patch tool-occluded areas by
leveraging information from adjacent frames, ensuring a comprehensive and
watertight representation of the dynamic soft tissue. In order to ensure that
the reconstructed surface is suitable for simulation purposes in contemporary
simulation engines, we have employed a mesh extraction scheme capable of
constructing high-resolution meshes with intricate textures and shapes from
the reconstructed point clouds. Furthermore, our proposed mesh-open2closed
algorithm facilitates the creation of a closed structure by appending a base
to the mesh surface. This closed structure is essential for enabling accurate
simulations in the chosen environment. In Figure 3, we run our method and the
original EndoNeRF [7] on the same NVIDIA RTX 3090 GPU for 3 minutes and
compare their training efficiency. Due to the limited training time, the
reconstruction results obtained with EndoNeRF remain noisy and blurry.
Conversely, our method demonstrates impressive performance even at an early
training stage (i.e., 10s to 60s), with the ability to approximate the scene’s
appearance and shape accurately. This validates the superior training
convergence speed of our proposed scene representation. It is noteworthy that
our model employs $\sim$160M parameters, consuming 4GB GPU memory for training
each case. Without factorizing the 4D deformation field, the 4D deformation
field would necessitate an allocation of over 12GB of memory during the
training procedure, which shows the effectiveness of our compact dynamic scene
representations.

Figure 4: Soft tissue simulation results. The first row exhibits real-time
interaction between surgical tools and reconstructed soft tissues in NVIDIA
Isaac Sim [15]. The second presents a simulation example of soft tissue
incision with MLS-MPM algorithm [14] implemented in Taichi [13].

Table 1 displays a quantitative comparison of the metrics PSNR, SSIM, LPIPS,
and training time. Both methods exhibit impressive photometric results when
compared to the traditional method of E-DSSR [6]. Despite a slight decrease in
performance, FastEndoNeRF achieves a remarkable training time improvement of
approximately 20 times faster than EndoNeRF. By training FastEndoNeRF for 27
minutes, we can achieve comparable quality to EndoNeRF trained for over 10
hours. This highlights the efficiency and effectiveness of the FastEndoNeRF
approach.

###  3.2 Initial Application for Surgical Scene Simulation

Virtual surgical training platforms have become increasingly significant in
surgery education and training [31, 32]. However, building a surgical
education and training platform is associated with several challenges,
including limited exposure to real-life surgical cases, and limited access to
high-fidelity simulation. Our proposed framework can overcome these challenges
by providing a reconstructed realistic environment for surgical trainees to
practice and master their skills.

Real-time FEM simulation. Here we first build a real-time virtual surgery
simulation in NVIDIA Isaac Sim [15], where FEM is the solver for simulating
reconstructed continuum objects. In the first row of Figure 4, we import a
reconstructed closed mesh into NVIDIA Isaac Sim and tune its physical
properties to make it behave like soft tissues. Owing to advanced GPU
acceleration, NVIDIA Isaac Sim enables real-time FEM simulation and rendering,
producing high-fidelity deformations under the dissection interaction. The
automatic reconstruction of the simulation environment from real surgical
videos ensures that the in-vivo textures are accurately preserved, thereby
enhancing the visual realism of surgical simulations. The proposed algorithm
for closed mesh extraction facilitates material domain discretization for the
FEM solver within Isaac Sim. If the imported meshes are not closed in Isaac
Sim, the mesh tetrahedralization procedure will fail, resulting in
unreasonable simulation effects. Moreover, the creation procedure for this
simulation environment is highly scalable, thanks to the efficiency of the
surgical reconstruction pipeline.

MPM simulation. While the FEM solver in NVIDIA Isaac Sim achieves basic soft-
body simulation, it lacks the ability to perform damage operations on
continuum objects, which is considered a crucial aspect of simulating soft
tissues. In order to address this limitation, we employ the Material Point
Method (MPM) [33], a hybrid grid-particle method that combines the strengths
of both Eulerian and Lagrangian approaches. This method enables us to handle
large deformations and complex material behavior, as demonstrated in recent
papers [34, 35]. To specifically support damage deformations on soft bodies
and achieve two-way coupling between rigid and non-rigid objects, we implement
the state-of-the-art MLS-MPM [14]. In Figure 4, the second row illustrates an
example of soft tissue damage resulting from dissection. It is evident that
MLS-MPM is capable of accurately capturing the incision behavior on the soft
tissues. While MPM offers soft tissue damaging simulation, it is characterized
by high computational costs and falls short of achieving real-time
simulations. In the simulation stage, $\sim$5M particles are generated for
simulation, resulting in a memory consumption of around 5GB.

##  4 Conclusion

We present an innovative and data-driven framework for constructing surgical
simulation environments using endoscopic videos. Our approach introduces a new
fast dynamic scene representation based on NeRF, which significantly
accelerates the 3D reconstruction process of surgical scenes. Additionally, we
propose a closed mesh extraction algorithm that converts reconstructed soft
tissue surfaces into simulation objects. To demonstrate the versatility and
applicability of our framework, we showcase multiple simulations of
reconstructed surgical environments for diverse clinical applications. Our
proposed methodology aims to inspire a significant advancement in the field of
surgical simulation and is poised to open up new possibilities for next-
generation surgical training and surgical robot learning.

Limitations and future work. There are still some under-explored problems with
our current methods. First, the de-occlusion of surgical tools relies on the
interpolation of radiance fields, which will cause artifacts in the textures
of occluded soft tissues. This could be solved by incorporating generative
models to inpaint the textures. Second, as an initial trial, our simulation is
based on the naive versions of FEM and MPM. In the future, we aim to test more
simulation algorithms on our reconstructed soft tissues, e.g., XFEM and XMPM,
to achieve more realistic simulation effects.

Acknowledgments

This work was supported in part by a grant from the Research Grants Council of
the Hong Kong Special Administrative Region, China (Project No. 24209223), in
part by the Hong Kong Innovation and Technology Fund (Project No. ITS/223/22),
in part by the Science, Technology and Innovation Commission of Shenzhen
Municipality Project No. SGDX20220530111201008, and in part by InnoHK Multi-
Scale Medical Robotics Center.

## References

  *   * Liu et al. [2020] Liu, X., Stiber, M., Huang, J., Ishii, M., Hager, G.D., Taylor, R.H., Unberath, M.: Reconstructing sinus anatomy from endoscopic video–towards a radiation-free approach for quantitative longitudinal assessment. In: MICCAI, pp. 3–13 (2020)
  * Chen et al. [2020] Chen, W., Liao, X., Sun, Y., Wang, Q.: Improved orb-slam based 3d dense reconstruction for monocular endoscopic image. In: ICVRV, pp. 101–106 (2020)
  * Recasens et al. [2021] Recasens, D., Lamarca, J., Fácil, J.M., Montiel, J., Civera, J.: Endo-depth-and-motion: Reconstruction and tracking in endoscopic videos using depth networks and photometric constraints. IEEE Robotics and Automation Letters 6(4), 7225–7232 (2021)
  * Wei et al. [2021] Wei, G., Yang, H., Shi, W., Jiang, Z., Chen, T., Wang, Y.: Laparoscopic scene reconstruction based on multiscale feature patch tracking method. In: EIECS, pp. 588–592 (2021). IEEE
  * Wei et al. [2022] Wei, R., Li, B., Mo, H., Lu, B., Long, Y., Yang, B., Dou, Q., Liu, Y., Sun, D.: Stereo dense scene reconstruction and accurate localization for learning-based navigation of laparoscope in minimally invasive surgery. IEEE Transactions on Biomedical Engineering 70(2), 488–500 (2022)
  * Long et al. [2021] Long, Y., Li, Z., Yee, C.H., Ng, C.F., Taylor, R.H., Unberath, M., Dou, Q.: E-dssr: efficient dynamic surgical scene reconstruction with transformer-based stereoscopic depth perception. In: MICCAI, pp. 415–425 (2021)
  * Wang et al. [2022] Wang, Y., Long, Y., Fan, S.H., Dou, Q.: Neural rendering for stereo 3d reconstruction of deformable tissues in robotic surgery. MICCAI (2022)
  * Mildenhall et al. [2020] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: ECCV, pp. 405–421 (2020)
  * Müller et al. [2007] Müller, M., Heidelberger, B., Hennix, M., Ratcliff, J.: Position based dynamics. Journal of Visual Communication and Image Representation 18(2), 109–118 (2007)
  * Sifakis and Barbic [2012] Sifakis, E., Barbic, J.: Fem simulation of 3d deformable solids: a practitioner’s guide to theory, discretization and model reduction. In: Acm Siggraph 2012 Courses, pp. 1–50 (2012)
  * Qian et al. [2017] Qian, K., Bai, J., Yang, X., Pan, J., Zhang, J.: Essential techniques for laparoscopic surgery simulation. Computer Animation and Virtual Worlds 28(2), 1724 (2017)
  * Qian et al. [2016] Qian, K., Jiang, T., Wang, M., Yang, X., Zhang, J.: Energized soft tissue dissection in surgery simulation. Computer Animation and Virtual Worlds 27(3-4), 280–289 (2016)
  * Hu et al. [2019] Hu, Y., Li, T.-M., Anderson, L., Ragan-Kelley, J., Durand, F.: Taichi: a language for high-performance computation on spatially sparse data structures. ACM Transactions on Graphics (TOG) 38(6), 1–16 (2019)
  * Hu et al. [2018] Hu, Y., Fang, Y., Ge, Z., Qu, Z., Zhu, Y., Pradhana, A., Jiang, C.: A moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions on Graphics (TOG) 37(4), 1–14 (2018)
  * Liang et al. [2018] Liang, J., Makoviychuk, V., Handa, A., Chentanez, N., Macklin, M., Fox, D.: Gpu-accelerated robotic simulation for distributed reinforcement learning. In: CoRL, pp. 270–282 (2018). PMLR
  * Müller et al. [2022] Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) 41(4), 1–15 (2022)
  * Sun et al. [2022] Sun, C., Sun, M., Chen, H.-T.: Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In: CVPR (2022)
  * Fridovich-Keil et al. [2022] Fridovich-Keil, S., Yu, A., Tancik, M., Chen, Q., Recht, B., Kanazawa, A.: Plenoxels: Radiance fields without neural networks. In: CVPR, pp. 5501–5510 (2022)
  * Fridovich-Keil et al. [2023] Fridovich-Keil, S., Meanti, G., Warburg, F.R., Recht, B., Kanazawa, A.: K-planes: Explicit radiance fields in space, time, and appearance. In: CVPR, pp. 12479–12488 (2023)
  * Pumarola et al. [2021] Pumarola, A., Corona, E., Pons-Moll, G., Moreno-Noguer, F.: D-nerf: Neural radiance fields for dynamic scenes. In: CVPR, pp. 10318–10327 (2021)
  * Park et al. [2021] Park, K., Sinha, U., Barron, J.T., Bouaziz, S., Goldman, D.B., Seitz, S.M., Martin-Brualla, R.: Nerfies: Deformable neural radiance fields. In: ICCV, pp. 5865–5874 (2021)
  * Chen et al. [2022] Chen, A., Xu, Z., Geiger, A., Yu, J., Su, H.: Tensorf: Tensorial radiance fields. In: ECCV, pp. 333–350 (2022)
  * Cao and Johnson [2023] Cao, A., Johnson, J.: Hexplane: A fast representation for dynamic scenes. In: CVPR, pp. 130–141 (2023)
  * Xu et al. [2023] Xu, H., Zhang, J., Cai, J., Rezatofighi, H., Yu, F., Tao, D., Geiger, A.: Unifying flow, stereo and depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
  * Li et al. [2021] Li, Z., Liu, X., Drenkow, N., Ding, A., Creighton, F.X., Taylor, R.H., Unberath, M.: Revisiting stereo depth estimation from a sequence-to-sequence perspective with transformers. In: ICCV, pp. 6197–6206 (2021)
  * Huber [1992] Huber, P.J.: Robust estimation of a location parameter. In: Breakthroughs in Statistics: Methodology and Distribution, pp. 492–518 (1992)
  * Wang et al. [2020] Wang, X., Qiu, Y., Slattery, S.R., Fang, Y., Li, M., Zhu, S.-C., Zhu, Y., Tang, M., Manocha, D., Jiang, C.: A massively parallel and scalable multi-gpu material point method. ACM Transactions on Graphics (TOG) 39(4), 30–1 (2020)
  * Hu et al. [2018] Hu, Y., Zhou, Q., Gao, X., Jacobson, A., Zorin, D., Panozzo, D.: Tetrahedral meshing in the wild. ACM Transactions on Graphics (TOG) 37(4), 60–1 (2018)
  * Hu et al. [2020] Hu, Y., Schneider, T., Wang, B., Zorin, D., Panozzo, D.: Fast tetrahedral meshing in the wild. ACM Transactions on Graphics (TOG) 39(4), 117–1 (2020)
  * Si [2015] Si, H.: Tetgen, a delaunay-based quality tetrahedral mesh generator. ACM Trans. Math. Softw. 41(2) (2015) https://doi.org/10.1145/2629697
  * Long et al. [2022] Long, Y., Li, C., Dou, Q.: Robotic surgery remote mentoring via ar with 3d scene streaming and hand interaction. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 1–6 (2022)
  * Long et al. [2023] Long, Y., Wei, W., Huang, T., Wang, Y., Dou, Q.: Human-in-the-loop embodied intelligence with interactive simulation environment for surgical robot learning. IEEE Robotics and Automation Letters (2023)
  * Sulsky et al. [1995] Sulsky, D., Zhou, S.-J., Schreyer, H.L.: Application of a particle-in-cell method to solid mechanics. Computer physics communications 87(1-2), 236–252 (1995)
  * Wolper et al. [2019] Wolper, J., Fang, Y., Li, M., Lu, J., Gao, M., Jiang, C.: Cd-mpm: continuum damage material point methods for dynamic fracture animation. ACM Transactions on Graphics (TOG) 38(4), 1–15 (2019)
  * Wolper et al. [2020] Wolper, J., Chen, Y., Li, M., Fang, Y., Qu, Z., Lu, J., Cheng, M., Jiang, C.: Anisompm: Animating anisotropic damage mechanics. ACM Transactions on Graphics (TOG) 39(4) (2020)


Intentional binding enhances hybrid BCI control

Tristan Venot1, Arthur Desbois1, Marie Constance Corsi1, Laurent Hugueville1,
Ludovic Saint-Bauzel2 Fabrizio De Vico Fallani1*

1 Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM,
CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière, F-75013, Paris,
France

2 Sorbonne Université, Institut des Systèmes intelligents et de robotiques
ISIR, F-75005, Paris, France

* Corresponding author<EMAIL_ADDRESS>

## Abstract

Mental imagery-based brain-computer interfaces (BCIs) allow to interact with
the external environment by naturally bypassing the musculoskeletal system.

Making BCIs efficient and accurate is paramount to improve the reliability of
real-life and clinical applications, from open-loop device control to closed-
loop neurorehabilitation. By promoting sense of agency and embodiment,
realistic setups including multimodal channels of communication, such as eye-
gaze, and robotic prostheses aim to improve BCI performance. However, how the
mental imagery command should be integrated in those hybrid systems so as to
ensure the best interaction is still poorly understood.

To address this question, we performed a hybrid EEG-based BCI experiment
involving healthy volunteers enrolled in a reach-and-grasp action operated by
a robotic arm. Main results showed that the hand grasping motor imagery timing
significantly affects the BCI accuracy as well as the spatiotemporal brain
dynamics. Higher control accuracy was obtained when motor imagery is performed
just after the robot reaching, as compared to before or during the movement.
The proximity with the subsequent robot grasping favored intentional binding,
led to stronger motor-related brain activity, and primed the ability of
sensorimotor areas to integrate information from regions implicated in higher-
order cognitive functions.

Taken together, these findings provided fresh evidence about the effects of
intentional binding on human behavior and cortical network dynamics that can
be exploited to design a new generation of efficient brain-machine interfaces.

## Introduction

By establishing direct communication pathways between the brain and external
devices, noninvasive brain-computer interfaces (BCIs) have the potential to
become one of the most disruptive technology of the century.

BCIs interpret and translate neural signals, typically recorded with
functional neuroimaging such as electroencephalography (EEG), into commands
that can be understood by computers or devices. Humans communicate their
intention by voluntarily modulating brain activity thanks to mental imagery
tasks, such as hand grasping or mathematical operations [1]. The potential
impact of BCIs is huge, from understanding brain mechanisms underlying
motor/cognitive control and neurofeedback, to the development of prototypes
for substitution and rehabilitation of lost functions (e.g., stroke
recovery)[2, 3].

A crucial requirement for BCIs is the usability in terms of performance, that
is the ability to correctly detect and recognize the user’s mental intent and
ensure a reliable communication. On the computer side, the challenge is to
correctly classify the user’s intent. Basic research in electrophysiology as
well as methodological advances in signal processing and machine learning have
made considerable progresses [4]. On the human side, the challenge is to learn
to generate accurate brain signals so as to ease their detectability from the
computer [5, 6]. Recent works on cognitive science demonstrated that the level
of subjects’ embodiment, i.e. the feeling of the environment as an extension
of the own body, plays a crucial role in facilitating the BCI skill
acquisition process [7]. To this end, the design of immersive setups including
hybrid communication channels and bodily-like robotic effectors, can provide
significant benefit in terms of BCI performance as compared to traditional
display-based configurations [8, 9]. For example, teleoperated androids and
head-mounted displays led to easiness in motor imagery training and higher
levels of embodiment resulting in an overall BCI performance improvement [10,
11, 12, 13, 14]. More in general, embodiment could be even obtained through
non-human looking robots [15, 16].

These findings demonstrate the beneficial effect of considering multimodal
setups in terms of classification accuracy but ignore how they eventually
affect the brain dynamics. Yet, a better understanding of the brain
organizational mechanisms associated with embodying BCIs would allow more
biologically-grounded design optimization. A particularly delicate question is
how the mental imagery process should be integrated in such sophisticated
realistic setups in order to favor the feeling of the user to be in control,
also called sense of agency [17, 18, 19, 20]. Here, we hypothesized that the
timing of the mental imagery process, i.e., the moment when subjects send the
mental imagery command, plays a fundamental role in determining the overall
BCI performance. In particular, the delay between the mental intention and the
related action might significantly affect the user’s sense of agency, a key
prerequisite for embodiment. To test this hypothesis, we realized a
longitudinal randomized hybrid BCI experiment involving a group of healthy
volunteers controlling a robotic arm to reach and grasp a target object. We
tested three different mental imagery timing strategies: i) before, ii)
during, and iii) after the reaching phase. We evaluated the BCI performance in
terms of classification accuracy and sensitivity, and we investigated the
brain dynamics in terms of motor-related activity and functional connectivity.
See Methods for details of the sample, experimental paradigm, and methods of
analysis.

## Results

### Behavioral performance

Fifteen healthy right-handed subjects (8 females) participated in a randomized
longitudinal EEG study consisting in controlling the reach-and-grasp action of
a robotic arm via an hybrid-BCI (Fig. 1a). The goal was to use the eye-gaze to
select a target object and grasp it by means of a right-hand motor imagery
(MI). Across sessions, subjects were instructed to perform the MI task in
different moments, i.e., S1) before, S2) during and S3) after the reaching
phase (Fig. 1b, S1a). Each session started with a calibration, where subjects
were instructed through a visual cue prompted on the table monitor to perform
several trials of MI (grasp) and resting state (no-grasp) tasks. At this stage
the robotic arm reached the target and the grasping depended on the given cue
and not on the recorded brain activity (neurofeedback off). After selecting
the most relevant controlling EEG channels in terms of discriminant power
spectra (File S1), subjects performed the same task but the robotic hand
action was now controlled by the brain activity (neurofeedback on). Based on
the input controlling features, a linear discriminant classifier determined
the type of action, i.e. grasp/no grasp. Two consecutive control blocks were
then realized to allow subjects to practice and to learn the task (Fig. S1b,
Methods).

To assess the role of the intrinsic subjects’ motivation on their ability to
control the BCI, we first measured their reward/effort ratio via an online
questionnaire before the experiment (Methods)[21]. Results showed that the
highest classification accuracies (correct/total trials) tended to be reached
by the most motivated subjects (Fig 2a). Then, we investigated how subjects
became proficient and whether one timing strategy gave better performance. In
average subjects exhibited a significant learning effect across the control
blocks regardless of the timing strategy. However, only S3 gave a
significantly higher accuracy at the end of the session (Fig. 2b, Tab S1). In
a supplementary analysis, we showed that this effect was not visible in the
absence of feedback (Fig. S2). In terms of sensitivity (correct/total MI
trials), the scores were in general very high ($>83\%$) and no significant
effects were reported across blocks or strategies.

### Motor-related spatiotemporal brain dynamics

To understand how the brain responded to the different timing strategies, we
focused on the last control block of the experiment corresponding to the best
achieved accuracy in average. First, we computed the autoregressive-based
power spectrum of the EEG signals corresponding to the MI and rest trials in
the four characteristic frequency bands theta ($4-7Hz$), alpha ($8-12Hz$),
beta ($13-25Hz$) and gamma ($26-35Hz$)[22]. Then, we considered the trial-
averaged power spectra so to have a more robust estimation of the MI and rest
condition for each subject and strategy (Methods).

At the group-level, all strategies exhibited a significant beta power decrease
in the MI condition as compared to the rest one, while no differences were
found in the other bands (Fig. 3a). In terms of spatial distribution, all
strategies involved the sensorimotor area of the brain, while S2 further
exhibited a wider extension notably including frontal premotor EEG channels in
both hemispheres. However, only the stronger and more focused activity of S3
translated into a significant source beta power decrement localized in the
sensorimotor area contralateral to the imagined hand grasping (Fig. 3b).

A more detailed time-frequency analysis of the EEG signals in the best
activated channel (C1) revealed that S3 also elicited an increasingly stronger
motor-related beta-activity as compared to strategies 1 and 2 (Fig. 4,
Methods). This findings indicated that performing MI during the reaching phase
generates stronger brain activity responses both in space and time. Although
strategy 2 elicited higher attentional levels as measured by the pupil
diameter derivative data (Fig. S2), that was not sufficient to generate better
motor-related activation and performance.

### Brain network changes during motor imagery

To better understand the brain organizational properties in the different
timing strategies, we performed a functional connectivity network analysis of
the recorded EEG signals. To this end, we computed the Welch-based spectral
coherence in the same frequency bands considered for the power spectra
(Methods). The resulting brain networks consisted of nodes (the EEG channels)
and weighted links measuring the amount of signal synchronization between
channels.

At the group-level, S3 elicited a higher number of motor-related functional
interactions as compared to the other timing strategies. While this tendency
was reported in every frequency band, a stronger effect was observed for the
beta band (Fig. S3). To quantify how these motor-related functional
connections were spatially distributed and whether they concentrated in
specific brain regions, we then computed the so-called node strength which
measured the total connection intensity for each node (Methods).

Here, only the beta-node strength in S3 showed significant increments in the
MI condition as compared to the rest one, while no differences were found in
the other bands or strategies. Notably, the most significant EEG channels were
all located over the sensorimotor area contralateral to the imagined movement
and exhibited a preferential information integration with EEG channels
covering prefrontal and frontal brain areas in the contralateral hemisphere
and, to a less extent, in the ipsilateral one (Fig. 5).

## Discussion

### Mental imagery and intentional binding

Mental imagery is a crucial element of noninvasive BCIs since it represents
the way the users communicate their intention to the computer. However, mental
imagery is a complex motor/cognitive task and it might be difficult to
generate detectable brain signal features. Maintaining a high level of
engagement during the task and feeling in control of an external device
significantly boosts the MI quality and elicit strong motor-related patterns.
To this end, immersive setups including hybrid BCI solutions and realistic
robotc effectors have effectively demonstrated their superiority compared to
more traditional display-based systems.

In such immersive setups, the role of the MI timing becomes crucial given the
number of interacting components. Notably, the period between the user’s
intent and the subsequent action significantly affects the sense of agency
[23]. According to intentional binding theories, when the action immediately
follows the intention there is a reduced time perception by the user who feels
more in control [24]. Thus, in a reach-and-grasp task, deciding the moment
when MI has to be performed might have important consequences on the final
performance. Despite the fundamental importance of this question, the MI
timing effects have been poorly investigated so far.

Here, we showed a significant difference in BCI learning depending on the
timing strategy. Performing MI after the reaching phase of the robotic arm
produced significantly a higher classification accuracy compared to executing
the MI before or during the reaching phase. This result is actually inline
with intentional binding phenomenon as by design the delay between the
intention and the grasping action was 5, 1, and 0 s for the before, during and
after strategy (Fig. S1a) . These findings are particularly relevant as in
most BCI protocols the MI command is inserted before the reaching phase and
provide a novel grounded criterion to boost sense of agency and therefore
optimize the design of noninvasive BCIs.

### Motor-related brain activity and embodiment

Characterizing the spatial and temporal features of the brain signals allows
to better understand the type of neural mechanisms subserving real and
imagined movements. Different types of motor-related features distinctively
occur before (pre-motor potential BP), during (e.g. event-related
desynchronization ERD) and also after the movement onset (e.g., event-related
synchronization ERS) [25]. These temporary signal changes code the different
phases of deliberate movements, both real or imagined, and typically emerge
from the sensorimotor cortex within the alpha and beta frequency bands. ERDs
are the most used features in BCIs since they take place during the mental
imagery task and can be used to directly control external applications, or be
targeted by neurofeedback training with broad benefit for brain diseases but
also sport activities [26].

In highly immersive setups, a natural question is how multimodality and the
presence of external interacting devices affects the ongoing mental imagery
process and more in general the motor-related brain response. Recent results
showed that including robotic effectors reinforces the sense of embodiment and
leads to better performance[27, 28]. Notably, the passive observation of robot
movement would be sufficient to generate ERD/ERS responses [29, 30]. Here, we
investigated how the MI motor-related brain activity were influenced by the
active control of a robotic arm operating reach-and-grasp goal-oriented task.
We found that the delay between the MI control task and the robot grasping is
crucial to generate spatially-focused and temporally-increasing motor-related
brain responses.

The best result was obtained when the MI is performed after the robot reaching
phase, immediately preceding the target grasping. A plausible explanation
would consist in the presence of a priming effect that the robot has on the
ability of the brain to generate stronger task-related activity. Indeed,
recent studies demonstrated that performing MI just after motor execution and
observation can have beneficial effects on the brain-related response [31,
32]. Altogether, these results indicated that the presence of a robot is not
sufficient per-se to create empowering embodiment effects on MI-related brain
activity. Brain responses can be too weak if MI is performed too much in
advance with respect to the robot movement, or too variable if MI occurs while
the robot is moving, which could induce spurious visual-related attentional
processes [33, 34].

### Brain connectivity networks and BCIs

Understanding how brain areas interact beyond their mere activation is
becoming more and more essential in both basic and clinical neuroscience [35].
Brain connectivity together with recent development in network science have
allowed researchers to gain insights into the organizational and
reorganizational processes of brain functioning across species, scales and
conditions [36]. Determining which are the most connected areas, how
efficiently they exchange information, or what is their tendency to form
highly connected clusters, are just few example witnessing the power of these
methods to quantify biologically-plausible properties such as hub formation,
segregation and integration of information, and modular specialization [37,
38].

The adoption of network-based approaches in BCI science is relatively recent,
mostly due to the historical focus put by the machine learning community on
the improvement of classification algorithms rather than on the identification
of new brain controlling mechanisms [4]. Recent works showed that sensorimotor
regions can modulate their connectivity in BCI-related tasks, such as MI and
resting states, and allow enriching the feature space so as to improve overall
performance. Besides performance, the study of BCI-related brain networks can
therefore provide precious information on how complex cognitive tasks
integrate information from distributed regions, as in BCI learning, and inform
the design of more reliable adaptive prototypes [39, 5].

Here, we showed how the hybrid BCI control of a robotic arm in a reach-and-
grasp task modulated brain connectivity networks. Performing MI after the
reaching phase, immediately preceding the robot grasping, elicited a more
connected network compared to before and during the reaching. Specifically,
the contralateral sensorimotor area increased its connectivity with frontal
and prefrontal areas in the same hemsiphere. These regions support intentional
binding due to their involvement in higher-order cognitive functions and
decision-making (pre-frontal), as well as in action prediction, temporal
integration, and causal inference (frontal) [40]. This increased integration
of information might explain the ability of sensorimotor areas to encode not
only action and sensing, but also subjective agency signals, as recently
demonstrated with intracortical recordings [41].

### Conclusion

The future of noninvasive BCIs depends on their ability to become reliable and
effective. By improving the user’s sense of agency and embodiment, immersive
multimodal prototypes can generate more accurate brain responses and enhance
performance. To this end, we showed that optimal setups should be designed in
order to maximize the user’s intentional binding by placing the motor-imagery
command close to the related action.

We hope that our work will stimulate further studies on the neural mechanisms
underlying hybrid-BCIs beyond the effects on performance. The related brain
dynamics will be crucial to build adaptive, interpretable and personalized BCI
prototypes. Although hybrid BCIs-brain relationships have not received much
attention, we suggest that they will likely have important theoretical and
practical consequences on the future generation of BCIs.

## Methods

### Experimental design

Fifteen healthy BCI-naive subjects (8 females), aged $25±1.5$ years, all
right-handed, provided informed consent and participated voluntarily in the
BRACCIO protocol. The protocol was approved by Inria’s national ethical
committee as part of the BCIPRO protocol ( authorization number 2021-35 - ref
SICOERLE n°179). Experiments took place in the extremely controlled
environment of the EEG/MEG center within the neuroimaging core facility of the
Paris Brain Institute. The experimental setup featured a robotic arm facing
subjects at the opposite side of an augmented table used to display stimuli
cues, visual neurofeedback, and supporting two target objects (two cans).
Subjects used their gaze to select the target to reach and modulated their
brain activity to determine and trigger the type of action (grasping or not)
operated by a robotic arm (Fig. 1a, File S1).

The grasping action was realized by performing a sustained right hand motor
imagery grasping (MI) of the right hand. The no grasping action was realized
by remaining at rest. The type of action, which was randomized within each
sessions, was cued by a visual stimulus on the augmented table as a halo
surrounding the cans (red for MI, blue for resting state). In the grasping
action, the robot reaches the target, grasps it, lifts it, puts it back into
the original position and returns to its starting position. In the no-grasping
action, the robot reaches the object, remains steady and returns to its
starting point. All participants were enrolled in three consecutive
experimental sessions over three weeks, each session dedicated to testing a
distinct timing strategy, i.e., performing the mental task (MI or rest)
before, during or after the robot arm reaching phase. The strategy order was
randomized across subjects.

### Session blocks

Each session was composed of four separate parts for a total duration of 3
hours in average (Fig S6). First, subjects were instructed to get familiar and
practice with the type of movement to be imagined in the subsequent blocks.
This part consisted of 1 run comprising 10 right hand grasping trials and 10
rest trials, each lasting 11 seconds (3 seconds cue, 4 seconds task, 3 seconds
end). In this block, the robot arm was not used. In the next calibration
block, the robotic hand action depended on the given cue (MI/grasping or
rest/no-grasping) and was not related to the user’s brain activity (feedback
off). This block consisted of 3 runs of 10 MI and 10 rest trials, lasting 21.5
seconds each (6.5 seconds cue, 4 seconds task, 11 seconds end).

In the first control block (control 1), the feedback is introduced based on a
LDA training of the EEG controlling features from the calibration block. The
robotic hand action depended on the user’s brain activity (feedback on). This
block includes 3 runs of 10 MI and 10 rest trials, lasting 21.5 seconds each
(6.5 seconds cue, 4 seconds task, 11 seconds end). In the second and last
control block (control 2), feedback is eventually adapted based on a new LDA
training of the EEG controlling features from control 1. The robotic hand
action kept depending on the user’s brain activity (feedback on). This block
includes 2 runs of 10 MI and 10 rest trials, lasting 21.5 seconds each (6.5
seconds cue, 4 seconds task, 11 seconds end).

### Hardware and software

The robotic device was a Pollen Robotic (7 DoF) right arm mounted in front of
the subject, with a claw gripper. The positions of the cans were predetermined
in the robot space. Inverse Kinematics with minimum jerk was used to find
optimal joints configuration for each can[42]. The robot went to its target
based on the choice issued for the eye tracker.

Eye-gaze was recorded and used as a command with a Tobii Pro Glasses 3 set to
50 Hz. Only the x-axis direction was used to choose between right and left.
Negative values were the indication to seize the left can and positive values
were the indication to seize the right can. The augmented table consisted of a
display monitor underneath a plexiglas screen to support cans. Neuro feedback
and stimuli appeared beneath the cans on the black screen. The robot could
reach elements of the table.

EEG signals were acquired using a 64 electrode BrainAmp system within a
magnetically shielded room, with TP9 and TP10 as Reference and Ground
respectively. The sampling rate was set to 500 Hz. Impedance level was set to
15 $k\Omega$ with a tolerance of 10 $k\Omega$ with ActiCap control software.
The EEG acquisition was realized using the OpenViBe 3.3.0 software with
BrainAmp drivers. Gaze flow was acquired continuously and sent to the robot
server that also received decisions coming from the BCI system. The server
made the synchronization between the different modalities by averaging on a
time window the gaze activity (corresponding to the apparition of the crosses
to select the can) and propagated the BCI order at the right moment to the
robot gripper.

### Online signal processing

The eye-gaze was used to communicate to the robotic arm the target it had to
reach. To do so, we infer the 2d gaze location from pupil positions with
regards to the infrared cameras. We establish for each eye a corresponding
vector pointing towards a direction, the crossing of the two eye vectors
determines the estimated gaze posititon.

The EEG online data processing was performed with OpenViBE 3.3.0 [43]. To
avoid possible transitory effects, we did not considered the first second
after the stimulus cue and only kept the remaining 3 s of each trial. In a
preliminary analysis, we actually confirmed higher classification accuracies
after the first second (Fig S4). As a first preprocessing, common average
reference (CAR) was applied to the EEG signals. The spectral analysis was
carried out using the autoregressive (AR) Burg method [44]. The parameters
chosen are the 19 for the model order, 250 ms for the windowing and 0.161 ms
for the overlap, and a 1 Hz frequency bin resolution in the 4-35 Hz range,
parameters were the same for all subjects. The visual feedback was given to
the subject every 250 ms by changing the halo radius proportionally to the
power spectrum of the best EEG controlling feature. To allow smooth changes,
we performed a logarithmic iteration consisting in buffering the power
spectrum and computing its logarithm to damp its variation as in [45] .

The type of action for the robotic hand was instead calculated at the end of
the trial (3 s). To this end, the average of all the accumulated power spectra
were sent for a 2-class LDA classification which determined the type of
action, i.e., grasping/no-grasping.

### Offline data analysis

The motivation score was inferred from online questionnaires that subjects
completed prior to the experimentation. Subjects answered 48 questions
consisting in choosing whether to complete or not a task for a certain reward.
This allowed to measure the ratio between reward and effort. Then, using a
logistic regression between the actual values and those in a previously-
obtained database , we computed a normalized ratio informing on the individual
relative amount of motivation [21].

Given the exceptional quality of the recorded data, no artefact rejection
procedure was applied to the EEG signals for the offline analysis. For the
power spectra estimation, we used the same parameters as in the online
analysis, but we then averaged all the segments and trials so to have a
characteristic MI and rest power spectrum for each subject and strategy.
Similarly, characteristic EEG time-frequency maps were obtained by computing
the AR-based power spectra with the following parameters: temporal resolution
of 0.011s and frequency resolution of 0.1 Hz, the Burg parameters were set to
250 ms for time window with 100 ms of overlap and a filter order set to 19
following [46].

The computation of the source space was done in two steps. In a first step, we
computed a forward model from the sensor space to a template brain based on a
combination of 40 healthy MRI scans from freesurfer software with 20484
vertices. In a second step, we project the sensor activity onto the source
space. The regularisation parameter $\lambda^{2}$ is set to $1/3$ for source
estimation following[47] using the weighted minimum norm method[48]. We
estimated the power spectrum in the source space using multitaper method with
an overlap of 0.5s corresponding to the degree of overlap between the tapered
time segments with Hann window.

Functional connectivity was computed from the segmented EEG signals by means
of the Welch-based spectral coherence [39]. The main parameters were: window
length 500 ms, overlap 250 ms and frequency resolution 1 Hz with a Hanning
window. The resulting networks consisted of connectivity matrices where the
$i,j$ entry contained the amount of signal synchronization between the EEG
channel $i$ and $j$ at a given frequency bin. From these networks, we further
computed the node strength as the sum of all the connectivity matrices’ rows
[49]. Node strengths allow to have quantitative measure of the amount of total
connectivity of each EEG channel. Both connectivity matrices and node
strengths, were finally averaged across segments and trials so to have
characteristic values for each subject, task (MI, rest) and strategy.

To investigate the group-level statistical difference between MI and rest
tasks, we used a paired cluster-based permutation t-test applied to the motor-
related score given by the relative difference between tasks

$\frac{MI-rest}{rest}=\frac{MI}{rest}-1$ (1)

This normalizing transformation was applied to every considered metric, i.e.
power, time-frequency power, functional connectivity and node strength. The
maximal statistical threshold level was set to $\alpha=0.05$.

## References

  * 1. J. Wolpaw, ed., Brain-Computer Interfaces: Principles and Practice.  Oxford, New York: Oxford University Press, Mar. 2012.
  * 2. S. Silvoni, A. Ramos-Murguialday, M. Cavinato, C. Volpato, G. Cisotto, A. Turolla, F. Piccione, and N. Birbaumer, “Brain-Computer Interface in Stroke: A Review of Progress,” Clin EEG Neurosci, vol. 42, pp. 245–252, Oct. 2011.  Publisher: SAGE Publications Inc.
  * 3. R. Mane, T. Chouhan, and C. Guan, “BCI for stroke rehabilitation: motor and beyond,” J. Neural Eng., vol. 17, p. 041001, Aug. 2020.  Publisher: IOP Publishing.
  * 4. F. Lotte, L. Bougrain, A. Cichocki, M. Clerc, M. Congedo, A. Rakotomamonjy, and F. Yger, “A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update,” J. Neural Eng., vol. 15, p. 031005, Apr. 2018.
  * 5. M.-C. Corsi, M. Chavez, D. Schwartz, N. George, L. Hugueville, A. E. Kahn, S. Dupont, D. S. Bassett, and F. De Vico Fallani, “Functional disconnection of associative cortical areas predicts performance during BCI training,” NeuroImage, vol. 209, p. 116500, Apr. 2020.
  * 6. J. D. Wander, T. Blakely, K. J. Miller, K. E. Weaver, L. A. Johnson, J. D. Olson, E. E. Fetz, R. P. N. Rao, and J. G. Ojemann, “Distributed cortical adaptation during learning of a brain–computer interface task,” Proceedings of the National Academy of Sciences, vol. 110, pp. 10818–10823, June 2013.  Publisher: Proceedings of the National Academy of Sciences.
  * 7. B. Serim, M. Spapé, and G. Jacucci, “Revisiting embodiment for brain–computer interfaces,” Human–Computer Interaction, vol. 0, pp. 1–27, Feb. 2023.  Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/07370024.2023.2170801.
  * 8. L. Cao, G. Li, Y. Xu, H. Zhang, X. Shu, and D. Zhang, “A brain-actuated robotic arm system using non-invasive hybrid brain–computer interface and shared control strategy,” J. Neural Eng., vol. 18, p. 046045, May 2021\.  Publisher: IOP Publishing.
  * 9. B. Xu, W. Li, D. Liu, K. Zhang, M. Miao, G. Xu, and A. Song, “Continuous Hybrid BCI Control for Robotic Arm Using Noninvasive Electroencephalogram, Computer Vision, and Eye Tracking,” Mathematics, vol. 10, p. 618, Jan. 2022.  Number: 4 Publisher: Multidisciplinary Digital Publishing Institute.
  * 10. M. Alimardani, S. Nishio, H. Ishiguro, M. Alimardani, S. Nishio, and H. Ishiguro, “Brain-Computer Interface and Motor Imagery Training: The Role of Visual Feedback and Embodiment,” in Evolving BCI Therapy - Engaging Brain State Dynamics, IntechOpen, Oct. 2018\.
  * 11. J. M. Juliano, R. P. Spicer, A. Vourvopoulos, S. Lefebvre, K. Jann, T. Ard, E. Santarnecchi, D. M. Krum, and S.-L. Liew, “Embodiment Is Related to Better Performance on a Brain–Computer Interface in Immersive Virtual Reality: A Pilot Study,” Sensors (Basel), vol. 20, p. 1204, Feb. 2020.
  * 12. R. Leeb, C. Keinrath, D. Friedman, C. Guger, R. Scherer, C. Neuper, M. Garau, A. Antley, A. Steed, M. Slater, and G. Pfurtscheller, “Walking by Thinking: The Brainwaves Are Crucial, Not the Muscles!,” Presence, vol. 15, pp. 500–514, Oct. 2006.
  * 13. D. Petit, P. Gergondet, A. Cherubini, and A. Kheddar, “An integrated framework for humanoid embodiment with a BCI,” in 2015 IEEE International Conference on Robotics and Automation (ICRA), (Seattle, WA, USA), pp. 2882–2887, IEEE, May 2015.
  * 14. N. Martens, R. Jenke, M. Abu-Alqumsan, C. Kapeller, C. Hintermuller, C. Guger, A. Peer, and M. Buss, “Towards robotic re-embodiment using a Brain-and-Body-Computer Interface,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, (Vilamoura-Algarve, Portugal), pp. 5131–5132, IEEE, Oct. 2012.
  * 15. L. Aymerich-Franch, D. Petit, G. Ganesh, and A. Kheddar, “Non-human Looking Robot Arms Induce Illusion of Embodiment,” Int J of Soc Robotics, vol. 9, pp. 479–490, Sept. 2017.
  * 16. J. L. Segil, L. M. Roldan, and E. L. Graczyk, “Measuring embodiment: A review of methods for prosthetic devices,” Frontiers in Neurorobotics, vol. 16, 2022.
  * 17. P. Haselager, “Did I Do That? Brain–Computer Interfacing and the Sense of Agency,” Minds & Machines, vol. 23, pp. 405–418, Aug. 2013\.
  * 18. J. Van Acken, Tracking the Sense of Agency in BCI Applications.  PhD thesis, Aug. 2012.
  * 19. C. Jeunet, B. N’Kaoua, and F. Lotte, “Chapter 1 - Advances in user-training for mental-imagery-based BCI control: Psychological and cognitive factors and their neural correlates,” in Progress in Brain Research (D. Coyle, ed.), vol. 228 of Brain-Computer Interfaces: Lab Experiments to Real-World Applications, pp. 3–35, Elsevier, Jan. 2016.
  * 20. P. Haggard, “Sense of agency in the human brain,” Nat Rev Neurosci, vol. 18, pp. 196–207, Apr. 2017.  Number: 4 Publisher: Nature Publishing Group.
  * 21. M. Pessiglione, F. Vinckier, S. Bouret, J. Daunizeau, and R. Le Bouc, “Why not try harder? Computational approach to motivation deficits in neuro-psychiatric diseases,” Brain, vol. 141, pp. 629–650, Mar. 2018.
  * 22. L. V. Marcuse, M. C. Fields, and J. J. Yoo, “2 - The normal adult EEG,” in Rowan’s Primer of EEG (Second Edition) (L. V. Marcuse, M. C. Fields, and J. J. Yoo, eds.), pp. 39–66, London: Elsevier, Jan. 2016.
  * 23. C. Haering and A. Kiesel, “Was it me when it happened too early? Experience of delayed effects shapes sense of agency,” Cognition, vol. 136, pp. 38–42, Mar. 2015.
  * 24. J. W. Moore and S. S. Obhi, “Intentional binding and the sense of agency: a review,” Conscious Cogn, vol. 21, pp. 546–561, Mar. 2012.
  * 25. G. Pfurtscheller, “Functional brain imaging based on ERD/ERS,” Vision Research, vol. 41, pp. 1257–1260, May 2001.
  * 26. J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan, “Brain-computer interfaces for communication and control,” Clin Neurophysiol, vol. 113, pp. 767–791, June 2002.
  * 27. A. Vavoulis, P. Figueiredo, and A. Vourvopoulos, “A Review of Online Classification Performance in Motor Imagery-Based Brain–Computer Interfaces for Stroke Neurorehabilitation,” Signals, vol. 4, pp. 73–86, Mar. 2023.  Number: 1 Publisher: Multidisciplinary Digital Publishing Institute.
  * 28. B. Xu, A. Song, G. Zhao, G. Xu, L. Pan, R. Yang, H. Li, J. Cui, and H. Zeng, “Robotic neurorehabilitation system design for stroke patients,” Advances in Mechanical Engineering, vol. 7, p. 1687814015573768, Mar. 2015.  Publisher: SAGE Publications.
  * 29. E. P. Lana, B. V. Adorno, and C. J. Tierra-Criollo, “An ERD/ERS analysis of the relation between human arm and robot manipulator movements,” in 2013 ISSNIP Biosignals and Biorobotics Conference: Biosignals and Robotics for Better and Safer Living (BRC), pp. 1–7, Feb. 2013.  ISSN: 2326-7844.
  * 30. T. Venot, M. C. Corsi, L. Saint-Bauzel, and F. d. Vico Fallani, “Towards multimodal BCIs: the impact of peripheral control on motor cortex activity and sense of agency,” in 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 5876–5879, Nov. 2021.  ISSN: 2694-0604.
  * 31. Z. Sun, Y.-c. Jiang, Y. Li, J. Song, and M. Zhang, “Short-term Priming Effects: an EEG Study of Action Observation on Motor Imagery,” IEEE Transactions on Cognitive and Developmental Systems, pp. 1–1, 2022\.  Conference Name: IEEE Transactions on Cognitive and Developmental Systems.
  * 32. M. Daeglau, C. Zich, R. Emkes, J. Welzel, S. Debener, and C. Kranczioch, “Investigating Priming Effects of Physical Practice on Motor Imagery-Induced Event-Related Desynchronization,” Front Psychol, vol. 11, p. 57, 2020.
  * 33. L. B. Stelmach, C. M. Herdman, and K. R. McNeil, “Attentional modulation of visual processes in motion perception,” Journal of Experimental Psychology: Human Perception and Performance, vol. 20, no. 1, pp. 108–121, 1994\.  Place: US Publisher: American Psychological Association.
  * 34. J. Thompson and R. Parasuraman, “Attention, biological motion, and action recognition,” NeuroImage, vol. 59, pp. 4–13, Jan. 2012.
  * 35. L. Pessoa, “Understanding brain networks and brain organization,” Phys Life Rev, vol. 11, pp. 400–435, Sept. 2014.
  * 36. N. Leeuwis, S. Yoon, and M. Alimardani, “Functional Connectivity Analysis in Motor-Imagery Brain Computer Interfaces,” Frontiers in Human Neuroscience, vol. 15, 2021.
  * 37. M. Milano, G. Agapito, and M. Cannataro, “Challenges and Limitations of Biological Network Analysis,” BioTech (Basel), vol. 11, p. 24, July 2022.
  * 38. M. Koutrouli, E. Karatzas, D. Paez-Espino, and G. A. Pavlopoulos, “A Guide to Conquer the Biological Network Era Using Graph Theory,” Frontiers in Bioengineering and Biotechnology, vol. 8, 2020.
  * 39. T. Cattai, S. Colonnese, M.-C. Corsi, D. S. Bassett, G. Scarano, and F. De Vico Fallani, “Phase/Amplitude Synchronization of Brain Signals During Motor Imagery BCI Tasks,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 29, pp. 1168–1177, 2021\.  Conference Name: IEEE Transactions on Neural Systems and Rehabilitation Engineering.
  * 40. A. Cavazzana, SENSE OF AGENCY AND INTENTIONAL BINDING: How does the brain link voluntary actions with their consequences? PhD thesis, Padua,Italy, Jan. 2016.
  * 41. A. Serino, M. Bockbrader, T. Bertoni, S. Colachis IV, M. Solcà, C. Dunlap, K. Eipel, P. Ganzer, N. Annetta, G. Sharma, P. Orepic, D. Friedenberg, P. Sederberg, N. Faivre, A. Rezai, and O. Blanke, “Sense of agency for intracortical brain–machine interfaces,” Nat Hum Behav, vol. 6, pp. 565–578, Apr. 2022.  Number: 4 Publisher: Nature Publishing Group.
  * 42. S. Mick, M. Lapeyre, P. Rouanet, C. Halgand, J. Benois-Pineau, F. Paclet, D. Cattaert, P.-Y. Oudeyer, and A. de Rugy, “Reachy, a 3D-Printed Human-Like Robotic Arm as a Testbed for Human-Robot Control Strategies,” Front Neurorobot, vol. 13, p. 65, 2019.
  * 43. Y. Renard, F. Lotte, G. Gibert, M. Congedo, E. Maby, V. Delannoy, O. Bertrand, and A. Lécuyer, “Openvibe: An open-source software platform to design, test, and use brain–computer interfaces in real and virtual environments,” Presence, vol. 19, no. 1, pp. 35–53, 2010.  Publisher: MIT press.
  * 44. D. Krusienski, D. Mcfarland, and J. Wolpaw, “An Evaluation of Autoregressive Spectral Estimation Model Order for Brain-Computer Interface Applications,” Conference proceedings : … Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Conference, vol. 1, pp. 1323–6, Feb. 2006.
  * 45. H. Ramoser, J. R. Wolpaw, and G. Pfurtscheller, “EEG-based communication: evaluation of alternative signal prediction methods,” Biomed Tech (Berl), vol. 42, pp. 226–233, Sept. 1997.
  * 46. S. Bufalari, D. Mattia, F. Babiloni, M. Mattiocco, M. G. Marciani, and F. Cincotti, “Autoregressive spectral analysis in Brain Computer Interface context,” in 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 3736–3739, Aug. 2006.  ISSN: 1557-170X.
  * 47. E. Vallarino, A. S. Hincapié, K. Jerbi, R. Leahy, A. Pascarella, A. Sorrentino, and S. Sommariva, “Tuning Minimum-Norm regularization parameters for optimal MEG connectivity estimation,” Neuroimage, p. 120356, Sept. 2023.
  * 48. R. Grech, T. Cassar, J. Muscat, K. P. Camilleri, S. G. Fabri, M. Zervakis, P. Xanthopoulos, V. Sakkalis, and B. Vanrumste, “Review on solving the inverse problem in EEG source analysis,” J Neuroeng Rehabil, vol. 5, p. 25, Nov. 2008.
  * 49. J. Gonzalez-Astudillo, T. Cattai, G. Bassignana, M.-C. Corsi, and F. D. V. Fallani, “Network-based brain–computer interfaces: principles and applications,” J. Neural Eng., vol. 18, p. 011001, Jan. 2021.  Publisher: IOP Publishing.
  * 50. M. Seeck, L. Koessler, T. Bast, F. Leijten, C. Michel, C. Baumgartner, B. He, and S. Beniczky, “The standardized EEG electrode array of the IFCN,” Clinical Neurophysiology, vol. 128, pp. 2070–2077, Oct. 2017.
  * 51. J. Song, C. Davey, C. Poulsen, P. Luu, S. Turovets, E. Anderson, K. Li, and D. Tucker, “EEG source localization: Sensor density and head surface coverage,” Journal of Neuroscience Methods, vol. 256, pp. 9–21, Dec. 2015\.
  * 52. L. Brinkman, A. Stolk, H. C. Dijkerman, F. P. de Lange, and I. Toni, “Distinct Roles for Alpha- and Beta-Band Oscillations during Mental Simulation of Goal-Directed Actions,” J Neurosci, vol. 34, pp. 14783–14792, Oct. 2014.

## Acknowledgements

We would like to thank Will Hopper and Jean Daunizeau for having provided the
code to compute the motivation score and for insightful discussion. FDVF
acknowledges support from the European Research Council (ERC), Grant Agreement
No. 864729 and from the Agence Innovation Defense DGA. All data needed to
evaluate the conclusions in the paper are present in the paper and/or the
Supplementary Materials. The data (power spectra, time-frequency, functional
connectivity, pupil diameter variations, motivation scores) can be provided by
Fabrizio, De Vico Fallani pending scientific review and a completed material
transfer agreement. Requests for these data should be submitted to fabrizio-
<EMAIL_ADDRESS>The authors declare that they have no competing
interests.

Fig 1: Structure of the experimental setup.
a) Hybrid-BCI system composed of a 64 EEG BrainAmp device, the Tobii Pro
Glasses 3 Eyetracker, the augmented table with the display monitor underneath
the glass and a Pollen Robotics’s Reachy arm facing the subject. The user
selects the target via the gaze position. The position is sent to the robot
which reaches the target. Once the robot reaches the target, its grasping
depends on the user’s motor imagery (MI) activity command.
b) The three different timings of control investigated in the protocol.
Strategy 1 (in green) - performing MI before the robot reaches the target can.
Strategy 2 (in red) - performing MI while the robot reaches the target,
strategy 3 (in blue) - performing MI after the robot has reached the target.
Among all strategies, the latter is the one temporally closest to the
subsequent grasping robot action (Fig. S1a).

Fig 2: Participants’ motivation and BCI performance.
a) Correlation between the highest performance of subjects across strategies
in terms of accuracy and sensitivity as a function of their motivation as
measured by the Reward/Effort test [21]. Spearman correlation: Sensitivity in
black $R=0.595$, $p_{val}=0.024$. Accuracy in blue $R=0.683$,$p_{val}=0.007$.
b) Group-averaged accuracy in the two control blocks for each timing strategy
. 2-way ANOVA reveals a significant learning effect (Block F=14.861523
$p<0.0002$), while there are no significant effects for the timing strategy. A
post-hoc analysis with Bonferroni correction indicates that only strategy 3
leads to a significant increase of classification accuracy at the end of the
session ($F=4.9837$, $p_{corrected}=0.003$). Control block 1 in blue, control
block 2 in orange. Asterisk denotes statistically significant difference with
$\alpha=0.05$ threshold.

Fig 3: Maps of motor-related brain activity for different MI timing
strategies.
a) Sensor space activity. Cluster-based permutation T-test on the relative
difference between the average power spectra of MI and Rest trials across
subjects (Methods). Data refer to the Control block 2 and beta frequency band
(13-25Hz). Only T-values corresponding to $p<0.05$ are shown. The distance
matrix for the cluster computation is set considering a threshold of 40 mm
between the EEG channels (based on the average distance between electrodes in
a 10-10 international system [50]). The cluster forming threshold is computed
using percent point function at $\alpha=0.05$ for $n=15$.
b) Source space activity. Cluster-based permutation T-test on the relative
difference between the average power spectra of MI and Rest trials across
subjects (Methods). Data refer to the Control block 2 and beta frequency band
(13-25Hz). Source space dipoles’ signals are estimated using a weight minimum
norm estimation (wMNE) and a average head model (Methods). The distance matrix
for the cluster computation is set considering a threshold of 20 mm between
the source dipoles (based on the average distance between dipoles in a forward
model[51]). The cluster forming threshold is computed using percent point
function at $\alpha=0.05$ for $n=15$. Only strategy 3 present significant
T-values ($p<0.05$), while the weaker activations in the other two other
strategies are shown only for illustrative purposes.

Fig 4: Time-frequency maps of motor-related activity for different MI timing
strategies Cluster-based permutation T-test on the relative difference between
the time-varying average power spectra of MI and Rest trials across subjects
(Methods). Data refer to the Control block 2 and are computed with an
autoregressive model (Burg) on a sliding window of 0.011 s and frequency
resolution 0.1 Hz (Methods). T-values corresponding to $p<0.05$ are
highlighted by dashed contours. The distance matrix for the cluster
computation at each point is given by closest pixels [52]. The cluster forming
threshold is computed using percent point function at $\alpha=0.05$ for
$n=15$. Vertical dashed lines indicate that significant motor-related activity
starts to appear after 1 s from the cue.

Fig 5: Motor-related brain connectivity networks for different MI timing
strategies.
Statistical connectivity matrices. Each entry corresponds to an EEG channel
pair. Left=left hemisphere. Right=right hemisphere. The color codes for the
permutation T-test value on the relative difference between the average
spectral beta (13-25 Hz) coherence in the MI and Rest trials across subjects
(Methods). Cluster based permutation T-test values on the relative difference
of the node strength between average MI and Rest trials across subjects
(Methods). Results refer to the beta band (13-25Hz) in last session block
(Control 2). Nodes correspond to EEG channels. The larger and darker is the
node the higher is the associated T-value. Only T-values corresponding to
$p<0.05$ are shown. The most significant nodes in the sensorimotor related
zone are CP3,C1, and C3. The distance matrix for the cluster computation is
set considering a threshold of 40 mm between the EEG nodes [50]. The cluster
forming threshold is computed using percent point function at $\alpha=0.05$
for $n=15$. Links are shown for illustrative purposes and correspond to the
strongest T-values from the related connectivity matrices in a), ie $p<0.001$
for strategy 1 and 2, $p<10e-06$ for strategy 3.