# Learning Generalizable Feature Fields for
Mobile Manipulation

Ri-Zhao Qiu∗1, Yafei Hu∗1,2, Ge Yang3,4, Yuchen Song1, Yang Fu1, Jianglong
Ye1, Jiteng Mu1, Ruihan Yang1,
Nikolay Atanasov1, Sebastian Scherer2, Xiaolong Wang1
∗equal contribution
1UC San Diego, 2CMU, 3MIT, 4IAIFI
https://geff-b1.github.io

###### Abstract

An open problem in mobile manipulation is how to represent objects and scenes
in a unified manner so that robots can use it both for navigating in the
environment and manipulating objects. The latter requires capturing intricate
geometry while understanding fine-grained semantics, whereas the former
involves capturing the complexity inherited to an expansive physical scale. In
this work, we present GeFF (Generalizable Feature Fields), a scene-level
generalizable neural feature field that acts as a unified representation for
both navigation and manipulation that performs in real-time. To do so, we
treat generative novel view synthesis as a pre-training task, and then align
the resulting rich scene priors with natural language via CLIP feature
distillation. We demonstrate the effectiveness of this approach by deploying
GeFF on a quadrupedal robot equipped with a manipulator. We evaluate GeFF’s
ability to generalize to open-set objects as well as running time when
performing open-vocabulary mobile manipulation in dynamic scenes.

Figure 1: GeFF, Generalizable Feature Fields, provide unified implicit scene
representations for both robot navigation and manipulation in real-time. We
demonstrate the efficacy of GeFF on open-world mobile manipulation and
semantic-aware navigation under diverse real-world scenes ((a) working in a
lab where a person walks in, (b) entering a meeting room with narrow entrance,
(c) operating in a community kitchen with various furniture, (d) grasping
objects from a car trunk in a parking lot, and (e) semantic-aware navigation
near a lawn outside of a building). The visualization of the feature fields is
obtained by PCA of neural-rendered features. For best illustration, we animate
feature fields built in real-time on the website.

##  I Introduction

Building a personal robot that can assist with common chores has been a long-
standing goal of robotics [1, 2, 3]. This paper studies the task of open-
vocabulary navigation and picking, where a robot needs to navigate through
diverse scenes to pick up objects based on language instructions. This task,
while seemingly easy for humans, remains quite challenging for autonomous
robots. We humans achieve such tasks by understanding the layout of rooms and
the affordances of objects without explicitly memorizing every aspect.
However, when it comes to autonomous robots, there does not exist a unified
scene representation that captures geometry and semantics for both navigation
and manipulation tasks.

Recent approaches in navigation seek representations such as geometric maps
(with semantic labels) [4, 5] and topological maps [6, 7] to handle large-
scale scenes, but are not well integrated with manipulation requirements.
Manipulation, on the other hand, often relies on continuous scene
representation such as implicit surfaces or meshes [8] to compute precise
grasping poses, which are not typically encoded in navigation representations.
More importantly, interpreting semantic task instructions requires grounding
of concepts with respect to geometric and semantic features of the
environment. Such discrepancy in representations leads to unsatisfactory
performance [9] in complex tasks that involve multiple visuomotor skills.
Performing coherent open-vocabulary perception for both navigation and
manipulation remains a significant challenge.

To this end, we present a novel scene-level Generalizable Feature Field (GeFF)
as a unified representation for navigation and manipulation. This
representation is trained with neural rendering using Neural Radiance Fields
(NeRFs) [10]. Instead of fitting a single NeRF with a static scene, our
representation can be updated in real-time as the robot moves and the
surroundings change. Inspired by recent works in Generalizable NeRFs (Gen-
NeRFs) [11, 12], we train our representation with an encoder, which allows one
feed-forward pass in the network to update the scene representation during
inference. Besides being a unified representation, GeFF stands out with two
more advantages: (i) GeFF is able to decode multiple 3D scene representations
from a posed RGB-D stream, including SDF, mesh, and pointcloud, and (ii) by
performing feature distillation from a pre-trained Vision-Language Model
(VLM), e.g., CLIP [13], the representation not only contains geometric
information but also language-conditioned semantics. These three key factors
mitigate the discrepancy as discussed in the previous paragraph.

We demonstrate GeFF using a quadrupedal mobile manipulator to execute object
discovery and manipulation tasks specified using language instructions. Our
mobile manipulation system works as follows. First, the robot scans part of
the scene which includes the target objects using an RGB-D camera and
constructs a 3D representation using GeFF. At the same time, GeFF enables
constructing a 3D feature field via feature distillation. The robot can then
identify the goal object by searching in the feature field given language
instructions. With the 3D map and an object goal, the robot can perform
semantic-aware planning for navigation to reach and grasp target objects. As
the robot moves in the scene, the RGB-D streams are fed into GeFF to extract
3D semantic features, and the pre-computed feature field is updated in real-
time. This brings two benefits: (i) when the object arrangement (goal object
or surroundings) changes in the scene, we can update the map in real-time and
perform re-planning; (ii) as the robot moves closer to the object, GeFF can
provide a more detailed description of the object given higher resolution
inputs, which is essential for grasping.

We experiment with a Unitree B1 quadrupedal robot where a Z1 robot arm is
attached on top, as shown in Fig. 1. We perform mobile manipulation with this
robot in diverse environments where the robot needs to navigate to different
receptacles, avoid dynamic obstacles, plan semantically to stay on walkways
away from the lawn, and search and pick up objects in an open-vocabulary
manner. We show that using the GeFF representation significantly improves over
baselines using standard NeRF with feature distillation (e.g., LeRF [14]):
GeFF achieves an average $52.9\%$ success rate while LeRF achieves an average
$30.7\%$ success rate in mobile manipulation. We further perform ablations in
simulation to validate the effectiveness of our approach. We plan to release
the pre-trained models and the source code.

##  II Related Work

Generalizable Neural Radiance Fields. Generalizable Neural Radiance Fields
extend conventional Neural Radiance Fields’ ability to render highly-detailed
novel views to scenes that come with just one or two images [11, 15, 16, 17,
18, 19, 20, 12]. They replace the time-consuming optimization of weights for
each scene with a single feed-forward process through a network. Existing
works [19, 21, 22] mainly focus on synthesizing novel views. Our focus is to
use novel view synthesis via generalizable neural fields as a generative pre-
training task. At test time, we use the produced network for real-time
semantic and geometric inference in a robotic mobile manipulation setting.

Feature Distillation in NeRF Beyond just synthesizing novel views, recent
works [14, 23, 24, 12] have also attempted to combine NeRF with 2D features
via feature distillation from 2D vision foundation models [13, 25, 26, 27] to
3D space to empower neural fields with semantic understanding of objects [23,
24, 12], scenes [14, 28] and downstream robotic applications [29, 28].
Nonetheless, these works cannot be easily adapted for mobile manipulation due
to the expensive per-scene optimization scheme [14, 23] or restrictions to
object-level representations [12]. Most closely related to our work, LERF [14]
and F3RM [28] distill CLIP features to create scene representations that can
be queried with natural language. F3RM adapts the feature fields for tabletop
manipulation. Nonetheless, both LERF and F3RM require expensive per-scene
optimization, which takes up to 40 minutes [14] to create a scene-level
representation. Such an inefficiency hinders practical downstream applications
on mobile robots. In stark contrast, our GeFF runs in real-time on mobile
robots.

Figure 2: Pre-trained as a generalizable NeRF encoder, GeFF provides unified
scene representations for robot tasks from onboard RGB-D stream, offering both
real-time geometric information for planning and language-grounded semantic
query capability. Compared to LERF [14], GeFF runs in real-time without costly
per-scene optimization, which enables many potential robotics applications. We
demonstrate the efficacy of GeFF on open-world language-conditioned mobile
manipulation. Feature visualizations are done by running PCA on high-
dimensional feature vectors and normalizing the 3 main components as RGB.

Object Navigation and Mobile Manipulation. Object navigation involves
controlling a robot to navigate in the environment and to find target objects.
Existing object navigation methods tackle this problem via modular approaches,
using vision and language foundation models, scene graphs, etc. [30, 31, 32,
33, 34, 35, 36, 37], or via learning-based methods [38, 39, 40, 41, 42].
However, most end-to-end learning-based methods were only tested in
constrained simulation environments. In addition, recently Gervet et al. [31]
found that these end-to-end methods show poor generalization in real-world
environments. Inspired by this finding, we follow the modular approach and
combine GeFF with a classical motion planner and controller for real-world
mobile manipulator deployment.

Beyond navigating to object goals, mobile manipulation requires a synergistic
combination of navigation and manipulation [43, 44, 45, 9, 46, 47]. Previous
works include learning-based methods [44, 48, 49, 45, 50, 51], and classical
approaches based on motion planning [52, 53] or visual servoing [54].
Nonetheless, these works are constrained to a closed-set setting, meaning that
they only work with a pre-defined range of objects that have seen during
training. In contrast, our work operates on an open set of objects in both
manipulation (specifying objects to manipulate) and navigation (instructing
robots to avoid objects). Most recent works like HomeRobot [9] show open-
vocabulary capabilities but have demonstrated only relative low success rate
in small-scale real-world environments. In comparison with our approach,
existing techniques lack a rich and unified 3D semantic and geometric
representation of the environment to support integrated robot navigation and
manipulation.

Open-Vocabulary Scene Representations. There have been some recent works [55,
56, 33, 57, 58] that leverage 2D foundation vision models to build open-
vocabulary 3D representations. These methods project predictions from large-
scale models such as CLIP [13] or SAM [59] directly onto explicit
representations (point-based or voxel-based). As the number of features stored
at each location increases, these explicit representation-based methods become
harder to scale and are mostly limited to room-scale environment. GeFF, on the
other hand, builds a latent and unified representation that conceptually scale
to larger environments. A concurrent work, OK-Robot [56], uses voxel-based
representations to perform open-vocabulary mobile manipulation, which is most
related to our work among existing methods. In turn, GeFF demonstrates ability
to operate in both room-scale environment and larger-scale outdoor environment
with the its perceptive capability and traversability of quadruped robots.

##  III Problem Formulation and Background

###  III-A Problem Statement

Let $\Omega$ be the space of RGB-D images. Consider $N$ posed RGB-D frames
$\mathcal{D}=\\{(F_{i},\mathbf{T}_{i})\\}_{i=1}^{N}$ obtained from a mobile
robot equipped with an RGB-D camera, where $F_{i}\in\Omega$ is the $i$-th
RGB-D frame and $\mathbf{T}_{i}\in\mathbf{SE}(3)$ is the camera extrinsics.
Our goal is to create a unified scene representation that captures geometric
and semantic properties for robot navigation and manipulation from
$\mathcal{D}$. More specifically, we aim to design an encoding function
$f_{enc}(\cdot):(\Omega\times\mathbf{SE}(3))^{N}\mapsto\mathbb{R}^{N\times C}$
that compresses $\mathcal{D}$ to a latent representation, and decoding
functions $g_{geo}(\cdot,\cdot):\mathbb{R}^{3}\times\mathbb{R}^{N\times
C}\mapsto\mathbb{R}^{m}$ and
$g_{sem}(\cdot,\cdot):\mathbb{R}^{3}\times\mathbb{R}^{N\times
C}\mapsto\mathbb{R}^{n}$ that decode the latents into different geometric and
semantic features at different positions in 3D space. Here, $C$, $m$, and $n$
are the dimensions of the latent representation, geometric feature, and
semantic feature, respectively. These geometric and semantic features can then
serve as the input to the downstream planner. We aim to design these three
functions to meet the following criteria.

  * •

Unified. The encoded scene representation $f_{enc}(\mathcal{D})$ is sufficient
for both geometric and semantic query (i.e., $g_{geo}$ and $g_{sem}$ are
conditioned on $\mathcal{D}$ only via $f_{enc}(\mathcal{D})$).

  * •

Incremental. The scene representation supports efficient incremental addition
of new observations, (i.e.,
$f_{enc}(\mathcal{D}_{1}\cup\mathcal{D}_{2})=f_{enc}(\mathcal{D}_{1})\oplus
f_{enc}(\mathcal{D}_{2})$)

  * •

Continuous. To support hierarchical, coarse-to-fine-grained robotic motion
planning and control, the query function should be continuous and queryable at
any coordinate (e.g., $g_{geo}(\mathbf{x},f_{enc}(\mathcal{D}))$, where
$\mathbf{x}=(x,y,z)$ is a location in 3-D space).

  * •

Implicit. The encoded latents $f_{enc}(D)$ should be organized in a sparse
implicit representation to enable more efficient scaling to large scenes than
storing $\mathcal{D}$.

  * •

Open-world. The semantic knowledge from $g_{sem}$ should be open-set and
aligned with language, so the robot can perform open-world perception. That
is, the feature vector output from $g_{sem}$ lies in a multi-modality that
aligns text and vision (e.g., CLIP [13]).

In this paper, we build GeFF upon generalizable NeRFs to satisfy all of these
requirements.

###  III-B Background: NeRF and Generalizable NeRF

Given a coordinate $\mathbf{x}\in\mathbb{R}^{3}$ and a unit-vector viewing
direction $\mathbf{d}\in\mathbf{S}^{2}$, the original NeRF [10] adopts two
parameterized networks, a density mapping
$\sigma_{\theta}(\mathbf{x}):\mathbb{R}^{3}\to\mathbb{R}[0,1]$, which predicts
the probability that $\mathbf{x}$ is occupied, and a color mapping
$\mathbf{c}_{\omega}(\mathbf{x},\mathbf{d}):\mathbb{R}^{3}\times\mathbf{S}^{2}\to\mathbb{R}^{3}$
which maps coordinate and viewing direction to color.

Consider a ray $\mathbf{r}$ from a camera viewport with camera origin
$\mathbf{o}$ and direction $\mathbf{d}$, which is conditioned by the camera
extrinsics $\mathbf{T}$ and intrinsics $\mathbf{K}$. NeRF estimates color
along $\mathbf{r}$ by

$\mathbf{\hat{C}}(\mathbf{r})=\int_{t_{n}}^{t_{f}}T(t)\alpha_{\theta}(\mathbf{r}(t))\mathbf{c}_{\omega}(\mathbf{r}(t),\mathbf{d})\mathrm{d}t\,,$
(1)

where $t_{n}$ and $t_{f}$ are minimum and maximum distances bounding depth,
$T(t)=\exp(-\int_{t_{n}}^{t}\sigma_{\theta}(s)\mathrm{d}s)$ is the
transmittance which denotes the observed occupancy so far to avoid rendering
voxels behind objects, and $\alpha_{\theta}(r(t))$ being the opacity value at
$r(t)$ (original NeRF [10] uses $\alpha_{\theta}=\sigma_{\theta}$).

NeRF then optimizes $\theta$ and $\omega$ w.r.t. color by

$\mathcal{L}_{col}(\theta,\omega)=\sum_{\mathbf{r}\in\mathcal{R}}\left|\left|\mathbf{C}(\mathbf{r})-\mathbf{\hat{C}}(\mathbf{r})\right|\right|_{2}^{2}\,,$
(2)

Figure 3: Generalizable Feature Fields Acquire Geometric and Semantic Priors.
RGB images are input views from ScanNet [60]. Color images are PCA
visualizations of feature volume projected to the input camera view encoded by
an RGB-D Gen-NeRF [61] encoder. Note how semantically similar structures
acquire similar features.

where $\mathbf{C}(\mathbf{r})$ denotes the ground truth color of the ray
$\mathbf{r}$, and $\mathcal{R}$ is a set of randomly sampled ray for training.
Note that the training process starts from scratch for every scene and may
take hours. To avoid costly per-scene optimization, generalizable NeRFs [11,
12] propose to condition the novel view on input frames instead of optimizing
the underlying parameters. During its training time, generalizable NeRFs learn
to incorporate scene priors into its encoder. More concretely, the occupancy
and the radiance networks are given by

$\displaystyle\sigma_{\theta}(\mathbf{x},\mathcal{D})$
$\displaystyle=g_{\sigma}(\mathbf{x},f_{\textit{enc}}(\mathcal{D}))$ (3)
$\displaystyle\mathbf{c}_{\omega}(\mathbf{x},\mathbf{d},\mathcal{D})$
$\displaystyle=g_{\mathbf{c}}(\mathbf{x},\mathbf{d},f_{\textit{enc}}(\mathcal{D}))\,,$
(4)

where $g_{\sigma}$ and $g_{\mathbf{c}}$ are MLPs that predict density and
color, $f_{\textit{enc}}$ being a neural encoder. Note that parameters are
learned during pre-training. During the testing stage, novel views are
efficiently rendered in a single-pass manner.

##  IV GeFF for Mobile Manipulation

We describe our approach, Generalizable Feature Fields (GeFF), and ways we
apply it to mobile manipulation tasks in the following subsections. An
overview of our method is shown in Fig. 2.

###  IV-A Learning Scene Priors via Neural Synthesis

Generalizable neural radiance fields acquire rich geometric and semantic
priors by learning to synthesize novel views in many scenes [61, 62, 12]. To
illustrate this point, we offer a motivating example in Fig. 3, where we
render the intermediate feature volume from an RGB-D Gen-NeRF encoder [61]
trained to synthesize novel views on the ScanNet [60] dataset. The colors
correspond to the principal components of the latent features. We observe
separations between objects and the background, despite no explicit semantic
supervision were provided during training (only the RGB-D views and the camera
poses were available). This example highlights the potential of using neural
synthesis as a generative pre-training task for learning scene priors.

We propose two types of training signals using both the 2D foundation models
and depth to provide supervision.

Supervision (i): Language-Alignment via Feature Distillation. Although we have
shown that Gen-NeRF encoders implicitly encode geometric and semantic cues,
the representation is less useful if it is not aligned to other feature
modalities, such as language. In order to further enhance the representation
capability of GeFF, we propose to use knowledge distillation to transfer
learned priors from 2D vision foundation models and align the 3D
representations with them. To the best of our knowledge, GeFF is the first
approach that combines scene-level generalizable NeRF with feature
distillation. In stark contrast to previous attempts [23, 14, 12], GeFF both
works in relatively large-scale environments and runs in real-time, making it
a powerful perception method for mobile manipulation.

Specifically, we build a feature decoder $g_{sem}(\mathbf{x},f_{enc}(D))$ on
top of the latent representation, which maps a 3D coordinate to a feature
vector. The output of $g_{sem}$ is trained to be aligned with the embedding
space of a teacher 2D vision foundation model, termed $f_{teacher}$. Note that
$g_{sem}$ is non-directional. Intuitively, the semantic properties of an
object should be view-independent (e.g., a cup is a cup regardless of the
viewing directions). Similar to color rendering in Eq. 1, we can render 2D
features for pre-training via

$\mathbf{\hat{F}}(r)=\int_{t_{n}}^{t_{f}}T(t)\alpha(r(t))g_{sem}(\mathbf{r}(t),f_{enc}(\mathcal{D}))\mathrm{d}t\,.$
(5)

To further enhance the fidelity of the 3D scene representation, we use the 2D
features of the input views computed by the teacher model as an auxiliary
input to $f_{enc}$, which is

$f_{enc}(D)=\textsc{ConCat}\left(\hat{f}_{enc}(\mathcal{D}),f_{teacher}(\mathcal{D})\right)\,,$
(6)

where $\hat{f}_{enc}$ is a trainable encoder and $f_{teacher}$ is a pre-
trained vision model with frozen weights. We ablate the effect of the
auxiliary input in the experiments. The final feature rendering loss is then
given by

$\mathcal{L}_{feat}=\sum_{\mathbf{r}\in\mathcal{R}}\left|\left|\mathbf{F}(\mathbf{r})-\mathbf{\hat{F}}(\mathbf{r})\right|\right|_{2}^{2}\,$
(7)

where $\mathbf{F}$ is the reference ground-truth features obtained by running
foundation models on ground-truth novel views. Compared to previous works that
use view-dependent features [14, 28], one strength of our training scheme is
that the encoded features are view-independent, which makes it a favorable
representation for downstream motion planners that often require 3D
information.

_Model for Distillation._ Our proposed feature distillation method for scene-
level generalizable NeRFs is generic and can be applied to many 2D vision
foundation models such as Stable Diffusion [27], SAM [59], and DINO [26]. In
this work, however, since we are interested in tasking robots to perform open-
vocabulary mobile manipulation, we need to choose a vision foundation model
that is aligned with language. Therefore, we choose MaskCLIP [63] as
$f_{teacher}$, which is a variant of CLIP [13] that exploits a
reparametrization trick [63, 28] to transform the output of CLIP from a single
feature vector to a feature map aligned with the CLIP text encoders. Though
the 2D feature maps from MaskCLIP are coarse (illustrated in Fig. 2), it is
efficient enough to run at real time on mobile robots and we show
qualitatively that GeFF learns to reconstruct fine details from multiple
views.

_Handling Language Query._ Following standard protocols [14, 28], GeFF takes
in positive text queries along with a few negative text queries (e.g., wall
and ceiling). These text queries are encoded using CLIP’s text encoders, which
send texts to an embedding space that is aligned with the distill features. To
rate the similarity of a coordinate with a positive text query, we use cosine
similarity to compute the rendered feature with each text query. A
temperatured softmax is then applied to the cosine similarity to form a
probability distribution. Finally, we sum up the probabilities of positive
queries to formulate the similarity score.

Supervision (ii): Depth Supervision via Neural SDF. Inspired by [18, 64, 61],
we introduce a signed distance network
$s(\mathbf{x})=g_{geo}(\mathbf{x},f_{enc}(\mathcal{D}))$ to encode depth
information. Doing so has two advantages over previous work [11]: 1) it
leverages depth information to efficiently resolve scale ambiguity for
building scene-level representation, rather than restricted to object-level
representation, and 2) it creates a continuous implicit SDF surface
representation, which is a widely used representation for robotics
applications such as computing collision cost in motion planning [64].

To provide supervision for $g_{geo}$ during pre-training, we follow iSDF [64]
and introduce an SDF loss $\mathcal{L}_{\text{sdf}}$ and an Eikonal
regularization loss [65] $\mathcal{L}_{\text{eik}}$ to ensure smooth SDF
values. The main difference with iSDF [64] is that we condition $g_{geo}$ with
$f_{enc}(\mathcal{D})$, which does not require optimization for novel scenes.
Instead of using a density network, we represent the opacity function $\alpha$
in Eq. 1 using $s(\mathbf{x})$

$\alpha(r(t))=\textsc{Max}\left(\frac{\sigma_{s}(s(\mathbf{x}))-\sigma_{s}(s(\mathbf{x}+\Delta))}{\sigma_{s}(s(\mathbf{x}))},0\right)\,,$
(8)

where $\sigma_{s}$ is a sigmoid function modulated by a learnable parameter
$s$. The depth along a ray $\mathbf{r}$ is then rendered by

$\mathbf{\hat{D}}(r)=\int_{t_{n}}^{t_{f}}T(t)\alpha(r(t))d_{i}\mathrm{d}t\,,$
(9)

where $d_{i}$ is the distance from current ray marching position to the camera
origin. Similar to Eq. 2, the rendered depth can be supervised via

$\mathcal{L}_{depth}=\sum_{\mathbf{r}\in\mathrm{R}}\left|\left|\mathbf{D}(\mathbf{r})-\mathbf{\hat{D}}(\mathbf{r})\right|\right|_{2}^{2}\,.$
(10)

Implementation Details. For a single posed RGB-D frame, $f_{enc}$ follows
existing works in 3D encoder [66, 67] and encodes a single view to a 3D volume
of shape $\mathbb{R}^{M\times C}$. Here, $M=512$ is a set of sparse points
obtained via the farthest point sampling and $C$ is the feature dimension of
each point. The obtained points are also transformed to the world frame using
camera intrinsics and extrinsics to build a consistent world representation
from multi-frame observations.

Features of these $M$ points are obtained by using PointCov [68] on the
downsampled 3D points and interpolating a dense feature map from a ResNet-50
[69] encoder. For a specific point query in generalizable NeRF decoding,
$f_{enc}$ interpolates features from nearby $K$ points. The decoders $g_{sem}$
and $g_{geo}$ are implemented as multi-layer MLPs. We will release the code
for details and reproducibility.

Final Training Objective. Combining all the above equations, the total loss we
used to train $f_{enc}$ for a unified latent scene representation is given by

$\begin{split}\mathcal{L}=\lambda_{1}\mathcal{L}_{col}+\lambda_{2}\mathcal{L}_{depth}&+\lambda_{3}\mathcal{L}_{sdf}\\\
&+\lambda_{4}\mathcal{L}_{eik}+\lambda_{5}\mathcal{L}_{feat}\ \end{split}$
(11)

where $\lambda_{i}$ are hyperparameters used to balance loss scales.

###  IV-B Open-Vocabulary Mobile Manipulation

Scene Mapping with GeFF. As detailed in previous sections, GeFF encodes a
single posed RGB-D frame to a latent 3D volume, which is represented as a
sparse latent point cloud. Since per-frame 3D latent volume is back-projected
to the world frame, we incrementally build the latent volume by concatenating
per-frame observations. The camera pose used to construct GeFF is provided by
an off-the-shelf Visual-Inertial Odometry (VIO) method [70]. The latent 3D
volume can then be decoded into geometric and semantic representations.

Decoded Representations. Though GeFF supports decoding to various
representations, it is inefficient and impractical to generate all possible
representations on-the-fly. For this work, we decode the latent representation
into a point cloud and an occupancy map as geometric representations for
navigation and manipulation. We then enhance basic units in these
representations (i.e., points and grid cells) with feature vectors from
$g_{sem}$, which can be compared with language queries encoded by the CLIP
[13] text encoder. The comparison results are per-point similarity scores with
the higher-score responses being the points more similar to the description in
the language instruction. For a visualization of the 3D map with score
responses, please refer to Fig. 7.

GeFF for Navigation. We consider the navigation of the base quadrupedal robot
as a 2D navigation problem due to the compute constraints of the robot. The 2D
occupancy grid map which provides the traversability for navigation is
downward projected by decoded 3D point cloud. The feature vector for each grid
cell is created by averaging the feature vectors of related points. The
location with the most points, whose feature vectors are top-$k$ similar to
the input language query, is chosen as the goal location. To support semantic-
aware planning, we take in text queries of objects to avoid (e.g., lawn) and
assign semantic affordances (i.e., cost to traverse over a grid cell) to every
grid cell using its similarity with the avoiding objects. The robot then uses
a cost-aware A∗ planner to plan a set of waypoints to the goal location. We
use a PD controller to track these waypoints.

Figure 4: The mobile manipulation platform. A 7-DOF Z1 robotic arm is mounted
on top of the B1 quadrupedal robot. A forward facing Kinect sensor and
RealSense camera are mounted at the front, and a Nvidia Orin onboard computer
is mounted at the rear.

Note that since GeFF runs in real-time, the goal location and obstacles are
dynamically updated so that the robot can react to scene changes on the fly,
which leads to the robot’s ability to avoid previously unseen obstacles and to
find new objects upon arriving at the specified receptacle. We evaluate GeFF’s
ability to handle scene change in V-D.

Figure 5: Overview of our mobile manipulation autonomy system for navigation
and manipulation.

GeFF for Manipulation. After the robot arrives at the coarse goal location on
the map, it aggregates the semantic point cloud decoded from GeFF with the
same clustering algorithm to refine the centroid of the object. We then adjust
the final pose of the robot base so that the centroid is within the
configuration space of the end gripper of the robot manipulator. The robot
then attempts to grasp the object via an open-push-close gripper action
sequence with trajectories computed by a sample-based planner (OMPL planner
[71]) to solve the Inverse Kinematic (IK) for the manipulator.

|  | Lab Env. | Meeting Room Env. | Kitchen Env. | Overall
---|---|---|---|---|---
Method | Latency | Nav. Succ. | Mani. Succ. | Nav. Succ. | Mani. Succ. | Nav. Succ. | Mani. Succ. | Nav. Succ. | Mani. Succ.
GeFF (Ours) | 0.4s | 94.4% | 61.1% | 86.7% | 53.3% | 66.7% | 44.4% | 82.6% | 52.9%
GeFF no auxiliary | 0.2s | 55.6% | 27.5% | 60.0% | 33.3% | 38.9% | 22.2% | 51.5% | 27.6%
LERF [14] | 2 hrs | 72.2% | 44.4% | 40.0% | 20.0% | 44.4% | 27.8% | 52.2% | 30.7%

TABLE I: Mobile manipulation success rate categorized by navigation success
rate (Nav. Succ.) and manipulation success rate (Mani. Succ.) under different
environments with different methods. Our method consistently outperforms
baseline methods.

##  V Experiments

###  V-A Experimental Setup

Real-world Evaluation. We deploy GeFF on a real robot to evaluate the efficacy
of GeFF in performing various tasks such as open-vocabulary mobile
manipulation and semantic-aware planning. In addition to working with the real
robot, we also ablate design choices (e.g., whether auxiliary inputs are used
and teacher models) in simulation with Gazebo [72] and Habitat [73]. For
quantitative experiments, we task the robot to perform mobile manipulation
using language conditioning in 3 environments: a 25 $m^{2}$ lab with
artificial obstacles, a 30 $m^{2}$ meeting room with chairs and a big
rectangular table, and a 60 $m^{2}$ community kitchen with various furniture.
We perform $3$ trials on a total of 17 objects (6 miscellaneous objects for
the lab, 5 office-related items for the meeting room, and 6 kitchen-related
items for the kitchen) including 8 out-of-distribution categories that GeFF
had not seen during pre-training on ScanNet [60]. For qualitative experiments,
we test the robot’s ability to navigate with language-conditioned semantic
affordance, map the environment when a person walks into the scene, builds
intricate geometry from multiple views, and entering narrow doorway.

Experimental Protocol. For each environment, we first manually drive the robot
to explore the environment and build an initial representation of the
environment. This process allows the robot to perceive the locations of fixed
structures (e.g., receptacles) and can be replaced by standard robotic
exploration algorithms. During this process, the robot is agnostic to the
final goal category. Instead, GeFF allows the robot to retrieve specific
objects using language online. To make the task setting realistic and to
demonstrate the capability of GeFF performing real-time mapping, we study a
more challenging task setting where there may be scene changes between the
initial mapping stage and the object retrieval stage.

Robotic Hardware Platforms. The hardware setup is shown in Fig. 4. We use
Unitree B1 as the base robot and mount an Unitree Z1 arm on top of it. The
robot is equipped with a Microsoft Kinect RGBD sensor to provide depth for
GeFF, and an Intel RealSense D455 stereo camera to run VIO to localize the
robot for GeFF. The onboard computer is an NVIDIA Jetson AGX Orin.

Figure 6: Communication system setup based on ROS

Metrics. Following protocols in existing work in mobile manipulation [49, 9,
44], we use success rate as the main evaluation metric. We define success for
navigation as stopping the navigation process when the robot is within 1$m$ of
its front-facing goal object. We define success for mobile manipulation as
navigating and grasping the specified object, lifting it off the surface, and
holding it.

###  V-B Autonomy System Overview

We use a modular approach and divide the autonomy system into the perception,
motion planning, and control modules, which is illustrated in Fig. 5.

Method | Scene Change | Lab | Meeting Rm. | Kitchen
---|---|---|---|---
GeFF (Ours) | ✗ | 7/9 | 7/9 | 8/9
| ✓ | 4/9 | 6/9 | 8/9
LERF [14] | ✗ | 6/9 | 7/9 | 4/9
| ✓ | NA∗ | NA∗ | NA∗

TABLE II: Mobile manipulation success rate under scene change, where objects
are added to scenes after the robot builds an initial map. The results are
reported on three objects (hand lotion, dog toy, and toy duck) over three
trials per object. Note that LERF [14] requires costly per-scene optimization
and thus can not handle scene change. Our method consistently outperforms the
baselines.

###  V-C Communication System Setup

Based on ROS, we set up our communication system. We use the Nvidia Orin as
the main computer, an (optional) Ubuntu 20.04 laptop as the base station to
visualize the online results and performance, and the onboard Nvidia XavierNX
of the robot as the side computer. We use the main computer to perform all the
computations for the autonomy system, the base station to visualize the online
results, and the side computer to receive velocity commands to execute actions
for the robot base. The conversion from velocity commands to motor commands is
done by Unitree SDK. The Z1 manipulator receives commands directly from the
main computer. An overview of our communication system is shown in Fig. 6.

Figure 7: Qualitative results of GeFF for diverse tasks. (a): using GeFF, the
quadrupedal robot can build feature fields in real time, which allows
detection of passer-by in real time. (b): building high-quality feature fields
from multi-frame inputs. (c): entering a narrow doorway. (d): semantic-aware
planning where the robot avoids lawn and stays on walkway. For best
illustration, please refer to the website. (From top to bottom: side views of
the robot; first-person views of the robot; PCA visualization of the first-
person GeFF features; similarity response of feature point clouds to positive
text queries).

###  V-D Quantitative Results on Real Robot

We evaluate GeFF’s capability to help the robot perform open-set mobile
manipulation. Since previous mobile manipulation methods either work on only a
pre-defined set of objects [49] or have a low success rate and require
specialized hardware [9], we use LERF [14], another feature field method, as
the main baseline method. Since LERF is an RGB-only method with view-dependent
features, we use metric poses estimated by the same VIO algorithm to alleviate
scale ambiguity and select the point with maximum responses in features
rendered from training views as the goal location. We evaluate the success
rate of navigation and mobile manipulation.

Open-Set Mobile Manipulation. We test the mapping latency, navigation and
manipulation success rates with a total of 17 objects in 3 different
environments. The results are given in Tab. I. We compare a baseline method
without using auxiliary input shown in Eq. 6, as well as LERF [14]. Most
methods show the highest success rates on both tasks in the Lab, which is a
controlled environment with consistent lighting. On the other hand, Kitchen, a
realistic scene with challenging lighting conditions and complex scene
layouts, poses challenges to all methods. Our method, GeFF augmented with
auxiliary foundation model input, consistently achieves the best performance
across all task settings.

Open-Set Mobile Manipulation with Scene Change. One notable characteristic of
generalizable neural fields is that they do not require costly per-scene
optimization. This is a desirable property for robotics applications, as we
want robots to be able to respond to scene changes on the fly. In this part,
we evaluate the performance of GeFF for open-set mobile manipulation where
objects are added to the scenes after initial mapping. Specifically, we put
objects on tables after the initial mapping stage. With new objects and
language descriptions, the robot should be able to find the new objects,
navigate to them, and then grasp them. We use a subset of objects from the
main experiments and test the mobile manipulation success rates under three
environments. Tab. II shows the results of open-set mobile manipulation with
scene changes. LERF [14], being a conventional feature field method, requires
costly per-scene optimization and is not applicable to online response to
scene changes. On the other hand, our method, GeFF, successfully copes with
scene changes.

###  V-E Qualitative Results on Real Robot

In this section we provide qualitative results of GeFF to demonstrate
different potential applications. In Fig. 7, we show qualitative results of
dynamic obstacle avoidance ability in the lab environment, localizing objects
with good geometric reconstruction by fusing multiple views, ability to go
through a narrow doorway, and semantic-aware planning to avoid terrains that
the robot should semantically avoid.

Dynamic Obstacle Avoidance. We first construct the map and while the robot
moves toward the goal object, one person comes in front of the robot. From the
first sub-figure in the 3rd row of Fig. 7 (a), we notice that GeFF recognizes
people and can assign higher affordances in real time, which is challenging
for per-scene optimization feature fields such as LERF [14].

Intricate Geometry. GeFF is capable of fusing features from multi-views to
create a fine-grained semantic representation than a single-view projection.
Illustrated in Fig. 7 (b) and Fig. 8, we demonstrate clear semantic and
geometric boundaries from reconstructed objects.

Narrow Passage. Since GeFF can produce a fine-grained scene representation, it
allows the robot to pass through a narrow doorway without collisions in the
meeting room. This result is illustrated in Fig 7 (c).

Semantic-aware Planning. It is natural to integrate semantic affordances with
the feature-enhanced point cloud and occupancy map provided by GeFF. In Fig. 7
(d), we test the capability of GeFF in an outdoor scene. With the
traversability of the quadrupedal robot, the robot can directly step over lawn
towards the goal object. However, it may be undesirable for robots to step on
lawns. Therefore, we assign higher semantic affordances for grass surface,
which encourages the robot to stay on the walkways.

Figure 8: GeFF fuses low-resolution coarse 2D features from multiple views for
refinement. (a) A single RGB view of the object. (b) coarse 2D heatmap with
text query ‘toy duck’ by CLIP [13]. (c) 3D heatmap from GeFF with clean and
sharp object boundary. (Best viewed when zoomed in.)

###  V-F Ablation Studies in Simulation

Figure 9: Snapshots of the simulation environments and objects we use in our
experiment. From this figure we are able to see the large domain gap between
real-world and simulation, yet our method is able to handle the objects in
simulation successfully.

We use Gazebo [72] and Habitat [73] with OVMM extended data [73, 9] as our
simulators. We use two environments (office and warehouse) with 8 objects in
Gazebo, and use 7 scenes with 31 objects in Habitat. We also test the results
with scene change in Gazebo. The scene change mainly consists of two
categories: (1) adding dynamic obstacles and (2) add new target object. We use
large objects such as bookshelf and big box to test the obstacle avoidance
ability. We also test small-size objects like soda to indicate the open-set
ability of GeFF. The results are shown in Tabs. III and IV.

We notice that the full GeFF with auxiliary inputs consistently outperforms
the no auxiliary input version, which confirms the necessity of using
foundation models during inference time. In addition, given perfect camera
poses and depth, the success rate of goal localization and navigation are good
even when there is scene change, which identify the bottleneck for GeFF in
real world as the localization accuracy and depth quality. In addition, we
notice that the choice of vision foundation models matters. We use DinoV2
[26], which generates feature maps with higher resolution but is not aligned
with languages, for goal navigation on Habitat. The feature of the goal object
is specified using an image where DinoV2 [26] averages goal object features.
For the Habitat OVMM dataset [9], DinoV2 shows good sucess rate, which hints
future research to fuse features from multiple foundation models for GeFF.
However, as illustrated in Fig. 9, the simulation environment has a large
domain gap from real-world data. Therefore, in habitat, GeFF using CLIP and
queried with text demonstrates unsatisfactory performance in finding goal
objects.

|  | GeFF (Ours) | GeFF no aux.
---|---|---|---
Env. | Scene Change | Goal Succ. | Nav. Succ. | Goal Succ. | Nav. Succ.
Office | ✗ | 100.0% | 75.0% | 66.7% | 25.0%
| ✓ | 75.0% | 66.7% | 63.6% | 8.3%
Warehouse | ✗ | 88.9% | 66.7% | 66.7% | 44.4%
| ✓ | 88.9% | 77.8% | 66.7% | 33.3%
Overall | ✗ | 94.4% | 70.8% | 66.7% | 34.7%
| ✓ | 81.3% | 72.2% | 65.2% | 20.8%

TABLE III: We compare with the baseline method without auxiliary (aux.) inputs
of GeFF encoder. We evaluate the success rate of finding goal objects (Goal
Succ.) as well as navigation success rate (Nav. Succ.). Note that in warehouse
with scene change the navigation performs slightly better. This is due to the
robot hit on the rack once while navigating and the case may happen when there
are changes in scenes too.

Distill. Model | DinoV2 [26] | CLIP-Text [13] | CLIP-Image [13]
---|---|---|---
Nav. Succ. | 67.6% | 10.8% | 27.0%

TABLE IV: Open-set object navigation results on habitat. We perform ablation
study by distilling different vision foundation models.

###  V-G Failure Analysis

We perform an in-depth analysis of failure cases to facilitate future
research. Perception failure, including both the failure to precisely
determine objects’ goal locations or the inability to localize the robot, is
the critical factor that leads to navigation failure in complex scenes (i.e.,
kitchen). In challenging lighting conditions, such as the kitchen where the
spotlights may directly illuminate the localization camera, the VIO algorithm
could be inaccurate due to its internal assumption of consistent lighting.
This leads to imprecise world representation and undesired errors when solving
inverse kinematics for manipulation. Future work could improve this by either
1) using an in-hand camera for manipulation or 2) designing a high-level
policy that does not rely on accurate camera poses over the long horizon.

Besides perception failures, the manipulation also sometimes fails due to the
current open-loop manipulation scheme, especially the end effector of the Z1
arm often fails on low-friction objects (e.g., plastic drink bottles). Future
work could include transforming the current manipulation solution to a close-
loop system.

Failure Analysis in Scene Change. Scene changes pose unique challenges to
perception. Though we can instruct the robot to navigate to furniture to map
and manipulate new objects, the robot obtains only a single view of the
object, which may lead to inaccurate representation unlike Fig. 8. Future work
could design an exploration policy that attempts to automatically obtain
multi-views of the object or learn priors for shape completion.

##  VI Conclusion

In this paper, we present GeFF, a scene-level generalizable neural feature
field with feature distillation from VLM that provides a unified
representation for robot navigation and manipulation. Deployed on a
quadrupedal robot with a manipulator, GeFF demonstrates zero-shot object
retrieval ability in real-time in real-world environments. Using common motion
planners and controllers powered by GeFF, we show competitive results in the
open-set mobile manipulation tasks.

A promising future direction that may potentially address both issues is to
learn a unified control policy on top of GeFF features to close the control
loop, which can address the aforementioned failure reasons.

## References

  * [1] Abhinav Gupta, Adithyavairavan Murali, Dhiraj Prakashchand Gandhi, and Lerrel Pinto.  Robot learning in homes: Improving generalization and reducing dataset bias.  Advances in Neural Information Processing Systems, 31:9094–9104, 2018.
  * [2] Joao MC Marques, Jing-Chen Peng, Patrick Naughton, Yifan Zhu, James S Nam, and Kris Hauser.  Commodity telepresence with team avatrina’s nursebot in the ana avatar xprize finals.  In ICRA 2023 2nd Workshop on Toward Robot Avatars, 2023.
  * [3] Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser.  Tidybot: Personalized robot assistance with large language models.  Autonomous Robots, 2023.
  * [4] Yulun Tian, Yun Chang, Fernando Herrera Arias, Carlos Nieto-Granda, Jonathan P. How, and Luca Carlone.  Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems.  IEEE Transactions on Robotics (T-RO), 38(4):2022–2038, 2022.
  * [5] Arash Asgharivaskasi and Nikolay Atanasov.  Semantic OcTree Mapping and Shannon Mutual Information Computation for Robot Exploration.  IEEE Transactions on Robotics (T-RO), 39(3):1910–1928, 2023.
  * [6] Dhruv Shah, Ajay Sridhar, Arjun Bhorkar, Noriaki Hirose, and Sergey Levine.  Gnm: A general navigation model to drive any robot.  In ICRA, 2023.
  * [7] Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, and Sergey Levine.  Vint: A foundation model for visual navigation.  In CORL, 2023.
  * [8] Fan Wang and Kris Hauser.  Stable bin packing of non-convex 3d objects with a robot manipulator.  In ICRA, 2019.
  * [9] Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alexander William Clegg, John Turner, et al.  Homerobot: Open-vocabulary mobile manipulation.  arXiv preprint arXiv:2306.11565, 2023.
  * [10] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [11] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In CVPR, 2021.
  * [12] Jianglong Ye, Naiyan Wang, and Xiaolong Wang.  Featurenerf: Learning generalizable nerfs by distilling foundation models.  In ICCV, 2023.
  * [13] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML. PMLR, 2021.
  * [14] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik.  Lerf: Language embedded radiance fields.  In ICCV, 2023.
  * [15] Alex Trevithick and Bo Yang.  Grf: Learning a general radiance field for 3d representation and rendering.  In ICCV, 2021.
  * [16] Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, and Wenping Wang.  F2-nerf: Fast neural radiance field training with free camera trajectories.  In CVPR, 2023.
  * [17] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Zip-nerf: Anti-aliased grid-based neural radiance fields.  arXiv preprint arXiv:2304.06706, 2023.
  * [18] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  arXiv preprint arXiv:2106.10689, 2021.
  * [19] Mukund Varma, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang.  Is attention all that nerf needs?  In ICLR, 2023.
  * [20] Jiteng Mu, Shen Sang, Nuno Vasconcelos, and Xiaolong Wang.  Actorsnerf: Animatable few-shot human rendering with generalizable nerfs.  In ICCV, pages 18391–18401, 2023.
  * [21] A. Tewari, J. Thies, B. Mildenhall, P. Srinivasan, E. Tretschk, W. Yifan, C. Lassner, V. Sitzmann, R. Martin-Brualla, S. Lombardi, T. Simon, C. Theobalt, M. Nießner, J. T. Barron, G. Wetzstein, M. Zollhöfer, and V. Golyanik.  Advances in neural rendering.  In arXiv:2111.05849, 2021.
  * [22] Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, and Andrea Tagliasacchi.  Lolnerf: Learn from one look.  In CVPR, 2022.
  * [23] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann.  Decomposing nerf for editing via feature field distillation.  NeurIPS, 2022.
  * [24] Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi.  Neural feature fusion fields: 3d distillation of self-supervised 2d image representations.  In International Conference on 3D Vision (3DV), 2022.
  * [25] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
  * [26] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  arXiv preprint arXiv:2304.07193, 2023.
  * [27] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [28] William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, and Phillip Isola.  Distilled feature fields enable few-shot language-guided manipulation.  arXiv preprint arXiv:2308.07931, 2023.
  * [29] Yanjie Ze, Ge Yan, Yueh-Hua Wu, Annabella Macaluso, Yuying Ge, Jianglong Ye, Nicklas Hansen, Li Erran Li, and Xiaolong Wang.  Gnfactor: Multi-task real robot learning with generalizable neural feature fields.  In CoRL, 2023.
  * [30] Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad Al-Halah, Jitendra Malik, and Kristen Grauman.  Poni: Potential functions for objectgoal navigation with interaction-free learning.  In CVPR, 2022.
  * [31] Theophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, and Devendra Singh Chaplot.  Navigating to objects in the real world.  In SCIENCE ROBOTICS, 2023.
  * [32] Nur Muhammad (Mahi) Shafiullah, Chris Paxton, Lerrel Pinto1 Soumith Chintala, and Arthur Szlam.  Clip-fields: Weakly supervised semantic fields for robotic memory.  In RSS, 2023.
  * [33] Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S Ryoo, Austin Stone, and Daniel Kappler.  Open-vocabulary queryable scene representations for real world planning.  In ICRA, 2023.
  * [34] Haokuan Luo, Albert Yue, Zhang-Wei Hong, and Pulkit Agrawal.  Stubborn: A strong baseline for indoor object navigation.  In IROS, 2022.
  * [35] Dhruv Shah, Michael Equi, Błażej Osiński, Fei Xia, Brian Ichter, and Sergey Levine.  Navigation with large language models: Semantic guesswork as a heuristic for planning.  In Conference on Robot Learning (CoRL), 2023.
  * [36] Saeid Amiri, Kishan Chandan, and Shiqi Zhang.  Reasoning with scene graphs for robot planning under partial observability.  IEEE Robotics and Automation Letters (RAL), 7(2):5560–5567, 2022.
  * [37] Zachary Ravichandran, Lisa Peng, Nathan Hughes, J Daniel Griffith, and Luca Carlone.  Hierarchical representations and explicit memory: Learning effective navigation policies on 3d scene graphs using graph neural networks.  In IEEE International Conference on Robotics and Automation (ICRA), pages 9272–9279, 2022.
  * [38] Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov.  Object goal navigation using goal-oriented semantic exploration.  In In Neural Information Processing Systems (NeurIPS), 2020.
  * [39] Karmesh Yadav, Arjun Majumdar, Ram Ramrakhya, Naoki Yokoyama, Alexei Baevski, Zsolt Kira, Oleksandr Maksymets, and Dhruv Batra.  Ovrl-v2: A simple state-of-art baseline for imagenav and objectnav.  In arXiv:2303.07798, 2023.
  * [40] Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad Al-Halah, Jitendra Malik, and Kristen Grauman.  Poni: Potential functions for objectgoal navigation with interaction-free learning.  In CVPR, 2022.
  * [41] Nuri Kim, Obin Kwon, Hwiyeon Yoo, Yunho Choi, Jeongho Park, and Songhwai Oh.  Topological semantic graph memory for image-goal navigation.  In CoRL, 2022.
  * [42] Pierre Marza, Laetitia Matignon, Olivier Simonin, and Christian Wolf.  Multi-object navigation with dynamically learned neural implicit representations.  In ICCV, 2023.
  * [43] Leslie Pack Kaelbling and Tomas Lozano-Perez.  Unifying perception, estimation and action for mobile manipulation via belief space planning.  In ICRA, 2012.
  * [44] Charles Sun, Jędrzej Orbik, Brian Yang Coline Devin, Abhishek Gupta, Glen Berseth, and Sergey Levine.  Fully autonomous real-world reinforcement learning with applications to mobile manipulation.  In CoRL, 2021.
  * [45] Josiah Wong, Albert Tung, Andrey Kurenkov, Ajay Mandlekar, Li Fei-Fei, Silvio Savarese, and Roberto Mart´ın-Mart´ın.  Error-aware imitation learning from teleoperation data for mobile manipulation.  In CoRL, 2021.
  * [46] Jiayuan Gu, Devendra Singh Chaplot, Hao Su, and Jitendra Malik.  Multi-skill mobile manipulation for object rearrangement.  In ICLR, 2023.
  * [47] Priyam Parashar, Vidhi Jain, Xiaohan Zhang, Jay Vakil1, Sam Powers, Yonathan Bisk, and Chris Paxton.  Slap: Spatial-language attention policies.  In CoRL, 2023.
  * [48] Fei Xia, Chengshu Li, Roberto Martın-Martın, Or Litany, Alexander Toshev, and Silvio Savarese.  Relmogen: Integrating motion generation in reinforcement learning for mobile manipulation.  In ICRA, 2021.
  * [49] Naoki Yokoyama, Alex Clegg, Joanne Truong, Eric Undersander, Tsung-Yen Yang, Sergio Arnaud, Sehoon Ha, Dhruv Batra, and Akshara Rai.  Asc: Adaptive skill coordination for robotic mobile manipulation.  In arXiv:2304.00410, 2023.
  * [50] Xiaoyu Huang, Dhruv Batra, Akshara Rai, and Andrew Szot.  Skill transformer: A monolithic policy for mobile manipulation.  In ICCV, 2023.
  * [51] Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, Chelsea Finn, and Karol Hausman.  Open-world object manipulation using pre-trained vision-language model.  In CoRL, 2023.
  * [52] Kenneth Blomqvist, Michel Breyer, Andrei Cramariuc, Julian Forster, Margarita Grinvald, Florian Tschopp, Jen Jen Chung, Lionel Ott, Juan Nieto, and Roland Siegwart.  Go fetch: Mobile manipulation in unstructured environments.  In arXiv:2004.00899, 2020.
  * [53] Simon Zimmermann, Roi Poranne, and Stelian Coros.  Go fetch! - dynamic grasps using boston dynamics spot with external robotic arm.  In ICRA, 2021.
  * [54] Riccardo Parosi, Mattia Risiglione, Darwin G. Caldwell, Claudio Semini, and Victor Barasuol.  Kinematically-decoupled impedance control for fast object visual servoing and grasping on quadruped manipulators.  In IROS, 2023.
  * [55] Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song.  Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation.  In CVPR, 2023.
  * [56] Peiqi Liu, Yaswanth Orru, Chris Paxton, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto.  Ok-robot: What really matters in integrating open-knowledge models for robotics.  arXiv preprint arXiv:2401.12202, 2024.
  * [57] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard.  Visual language maps for robot navigation.  In ICRA, 2023.
  * [58] Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, et al.  Conceptfusion: Open-set multimodal 3d mapping.  arXiv preprint arXiv:2302.07241, 2023.
  * [59] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  arXiv preprint arXiv:2304.02643, 2023.
  * [60] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In CVPR, 2017.
  * [61] Yang Fu, Shalini De Mello, Xueting Li, Amey Kulkarni, Jan Kautz, Xiaolong Wang, and Sifei Liu.  3d reconstruction with generalizable neural fields using scene priors.  arXiv preprint arXiv:2309.15164, 2023.
  * [62] Di Huang, Sida Peng, Tong He, Honghui Yang, Xiaowei Zhou, and Wanli Ouyang.  Ponder: Point cloud pre-training via neural rendering.  In ICCV, 2023.
  * [63] Chong Zhou, Chen Change Loy, and Bo Dai.  Extract free dense labels from clip.  In ECCV, 2022.
  * [64] Joseph Ortiz, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoefer, and Mustafa Mukadam.  isdf: Real-time neural signed distance fields for robot perception.  In RSS, 2022.
  * [65] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman.  Implicit geometric regularization for learning shapes.  In ICML. PMLR, 2020.
  * [66] Xuran Pan, Zhuofan Xia, Shiji Song, Li Erran Li, and Gao Huang.  3d object detection with pointformer.  In CVPR, 2021.
  * [67] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun.  Point transformer.  In ICCV, 2021.
  * [68] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In CVPR, 2017.
  * [69] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [70] Otto Seiskari, Pekka Rantalankila, Juho Kannala, Jerry Ylilammi, Esa Rahtu, and Arno Solin.  Hybvio: Pushing the limits of real-time visual-inertial odometry.  In WACV, 2022.
  * [71] Ioan A. Şucan, Mark Moll, and Lydia E. Kavraki.  The Open Motion Planning Library.  IEEE Robotics & Automation Magazine, 19(4):72–82, December 2012.  https://ompl.kavrakilab.org.
  * [72] https://gazebosim.org/home.
  * [73] Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra.  Habitat 2.0: Training home assistants to rearrange their habitat.  In Advances in Neural Information Processing Systems (NeurIPS), 2021.


Best Simultaneous Approximation of Functions and
a Generalized Minimax Theorem
Shinji Tanimoto
Department of Mathematics, University of Kochi,

Kochi 780-8515, Japan***Former affiliation.

###### Abstract

Best simultaneous approximation (BSA) for finitely or infinitely many
functions are considered under the uniform norm and other important norms.
Characterization theorems for a BSA from a finite-dimensional subspace are
obtained by a generalized minimax theorem. From the characterization theorem a
strong unicity theorem is also deduced for a BSA.

1\. Introduction

Let $\\{f_{a}\\}$ be a family of functions obtained in association with each
element $a$ in a set $A$. The purpose is to approximate these functions
$\\{f_{a}\\}_{a\in A}$ simultaneously from a subspace $H$ contained in a
function space. In this scetion $X$ is a compact Hausdorff space and $C(X)$
denotes the set of all real-valued continuous functions on $X$.
In [6] such an approximation problem was considered for real-valued functions
$\\{f_{a}\\}_{a\in A}$ defined on $X$. The continuity of functions themselves
is not supposed, but we assume uniform boundedness of the functions. For a
specified subspace $H$ of finite dimension in $C(X)$, we say that $f^{*}\in H$
is a best simultaneous approximation $($BSA$)$ for $\\{f_{a}\\}_{a\in A}$ from
$H$, whenever $f^{*}$ satisfies the inequality

$\max_{a\in A,\;x\in X}|f_{a}(x)-f^{*}(x)|\leq\max_{a\in A,\;x\in
X}|f_{a}(x)-f(x)|{\rm~{}~{}for~{}all}~{}f\in H.$

In [6] a characterization theorem for a BSA was deduced under the following
conditions:

  * •

both functions (of $x$) $\inf_{a\in A}f_{a}(x)$ and $\sup_{a\in A}f_{a}(x)$
belong to $C(X)$;

  * •

for each $x\in X$, the infimum and supremum of $f_{a}(x)$ are, respectively,
attained by some $f_{a}(x)$.

Moreover, if $H$ is a Haar subspace, a strong unicity theorem for a BSA was
obtained from the characterization theorem (see Section 3). When $X$ is a
finite closed interval, an alternation theorem for a BSA was also obtained
that is similar to the ordinary one (see [1]).
In the next section we consider a BSA problem in a function space $C(X,Y)$
(the set of all continuous functions from $X$ to $Y$), $Y$ being a normed
linear space over the real field $\mathbb{R}$ with norm
$\parallel\cdot\parallel$. When a family of functions $\\{f_{a}\\}_{a\in
A}\subset C(X,Y)$ and a finite-dimensional subspace $H\subset C(X,Y)$ are
given, $f^{*}\in H$ is said to be a BSA to the functions $\\{f_{a}\\}_{a\in
A}$ from $H$, if the inequality

$\max_{a\in A,\;x\in X}\parallel f_{a}(x)-f^{*}(x)\parallel\>\leq\max_{a\in
A,\;x\in X}\parallel f_{a}(x)-f(x)\parallel$

holds for all $f\in H$. In this setting we will deduce a characterization
theorem of a BSA for $\\{f_{a}\\}_{a\in A}$ that corresponds to the one of
[6].
In Section 3, from this characterization theorem, a strong unicity theorem is
derived in the function space $C(X)$.
In Section 4 we treat another BSA problem in $L^{p}$-approximation and obtain
a characterization theorem of a BSA for finitely or infinitely many functions
$\\{f_{a}\\}\subset L^{p}$. These characterization theorems are proved by
means of a generalized minimax theorem ([4, Corollary 3.3]). For convenience
sake we restate it as a lemma.

Lemma 1. Let $U$ be an $n$-dimensional, compact convex subset of a Hausdorff
topological vector space, $V$ a compact Hausdorff space, and let $J:U\times
V\to\mathbb{R}$ be a jointly continuous function. An element $u^{*}\in U$
minimizes $\max_{v\in V}J(u,v)$ over $U$, if and only if there exist
nonnegative numbers $\lambda_{1},\ldots,\lambda_{n+1}$ with sum one, and
$v^{*}_{1},\ldots,v^{*}_{n+1}\in V$ such that

$\displaystyle\sum_{i=1}^{n+1}\mu_{i}J(u^{*},v_{i})\leq\sum_{i=1}^{n+1}\lambda_{i}J(u^{*},v^{*}_{i})\leq\sum_{i=1}^{n+1}\lambda_{i}J(u,v^{*}_{i})$
(1)

holds for all $u\in U$, $v_{1},\ldots,v_{n+1}\in V$, and for all nonnegative
numbers $\mu_{1},\ldots,\mu_{n+1}$ with sum one.

As a useful remark we add that, ignoring all $i$ such that $\lambda_{i}=0$ and
rearranging the suffix, (1) can be described as

$\sum_{i=1}^{n+1}\mu_{i}J(u^{*},v_{i})\leq\sum_{i=1}^{k}\lambda_{i}J(u^{*},v^{*}_{i})\leq\sum_{i=1}^{k}\lambda_{i}J(u,v^{*}_{i}),$

for some $k\,(1\leq k\leq n+1)$ with
$\sum_{i=1}^{k}\lambda_{i}=1~{}(\lambda_{i}>0)$.

2\. Characterization theorem

For a compact Hausdorff space $X$ and a normed linear space $Y$ over the real
field $\mathbb{R}$ with norm $\parallel\cdot\parallel$, we consider the set
$C(X,Y)$ of all continuous functions from $X$ to $Y$. A family of functions
$\\{f_{a}\\}_{a\in A}\subset C(X,Y)$ and an $n$-dimensional subspace $H\subset
C(X,Y)$ are given, where $n$ is a positive integer. For $f\in C(X,Y)$ we
define the uniform norm of $f$ by

$|||f|||=\max_{x\in X}\parallel f(x)\parallel,$

and we endow the function space $C(X,Y)$ with this norm. Therefore, a BSA
$f^{*}\in H$ is characterized by

$\max_{a\in A}|||f_{a}-f^{*}|||\>\leq\max_{a\in
A}|||f_{a}-f|||{\rm~{}~{}for~{}all}~{}f\in H.$

We assume that $A$ is a Hausdorff topological space and impose the two
conditions:

  * (a)

$A$ is compact;

  * (b)

the mapping $A\to C(X,Y)$ defined by $a\mapsto f_{a}$ is continuous.

Now let us introduce the following function

$J(f,a,x)=\;\parallel f_{a}(x)-f(x)\parallel$

defined on $H\times A\times X$. It is a jointly continuous function and convex
in the argument $f$. Moreover, $A\times X$ is a compact set with respect to
the product topology. Under this setting we have the following
characterization theorem for a BSA.

Theorem 2. An element $f^{*}\in H$ is a BSA to $\\{f_{a}\\}$ from $H$ if and
only if, for some positive integer $k\,(1\leq k\leq n+1)$, there exist
$a_{1},\ldots,a_{k}\in A$, $x_{1},\ldots,x_{k}\in X$ and positive numbers
$\lambda_{1},\ldots,\lambda_{k}$, whose sum is one, such that

  * (i)

$\sum_{i=1}^{k}\lambda_{i}\parallel
f_{a_{i}}(x_{i})-f^{*}(x_{i})\parallel\;\leq\sum_{i=1}^{k}\lambda_{i}\parallel
f_{a_{i}}(x_{i})-f(x_{i})\parallel~{}~{}{\it for~{}all}~{}f\in H;$

  * (ii)

$\parallel
f_{a_{i}}(x_{i})-f^{*}(x_{i})\parallel\;=|||f_{a_{i}}-f^{*}|||=\max_{a\in
A}|||f_{a}-f^{*}|||~{}for~{}all~{}i\,(1\leq i\leq k).$

(Proof) Let $f^{*}$ be a BSA. We define $U=\\{f\in H:|||f-f^{*}|||\leq 1\\}$.
Then $U$ is a compact convex set of $H$, since $H$ is finite-dimensional.
First we consider the approximation problem over the set $U$ in place of $H$.
Then $f^{*}$ is also a minimizer of $\max_{(a,x)\in A\times X}J(f,a,x)$ over
$U$. Applying Lemma 1 and its remark to this situation, we see that, for some
$k\,(1\leq k\leq n+1)$, there exist $(a_{1},x_{1}),\ldots,(a_{k},x_{k})\in
A\times X$, and positive numbers $\lambda_{1},\ldots,\lambda_{k}$ with
$\sum_{i=1}^{k}\lambda_{i}=1$ such that the following two inequalities hold:

$\displaystyle\sum_{i=1}^{k}\lambda_{i}J(f^{*},a_{i},x_{i})\leq\sum_{i=1}^{k}\lambda_{i}J(f,a_{i},x_{i}){\rm~{}for~{}all}~{}f\in
U;$ (2)
$\displaystyle\sum_{i=1}^{n+1}\mu_{i}J(f^{*},b_{i},y_{i})\leq\sum_{i=1}^{k}\lambda_{i}J(f^{*},a_{i},x_{i})$
(3)

for all $b_{1},\ldots,b_{n+1}\in A$, $y_{1},\ldots,y_{n+1}\in X$ and all
nonnegative numbers $\mu_{1},\ldots,\mu_{n+1}$ with sum one.
The right-hand side of (2) is a convex function of $f$ and has a local minimum
at $f^{*}\in U$. By a property of convex functions it follows that it has a
global minimum at $f^{*}\in H$, which implies (i). Next in (3) putting
$\mu_{1}=1$ while other $\mu_{i}=0$, and $b_{1}=a$ for any $a\in A$, we have
$\parallel f_{a}(y)-f^{*}(y)\parallel\;\leq\sum_{i=1}^{k}\lambda_{i}\parallel
f_{a_{i}}(x_{i})-f^{*}(x_{i})\parallel$ for all $y\in X$, and hence for every
$a\in A$

$|||f_{a}-f^{*}|||\leq\sum_{i=1}^{k}\lambda_{i}\parallel
f_{a_{i}}(x_{i})-f^{*}(x_{i})\parallel.$

This shows that $\max_{a\in
A}|||f_{a}-f^{*}|||\leq\sum_{i=1}^{k}\lambda_{i}\parallel
f_{a_{i}}(x_{i})-f^{*}(x_{i})\parallel$. Using $\sum_{i=1}^{k}\lambda_{i}=1$
$(\lambda_{i}>0)$, we conclude that

$\max_{a\in A}|||f_{a}-f^{*}|||\leq\sum_{i=1}^{k}\lambda_{i}\parallel
f_{a_{i}}(x_{i})-f^{*}(x_{i})\parallel\,\leq\max_{a\in A}|||f_{a}-f^{*}|||,$

which implies (ii).
Conversely, suppose that $f^{*}\in H$ satisfies conditions (i) and (ii) for
$a_{i}$’s in $A$, $x_{i}$’s of $X$ and positive numbers $\lambda_{i}$’s such
that $\sum_{i=1}^{k}\lambda_{i}=1$. Then these conditions imply that, for any
$f\in H$,

$\displaystyle\max_{a\in
A}|||f_{a}-f^{*}|||=\sum_{i=1}^{k}\lambda_{i}\parallel
f_{a_{i}}(x_{i})-f^{*}(x_{i})\parallel\,\leq\sum_{i=1}^{k}\lambda_{i}\parallel
f_{a_{i}}(x_{i})-f(x_{i})\parallel\,\leq\max_{a\in A}|||f_{a}-f|||,$

showing that $f^{*}$ becomes a BSA. This completes the proof.

Next we consider the case where $A$ is a finite set, as discussed in [5]. Let
$g_{1},\ldots,g_{\ell}\in C(X,Y)$ be given. In order to consider BSA to
$\\{g_{j}\\}$, we introduce a compact set

$A=\Big{\\{}a=(\alpha_{1},\ldots,\alpha_{\ell}):\sum_{j=1}^{\ell}\alpha_{j}=1,~{}\alpha_{j}\geq
0~{}(1\leq j\leq\ell)\Big{\\}}.$

For each $a\in A$ we set $g_{a}=\sum^{\ell}_{j=1}\alpha_{j}g_{j}$. For $f\in
H$, an $n$-dimensional subspace of $C(X,Y)$, we have, using the convexity of
norm

$\max_{1\leq j\leq\ell}|||g_{j}-f|||\leq\max_{a\in A}|||g_{a}-f|||=\max_{a\in
A}|||\sum^{\ell}_{1\leq j\leq\ell}\alpha_{j}(g_{j}-f)|||\leq\max_{1\leq
j\leq\ell}|||g_{j}-f|||.$

Thus our approximation problem is reduced to simultaneously approximate
$\\{g_{a}\\}~{}(a\in A)$ from $H$. Then as a special case of Theorem 2 follows
the characterization theorem in [5].

3\. Strong unicity theorem

Suppose that the norm of $Y$ is defined by means of an inner product
$\langle~{},~{}\rangle$ so that ${\parallel y\parallel}^{2}=\langle
y,y\rangle$ for $y\in Y$. The condition (i) of Theorem 2 is equivalent to the
assertion; the function of a real variable $t$

$\sum_{i=1}^{k}\lambda_{i}\parallel
f_{a_{i}}(x_{i})-f^{*}(x_{i})+tf(x_{i})\parallel$

attains the minimum at $t=0$ for every $f\in H$. By a simple calculation we
obtain the next corollary, using the inner product.

Corollary 3. Let $Y$ be an inner product space. An element $f^{*}\in H$ is a
BSA to $\\{f_{a}\\}$ from $H$ if and only if, for some positive integer
$k\,(1\leq k\leq n+1)$, there exist $a_{1},\ldots,a_{k}\in A$,
$x_{1},\ldots,x_{k}\in X$ and positive numbers
$\lambda_{1},\ldots,\lambda_{k}$, whose sum is one, such that
$\displaystyle~{}~{}{\rm(i^{\prime})}~{}~{}~{}\sum_{i=1}^{k}\lambda_{i}\langle
f_{a_{i}}(x_{i})-f^{*}(x_{i}),f(x_{i})\rangle=0~{}~{}for~{}all~{}f\in H;$
$\displaystyle~{}~{}{\rm(ii)}~{}~{}\parallel
f_{a_{i}}(x_{i})-f^{*}(x_{i})\parallel\;=|||f_{a_{i}}-f^{*}|||=\max_{a\in
A}|||f_{a}-f^{*}|||~{}for~{}all~{}i\,(1\leq i\leq k).$

In what follows we consider the case of $Y=\mathbb{R}$ and so we deal with the
function space $C(X)$ as in Section 1. Referring to [2, p.91] or [4, Section
5], an $n$-dimensional subspace $H\subset C(X)$ is called a Haar subspace if,
for $n$ distinct elements $x_{1},\ldots,x_{n}\in X$ and for $n$ arbitrary
numbers $r_{1},\ldots,r_{n}\in\mathbb{R}$, there exists a unique $f\in H$ such
that $f(x_{k})=r_{k}\;(1\leq k\leq n)$.
A one-dimensional Haar subspace is obviously spanned by any function that does
not vanish in $X$. A two-dimensional Haar subspace is spanned by every pair of
functions $f,\;g\in C(X)$ satisfying $f(x)g(y)\neq f(y)g(x)$ whenever $x\neq
y$, and so on (by linear algebra). Here the uniform norm of $C(X)$ is defined
by

$\parallel f\parallel=\max_{x\in X}|f(x)|$

for $f\in C(X)$. Now we prove the following strong unicity theorem.

Theorem 4. Suppose that the compact set $X$ contains at least $n+1$ elements,
where $n\geq 1$. Let a BSA $f^{*}\in H$ to $\\{f_{a}\\}_{a\in A}$ from an
$n$-dimensional Haar subspace $H$ satisfy (i’) and (ii) of Corollary 3, for
some integer $k\,(1\leq k\leq n+1)$. If $k$ elements $x_{1},\ldots,x_{k}\in X$
are all distinct and the common value of (ii) is not zero, then there exists a
positive number $\gamma$ such that

$\max_{a\in A}\parallel f_{a}-h\parallel\geq\max_{a\in A}\parallel
f_{a}-f^{*}\parallel+\,\gamma\parallel f^{*}-h\parallel~{}~{}for~{}all~{}h\in
H.$

(Proof) First we show that $k=n+1$. If $k\leq n$, there is a function $f\in H$
such that $f(x_{i})=f_{a_{i}}(x_{i})-f^{*}(x_{i})$ for $i\,(1\leq i\leq k)$,
since $H$ is a Haar subspace. Inserting this $f$ into the equality (i’) we
have

$\displaystyle\sum_{i=1}^{k}\lambda_{i}|f_{a_{i}}(x_{i})-f^{*}(x_{i})|^{2}=|f_{a_{i}}(x_{i})-f^{*}(x_{i})|^{2}=0.$

However, we assumed that $\delta=|f_{a_{i}}(x_{i})-f^{*}(x_{i})|=\max_{a\in
A}\parallel f_{a}-f^{*}\parallel$ is not zero. Hence we must have $k=n+1$.
Letting $h$ be an arbitrary element in $H$ such that $\parallel h\parallel=1$,
condition (i’) can be written as

$\displaystyle\sum_{i=1}^{n+1}\lambda_{i}\sigma_{i}h(x_{i})=0,~{}~{}\sigma_{i}=(f_{a_{i}}(x_{i})-f^{*}(x_{i}))/\delta~{}~{}(1\leq
i\leq n+1).$

Since $H$ is a Haar subspace and $\parallel h\parallel=1$, it follows that
$\max_{1\leq i\leq n+1}\sigma_{i}h(x_{i})>0$. If we set

$\displaystyle\gamma=\min_{\parallel h\parallel=1}\,\max_{1\leq i\leq
n+1}\sigma_{i}h(x_{i}),$

then $\gamma$ is positive, since the set $\\{h\in H:\;\parallel
h\parallel=1\\}$ is compact.
Let $f\neq f^{*}$ be any element in $H$ and set $h=(f^{*}-f)/\parallel
f^{*}-f\parallel$. Then there exists at least one $i$ satisfying

$\displaystyle\sigma_{i}h(x_{i})=\sigma_{i}(f^{*}(x_{i})-f(x_{i}))/\parallel
f^{*}-f\parallel\geq\,\gamma,$

and the required inequality follows using this $i$ and $|\sigma_{i}|=1$:

$\displaystyle\max_{a\in A}\parallel
f_{a}-f\parallel\geq\sigma_{i}(f_{a_{i}}(x_{i})-f(x_{i}))=\sigma_{i}(f_{a_{i}}(x_{i})-f^{*}(x_{i}))+\sigma_{i}(f^{*}(x_{i})-f(x_{i}))$
$\displaystyle\geq\max_{a\in A}\parallel
f_{a}-f^{*}\parallel+\,\gamma\parallel f^{*}-f\parallel.$

4\. BSA on $L^{p}$-spaces

Let $(S,m)$ be a $\sigma$-finite positive measure space and
$L^{p}(S,m)~{}(1\leq p<\infty)$ the set of all real-valued measurable
functions $f$ such that $|f|^{p}$ are integrable over $S$. For such $p$ let
$q$ be the real number determined by $p^{-1}+q^{-1}=1$ for $p>1$, and
$q=\infty$ for $p=1$. We use the following notation (similarly for $\parallel
g\parallel_{q}$ of $g\in L^{q}$),

$\displaystyle\parallel
f\parallel_{p}\,=\,\big{(}\int_{S}|f|^{p}dm\big{)}^{1/p},$

and $\parallel f\parallel_{\infty}\,={\rm ess~{}sup}_{x\in S}|f(x)|$.
We also assume that $A$ is a compact set of a Hausdorff topological space and
to each $a\in A$ there corresponds a function $f_{a}$ which belongs to
$L^{p}(S,m)$, and that the mapping $A\to L^{p}(S,m)$ so defined is continuous.
Let $H$ be an $n$-dimensional subspace of $L^{p}(S,m)$, where $n\geq 1$. The
problem is to approximate simultaneously the functions $\\{f_{a}\\}$ by
elements of $H$. If $f^{*}\in H$ satisfies

$\displaystyle{\rm max}_{a\in A}\parallel f_{a}-f^{*}\parallel_{p}\;\leq\;{\rm
max}_{a\in A}\parallel f_{a}-f\parallel_{p}$ (4)

for all $f\in H$, we say that $f^{*}$ is a BSA to the functions $\\{f_{a}\\}$
from $H$.
In order to formulate this problem in relation to Lemma 1, we need the
following well-known facts. Property (a) is the Banach-Alaoglu theorem (see
[3, p.68]) and property (b) is the duality pairing (see [7, p.115]).

  * (a)

The dual space of $L^{p}(S,m)$ is $L^{q}(S,m)$ and $G=\\{g\in
L^{q}(S,m):\,\parallel g\parallel_{q}\,\leq 1\\}$ is a compact set in the
weak*-topology $\sigma(L^{q}(S,m),L^{p}(S,m))$.

  * (b)

For each $f\in L^{p}(S,m)$ we have $\parallel f\parallel_{p}\;=\,{\rm
max}_{g\in G}\int_{S}(gf)dm$.

Therefore, (4) is equivalent to the following:

$\displaystyle\max_{(a,g)\in A\times
G}\int_{S}g(f_{a}-f^{*})dm\leq\max_{(a,g)\in A\times G}\int_{S}g(f_{a}-f)dm$
(5)

for all $f\in H$, and the problem is to find a function $f^{*}$ satisfying
(5).
It follows from (a) that $A\times G$ is a compact set in the product topology
and it is easy to see that

$J(f,a,g)=\int_{S}g(f_{a}-f)dm$

is a jointly continuous function of the three variables $a\in A$, $g\in G$ and
$f\in H$, using Hölder’s inequality and the definition of the weak*-topology.
Moreover, it is a convex function with respect to $f$. Hence we can again
invoke Lemma 1 for the characterization of best approximations.

Theorem 5. An element $f^{*}\in H$ is a BSA to $\\{f_{a}\\}$ from $H$ if and
only if, for some positive integer $k\,(1\leq k\leq n+1)$, there exist
$a_{1},\ldots,a_{k}\in A$, $g_{1},\ldots,g_{k}\in G$ and positive numbers
$\lambda_{1},\ldots,\lambda_{k}$ with sum one, satisfying the following two
conditions:
(i)
$\displaystyle\int_{S}\big{(}\sum_{i=1}^{k}\lambda_{i}g_{i}\big{)}h\,dm=0~{}~{}for~{}all~{}h\in
H;$
(ii) $\displaystyle\int_{S}g_{i}(f_{a_{i}}-f^{*})dm=\,\parallel
f_{a_{i}}-f^{*}\parallel_{p}\;=\max_{a\in A}\parallel
f_{a}-f^{*}\parallel_{p}\,~{}for~{}all~{}i\,(1\leq i\leq k).$

(Proof) Let $f^{*}$ be a BSA. Define the set $U=\\{f\in H:{\parallel
f-f^{*}\parallel}_{p}\leq 1\\}$. Then $U$ is compact and convex, for $H$ is a
finite-dimensional subspace of $L^{p}(S,m)$. The convexity follows from
Minkowski’s inequality (see [7, p.33]). It is obvious that $f^{*}$ also
minimizes $\max_{(a,g)\in A\times G}J(f,a,g)$ over the set $U$. It follows
from Lemma 1, for some $k\,(1\leq k\leq n+1)$, that there exist
$a_{1},\ldots,a_{k}\in A$, $g_{1},\ldots,g_{k}\in G$, and numbers
$\lambda_{1},\ldots,\lambda_{k}>0$ with $\sum_{i=1}^{k}\lambda_{i}=1$ such
that the following two inequalities hold:

$\displaystyle\sum_{i=1}^{k}\int_{S}\lambda_{i}g_{i}(f_{a_{i}}-f^{*})dm\leq\sum_{i=1}^{k}\int_{S}\lambda_{i}g_{i}(f_{a_{i}}-f)dm$
(6)

for all $f\in U$; and

$\displaystyle\sum_{i=1}^{n+1}\int_{S}\mu_{i}h_{i}(f_{b_{i}}-f^{*})dm\leq\sum_{i=1}^{k}\int_{S}\lambda_{i}g_{i}(f_{a_{i}}-f^{*})dm$
(7)

for all $(b_{1},h_{1}),\ldots,(b_{n+1},h_{n+1})\in A\times G$, and all
nonnegative numbers $\mu_{1},\ldots,\mu_{n+1}$ with sum one.
Inequality (6) implies

$\displaystyle\sum_{i=1}^{k}\int_{S}\lambda_{i}g_{i}(f-f^{*})dm\leq
0{~{}~{}\rm for~{}all}\>f\in U.$

Let us put $f=f^{*}+th$, where $h\in H$ is arbitrary and $t>0$ is so small
that this $f$ belongs to $U$. Then we get

$\displaystyle\sum_{i=1}^{k}\int_{S}\lambda_{i}g_{i}h\,dm=\int_{S}\big{(}\sum_{i=1}^{k}\lambda_{i}g_{i}\big{)}h\,dm\leq
0{~{}~{}\rm for~{}all}\>h\in H,$

which means condition (i), since the left-hand side of the inequality must be
zero.
By setting $b_{i}=a_{i}$ and $\mu_{i}=\lambda_{i}$ for all $i$ ($1\leq i\leq
k$) in (7) and remarking property (b), we see that (7) implies

$\displaystyle\sum_{i=1}^{k}\lambda_{i}{\parallel
f_{a_{i}}-f^{*}\parallel}_{p}\leq\sum_{i=1}^{k}\int_{S}\lambda_{i}g_{i}(f_{a_{i}}-f^{*})dm.$

Since the reverse inequality always holds, we conclude

$\displaystyle\sum_{i=1}^{k}\lambda_{i}{\parallel
f_{a_{i}}-f^{*}\parallel}_{p}=\sum_{i=1}^{k}\int_{S}\lambda_{i}g_{i}(f_{a_{i}}-f^{*})dm\leq\max_{a\in
A}\parallel f_{a}-f^{*}\parallel_{p}.$

Next in (7) putting $\mu_{1}=1$ (so $\mu_{i}=0$ for $i\neq 1$) and $b_{1}=a$
for any $a\in A$, we have for any $a\in A$

$\displaystyle\max_{g\in G}\int_{S}g(f_{a}-f^{*})dm=\;\parallel
f_{a}-f^{*}\parallel_{p}\>\leq\sum_{i=1}^{k}\int_{S}\lambda_{i}g_{i}(f_{a_{i}}-f^{*})dm,$

hence

$\displaystyle\max_{a\in A}\parallel
f_{a}-f^{*}\parallel_{p}\>\leq\sum_{i=1}^{k}\int_{S}\lambda_{i}g_{i}(f_{a_{i}}-f^{*})dm\leq\max_{a\in
A}\parallel f_{a}-f^{*}\parallel_{p}.$ (8)

Then we conclude from (8) and $\lambda_{i}>0$ that for all $i\,(1\leq i\leq
k)$

$\displaystyle\parallel
f_{a_{i}}-f^{*}\parallel_{p}\;=\int_{S}g_{i}(f_{a_{i}}-f^{*})dm=\max_{a\in
A}\parallel f_{a}-f^{*}\parallel_{p}\;,$

which is condition (ii).
Conversely, suppose that $f^{*}$ satisfies conditions (i) and (ii). Let $f\in
H$ be any element. We have by (i)

$\displaystyle\sum_{i=1}^{k}\int_{S}\lambda_{i}g_{i}(f_{a_{i}}-f)dm-\sum_{i=1}^{k}\int_{S}\lambda_{i}g_{i}(f_{a_{i}}-f^{*})dm=\int_{S}\big{(}\sum_{i=1}^{k}\lambda_{i}g_{i}\big{)}(f^{*}-f)dm=0.$
(9)

On the other hand, using (ii),

$\displaystyle\sum_{i=1}^{k}\int_{S}\lambda_{i}g_{i}(f_{a_{i}}-f)dm-\sum_{i=1}^{k}\int_{S}\lambda_{i}g_{i}(f_{a_{i}}-f^{*})dm\leq\max_{1\leq
i\leq k}\parallel f_{a_{i}}-f\parallel_{p}$ $\displaystyle-\max_{a\in
A}\parallel f_{a}-f^{*}\parallel_{p}\;\leq\max_{a\in A}\parallel
f_{a}-f\parallel_{p}-\max_{a\in A}\parallel f_{a}-f^{*}\parallel_{p},$

where the condition $\sum_{i=1}^{k}\lambda_{i}=1~{}(\lambda_{i}>0)$ is used.
Immediately we conclude that $f^{*}$ is a BSA in view of (9), thereby
completing the proof.

References

  * [1]

E. W. Cheney, Introduction to Approximation Theory, McGraw-Hill, 1966.

  * [2]

R. B. Holmes, A Course on Optimization and Best Approximation, Lecture Notes
in Mathematics, Vol. 257, Springer-Verlag, 1972.

  * [3]

W. Rudin, Functional Analysis (2nd ed.), McGraw-Hill, 1991.

  * [4]

S. Tanimoto, Uniform approximation and a generalized minimax theorem, Journal
of Approximation Theory 45, 1–10, 1985.

  * [5]

S. Tanimoto, A characterization of best simultaneous approximations, Journal
of Approximation Theory 59, 359–361, 1989.

  * [6]

S. Tanimoto, On best simultaneous approximation, Mathematica Japonica 48,
275–279, 1998.

  * [7]

K. Yosida, Functional Analysis (3rd ed.), Springer-Verlag, 1971.