# Mock-local energy density of gravitational waves

Antoine Rignon-Bret and Simone Speziale
Aix Marseille Univ., Univ. de Toulon, CNRS, CPT, UMR 7332, 13288 Marseille,
France

###### Abstract

We propose a new set of BMS charges at null infinity, characterized by a
super-translation flux that contains only the ‘hard’ term. This is achieved
with a specific corner improvement of the symplectic 2-form, and we spell the
conditions under which it is unique. The charges are associated to a Wald-
Zoupas symplectic potential, and satisfy all standard criteria: they are
covariant, provide a center-less realization of the symmetry algebra, have
vanishing flux in non-radiative spacetimes, and vanish in Minkowski. We use
them to define a certain notion of localized energy density of gravitational
waves. They have potential applications to the generalized second law and to
soft theorems.

##  1 Introduction

Bondi’s energy loss formula is a cornerstone of our physical understanding of
general relativity. It was the historical proof that gravitational waves
dissipate energy, ending any lingering doubt on their physical existence. The
proof of dissipation refers to the _total_ gravitational energy; there is no
equivalent statement for a local notion of energy, in line with standard
intuition that general covariance prevents the existence of a local energy-
momentum tensor for the gravitational field. In this paper we report that
general covariance allows an additional step: it is possible to provide a
formula for a _local_ energy density of gravitational waves at future null
infinity which is necessarily dissipated by physical processes. The non-
locality is confined to the relation between its flux, or rate of change, and
the spacetime curvature. We further argue that even the flux can actually be
determined by a local measurement in space, provided a non-local initial
condition in time. The result is based on taking into account so-called
‘corner degrees of freedom’, and leads to an estimate of the amount of energy
that can be absorbed by a local experiment for a given flux.

The group of gravitational symmetries at null infinity has been known since
the seminal work of Bondi, van der Burg, Metzner and Sachs (BMS) [1, 2] and
extensively studied since. It is a generalization of the Poincaré group to
angle-dependent translations, known as super-translations. To these asymptotic
symmetries are associated asymptotic charges which are conserved in the
absence of radiation, and satisfy flux-balance laws in the presence of
radiation. While there is a certain amount of ambiguity in the definition of
Noether charges and their fluxes, a unique set satisfying certain physical
requirements has been identified as early as [3, 4, 5, 6, 7], and later
related to Noether charges and canonical generators in [8], see also
discussion in [9]. Bondi’s energy loss formula fits elegantly this framework
as the flux-balance law for super-translations. It has two contributions, a
‘hard’ term which is monotonic, and a ‘soft’ term which is not. The soft term
however vanishes for global translations, and this is crucial to establish
Bondi’s proof of dissipation for the total energy.

The uniqueness of the BMS charges relies on a technical assumption concerning
the choice of symplectic structure used to define the canonical generators.
There is a growing body of evidence that this assumption can be relaxed,
allowing for so called ‘corner improvements’, which are important to capture
the physics of gauge and gravitational theories in the presence of boundaries,
see e.g. [10, 11, 12, 13, 14, 15, 16, 17, 18]. This is the first technical
ingredient to our result. The second is the use of the ‘covariant shear’
introduced in [19], which is related to previous work done in [4] and to the
notion of super-translation Goldstone field of [20], which is a corner degrees
of freedom. With these two inputs, we are able to identify a unique corner-
improved symplectic structure whose canonical generators satisfy all the
properties of the standard BMS charges, plus the new property of having a
purely hard flux. The new super-momentum charge associated with this flux is
the local energy density. We conclude the paper with a brief discussion of
potential applications.

##  2 BMS flux-balance laws

We assume that the reader has a certain familiarity with null infinity, and
refer to the reviews in [21, 22, 23, 9] for the necessary background. We use
the conventions of [24], to which we refer for further details. The Ashtekar-
Streubel (AS) flux for a BMS symmetry $\xi$ between two cross-sections $S_{1}$
and $S_{2}$ of ${\mathscr{I}}$ is given by [5]

$F_{\xi}^{\scriptscriptstyle\rm
AS}=-\frac{1}{16\pi}\int_{S_{1}}^{S_{2}}N_{ab}\delta_{\xi}\sigma^{ab}\epsilon_{\mathscr{I}}.$
(2.1)

Here $\sigma_{ab}$ is the asymptotic shear, which can be assumed without loss
of generality to refer to an affine foliation of ${\mathscr{I}}$, and then the
news tensor is $N_{ab}=2\pounds_{n}\sigma_{ab}-\rho_{\langle ab\rangle}$, wth
$n$ the null tangent to ${\mathscr{I}}$, and $\rho_{\langle ab\rangle}$ the
trace-less part of Geroch’s tensor, which vanishes if the conformal frame is a
round sphere, also known as a Bondi frame. The explicit form of the BMS
transformation $\delta_{\xi}\sigma_{ab}$ can be found in the references, and
will not be needed. The foliation induces a family of Lorentz subgroups of the
BMS group, one per cross-section. Picking coordinates $(u,x^{A})$ so that
$n\,\smash{\stackrel{{\scriptstyle\scriptscriptstyle\rm{{\mathscr{I}}}}}{{=}}}\,\partial_{u}$,
we can parametrize
$\xi\,\smash{\stackrel{{\scriptstyle\scriptscriptstyle\rm{{\mathscr{I}}}}}{{=}}}\,f\partial_{u}+Y^{A}\partial_{A}$,
with $f=T+\frac{u}{2}{\mathscr{D}}Y$. Here $T=T(x^{A})$ is the super-
translation parameter, and $Y^{A}(x^{A})$ is the Lorentz parameter,
represented by a globally defined conformal Killing vector on the cross-
sections. These are topological spheres, with time-independent metric
$q_{AB}$, and we denote ${\mathscr{D}}$ its covariant derivative and ${\cal
R}$ is Ricci scalar. We can then write the charges on cross sections of
constant $u$ as

$\displaystyle Q^{\scriptscriptstyle\rm BMS}_{\xi}=\frac{1}{8\pi
G}\oint_{S}(2fM_{\rho}+Y^{A}J_{A})\epsilon_{S},\qquad
M_{\rho}:=M+\frac{1}{4}\rho_{ab}\sigma^{ab}.$ (2.2)

They correspond to Geroch’s super-momentum [3] and to the Dray-Streubel
Lorentz charge [6], as proved in [5, 7]. The aspects $M$ and $J_{A}$ can be
written in terms of tensors e.g. [3, 9] or in terms of Newman-Penrose
formalism e.g. [6, 24]. It is also possible to express them in terms of the
metric components of an asymptotic expansion in Bondi coordinates [25, 22]
(but we the caveats explained in [24]). See the Appendix for more details.

The original papers singled out the unique choices (2.1) and (2.2) by a series
of physical requirements: they are covariant; they are conserved in non-
radiative spacetimes; they vanish in Minkowski spacetime, for any $\xi$ and at
any cut. There were also technical requirements, such as a number of
derivatives compatible with second order field equations, and a choice of
canonical symplectic form associated with the Einstein-Hilbert Lagrangian. The
later Wald-Zoupas insight [8] is that they are associated to a unique
symplectic potential, given by

$\theta^{\scriptscriptstyle\rm
BMS}=-\frac{1}{16\pi}N_{ab}\delta\sigma^{ab}\epsilon_{\mathscr{I}},$ (2.3)

so that the integrand of the flux is the Noether current
$j_{\xi}=I_{\xi}\theta^{\scriptscriptstyle\rm BMS}$. The covariance of the
charges means that they are independent of the background structures given by
the choice of conformal factor and choice of foliation for the shear. The only
background structure they depend on is the symmetry vector field $\xi$, and
for the Lorentz part of the charge also the choice of cross section
determining the Lorentz subgroup of $\xi$. Covariance can be expressed in
terms of the Barnich-Troessaert bracket [25] as

$\displaystyle\\{Q_{\xi},Q_{\chi}\\}_{*}:=\delta_{\chi}Q_{\xi}-\oint
i_{\chi}j_{\xi}\,\hat{=}\,Q_{[\xi,\chi]}.$ (2.4)

This property can be explicitly checked [24], and shown to be a consequence of
the covariance of (2.3) and of the BMS boundary conditions [18]. It
generalizes the standard Hamiltonian action to dissipative situations with a
non-zero current at the cross section.

Let us now focus attention on the super-translation charges. Their flux is

$Q^{\scriptscriptstyle\rm BMS}_{T}[S_{2}]-Q^{\scriptscriptstyle\rm
BMS}_{T}[S_{1}]\,\hat{=}\,-\frac{1}{32\pi}\int\left(TN^{ab}N_{ab}+2N^{ab}({\mathscr{D}}_{a}{\mathscr{D}}_{b}+\frac{1}{2}\rho_{ab})T\right)\epsilon_{\mathscr{I}}.$
(2.5)

The first is the ‘hard term’, squared in the news. The second is the ‘soft
term’, linear in the news. It follows from the time-independence of $T$ and of
the volume form, that it corresponds to the displacement memory

${\mathbbm{M}}=-\frac{1}{8\pi}\oint
T{\mathbbm{m}}\epsilon_{S},\qquad{\mathbbm{m}}:={\mathbbm{D}}_{\rho}^{ab}({\sigma}_{ab}-\frac{u}{2}\rho_{ab})\big{|}^{S_{2}}_{S_{1}}.$
(2.6)

where we introduced the short-hand notation
${\mathbbm{D}}_{\rho}{}_{ab}:={\mathscr{D}}_{\langle
a}{\mathscr{D}}_{b\rangle}+\frac{1}{2}\rho_{\langle ab\rangle}$. When acting
on $T$, the four zero modes of this operator define global translations, which
form an ideal of the BMS algebra, and can be identified with the first four
harmonics $l=(0,1)$ in Bondi frames. It follows that the soft term vanishes
for global translations, resulting in a ‘purely hard’ flux. Furthermore, it is
strictly negative for a future-pointing global time translation: this is
Bondi’s celebrated result proving that gravitational waves dissipate energy.

Modifying the flux-balance law so that it has a purely hard flux for any $T$
is at first sight straightforward. Since the soft term is a total derivative
in $u$, it is possible to reabsorb it in the left-hand side, and consider the
charge

$Q^{\scriptscriptstyle\rm
M}_{T}=\frac{1}{4\pi}\oint_{S}T\Big{(}M_{\rho}+\frac{1}{2}{{\mathbbm{D}}_{\rho}}^{ab}(\sigma_{ab}-\frac{u}{2}\rho_{\langle
ab\rangle})\Big{)}\epsilon_{S}.$ (2.7)

If we restrict this expression to a Bondi frame, we recognize the Moreschi
mass

$M^{\scriptscriptstyle\rm
M}:=M+\frac{1}{2}{{\mathscr{D}}}_{a}{{\mathscr{D}}}_{b}\sigma^{ab},$ (2.8)

that was proposed as a super-momentum charge in [26, 27]. The problem with
this proposal is that it is manifestly non-covariant. Adding Geroch’s tensor
fixes the limitation of the original expression to Bondi frames and achieves
full conformal invariance. However it is still foliation dependent, or
equivalently $l$ dependent, via the shear and $u$. Foliation-dependence
carries an intuitive meaning of non-covariance, which can be made sharper
observing that two super-translations vector fields commute, hence a
foliation-dependent super-momentum charge would fails to capture this basic
property of the algebra. The lack of covariance can be made explicit computing
its transformation law. This gives

$\delta_{\chi}Q^{\scriptscriptstyle\rm
M}_{T}\,\hat{=}\,Q^{\scriptscriptstyle\rm
M}_{[\xi_{T},\chi]}+K_{(\xi_{T},\chi)}+{\rm radiation\ terms},$ (2.9)

where the term

$K_{(\xi_{T},\chi)}=\frac{1}{16\pi}\oint_{S}T\left(({\mathscr{D}}^{2}+{\cal
R}){\mathscr{D}}^{2}f+2{\mathscr{D}}_{a}{\cal
R}{\mathscr{D}}^{a}f-4\sigma^{ab}{\mathscr{D}}_{a}{\mathscr{D}}_{b}f\right)\epsilon_{S}.$
(2.10)

prevents the recovery of the correct symmetry action on non-radiative
spacetimes, and shows up as a field-dependent cocycle on the right-hand side
of the Barnich-Troessaert bracket. In the rest of this communication we show
how this problem can be resolved, to obtain a covariant charge with a purely
hard flux. The first step is to use the ‘covariant shear’ of [19].

##  3 Super-translation Goldstone and covariant shear

The AS radiative phase space is supplemented with the late time boundary
conditions

$\lim_{u\to\infty}N_{ab}\sim\frac{1}{u^{1+\epsilon}},\qquad\lim_{u\to\infty}\mathrm{Im}(\psi_{2})\sim\frac{1}{u^{\epsilon}},\qquad\epsilon>0,$
(3.1)

namely no radiation and a vacuum shear ${\smash{\overset{\circ}{\sigma}}}$
that is purely electric,

$\lim_{u\to\infty}\sigma_{ab}-\frac{1}{2}u\rho_{\langle
ab\rangle}={\smash{\overset{\circ}{\sigma}}}_{ab}-\frac{1}{2}u\rho_{\langle
ab\rangle}=-({{\mathscr{D}}}_{\langle
a}{{\mathscr{D}}}_{b\rangle}+\frac{1}{2}\rho_{\langle ab\rangle})u_{0}.$ (3.2)

The system thus settles down to equilibrium, and the radiative degrees of
freedom reduce to a vacuum parametrized by $u_{0}(x^{A})$, whose four-
parameter family of zero modes are the famous ‘good cuts’. The boundary
condition $u_{0}$ can be interpreted as a choice of ‘bad cut’ [24], as a
super-translation field parametrizing the vacua [19], or as Goldstone mode for
the breaking of the super-translation symmetry [20] caused by the choice of
final vacuum state ${\smash{\overset{\circ}{\sigma}}}$. Its transformation
rule induced by that of a vacuum shear is [24]

$\delta_{\xi}u_{0}=-T-u_{0}\dot{f}.$ (3.3)

We can use the boundary condition to define the relative shear

${\cal
S}_{ab}:=\sigma_{ab}-{\smash{\overset{\circ}{\sigma}}}_{ab}=\sigma_{ab}-\frac{1}{2}u\rho_{\langle
ab\rangle}+({{\mathscr{D}}}_{\langle
a}{{\mathscr{D}}}_{b\rangle}+\frac{1}{2}\rho_{\langle
ab\rangle})u_{0}\equiv-\frac{1}{2}{\cal C}_{ab},$ (3.4)

which is nothing but the ‘covariant shear’ ${\cal C}_{ab}$ introduced in [19].
The term covariant refers to the fact that it is invariant under super-
translations, hence independent of the background structure given by the $u$
foliation. This property follows from the definition, and can be verified
using (3.3). Notice that with this definition, the relative shear vanishes in
any non-radiative spacetime, since the shear has to match its own ‘bad cut’ in
the absence of radiation. The relative shear measures the same memory effect,

${\mathbbm{m}}={\mathbbm{D}}_{\rho}^{ab}{{\cal
S}}_{ab}\big{|}^{S_{2}}_{S_{1}}.$ (3.5)

It has conformal weight 1, and satisfies

$N_{ab}=2\pounds_{n}{\cal S}_{ab}.$ (3.6)

It is thus a covariant choice of ‘time potential’ for the news. Since it
differs from the one used to obtain (2.7) by a time-independent term, it is
natural to ask whether one can use the ambiguity to add time-independent terms
to the charges to achieve covariance. This cannot be done in the context of
[5, 8], since the relation to the canonical choice for the symplectic 2-form
would be lost. The BMS charges are unique, after all.

##  4 Corner improvement and new charges

The way out of the uniqueness result for the standard BMS charges is that we
will allow for corner modifications of the symplectic 2-forms. These
modifications are compatible with the field equations, and can be motivated by
situations in which stationarity and covariance, or finiteness, cannot be
achieved otherwise, see e.g. [10, 11, 12, 13, 14, 15, 16, 17, 18]. For BMS all
these properties are already satisfied with the standard symplectic 2-form,
and our reason to consider a different one comes from the additional
requirement of having a purely hard flux.

The corner term modification that we introduce is

$\bar{\Omega}:=\Omega-\delta\oint_{S}\vartheta,\qquad\vartheta=\frac{1}{8\pi}{\cal
S}_{ab}\delta({\mathbbm{D}}_{\rho}^{ab}u_{0})\epsilon_{S}=\frac{1}{8\pi}{\cal
S}_{ab}\,\delta{\smash{\overset{\circ}{\sigma}}}^{ab}\epsilon_{S}.$ (4.1)

The new symplectic structure is not defined in the Ashtekar-Streubel radiative
phase space at ${\mathscr{I}}$, but in the enlarged phase space in which we
include as late time boundary condition a choice of vacuum. It makes the
Goldstone mode $u_{0}$ canonically conjugated to the displacement memory
effect, as can be easily seen integrating on (a region of) ${\mathscr{I}}$,

$\bar{\Omega}_{\mathscr{I}}=-\frac{1}{16\pi}\int\delta N_{ab}\delta{\cal
S}^{ab}\epsilon_{\mathscr{I}}=\Omega^{\scriptscriptstyle\rm
AS}-\frac{1}{8\pi}\oint_{S}\delta{\mathbbm{M}}\delta
u_{0}\epsilon_{S},\qquad{\mathbbm{M}}:={\mathbbm{D}}_{\rho}^{ab}{\sigma}_{ab}\big{|}^{S_{2}}_{S_{1}}.$
(4.2)

To write the fluxes and charges we need on top of the symplectic 2-form also a
choice of preferred symplectic potential.111For BMS, this step can be replaced
by a choice of topology in the phase space [9]. It would be interesting to
know if this alternative exists also for the new symplectic structure. We take

$\bar{\theta}=\theta^{\scriptscriptstyle\rm
BMS}-d\vartheta=-\frac{1}{16\pi}N_{ab}\delta{\cal
S}^{ab}\epsilon_{\mathscr{I}},$ (4.3)

where in the second equality we used (3.6). This choice is manifestly
covariant and vanishes in non-radiative spacetimes, hence it satisfies the
conditions for the generalized Wald-Zoupas prescription. The associated fluxes
are

$\bar{F}_{\xi}=-\frac{1}{16\pi}\int N_{ab}\delta_{\xi}{\cal
S}^{ab}\epsilon_{\mathscr{I}}=F^{\scriptscriptstyle\rm
BMS}_{\xi}-\frac{1}{8\pi}\oint^{S_{2}}_{S_{1}}\delta_{\xi}u_{0}\,{\mathbbm{D}}_{\rho}^{ab}{\cal
S}_{ab}\,\epsilon_{S},$ (4.4)

with charges

$\bar{Q}_{\xi}=\frac{1}{8\pi}\oint_{S}[(2fM_{\rho}+f|_{u_{0}}{{\mathbbm{D}}_{\rho}}^{AB}{\cal
S}_{AB})+Y^{A}(J_{A}-{\mathbbm{D}}_{\rho}^{BC}{\cal
S}_{BC}{\mathscr{D}}_{A}u_{0})]\epsilon_{S}.$ (4.5)

Since the symplectic potential is Wald-Zoupas, the fluxes are guaranteed to be
covariant, and the charge algebra admits at most a time-independent cocycle
[18]. Since the BMS algebra is center-less [18, 24], the only cocycle can come
from an anomalous $\vartheta$ [28, 29, 30, 15]. But the choice (4.1) is
covariant, hence the new charges (4.5) satisfy (2.4). Furthermore they differ
from the standard BMS charges by a term that vanishes in any non-radiative
spacetime, hence they also share with the BMS charges the property of
vanishing in Minkowski, for any symmetry parameter, at any cut.

Let us discuss the uniqueness of our proposal. The potential ambiguity in the
charges is the addition of time-independent terms. These are related to the
freedom of adding time-independent terms to $\vartheta$, in order to keep the
Wald-Zoupas requirement that the charges match the canonical generators on
partial Cauchy slices intersecting ${\mathscr{I}}$. Requiring covariance of
the charges restricts the allowed terms to be covariant themselves. But the
only time independent fields are $u_{0}$, $\rho_{ab}$ and $q_{ab}$, and a
moment of reflection shows that it is not possible to write something
covariant. Hence any ambiguity is ruled out by the requirement to have
covariant charges. Similar considerations show that (4.3) is the only
covariant symplectic potential with a purely hard flux. We conclude that the
symplectic potential (4.2) and charges (4.5) are unique. What distinguishes
them from their standard BMS analogue is only the modification of the
symplectic structure.

The new super-momentum charge is

$\bar{Q}_{T}=\frac{1}{4\pi}\oint_{S}T\left(M_{\rho}-\frac{1}{4}{\mathbbm{D}}_{\rho}^{ab}{\cal
C}_{ab}\right)\epsilon_{S}.$ (4.6)

It is covariant, a canonical generator of super-translations with respect to
the corner-improved symplectic structure (4.2), and has a purely hard flux

$\bar{Q}_{T}[S_{2}]-\bar{Q}_{T}[S_{1}]\,\hat{=}\,-\frac{1}{32\pi}\int
TN^{ab}N_{ab}\epsilon_{\mathscr{I}}.$ (4.7)

It can be seen as a version of the Moreschi mass made covariant under
conformal transformations and super-translations thanks to $\rho_{ab}$ and
$u_{0}$. The precise relation between the two is

$\bar{M}=M^{\scriptscriptstyle\rm
M}+\frac{1}{8\pi}\oint_{S}T\left(\rho^{ab}{\sigma}_{ab}-\frac{u}{2}{\mathbbm{D}}_{\rho}^{ab}\rho_{ab}+{\mathbbm{D}}_{\rho}^{2}u_{0}\right)\epsilon_{S}.$
(4.8)

On round spheres, the required extra term for foliation-independence reduces
to $\oint T({\mathscr{D}}^{2}+2){\mathscr{D}}^{2}u_{0}$, and vanishes for
global translations.

##  5 Applications

With the standard BMS charges, only the total energy has a monotonic flux.
This corresponds to the lowest mode of the super-translation parameter, or
$T=1$ on a Bondi frame. Consider instead a localized super-momentum charge,
corresponding for instance to a parameter $T$ with a Gaussian-like profile
peaked on some point on the celestial sphere. This profile necessarily
involves higher modes whose flux is not monotonic, hence the charge cannot be
interpreted as a localized energy, because it may be increasing even though
the system is dissipating. Indeed, the standard interpretation of the higher
super-momentum modes is not related to the energy, but rather to the notion of
mass multipoles. Their flux is not monotonic, and corresponds to the memory
effect.

The new charge (4.6) has a monotonic flux for any positive super-translation
parameter. It follows that we can take any peaked Gaussian-like function $T$,
and extend Bondi’s result about dissipation from the total energy to a local
energy! We stress that this construction does not violate the principle of
general covariance, because some aspects of non-locality do remain. We have
been able to define a local energy density at ${\mathscr{I}}$, using the
background structure there at disposal, and its flux is local in the news
tensors. But the news tensor itself is a _non-local_ function of the Weyl
curvature, in both space and time. E.g. in Newman-Penrose formalism,

$\psi_{3}=\eth N,\qquad\psi_{4}=\dot{N}.$ (5.1)

Therefore the flux of the local energy still requires non-local knowledge of
the curvature. For this reason, our construction does not achieve a full
localization, hence the title.

A monotonic flux for all future-pointing super-translations has further
potential applications. Let us mention two. The first comes from the work of
Wall [31]. His proof of the generalized second law requires the axioms of
ultra-locality and stability. Ultra-locality means that the null geodesics
spanning a null hypersurface can be treated as independent subsystems, and
stability implies that the Hamiltonian on each subsystem has positive
eigenvalues. Identifying the Hamiltonian density with the super-translation
Noether current $j_{T}$, we can state the axioms as

$\int_{\mathcal{N}}j_{T}\epsilon_{\mathcal{N}}\geq 0,$ (5.2)

for any $T>0$, which includes any approximation of a delta function localized
at a point. In Wall’s work the null hypersurface ${\cal N}$ is an horizon, for
which a monotonic flux has been studied in e.g. [32, 33, 34, 35, 36]. Our work
shows that the axioms can apply to ${\mathscr{I}}$ as well, if the right
symplectic structure is chosen. This is in line with the interpretation of
${\mathscr{I}}$ as an horizon (in the conformal spacetime [37, 9]), and can be
relevant for studying the generalized second law at ${\mathscr{I}}$ (e.g. [38,
39]).

A second application comes from soft theorems. It was pointed out that the
soft terms in the flux are related to infra-red divergences [40], see also
[41, 17], hence a covariant remotion of the soft flux may lead to infra-red
finiteness. We also remark that our improved symplectic structure appears to
be consistent with the corner bracket posited in [42]. However, we stress that
our proposal is valid for BMS, and not for larger symmetry groups such as eBMS
[43, 44, 45], gBMS [46, 47] or BMSW [30]. The reason for it is that the proof
of covariance relies on the universality of Geroch’s tensor, used explicitly
in the second equality of (4.1), and which is lost for the larger symmetry
groups. Covariant BMS symplectic potentials for eBMS and (a group isomorphic
to) gBMS have been recently found [13, 17, 18, 24], but do not have the
property of being purely hard.

##  6 Conclusions

The new super-momentum charge has the beautiful property that it provides a
notion of localized energy density, namely a quantity that can be defined on
arbitrarily small regions of the celestial sphere, and whose behaviour in
radiative spacetimes is monotonically decreasing. This property gives it a
compelling operational definition. Consider an observer with a container
filled of water and a very precise thermometer. When gravitational waves reach
the observer they will heat the water. By what amount? Bondi’s formula only
gives us the total amount of energy released, proportional to the integral of
the squared news on the whole celestial sphere. But this is hardly relevant to
the container, which is a sharply localized object on the celestial sphere.
Intuitively, the relevant flux should depend only on the news at the specific
direction $(\theta,\varphi)$ of the observer. As any function on the sphere
$T(x^{A})$ is an asymptotic symmetry of ${\mathscr{I}}$, we can choose one
that is sharply peaked at the given direction. Let us idealize it with a Dirac
delta, $T=\delta^{(2)}(\theta,\phi)$. Then the increase of water temperature
$T_{w}$ is

$\displaystyle\Delta T_{w}=\frac{\alpha_{w}}{32\pi
G\rho_{w}c_{w}}\int_{u_{1}}^{u_{2}}N^{2}_{ab}(u,\theta,\phi)du,$ (6.1)

where $\alpha_{w}$ is a conversion factor measuring the fraction of energy
that can be extracted from the radiation and converted into heat in the water
by unit of volume, $\rho_{w}$ is the water density and $c_{w}$ is the thermal
mass capacity. The question is then whether a local experiment can verify this
formula, since as remarked earlier, the news depends non-locally on the
curvature, via (5.1). We claim that this is possible, as long as $u_{1}$ was
in the non-radiative regime. With this assumption,

$N(u,\theta,\phi)=\int^{u}_{u_{1}}\psi_{4}(u^{\prime},\theta,\phi)du^{\prime}.$
(6.2)

Hence a _spatially local_ measurement of curvature is enough to determine the
energy dissipation (6.1). The only left-over non-locality is that the
characterization of a non-radiative cross-section is still requiring
$\psi_{3}$ on the whole sphere. The non-locality can be thus confined to the
initial conditions for the experiment.

The localized charge (4.6) associated to this flux meets various criteria in
order to be interpreted as an energy. First, it is the Noether charge and
canonical generator for the associated super-translation symmetry. Second, it
is always positive, thanks to the negativity of the flux, if we assume that
the final state has a positive Bondi mass aspect, as for a stationary black
hole. Third, it is extensive. This follows from the fact that it is
functionally linear in $T$. Extensivity may come as a surprise, since the
theory is non-linear and it is not possible to screen gravitational
interaction. But here we are talking about extensivity at ${\mathscr{I}}$,
where different regions on the celestial sphere are infinitely far away and
_causally disconnected_ , hence extensivity is a natural property.222A similar
construction could be done also on local null hypersurfaces if one uses a
monotonic flux, however only within domains free of caustics or crossings, as
to guarantee causal disconnection. Otherwise extensivity would no longer be
possible. Super-translations are in fact not even admissable symmetries of
local null surfaces [48].

Since the asymptotic observers cannot synchronize their clock, their notions
of future directed time translations are independent and equivalent. The
canonical generator associated to these symmetries should then capture this
property, and this is what our notion of local energy density does. The local
version of the Bondi energy loss formula reflects the basic fact that there is
no universal notion of energy in general relativity. Therefore, the new
charges that we constructed have all the properties of localized energy. If
this energy transfer is observable, it is therefore a good argument in favor
of the new symplectic stuctrure and of the introduction of the edge mode
$u_{0}$ in the phase space, as it seems impossible to obtained a covariant and
stationary symplectic flux and charge without it.

### Acknowledgements

We thank Abhay Ashtekar and Jurek Lewandowski for comments.

## Charge aspects

The charges can be written without reference to any coordinate system on
${\mathscr{I}}$ if we introduce an auxiliary null vector $l$ such that $n\cdot
l=-1$. This vector can be used to identify the shear and Lorentz subgroups,
and to that end it is convenient to take it hypersurface orthogonal and Lie
dragged by $n$, so that it is adapted to leafs of an affine foliation. The
pair $(n,l)$ can be completed to a Newman-Penrose basis at ${\mathscr{I}}$
with a complex dyad $(m,\bar{m})$. The charge aspects can then be written
using the Newman-Penrose formalism as

$M_{\rho}=-\mathrm{Re}(\psi_{2}-\sigma N),\qquad
m^{A}J_{A}=-\left(\psi_{1}+\sigma\eth\bar{\sigma}+\frac{1}{2}\eth(\sigma\bar{\sigma})\right),$

where $\sigma=-m^{a}m^{b}\sigma_{ab}$ and
$N=\frac{1}{2}\bar{m}^{a}\bar{m}^{b}N_{ab}$. This is the form in which they
appear for instance in [6]. Fixing $l$ adapted to the cross-section is not
necessary for the super-momentum, but it is for the Lorentz charge, and it is
ultimately related to the fact that there is no preferred Lorentz subgroup of
the BMS group in radiative spacetimes, one can speak of a Lorentz group only
in relation to a chosen cross section.

The aspects can also be related to the metric coefficients of an asymptotic
expansion of the bulk metric. To that end, we introduce Bondi coordinates
$(u,r,x^{A})$, and the expansion

$\displaystyle g_{uu}$ $\displaystyle=-\frac{{\cal
R}}{2}+\frac{2M}{r}+O(r^{-2}),\qquad
g_{ur}=-1-\frac{2\beta}{r^{2}}+O(r^{-3}),\qquad\beta:=-\frac{1}{32}C^{AB}C_{AB},$
$\displaystyle g_{uA}$
$\displaystyle=-U_{A}+\frac{2}{3r}(J_{A}+\partial_{A}\beta-\frac{1}{2}C_{AB}U^{B})+O(r^{-2}),\qquad
U_{A}:=-\frac{1}{2}{\mathscr{D}}^{B}C_{AB},$ $\displaystyle g_{AB}$
$\displaystyle=r^{2}q_{AB}+rC_{AB}+O(1).$

The bulk Bondi coordinates induce affine coordinates $(u,x^{A})$ on
${\mathscr{I}}$ such that
$n\,\,\smash{\stackrel{{\scriptstyle\scriptscriptstyle\rm{{\mathscr{I}}}}}{{=}}}\,\,\partial_{u}$,
and an affine foliation given by the level sets of $u$, whose shear is
$\sigma_{ab}=-1/2\delta_{a}^{A}\delta_{b}^{B}C_{AB}$, and the aspects $M$ and
$J_{A}$ appear at first order in the conformal factor $\Omega:=1/r$.

## References

  * [1] H. Bondi, M. G. J. van der Burg and A. W. K. Metzner, Gravitational waves in general relativity. 7. Waves from axisymmetric isolated systems, Proc. Roy. Soc. Lond. A 269 (1962) 21–52.
  * [2] R. Sachs, Asymptotic symmetries in gravitational theory, Phys. Rev. 128 (1962) 2851–2864.
  * [3] R. Geroch, Asymptotic Structure of Space-Time, ch. 1, pp. 1–106.  Springer US, Boston, MA, 1977.
  * [4] A. Ashtekar, Radiative Degrees of Freedom of the Gravitational Field in Exact General Relativity, J. Math. Phys. 22 (1981) 2885–2895.
  * [5] A. Ashtekar and M. Streubel, Symplectic Geometry of Radiative Modes and Conserved Quantities at Null Infinity, Proc. Roy. Soc. Lond. A 376 (1981) 585–607.
  * [6] T. Dray and M. Streubel, Angular momentum at null infinity, Class. Quant. Grav. 1 (1984), no. 1 15–26.
  * [7] T. Dray, Momentum Flux At Null Infinity, Class. Quant. Grav. 2 (1985) L7–L10.
  * [8] R. M. Wald and A. Zoupas, A General definition of ’conserved quantities’ in general relativity and other theories of gravity, Phys. Rev. D 61 (2000) 084027 [gr-qc/9911095].
  * [9] A. Ashtekar and S. Speziale, Null Infinity as a Weakly Isolated Horizon, 2402.17977.
  * [10] G. Compere and D. Marolf, Setting the boundary free in AdS/CFT, Class. Quant. Grav. 25 (2008) 195014 [0805.1902].
  * [11] D. Harlow and J.-Q. Wu, Covariant phase space with boundaries, JHEP 10 (2020) 146 [1906.08616].
  * [12] L. Freidel, M. Geiller and D. Pranzetti, Edge modes of gravity. Part I. Corner potentials and charges, JHEP 11 (2020) 026 [2006.12527].
  * [13] M. Campiglia and J. Peraza, Generalized BMS charge algebra, Phys. Rev. D 101 (2020), no. 10 104039 [2002.06691].
  * [14] G. Odak and S. Speziale, Brown-York charges with mixed boundary conditions, JHEP 11 (2021) 224 [2109.02883].
  * [15] V. Chandrasekaran, E. E. Flanagan, I. Shehzad and A. J. Speranza, A general framework for gravitational charges and holographic renormalization, Int. J. Mod. Phys. A 37 (2022), no. 17 2250105 [2111.11974].
  * [16] G. Odak, A. Rignon-Bret and S. Speziale, Wald-Zoupas prescription with soft anomalies, Phys. Rev. D 107 (2023), no. 8 084028 [2212.07947].
  * [17] L. Donnay, K. Nguyen and R. Ruzziconi, Loop-corrected subleading soft theorem and the celestial stress tensor, JHEP 09 (2022) 063 [2205.11477].
  * [18] A. Rignon-Bret and S. Speziale, Covariance and symmetry algebras, 2403.00730.
  * [19] G. Compère, A. Fiorucci and R. Ruzziconi, Superboost transitions, refraction memory and super-Lorentz charge algebra, JHEP 11 (2018) 200 [1810.00377]. [Erratum: JHEP 04, 172 (2020)].
  * [20] A. Strominger, On BMS Invariance of Gravitational Scattering, JHEP 07 (2014) 152 [1312.2229].
  * [21] A. Ashtekar, Geometry and Physics of Null Infinity, 1409.1800.
  * [22] E. E. Flanagan and D. A. Nichols, Conserved charges of the extended Bondi-Metzner-Sachs algebra, Phys. Rev. D 95 (2017), no. 4 044002 [1510.03386].
  * [23] A. M. Grant, K. Prabhu and I. Shehzad, The Wald-Zoupas prescription for asymptotic charges at null infinity in general relativity, Class. Quant. Grav. 39 (2022), no. 8 085002 [2105.05919].
  * [24] A. Rignon-Bret and S. Speziale, Center-less BMS charge algebra, 2405.01526.
  * [25] G. Barnich and C. Troessaert, BMS charge algebra, JHEP 12 (2011) 105 [1106.0213].
  * [26] O. M. Moreschi and S. Dain, Rest frame system for asymptotically flat space-times, J. Math. Phys. 39 (1998) 6631–6650 [gr-qc/0203075].
  * [27] S. Dain and O. M. Moreschi, General existence proof for rest frame systems in asymptotically flat space-time, Class. Quant. Grav. 17 (2000) 3663–3672 [gr-qc/0203048].
  * [28] V. Chandrasekaran and A. J. Speranza, Anomalies in gravitational charge algebras of null boundaries and black hole entropy, JHEP 01 (2021) 137 [2009.10739].
  * [29] L. Freidel, R. Oliveri, D. Pranzetti and S. Speziale, Extended corner symmetry, charge bracket and Einstein’s equations, JHEP 09 (2021) 083 [2104.12881].
  * [30] L. Freidel, R. Oliveri, D. Pranzetti and S. Speziale, The Weyl BMS group and Einstein’s equations, JHEP 07 (2021) 170 [2104.05793].
  * [31] A. C. Wall, A proof of the generalized second law for rapidly changing fields and arbitrary horizon slices, Phys. Rev. D 85 (2012) 104049 [1105.3445]. [Erratum: Phys.Rev.D 87, 069904 (2013)].
  * [32] A. Ashtekar, N. Khera, M. Kolanowski and J. Lewandowski, Charges and fluxes on (perturbed) non-expanding horizons, JHEP 02 (2022) 066 [2112.05608].
  * [33] A. Rignon-Bret, Second law from the Noether current on null hypersurfaces, Phys. Rev. D 108 (2023), no. 4 044069 [2303.07262].
  * [34] G. Odak, A. Rignon-Bret and S. Speziale, General gravitational charges on null hypersurfaces, JHEP 12 (2023) 038 [2309.03854].
  * [35] L. Ciambelli, L. Freidel and R. G. Leigh, Null Raychaudhuri: canonical structure and the dressing time, JHEP 01 (2024) 166 [2309.03932].
  * [36] S. Hollands, R. M. Wald and V. G. Zhang, The Entropy of Dynamical Black Holes, 2402.00818.
  * [37] A. Ashtekar and S. Speziale, Horizons and Null Infinity: A Fugue in 4 voices, 2401.15618.
  * [38] R. Bousso, Asymptotic Entropy Bounds, Phys. Rev. D 94 (2016), no. 2 024018 [1606.02297].
  * [39] T. Faulkner and A. J. Speranza, Gravitational algebras and the generalized second law, 2405.00847.
  * [40] N. Arkani-Hamed, M. Pate, A.-M. Raclariu and A. Strominger, Celestial amplitudes from UV to IR, JHEP 08 (2021) 062 [2012.04208].
  * [41] M. Campiglia and A. Laddha, BMS Algebra, Double Soft Theorems, and All That, 2106.14717.
  * [42] T. He, V. Lysov, P. Mitra and A. Strominger, BMS supertranslations and Weinberg’s soft graviton theorem, JHEP 05 (2015) 151 [1401.7026].
  * [43] G. Barnich and C. Troessaert, Symmetries of asymptotically flat 4 dimensional spacetimes at null infinity revisited, Phys. Rev. Lett. 105 (2010) 111103 [0909.2617].
  * [44] G. Barnich and C. Troessaert, Aspects of the BMS/CFT correspondence, JHEP 05 (2010) 062 [1001.1541].
  * [45] S. Pasterski, A. Strominger and A. Zhiboedov, New Gravitational Memories, JHEP 12 (2016) 053 [1502.06120].
  * [46] M. Campiglia and A. Laddha, Asymptotic symmetries and subleading soft graviton theorem, Phys. Rev. D 90 (2014), no. 12 124028 [1408.2228].
  * [47] M. Campiglia and A. Laddha, New symmetries for the Gravitational S-matrix, JHEP 04 (2015) 076 [1502.02318].
  * [48] V. Chandrasekaran, E. E. Flanagan and K. Prabhu, Symmetries and charges of general relativity at null boundaries, JHEP 11 (2018) 125 [1807.11499].


11institutetext: NYU Multimedia and Visual Computing Lab
NYU Tandon School of Engineering, New York University, USA
New York University Abu Dhabi, UAE
NYU Langone Health, USA
11email: {yh3252, jf4151, yw523<EMAIL_ADDRESS>
11email<EMAIL_ADDRESS>
# Detect and Approach: Close-Range Navigation Support for People with
Blindness and Low Vision

Yu Hao 1122 Junchi Feng 22 John-Ross Rizzo 2244 Yao Wang 22 Yi Fang
11223^(✉)3^(✉)

###### Abstract

People with blindness and low vision (pBLV) experience significant challenges
when locating final destinations or targeting specific objects in unfamiliar
environments. Furthermore, besides initially locating and orienting oneself to
a target object, approaching the final target from one’s present position is
often frustrating and challenging, especially when one drifts away from the
initial planned path to avoid obstacles. In this paper, we develop a novel
wearable navigation solution to provide real-time guidance for a user to
approach a target object of interest efficiently and effectively in unfamiliar
environments. Our system contains two key visual computing functions: initial
target object localization in 3D and continuous estimation of the user’s
trajectory, both based on the 2D video captured by a low-cost monocular camera
mounted on in front of the chest of the user. These functions enable the
system to suggest an initial navigation path, continuously update the path as
the user moves, and offer timely recommendation about the correction of the
user’s path. Our experiments demonstrate that our system is able to operate
with an error of less than 0.5 meter both outdoor and indoor. The system is
entirely vision-based and does not need other sensors for navigation, and the
computation can be run with the Jetson processor in the wearable system to
facilitate real-time navigation assistance.

###### Keywords:

Assistive technology, Object localization from video, Navigation

##  1 Introduction

According to 2020 WHO estimates, 295 million people suffer from moderate to
severe visual impairment, while 43.3 million people are presently blind [22].
Globally, between 1990 to 2020, the number of moderate to severely visually
impaired increased by 91.7%, and the number of people who were blind increased
by 50.6% [8]. This trend is predicted to continue with estimates approaching
474 million people with moderate to severe visual impairment and 61 million
people with blindness by 2050 [21]. Blindness and low vision poses significant
challenges for nearly every activities of daily living [17]. One critical task
element of most activities in daily living is visual search or a goal-oriented
activity that involves the active scanning of the environment to locate a
particular target among irrelevant distractors [28]. Performing visual search
can be demanding in complex environments, even for those with normal vision.
It is even more challenging for the pBLV [15]. For people with moderate to
severe peripheral vision loss, central vision loss, and hemi-field vision
loss, due to reductions in the field of view, most have difficulty in
isolating a particular location when searching for an object of interest and
may need help in locating the object. For people experiencing blurred vision
or nearsightness, they may have difficulty in identifying object at relatively
far distances. For people with color deficient vision and low contrast vision,
it may be difficult for them to distinguish objects from background when the
object and background share similar colors. Aside from isolating the
particular location of an object, closing the distance between one’s current
position and the object itself is also a challenge. pBLV often want more than
just information about the initial location of the object relative to their
current position, but also continuous help in navigating to the object along
the way [4].

Figure 1: Challenges for pBLV exist in various scenarios: Searching for
objects of interests and walking to the objects (Left). Our wearable system
contains a backpack with a monocular camera, an Nvidia Jetson Xavier NX
Developer kit, and battery. The camera is placed on the chest of the user
(Right).  Figure 2: Our system is able to detect and locate an object of
interest and guide a person with BLV to the target object. Initially, an
object detection module will detect all possible interesting objects. Once the
person selects a target object of interest, the object localization module
will provide the 3D location of the object and plan the path for the person to
reach the object. The trajectory estimation module will then continuously
estimate the person’s movement between two time points, update the object
location (relative to the user), and send path correction feedback to the user
when necessary.

Solutions to aid this enormous and ever-growing problem of blindness and low
vision are desperately needed. In the context of navigation and overcoming the
close-range challenge, assistive technologies may help close the gap, and aid
pBLV attain functional independence with better quality of life [18]. However,
for the pBLV, only a limited number of tools have modest market traction and
very few, if any, are able to support precise interaction with objects of
interest in the surrounding environment.

Many of the present mobile apps for way-finding have decent success at leading
end users to a general vicinity of a target location but few are able to
precisely provide instructions as one approaches the target. As most apps are
focused on outdoor use and are predicated on GPS technology, they lack the
accuracy required to support close-range navigation, which is necessary for
pBLV to approach their final destination. Adding insult to injury, as most
pBLV live in metropolitan environments, the accuracy of GPS-enabled
smartphones reduces from a 4.9 meters radius under the open sky to a 20 meters
radius in an urban setting, which is insufficient for reaching exact location
and/or specific objects of interest [6].

In this work, we develop a new wearable navigation solution to augment
perceptive ability for pBLV. The system will help a user to locate a target
object of interest and provide guidance to reach the target efficiently and
effectively in unfamiliar environments. Our wearable system, as shown in
Figure 1, contains a backpack with a monocular camera, an Nvidia Jetson Xavier
NX Developer kit, and a battery. The camera is placed in front of the chest of
the user in a custom scaffold that can be mounted on the shoulder strap of a
backpack housing the Jetson board and the battery. With the sequence of images
captured by the camera, our system is be able to detect the target object and
provide real-time path planing and updated guidance to the end user as the
user approaches the target object, with an accuracy of less than 0.5 meter.

The visual processing part of our system contains three main modules, as
illustrated in Figure 2: object detection, object localization, and trajectory
estimation. The object detection module is implemented by a pretrained YOLOv5
detection model [26], which is responsible to detect all possible objects of
interests. After the user selects an object as the target, the object
localization module will provide the initial 3D coordinate of the target
object and suggest an initial path for navigation from the user’s current
location to the target (e.g. the first purple path in the figure). The
trajectory estimation module will then continuously estimate the movement of
the user (or more precisely the camera) and consequently update the desired
path to the target (the second purple path in the figure). If the angle
between the updated path and the estimated user’s path (the yellow path) is
higher than a pre-defined threshold, our system will send an alert message to
the user. In this example, the system may say “Please head towards your left
slightly by about 30 degrees”.

To reduce the system cost and computational load, we only use a deep-learning
model for object detection in the first frame. We estimate the initial 3D
coordinates of the target object using the corresponding 2D locations of the
object in two initial frames as shown in Figure 3, to alleviate the need for a
stereo camera for depth sensing. Given the initial position of the target
object, we estimate the camera motion between successive frames to determine
the trajectory of the user, and update the object position relative to the
user for continuous path updating as shown in Figure 3.

Our experiments demonstrate that our system is able to detect objects of
interests and provide real-time update of the object location relative to the
user as the user moves towards the object, with an error of less than 0.5
meter. The system is entirely vision-based and does not need other sensors for
navigation (e.g. IMUs and range sensors), and the computation can be run with
the Jetson processor in the wearable system to facilitate real-time navigation
assistance.

##  2 Related Works

Considering the growing prevalence of smartphones in general and in pBLV
population [12] [14] [7], mobile applications can be a potential solution to
address the needs of localization and navigation for the pBLV [25]. The All
Aboard [10], developed by Massachusetts Eye and Ear, utilizes computer vision
to detect bus stop sign in the vicinity of the users and guide the users to
the precise location of the bus stop by providing the distance estimations of
the bus stop sign using computer vision algorithms. The drawback of this app
is that it only detect the bus stop sign.

Figure 3: The processing flow for the target object localization module (top)
and trajectory estimation module (bottom).

Another example of the computer vision-based app for the pBLV is Virtual Touch
[12]. This app utilizes the smartphone’s camera to capture the surrounding
environment of the user, and detects objects of interest in the scene. This
app enables the users to interact with the environment by pointing their
fingers to an object of interest and the app will tell the users what object
they are pointing to. However, this app lacks the abilities of distance
estimation and navigation.

Another category of assistive technology for navigation is light-based indoor
positioning [16]. This technology requires illuminating equipment such as LED
light to illuminate the environment and transmit infrared signals at the same
time. The user holds a receiver such as the smartphone to receive and decode
the light signals. By calculating the angles of the received signals, it is
possible to accurately localize the user and provide navigation for indoor
locations where GPS doesn’t work well. However, the cost of this system can be
a concern as it requires significant efforts in establishing the qualified
illuminating infrastructure.

On the other hand, our system combines object detection and trajectory
estimation features to provide all the functions necessary for detecting and
navigating to a target object using a monocular camera only. It can detect
different types of target, whereas the All Aboard app can only detect a
specific type of object. Our system augments the pBLV’s perceptive ability
more than the Virtual Touch app because our system also provides real-time
path correction function to guide users to the target object. Moreover, our
system is wearable and all the processing can be done locally, and hence can
provide navigation without the need of new infrastructure or other sensors
except a monocular camera.

Our methodology for object localization and trajectory estimation from 2D
video is similar to that used for visual SLAM (simultaneous localization and
mapping) [24] [19], typically used to track the camera pose of a field robot
and map the sounding environment relative to the robot. Here we use visual
SLAM to estimate the movement of a user wearing a camera and the location of a
particular stationary object relative to the user. Therefore, the proposed
system is an innovative integration of object detection and visual SLAM for
assisting pBLV in detecting and approaching a target object.

##  3 Methods

As shown in Figure 2, our system consists of three main visual processing
modules: object detection, object localization and trajectory estimation. We
use a pretrained YOLOv5 model to detect all possible interesting objects on
the first frame. After a user selects an object as the target object, the
object localization module will determine the 3D coordinate of the object
relative to the user (more precisely the camera center) based on first two
frames of the video. This module is applied only in the first two frames at
the start of the navigation. Then the trajectory estimation module will
continuously estimate the movement of the user between frames and consequently
update the location of the target relative to the user’s current position.
Based on the updated object location, the system may provide path correction
suggestions to the user. There are two options for selecting target object
from all detected objects by the object detection module: 1) Use audio play
all detected objects to the user and user uses the microphone to select the
target object by existing audio to text API. 2) Use Virtual Touch [12] to
interact with the environment by pointing their fingers to the target object.
The details of the object localization and trajectory estimation modules are
described in the following subsections. Because the object localization module
makes use of the feature correspondence and camera motion estimation
approaches used for trajectory estimation, we will first describe the
trajectory estimation module in Sec. 3.1, and then present the object
localization module in Sec. 3.2.

###  3.1 Trajectory Estimation

In this section, we introduce the trajectory estimation module, which aims to
determine the movement of the user between two video frames with a chosen
frame interval.111The video is typically captured between 15 to 30 frames per
second. But this processing may be done at a slower speed, e.g. every 0.5 to 1
second. We make use of the fact that the camera is mounted in front of the
user’s chest and therefore the camera’s movement is a good proxy for the
user’s movement. We adopt a classical approach for determining the rotation
and translation of the camera between two camera views based on the
correspondence of selected features points. The user’s movement between the
two frames is assumed to be equal to the estimated camera translation.

Our trajectory estimation module includes three components: feature point
detection and feature descriptor extraction; feature point matching; and
camera motion estimation based on the epipolar constraint, as shown in Figure
3 (bottom). The following subsections describe these components.

####  3.1.1 Feature Descriptor Extraction and Matching

In recent years, many local feature detectors and descriptors, such as SIFT
[13], SURF [2] and ORB[27], have been developed and used for object
recognition, image registration, classification, or 3D reconstruction. To
enable real-time navigation assistance, we chose the ORB feature descriptor
[27], which are oriented multi-scale FAST [1] corners with a 256-bits
descriptor associated. There are two main advantages of ORB: 1) ORB uses an
orientation compensation mechanism, making it rotation invariant; 2) ORB
learns the optimal sampling pairs, whereas other descriptors like BRIEF [3]
uses randomly chosen sampling pairs. These strategies boost the accuracy and
efficiency of feature detection and matching.

Based on the feature descriptors, we establish the correspondences between the
features in the current frame and the reference frame of the same scene. We
first use a brute force matching algorithm to calculate the similarity between
all descriptors in the current frame and all descriptors in the reference
frame and determine an initial set of pairs of 2D coordinates of corresponding
features. RANdom SAmple Consensus (RANSAC) [5] algorithm is then utilized to
exclude the matching outliers and furthermore estimate the essential matrix
that best describes the geometric relation between corresponding 2D
coordinates, to be introduced in the next subsection.

####  3.1.2 Determining the camera rotation and translation

When a monocular camera views a 3D scene from two distinct positions and
orientations, there are a number of geometric constraints between the
projections of the same 3D points onto the 2D images [9]. Let $p$ and $q$
denote the homogeneous coordinates of the 2D projections of the same 3D point
$P$ in the reference and the current frame. They are related by the
Longuet–Higgins equation [9]:

$q^{t}Ep=0$ (1)

where the matrix $E$ is known as the essential matrix, which depends on the
camera rotation and translation between the two frames and the camera’s
intrinsic parameters. As described previously, we can use RANSAC to determine
the best $E$ matrix given the set of corresponding features points in the two
frames.

It is well-known [20] that we can use singular value decomposition (SVD) of
the essential matrix $E$ to determine the camera rotation matrix $R$ and
translation vector $t$. Specifically, we use SVD to obtain matrix $U$ and $V$
so that:

$E=UDV^{T}$ (2)

The rotation $R$ and translation $t$ can be computed from $U$ and $V$ as:

$R=UWV^{T},\;\;t=U_{3},\;\;{\rm with}\;\;W=\left[\begin{array}[]{ccc}0&-1&0\\\
1&0&0\\\ 0&0&1\\\ \end{array}\right]$ (3)

The results are algebraically correct also with $-t$ and $W^{T}$, so we try
all possible solutions on the matching descriptors to choose the $R$ and $t$
that leads to the least fitting error for Eq. (1). For implementation, we use
the openCV library function [23] to calculate the essential matrix and camera
rotation and translation.

Once the camera translation $t$ is determined, we update the target object
location by the estimated camera translation, i.e., $o^{\prime}=o-t$, where
$o$ is the object location in the reference frame and $o^{\prime}$ is its
location in the current frame, relative to the camera center and hence the
user. The straight line connecting the object location $o^{\prime}$ in the
ground plane (i.e. the $X$ and $Z$ coordinate) and the user is the updated
path.222Here we assume that there is an open space between the target and the
user for simplicity. In practice, more sophisticated algorithms that detect
obstacles between the target and the user and plan the path accordingly are
needed. In this work, we focus on the visual processing components. On the
other hand, the camera translation $t$ indicates the direction of the user’s
latest movement between the current frame and the last frame. We evaluate the
angle between $t$ and $o^{\prime}$. If the angle is larger than a pre-defined
threshold, our system will sent out a friendly alert message to the user.

###  3.2 Object Localization

In this section, we introduce our object localization module, which aims to
determine the 3D coordinate of the target object at the start of the
navigation. Given that we only have a monocular camera, one potential option
is to use a deep-learning model for determining the depth from 2D images. This
is however computationally demanding. Instead, we take advantage of the fact
that we have a video sequence captured while the user is moving, and use the
two adjacent video frames to determine the object location. Specifically, we
first determine the 2D coordinates of the object center in the two initial
frames and the camera motion between the two frames. We then determine the 3D
coordinate of the object center through a triangulation algorithm, as
illustrated in Figure 3 (top).

We use the same algorithm described in Sec. 3.1 to determine the corresponding
features in the first two frames and the camera motion (rotation and
translation) between the two frames, except that, for the first frame, we only
perform feature extraction within the bounding box of the detected object. We
use the centroid of the 2D coordinates of all the feature points in the object
region in the first frame, as the coordinate of the object center in the first
frame, denoted by $p$. Similarly, we determine the object center coordinate in
the second frame, denoted by $q$, using feature points that correspond to the
features belonging to the object in the first frame.

Given the camera rotation $R$ and translation $t$ and the 2D positions of the
object center, $p$ and $q$, in their homogeneous representations, we utilize
triangulation [9] to obtain the 3D coordinate of the object center $P$ (in the
homogeneous representation) with respect to the camera center in the first
frame. Specifically, given the camera pose $R$ and $t$, we compute the
projection matrix $J_{1}$ for the first frame and $J_{2}$ for the second
frame:

$J_{1}=K\cdot[I,0],\;\;J_{2}=K\cdot[R,t]$ (4)

where $K$ is the intrinsic matrix of camera and $I$ is the identity matrix.
Since the cross-product between two parallel vectors equals to zero, we have:

$\displaystyle p\times(J_{1}P)=0,\;\;q\times(J_{2}P)=0$ (5)

where $p=(u_{1},v_{1},1)$ and $q=(u_{2},v_{2},1)$. This equation can also be
written as follows:

$\displaystyle\left(\begin{array}[]{c}u_{1}J_{1}^{3}-J_{1}^{1}\\\
v_{1}J_{1}^{3}-J_{1}^{2}\\\ u_{2}J_{2}^{3}-J_{2}^{1}\\\
v_{2}J_{2}^{3}-J_{2}^{2}\\\ \end{array}\right)\cdot P=A\cdot P=0$ (10)

Then we apply SVD on $A$ to obtain $C,S,$ and $D$ so that

$A=CSD^{T}$ (11)

The third column of matrix $D$ is $P$:

$P=(X,Y,Z,W)=D_{3}$ (12)

Finally, we can transform the homogeneous coordinate to the Cartesian
coordinate using

${\tilde{P}}=(X/W,Y/W,Z/W)$ (13)

##  4 Experiments

We carried out a set of experiments to evaluate the performance of our
proposed system. We first use the KITTI odometry data to evaluate our system,
where the video sequences are captured by a moving vehicle. We also run an
experiment simulating a user walking towards a target object in an indoor
environment and evaluate the performance of our algorithms. We describe these
two experiments and their results separately.

Figure 4: Example results of object detection and object localization for the
KITTI dataset. Yellow rectangles denote the bounding box of the detected
objects (car and motorcycle). We also show the estimated location $(X,Z)$ of
the detected objects. Since we are only interested in the ground position of
the objects, we only show the $X$ and $Z$ coordinate for visualization.

###  4.1 Experiment with the KITTI dataset

####  4.1.1 KITTI Dataset:

The odometry benchmark from the KITTI dataset contains 11 sequences from a car
driven around a residential area with accurate ground truth from GPS and a
Velodyne laser scanner. We choose the car, the motorbike, the pedestrian and
the traffic light as possible target objects. We extract four video sequences
from the KITTI dataset each containing a target object. Specifically, for
video 1, we use frames 3-18 in Sequence 06 of KITTI visual odometry dataset
and select the car as the target object. For video 2, we use frames 2360-2370
from sequence 08 and select the motorcycle as the target object. For video 3,
we use frames 3416-3431 in Sequence 08 and select the person as the target.
For video 4, we use frames 3970-3980 in Sequence 08 and select the traffic
light as the target.

To generate the ground truth for object location, we use the corresponding 3D
scan of velodyne laser data in each frame for reference. Specifically, we
annotate a 3D bounding box for each object of interest and calculate the
centroid of the 3D coordinates of all points in the bounding box as the ground
truth object location.

Data | Mean Absolute Error
---|---
Car | 0.23
Motorcycle | 0.27
Person | 0.14
Traffic Light | 0.67
Mean | 0.39
Table 1:  Accuracy of object localization for 4 videos on the KITTI odometry
dataset. MAE in meter.

Data | MAE | RMSE
---|---|---
Video 1 | 0.056 | 0.091
Video 2 | 0.052 | 0.085
Video 3 | 0.063 | 0.094
Video 4 | 0.055 | 0.092
Mean | 0.056 | 0.090
Table 2: Accuracy of trajectory estimation for 4 videos in the KITTI odometry
dataset. MAE and RMSE in meter

For evaluation of trajectory estimation, we use the root mean squared error
(RMSE) and mean absolute error (MAE) between the predicted translational
movement and ground truth camera movement between two frames, considering only
the X- and Z- coordinate. For evaluation of object localization, we use the
mean absolute error (MAE) between the estimated object location and ground
truth location. Lower values indicate better performance.

####  4.1.2 Results:

We report the MAE between the predicted object location and ground truth
location to validate the effectiveness of our object localization module in
Table 2.Even when estimating small objects that are far away such as the
traffic light, our system still achieves a small error of 0.67 meter.

Table 2 reports the RMSE and MAE between the predicted trajectory between two
successive frames and ground truth trajectory. Our system is able to estimate
the trajectory accurately and achieve promising results with 0.056 meter for
MAE and 0.090 meter for RMSE on average over the 4 videos.

In addition to the quantitative results discussed above, we also show example
visual results in Figure 4. Our system can successfully detect the bounding
boxes and categories of the target objects by the object detection module and
estimate the object location by the object localization module.

Figure 5: Examples of target object detection and object localization for our
dataset. Yellow rectangles denote the bounding boxes of the detected objects
(car and motorcycle). We also show the estimated location $(X,Z)$ of the
detected objects.

###  4.2 Simulated Navigation Experiments

####  4.2.1 Experimental Setting:

We also conducted experiments to evaluate the proposed system in a simulated
navigation experiment. We record 4 videos by the ZED camera [11] in an office
room. Specifically, we select 2 objects of interest (laptop and chair) and
record 2 videos while we walk towards each object. We use the trajectory
captured by the positional tracking system of the ZED camera as the ground
truth. To validate the effectiveness of our object localization module, we use
the depth sensing system of ZED camera to obtain the ground truth of object
position. For each object, we design 2 different test scenarios. In the first
scene, the object is located in front of the user. In the second scenario, the
object is located to the left front or right front of the user. In each case,
the video is captured while a user wearing the camera is walking straight to
the front.

Data | Mean Absolute Error
---|---
Chair 1 | 0.30
Chair 2 | 0.33
Laptop 1 | 0.18
Laptop 2 | 0.20
Mean | 0.25
Table 3: Accuracy of object localization using sequences in our dataset. MAE
in meter between the ground truth location and the estimated location in $X$
and $Z$.

Data | MAE | RMSE
---|---|---
Video 1 | 0.094 | 0.139
Video 2 | 0.088 | 0.138
Video 3 | 0.077 | 0.122
Video 4 | 0.083 | 0.127
Mean | 0.086 | 0.132
Table 4: Accuracy of trajectory estimation using sequences on our dataset.
RMSE and MAE in meter between the ground truth translation and the estimated
translation.

Figure 6: Examples of trajectory estimation, path update, and path correction alert on our dataset. We show frames 2, 4, 6, and 8 in Video 4. Purple arrow denotes the updated desired path. Yellow arrow denotes the actual path of the user. As an example, in frame 8, the angle between the planned path and the user’s path is larger than 30 degree, our system will sent out an alert message to the user, and suggesting the user to veer left slightly.  Table 5: Running time of different modules in second. Object detection module is tested on Jetson Xavier NX with NVIDIA Volta GPU. Trajectory estimation and object localization is tested with the ARM Cortex®-A57 MPCore CPU. |  | Image
---
Resolution
| Object
---
Detection
| Object
---
Localization
| Trajectory
---
Estimation
KITTI dataset | 1226x370 | 0.18 | 0.62 | 0.58
Our dataset | 1920x1080 | 0.18 | 0.98 | 0.92

####  4.2.2 Results:

We first examine the accuracy of object localization module in Table 4.
Moreover, we show some sample frames with object localization results in
Figure 5. We observe that our system successfully detect the bounding box and
predict the initial coordinate of the target object in the first frame.

Table 4 reports the accuracy for trajectory estimation. We illustrate a few
examples frames in Figure 6. As we can see from the figure, our system can
provide accurate path planing and path correction. If the angle between the
planned path (purple arrow) and the user’s path (yellow arrow) is larger than
30 degree, our system will sent out an alert message to the user.

###  4.3 Run time Analysis

The run time for the three computation modules for the KITTI video and our
video are summarized in Table 5. We expect the run time using the Jetson
processor to be slightly higher than using our CPU. Therefore, we expect the
navigation initialization (including object detection and localization) takes
less than 2 sec. and trajectory estimation takes less than 1 sec. with the
Jetson processor. This should be sufficient for real-time navigation
assistance when one walks towards an object. These times can be further
shortened with the optimization of the software implementation.

##  5 Conclusions

In this paper, we present a novel wearable navigation assistive system for
pBLV, which augments their perceptive power so that they can perceive objects
in their surrounding environment and reach the object of interest easily. Our
light-weight wearable system consists of a monocular camera mounted in front
of the chest of the user, a Jetson board for computation, and a battery. To
reduce the system cost and computation load, the system performs object
detection and localization only at the start of the navigation using the first
two captured frames, and then continuously update the object location relative
to the user by estimating the camera motion between frames. This is akin to
visual SLAM for tracking the pose of a moving camera, but here we use the
visual SLAM approach to update the user’s location and correspondingly the
object location relative to the user. Such continuously updated user and
object locations then enable real-time navigation path update and feedback to
the user.

Our experimental results on the KITTI odometry video dataset and simulated
indoor navigation videos dataset demonstrate that the proposed system can
accurately detect and localize the target object at the start of the
navigation, and estimate the user movement continuously, with an error well
within 0.5 meter, both outdoor and indoor.333 Note that in KITTI video and our
video, the camera motion between successive frames is relatively small,
leading to very small motion estimation error as well. For the intended
navigation application, such analysis only need to be run between frames with
a larger interval, and hence larger errors are likely, but we expect them to
on the same order as the localization error, which is within 0.5 meter. The
system is entirely vision-based and does not need other sensors for navigation
(e.g. IMUs and range sensors), and the computation can be run with the Jetson
processor in the wearable system to facilitate real-time navigation
assistance. Such a system holds great promise for assisting pBLV in their
daily living. Future research may develop a system where the video is uploaded
to an edge server for conducting all computation tasks, to further reduce the
wearable system weight [29].

##### Acknowledgments:

Research reported in this publication was supported in part by the NSF grant
1952180 under the Smart and Connected Community program, the National Eye
Institute of the National Institutes of Health under Award Number R21EY033689,
and DoD grant VR200130 under the Delivering Sensory and Semantic Visual
Information via Auditory Feedback on Mobile Technology” The content is solely
the responsibility of the authors and does not necessarily represent the
official views of the National Institutes of Health and NSF, and DoD. Yu Hao
was partially supported by NYUAD Institute (Research Enhancement Fund -
RE132).

##### Conflict of Interest:

New York University (NYU) and John-Ross Rizzo (JRR) have financial interests
in related intellectual property. NYU owns a patent licensed to Tactile
Navigation Tools. NYU, JRR are equity holders and advisors of said company.

## References

  * [1] Alcantarilla, P.F., Solutions, T.: Fast explicit diffusion for accelerated features in nonlinear scale spaces. IEEE Trans. Patt. Anal. Mach. Intell 34(7), 1281–1298 (2011)
  * [2] Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-up robust features (surf). Computer vision and image understanding 110(3), 346–359 (2008)
  * [3] Calonder, M., Lepetit, V., Strecha, C., Fua, P.: Brief: Binary robust independent elementary features. In: European conference on computer vision. pp. 778–792. Springer (2010)
  * [4] Fernandes, H., Costa, P., Filipe, V., Paredes, H., Barroso, J.: A review of assistive spatial orientation and navigation technologies for the visually impaired. Universal Access in the Information Society 18(1), 155–168 (2019)
  * [5] Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM 24(6), 381–395 (1981)
  * [6] GPS.gov: Gps accuracy. Official U.S. government information about the Global Positioning System (GPS) and related topics
  * [7] Griffin-Shirley, N., Banda, D.R., Ajuwon, P.M., Cheon, J., Lee, J., Park, H.R., Lyngdoh, S.N.: A survey on the use of mobile applications for people who are visually impaired. Journal of Visual Impairment & Blindness 111(4), 307–323 (2017)
  * [8] Hakobyan, L., Lumsden, J., O’Sullivan, D., Bartlett, H.: Mobile assistive technologies for the visually impaired. Survey of ophthalmology 58(6), 513–528 (2013)
  * [9] Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge university press (2003)
  * [10] Jiang, E., Ma, Z., Singh, A., Bobba, A., Park, S., Pundlik, S., Luo, G.: Field testing of all aboard, an ai app for helping blind individuals to find bus stops. Investigative Ophthalmology & Visual Science 62(8), 3529–3529 (2021)
  * [11] Labs, S.: ZED 2 Camera product page. https://www.stereolabs.com/zed-2
  * [12] Liu, X.J., Fang, Y.: Virtual touch: Computer vision augmented touch-free scene exploration for the blind or visually impaired. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1708–1717 (2021)
  * [13] Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International journal of computer vision 60(2), 91–110 (2004)
  * [14] Lu, D., Fang, Y.: Audi-exchange: Ai-guided hand-based actions to assist human-human interactions for the blind and the visually impaired. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1718–1726 (2021)
  * [15] MacKeben, M., Fletcher, D.C.: Target search and identification performance in low vision patients. Investigative Ophthalmology & Visual Science 52(10), 7603–7609 (2011)
  * [16] Maheepala, M., Kouzani, A.Z., Joordens, M.A.: Light-based indoor positioning systems: A review. IEEE Sensors Journal 20(8), 3971–3995 (2020). https://doi.org/10.1109/JSEN.2020.2964380
  * [17] Massiceti, D., Hicks, S.L., van Rheede, J.J.: Stereosonic vision: Exploring visual-to-auditory sensory substitution mappings in an immersive virtual reality navigation paradigm. PloS one 13(7), e0199389 (2018)
  * [18] Montello, D.R.: Cognitive research in giscience: Recent achievements and future prospects. Geography Compass 3(5), 1824–1840 (2009)
  * [19] Mur-Artal, R., Montiel, J.M.M., Tardos, J.D.: Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics 31(5), 1147–1163 (2015)
  * [20] Nistér, D.: An efficient solution to the five-point relative pose problem. IEEE transactions on pattern analysis and machine intelligence 26(6), 756–770 (2004)
  * [21] Organization, W.H., et al.: Visual impairment and blindness fact sheet n 282. World Health Organization (2014)
  * [22] Pascolini, D., Mariotti, S.P.: Global estimates of visual impairment: 2010. British Journal of Ophthalmology 96(5), 614–618 (2012)
  * [23] Pulli, K., Baksheev, A., Kornyakov, K., Eruhimov, V.: Real-time computer vision with opencv. Communications of the ACM 55(6), 61–69 (2012)
  * [24] Qin, T., Li, P., Shen, S.: Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics 34(4), 1004–1020 (2018)
  * [25] Real, S., Araujo, A.: Navigation systems for the blind and visually impaired: Past work, challenges, and open problems. Sensors 19(15), 3404 (2019)
  * [26] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 779–788 (2016)
  * [27] Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: Orb: An efficient alternative to sift or surf. In: 2011 International conference on computer vision. pp. 2564–2571. Ieee (2011)
  * [28] Treisman, A.M., Gelade, G.: A feature-integration theory of attention. Cognitive psychology 12(1), 97–136 (1980)
  * [29] Yuan, Z., Azzino, T., Hao, Y., Lyu, Y., Pei, H., Boldini, A., Mezzavilla, M., Beheshti, M., Porfiri, M., Hudson, T.E., et al.: Network-aware 5g edge computing for object detection: Augmenting wearables to “see” more, farther and faster. IEEE Access 10, 29612–29632 (2022)