# Scaling of selectivity in uniformly charged nanopores through a modified
Dukhin number for 1:1 electrolytes

Zsófia Sarkadi  Dávid Fertig  Mónika Valiskó  Dezső Boda<EMAIL_ADDRESS>1Department of Physical Chemistry, University of Pannonia, P.O. Box 158,
H-8201 Veszprém, Hungary

###### Abstract

We show that a modified version of the Dukhin number is an appropriate scaling
parameter for the ionic selectivity of uniformly charged nanopores. The
modified Dukhin number is an unambiguous function of the variables $\sigma$
(surface charge), $R$ (pore radius), and $c$ (salt concentration), and defined
as $\mathrm{mDu}=|\sigma|/e(R/\lambda)$, where $\lambda$ is the screening
length of the electrolyte carrying the $c$ dependence ($\lambda\sim
c^{-1/2}$). Scaling means that the device function (selectivity) is a smooth
and (in this case) monotonic function of mDu. The original Dukhin number
defined as $\mathrm{Du}=|\sigma|/eRc$ ($c^{-1}$ dependence) was introduced to
indicate whether the surface or the volume conduction is dominant in the pore.
The modified version satisfies scaling and characterizes selectivity in the
intermediate regime, where both surface and bulk conductions are present and
the pore is neither perfectly selective, nor perfectly non-selective. Our
modeling study using the Local Equilibrium Monte Carlo method and the Poisson-
Nernst-Planck theory provides the radial flux profiles from which the radial
selectivity profile can be computed. These profiles show in which region of
the nanopore the surface or the volume conduction dominates for a given
combination of the variables $\sigma$, $R$, and $c$. We show that the
inflection point of the scaling curve may be used to characterize the
transition point between the surface and volume conductions.

##  I Introduction

When the function of a device is determined by a few well-defined variables,
$a_{1},a_{2},\dots$, it is often possible to group them into a composite
parameter, $\xi$, that also determines the device’s behavior. This scaling
parameter is a well-defined function of the independent variables:
$\xi=\xi(a_{1},a_{2},\dots)$. The device function, $F$, is an obervable
property of the device. Scaling of the device function means that $F$ is a
smooth unambiguous function of the scaling parameter:
$F=f\left[\xi(a_{1},a_{2},\dots)\right]$.

Nanopores are located in a membrane and connect two bath electrolytes. They
facilitate the controlled movement of ions from one side of the membrane to
the other side. Tunable input parameters of this device are the radius of the
pore, $R$, voltage applied across the membrane, $U$, the surface charge on the
wall of the nanopore, $\sigma$, ionic concentrations in the baths, $c$, and
properties of the electrolyte, like ionic valences, $z_{i}$, for example ($i$
indexes the ionic species). The measurable output parameters are the currents
carried by the ions, $I_{i}$.

The structural properties of the nanopore (charge pattern and geometry)
determine what is a practical choice for the device function. In the case of a
uniformly charged nanopore ($\sigma<0$ is constant) studied here (cation)
selectivity defined as

$S_{+}=\frac{I_{+}}{I_{+}+I_{-}}$ (1)

is an appropriate device function. It is an unambiguous function of the
currents and is well-measurable via the reversal potential. For 1:1
electrolytes, if this number is $\approx 0.5$, the pore is non-selective,
while if it is $1$, the pore is perfectly cation selective. Note that
$S_{-}=1-S_{+}$.

In a previous paper, Fertig  _et al._ (2019) we introduced the scaling
parameter

$\xi=\dfrac{R}{\lambda\sqrt{z_{+}|z_{-}|}},$ (2)

where $\lambda$ is a characteristic screening length of the electrolyte for
which the Debye length is an obvious choice:

$\lambda_{\mathrm{D}}=\left(\dfrac{ce^{2}}{\epsilon_{0}\epsilon
kT}\sum_{i}z_{i}^{2}\nu_{i}\right)^{-1/2},$ (3)

where $k$ is Boltzmann’s constant, $T$ is the absolute temperature (it is
$298.15$ K in this work), $e$ is the elementary charge, $\nu_{i}$ is the
stoichiometric coefficent of ionic species $i$, $c$ is the salt concentration
($c_{i}=\nu_{i}c$ is the bath concentration of species $i$), $\epsilon$ is the
dielectric constant of the solvent (it is $78.45$ in this work), and
$\epsilon_{0}$ is the permittivity of vacuum.

The Debye length depends on the square root of the concentration,
$\lambda_{\mathrm{D}}\sim 1{/}\sqrt{c}$, and characterizes the width of the
double layer (DL) formed at the charged wall of the nanopore. Another choice
Fertig  _et al._ (2019) for the screening length is the one obtained from the
Mean Spherical Approximation (MSA) Blum (1975); Blum and Hoeye (1977); Nonner,
Catacuzzeno, and Eisenberg (2000) denoted by $\lambda_{\mathrm{MSA}}$.

We showed Fertig  _et al._ (2019) that in a pore with a bipolar charge pattern
(positive/negative) an obvious device function is the ratio of currents at
forward- and reverse-biased values of voltage (rectification) and it scales
with $\xi$. In another paper, Mádai  _et al._ (2018) we showed that in a pore
with a transistor-like charge pattern (positive/negative/positive) the ratio
of currents in open and closed states (switching) scales with
$R/\lambda_{\mathrm{D}}$ for a 1:1 electrolyte.

In these papers, Fertig  _et al._ (2019); Mádai  _et al._ (2018) we studied
nanopores with dimensions small enough that regions in the pore with very
small ionic concentrations (depletion zones) form and determine device
behavior. The regions of a nanopore along $z$-axis can be considered as
resistors connected in series. If the resistance of one segment is large due
to the low concentration of an ionic species there (depletion zone), then the
resistance is large for the whole pore for that ionic species. Rectification
of a bipolar nanopore, for example, is based on the fact that the depletion
zones are deeper at one sign of the voltage than at the opposite sign. A
depletion zone for a given ionic species in a given region may form if that
ionic species is the coion with respect to the surface charge in that region.

Formation of DLs, their overlap, and the resulting exclusion of coions are
also the key factor determining selectivity. If the DL at the wall is wide
compared to the pore radius ($\lambda\gg R$), the DLs overlap and a bulk
electrolyte is not formed around the centerline. The concentration of the
coion is small not only close to the surface but also at the centerline (Fig.
1). The coions are depleted, counterions are in excess, and the surface
conduction dominates.

If the width of the DL is small compared to the pore radius ($\lambda\ll R$),
the DL is restricted to the region close to the surface and the region along
the centerline contains enough coions so they can carry current in that
region. In the region where both coions and counterions are present, volume
conduction dominates.

Between the two limiting cases both surface and volume conductions are present
and our negative pore is not perfectly cation selective ($S_{+}$ is in between
$0.5$ and $1$ for the 1:1 electrolyte consedered here).

The calculations reported in Ref. Fertig  _et al._ , 2019 were performed for a
fixed value of the surface charge $\sigma{\pm}1$ $e$/nm2, so the $\xi$ scaling
parameter does not contain $\sigma$. Here, we pursue a scaling parameter that
also includes $\sigma$ in addition to $R$ and $c$.

##  II Modified Dukhin number as scaling parameter

Because there is a monotonic relationship between ionic selectivity and the
ratio of surface and volume conductances that are present in the nanopore at
the same time, the Dukhin number Bazant, Thornton, and Ajdari (2004); Chu and
Bazant (2006); Bocquet and Charlaix (2010) defined as

$\mathrm{Du}\equiv\frac{|\sigma|}{eRc}$ (4)

offers itself to be the scaling parameter we are after. This definition of Du
contains exactly those variables that we want to see in our scaling parameter,
$\sigma$, $R$, and $c$.

Du was originally introduced by Bikerman Bikerman (1940) to characterize the
ratio of the surface and volume conductances focusing on electrokinetic
phenomena. Later, Dukhin adopted the idea (see Ref. Dukhin, 1993 and
references therein) to study electrophoretic phenomena. Although the name
Bikerman number also occurs in the literature, Lyklema introduced the name
Dukhin number to salute Dukhin. Lyklema  _et al._ (1995) The Dukhin number
(and a characteristic length called Dukhin length,
$l_{\mathrm{Du}}=\mathrm{Du}\,R$) was used in several modeling studies
describing nanopores, more specifically, when volume and surface transport
processes competed inside the pore. Bazant, Thornton, and Ajdari (2004); Chu
and Bazant (2006); Khair and Squires (2008); Das and Chakraborty (2010);
Bocquet and Charlaix (2010); Zangle, Mani, and Santiago (2010); Lee  _et al._
(2012); Yeh  _et al._ (2014); Ma  _et al._ (2017); Xiong  _et al._ (2019);
Poggioli, Siria, and Bocquet (2019); Cengio and Pagonabarraga (2019);
Kavokine, Netz, and Bocquet (2020); Noh and Aluru (2020)

The definition in Eq. 4 is in agreement with the traditional definition of
Bikerman, because it can be computed from the ratio of the surface excess of
the cations, $|\sigma|2\pi RH$ ($H$ is the length of the pore) assuming
perfect exclusion of the anions, and the number of charge carriers assuming a
bulk electrolyte in the pore, $2cR^{2}\pi H$ (the factor $2$ is needed because
both cations and anions carry current).

So, we have a parameter that seems to be appropriate for our purposes, we just
need to check whether it works as a scaling parameter. We will see in Section
III that it does not.

Therefore, we define a modified Dukhin number as

$\mathrm{mDu}\equiv\frac{|\sigma|}{eR/\lambda}.$ (5)

As defined, mDu is not strictly a “number,” as it has units of m-2. Below, we
show that mDu is a scaling parameter for $S_{+}$. Therefore, it captures the
essential physics of this device function. mDu can be multiplied by any length
squared that does not depend on $R$ or $c$ (e.g., the Bjerrum length) to make
it dimensionless, but this factor is irrelevant for the scaling so we will use
the dimensional version throughout (except Section III.5).

Because $\lambda\sim 1/\sqrt{c}$, we should realize that

$\mathrm{mDu}\sim\frac{|\sigma|}{eR\sqrt{c}},$ (6)

namely, mDu corresponds to the original Dukhin number (Eq. 4) with $\sqrt{c}$
in the denominator instead of $c$. The importance of $\sqrt{c}$ is not a new
idea. Bikerman writes in his 1940 paperBikerman (1940): “At a constant $\zeta$
potential the surface conductivity $\chi$ is nearly proportional to the square
root of the ionic concentration ($C^{\frac{1}{2}}$) whilst [the volume
conductivity] $\kappa$ in dilute solutions is nearly proportional to the
concentration $C$ itself. Thus,
$\chi/\kappa=\mathrm{const}.\;C^{-\frac{1}{2}}$.”

The original and modified Dukhin numbers behave the same way in the two
limiting cases, at very low and very large concentrations ($\lambda{\gg}R$ and
$\lambda{\ll}R$). In between the limiting cases, however, they behave
differently. Both contain $\sigma$ as a parameter, so both can account for the
fact that the surface charge produces the DLs, and, thus, selectivity.

The original Dukhin number is obtained from the relation of the two limiting
cases. In these limiting cases the screening length has no importance, because
there is no DL in the case of perfect volume conductance, while there are no
coions in the case of perfect surface conductance (perfect DL overlap), so the
ratio of $\lambda$ to $R$ does not matter.

The modified Dukhin number brings in the Debye length that, in our opinion, is
vital in constructing a proper scaling parameter. In the realm between the
limiting cases DLs may just partially overlap, so the ratio of $\lambda$ to
$R$ characterizing the extent of DL overlap matters. We will show in
Subsection III.3 that mDu is a proper scaling parameter in the intermediate
domain.

The ratio of the surface and volume conductions is also interesting, but they
are consequences. Surface conduction dominates because the DLs overlap and not
vice versa. In the intermediate domain, the nanopore contains both a region
where rather the surface conduction dominates (near the wall) and a region
where rather volume conduction dominates (in the middle of the pore around the
centerline).

The boundary between these regions, however, is not sharp, so any definition
of surface and volume conductions is arbitrary, while selectivity is a well-
defined measurable quantity. That is why we focus on selectivity as our main
device function. We also discuss surface and volume conductions in Subsection
III.4, however, by analyzing our simulation results that allow us to look into
the black box and see what is going on at the molecular level. Concentration
and flux profiles reveal the existence of DL and bulk regions inside the pore.

We think that a parameter intended to characterize surface vs. volume
conduction should satisfy scaling. The purpose of this paper is to show, using
computer simulations, that mDu is an appropriate scaling parameter for the
selectivity of a uniformly charged pore.

##  III Results

###  III.1 Model of the nanopore

A cylindrical nanopore of length $H$ and radius $R$ spans a membrane that
separates two baths. The cylindrical wall of the nanopore and the flat
parallel walls confining the membrane are assumed to be hard. The $z$
dimension is the one perpendicular to the membrane along the pore. Because the
system has rotational symmetry about the $z$ axis, the other relevant
coordinate is the radial one, $r$, that represents the distance from the $z$
axis.

The electrolyte is modeled in the implicit solvent framework, namely, the
interaction potential between two hard-sphere ions is defined by Coulomb’s law
in a dielectric background:

$u_{ij}(r)=\left\\{\begin{array}[]{ll}\infty&\quad\mathrm{if}\quad
r<(d_{i}+d_{j})/2\\\
\dfrac{1}{4\pi\epsilon_{0}\epsilon}\dfrac{z_{i}z_{j}e^{2}}{r}&\quad\mathrm{if}\quad
r\geq(d_{i}+d_{j})/2\\\ \end{array}\right.$ (7)

where $d_{i}$ is the diameter of ionic species $i$, and $r$ is the distance
between two ions. Here, we consider only 1:1 electrolytes, namely, $z_{+}=1$
and $z_{-}=-1$. For the ionic diameters, we use $d_{+}=d_{-}=0.3$ nm. A
uniform negative surface charge $\sigma$ is placed on the wall of the
nanopore.

Figure 1: Radial concentration profiles as obtained for various combinations
of $\sigma$ and $R/\lambda$. Different columns have different values of
$R/\lambda$, while different rows have different values of $\sigma$. In these
cases $R=1$ nm. The concentrations that correspond to the $R/\lambda=0.526$,
$0.912$, and $2.408$ values are $c=0.03$, $0.1$, and $1$ M, respectively. Blue
and red curves refer to the cations and the anions, respectively. The curves
are mirrored to $r=0$ for better visualization, while strictly speaking
$r{\geq}0$. The applied method is NP+LEMC, so
$\lambda=\lambda_{\mathrm{MSA}}$.

The description of the methods with which we study this model is included in
the Appendix. Both methods are based on the Nernst-Planck (NP) transport
equation (Eq. 9). In one method, we relate the concentration profile,
$c_{i}(\mathbf{r})$, to the electrochemical potential profile,
$\mu_{i}(\mathbf{r})$ with the Poisson-Boltzmann (PB) theory. This continuum
theory is known as the Poisson-Nernst-Planck (PNP) theory.

The other method is based on a Monte Carlo (MC) technique that is an
adaptation of the Grand Canonical Monte Carlo (GCMC) method to a non-
equilibrium situation, where $\mu_{i}(\mathbf{r})$ is not constant system-
wide, so the system is not in global equilibrium, only in local equilibrium.
The method is called the Local Equilibrium Monte Carlo (LEMC) technique, while
it is called NP+LEMC when we couple it to the NP equation.

Because the difference between the computational methods is that NP+LEMC
includes all the ionic correlations present in the system and PNP is mean
field, we use different screening lengths that reflect this difference. So
$\lambda=\lambda_{\mathrm{MSA}}$ if we use NP+LEMC, while
$\lambda=\lambda_{\mathrm{D}}$ if we use PNP.

Figure 2: The selectivity curves plotted against the Dukhin number,
$\mathrm{Du}=|\sigma|/eRc$ (left panel), against the parameter in which there
is $\sqrt{c}$ in place of $c$ in Du (middle panel), and against the modified
Dukhin number, $\mathrm{mDu}=|\sigma|/e(R/\lambda)$ (right panel). Symbols and
lines refer to NP+LEMC and PNP results, respectively. Different colors mean
different concentrations, while points with a given color correspond to
different surface charges. The pore radius is constant, $R=1$ nm. In the right
panel, the MSA screening length ($\lambda_{\mathrm{MSA}}$) is used for the
NP+LEMC data, while the Debye length ($\lambda_{\mathrm{D}}$) for the PNP
data.

###  III.2 Double layer formation and double layer overlap

Figure 1 illustrates the effect of the two relevant parameters ($\sigma$ and
$R/\lambda$). The surface charge produces the separation of counterions and
coions (DL formation), while the $R/\lambda$ parameter accounts for the degree
of overlap of the DL. The figure shows radial concentration profiles to depict
the interplay of these two effects.

The columns correspond to different $R/\lambda$ values (a small $R/\lambda$
value means stronger DL overlap), while the rows correspond to different
surface charges (a larger $|\sigma|$ value results in a stronger separation of
cation and anion profiles).

As we go from left to right (increasing $R/\lambda$), the gap between the
cation and anion profiles around the centerline $r\approx 0$ decreases (note
the log scale). As we go from top to bottom (increasing $|\sigma|$), the gap
between the cation and anion profiles near the wall ($r\approx R$) increases.

This implies that the two parameters have two relatively distinct effects. The
$R/\lambda$ parameter rather determines the behavior in the middle of the
pore, while $\sigma$ rather determines the behavior in the DL. It is common to
express this distinction in terms of volume (or bulk) and surface conductions.
The DL region (if exists) is responsible for the surface conduction, while the
bulk region in the middle (if exists) is responsible for the volume
conduction. With $R/\lambda$ and $\sigma$, therefore, we have two parameters
with which we can tune the weights of the volume and surface conductions in
the total conduction. Later in this paper, we will provide a quantitative
analysis for this.

###  III.3 Scaling as a function of the modified Dukhin number

To check whether Du is an appropriate scaling parameter for our interests, we
performed a large number of simulations (using both NP+LEMC and PNP) first for
a fixed pore radius $R=1$ nm for varying values of $\sigma$ and $c$. The
results for our chosen device function, selectivity ($S_{+}$) are shown in the
left panel of Fig. 2. The different curves are for different concentrations. A
given curve was obtained by varying surface charge. The results show that Du
is not appropriate for our purposes, because the curves do not coincide: they
are spread along with the abscissa larger concentrations belonging to the
smaller Du values.

Eq. 6 implies that the modified Dukhin number with $\lambda_{\mathrm{D}}$ in
it is proportional to the original Dukhin number substituted $\sqrt{c}$ for
$c$. The results plotted as functions of $|\sigma|/eR\sqrt{c}$ are shown in
the middle panel of Fig. 2. Now the curves nicely coincide, implying that the
scaling factor we are looking for indeed depends on $\sqrt{c}$, and, thus, on
$\lambda_{\mathrm{D}}$.

The $Rc^{1/2}$ product in the denominator is proportional to
$R/\lambda_{\mathrm{D}}$, so it is conceptually advantageous to include the
$R/\lambda$ term in the new scaling parameter (Eq. 5) because it expresses the
tendency of the DLs to overlap.

The right panel of Fig. 2 shows the results as functions of this new scaling
parameter, mDu. Apart from a rescaled abscissa, this panel is the same as the
middle one with one notable difference that is hardly noticable with the naked
eye. This panel defines the screening length differently for the NP+LEMC and
PNP data. In the case of NP+LEMC we use the MSA screening length
($\lambda_{\mathrm{MSA}}$), while in the case of PNP we use the Debye length
($\lambda_{\mathrm{D}}$). The difference between these is small especially
using a logarithmic scale at the abscissa. The difference, however, is larger
at larger concentrations.

This is better seen in Fig. 3 where only the data for $c=0.3$ and $1$ M are
shown with a linear scale at the abscissa. If one observes, for example, the
red symbols and curves for $c=1$ M, one can see that the agreement between PNP
(curve) and NP+LEMC is better if we use $\lambda_{\mathrm{MSA}}$ for NP+LEMC
(filled symbols) than using $\lambda_{\mathrm{D}}$ (open symbols).

Figure 3: The blue ($c=0.3$ M) and red ($c=1$ M) curves of Fig. 2 plotted on a
linear scale of the abscissa (mDu). The open and filled symbols show NP+LEMC
values computed with $\lambda_{\mathrm{D}}$ and $\lambda_{\mathrm{MSA}}$,
respectively. The curves refer to PNP results using $\lambda_{\mathrm{D}}$.

The radius of the pore was constant in Fig. 2. Figure 4 shows scaling for
varying pore radii at a fixed concentration ($c=0.1$ M). The left panel shows
the $S_{+}$ curves as functions of $|\sigma|$, while the right panel shows
them rescaled with $R/\lambda$. Because $\lambda$ is constant ($c$ is
constant) here, in effect this means a rescaling with $R$.

In summary, scaling works for selectivity using an appropriately chosen
scaling parameter that we call the modified Dukhin number, mDu.

Figure 4: Selectivity ($S_{+}$) curves plotted against the surface charge
(left panel), and the modified Dukhin number (right panel) for different pore
radii (different colors). The difference between the two panels is that the
abscissa is rescaled by dividing $|\sigma|$ with $R/\lambda$. The unit of
$\sigma$ is $e$/nm2. The concentration is constant, $c=0.1$ M. Only NP+LEMC
results are shown with $\lambda=\lambda_{\mathrm{MSA}}$.

###  III.4 Radial selectivity profile to characterize surface and volume
conductions

If we have a selectivity value for a given situation of, for example,
$S_{+}=0.85$, it does not necessarily mean that there is no volume conduction
in the pore. Similarly, if we have a mildly selective value $S_{+}=0.65$, it
does not necessarily mean that there is no surface conduction. We cannot
conclude from the “global” selectivity value ($S_{+}$) alone to what degree
surface and volume conductions are present in the two respective regions of
the pore (close and far from the surface).

How can we characterize the share of the two types of conductions on the basis
of the simulation results? We define the radial selectivity profile as

$s_{+}(r)=\dfrac{j_{+}(r)}{j_{+}(r)+j_{-}(r)},$ (8)

where $j_{+}(r)$ and $j_{-}(r)$ are the $z$-components of the flux profiles
for the cation and the anion, respectively, averaged over the pore in the $z$
dimension (from $-H/2$ to $H/2$). It is a quantity that depends on $r$ and
characterizes to what degree a region of the pore at a distance $r$ from the
$z$ axis contributes to the “global” selectivity, $S_{+}$. Note that the
average of $s_{+}(r)$ is not equal to $S_{+}$, but we can draw conclusions
from the shapes of the curves nevertheless.

Figure 5: The radial selectivity profiles, $s_{+}(r)$, for different surface
charges (left panel), concentrations (middle panel), and pore radii (right
panel). In each case, the other two parameters are kept fixed at the values
indicated above the panels. The arrows show the direction of increasing the
respective parameter.

First, let us investigate the effect of the parameters $\sigma$, $c$, and $R$
on the $s_{+}(r)$ function. Figure 5 shows the dependence of these curves on
one of these parameters while the other two are kept constant. The left panel
shows that the $s_{+}(r)$ curves are shifted towards larger values as $\sigma$
increases. A larger surface charge separates cation and anion profiles and
excludes anions more efficiently. The middle panel shows that the $s_{+}(r)$
curves are shifted towards smaller values as $c$ increases. At larger
concentrations, the DLs are thinner, so a wider non-selective bulk region
forms in the middle of the pore thus bringing selectivity down. The right
panel shows that the $s_{+}(r)$ profiles are shifted towards smaller values as
$R$ increases. In wider pores there is more space for the formation of a bulk
region in the middle of the pore.

Figure 5 also shows that there are cases when both a surface conduction region
(large selectivity near the surface) and a volume conduction region (small
selectivity in the center of the pore) are present. For moderate overlap (left
panel, $R/\lambda=0.912$) it occurs if $\sigma$ is large enough because large
$\sigma$ increases selectivity near the wall. For a relatively large surface
charge (middle panel, $\sigma=-1$ $e$/nm2), it occurs at large concentrations
that produces small screening lengths and confine the DL near the wall region.
The pore radius has similar effects with the difference that now we tune the
$R/\lambda$ ratio with $R$ (right panel).

Figure 6 illustrates the notion that a given “global” selectivity can be
achieved with different sets of parameters that correspond to different
$s_{+}(r)$ profiles. The figure shows curves for the same mDu parameter but
with different combinations of $\sigma$, $R$, and $c$ producing the same mDu.

Figure 6A shows results for two simulations that, due to scaling, produce
about the same “global” selectivity, $S_{+}=0.787$ (blue) and $0.757$ (red).
Both the blue and red curves have the same mDu number ($\mathrm{mDu}=0.2$
nm-2) and the same pore radius ($R=1$ nm). The blue curve is for a smaller
surface charge ($\sigma=-0.1$ $e$/nm2) and a smaller concentration ($c=0.025$
M), while the red curve is for a larger surface charge ($\sigma=-0.5$ $e$/nm2)
and a larger concentration ($c=1$ M).

(A)

(B)

Figure 6: Radial selectivity profiles ($s_{+}(r)$, left panel), radial flux
profiles (the $z$-coordinate averaged over the pore in the axial direction)
multiplied by $2\pi r$ ($2\pi rj_{i}(r)$, middle panel), and radial
concentration profiles ($c_{i}(r)$, right panel) for parameters indicated in
the figure. (A) The pore radius is fixed ($R=1$ nm), while $\sigma$ and $c$
are changed in a way that mDu is the same ($\mathrm{mDu}=0.2$ nm-2) for the
two sets of curves (red and blue). (B) The concentration is fixed ($c=0.1$ M),
while $\sigma$ and $R$ are changed in a way that mDu is the same
($\mathrm{mDu}=0.137$ nm-2) for the two sets of curves (red and blue). In the
middle and right panels, solid and dashed lines refer to cations and anions,
respectively.

The $s_{+}(r)$ curves (left panel) show that approximately the same $S_{+}$
value can be achieved in two ways. In the case of the blue curve, the DL
overlaps (small $c$, large $\lambda$) so the coion is excluded over the whole
cross section and selectivity is at a high level even if $\sigma$ is
relatively small. In the case of the red curve, the DL overlaps less (large
$c$, small $\lambda$) so a less selective bulk region is formed in the middle,
but the larger surface charge ($\sigma=-0.5$ $e$/nm2) produces a large
counterion-coion separation near the wall and pulls up selectivity there.

The concentration and flux profiles in the other two panels show how this
comes about. Different degree of overlapping can be seen better in the
concentration profiles (right panel). In the middle panel, the $2\pi
rj_{i}(r)$ curves are shown on a logarithmic scale. The gap between the cation
(solid) and anion (dashed) profiles shows the degree of selectivity at $r$. In
the case of the blue curves, the gap is the same all along the pore in the
radial dimension, while in the case of the red curves, the gap opens as the
pore wall is approached.

Figure 6B shows results for two simulations that have the same mDu number
($\mathrm{mDu}=0.137$ nm-2) and the same concentration ($c=0.1$ M). Because
$R$ changes, we show the profiles as functions of $r/R$ here. They produce
approximately the same selectivities, $S_{+}=0.77$ (blue) and $0.772$ (red).
The blue curve is for a smaller surface charge ($\sigma=-0.25$ $e$/nm2) and a
smaller pore radius ($R=2$ nm), while the red curve is for a larger surface
charge ($\sigma=-0.5$ $e$/nm2) and a larger pore radius ($R=4$ nm).

The $s_{+}(r)$ curves (left panel) show similar behavior as in Fig. 6A. In the
case of the blue curve, the DL overlaps (small $R$) so there is less space for
the formation of a bulk region in the middle. Therefore, the selectivity is
uniformly large even if $\sigma$ is relatively small. In the case of the red
curve, the degree of overlap of the DL is smaller (large $R$) so a less
selective bulk region is formed in the middle, but the larger surface charge
($\sigma=-0.5$ $e$/nm2) produces a large counterion-coion separation near the
wall and pulls up selectivity there.

The concentration profiles (right panel) show the different degrees of overlap
in the two cases. The flux profiles (middle panel) show as the gap opens more
widely in the case of the red curves than in the case of the blue curves.

###  III.5 Transition point between surface and volume conductions

By plotting selectivity as a function of mDu using a logarithmic scale on the
mDu axis, we get a curve with an inflection point that belongs to a
selectivity $S_{+}\approx 0.75$. If we fit a sigmoid curve too all the points
of Fig. 2, the resulting selectivity value for the inflection point is
$S_{+}\approx 0.753$. The inflection point offers itself as a transition point
separating “rather non-selective” and “rather selective” regions. The
derivative of the $S_{+}$ curve as a function of $\lg(\mathrm{mDu})$ is
maximal in the inflection point where small changes in mDu lead to relatively
large changes in selectivity.

The inflection point is at $0.133$ nm-2, so if we normalize $\sigma$ in mDu
with $\sigma_{0}\equiv-0.133$ $e$/nm2 as $(\sigma/\sigma_{0})/(R/\lambda)$, we
obtain a rescaled mDu that has the inflection point at $1$ as shown by the
left panel of Fig. 7. This panel shows the same data that the right panel of
Fig. 2 showed with mDu rescaled and visualizing the results differently. A
given color now corresponds to a given surface charge with concentration
changing in the range $c=0.01-1$ M. Black, red, and blue colors correspond to
surface charges $\sigma=-0.01$, $-0.133$, and $-2$ $e$/nm2, respectively.

The slope is maximal in the case of $\sigma=-0.133$ $e$/nm2 (red curve).
$S_{+}$ changes in a wide range with varying $R/\lambda$ with $S_{+}=0.75$
being the mean value. At this surface charge, $S_{+}$ is far from the limiting
values $0.5$ and $1$, at least for the simulated values of $c$ and $R$. Low
and large surface charges ($\sigma=-0.01$ and $-2$ $e$/nm2, black and blue
curves), on the other hand, are restricted to small ($\sim 0.5$) and large
($\sim 1$) selectivities.

Figure 7:  Selectivity, $S_{+}$, plotted against the modified Dukhin number
normalized as , $(\sigma/\sigma_{0})/e(R/\lambda)$, where
$\sigma_{0}\equiv-0.133$ $e$/nm2 (left panel). Different colors mean different
surface charges, while points with a given color correspond to different
concentrations. The values $\sigma=-0.01$, $-0.133$, and $-2$ $e$/nm2 are
highlighted with different colors (black, red, and blue), while other surface
charges are shown with gray for reference. The right panel shows the radial
selectivity profiles, $s_{+}(r)$, for these surface charges for concentrations
$c=0.01$ and $1$ M (filled and open symbols). The MSA screening length for
$c=1$ M is indicated by the horizontal dashed line with arrows.

The right panel of Fig. 7 shows the $s_{+}(r)$ selectivity profiles for the
largest ($c=1$ M, dashed lines) and smallest ($c=0.01$ M, solid lines)
simulated concentrations for the three surface charges in focus. Selectivity
is small for $\sigma=-0.01$ $e$/nm2 (black) even for $c=0.01$ M, where DL
overlap tend to increase the selectivity. For the value $\sigma=-0.133$
$e$/nm2, selectivity changes in a wide range as a result of changing the
concentration between $c=0.01$ and $1$ M. In the case of $\sigma=-2$ $e$/nm2,
the large DL overlap at $c=0.01$ M results in a closely perfect selectivity.

The curve for $\sigma=-2$ $e$/nm2 and $c=1$ M is more interesting. A clean
separation of volume and surface conductions can be observed in this case
because the DL is restricted to the surface due to the small value of
$\lambda$ (large $c$). This curve corresponds to a “global” selectivity
$S_{+}=0.933$ that is not readily apparent from the $s_{+}(r)$ curve. As noted
earlier, $S_{+}$ is not the average of $s_{+}(r)$; regions near the wall
contribute more to current, and, thus, to $S_{+}$, because those regions have
larger cross section (see middle panels of Fig. 6). This result emphasizes
that surface conduction tends to have a larger weight compared to the volume
conduction due to the fact that cross sections are larger near the pore wall
than near the centerline. Also, these results imply that a sufficiently large
surface charge is needed if we want to obtain a selective surface-conduction
region near the wall.

##  IV Summary

We showed that cation selectivity as our chosen device function scales
properly with the scaling parameter inspired by the Dukhin number. Continuing
the tradition of saluting Dukhin, we call this paremeter the modified Dukhin
number. This new parameter differs from the original one that it contains the
$R/\lambda$ ratio instead of the $Rc$ product. As a consequence, it is
proportional to $c^{-1/2}$ instead of $c^{-1}$.

The modified version satisfies scaling and characterizes selectivity in the
intermediate regime, where both surface and bulk conductions may be present
and the pore may be neither perfectly selective, nor perfectly non-selective.
Our modeling study using the LEMC simulation method and the PNP theory
provides the radial flux profiles from which the radial selectivity profile
can be computed. These profiles show in which region of the nanopore the
surface or the volume conduction dominates for a given combination of the
variables $\sigma$, $R$, and $c$.

To characterize the transition point that separates the “rather selective” and
the “rather non-selective” state points, the inflection point of the $S_{+}$
vs. $\lg(\mathrm{mDu})$ function is a natural, and mathematically well
defined, choice. The inflection point depends on system parameters ($d_{i}$,
$H$, $U$, $T$, and $\epsilon$), but otherwise it is a well-defined universal
feature of this system for 1:1 electrolytes.

For electrolytes containing multivalent ions (2:2, 2:1, and 3:1, for example)
interesting phenomena beyond the mean-field treatment may occur due to strong
ionic correlations such as overcharging and charge inversion. Fertig, Valiskó,
and Boda (2020) These will be reported in subsequent publications.

## Acknowledgements

We gratefully acknowledge the financial support of the National Research,
Development and Innovation Office – NKFIH K124353. Present article was
published in the frame of the project GINOP-2.3.2-15-2016-00053 (“Development
of engine fuels with high hydrogen content in their molecular structures
(contribution to sustainable mobility)”).

##  Appendix A Computational methods

In this work, we use two methods that have the common denominator that both
apply the Nernst-Planck (NP) transport equation Nernst (1888); Planck (1890)
to compute the ionic flux:

$\mathbf{j}_{i}(\mathbf{r})=-\frac{1}{kT}D_{i}(\mathbf{r})c_{i}(\mathbf{r})\nabla\mu_{i}(\mathbf{r}),$
(9)

where $\mathbf{j}_{i}(\mathbf{r})$, $D_{i}(\mathbf{r})$, $c_{i}(\mathbf{r})$,
and $\mu_{i}(\mathbf{r})$ are the flux density, the diffusion coefficent
profile, the concentration profile, and the electrochemical potential profile
of ionic species $i$, respectively. To make use of this equation, we need a
relation between the concentration profile, $c_{i}(\mathbf{r})$, and the
electrochemical potential profile, $\mu_{i}(\mathbf{r})$.

The LEMC method Boda and Gillespie (2012) is a particle simulation technique
devised for a non-equilibrium situation, where $\mu_{i}(\mathbf{r})$ is not
constant globally. We divide the simulation into small volume elements,
$V^{\alpha}$, and assume local thermodynamic equilibrium in each, namely, we
assign $\mu_{i}^{\alpha}$ values to each. Then, we perform particle
displacements and particle insertions/deletions with the same acceptannce
criteria as we do in a GCMC simulation, but with the volume, particle number
($N_{i}^{\alpha}$), and chemical potential of the volume element in which we
peform an MC step. Therefore, the LEMC technique is an adaptation of the GCMC
technique for a system that is not at equilibrium globally.

The resulting method, coined NP+LEMC, solves the problem iteratively on the
basis of the scheme

$\mu^{\alpha}_{i}[n]\,\xrightarrow{\mathrm{LEMC}}\,c^{\alpha}_{i}[n]\,\xrightarrow{\mathrm{NP}}\,\mathbf{j}^{\alpha}_{i}[n]\,\xrightarrow{\nabla\cdot\mathbf{j}=0}\,\,\mu^{\alpha}_{i}[n+1],$
(10)

where $c_{i}^{\alpha}[n]$ is the concentration in volume element $V^{\alpha}$
obtained from an LEMC simulation in the $n$th iteration. The chemical
potential for the next $[n{+}1]$th iteration is obtained by assuming that the
flux $\mathbf{j}_{i}^{\alpha}$ computed from it and from $c_{i}^{\alpha}[n]$
satisfies the continuity equation, $\nabla\cdot\mathbf{j}_{i}=0$. Details are
found in Refs. Boda and Gillespie, 2012; Boda  _et al._ , 2014; Boda, 2014;
Fertig  _et al._ , 2017.

The importance of the LEMC technique is that it can take into account the
correlations between ions beyond the mean-field approximation including the
finite size of ions.

In the Poisson-Nernst-Planck (PNP) theory, we relate $c_{i}(\mathbf{r})$ to
$\mu_{i}(\mathbf{r})$ via the Poisson-Boltzmann (PB) theory where the ions are
modeled as point charges interacting with the average electrical potential,
$\Phi(\mathbf{r})$, exerted by all the ions in the system. In this mean-field
approach, the electrolyte is assumed to be an ideal solution with the
electrochemical potential

$\mu_{i}(\mathbf{r})=\mu_{i}^{0}+kT\ln
c_{i}(\mathbf{r})+z_{i}e\Phi(\mathbf{r}),$

where $\mu_{i}^{0}$ is a reference chemical potential independent of the
location. Note that an excess chemical potential
$\mu_{i}^{\mathrm{ex}}(\mathbf{r})$, is added to this expression when ionic
correlations are taken into account as they are in the LEMC method. Poisson’s
equation and the continuity equation are also satisfied in the solution of the
PNP theory.

Here, we solve the system with the Scharfetter–Gummel scheme. Gummel (1964) A
2D finite element method is used with $20-60$ thousands elements in a
triangular mesh. Details are found in Ref. Matejczyk  _et al._ , 2017.

The constant $\sigma$ surface charge is assured via a Neumann boundary
condition in PNP, while fractional point charges are placed on a rectangular
grid of width $0.2$ nm in LEMC.

In both methods, proper boundary conditions were applied in the two baths at
the wall of the cylinder that confines the finite simulation cell. Dirichlet
boundary conditions were applied for $\Phi(\mathrm{r})$; the difference of the
applied potential on the two sides of the membrane specifies the applied
voltage, $U$. Bath concentrations were assumed to be the same on the two sides
of the membrane, $c$, though the methodology would be able to handle
asymmetrical systems as well. In the 1:1 electrolyte considered here
$c_{+}=c_{-}=c$ in both baths.

For the diffusion coefficient profile, $D_{i}(\mathbf{r})$, we use a piecewise
constant function, where the value in the baths is $1.334\times 10^{-9}$ m2s-1
for both ionic species, while it is the tenth of that inside the pore,
$D_{i}^{\mathrm{pore}}$, as in our earlier works. Matejczyk  _et al._ (2017);
Mádai  _et al._ (2017, 2018); Fertig  _et al._ (2019); Fertig, Valiskó, and
Boda (2020) These particular choices do not qualitatively affect our
conclusions.

In this model calculation, where $d_{+}=d_{-}$ and
$D_{+}(\mathbf{r})=D_{-}(\mathbf{r})$ for a 1:1 electrolyte, so $S_{+}=0.5$
exactly for a perfectly non-selective pore ($\sigma=0$, for example)

## References

  * Fertig  _et al._ (2019) D. Fertig, B. Matejczyk, M. Valiskó, D. Gillespie, and D. Boda, J. Phys. Chem. C 123, 28985 (2019).
  * Blum (1975) L. Blum, Mol. Phys. 30, 1529 (1975).
  * Blum and Hoeye (1977) L. Blum and J. S. Hoeye, J. Phys. Chem. 81, 1311 (1977).
  * Nonner, Catacuzzeno, and Eisenberg (2000) W. Nonner, L. Catacuzzeno, and B. Eisenberg, Biophys. J. 79, 1976 (2000).
  * Mádai  _et al._ (2018) E. Mádai, B. Matejczyk, A. Dallos, M. Valiskó, and D. Boda, Phys. Chem. Chem. Phys. 20, 24156 (2018).
  * Bazant, Thornton, and Ajdari (2004) M. Z. Bazant, K. Thornton, and A. Ajdari, Phys. Rev. E 70, 021506 (2004).
  * Chu and Bazant (2006) K. T. Chu and M. Z. Bazant, Phys. Rev. E 74, 011501 (2006).
  * Bocquet and Charlaix (2010) L. Bocquet and E. Charlaix, Chem. Soc. Rev. 39, 1073 (2010).
  * Bikerman (1940) J. J. Bikerman, Trans. Farad. Soc. 35, 154 (1940).
  * Dukhin (1993) S. Dukhin, Adv. Coll. Interf. Sci. 44, 1 (1993).
  * Lyklema  _et al._ (1995) J. J. Lyklema, A. de Keizer, B. Bijsterbosch, G. Fleer, and M. C. S. (Eds.),  _Solid-Liquid Interfaces_, Fundamentals of Interface and Colloid Science 2 (Elsevier, Academic Press, 1995).
  * Khair and Squires (2008) A. S. Khair and T. M. Squires, J. Fluid Mech. 615, 323 (2008).
  * Das and Chakraborty (2010) S. Das and S. Chakraborty, Langmuir 26, 11589 (2010).
  * Zangle, Mani, and Santiago (2010) T. A. Zangle, A. Mani, and J. G. Santiago, Chem. Soc. Rev. 39, 1014 (2010).
  * Lee  _et al._ (2012) C. Lee, L. Joly, A. Siria, A.-L. Biance, R. Fulcrand, and L. Bocquet, Nano Lett. 12, 4037 (2012).
  * Yeh  _et al._ (2014) H.-C. Yeh, M. Wang, C.-C. Chang, and R.-J. Yang, Israel J. Chem. 54, 1533 (2014).
  * Ma  _et al._ (2017) Y. Ma, J. Guo, L. Jia, and Y. Xie, ACS Sensors 3, 167 (2017).
  * Xiong  _et al._ (2019) T. Xiong, K. Zhang, Y. Jiang, P. Yu, and L. Mao, Sci. China Chem. 62, 1346 (2019).
  * Poggioli, Siria, and Bocquet (2019) A. R. Poggioli, A. Siria, and L. Bocquet, J. Phys. Chem. B 123, 1171 (2019).
  * Cengio and Pagonabarraga (2019) S. D. Cengio and I. Pagonabarraga, J. Chem. Phys. 151, 044707 (2019).
  * Kavokine, Netz, and Bocquet (2020) N. Kavokine, R. R. Netz, and L. Bocquet, Annu. Rev. Fluid Mech. 53 (2020), 10.1146/annurev-fluid-071320-095958.
  * Noh and Aluru (2020) Y. Noh and N. R. Aluru, ACS Nano 14, 10518 (2020).
  * Fertig, Valiskó, and Boda (2020) D. Fertig, M. Valiskó, and D. Boda, Phys. Chem. Chem. Phys. 22, 19033 (2020).
  * Nernst (1888) W. Nernst, Zeitschrift für Physikalische Chemie 2 (1888), 10.1515/zpch-1888-0174.
  * Planck (1890) M. Planck, Annalen der Physik und Chemie 275, 161 (1890).
  * Boda and Gillespie (2012) D. Boda and D. Gillespie, J. Chem. Theor. Comput. 8, 824 (2012).
  * Boda  _et al._ (2014) D. Boda, R. Kovács, D. Gillespie, and T. Kristóf, J. Mol. Liq. 189, 100 (2014).
  * Boda (2014) D. Boda, in  _Ann. Rep. Comp. Chem._, Vol. 10, edited by R. A. Wheeler (Elsevier, 2014) Chap. 5 Monte Carlo Simulation of Electrolyte Solutions in Biology: In and Out of Equilibrium, pp. 127–163.
  * Fertig  _et al._ (2017) D. Fertig, E. Mádai, M. Valiskó, and D. Boda, Hung. J. Ind. Chem. 45, 73 (2017).
  * Gummel (1964) H. K. Gummel, IEEE Transactions on electron devices 11, 455 (1964).
  * Matejczyk  _et al._ (2017) B. Matejczyk, M. Valiskó, M.-T. Wolfram, J.-F. Pietschmann, and D. Boda, J. Chem. Phys. 146, 124125 (2017).
  * Mádai  _et al._ (2017) E. Mádai, M. Valiskó, A. Dallos, and D. Boda, J. Chem. Phys. 147, 244702 (2017).


# Forecasting Time-to-Collision from Monocular Video:
Feasibility, Dataset, and Challenges

Aashi Manglik, Xinshuo Weng, Eshed Ohn-Bar and Kris M. Kitani1 1Authors are
affiliated with the Robotics Institute, Carnegie Mellon University,
Pittsburgh, PA 15213, USA. {amanglik, xinshuow, eohnbar,
<EMAIL_ADDRESS>Eshed Ohn-Bar is now at the Max Planck Institute for
Intelligent Systems.

###### Abstract

We explore the possibility of using a single monocular camera to forecast the
time to collision between a suitcase-shaped robot being pushed by its user and
other nearby pedestrians. We develop a purely image-based deep learning
approach that directly estimates the time to collision without the need of
relying on explicit geometric depth estimates or velocity information to
predict future collisions. While previous work has focused on detecting
immediate collision in the context of navigating Unmanned Aerial Vehicles, the
detection was limited to a binary variable (_i.e._ , collision or no
collision). We propose a more fine-grained approach to collision forecasting
by predicting the exact time to collision in terms of milliseconds, which is
more helpful for collision avoidance in the context of dynamic path planning.
To evaluate our method, we have collected a novel dataset of over 13,000
indoor video segments each showing a trajectory of at least one person ending
in a close proximity (a near collision) with the camera mounted on a mobile
suitcase-shaped platform. Using this dataset, we do extensive experimentation
on different temporal windows as input using an exhaustive list of state-of-
the-art convolutional neural networks (CNNs). Our results show that our
proposed multi-stream CNN is the best model for predicting time to near-
collision. The average prediction error of our time to near-collision is
$0.75$ seconds across the test videos. The project webpage can be found at
https://aashi7.github.io/NearCollision.html.

##  I INTRODUCTION

Automated collision avoidance technology is an indispensable part of mobile
robots. As an alternative to traditional approaches using multi-modal sensors,
purely image-based collision avoidance strategies [1, 2, 3, 4] have recently
gained attention in robotics. These image-based approaches use the power of
large data to detect immediate collision as a binary variable - collision or
no collision. In this work, we propose a more fine-grained approach to predict
the exact time to collision from images, with a much longer prediction
horizon.

Figure 1: Forecasting time to near-collision between a suitcase-shaped robot
being pushed by its user and nearby pedestrian.

A method frequently used [5, 6] for forecasting time to collision is to track
[7, 8, 9, 10] the 3D location of the surrounding pedestrians and extrapolate
their trajectories using a constant velocity model. When it is possible to use
high quality depth imaging devices, this type of physics-based modeling can be
very accurate. However, physics-based approach can also be prone to failure in
the presence of sensor noise and uncertainty in detection of nearby
pedestrians. Small mistakes in the estimation of depth (common to low-cost
depth sensors) or noise in 2D bounding box detection (common to image-based
object detection algorithms) can be misinterpreted to be very large changes of
velocity. Many physics-based approaches can be brittle in the presence of such
noise. Accordingly, errors in either pedestrian detection, tracking or data
association can result in very bad future trajectory estimates. Other more
advanced physics-based models and decision-theoretic models also depend
heavily on accurate state estimates of nearby people and can be significantly
influenced by sensor and perception algorithm noise. We propose to address the
issue of sensor noise and perception algorithm noise by directly estimating
the time to collision from a sequence of images. Using a monocular camera is
an alternate solution to using inexpensive RGB-D cameras like Kinect which are
inapplicable for outdoor use.

To create a dataset for learning time to collision, we designed a training
prototype that facilitates efficient annotation of large amounts of data. It
is both unnatural and infeasible to record or insist that people actually
collide with the mobile platform to collect large scale data. As an
abstraction, we define the presence of a person within a 1 meter radius around
the mobile platform as a near-collision. If a person is present within this
radius, we mark it as a near-collision that should be forecasted using an
earlier segment of video. The proposed approach is designed for a mobile robot
that is being pushed by a person with visual impairment as shown in Fig. 1.
The goal of the system is to forecast the time to near-collision, few seconds
before the near-collision event. While most of the existing datasets for human
trajectory prediction are from a fixed overhead camera [11], [12], our dataset
of 13,658 video segments targets the first-person view which is more intuitive
for mobile robots. In the work on robust multi-person tracking from mobile
platforms [7], the dataset is egocentric at walking speed but with 4788 images
it is insufficient to deploy the success of deep learning.

We formulate the forecasting of time to near-collision as a regression task.
To learn the mapping from spatial-temporal motion of the nearby pedestrians to
time to near-collision, we learn a deep network which, takes a sequence of
consecutive frames as input and outputs the time to near-collision. To this
end, we evaluate and compare two popular video network architectures in the
literature: (1) The high performance of the image-based network architectures
makes it appealing to reuse them with as minimal modification as possible.
Thus, we extract the features independently from each frame using an image-
based network architecture (_e.g._ , VGG-16) and then aggregate the features
across the temporal channels; (2) It is also natural to directly use a 3D
ConvNet (_e.g._ , I3D [13]) to learn the seamless spatial-temporal features.
Moreover, it is a nontrivial task to decide how many past frames should form
the input. Thus, we do extensive experimentation on different temporal windows
as input using aforementioned video network architectures. Our results show
that our proposed multi-stream CNN trained on the collected dataset is the
best model for predicting time to near-collision.

In summary, the contributions of our work are as follows: (1) We contribute a
large dataset of 13,658 egocentric video snippets of humans navigating in
indoor hallways. In order to obtain ground truth annotations of human pose,
the videos are provided with the corresponding 3D point cloud from LIDAR; (2)
We explore the possibility of forecasting the time to near-collision directly
from monocular images; (3) We provide an extensive analysis on how current
state-of-the-art video architectures perform on the task of predicting time to
near-collision on the proposed dataset and how their performance varies with
different temporal windows as input.

##  II Related Work

Monocular-Based Collision Avoidance. Existing monocular collision avoidance
systems mostly focus on avoiding the immediate collision at the current time
instant. Learning to fly by crashing [1] presented the idea of supervised
learning to navigate an unmanned aerial vehicle (UAV) in indoor environments.
The authors create a large dataset of UAV crashes and train an AlexNet [14]
with single image as input to predict from one of these three actions - go
straight, turn left or turn right. Similarly, DroNet [2] trains a ResNet-8
[15] to safely navigate a UAV through the streets of the city. DroNet takes
the images from an on-board monocular camera on UAV and outputs a steering
angle along with the collision probability.

While predicting an action to avoid the immediate collision is useful, it is
more desirable to predict a possible collision in the short future. To
anticipate traffic accidents, Kataoka et al. [16] constructed a near-miss
incident database of traffic scenes annotated with near-miss incident
duration. By passing a video input to a quasi-recurrent neural network, their
approach outputs a probability that an accident will occur in future. On an
average, their approach is shown to anticipate a near-miss incident or
accident 3.65 seconds in advance. While their study explored how early can a
collision be anticipated, the exact time to collision was not predicted.
Therefore, our work focuses on forecasting the exact time to a possible
collision occurring within next 6 seconds, which will be helpful for collision
avoidance in the context of dynamic path planning [5], [17], [18].

Predicting Time to Collision by Human Trajectory. Instead of predicting the
time to collision directly from images, one can also predict the human
trajectories [19, 20, 21, 11, 22] as the first step, then the time to
collision can be predicted based on the speed and heading directions of all
the surrounding pedestrians. [11] introduces a dynamic model for human
trajectory prediction in a crowd scene by modeling not only the history
trajectory but also the surrounding environment. [20] proposes a LSTM model to
learn general human movement pattern and thus can predict better future
trajectory. As there are many plausible ways that humans can move, [19]
proposes to predict diverse future trajectories instead of a deterministic
one. [21] proposes an attention model, which captures the relative importance
of each surrounding pedestrian when navigating in the crowd, irrespective of
their proximity.

However, in order to predict future trajectory reliably, these methods rely on
accurate human trajectory history. This usually involves multi-people
detection [23, 24, 25, 26] and tracking, and thus has two major disadvantages:
(1) Data association is very challenging in crowded scenarios. Small mistakes
can be misinterpreted to be very large changes of velocity, resulting in very
bad trajectory estimate; (2) Robust multi-person tracking from mobile platform
is often time-consuming. For example, [7] takes 300ms to process one frame on
a mobile GPU, making it impossible to achieve real-time collision forecasting.

In contrast, our approach can predict the time to collision directly from a
sequence of images, without requiring to track the surrounding pedestrians
explicitly. We demonstrate that our data-driven approach can implicitly learn
reliable human motion and also achieve real-time time to collision.

Learning Spatio-Temporal Feature Representation. Existing video architectures
for spatio-temporal feature learning can be split into two major categories.
To leverage the significant success from image-based backbone architectures
(_e.g._ , VGG and ResNet) pre-trained on large-scale image datasets such as
ImageNet [27] and PASCAL VOC [28], methods in the first category reuse the 2D
ConvNet to extract features from a sequence of images with as minimal
modification as possible. For example, [29] proposes to extract the image-
based features independently from each frame using GoogLeNet and then apply a
LSTM [30] on the top for feature aggregation for video action recognition.

The second category methods explore the use of 3D ConvNets for video tasks
[31, 32, 33, 34, 35] that directly operate 3D spatio-temporal kernels on video
inputs. While it is natural to use 3D ConvNets for spatio-temporal feature
learning, 3D ConvNets are unable to leverage the benefits of ImageNet
pretraining easily and often have huge number of parameters which makes it
most likely to overfit on small datasets. Recently, the two-stream inflated 3D
ConvNet (I3D) [13] is proposed to mitigate these disadvantages by inflating
the ImageNet-pretrained 2D weights to 3D. Also, the proposed large-scale video
dataset, Kinetics, has shown to be very successful for 3D kernel pre-training.

To validate how current state-of-the-art video-based architectures perform on
the novel task of predicting time to near-collision on the proposed dataset,
we evaluate methods from both categories in our experiments.

##  III DATASET

TABLE I: Public Video Datasets with Egocentric Viewpoint Dataset |  Number of near-collision video sequences |  Structure of scenes |  Setup for recording |  Modalities Available
---|---|---|---|---
Ours (Near-collision) |  13,685 |  Indoor hallways |  Suitcase |  Stereo Depth + 3D Point Cloud
UAV crashing [1] |  11,500 |  Indoor hallways |  UAV |  Monocular
DroNet [2] |  137 |  Inner-city |  Bicycle |  Monocular
Robust Multi-Person Tracking [7] |  350 |  Busy inner-city |  Chariot |  Stereo Depth

We sought to analyze a large-scale, real-world video dataset in order to
understand challenges in prediction of near-collision events. However, based
on our survey, existing datasets had a small number of interaction events as
reported in Table I and lacked diversity in the capture settings. Therefore,
in order to train robust CNN models that can generalize across scenes, ego-
motion, and pedestrian dynamics, we collected an extensive dataset from a
mobile perspective. Next, we describe our hardware setup and methodology for
data collection.

###  III-A Hardware Setup

The mobile platform used for data collection is shown in Fig. 2, and includes
a stereo camera and LIDAR sensor. While during inference we only utilize a
monocular video, the stereo camera serves two purposes. First, it provides a
depth map to help in automatic ground truth annotation. Second, it doubles the
amount of training data by providing both a left and right image perspective
which can be used as separate training samples. However, during the process of
automatic data annotation, it was observed that the depth maps from stereo
camera are insufficient for extracting accurate distance measurements. In
particular, when the pedestrian is close to camera the depth values are
missing at corresponding pixels due to motion blur. To ensure stable and
reliable depth annotations, we utilize a LIDAR sensor which is accurate to
within a few centimeters. The images and corresponding 3D point clouds are
recorded at the rate of 10Hz.

###  III-B Camera-LIDAR Extrinsic Calibration

We use the camera and the LIDAR for automatic ground truth label generation.
The two sensors can be initially calibrated with correspondences [36, 37]. An
accurate calibration is key to obtaining the 3D position of surrounding
pedestrians and annotating the large number of videos in our dataset. Let $R$
and $t$ denote the rotation matrix and the translation vector defining the
rigid transformation between the LIDAR to the camera frame and $K$ the
$3\times 3$ intrinsic matrix of camera. Then, the LIDAR 3D coordinates
$(x,y,z)$ can be related to a pixel in the image with coordinates
$(U,V)=(\frac{u}{w},\frac{v}{w})$ using following transformation:

$\begin{bmatrix}u\\\ v\\\ w\end{bmatrix}=K[R\mid-R^{T}t]\begin{bmatrix}x\\\
y\\\ z\\\ 1\end{bmatrix}$ (1)

Given this calibration, we can now project LIDAR points onto the image and
obtain estimated depth values for the image-based pedestrian detection.

###  III-C Data Collection and Annotation

The platform is pushed through three different university buildings with low-
medium density crowd. Our recorded videos comprise of hallways of varying
styles. We experimented with several techniques for obtaining pedestrian
detections [23, 24] in the scene from the image and LIDAR data. As 2D person
detection is a well-studied problem, we found an image-based state-of-the-art
person detection (Faster-R-CNN [23]) to perform well in most cases, and
manually inspect and complete any missing detections or false positives. To
obtain the 3D position of each detected bounding box, we compute a median
distance of its pixels using the 3D point cloud. An illustration of the
resulting processing is shown in Fig. 3. Each image is annotated with a binary
label where a positive label indicates the presence of at least one person
within a meter distance from setup. We understand that some people in
immediate vicinity moving in the same direction as camera might not be
important for collision, but due to close proximity of 1 meter should still be
recognized and planned over carefully.

Figure 2: Left: We show an assistive suitcase with a camera sensor and speaker
to guide people. Right: We show the corresponding suitcase-shaped training
prototype mounted with stereo camera and LIDAR for data collection.

Now we want to estimate the time to near-collision, in terms of milliseconds,
based on a short temporal history of few RGB frames. Let us consider a tuple
of $N$ consecutive frames $(I_{1},I_{2},\ldots,I_{N})$ and using this sequence
as history we want to estimate if there is a proximity over the next 6
seconds. Since the framerate is $10$ fps, we look at the next 60 binary labels
in future annotated as $\\{label_{n+1},label_{n+2},\ldots,label_{n+60}\\}$. If
we denote the index of first positive label in this sequence of labels as $T$
then our ground truth time to near-collision is $t=\frac{T}{10}$ seconds.

Figure 3: Multi-modal ground truth generation. The two red bounding boxes
indicate people detected by Faster R-CNN. The green points are projected to
the image from the LIDAR, with the relative distance between LIDAR and person
shown as well.

###  III-D Comparison with Existing Datasets

In Table I, we compare our proposed dataset with existing datasets recorded
from egocentric viewpoint in terms of (1) number of near-collision video
sequences, (2) structure of scenes, and (3) setup used for recording. UAV
crashing [1] dataset is created by crashing the drone 11,500 times into random
objects. DroNet [2] has over 137 sequences of starting far way from an
obstacle and stopping when the camera is very close to it. The two main
reasons for collecting proposed dataset over existing datasets of UAV crashing
and DroNet are: (1) applicability to assistive suitcase system [6], and (2)
focus on pedestrian motion. While the dataset provided by Ess et al [7] suited
to our application, we find only $350$ near-collision instances making it
infeasible to deploy CNNs.

##  IV APPROACH

Figure 4: Our model with VGG-16 as backbone where the final output is time to
collision denoted as $t$

Our goal is to predict the time at which at least one person is going to come
within a meter distance of the mobile setup using only a monocular image
sequence of $N$ frames. The video is recorded at $10$ fps and thus a sequence
of $N$ frames, including the current frame and past $(N-1)$ frames, correspond
to the history of $(N-1)/10$ seconds. We first provide a formal definition of
the task and then the details of network architecture for reproducibility.

###  IV-A Problem Formulation - Classification or Regression?

Learning time to near-collision can be formulated as a multi-class
classification into one of the 60 classes where $i^{th}$ class corresponds to
time range between $((i-1)/10,i/10]$ seconds. The disadvantage of training it
as a classification task is that all the mispredictions are penalized equally.
For example, let us consider two different mispredictions given the same
ground truth of 0.5 seconds - one where the network categorized it into the
class $(0.6,0.7]$ and other when the network predicted $(5.5,5.6]$. The multi-
class cross-entropy loss on both of these will be equal while we want the
latter to be penalized much more than the former. One of the solutions is to
design a differentiable loss function which keeps this preference in mind.
Another solution is to formulate it as a regression problem and use the mean-
squared error as the loss function. In this paper, we formulated it as a
regression problem as follows:

$t=f(I_{1},I_{2},\ldots,I_{N})\text{ where }t\in[0,6]$

###  IV-B Network Architecture

VGG-16 [38] is a 16-layer convolutional neural network which won the
localization task in ImageNet Challenge 2014. It used parameter efficient
$3\times 3$ convolutional kernels pushing the depth to 16 weight layers. It
was shown that its representations generalize well to other datasets achieving
state-of-the-art results. We propose a multi-stream VGG architecture as shown
in Fig. 4 where each stream takes a $224\times 224$ RGB frame as input to
extract spatial features . These spatial features are then concatenated across
all frames preserving the temporal order and then fed into a fully-connected
layer to output time to collision.

Feature Extraction from VGG-16 We extracted the features of dimensions
7$\times$7$\times$512 from the last max pool layer of VGG-16. These features
pass through an additional convolution layer to reduce the feature size to
7$\times$7$\times$16 and then flattened. These flattened features for each
frame are concatenated into a vector and fed into the successive fully-
connected layer of size 2048 which finally leads to a single neuron denoted as
$t$ in Fig. 4.

In this network, the convolutional operators used spatial 2D kernels. A major
question in current video architectures is whether these 2D kernels should be
replaced by 3D spatio-temporal kernels [13]. To address this question, we also
experimented with 3D spatio-temporal kernels and report the results in
following section.

Training N-stream VGG We initialized the VGG-16 network using ImageNet-
pretrained weights. As the ImageNet dataset does not have a person class, we
fine-tuned the network weights on PASCAL VOC [28] dataset. Using these weights
as initialization, we train a multi-stream architecture with shared weights.
The network is trained using the following loss function.

$L_{MSE}=\frac{1}{2}||t_{true}-f(I_{1},I_{2},\ldots,I_{N})||^{2}$

Here, $L$ is the mean squared loss between the predicted time, _i.e._ ,
$f(I_{1},I_{2},\ldots,I_{N})$ and ground truth time denoted as $t_{true}$.
The loss is optimized using mini-batch gradient descent of batch size 24 with
the learning rate of $0.001$. The training data is further doubled by applying
horizontal flip transformation.

##  V EXPERIMENTAL EVALUATION

We now describe our evaluation procedure to decide the optimum temporal window
as input on two different video network architectures. We further compare the
performance with strong collision prediction baselines.

###  V-A Different temporal windows as input

A single image can capture spatial information but no motion characteristics.
Thus, we propose to use a sequence of image frames as history. By feeding $N$
image frames, we consider a history of $(N-1)/10$ seconds. The temporal window
of input frames was gradually increased from 2 frames (0.1 sec) to 9 frames
(0.8 sec). To quantify the performance, we measure the mean absolute error
(MAE) for the predictions on the test set and the standard deviation in error.
From Table II, it is empirically concluded to use a temporal window of 0.5
seconds, i.e, 6 frames in multi-stream VGG for most accurate predictions.

TABLE II: Distribution of absolute error (mean $\pm$ std) on near-collision dataset using different number of frames Number of frames |  Multi-stream VGG |  I3D
---|---|---
1 |  0.879 $\pm$ 0.762s |  0.961 $\pm$ 0.707s
2 |  0.828 $\pm$ 0.739s |  0.879 $\pm$ 0.665s
3 |  0.826 $\pm$ 0.647s |  0.914 $\pm$ 0.659s
4 |  0.866 $\pm$ 0.696s |  0.811 $\pm$ 0.642s
5 |  0.849 $\pm$ 0.734 |  0.845 $\pm$ 0.658s
6 |  0.753 $\pm$ 0.687s |  0.816 $\pm$ 0.663s
7 |  0.757 $\pm$ 0.722s |  0.848 $\pm$ 0.733s
8 |  0.913 $\pm$ 0.732s |  0.811 $\pm$ 0.647s
9 |  0.817 $\pm$ 0.738s |  0.855 $\pm$ 0.670s

###  V-B Experimental comparison of architectures

We show a comparison of the performance of multi-stream VGG model and
baselines including state-of-the-art methods in Table III.

TABLE III: Distribution of absolute error (mean $\pm$ std) on regression task compared with different baselines Method |  Mean (in s) |  Std (in s)
---|---|---
Constant baseline ($\mathbb{E}[y_{true}]$) |  1.382 |  0.839
Tracking + Linear Model [6] |  1.055 |  0.962
DroNet [2] |  1.099 |  0.842
Gandhi et al [1] |  0.884 |  0.818
Single Image VGG-16 |  0.879 |  0.762
I3D (4 frames) [13] |  0.811 |  0.642
Multi-stream VGG (6 frames) |  0.753 |  0.687
Figure 5: Example prediction results over consecutive frames. The
discontinuities in the ground truth occur when the nearest pedestrian exits
the view. As shown, the multi-stream VGG approach better adheres to the ground
truth compared to I3D both when the pedestrian is far and near the camera,
while providing smoother and more temporally-consistent predictions.  Figure
6: Predictions on four different test videos

####  V-B1 Constant Baseline

On the training data of 12,620 samples, we compute the mean time to near-
collision denoted by $\mathbb{E}[y_{true}]$ as a weak baseline. For each test
input, we predict $\mathbb{E}[y_{true}]$ which was found to be $2.23$ seconds.

####  V-B2 Tracking followed by constant velocity model

In dynamic environments, pedestrians are often tracked using a stereo camera
or LIDAR. By saving few previous locations (0.5-2 seconds), a linear
regression fit is used to predict the velocity vector of person [6, 5]. This
velocity vector is then linearly extrapolated to predict where the pedestrian
will be over the next 6 seconds and the corresponding accuracy is reported in
Table III. A major disadvantage in this method is the need for image-based
tracking which is less reliable at low framerate of $10$ fps.

####  V-B3 Deep learning for collision avoidance

Collision avoidance using deep learning has been previously proposed in [1]
and [2]. Gandhi et al [1] created a UAV crash dataset to train AlexNet for
classifying the current image into one of these three categories of drone
policy: go straight, turn left or turn right. For learning time to near-
collision using their approach, we take a single image as input and use
AlexNet architecture with ImageNet-pretrained weights as initialization. The
only difference lies in the output layer which is a single neuron regression
instead of a three-neuron classifier. Our multi-stream VGG outperformed
current-frame AlexNet as reported in Table III.

ResNet-8 architecture used in DroNet [2] takes in the single image and after
the last ReLU layer splits into two fully-connected streams outputting
steering angle and collision probability respectively. To experiment with
their learning approach, we used ResNet-8 architecture with only one output,
i.e., time to near-collision. The performance is close to the constant
velocity prediction model as reported in Table III and thus it can be seen
that it is unable to leverage real-world data. One of the reasons for its low
performance could be the unavailability of ImageNet-pretrained weights for
ResNet-8 architecture and thus it has to be trained from scratch on our
dataset which is much smaller than the Udacity’s car-driving dataset of 70,000
images used for training steering angle stream in DroNet.

####  V-B4 I3D for action classification in videos

Two-Stream Inflated 3D ConvNet (I3D) [13] is a strong baseline to learn a task
from videos. All the $N\times N$ filters and pooling kernels in ImageNet-
pretrained Inception-V1 network [39] are inflated with an additional temporal
dimension to become $N\times N\times N$. I3D has two streams - one trained on
RGB inputs and other on optical flow inputs. To avoid adding the latency of
optical flow computation for real-time collision forecasting, we only used the
RGB stream of I3D. We fine-tuned the I3D architecture which was pre-trained on
Kinetics Human Action Video dataset [13] on our near-collision dataset by
sending $N$ RGB frames as input where $N=\\{1,2,\ldots,8,9\\}$ as reported in
Table II. The outermost layer is modified from 400-neuron classifier to
1-neuron regressor. Since our $N$-frame input is smaller than the original
implementation on 64-frame input, we decreased the temporal stride of last
max-pool layer from 2 to 1. While 6 input frames were found to be the best for
proposed multi-stream VGG network, we experimented again with the optimal
history on I3D. The performance of I3D with varying number of input frames is
reported in Table II. For $N=4,6,8$, I3D is found to give the best results
among 1-9 frames though our multi-stream VGG prediction for $N=6$ outperformed
the I3D prediction in best case.

###  V-C Qualitative Evaluation

From the plots shown in Fig. 5 we can observe that the predictions given by
multi-stream VGG on 6 frames give smoother output as compared to undesired
fluctuations in I3D output. We also qualitatively show in Fig. 6 the
comparison of time to near-collision predicted by our method vs the ground
truth.

###  V-D Forecast Horizon

In Table IV, we report the error at different time-to-collision intervals. It
is easiest to forecast the collision a second away and hence the error is
least. In general, as forecast horizon increases it keeps on getting more
difficult to forecast the exact time-to-collision.

TABLE IV: How error in prediction varies with time-to-collision? Ground Truth Time-to-Collision Interval (in s) |  Number of Test Samples |  Mean Absolute Error in Predictions (in s)
---|---|---
0-1 |  348 |  0.6027
1-2 |  211 |  0.6446
2-3 |  188 |  0.7547
3-4 |  147 |  0.7189
4-5 |  86 |  0.9502
5-6 |  58 |  1.8369

##  VI Discussion

We return to the question posed in introduction, ’Is it possible to predict
the time to collision from a single camera?’. The answer is that the proposed
model is able to leverage spatio-temporal cues for predicting the time to
collision. Also, we observed that the multi-stream network of shared weights
performed the task of collision forecasting better than I3D on the proposed
dataset.

Regarding the temporal window of input history, it is evident that using a
sequence of images has a considerable benefit over prediction from single
frame only. Though a temporal window incorporating 0.5 seconds of history
performed best for our task, we do not observe a piecewise monotonic relation
between input frames and error in prediction. This observation aligns with the
performance of constant velocity model where accuracy in prediction does not
necessarily increase or decrease with the temporal footprint of past
trajectory.

During data collection, the pushing speed of suitcase varies between $0.2-1.5$
meters per second which is similar in range to human walking speed. We
understand that the proposed model might not generalize well when there are
significant changes in camera’s height, pushing speed of suitcase or structure
of the scenes in comparison to provided dataset. In the scenarios where
assistive robot operates in a constrained domain like museums and airports,
the proposed approach will be suitable for predicting time to collision
requiring only a low-cost monocular camera. We believe that the proposed
approach could be extended for outdoor use given enough training data of
outdoor scenes.

One of the scenarios where the current approach fails is when a person is
moving away from the user in the same direction. The pedestrian detector
detects the person and thus the image is passed through the proposed multi-
stream network. The network outputs an inaccurate estimate of time-to-
collision within the trained forecast horizon. In the future work, we can
instead try to output two values - mean and variance in time-to-collision. The
variance can act as an indicator on how confident or reliable is the
prediction. To train the output tuple of mean and variance, the negative of
gaussian log-likelihood loss can be minimized over the training data which
includes such failure cases.

## ACKNOWLEDGMENT

This work was sponsored in part by NIDILRR (90DPGE0003), JST CREST
(JPMJCR14E1) and NSF NRI (1637927).

## References

  * [1] D. Gandhi, L. Pinto, and A. Gupta, “Learning to Fly by Crashing,” _IROS_ , 2017.
  * [2] A. Loquercio, A. I. Maqueda, C. R. del-Blanco, and D. Scaramuzza, “DroNet: Learning to Fly by Driving,” _Robotics and Automation Letters_ , 2018.
  * [3] F.-H. Chan, Y.-T. Chen, Y. Xiang, and M. Sun, “Anticipating accidents in dashcam videos,” in _Asian Conference on Computer Vision_. Springer, 2016, pp. 136–153.
  * [4] Q. Fu, N. Bellotto, H. Wang, F. C. Rind, H. Wang, and S. Yue, “A Visual Neural Network for Robust Collision Perception in Vehicle Driving Scenarios,” _arXiv:1904.02074_ , 2019.
  * [5] M. Phillips and M. Likhachev, “SIPP: Safe Interval Path Planning for Dynamic Environments,” _ICRA_ , 2011.
  * [6] S. Kayukawa, K. Higuchi, J. Guerreiro, S. Morishima, Y. Sato, K. Kitani, and C. Asakawa, “BBeep: A Sonic Collision Avoidance System for Blind Travellers and Nearby Pedestrians,” _CHI_ , 2019.
  * [7] A. Ess, B. Leibe, K. Schindler, and L. van Gool, “Robust Multi-Person Tracking from a Mobile Platform,” _TPAMI_ , 2009.
  * [8] A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft, “Simple Online and Realtime Tracking,” _ICIP_ , 2016.
  * [9] S. Sharma, J. A. Ansari, J. K. Murthy, and K. M. Krishna, “Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Tracking,” _ICRA_ , 2018.
  * [10] X. Weng and K. Kitani, “A Baseline for 3D Multi-Object Tracking,” _arXiv:1907.03961_ , 2019.
  * [11] S. Pellegrini, A. Ess, K. Schindler, and L. V. Gool, “You’ll Never Walk Alone: Modeling Social Behavior for Multi-Target Tracking,” _ICCV_ , 2009\.
  * [12] A. Sadeghian, V. Kosaraju, A. Gupta, S. Savarese, and A. Alahi, “TrajNet: Towards a Benchmark for Human Trajectory Prediction,” _arXiv:1805.07663_ , 2018.
  * [13] J. Carreira and A. Zisserman, “Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset,” _CVPR_ , 2017.
  * [14] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet Classification with Deep Convolutional Neural Networks,” _NIPS_ , 2012.
  * [15] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” _CVPR_ , 2016.
  * [16] T. Suzuki, H. Kataoka, Y. Aoki, and Y. Satoh, “Anticipating Traffic Accidents with Adaptive Loss and Large-Scale Incident DB,” _CVPR_ , 2018.
  * [17] A. Kushleyev and M. Likhachev, “Time-Bounded Lattice for Efficient Planning in Dynamic Environments,” _ICRA_ , 2009.
  * [18] A. Vemula, K. Muelling, and J. Oh, “Path Planning in Dynamic Environments with Adaptive Dimensionality,” _Ninth Annual Symposium on Combinatorial Search_ , 2016.
  * [19] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi, “Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks,” _CVPR_ , 2018.
  * [20] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese, “Social LSTM: Human Trajectory Prediction in Crowded Spaces,” _CVPR_ , 2016.
  * [21] A. Vemula, K. Muelling, and J. Oh, “Social Attention: Modeling Attention in Human Crowds,” _ICRA_ , 2018.
  * [22] B. D. Ziebart, N. Ratliff, G. Gallagher, C. Mertz, K. Peterson, J. A. Bagnell, M. Hebert, A. K. Dey, and S. Srinivasa, “Planning-Based Prediction for Pedestrians,” _IROS_ , 2009.
  * [23] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” _NIPS_ , 2015.
  * [24] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [25] X. Weng, S. Wu, F. Beainy, and K. Kitani, “Rotational Rectification Network: Enabling Pedestrian Detection for Mobile Vision,” _WACV_ , 2018.
  * [26] X. Weng and K. Kitani, “Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud,” _arXiv:1903.09847_ , 2019.
  * [27] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” _CVPR_ , 2009.
  * [28] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The Pascal Visual Object Classes (VOC) Challenge,” _IJCV_ , 2010.
  * [29] J. Y.-H. Ng, M. J. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici, “Beyond Short Snippets: Deep Networks for Video Classification,” _CVPR_ , 2015.
  * [30] S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” _Neural Computing_ , 1997.
  * [31] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning SpatioTemporal Features with 3D Convolutional Networks,” _ICCV_ , 2015.
  * [32] C. Feichtenhofer, A. Pinz, and R. P. Wildes, “SpatioTemporal Residual Networks for Video Action Recognition,” _NIPS_ , 2016.
  * [33] Z. Qiu, T. Yao, and T. Mei, “Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks,” _ICCV_ , 2017.
  * [34] K. Simonyan and A. Zisserman, “Two-stream Convolutional Networks for Action Recognition in Videos,” _NIPS_ , 2014.
  * [35] X. Weng and K. Kitani, “Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading,” _arXiv:1905.02540_ , 2019.
  * [36] S. Kato, S. Tokunaga, Y. Maruyama, S. Maeda, M. Hirabayashi, Y. Kitsukawa, A. Monrroy, T. Ando, Y. Fujii, and T. Azumi, “Autoware on Board: Enabling Autonomous Vehicles with Embedded Systems,” _ICCPS_ , 2018.
  * [37] L. Zhou, Z. Li, and M. Kaess, “Automatic Extrinsic Calibration of a Camera and a 3D LiDAR Using Line and Plane Correspondences,” _IROS_ , 2018.
  * [38] K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” _ICLR_ , 2015.
  * [39] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going Deeper with Convolutions,” _CVPR_ , 2015.