# On the Role of Non-Localities in Fundamental Diagram Estimation

Jing Liu  Fangfang Zheng<EMAIL_ADDRESS>Boxi Yu  Saif Jabari  School of
Transportation and Logistics, National Engineering Laboratory of Integrated
Transportation Big Data Application Technology, National United Engineering
Laboratory of Integrated and Intelligent Transportation, Southwest Jiaotong
University Western Hi-tech Zone Chengdu, Sichuan 611756, P.R. China  Division
of Engineering, New York University Abu Dhabi, Saadiyat Island PO Box 129188 -
Abu Dhabi, United Arab Emirates

###### Abstract

We consider the role of non-localities in speed-density data used to fit
fundamental diagrams from vehicle trajectories. We demonstrate that the use of
anticipated densities results in a clear classification of speed-density data
into stationary and non-stationary points, namely, acceleration and
deceleration regimes and their separating boundary. The separating boundary
represents a locus of stationary traffic states, i.e., the fundamental
diagram. To fit fundamental diagrams, we develop an enhanced cross entropy
minimization method that honors equilibrium traffic physics. We illustrate the
effectiveness of our proposed approach by comparing it with the traditional
approach that uses local speed-density states and least squares estimation.
Our experiments show that the separating boundary in our approach is invariant
to varying trajectory samples within the same spatio-temporal region,
providing further evidence that the separating boundary is indeed a locus of
stationary traffic states.

###### keywords:

Non-Local Density-speed Samples , Enhanced Cross-entropy Loss , Equilibrium
Speed-density Relationship , Acceleration

††journal: Transportation Research Part C

## Highlights

  * 1.

We approach the modeling of the FD as a binary classification problem,
specifically focusing on distinguishing acceleration regimes.

  * 2.

Our proposed method introduces a set of empirical samples that consider non-
localities, allowing us to model the FD while maintaining its inherent
physical equilibrium.

  * 3.

We propose an enhanced cross-entropy loss function for fitting the FD model
using these non-locality samples.

  * 4.

Our approach consistently generates an invariant FD across various datasets
collected at the same location.

##  1 Introduction

Fundamental diagrams (FDs) date back to Greenshields et al. (1935), and they
remain an area of continuing research interest, particularly, methods of
fitting FDs (Bramich et al., 2022, 2023; Liu et al., 2023). The enduring
appeal of FDs lies in their ability to summarize the functional relationship
between macroscopic traffic flow variables in equilibrium. FDs serve as
essential inputs for continuous traffic flow models (Makridis et al., 2020)
and find extensive applications in areas such as traffic control (Wang et al.,
2014; Heydecker and Addison, 2011; D. Frejo et al., 2019), capacity analysis
(Qin and Wang, 2023), traffic state estimation (Thodi et al., 2022; Zheng et
al., 2018), prediction (Liu et al., 2021; Yang et al., 2021), and
identification (Kalair and Connaughton, 2021).

In the context of FD fitting methods, all approaches in the literature attempt
to relate variables locally, i.e., they relate estimated densities and speeds
in the same spatio-temporal location (e.g.,
($k_{\mathrm{A}}$,$v_{\mathrm{A}}$) and ($k_{\mathrm{B}}$,$v_{\mathrm{B}}$)
depicted in Fig. 1). We will refer to samples created this way as local
density ($K$) speed ($V$) samples, or LKV for short. The common fitting
techniques use some variant of least square estimation (LSE) to determine the
parameters of a specific FD model. The fitting methods implicitly assume that
the dependent variable (i.e., speed) is independent across different densities
and follows a Gaussian distribution. Additionally, they assume that the
Gaussian distribution has a mean that aligns with the equilibrium speed as
modeled by the FD, and a constant variance that is independent of density
(although few papers have allowed for heteroskedasticity (Jabari et al.,
2014), a constant variance remains a common implicit assumption). The
estimation of model parameters is then performed using maximum likelihood
estimation (MLE) (Bramich et al., 2022).

The nature of LKV samples is incompatible with the implicit assumptions of
LSE-based techniques summarized above. First, traffic flow cannot
instantaneously transition from one equilibrium state to another. The non-
stationary transition speed is influenced by the desired speed, which is
determined by downstream traffic conditions (as depicted in Fig. 1, where the
density in region B exceeds that in the upstream region A, causing upstream
vehicles to adjust their speed in anticipation of traffic in region B rather
than region A), thus violating the implicit independence assumption. Several
researchers attempted to address this issue by excluding non-stationary data
from FD fitting (Jabari et al., 2014). This results in removal of a large
amount of data from the sample, which is impractical. As the LKV samples do
not contain equilibrium information, other approaches consider external
information (e.g., the arrival rate (Cassidy, 1998)) or make assumptions about
the relationship between equilibrium state and the coefficient of speed
variation (Seo et al., 2019). Second, although the speed near a certain
density can be modeled as a Gaussian distribution, the parameters (mean and
variance) vary with density (Helbing, 1997; Jabari and Liu, 2013). Therefore,
despite the popularity of LSE methods applied to LKV data for FD modeling,
these drawbacks cannot be ignored.

Figure 1: Speed adaptation in response to downstream traffic conditions

Estimating the desired speed of traffic flow under a specific density is
challenging due to the hidden nature of this variable, which can only be
observed in rare instances of ideal equilibrium conditions. In most
situations, traffic conditions are dynamic, and it takes time for the traffic
speed to align with the corresponding desired speed, resulting in hysteresis
(Laval, 2011). However, the relationship between the desired speed and the
current speed can be deduced from the acceleration. When the desired speed
associated with the anticipated density is lower than the current speed,
negative acceleration occurs in the traffic flow, whereas positive
acceleration is observed when the desired speed exceeds the current speed.
Hence, acceleration/deceleration behavior solely depends on current and
desired speeds in each sample independently. Thus, adding
acceleration/deceleration behavior overcomes the dependence drawback mentioned
above. It also naturally delineates stationary and non-stationary data in the
macroscopic relation, thereby allowing for fitting of true equilibrium
relations (i.e., FDs) to the data.

In this paper, we propose a novel way to create samples by substituting the
current density with the anticipated density and utilize the acceleration
property as a label, to model the FD. We will refer to these samples as non-
local density (K) speed (V) samples, or NLKV for short. By doing so, we can
frame the equilibrium relationship as a binary classification problem, using
the sign of the acceleration label. Additionally, we propose an enhanced
cross-entropy (ECE) loss as the optimization objective to minimize and fit the
FD model parameters using the NLKV samples.

The rest of this paper is organized as follows. Sect. 2 provides a review of
the relevant research on the estimation of empirical samples, FD models, and
optimization methods employed to fit the models. In Sect. 3, we present the
assumptions of this study and outline the structure of NLKV samples and the
ECE loss. Sect. 4 describes the method for creating the NLKV samples from
trajectories, including the estimation of continuous speed, density, and
acceleration fields of the trajectories, as well as the anticipated density
for each spatio-temporal location. The NLKV samples fitted with the ECE loss
(NLKV+ECE) can be regarded as a parallel approach to fitting the LKV samples
using LSE (LKV+LSE). Sect. 5 compares the two approaches, examining the
properties of the samples themselves and the fitting results with FD models.
Finally, Sect. 6 concludes this paper and provides additional discussions.

##  2 Related Research

The development of FDs of real-world traffic involves the use of local samples
created from field-collected data within small spatio-temporal locations,
including estimated flow rates ($q$), densities ($k$) and macroscopic speeds
($v$). Each $(v,k,q)$ tuple represents a sample point in empirical $v-k$ (or
$q-k$ and $q-v$) plots. When point sensors like loop detectors are used to
collect the field data, the spatial extent is defined by the detectors
themselves. One estimates flow rates from the traffic counts, densities from
sensor occupancies (Papageorgiou and Vigos, 2008), and speed from individual
vehicle speeds (Treiber and Kesting, 2013) (or utilizing the relation $q=kv$).
In the case of trajectory data, the $(v,k,q)$ values for each spatio-temporal
location can be estimated using Edie’s method (Edie, 1963).

Macroscopic relations can be classified into two types: physical and empirical
models. Physical models involve the development of theories of traffic
dynamics, such as car-following behavior (Gazis et al., 1961; Newell, 1961;
Del Castillo and Benitez, 1995; Zheng et al., 2023) and continuous fluid
approximations (Greenberg, 1959), to establish functional forms for the
macroscopic relations. This approach is advantageous as it allows traffic flow
theories to be tested against empirical observations. Physical models also
allow for the development of FD relationships (i.e., macroscopic relations of
equilibrium traffic states). On the other hand, empirical models formulate
analytical expressions for macroscopic relationships with free parameters that
are adjusted to provide a best fit to the empirical samples. These free
parameters are then interpreted in terms of traffic flow properties, such as
free-flow speed, capacity, and jam density, to provide the analytical
expression with phenomenological meaning (e.g., Greenshields et al. (1935);
Underwood (1961); Drake et al. (1965)). A subclass of empirical models uses
data-mining methods (e.g., Sun et al. (2014); Hadiuzzaman et al. (2018); Shi
et al. (2021); Bramich et al. (2023)), sometimes referred to as non-parametric
models, where the free parameters generally lack interpretability in the
context of traffic flow properties. However, when provided with sufficient
empirical samples, these methods can demonstrate remarkable flexibility and
fitting performance. In a study conducted by Bramich et al. (2022), it was
observed that Sun’s empirical-based model (Sun et al., 2014) exhibited the
best fitting performance among the 50 models using loop detector data from 25
cities. This finding underscores the potential and efficacy of empirical
models.

Most techniques utilize a variant of LSE to fit macroscopic models to
empirical samples. This approach involves determining the model parameters by
minimizing the sum of squared estimation errors. The accuracy of the fitting
result is influenced by the distribution of the empirical samples. Qu et al.
(2015) acknowledged the existence of different speed distributions at various
densities and proposed a density-weighted optimization function.
Traditionally, the error distribution is assumed to follow a Gaussian
distribution. However, it is now recognized that this assumption is
insufficient for modeling LKV samples due to the complex nature of the
observed noise, which can be attributed to various factors such as driver
behavior, traffic dynamics, and hysteresis effects (Bramich et al., 2022).

##  3 Preliminary and Methodology

###  3.1 The equilibrium state and NLKV samples

Equilibrium states in traffic are stationary states (i.e. when the vehicles
are not accelerating or decelerating). Classically, one also assumes
homogeneous drivers (e.g., identical reaction times, identical desired speeds,
and identical safety distances). Our focus is on the former: stationary
states. Naturally, the desired speed associated with an increasing density
follows a monotonically decreasing and continuous function (Bramich et al.,
2022; Liu et al., 2023; van Wageningen-Kessels et al., 2015). As drivers
anticipate the traffic state they are about to enter, they assign an
anticipated density to the current traffic state. Subsequently, this
anticipated density determines the corresponding desired speed (modeled by an
FD). As a result, drivers proactively adjust their current speed to approach
the desired speed. This adjustment is manifested as follows: if the current
speed exceeds the desired speed corresponding to the anticipated density, the
driver decelerates (2(a)). Conversely, they accelerate if the current speed
falls below the desired speed (2(b)). In cases where the current speed is
close to the desired speed, slight acceleration or deceleration may occur due
to random factors.

(a) deceleration

(b) acceleration

Figure 2: Illustration of speed adaptation and anticipation: (a) deceleration;
(b) acceleration

We propose a novel way to create samples (NLKV samples) from trajectory data,
considering non-localities, to model the equilibrium FD relationship. The NLKV
samples have anticipated density ($k^{\mathrm{a}}$) and current speed ($v$) as
variables and their acceleration/deceleration state (0 if accelerating, 1 if
decelerating) as labels. Table 1 provides the comparison of the structural
differences between the LKV and NLKV samples.

Table 1: Data form of LKV and NLKV samples Sample | LKV |  | NLKV |  |
---|---|---|---|---|---
| Density | Speed | Anticipated density | Speed | Acceleration sign (0 or 1)
Spatio-temporal |  |  |  |  |
location 1 | $k_{1}$ | $v_{1}$ | $k^{\mathrm{a}}_{1}$ | $v_{1}$ | $y_{1}$
$\vdots$ | $\vdots$ | $\vdots$ | $\vdots$ | $\vdots$ | $\vdots$
Spatio-temporal |  |  |  |  |
location $n$ | $k_{n}$ | $v_{n}$ | $k^{\mathrm{a}}_{n}$ | $v_{n}$ | $y_{n}$

The following assumptions serve as guiding principles in our proposed model:

  1. 1.

The anticipated density $k^{\mathrm{a}}$ is determined by the current traffic
state. Drivers accurately anticipate $k^{\mathrm{a}}$ based on their current
speed $v$, downstream density, and a transition time $t^{\mathrm{m}}$.

  2. 2.

The desired speed $v^{\mathrm{a}}$ corresponding to $k^{\mathrm{a}}$ exists
and is determined by the function $v^{\mathrm{a}}=f(k^{\mathrm{a}})$, where
$f(\cdot)$ is the $v-k$ relationship to be determined.

  3. 3.

The desired speed $v^{\mathrm{a}}$ corresponding to the anticipated density
$k^{\mathrm{a}}$ is adopted as the target for the current traffic to adjust
its speed accordingly.

  4. 4.

The density variation from $k$ to $k^{\mathrm{a}}$ is monotone, either
increasing or decreasing.

Assumption 1) states that drivers possess accurate anticipation skills,
enabling them to estimate $k^{\mathrm{a}}$ correctly. We divide space and time
into discrete intervals. For a specific space interval $j$ and time interval
$i$, $k^{\mathrm{a}}$ is estimated as the density of space interval
$j+vt^{\mathrm{m}}$ and time interval $i+t^{\mathrm{m}}$. The parameter
$t^{\mathrm{m}}$ is inspired by the definition of decision sight distance,
which refers to the distance required to detect an unexpected source or hazard
on a roadway, recognize its potential threat, select an appropriate speed and
path, and safely and efficiently complete the required maneuver (Green Boook,
2004). Decision sight distance is estimated as
$v_{\mathrm{des}}t^{\mathrm{m}}$, where $v_{\mathrm{des}}$ is the design speed
(McGee, 1979). It varies from 10.2 to 14.5 seconds depending on the type of
road (i.e., rural, suburban, or urban roads) (Green Boook, 2004).

Assumptions 2) and 3) reflect the definition of stationary states and the
dynamical characteristics of traffic flow, respectively. With the monotone
decreasing property of the $v-k$ relationship, assumption 4) ensures that the
acceleration $a$ during the adjustment of traffic flow from the current speed
$v$ to the anticipated desired speed $v^{\mathrm{a}}=f(k^{\mathrm{a}})$
maintains a consistent sign. If $v$ is perceived to be higher than
$v^{\mathrm{a}}$ corresponding to $k^{\mathrm{a}}$, vehicles decelerate and
continue to decelerate until they reach speed $v^{\mathrm{a}}$. If $v$ is
perceived to be lower than $v^{\mathrm{a}}$, vehicles accelerate and continue
to accelerate until they reach speed $v^{\mathrm{a}}$. During this process, no
alternating acceleration and deceleration behaviors occur. This is summarized
by the following condition:

$\mathrm{sgn}\leavevmode\nobreak\
(v^{\mathrm{a}}-v)=\mathrm{sgn}\leavevmode\nobreak\ a,$ (1)

where $a$ represents the acceleration during the adjustment from $v$ to
$v^{\mathrm{a}}$ and $\mathrm{sgn}$ is the sign function. According to the
assumptions, the desired speed $v^{\mathrm{a}}$ under the anticipated density
$k^{\mathrm{a}}$ satisfies

$\left\\{\begin{aligned} v^{\mathrm{a}}&<v\mbox{ if and only if }a<0\\\
v^{\mathrm{a}}&>v\mbox{ if and only if }a>0\\\ v^{\mathrm{a}}&=v\mbox{ if and
only if }a=0\\\ \end{aligned}\right..$ (2)

Eq. 2 says that the desired speed of vehicles is higher than the current speed
if they have positive acceleration. Conversely, if they are decelerating, the
current speed is higher than the desired speed. At equilibrium, the
acceleration is 0, signifying that there is no change in speed over time. In
this state, the current speed matches the desired speed corresponding to the
anticipated density.

By plotting the anticipated density against the current speed and labeling the
data points with the corresponding sign of acceleration (referred to as the
NLKV samples), a scatter plot can be generated to visually represent the
relationship between anticipated density, current speed, and the sign of
acceleration. The boundary in the scatter plot is determined by the
anticipated density and its corresponding desired speed, which serves as a
separator, dividing the samples into two distinct regions. The region above
the boundary represents the samples where the current speed is higher than the
desired speed (indicating negative acceleration), while the region below the
boundary represents samples where the current speed is lower than the desired
speed (indicating positive acceleration). By modeling the separation curve for
samples with positive and negative accelerations, the desired speed
corresponding to the anticipated density represented by the $v-k$ relationship
can be obtained. The points on the curve indicate the scenario where
acceleration is zero, representing a state of equilibrium.

However, it is important to note that the NLKV samples created in the field
traffic may not exhibit ideal properties described above. In practical
scenarios, the regions of acceleration and deceleration often overlap as the
system approaches stationary. This overlap can be attributed to unobserved
features (random features) and the heterogeneity of vehicles in the system.

Taking these factors into consideration, we further make the assumption that
the probability of deceleration, denoted as $\pi$, is related to the
difference between the current speed $v$ and the desired speed
$v^{\mathrm{a}}$, following a logistic distribution. Specifically, we have

$p(y|z)=\pi^{y}(1-\pi)^{1-y}=\bigg{(}\frac{1}{1+e^{-z}}\bigg{)}^{y}\bigg{(}\frac{e^{-z}}{1+e^{-z}}\bigg{)}^{1-y},$
(3)

where $y$ represents the sign of acceleration, with $y=1$ indicating a
negative acceleration in the current traffic flow, and $y=0$ representing a
positive acceleration. The variable $z$ is the difference between the current
speed and their desired speed under the anticipated density $k^{\mathrm{a}}$.
Eq. 3 implies that the deceleration probability $\pi<0.5$ if $z<0$, whereas
$\pi>0.5$ if $z>0$. Additionally, if $v\ll v^{\mathrm{a}}$, then $\pi\to 0$.
This indicates a very small probability of negative acceleration and a high
probability of positive acceleration. Conversely, if $v\gg v^{\mathrm{a}}$, we
have that $\pi\to 1$, i.e., an overwhelmingly high probability of
deceleration.

We now consider the $v-k$ relationship
$v^{\mathrm{a}}=f(k^{\mathrm{a}};\vec{\theta})$, where $\vec{\theta}$, a
vector, represents the parameters of $f$. The sign of acceleration given
$z=v-v^{\mathrm{a}}$, a binary variable, naturally follows a binomial
distribution with trial parameter 1 and success probability

$\pi=\frac{1}{1+e^{-(v-f(k^{\mathrm{a}};\vec{\theta}))}}$ (4)

(i.e., a Bernoulli distribution):
$\\{y|f(k^{\mathrm{a}};\vec{\theta}),v\\}\sim\mathrm{Binomial}(1,\pi)$. The
probability density function of this distribution is given by

$p(y|f(k^{\mathrm{a}};\vec{\theta}),v)=\pi^{y}(1-\pi)^{1-y}=\bigg{(}\frac{1}{1+e^{-(v-f(k^{\mathrm{a}};\vec{\theta}))}}\bigg{)}^{y}\bigg{(}\frac{e^{-(v-f(k^{\mathrm{a}};\vec{\theta}))}}{1+e^{-(v-f(k^{\mathrm{a}};\vec{\theta}))}}\bigg{)}^{1-y},\leavevmode\nobreak\
y\in\\{0,1\\}.$ (5)

###  3.2 The loss function with NLKV samples

In the last subsection, we have established the NLKV samples and determined
the distribution of the sign of their acceleration $y$. The $v-k$ relationship
can be modeled as a binary classification problem, with $y$ as the dependent
variable. To estimate the parameters of a specific FD model, the likelihood
function can be formulated assuming that $y$ follows a Bernoulli distribution
with a deceleration probability $\pi$, that is
$\\{y|f(k^{\mathrm{a}};\vec{\theta}),v\\}\sim\mathrm{Binomial}(1,\pi)$, based
on the discussion above. From Eq. 5, the likelihood function, denoted as
$L(\vec{\theta})$, is defined as the probability of observing the given sample
(of size $m$) $\\{k^{\mathrm{a}}_{i},v_{i},y_{i}\\}_{i=1}^{m}$ conditioned on
the model parameters $\vec{\theta}$ and the $v-k$ relationship $f(\cdot)$:

$L(\vec{\theta}):=\prod_{i=1}^{m}\bigg{(}\frac{1}{1+e^{-(v_{i}-f(k^{\mathrm{a}}_{i};\vec{\theta}))}}\bigg{)}^{y_{i}}\bigg{(}\frac{e^{-(v_{i}-f(k^{\mathrm{a}}_{i};\vec{\theta}))}}{1+e^{-(v_{i}-f(k^{\mathrm{a}}_{i};\vec{\theta}))}}\bigg{)}^{1-y_{i}}.$
(6)

By maximizing the likelihood function with respect to the parameter
$\vec{\theta}$, we can estimate the model parameters that best fit the
observed samples and the FD model using maximum likelihood estimation (MLE).
This is equivalent to minimizing the negative log-likelihood function, that
is, solving the following optimization problem:

$\displaystyle\underset{\vec{\theta}}{\mathrm{minimize}}\leavevmode\nobreak\
-\log\big{(}L(\vec{\theta})\big{)}$
$\displaystyle=\sum_{i=1}^{m}\bigg{(}y_{i}\log\Big{(}1+e^{-(v_{i}-f(k^{\mathrm{a}}_{i};\vec{\theta}))}\Big{)}+(1-y_{i})\big{(}v_{i}-f(k^{\mathrm{a}}_{i};\vec{\theta})\big{)}+(1-y_{i})\log\Big{(}1+e^{-(v_{i}-f(k^{\mathrm{a}}_{i};\vec{\theta}))}\Big{)}\bigg{)}$
$\displaystyle=\sum_{i=1}^{m}\bigg{(}(1-y_{i})\big{(}v_{i}-f(k^{\mathrm{a}}_{i};\vec{\theta})\big{)}+\log\Big{(}1+e^{-(v_{i}-f(k^{\mathrm{a}}_{i};\vec{\theta}))}\Big{)}\bigg{)}.$
(7)

Sect. 3.2 sums the cross-entropy loss of each sample and can, therefore, be
influenced by bias in the sample of acceleration and deceleration data (as can
be seen in the second line after canceling similar terms). To address this
issue, we enhance the objective function by introducing sample rate weighting.
Additionally, we aim to increase the convergence speed towards the minimum of
Sect. 3.2 and impose a greater penalty in case of a false classification when
there is a significant absolute distance between the current speed and the
anticipated desired speed. Therefore, we propose to minimize the following
enhanced cross entropy (ECE) loss to estimate the parameters of the FD model:

$\underset{\vec{\theta}}{\mathrm{minimize}}\leavevmode\nobreak\
\frac{1}{m}\sum_{i=1}^{m}\bigg{(}\omega
y_{i}\log\Big{(}1+e^{-(v_{i}-f(k^{\mathrm{a}}_{i};\vec{\theta}))}\Big{)}+(1-\omega)(1-y_{i})\big{(}v_{i}-f(k^{\mathrm{a}}_{i};\vec{\theta})\big{)}+(1-\omega)(1-y_{i})\log\Big{(}1+e^{-(v_{i}-f(k^{\mathrm{a}}_{i};\vec{\theta}))}\Big{)}\bigg{)},$
(8)

where

$\omega:=\frac{\\#\left\\{y_{i}|y_{i}=0\right\\}}{\\#\left\\{y_{i}\right\\}}=\frac{\\#\left\\{y_{i}|y_{i}=0\right\\}}{m}$
(9)

represents the fraction of positive accelerations in the sample. We also
divided the loss by the number of samples $m$, which does not affect the
optimization problem but allows us to interpret the loss function obtained in
this way.

##  4 NLKV Samples Estimated from Trajectories

In order to estimate the NLKV samples from trajectories, it is necessary to
first estimate the macroscopic fields for speed, density, and acceleration.
These fields should exhibit a sufficient level of smoothness to accurately
capture the overall patterns and trends in the traffic flow. Subsequently, the
anticipated density for each spatio-temporal state can be obtained based on
the previously estimated macroscopic fields. The anticipated density
represents the density that vehicles expect to reach at each specific spatio-
temporal location, considering their current speeds. Finally, the estimated
anticipated density can be paired with the corresponding speed and sign of
acceleration for each state of the traffic flow, enabling the formation of the
NLKV samples.

###  4.1 Estimation of the speed, density, and acceleration fields

We denote the spatio-temporal domain by
$\mathcal{D}=\mathcal{X}\times\mathcal{T}$ representing a road section
$\mathcal{X}$ and time interval $\mathcal{T}$, which we divide into
homogeneous spatio-temporal intervals $\mathcal{D}(i,j)$, where
$i\in\\{0,...,I\\}$, $j\in\\{0,...,J\\}$, and $i$,$j$ are integers, as shown
in Fig. 3. The intervals $\mathcal{D}(i,j)$ are $\Delta x$ long in the space
dimension and $\Delta t$ long in the time dimension. We consider sliding step
sizes to capture non-localities: we denote the _sliding_ space and time step
sizes by $x^{\mathrm{s}}\ll\Delta x$ and $t^{\mathrm{s}}\ll\Delta t$,
respectively. Thus, the spatio-temporal interval $\mathcal{D}(i,j)$ covers
$it^{\mathrm{s}}$ to $it^{\mathrm{s}}+\Delta t$ in time and $jx^{\mathrm{s}}$
to $jx^{\mathrm{s}}+\Delta x$ in space. With the sliding intervals, we can
discretize the entire spatio-temporal domain $\mathcal{D}$ into $I\times J$
subdomains with overlaps, where

$I=\left\lfloor\frac{T-\Delta
t}{t^{\mathrm{s}}}\right\rfloor,\leavevmode\nobreak\
J=\left\lfloor\frac{X-\Delta x}{x^{\mathrm{s}}}\right\rfloor,$ (10)

and $T:=|\mathcal{T}|$ and $X:=|\mathcal{X}|$. Due to the overlapping spatio-
temporal subdomains, there are partially duplicated trajectories between
$\mathcal{D}(i,j)$ and $\mathcal{D}(i+1,j)$ as well as between
$\mathcal{D}(i,j)$ and $\mathcal{D}(i,j+1)$. This enhances the smoothness of
macroscopic traffic flow parameters estimated from subdomains in both time and
space. As $t^{\mathrm{s}}$ and $x^{\mathrm{s}}$ approach zero in the limit, a
continuous field of traffic flow states can be obtained within the
corresponding $\\{\mathcal{D}(i,j)\\}$.

Figure 3: Discretization of the spatio-temporal domain

According to the general definition proposed by Edie (1963), the macroscopic
parameters $k_{i,j}$, and $v_{i,j}$ of each subdomain $\mathcal{D}(i,j)$ can
be calculated as

$k_{i,j}=\frac{\sum_{n=1}^{N_{i,j}}t_{n}}{A_{i,j}},$ (11)

and

$v_{i,j}=\frac{\sum_{n=1}^{N_{i,j}}x_{n}}{\sum_{n=1}^{N_{i,j}}t_{n}},$ (12)

where $N_{i,j}$ is the total number of vehicles in $\mathcal{D}(i,j)$. $x_{n}$
and $t_{n}$ are the travel distance and travel time of vehicle $n$ in
$\mathcal{D}(i,j)$, respectively. $A_{i,j}=\Delta x\Delta t$ is the area of
$\mathcal{D}(i,j)$. The speed field $\mathcal{V}\in\mathbb{R}^{I\times J}$ and
density field $\mathcal{K}\in\mathbb{R}^{I\times J}$ can be obtained by
applying Eq. 11 and Eq. 12 to each subdomain: $\mathcal{V}(i,j):=v_{i,j}$,
$\mathcal{K}(i,j):=k_{i,j}$.

The acceleration $a_{i,j}$ in $\mathcal{D}(i,j)$ can be approximated as

$a_{i,j}\approx\frac{v_{i+1,j+b}-v_{i,j}}{t^{\mathrm{s}}},$ (13)

where

$b:=\left\lfloor\frac{v_{i,j}t^{\mathrm{s}}}{x^{\mathrm{s}}}\right\rfloor$
(14)

is the estimated number of spatial intervals traversed at the current speed
within $t^{\mathrm{s}}$ units of time. Eq. 13 assumes that the speed change
between two adjacent temporal steps is linear. The acceleration field obtained
by traversing all $\mathcal{D}(i,j)$ is represented by the acceleration field
$\mathcal{A}\in\mathbb{R}^{I\times J}$, which is given by

$\mathcal{A}\left(i,j\right)=\begin{cases}a_{i,j}&\leavevmode\nobreak\ i\leq
I-1,j\leq J-b\\\ \emptyset&\mbox{otherwise}\end{cases}.$ (15)

Finally, we denote by $y(i,j)$ the sign of acceleration of subarea
$\mathcal{D}(i,j)$ i.e.,

$y(i,j)=\begin{cases}0&\mathcal{A}(i,j)>0\\\ 1&\mathcal{A}(i,j)<0\\\
\emptyset&\mathcal{A}(i,j)=\emptyset\end{cases}.$ (16)

###  4.2 Estimation of the anticipated density

As per assumption 1), the anticipated density $k^{\mathrm{a}}$ of
$\mathcal{D}(i,j)$ is estimated as $k_{i+t^{\mathrm{m}},j+vt^{\mathrm{m}}}$,
i.e., we have a shift in indices given as $i\mapsto i+t^{\mathrm{m}}$ and
$j\mapsto j+vt^{\mathrm{m}}$. To align with the discrete spatio-temporal
location, the anticipated density is adjusted as

$k^{\mathrm{a}}_{i,j}=k_{i+\left\lfloor\frac{t^{\mathrm{m}}}{t^{\mathrm{s}}}\right\rfloor
t^{\mathrm{s}},j+\left\lfloor\frac{v_{i,j}t^{\mathrm{m}}}{x^{\mathrm{s}}}\right\rfloor
x^{\mathrm{s}}}.$ (17)

That is, $k^{\mathrm{a}}_{i,j}=k_{i^{\prime},j^{\prime}}$, where
$i^{\prime}:=i+\left\lfloor t^{\mathrm{m}}/t^{\mathrm{s}}\right\rfloor
t^{\mathrm{s}}$ and $j^{\prime}:=j+\left\lfloor
v_{i,j}t^{\mathrm{m}}/x^{\mathrm{s}}\right\rfloor x^{\mathrm{s}}$.
Consequently, the _anticipated densities_ field,
$\mathcal{K}^{\mathrm{a}}\in\mathbb{R}^{I\times J}$, is

$\mathcal{K}^{\mathrm{a}}(i,j)=\begin{cases}k^{\mathrm{a}}_{i,j}&i\leq
I-\left\lfloor\dfrac{t^{\mathrm{m}}}{t^{\mathrm{s}}}\right\rfloor
t^{\mathrm{s}},\ j\leq
J-\left\lfloor\dfrac{v_{i,j}t^{\mathrm{m}}}{x^{\mathrm{s}}}\right\rfloor
x^{\mathrm{s}}\\\ \emptyset&\mbox{otherwise}\end{cases}.$ (18)

With these definitions, we obtain a complete NLKV sample
$\left\\{\mathcal{K}^{\mathrm{a}}(i,j),\mathcal{V}(i,j),y(i,j)\right\\}$ for
the entire spatio-temporal domain. We only samples from domain intervals
$\mathcal{D}(i,j)$ with $i\leq\min\left(I-1,I-\left\lfloor
t^{\mathrm{m}}/t^{\mathrm{s}}\right\rfloor t^{\mathrm{s}}\right)$ and
$j\leq\min\left(J-b,J-\left\lfloor vt^{\mathrm{m}}/x^{\mathrm{s}}\right\rfloor
x^{\mathrm{s}}\right)$ for FD modeling.

##  5 Comparison between the Fitting Approaches of LKV+LSE and NLKV+ECE

This section provides a systematic comparison between fitting FDs using
(classical) LKV samples and the proposed NLKV samples. For the former, we use
traditional LSE; for the latter, we employ the ECE minimization method
developed in Sect. 3.2. We introduce the field trajectory datasets used to
perform the comparisons in Sect. 5.1. Moving on to Sect. 5.2, we estimate the
LKV and NLKV samples for different trajectory datasets. We then compare the
distribution of LKV and NLKV features for a specific dataset. Sect. 5.3
introduces our FD models and the LSE optimization model used for FD fitting.
The fitting results of the LKV+LSE and NLKV+ECE approaches with the same FD
model are presented in Sect. 5.4. Finally, in Sect. 5.5, we compare the
fitting results of different FD models using the LKV+LSE and NLKV+ECE
approaches.

###  5.1 The field trajectory dataset

This study utilizes four trajectory datasets obtained from different freeway
road links. The datasets are as follows:

  * 1.

Dataset 1: This dataset is obtained from the Hanshin Expressway link #11,
which was collected in the Zen Traffic Data (ZTD) source (Hanshin Expressway
Company, 2018). In the ZTD dataset, link #11 is labeled as L001. Dataset 1
specifically corresponds to the trajectory dataset L001F001, representing one
hour of data collection for this particular link.

  * 2.

Dataset 2: This dataset is obtained from the Hanshin Expressway link #4, also
collected from the ZTD source. In the ZTD dataset, link #4 is labeled as L002.
Dataset 2 corresponds to the trajectory dataset L002F001, representing one
hour of data collection for this particular link.

  * 3.

Dataset 3: This dataset is obtained from the Next Generation Simulation
Program (NGSIM, United States Department of Transportation (2006)),
specifically collected from the US101 link.

  * 4.

Dataset 4: Trajectories from the NGSIM, specifically collected from the I80
link.

We begin by estimating the speed, density, and acceleration fields for each of
the four trajectory datasets. The spatio-temporal domain is divided into
subdomains with a time interval of $\Delta t=50s$ and a spatial interval of
$\Delta x=300m$. This division ensures sufficient coverage for estimating the
macroscopic traffic flow parameters within each subdomain. To balance
continuity in the parameter fields and computational efficiency, smaller
sliding steps in space and time are chosen, with $t^{\mathrm{s}}=2s$ and
$x^{\mathrm{s}}=3m$. For the anticipated density estimation, the transition
time $t^{\mathrm{m}}$ of 12 seconds is selected, which falls within the
recommended range (Green Boook, 2004). The speed ($\mathcal{V}$), density
($\mathcal{K}$), and acceleration ($\mathcal{A}$) fields for the four
datasets, developed using Eq. 11-Eq. 15, are presented in Fig. 4.

(a) Dataset 1: $\mathcal{V}$

(b) Dataset 1: $\mathcal{K}$

(c) Dataset 1: $\mathcal{A}$

(d) Dataset 2: $\mathcal{V}$

(e) Dataset 2: $\mathcal{K}$

(f) Dataset 2: $\mathcal{A}$

(g) Dataset 3: $\mathcal{V}$

(h) Dataset 3: $\mathcal{K}$

(i) Dataset 3: $\mathcal{A}$

(j) Dataset 4: $\mathcal{V}$

(k) Dataset 4: $\mathcal{K}$

(l) Dataset 4: $\mathcal{A}$

Figure 4: The speed, density, and acceleration fields of the different
trajectory datasets: (a)-(c) the fields of dataset 1; (d)-(f) the fields of
dataset 2; (g)-(i) the fields of dataset 3; (j)-(l) the fields of dataset 4.

We note that the blanks present in (g)-(l) of Fig. 4 are due to the gaps in
the NGSIM data. The NGSIM dataset provides trajectories in separate 15-minute
intervals. Therefore, the fields are estimated individually for each 15-minute
subset of trajectories and then concatenated together. The blank areas in Fig.
4 represent the boundaries between each 15-minute subset of trajectories
resulting from the subdomain of the space-time field estimation. The blank
areas do not affect our tests.

###  5.2 LKV and NLKV samples of different trajectory datasets

We begin by comparing the properties of LKV and NLKV samples across datasets 1
to 4, as shown in Fig. 5. The figures in the first column of Fig. 5 depict the
scattered LKV samples of each subdomain $\mathcal{D}(i,j)$, representing
$\mathcal{K}(i,j)$–$\mathcal{V}(i,j)$ pairs, with colors corresponding to
their respective acceleration $\mathcal{A}(i,j)$. The presence of traffic
hysteresis can be clearly observed in (a), (d), (g), and (j) of Fig. 5 by the
absence of separation between acceleration regimes. This indicates that the
speeds are transient, which is not suitable for fitting equilibrium relations.

The second column of Fig. 5 depicts scatter-plots of
$\mathcal{K}^{\mathrm{a}}(i,j)$–$\mathcal{V}(i,j)$ pairs for each subdomain
$\mathcal{D}(i,j)$, with colors representing their corresponding acceleration
$\mathcal{A}(i,j)$. It is evident from these figures that with the non-
locality, a distinct separation between acceleration, deceleration, and nearly
steady state traffic flow area can be observed. The color gradient in this
column of figures correlates with the following driver behaviors: 1) Negative
(positive) acceleration occurs when a higher (lower) density than current is
anticipated. 2) A higher absolute value of acceleration occurs when
significant traffic state changes are expected.

The last column of figures in Fig. 5 displays the NLKV samples, which
represent scatter-plots of $\mathcal{K}^{\mathrm{a}}(i,j)$–$\mathcal{V}(i,j)$
pairs for each subdomain $\mathcal{D}(i,j)$, with colors indicating _the sign_
of their acceleration, denoted as $y(i,j)$. By classifying the empirical
samples into two distinct sets of acceleration and deceleration, the
separation becomes more pronounced. It is evident from Fig. 5 that NLKV
samples capture equilibrium states uninfluenced by hysteresis, making them
more suitable for FD modeling.

(a) Dataset 1: LKV sample pairs

(b) Dataset 1: NLKV sample pairs + $\mathcal{A}(i,j)$

(c) Dataset 1: NLKV sample pairs

(d) Dataset 2: LKV sample pairs

(e) Dataset 2: NLKV sample pairs + $\mathcal{A}(i,j)$

(f) Dataset 2: NLKV sample pairs

(g) Dataset 3: LKV sample pairs

(h) Dataset 3: NLKV sample pairs + $\mathcal{A}(i,j)$

(i) Dataset 3: NLKV sample pairs

(j) Dataset 4: LKV sample pairs

(k) Dataset 4: NLKV sample pairs + $\mathcal{A}(i,j)$

(l) Dataset 4: NLKV sample pairs

Figure 5: The speed-density relationship derived using LKV samples and NLKV
samples: (a), (d), (g), and (j) the LKV samples of datasets1- 4 colored by
acceleration; (b), (e), (h), and (k) the NLKV samples of datasets 1- 4 colored
by acceleration; (c), (f), (i), and (l) the NLKV samples of datasets 1- 4
indicated with acceleration and deceleration.

We further analyze the distribution of LKV and NLKV samples in dataset 1, and
illustrate the results in Fig. 6. As mentioned in Sect. 1, the traditional
fitting approach (LKV+LSE) implicitly assumes uniform variance in the sample.
6(a) illustrates that speed variance depends on traffic density and varies
substantially from one density to another (speeds in the figure are centered
around the mean). As a result, standard LSE-based fitting techniques are not
suitable for LKV data.

On the other hand, the underlying statistical assumption in our ECE
minimization method is that the signs of acceleration, given anticipated
density and speed, follow an identical Bernoulli distribution. To test this
assumption, we calculate deceleration probabilities using
$p=n_{\mathrm{d}}/n$, where $n_{\mathrm{d}}$ represents the number of
deceleration samples in the subarea, and $n$ is the total number of samples in
the same subarea. These probabilities are depicted in 6(b) for varying
anticipated density–speed pairs. We observe a similar pattern across different
anticipated density–speed pairs (the width of the separating band across the
diagonal does not change). Further, upon plotting the deceleration
probabilities after centering the speeds in 6(c), we observe logistic
distributions that appear invariant to anticipated density, which corroborates
the implicit statistical assumption underlying our approach.

(a) Observed heteroskedasticity in LKV data

(b) Probabilities of deceleration in NLKV data

(c) Probabilities of deceleration in NLKV data with center speeds

Figure 6: The distribution of LKV and NLKV samples: (a) centered speeds vs.
densities in LKV data; (b) probabilities of deceleration for varying
anticipated density–speed pairs in NLKV data; (c) probabilities of
deceleration with speeds centered around the $p=0.5$ points.

###  5.3 FD models and the LSE for comparison

To compare the performance of the LKV+LSE and the proposed NLKV+ECE
approaches, we select three FD models: Greenberg’s model (Greenberg, 1959),
Smulders’ model (Smulders, 1990), and the Franklin-Newell model (Newell, 1961;
Franklin, 1961). These are classical parametric models of the FD. We note that
Greenberg’s model does not respect boundedness axioms in (Del Castillo and
Benitez, 1995); the other two models do, as well as the concavity axiom. This
is immaterial in this study as our purpose is to illustrate the value of non-
localities in speed-density data and not to promote a particular parametric
model. The functional forms of these models are given below:

$\mbox{Greenberg's FD: }\leavevmode\nobreak\
\hat{v}(k;v_{0},k_{\mathrm{jam}})=v_{0}\ln\left(\frac{k_{\mathrm{jam}}}{k}\right),$
(19) $\mbox{Smulders' FD: }\leavevmode\nobreak\
\hat{v}(k;v_{0},k_{\mathrm{crit}},k_{\mathrm{jam}})=\left\\{\begin{array}[]{cl}v_{0}\left(1-\dfrac{k}{k_{\mathrm{jam}}}\right)&k<k_{\mathrm{crit}}\\\
v_{0}k_{\mathrm{crit}}\left(\dfrac{1}{k}-\dfrac{1}{k_{\mathrm{jam}}}\right)&k\geq
k_{\mathrm{crit}}\\\ \end{array},\right.$ (20)

$\mbox{Franklin-Newell FD: }\leavevmode\nobreak\
\hat{v}(k;v_{0},\lambda,k_{\mathrm{jam}})=v_{0}\left(1-e^{-\frac{\lambda}{v_{0}}\left(\frac{1}{k}-\frac{1}{k_{\mathrm{jam}}}\right)}\right),$
(21)

where $\hat{v}$ is a speed-density FD and $v_{0}$, $k_{\mathrm{jam}}$,
$k_{\mathrm{crit}}$ and $\lambda$ are interpretable parameters of FD models.
For the LKV+LSE approach, the fitting problem is given as

$\underset{\vec{\theta}}{\mathrm{minimize}}\leavevmode\nobreak\
\frac{1}{m}\sum_{i=1}^{m}\Big{(}v_{i}-f\left(k_{i};\vec{\theta}\right)\Big{)}^{2}.$
(22)

We emphasize that Eq. 22 is representative of the state of the art and not
just one FD fitting method. Our tests use parametric models for $f$, but $f$
can be any non-parametric model as well. In either case, what we see is that
Eq. 22 broadly overlooks equilibrium characteristics in the traffic. While our
comparisons are focused on parametric FDs, one can easily argue that our
findings generalize to any type of model $f$.

###  5.4 Fitting results of LKV+LSE and NLKV+ECE approaches

####  5.4.1 Results of the same trajectory dataset and same FD model

In this subsection, we fit the parameters of Smulders’ FD model on LKV and
NLKV samples from dataset 1. The fitting results of the two approaches, namely
LKV+LSE and NLKV+ECE, are presented in Fig. 7. The black solid curve
represents the fitting result using LKV samples, while the red dashed curve
represents the fitting result using the NLKV samples. In 7(a), we superimpose
the two fits on the LKV samples and we superimpose the two on NLKV samples in
7(b). We see a substantial difference in the estimated critical and jam
densities in the two figures. The FD fitted using LKV samples underestimates
the critical density and overestimates the jam density. The model fitted using
LKV samples also underestimates the free flow speed. The FD fitted using NLKV
samples separates acceleration and deceleration regimes better, suggesting
that it captures equilibrium conditions better.

(a) Fitted FDs with the LKV samples

(b) Fitted FDs with the NLKV samples

Figure 7: Comparison of fitting results using different fitting approaches:
(a) Fitted FDs with the LKV samples; (b) Fitted FDs with the NLKV samples.

####  5.4.2 Results of different trajectory datasets of the same road link and
the same FD model

We further compare the fitting results of the two methods using a different
sample of trajectories from the same location and time period, specifically,
L001 of ZTD. We employ the Smulders’ and the Franklin-Newell FD models. ZTD
provides five isolated detected trajectory datasets for link L001, each
collected over a duration of one hour. These datasets are denoted as L001F001,
L001F002, L001F003, L001F004, and L001F005. The macroscopic fields and LKV and
NLKV samples of L001F001 are shown in 4(a)-4(c) and 5(a) and 5(c). The speed
fields and samples of L001F002-L001F005 are presented in Fig. 8. It is evident
from Fig. 8 that all the datasets exhibit a transition from free flow to
congestion, but with varying characteristics. For instance, L001F002 and
L001F005 show a gradual buildup of congestion over time. L001F004 demonstrates
congestion occurring in specific spatial areas, while L001F003 experiences
congestion in both space and time. These variations in traffic characteristics
result in distinct shapes and patterns in both the LKV and NLKV samples. The
changes observed in the LKV samples have a significant impact on the
distributions of traffic density and, consequently, the fitted FDs. In
contrast, the NLKV samples display relatively stable separating boundaries
between acceleration and deceleration regimes. Therefore, modeling the
separating boundaries using NLKV samples, hence the locus of equilibrium
states, is invariant to the trajectory samples compared to modeling the mean
speeds using LKV samples. This is because the true FD is an emergent property
of road traffic (i.e., independent of its microscopic constituents).

(a) L001F002: $\mathcal{V}$

(b) L001F002: LKV sample points

(c) L001F002: NLKV sample points

(d) L001F003: $\mathcal{V}$

(e) L001F003: LKV sample points

(f) L001F003: NLKV sample points

(g) L001F004: $\mathcal{V}$

(h) L001F004: LKV sample points

(i) L001F004: NLKV sample points

(j) L001F005: $\mathcal{V}$

(k) L001F005: LKV sample points

(l) L001F005: NLKV sample points

Figure 8: The speed field, LKV and NLKV samples of L001F002-L001F005: (a)-(c)
L001F002; (d)-(f) L001F003; (g)-(i) L001F004; (j)-(l) L001F005.

We proceed to fit the parameters of the Smulders’ FD model and the Franklin-
Newell’s FD model on these five datasets (L001F001 to L001F005) using the
LKV+LSE and NLKV+ECE approaches, respectively (note that Smulders’ FD fitted
to L001F001 has already been conducted in Sect. 5.4.1). The fitting results
are as shown in Fig. 9, where the curves represent the fitted FDs on the
different L001 datasets. 9(a) and 9(c) depict Smulders’ FDs and the Franklin-
Newell FDs obtained using the LKV+LSE approach, while 9(b) and 9(e) display
the Smulders’ FDs and Franklin-Newell’s FDs obtained through the NLKV+ECE
approach. A careful observation of the figures that the FDs generated by the
LKV+LSE approach exhibit more diversity, whereas the FDs generated by the
NLKV+ECE approach demonstrate greater consistency and similarity across the
different datasets.

Irrespective of environmental variations and the heterogeneity of traffic
flow, the $v-k$ relationship should be determined solely by the
characteristics of the road traffic and should not be influenced by the
collected data. However, the fitting approach that lacks equilibrium
information (i.e., LKV+LSE) heavily relies on the detected features, leading
to varying fitted FDs for the same road link with different trajectories. On
the other hand, the FDs fitted using NLKV samples exhibit invariance to
trajectory samples from the same spatio-temporal region.

(a) Smulders’ FD model with the LKV+LSE approach

(b) Smulders’ FD model with the NLKV+ECE approach

(c) the Franklin-Newell FD fitting using the LKV+LSE

(d) the Franklin-Newell FD fitted using NLKV+ECE minimization (e)

Figure 9: Comparison of FDs fitted using different datasets from L001: (a)
Smulders’ FD model with the LKV+LSE approach; (b) Smulders’ FD model with the
NLKV+ECE approach; (c) the Franklin-Newell FD fitting using the LKV+LSE; (d)
the Franklin-Newell FD fitted using NLKV+ECE minimization.

###  5.5 Comparison of different FD models using LKV+LSE and NLKV+ECE
approaches

In the comparison of fitting results considering FD model selection, we first
fit Greenberg’s, Smulders’, and the Franklin-Newell FD models using LKV
samples from dataset 1, using the LKV+LSE approach. We then select the FD with
the least squared error as the optimal FD under this approach. Similarly, we
fit these three models using NLKV samples and select the FD with the least ECE
error as the optimal FD under the NLKV+ECE approach. The fitting results are
presented in Fig. 10 and Table 2.

Fig. 10 illustrates the fitted FDs obtained through different approaches,
capturing the characteristics of their corresponding samples. In 10(a), the
fitted FDs using the LKV+LSE approach exhibit a diverse free flow speed and a
higher jam density compared to the FDs in 10(b). Table 2 provides a
quantitative comparison of the fitted FDs, including the model parameters, as
well as LSE loss and the ECE loss. Among the FDs fitted by LKV+LSE approach,
Smulders’ FD has the lowest LSE loss, indicating a closer fit to the LKV
samples of dataset 1. Consequently, the Smulders model can be considered as
the most suitable model among the three models for the LKV samples of dataset
1. However, in the NLKV+ECE approach, the Franklin-Newell FD exhibits the
lowest ECE loss among all the fitted FDs. These results are logical since the
Franklin-Newell model accounts for driver behavior.

(a) LKV fitting

(b) NLKV fitting

Figure 10: The fitting results of the different FD models on the LKV and NLKV samples: (a) LKV fitting; (b) NLKV fitting. Table 2: The FD comparison of the different samples and models | LKV+LSE | NLKV+ECE
---|---|---
| $v_{f}$ | $v_{0}$ | $k_{c}$ | $k_{j}$ |  | LSE
---
loss
$v_{f}$ | $v_{0}$ | $k_{c}$ | $k_{j}$ |  | ECE
---
loss
Greenberg | - | 42.5 | - | 208.3 | 21.627 | - | 46.3 | - | 189.9 | 2.635
Smulders | 77.4 | - | 52 | 381.1 | 15.121 | 86.8 | - | 65 | 199.9 | 1.336
Franklin-Newell | 81.4 | - | - | 187.9 | 16.384 | 80.2 | - | - | 168.3 | 1.206

##  6 Concluding Remarks

This paper proposes a novel way to create samples from trajectory datasets,
which considers non-localities, to estimate the fundamental diagram (FD) of
traffic flow. The proposed non-local density-speed samples, dubbed NLKV
samples, incorporate anticipation by replacing the local density in density-
speed samples, dubbed LKV, with the anticipated density. Additionally, we
label the acceleration and deceleration behaviors of traffic flow.
Specifically, if the current traffic flow is moving faster than the desired
speed associated with its anticipated density, a negative acceleration is
observed. Conversely, a positive acceleration is inferred if the current
traffic flow is moving slower than the desired speed. If the traffic flow
tends to maintain speed, this signifies that the current speed corresponds to
the desired speed associated with the anticipated density.

By utilizing the NLKV samples, the FD modeling task can be formulated as a
binary classification problem. The FD is fitted to the separating boundary
between acceleration and deceleration regimes. We developed an enhanced cross
entropy (ECE) loss as our statistical model from basic statistical assumptions
that account for basic equilibrium traffic physics, which not only accounts
for the cost of misclassification but also addresses sample bias. This
approach stands in contrast to the conventional method employing LKV samples
and least squares estimation (LSE). The NLKV samples contain equilibrium
information and can be assumed to conform to the independent identically
distributed hypothesis required for maximum likelihood estimation (MLE).
Consequently, our proposed approach provides a more rational and
methodologically sound framework for FD fitting.

Through our comparison analysis of the LKV+LSE and NLKV+ECE approaches using
field trajectories, we observe distinct differences in their fitting
performance. Our investigation of four different links reveals that
accelerations differ across different pairs of anticipated density and current
speed, exhibiting a gradient ranging from acceleration to near-equilibrium and
ultimately to deceleration. Notably, the NLKV samples successfully segregate
the acceleration and deceleration regions, yielding a more accurate
representation of the “true” FD under equilibrium. In contrast, the LKV
samples are heavily influenced by hysteresis effects, leading to relations
that do not reflect equilibrium conditions.

Observing the same FD model fitted using these two approaches, we find that
parameters can differ substantially. Specifically, the FD model fitted using
LKV+LSE tends to underestimate the free flow speed and overestimate the jam
density compared to the model fitted using NLKV+ECE. Additionally, when
analyzing trajectories collected in the same road link but at different times,
we note that the FDs fitted by NLKV+ECE demonstrate invariance to the
trajectory sample compared to those fitted by LKV+LSE. Finally, we find that
the best-fit models may vary when using these two approaches.

The use of NLKV samples open up several interesting possibilities for future
research. Researchers can develop models to describe the separating boundaries
of NLKV, which could provide a more direct representation of the “true”
equilibrium FD instead of relying on a parametric models such as Greenberg’s,
Smulders’, and the Franklin-Newell model. Additionally, since NLKV samples
consider traffic dynamics, they can be coupled with continuous traffic flow
modeling to enhance prediction accuracy. Furthermore, we replace the LSE loss
with ECE loss, which circumvents criticisms regarding the Gaussian assumption
of the noise distribution. Notably, by eradicating the hysteresis effect
present in LKV samples, NLKV samples provide a promising avenue for improving
the noise distribution of speed.

## Acknowledgments

This work is supported by National Key R&D Program of China under grant
2021YFB1600100, and Chinese Scholarship Council under grant 202207000057. The
work is also supported by the NYUAD Center for Interacting Urban Networks
(CITIES), funded by Tamkeen under the NYUAD Research Institute Award CG001.

## Author Contribution Statement

The authors confirm contribution to the paper as follows: study conception and
design: J. Liu, F. Zheng, B. Yu, S. Jabari; data collection: J. Liu, B. Yu;
analysis and interpretation of results: J. Liu, F. Zheng, S. Jabari; draft
manuscript preparation: J. Liu, F. Zheng, S. Jabari. All authors reviewed the
results and approved the manuscript.

## Conflict of interest

The author(s) declared no potential conflicts of interest with respect to the
research, authorship, and/or publication of this article.

## Appendix

We provide Python codes to generate NLKV samples from trajectories, along with
a subset of the LKV and NLKV samples of L001F001 for reference. These
resources are available at https://github.com/AziaJingLiu/NLKV-sample-create.

## References

  * Bramich et al. (2023) Bramich, D., Menéndez, M., Ambühl, L., 2023.  Fitfun: A modelling framework for successfully capturing the functional form and noise of observed traffic flow–density–speed relationships.  Transportation research part C: emerging technologies 151, 104068.
  * Bramich et al. (2022) Bramich, D.M., Menéndez, M., Ambühl, L., 2022.  Fitting empirical fundamental diagrams of road traffic: A comprehensive review and comparison of models using an extensive data set.  IEEE Transactions on Intelligent Transportation Systems 23, 14104–14127.  doi:10.1109/TITS.2022.3142255.
  * Cassidy (1998) Cassidy, M.J., 1998.  Bivariate relations in nearly stationary highway traffic.  Transportation Research Part B: Methodological 32, 49–59.
  * D. Frejo et al. (2019) D. Frejo, J.R., Papamichail, I., Papageorgiou, M., De Schutter, B., 2019\.  Macroscopic modeling of variable speed limits on freeways.  Transportation Research Part C: Emerging Technologies 100, 15–33.  URL: https://www.sciencedirect.com/science/article/pii/S0968090X18303310, doi:https://doi.org/10.1016/j.trc.2019.01.001.
  * Del Castillo and Benitez (1995) Del Castillo, J.M., Benitez, F., 1995\.  On the functional form of the speed-density relationship—i: General theory.  Transportation Research Part B: Methodological 29, 373–389.
  * Drake et al. (1965) Drake, J., Schofer, J., May, A., 1965.  A statistical analysis of speed-density hypotheses.  Traffic Flow and Transportation .
  * Edie (1963) Edie, L.C., 1963.  Discussion of traffic stream measurements and definitions.  Port of New York Authority New York.
  * Franklin (1961) Franklin, R., 1961.  The structure of a traffic shock wave.  Civil Engineering Pulb. Wks. Rev 56, 1186–1188.
  * Gazis et al. (1961) Gazis, D.C., Herman, R., Rothery, R.W., 1961.  Nonlinear follow-the-leader models of traffic flow.  Operations research 9, 545–567.
  * Green Boook (2004) Green Boook, A., 2004.  A policy on geometric design of highways and streets.
  * Greenberg (1959) Greenberg, H., 1959.  An analysis of traffic flow.  Operations research 7, 79–85.
  * Greenshields et al. (1935) Greenshields, B.D., Bibbins, J., Channing, W., Miller, H., 1935\.  A study of traffic capacity, in: Highway research board proceedings, Washington, DC. pp. 448–477.
  * Hadiuzzaman et al. (2018) Hadiuzzaman, M., Siam, M.R.K., Haque, N., Shimu, T.H., Rahman, F., 2018.  Adaptive neuro-fuzzy approach for modeling equilibrium speed–density relationship.  Transportmetrica A: Transport Science 14, 784–808.
  * Hanshin Expressway Company (2018) Hanshin Expressway Company, 2018.  Zen Traffic Data (ZTD).  Available: https://zen-traffic-data.net.  Accessed on 2023-08-01.
  * Helbing (1997) Helbing, D., 1997.  Fundamentals of traffic flow.  Physical Review E 55, 3735\.
  * Heydecker and Addison (2011) Heydecker, B., Addison, J., 2011\.  Analysis and modelling of traffic flow under variable speed limits.  Transportation Research Part C: Emerging Technologies 19, 206–217.  URL: https://www.sciencedirect.com/science/article/pii/S0968090X10000926, doi:https://doi.org/10.1016/j.trc.2010.05.008. emerging theories in traffic and transportation and methods for transportation planning and operations.
  * Jabari and Liu (2013) Jabari, S.E., Liu, H.X., 2013\.  A stochastic model of traffic flow: Gaussian approximation and estimation.  Transportation Research Part B: Methodological 47, 15–41.
  * Jabari et al. (2014) Jabari, S.E., Zheng, J., Liu, H.X., 2014.  A probabilistic stationary speed–density relation based on newell’s simplified car-following model.  Transportation Research Part B: Methodological 68, 205–223.
  * Kalair and Connaughton (2021) Kalair, K., Connaughton, C., 2021\.  Anomaly detection and classification in traffic flow data from fluctuations in the flow–density relationship.  Transportation Research Part C: Emerging Technologies 127, 103178.  URL: https://www.sciencedirect.com/science/article/pii/S0968090X21001947, doi:https://doi.org/10.1016/j.trc.2021.103178.
  * Laval (2011) Laval, J.A., 2011.  Hysteresis in traffic flow revisited: An improved measurement method.  Transportation Research Part B: Methodological 45, 385–391.
  * Liu et al. (2021) Liu, J., Zheng, F., Liu, X., Guo, G., 2021.  Dynamic traffic flow prediction based on long-short term memory framework with feature organization.  IEEE Intelligent Transportation Systems Magazine 14, 221–236.
  * Liu et al. (2023) Liu, J., Zheng, F., Wei, M., Bai, L., Jabari, S., 2023\.  A linearizing transformation-based evaluation approach of the fundamental diagram.  Transportation Research Record , 03611981231166385\.
  * Makridis et al. (2020) Makridis, M., Leclercq, L., Ciuffo, B., Fontaras, G., Mattas, K., 2020.  Formalizing the heterogeneity of the vehicle-driver system to reproduce traffic oscillations.  Transportation Research Part C: Emerging Technologies 120, 102803.  URL: https://www.sciencedirect.com/science/article/pii/S0968090X20307129, doi:https://doi.org/10.1016/j.trc.2020.102803.
  * McGee (1979) McGee, H.W., 1979.  Decision sight distance for highway design and traffic control requirements.  Transportation research record 736, 11–13.
  * Newell (1961) Newell, G.F., 1961.  Nonlinear effects in the dynamics of car following.  Operations research 9, 209–229.
  * Papageorgiou and Vigos (2008) Papageorgiou, M., Vigos, G., 2008\.  Relating time-occupancy measurements to space-occupancy and link vehicle-count.  Transportation Research Part C: Emerging Technologies 16, 1–17.  URL: https://www.sciencedirect.com/science/article/pii/S0968090X0700037X, doi:https://doi.org/10.1016/j.trc.2007.06.001.
  * Qin and Wang (2023) Qin, Y., Wang, H., 2023.  Stabilizing mixed cooperative adaptive cruise control traffic flow to balance capacity using car-following model.  Journal of intelligent transportation systems 27, 57–79.
  * Qu et al. (2015) Qu, X., Wang, S., Zhang, J., 2015.  On the fundamental diagram for freeway traffic: A novel calibration approach for single-regime models.  Transportation Research Part B: Methodological 73, 91–102.
  * Seo et al. (2019) Seo, T., Kawasaki, Y., Kusakabe, T., Asakura, Y., 2019\.  Fundamental diagram estimation by using trajectories of probe vehicles.  Transportation Research Part B: Methodological 122, 40–56.
  * Shi et al. (2021) Shi, R., Mo, Z., Huang, K., Di, X., Du, Q., 2021\.  A physics-informed deep learning paradigm for traffic state and fundamental diagram estimation.  IEEE Transactions on Intelligent Transportation Systems 23, 11688–11698.
  * Smulders (1990) Smulders, S., 1990.  Control of freeway traffic flow by variable speed signs.  Transportation Research Part B: Methodological 24, 111–132.
  * Sun et al. (2014) Sun, L., Pan, Y., Gu, W., 2014\.  Data mining using regularized adaptive b-splines regression with penalization for multi-regime traffic stream models.  Journal of Advanced Transportation 48, 876–890.
  * Thodi et al. (2022) Thodi, B.T., Khan, Z.S., Jabari, S.E., Menéndez, M., 2022\.  Incorporating kinematic wave theory into a deep learning method for high-resolution traffic speed estimation.  IEEE Transactions on Intelligent Transportation Systems 23, 17849–17862.
  * Treiber and Kesting (2013) Treiber, M., Kesting, A., 2013\.  Traffic flow dynamics.  Traffic Flow Dynamics: Data, Models and Simulation, Springer-Verlag Berlin Heidelberg , 983–1000.
  * Underwood (1961) Underwood, R.T., 1961.  Speed, volume, and density relationships.  Quality and theory of traffic flow .
  * United States Department of Transportation (2006) United States Department of Transportation, 2006\.  Next Generation Simulation (NGSIM).  Available: https://ops.fhwa.dot.gov/trafficanalysistools/ngsim.htm.  Accessed on 2023-08-01.
  * van Wageningen-Kessels et al. (2015) van Wageningen-Kessels, F., Van Lint, H., Vuik, K., Hoogendoorn, S., 2015\.  Genealogy of traffic flow models.  EURO Journal on Transportation and Logistics 4, 445–473.
  * Wang et al. (2014) Wang, Y., Kosmatopoulos, E.B., Papageorgiou, M., Papamichail, I., 2014\.  Local ramp metering in the presence of a distant downstream bottleneck: Theoretical analysis and simulation study.  IEEE Transactions on Intelligent Transportation Systems 15, 2024–2039.
  * Yang et al. (2021) Yang, H., Du, L., Mohammadi, J., 2021.  A shock wave diagram based deep learning model for early alerting an upcoming public event.  Transportation Research Part C: Emerging Technologies 122, 102862.  URL: https://www.sciencedirect.com/science/article/pii/S0968090X20307622, doi:https://doi.org/10.1016/j.trc.2020.102862.
  * Zheng et al. (2018) Zheng, F., Jabari, S.E., Liu, H.X., Lin, D., 2018.  Traffic state estimation using stochastic lagrangian dynamics.  Transportation Research Part B: Methodological 115, 143–165.
  * Zheng et al. (2023) Zheng, S.T., Jiang, R., Jia, B., Tian, J., Bouadi, M., Makridis, M.A., Kouvelas, A., 2023.  A parsimonious enhanced newell’s model for accurate reproduction of driver and traffic dynamics.  Transportation Research Part C: Emerging Technologies 154, 104276.  URL: https://www.sciencedirect.com/science/article/pii/S0968090X23002656, doi:https://doi.org/10.1016/j.trc.2023.104276.