# Incorporating Data Uncertainty in Object Tracking Algorithms

Anish Muthali1, Forrest Laine2, Claire Tomlin3 *This work was supported by the
DARPA Assured Autonomy Program.1Anish Muthali is a student in the department
of Electrical Engineering and Computer Sciences, UC Berkeley
<EMAIL_ADDRESS>Laine is with the Faculty of Computer
Science, Vanderbilt University<EMAIL_ADDRESS>Tomlin is
with the Faculty of Electrical Engineering and Computer Sciences, UC Berkeley
<EMAIL_ADDRESS>
###### Abstract

Methodologies for incorporating the uncertainties characteristic of data-
driven object detectors into object tracking algorithms are explored. Object
tracking methods rely on measurement error models, typically in the form of
measurement noise, false positive rates, and missed detection rates. Each of
these quantities, in general, can be dependent on object or measurement
location. However, for detections generated from neural-network processed
camera inputs, these measurement error statistics are not sufficient to
represent the primary source of errors, namely a dissimilarity between run-
time sensor input and the training data upon which the detector was trained.
To this end, we investigate incorporating data uncertainty into object
tracking methods such as to improve the ability to track objects, and
particularly those which out-of-distribution w.r.t. training data. The
proposed methodologies are validated on an object tracking benchmark as well
on experiments with a real autonomous aircraft.

##  I Introduction

Object tracking can be thought of as the process of inferring the location of
objects based on received measurements. This inference problem is naturally
posed in a Bayesian framework, but such a setup requires a model about how
measurements can be expected to be generated from objects. This is usually
modeled in terms of the rate at which false detections are made, the rate at
which true detections are missed, and the spatial noise (in measurement space)
associated with true detections. Typically all of these statistics are assumed
to be the same for all measurements received, although in some cases they can
depend on the location of the object which the measurement is associated with,
or the location of the measurement itself. One such justification for the
former case is that it may be be more likely to miss a detection on objects
that are very far away than it is for objects nearby.

Modeling detections in this way is appropriate for detections coming from
sensors such as Radar or Lidar—the errors in measurements are typically due to
physical phenomena such as multi-path returns or scattering, which can be
related to object or measurement space. However, modern sensors like cameras
rely on learned functionality as part of the detection process. This typically
takes the form of a Convolutional Neural Network (CNN) mapping the pixel
values of the image to 2D boxes which bound detected objects. Considering the
CNN as part of the sensor itself allows the resultant bounding boxes to be
tracked with standard methods, although we argue that doing so also
invalidates the typical measurement model assumed by those tracking methods. A
CNN will fail to correctly generate bounding boxes for an image, not
necessarily because the objects in that image are near or far, but because
there are not similar images represented in the training data. In other words,
the measurement model should reflect the fact that learned models may have
uncertainty due to the relationship between run-time sensor data and offline
training data, not just spatial information like the distance from an object
to the sensor.

In this work, we present our investigation into the hypothesis that accounting
for data uncertainty in object tracking methods will result in improved
tracking performance, in particular when the task requires tracking objects
that are not well-represented in the detector training data. Specifically, we
present two object tracking methods which leverage the data uncertainty of
CNN-based object detectors for camera inputs: one which is theoretically
principled in capturing and leveraging the uncertainty of interest, and
another which is a computationally tractable approximation for use in systems
such as autonomous vehicles. We evaluate the performance of our proposed
framework on a standard benchmark tracking task, as well as by deploying it on
an autonomous aircraft tasked with tracking an out-of-distribution object.

##  II Background and Related Work

The methods we introduce stand on a foundation of existing techniques for a)
detecting objects from camera data using CNNs, b) generating estimates of
network prediction uncertainty, and c) tracking multiple objects in a Bayesian
inference framework. We discuss in this section relevant existing work
regarding each of these pillars.

###  II-A Camera-Based Object Detection

Effectively all state-of-the art methods for object detection in images rely
on deep learning to achieve strong performance. There are a multitude of
proposed neural network models and architectures designed for object
detection, such as those in [1] [2] [3] [4] [5] to cite just a few. In this
paper we are primarily interested in using the outputs of these object
detectors in a downstream tracking framework, and as such, rely on “off-the-
shelf” methods for our purposes. In later sections, we detail the particular
object detectors used for various implementations of our tracking method.

###  II-B Uncertainty Characterization in Deep Neural Networks

Characterization of uncertainty present in deep neural network models is the
promise of Bayesian Neural Networks [6]. Producing accurate and
computationally efficient estimates of model uncertainty is very much an
active problem. We do not claim to solve or contribute towards this goal, but
again, leverage existing “off-the-shelf” methods for approximating network
uncertainty, which we use for downstream tracking tasks. To this end, we
primarily investigated using Monte Carlo Dropout to generate estimates of
network uncertainty, which has been proposed as a tool for approximating
Bayesian Neural Networks [7].

Using such a technique, multiple evaluations of a network are made at
inference time with dropout enabled, and output variance is estimated using
the resultant sampled outputs. The authors in [8] proposed a network
architecture for training a network to produce a variance associated with each
its output. This method also requires sampling to produce these values of
uncertainty. For speed-critical scenarios, sampling-based methodologies may
not be suitable, due to the expensive computation required to evaluate the
network multiple times. Sampling-free methods such as that proposed by [5]
eliminate the need for repeated sampling. This method injects noise with known
variance at the input layer and “propagates” the variance using gradients such
that larger gradients contribute to a higher variance. In our framework, we
explore both sampling-based and sample-free approaches to estimating network
uncertainty.

###  II-C Multiple Object Tracking Methods

The methods for multiple object tracking (MOT) that we derive our approach
from can be categorized into two main groups. The first decouples the
_measurement association_ problem from the track inference problem. In the
association phase, the measurements produced by a detector are classified as
originating from an object already being tracked, originating from an
untracked object, or corresponding to clutter (a false positive). After new
tracks are generated if necessary, the inference phase updates object tracks
to incorporate the information provided by the newly received measurements, or
lack thereof. In this class of methods, the measurement association phase can
be completed in a greedy manner as demonstrated in [9], or in a probabilistic
manner such as in the PDAF [10] and JPDAF [11] methods. Given measurement
assignments, the position update is performed independently for each track.
For objects which are assumed to move according to a linear-Gaussian process
model, and detections which are assumed to be generated according to a linear-
Gaussian measurement model, this process reduces to performing a step of the
well-known Kalman filter update each time a detection is associated to a track
[12]. For more complicated process or measurement models, Extended or
Unscented Kalman filter approaches can be used [13] [14]. Alternatively, a
fixed-lag smoothing approach can be considered, which at each update uses a
batch of previous measurements to infer the current state of each object being
tracked [15] [16].

The second class of methods requiring mention are those which reason about the
measurement association and track update steps jointly. These approaches model
both the number of obstacles and their position as a single random finite set
(RFS) [17]. The characteristic statistics of the RFS are then updated using
the collection of all received measurements, which is also modeled as a RFS.
Updating the object RFS is in general not possible in closed form, so tracking
algorithms in this class rely on some way of approximately fulfilling the
update steps. Examples of such a method include the probability hypothesis
density (PHD) filter [17] and it’s many implementations, such as [18] [19]
[20] [21] [22] [23] [24].

An emerging third class of tracking methods includes deep learning based
tracking algorithms, which have been shown to perform well on tracking
benchmarks [25]. Generally, these deep learning methods may themselves be
susceptible to data uncertainty. Therefore we consider the development of
these methods as orthogonal to the goal of this paper.

In all of the existing object tracking methods we are aware of, none are
explicitly designed to incorporate the data uncertainty introduced by
learning-based object detectors, like we investigate in this work.

##  III Method

The development of our approach for incorporating object detector data
uncertainty in a tracking framework is outlined in what follows. We first
characterize the approach we chose for training an object detector which can
produce uncertainty estimates along with its outputs. We then describe an
ideal tracking method which could account for that uncertainty. We show in
Section IV that accounting for network uncertainty significantly improves
tracking performance on out-of-distribution (OOD) tracking tasks.

###  III-A Neural Network Model

Our method assumes the existence of neural network trained to generating
bounding boxes around detected objects. We require that, in addition to
predicted bounding boxes, the network produces an estimate of the uncertainty
it has in its detections. This uncertainty could arise from the relationship
of the input image to the training data (epistemic uncertainty) or the
ambiguity in the detection task on the particular image (aleatoric
uncertainty) [8]. Due to its popularity and ease of implementation, we rely on
Monte Carlo Dropout methods for producing these estimates of network
uncertainty, as discussed in Section II.

The object detection networks we consider here produce a list of bounding
boxes, each corresponding to a location within the input image, as well as a
confidence score, valued between $0$ and $1$ [4]. Typically a fixed threshold
is chosen such that all bounding boxes with confidence score above the
threshold are considered true detections. In our proposed Monte Carlo
framework, we introduce dropout layers only in the network path responsible
for producing bounding box locations. To introduce randomness in which
bounding boxes are considered detections or not, we flip the interpretation of
the confidence score to instead represent the threshold for considering that
output a detection—for each pass of the network, a random variable is sampled
from the uniform distribution $U:[0,1]$ for each bounding box output, and if
that random variable is less than the detection threshold, that bounding box
is considered a detection. If not, the particular bounding box is not included
as a detection.

Running many passes of the network in this way for any particular input image
produces a distribution over possible way the network could generate
detections for that image, in the form of particles.

###  III-B Multiple Object Tracking with Detection Network Uncertainty

The objective of a multiple object tracking method is typically to maintain a
representation of the distribution

$p(\mathbf{X}_{k}|\mathbf{Z}_{1:k})$ (1)

where $\mathbf{X}_{1:k}$ represents the set of all object tracks present at a
discrete stage $k$, and $\mathbf{Z}_{1:k}$ is the set of all measurements
received between the first stage and stage $k$. This distribution can be
expressed, using Bayes rule, as

$\displaystyle p(\mathbf{X}_{k}|\mathbf{Z}_{1:k})$
$\displaystyle=\frac{p(\mathbf{Z}_{k}|\mathbf{Z}_{1:k-1},\mathbf{X}_{k})p(\mathbf{X}_{k}|\mathbf{Z}_{1:k-1})}{p(\mathbf{Z}_{1:k})}$
(2)
$\displaystyle=\frac{p(\mathbf{Z}_{k}|\mathbf{X}_{k})p(\mathbf{X}_{k}|\mathbf{Z}_{1:k-1})}{p(\mathbf{Z}_{1:k})}.$

Here we have used the fact that the measurements at stage $k$ are independent
of all previous measurements. The distribution
$p(\mathbf{X}_{k}|\mathbf{Z}_{1:k-1})$ is the predicted distribution of object
states at stage $k$, using measurements up to stage $k-1$. This can be
expressed as

$p(\mathbf{X}_{k}|\mathbf{Z}_{1:k-1})=\int_{\mathcal{X}}p(\mathbf{X}_{k-1}|\mathbf{Z}_{1:k-1})p(\mathbf{X}_{k}|\mathbf{X}_{k-1})\delta\mathbf{X}_{k-1}.$
(3)

Combining these equations, a recursive update can be derived to relate
$p(\mathbf{X}_{k-1}|\mathbf{Z}_{1:k-1})\to
p(\mathbf{X}_{k}|\mathbf{Z}_{1:k})$. This update serves as the foundation for
all Bayesian object tracking methods.

In the context of tracking bounding box detections arising from a neural
network, the bounding boxes produced by the network at stage $k$ would
constitute $\mathbf{Z}_{k}$ in this framework. However, in the framework we
are proposing, we are assuming that we no longer work with particular
measurements $\mathbf{Z}_{k}$, but rather a distribution of measurements
induced by our network for a particular input. If we let $o_{k}$ represent the
input image to our network at stage $k$, we argue that the distribution we
seek to represent is better represented as $p(\mathbf{X}_{k}|o_{1:k})$. We can
represent this distribution as

$\displaystyle
p(\mathbf{X}_{k}|o_{1:k})=\int_{\mathcal{Z}^{k}}p(\mathbf{X}_{k}|o_{1:k},\mathbf{Z}_{1:k})p(\mathbf{Z}_{1:k}|o_{1:k})\delta\mathbf{Z}_{1:k}$
(4)
$\displaystyle=\int_{\mathcal{Z}^{k}}p(\mathbf{X}_{k}|\mathbf{Z}_{1:k})p(\mathbf{Z}_{1:k}|o_{1:k})\delta\textbf{Z}_{1:k}$
$\displaystyle\underset{\begin{subarray}{c}\uparrow\\\ \mathrlap{\text{eq.
(2)}}\end{subarray}}{=}\left(\int_{\mathcal{Z}}\int_{\mathcal{X}}\frac{p(\mathbf{Z}_{k}|\mathbf{X}_{k})p(\mathbf{X}_{k}|\mathbf{X}_{k-1})p(\mathbf{Z}_{k}|o_{k})}{p(\mathbf{Z}_{1:k})}\delta\mathbf{X}_{k-1}\delta\mathbf{Z}_{k}\right)$
$\displaystyle\ \ \ \ \
\cdot\Big{(}\underbrace{\int_{\mathcal{Z}^{k-1}}p(\mathbf{X}_{k-1}|\mathbf{Z}_{1:k-1})p(\mathbf{Z}_{1:k-1}|o_{1:k-1})\delta\mathbf{Z}_{1:k-1}}_{=p(\mathbf{X}_{k-1}|o_{1:k-1})}\Big{)}$
$\displaystyle=\mathbb{E}_{(\mathbf{X}_{k-1},\mathbf{Z}_{k})\sim
P_{k}}\left[\frac{p\left(\mathbf{Z}_{k}|\mathbf{X}_{k}\right)p\left(\mathbf{X}_{k}|\mathbf{X}_{k-1}\right)}{p\left(\mathbf{Z}_{1:k}\right)}\right],$

where we define the distribution

$P_{k}=p\left(\mathbf{X}_{k-1}|o_{1:k-1}\right)\cdot
p\left(\mathbf{Z}_{k}|o_{k}\right).$ (5)

We note that the detections at stage $k$ is conditionally independent of the
observations from stages 1 through $k-1$ ($o_{1:k-1}$), given the observation
at stage $k$ ($o_{k}$). The tracks $\mathbf{X}_{1:k}$ are conditionally
independent of the observations $o_{1:k}$, given $\mathbf{Z}_{1:k}$. The form
of the resultant distribution in Eq. 4 provides a natural framework a particle
filter method.

Specifically, we base our approach on the RFS bootstrap filter described by
Ristic et al. [21]. At stage $k$, we simply sample $N$ particles from the
distribution $P_{k}$, meaning each particle $i\in\\{1,...,N\\}$ is associated
with a particular instance of the joint obstacle states $\mathbf{X}_{k-1}$ and
bounding box outputs $\mathbf{Z}_{k}$, generated from the network with input
image $o_{k}$.

Sampling bounding box outputs from the distribution
$p\left(\mathbf{Z}_{k}|o_{k}\right)$ is as simple as evaluating our network
according to the process outlined in Section III-A. Other than sampling a
different observation $\mathbf{Z}_{k}$ for each particle, our method is as
described in [21]. Each particle performs its own track birth and track death
process, according to the update rules defined by that method.

###  III-C Track Labeling and Outputting

An important aspect of the particle filtering approach is to be able to output
a consistent “most-probable” object representation, which has consistent
labels from discrete stage to stage.

To accomplish this, assume that a most-probable representation of the objects
was known at stage $k-1$, and that each of these tracks was labeled with a
unique identifier. We refer to this representation as the output frame. The
particles that are sampled from the distribution $P_{k}$, to be used in the
measurement update at stage $k$, in general may have more or less objects
associated with them than those in the output frame. We assume that the
objects contained in any of these particles may or may not correspond to the
tracks in the output frame, and if they do, then they share the same label
(the procedure for ensuring this is described below).

After each particle undergoes the birth, death, and measurement update
process, and receives a weight, a new output frame needs to be generated. To
do this, particles are first binned by the number of tracks they contain. Each
bin is associated with a cumulative weight, which corresponds to the total
weight of all particles in that bin. The bin with the highest weight is used
to generate the new output frame. Some of the tracks of particles in this bin
may correspond to previously labeled objects, but some may not. For each
object that is labeled, we fit a Gaussian distribution is fit to its location,
using the particles in the bin that also contain that object with the same
label. A Gaussian Mixture Model is fit to the remaining unlabeled objects
appearing in all particles in the bin. We use a number of mixture components
equal to the cardinality of the bin (prior to removing labeled objects). If
$C$ is the cardinality of this bin, the $C$ highest weighted components are
selected for the output frame. Some of these may correspond to previously
labeled objects, and some may correspond to components of the newly fit GMM.
The components from the GMM are labeled with new unique identifiers. The mean
of the selected components are used in the output frame.

To sample from the new distribution $P_{k+1}$, the cardinality of a new
particle is first selected by sampling from a categorical distribution defined
by the cumulative bin weights. From the corresponding bin, object locations
are sampled from a GMM representing the object locations in that bin. If a
particular object in this newly generated particle is close to any labeled
object in the output frame, it is assigned the same label, and is left
unlabeled otherwise.

##  IV Numerical Results

We benchmark our uncertainty-enabled tracker using the tools provided by the
Waymo Open Dataset 2D Tracking challenge.

###  IV-A Neural Network Parameters and Performance

We trained our bounding box network (RetinaNet with a ResNet-34 backbone) on
the entire Waymo Open Dataset training set of 2D images, excluding all images
which have pedestrians or cyclists in them. For performance reasons, the
images were downsampled to a resolution of $960\times 640$ instead of the
original $1920\times 1280$. We chose a dropout probability of 0.25 for our
network and placed dropout layers after every convolutional layer in the
regression component. Our network was trained for 10 epochs and achieved a
mean average precision (mAP) score of 0.4582 at 0.5 IoU on the training set.

###  IV-B Tracker Parameters

We define the baseline tracker to be a version of our tracker that does not
perform Monte Carlo sampling on the network and does not sample a random
variable for detection thresholds. Instead, it uses the same set of detections
for each particle and uses a fixed threshold for accepting and rejecting the
network’s thresholds. We performed hyperparameter search for both trackers. We
chose $N=50$ particles for both trackers as this provided a balanced trade-off
between efficiency and performance. We used the metrics described in Section
IV-C and determined that the optimal (static) confidence threshold for the
baseline tracker is 0.7.

###  IV-C Tracker Performance

We tested the performance of our trackers on the validation set provided by
the Waymo Open Dataset. The goal is to track the location of objects up to
stage $k+1$ given camera images for up to stage $k$, repeating this process
for $k=1,\ldots,200$ in each segment of data. We used Waymo’s publicly
available tracking metrics computation tools, which computes mean object
tracking accuracy (MOTA), mean object tracking precision error (MOTP),
fraction of missed objects (Miss), fraction of tracks where the tracking
hypothesis changed from previous frames (Mismatch), and fraction of false
positives (FP) [26]. When testing our tracker, we had to do an extra
preprocessing step of relabeling all of the ground truth pedestrians and
cyclists as “vehicles”, since our network was only trained to detect vehicles
and thus does not have the capability of differentiating between multiple
classes. We also separately calculated performance metrics on segments that
contain OOD objects (i.e. pedestrians or cyclists).

###  IV-D Analysis of Results

We notice our implementation significantly improves the MOTA, Mismatches, and
FP over the baseline. We can hypothesize that our method is likely to be
robust to confidence score fluctuations from frame to frame because of the
weighting distribution. For example, if a detection has confidence score 0.71
in one timestep but 0.69 in the next, it will no longer be seen by the
baseline tracker, but the tracker incorporating data uncertainty would be more
robust to small fluctuations such as these. Despite these improvements, we
notice a small but non-negligible decrease in performance in MOTP which is
likely a consequence of spatial uncertainty.

In an example scene depicted in figures 1(a) and 1(b), we can see that, at the
same timestep, our tracker is able to pick up the silver and yellow car while
the baseline tracker cannot. This is likely a consequence of confidence score
fluctuations. Furthermore, our tracker appears to handle OOD objects better,
like the two pedestrians in the middle of the road while the network likely
cannot detect the pedestrians with high enough confidence consistently to
surpass the baseline tracker’s threshold. We also notice that the bounding box
for track 2 is not definitively encompassing either the yellow or silver car
which is likely a consequence of the spatial uncertainty for those two
objects.

TABLE I: Tracker Performance on All Segments Tracker | MOTA | MOTP | Miss | Mismatch | FP
---|---|---|---|---|---
Uncertainty | 0.0853 | 0.5515 | 0.8282 | 0.1654 | 0.2393
Baseline | 0.0708 | 0.5516 | 0.8433 | 0.2213 | 0.2923
Improvement | 20.5% | 0.02% | 1.79% | 25.3% | 18.1%
TABLE II: Tracker Performance on OOD Segments Tracker | MOTA | MOTP | Miss | Mismatch | FP
---|---|---|---|---|---
Uncertainty | 0.0741 | 0.5689 | 0.8552 | 0.1657 | 0.2248
Baseline | 0.0582 | 0.5582 | 0.8644 | 0.2148 | 0.2786
Improvement | 27.4% | -1.93% | 1.06% | 22.8% | 19.3%

(a)

(b)

Figure 1: (a) Representative output frame track locations when tracking
detections with uncertainty Section III. (b) Output frame track locations
without accounting for uncertainty.

##  V Computationally Efficient Approximation

As demonstrated in the previous section, the method presented in Section III
improved performance on out-of-distribution tracking tasks w.r.t. a baseline
implementation not accounting for uncertainty. However, the computational
expense of running the algorithm is not tractable for interactive systems,
such as autonomous vehicles. To bridge this gap, we developed an alternate
method, which retains the conceptual core of the idealized approach, but is
amenable to actual deployment. The particular method we develop is tailored
for the application of autonomous aircraft taxiing, although many of the
design choices made could be generalized to other applications.

###  V-A Pixel-Level Classification

A key aspect of the approach presented in Section III is the ability to relate
object detector data uncertainty with the location of detections in
measurement space. While Monte Carlo generation of bounding boxes is one way
to accomplish this, it relies on evaluating the neural network many times,
which is not feasible for real-time systems. Therefore we turned to the
sample-free variance propagation technique [5] to generate estimates of
uncertainty in a single pass of the network. This method approximates the
variance of a univariate Gaussian distribution centered at each output of the
network, representing the network uncertainty for that value.

For bounding box outputs, however, we found Gaussian approximations of each
bounding box location to be a poor representation of the distribution of
possible network outputs. To address this, we propose training a pixel-level
occupancy classifier, which is tasked with classifying whether each pixel in
the input image corresponds to empty space or an obstacle. Variance
propagation can be used to generate a 2D Gaussian distribution over the
penultimate layer of the classifier, which can be propagated through the final
soft-max layer to generate a value which can be interpreted as the probability
of that particular pixel being occupied.

Applying this computation to the output of every pixel generates a probability
map of obstacle occupancy over the entire image. For any given input pixel
with resultant probability of occupancy $p$, we convert that value $p$ into a
binary occupancy variable $o\in\\{0,1\\}$, where $o=1$ (representing occupied)
if $p\geq 0.5$, else $o=0$, and an uncertainty measure $u$, where $u=2(1-p)$
if $o=1$, and $u=2p$ if $o=0$. In this way, the uncertainty value $u$ is close
to $1$ when $p\approx 0.5$, and $u$ is close to $0$ when $p\approx 0$ or
$p\approx 1$. This change of variable makes it easier to reason about these
outputs in the object tracking procedure.

###  V-B Tracking Occupancy Grids

This transformed output can be interpreted as a binary mask, indicating the
pixel locations of obstacles, along with uncertainty estimates, where lower
uncertainty corresponds to higher confidence in the binary classification.
This form of the network output, in theory, contains information regarding the
spatial distribution of false negatives and false positives due to data
uncertainty. For a particular image, if the output of the detector
corresponding to a particular region of the image is classified as non-
occupied, but has high uncertainty values, this region may correspond to a
missed detection. Conversely, a region classified as occupied with a high
uncertainty may correspond to a false positive.

The output at each pixel can be treated as an independent measurement on
potential objects being tracked, and standard measurement association and
measurement update methods can be applied. In particular, each pixel
corresponding to a detection can be assigned to either an existing track or
new track, and pixels corresponding to empty space can be assigned to all
nearby tracks.

As in Section III, we assume that objects are tracked in the image plane. We
assume that the uncertainty in object tracks locations are maintained as a
Gaussian distribution, and the uncertainty in object tracks is maintained as a
probability of existence. We assume that objects evolve according to a linear-
Gaussian process model. Under these modeling assumptions, we propose the
following particle-filter based method for efficiently updating these object
track representations using pixel level measurements.

Assume that a representation of an object track is known at some discrete time
stage $k$. To update this representation to reflect measurements received at
stage $k+1$, the Gaussian representation of the object state is propagated
through the linear-Gaussian process model, producing a Gaussian distribution
over the predicted object state at stage $k+1$. Using a particle approach,
many particles representing object track state vectors are sampled from this
distribution. Each particle is then weighted by the product of the
probabilities of measuring the independent measurements, conditioned on the
object track state represented by the particle.

We define conditional measurement likelihood for an individual pixel $i$ with
output $o_{i},u_{i}$, conditioned on a particular object track state $x$ as
the following:

$\displaystyle p(o_{i}=1,u_{i}|x)$ $\displaystyle=$ (6) $\displaystyle
p_{fp}+$
$\displaystyle(p_{tp}-p_{fp})e^{-\frac{d_{i}^{2}}{2\sigma^{2}}}(1-u_{i})+\frac{u_{i}}{2}$
$\displaystyle p(o_{i}=0,u_{i}|x)$ $\displaystyle=1-p(o_{i}=1,u_{i}|x),$

where $d_{i}$ is defined to be the set distance between the object track
bounding box and the 2D region of the image plane corresponding to the pixel
$i$, and $p_{tp}$ is the nominal probability of a true positive detection,
$p_{fp}$ is the nominal probability of a false positive, and $\sigma$ is a
shaping parameter. A visualization and rationalization of this measurement
likelihood function is given in Fig. 2.

Specifically, the particle weighting $w$ for a particle $x$ is defined as

$w=\prod_{i=1}^{N}\ p(o_{i},u_{i}|x),$ (7)

where $N$ is the number of pixels associated with the object track represented
by particle $x$. Particles which overlap with many pixels with $o_{i}=1$, and
don’t overlap with many pixels with $o_{i}=0$ will be weighted highly. Pixels
with very high uncertainty will effectively assign all particles the same
weight. After all particles are weighted, a posterior distribution of the
track distribution is generated by fitting a Gaussian distribution to the
weighted particles.

Figure 2: The measurement likelihood function $p(o=1,u|x)$ as a function of
the set distance $d$, for various uncertainty values. Here $p_{fp}=0.2$,
$p_{tp}=0.75$, and $\sigma=1$. When $u=0$, the likelihood peaks at $p_{tp}$
when $d=0$, and decays to $p_{fp}$ as $|d|$ increases. When $u\to 1$, the
likelihood function converges towards $p(o=1,u|x)=0.5$ for all $d$.

To assist in the measurement association process, we first cluster occupied
pixels with neighboring (or nearly neighboring) occupied pixels. We
additionally include in each cluster any unoccupied pixels which surround the
cluster. We allow the unoccupied pixels to be part of multiple clusters, but
occupied pixels must belong to exactly one cluster. These clusters are then
either associated to existing tracks, or used to birth new tracks. For the
association process, we use a greedy approach, although others could be
considered. The product in Eq. 7 is then taken over all pixels belonging to
the associated cluster.

The final step of the tracking procedure is to update the probability of
existence for all tracks. The probability of existence is maintained as the
ratio of probability that an object exists over probability that an object
does not exist. This ratio is updated according to the following update:

$LR_{k+1}=LR_{k}\cdot\frac{\bar{w}}{\prod_{i=1}^{N}p(o_{i},u_{i}|\emptyset)}.$
(8)

Here, $\bar{w}$ is the average weight taken over all particles, and

$\displaystyle p(o_{i}=1,u_{i}|\emptyset)$
$\displaystyle=p_{fp}(1-u_{i})+\frac{u_{i}}{2}$ (9) $\displaystyle
p(o_{i}=0,u_{i}|\emptyset)$ $\displaystyle=1-p(o_{i}=1,u_{i}|\emptyset),$

is defined to be the probability of measuring pixel $i$ with occupancy $o_{i}$
and uncertainty $u_{i}$ when there is no nearby object. In practice, when the
likelihood ratio $LR_{k+1}<<1$ after a measurement update, the object track
under consideration is killed.

##  VI Implementation on Autonomous Aircraft

To validate that the method proposed in Section V is suitable for deployment
on real autonomous systems, we deployed an implementation on an autonomous
Cessna Grand Caravan 208B developed by Boeing. This integration was performed
as a demonstration for the DARPA Assured Autonomy program.

The demonstration aimed to test the capability of our approach in tracking
out-of-distribution objects in an interactive, online setting. Similar to the
setup in Section IV, we trained an object detector based on the architecture
presented in [5] to produce pixel level detections on a data set provided by
Boeing, which consisted of ground vehicles on airport runways. This data set
comprised of images labeled in the form of bounding boxes. In order to produce
per-pixel labels, we converted the bounding boxes to a binary mask, where
pixels in the interior of labeled bounding boxes were labeled as occupied, and
all other pixels were labeled as unoccupied.

The training data held out all instances of other aircraft, so that at test
time, these objects were considered out-of-distribution. The particular task
posed to the object tracking system was to track an aircraft that entered the
field of view, and follow that aircraft down an airport runway. Video footage
of our implementation successfully accomplishing this task is included in the
supplementary video submission. Our method was able to detect and track the
aircraft, despite the detector never being trained on that type of object.
Furthermore, both the object detection network and object tracking algorithm
were able to run at a rate faster than 5hz, which was suitable for use in the
downstream control task.

Figure 3: Visualization of our method running on the Boeing platform. The
central window displays the RGB input fed into the neural-network based object
detector. The window on the left displays the binary mask indicating which
pixels are detected to be occupied by an object (in this case another
aircraft). Overlaid on top of the detections is the mean estimate of the
object track bounding box. The window on the right visualizes the per-pixel
uncertainty map associated with the network outputs.

##  VII Conclusion

We presented a claim that incorporating data uncertainty from learning based
object detectors in an object tracking framework can improve tracking
performance, especially when tracking objects not well represented in the
training data of the object detector. We supported this claim with results on
standard tracking benchmark. We then proposed an approximate but tractable
method, and implemented it on an autonomous aircraft.

## Acknowledgments

We thank Jim Paunicka, Isaac Chang, Michael McGivern, Matt Moser, Dragos
Margineantu, Jordan Stringfield, Ashley Abril, Doug Stuart, and and the entire
team at Boeing for all their help and contributions in demonstrating our
method on the Caravan Test Platform.

## References

  * [1] Ross Girshick  “Fast r-cnn”  In _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1440–1448
  * [2] Alexey Bochkovskiy, Chien-Yao Wang and Hong-Yuan Mark Liao  “Yolov4: Optimal speed and accuracy of object detection”  In _arXiv preprint arXiv:2004.10934_ , 2020
  * [3] Chenhan Jiang et al.  “SP-NAS: Serial-to-parallel backbone search for object detection”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11863–11872
  * [4] Tsung-Yi Lin et al.  “Focal loss for dense object detection”  In _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988
  * [5] Janis Postels et al.  “Sampling-free epistemic uncertainty estimation using approximated variance propagation”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 2931–2940
  * [6] Laurent Valentin Jospin et al.  “Hands-on Bayesian Neural Networks–a Tutorial for Deep Learning Users”  In _arXiv preprint arXiv:2007.06823_ , 2020
  * [7] Yarin Gal and Zoubin Ghahramani  “Dropout as a bayesian approximation: Representing model uncertainty in deep learning”  In _international conference on machine learning_ , 2016, pp. 1050–1059  PMLR
  * [8] Alex Kendall and Yarin Gal  “What uncertainties do we need in bayesian deep learning for computer vision?”  In _arXiv preprint arXiv:1703.04977_ , 2017
  * [9] Gurinderbeer Singh, Sreeraman Rajan and Shikharesh Majumdar  “A Greedy Data Association Technique for Multiple Object Tracking”  In _2017 IEEE Third International Conference on Multimedia Big Data (BigMM)_ , 2017, pp. 177–184  DOI: 10.1109/BigMM.2017.53
  * [10] Yaakov Bar-Shalom and Edison Tse  “Tracking in a cluttered environment with probabilistic data association”  In _Automatica_ 11.5, 1975, pp. 451–460  DOI: https://doi.org/10.1016/0005-1098(75)90021-7
  * [11] Yaakov Bar-Shalom, Fred Daum and Jim Huang  “The probabilistic data association filter”  In _IEEE Control Systems Magazine_ 29.6, 2009, pp. 82–100  DOI: 10.1109/MCS.2009.934469
  * [12] Rudolph Emil Kalman  “A new approach to linear filtering and prediction problems”, 1960
  * [13] Masaru Hoshiya and Etsuro Saito  “Structural identification by extended Kalman filter”  In _Journal of engineering mechanics_ 110.12 American Society of Civil Engineers, 1984, pp. 1757–1770
  * [14] Eric A Wan and Rudolph Van Der Merwe  “The unscented Kalman filter for nonlinear estimation”  In _Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No. 00EX373)_ , 2000, pp. 153–158  Ieee
  * [15] Tue-Cuong Dong-Si and Anastasios I Mourikis  “Motion tracking with fixed-lag smoothing: Algorithm and consistency analysis”  In _2011 IEEE International Conference on Robotics and Automation_ , 2011, pp. 5655–5662  IEEE
  * [16] Frank Dellaert  “Factor graphs and GTSAM: A hands-on introduction”, 2012
  * [17] Ronald P.. Mahler  “Statistical Multisource-Multitarget Information Fusion”  USA: Artech House, Inc., 2007
  * [18] Daniel E Clark, Kusha Panta and Ba-Ngu Vo  “The GM-PHD filter multiple target tracker”  In _2006 9th International Conference on Information Fusion_ , 2006, pp. 1–8  IEEE
  * [19] Daniel Edward Clark and Judith Bell  “Convergence results for the particle PHD filter”  In _IEEE Transactions on Signal Processing_ 54.7 IEEE, 2006, pp. 2652–2661
  * [20] Branko Ristic, Daniel Clark and Ba-Ngu Vo  “Improved SMC implementation of the PHD filter”  In _2010 13th International Conference on Information Fusion_ , 2010, pp. 1–8  IEEE
  * [21] Branko Ristic, Michael Beard and Claudio Fantacci  “An Overview of Particle Methods for Random Finite Set Models”  In _CoRR_ abs/1602.03945, 2016  arXiv: http://arxiv.org/abs/1602.03945
  * [22] Kusha Panta, Daniel E. Clark and Ba-Ngu Vo  “Data Association and Track Management for the Gaussian Mixture Probability Hypothesis Density Filter”  In _IEEE Transactions on Aerospace and Electronic Systems_ 45.3, 2009, pp. 1003–1016  DOI: 10.1109/TAES.2009.5259179
  * [23] Zia Khan, Tucker Balch and Frank Dellaert  “An MCMC-based particle filter for tracking multiple interacting targets”  In _European Conference on Computer Vision_ , 2004, pp. 279–290  Springer
  * [24] Aijun Bai, Reid Simmons and Manuela Veloso  “Multi-object tracking and identification via particle filtering over sets”  In _Proceedings of the 20th International Conference on Information Fusion (FUSION 2017), Xi’an, China_ , 2017, pp. 10–13
  * [25] Nicolai Wojke, Alex Bewley and Dietrich Paulus  “Simple online and realtime tracking with a deep association metric”  In _2017 IEEE international conference on image processing (ICIP)_ , 2017, pp. 3645–3649  IEEE
  * [26] Keni Bernardin, Alexander Elbs and Rainer Stiefelhagen  “Multiple object tracking performance metrics and evaluation in a smart room environment”  In _Sixth IEEE International Workshop on Visual Surveillance, in conjunction with ECCV_ 90.91, 2006  Citeseer


# Voltage control of frequency, effective damping and threshold current in
nano-constriction-based spin Hall nano-oscillators

Victor H. González  Roman Khymyn  Physics Department, University of
Gothenburg, 412 96 Gothenburg, Sweden.  Himanshu Fulara  Department of
Physics, Indian Institute of Technology Roorkee, Roorkee 247667, India  Ahmad
A. Awad  Johan Åkerman<EMAIL_ADDRESS>Physics Department,
University of Gothenburg, 412 96 Gothenburg, Sweden.

###### Abstract

Using micromagnetic simulations, we study the interplay between strongly
voltage-controlled magnetic anisotropy (VCMA), $\Delta K=\pm$200 kJ/m3, and
gate width, $w=$ 10–400 nm, in voltage-gated W/CoFeB/MgO based nano-
constriction spin Hall nano-oscillators. The VCMA modifies the local magnetic
properties such that the magnetodynamics transitions between regimes of _i_)
confinement, _ii_) tuning, and _iii_) separation, with qualitatively different
behavior. We find that the strongest tuning is achieved for gate widths of the
same size as the the constriction width, for which the effective damping can
be increased an order of magnitude compared to its intrinsic value. As a
consequence, voltage control remains efficient over a very large frequency
range, and subsequent manufacturing advances could allow SHNOs to be easily
integrated into next-generation electronics for further fundamental studies
and industrial applications.

Spin Hall nano-oscillators (SHNOs) Liu  _et al._ (2012); Demidov  _et al._
(2012); Liu, Lim, and Urazhdin (2013); Demidov  _et al._ (2014a, b); Duan  _et
al._ (2014); Sato  _et al._ (2019) are miniaturized ultra-broadband microwave
signal generators, typically based on bilayer stacks of nonmagnetic heavy-
metal (HM) and ferromagnetic (FM) thin films, where the spin Hall effect
Hirsch (1999); Hoffmann (2013); Sinova  _et al._ (2015) converts a direct
longitudinal current in the HM into a pure transverse spin current injected
into the FM, where the associated spin transfer torque Slonczewski (1996);
Berger (1996); Ralph and Stiles (2008) can drive auto-oscillations of the
local magnetization. Nanoconstriction-based SHNOs Demidov  _et al._ (2014b);
Awad  _et al._ (2017); Dürrenfeld  _et al._ (2017); Mazraati  _et al._ (2016);
Zahedinejad  _et al._ (2018); Mazraati  _et al._ (2018); Awad  _et al._ (2020)
have a variety of desirable properties, such as robust RF response Awad  _et
al._ (2020), ease of injection locking Hache  _et al._ (2019) and mutual
synchronization Awad  _et al._ (2017); Zahedinejad  _et al._ (2020), making
them ideal candidates for broadband signal processing and unconventional
oscillator computing, where multi-signal injection locking has shown potential
for neuromorphic computing Yogendra  _et al._ (2016); Romera  _et al._ (2018);
Zahedinejad  _et al._ (2020); McGoldrick, Sun, and Liu (2022). SHNOs are also
potentially appealing for oscillator network-based Ising machines to solve
combinatorial optimization problems.Albertsson  _et al._ (2021); Houshang  _et
al._ (2022)

To unlock their potential for computing, effective control of the functional
properties of each individual SHNO must be implemented. While optothermal SHNO
control was very recently experimentally demonstrated Muralidhar  _et al._
(2022), a more established approach has been to use gates and voltage
controlled magnetic anisotropy (VCMA) to tune the material properties of the
active region of the SHNO, and, through this, its microwave signal properties.
Back-gating a nanogap SHNOs achieved about 4.6 MHz/V of frequency tuning Liu
_et al._ (2017). In nanoconstriction SHNOs, a frequency tuning of similar
order of magnitude was recently demonstrated for CoFeB sandwiched between W
and MgO Fulara  _et al._ (2020), which, thanks to the interplay between the
device geometry and the VCMA, also resulted in a giant change in the effective
damping and the SHNO threshold current. By adding a memristive behavior to the
gate dielectric, the tuning could also be made non-volatile.Zahedinejad  _et
al._ (2022)

So far, only a relatively limited tuning range has been explored. Very
recently, Choi et al, presented giant SHNO frequency tuning of up to 2.1 GHz
using VCMA tuning in Co/Ni multilayers, with a reported a change of the
perpendicular magnetic anisotropy (PMA) as large as 28.2 kJ/m3V. Choi  _et
al._ (2022) This hence raises the question of how the damping, threshold
current, and overall auto-oscillation behavior would behave if such strong
tuning could be achieved with narrow gates, where geometrical effects should
further amplify the effect.

Figure 1: Schematic of the simulated voltage-controlled W(5 nm)/CoFeB(1.7
nm)/MgO(2 nm) SHNO.

Here, we use micromagnetic simulations to study voltage-gated SHNOs over a
very wide range of PMA tuning ($\Delta K=\pm$200 kJ/m3) and probe its effect
on the damping, the resonant frequency, and the excited modes for a wide range
of gate widths ($w=$ 10–400 nm). We find a broad range of tunability for both
frequency and damping due to changes in mode localization both depending on
voltage and gate width. Our results demonstrate the versatility of VCMA in
SHNOs and suggest new phenomena, not yet seen in experiments, which could be
employed in the efficient design of purpose-built devices for low-power
electronics and unconventional computing applications.

We simulated a W(5 nm)/CoFeB(1.7 nm)/MgO(2 nm) bow tie shaped bilayer with a
nanoconstriction (NC) of 150 nm width, curvature radius of 50 nm, and opening
angle of 22∘, as shown in Fig. 1. The geometry of the sample is taken from
experiments in earlier work.Fulara  _et al._ (2020) The in-plane current
density and the Oersted field were sampled using the COMSOL multiphysics
software COMSOL . The spin-polarized current density acting on the CoFeB layer
due to the spin Hall effect was calculated by multiplying the in-plane current
density at the W layer by a spin-Hall angle $\theta_{\mathit{{SH}}}=0.43$ and
the injected spin current polarized perpendicular to the local calculated
current density.

The geometry of the SHNO, with a total volume of 4096$\times$4096$\times$1.7
nm3, was discretized into 1024$\times$1024$\times$1 cells and loaded into the
GPU-accelerated micromagnetic simulator Mumax3 Vansteenkiste  _et al._ (2014).
The properties of the CoFeB layer were chosen as: exchange stiffness
$A_{ex}=19$ pJ/m, saturation magnetization $M_{S}=1050$ kA/m, Gilbert damping
constant $\alpha_{0}=9\times 10^{-3}$, and gyromagnetic ratio
$\gamma/2\pi=29.1$ GHz/T. An oblique magnetic field of $B_{ext}=$ 0.4T was
applied at polar $\theta=$ 60.5∘ and azimuthal $\phi=$ 22.0∘ angles consistent
with previous experiments.Fulara  _et al._ (2020) To reduce artifacts due to
spin wave reflections from the edge, periodic boundary conditions were used
with absorbing borders, in the form of quadratically interpolated increased
damping.

The VCMA was applied to a rectangular gated region by a voltage-dependent
uniaxial perpendicular magnetic anisotropy (PMA) of the form,

$K=645\frac{kJ}{m^{3}}-V_{G}\cdot 2.5\frac{kJ}{m^{3}V},$ (1)

where both the intrinsic $K(V_{G}=0)$ and its linear gate voltage dependence
were taken from experiments.Fulara  _et al._ (2020) $V_{G}$ was swept from –80
V to +80 V to investigate a much wider range than studied previously. Although
such high voltages are unlikely to be used, the corresponding change in $K$
can be realized at lower voltages using materials with higher VCMA
coefficients.Choi  _et al._ (2022)

Figure 2: Linear response of a voltage-gated SHNO with 96 nm of gate width.
(a) PSD (color map) and extracted frequencies (white lines) and (b) Effective
damping ($\alpha_{\mathit{{eff}}}$), as functions of gate voltage for the
pulsed current ($V_{\mathit{G}}$) and the change in PMA ($\Delta K$). (c)
Amplitude profiles of the magnetodynamical modes for selected gate voltages
showing the features of the three gating regimes, confinement, tuning, and
separation (black lines in (a,b). The inset zoom-in around the constriction
region.

For extraction of the linear response of the system, such as the dominant
resonances and their damping, the FM layer was first relaxed, then excited
with a short Gaussian pulse ($\sigma=$ 10 ps) and allowed to decay back to
equilibrium for 5 ns with a sampling rate of 10 ps. Then, the average
magnetization in the center of the gate was converted to the frequency domain
via a fast Fourier transform (FFT) and then fitted with a Lorentz distribution
centered on the FMR frequency $f$ and with linewidth $\Delta f$. From this
fitting, the effective damping constant was calculated as

$\alpha_{\mathit{{eff}}}=\frac{\gamma\mu_{0}}{2\pi}\frac{\Delta
f}{f}\frac{dH}{df}$ (2)

for all $V_{G}$.Fulara  _et al._ (2020) The average power was also estimated
using the relation $P\propto\mathcal{F}(m)^{2}$, where $\mathcal{F}(m)$ is the
Fourier transform of the magnetization, to study all magnetodynamic modes.

The power spectral density (PSD) map shown in Fig. 2(a) was constructed as a
function of applied voltage and a gate of 96 nm. The nanoconstriction
frequency, $f_{\mathit{{NC}}}$, and $\alpha_{\mathit{{eff}}}$ (Fig. 2(b)) were
extracted for each case. Both quantities show a non-monotonic behavior and
wide ranges of tunability, with a five-fold increase in damping within the -10
V to +10 V range and an accessible band of frequencies of approximately 800
MHz within the -5 V to +40 V range.

To investigate the origin of the nonmonotonic frequency behavior of the
constriction mode, we simulated a FM layer with a geometry restricted to the
gated region. The extracted frequency of this gate is plotted in Fig. 2(a),
along with the intrinsic frequency $f_{0}$ of an ungated NC. From comparing
the three frequency curves, we can conclude that the shape of
$f_{\mathit{{NC}}}$ emerges from the interplay of an intrinsic mode $f_{0}$
and a gated non-linear mode $f_{G}$. In principle, we can switch between these
two modes with voltage alone.

To gain more insight into the VCMA-mediated modulation of the effective
damping and dynamics of the SHNO, we compared the spatially resolved mode
profile amplitudes at $f_{\mathit{{NC}}}$ for all $V_{G}$. The mode profiles
were constructed through a point-wise FFT, performed on magnetization
snapshots sampled at 10 ps intervals, producing both the amplitude and phase
maps of the oscillation modes. The mode amplitudes are shown in Fig. 2(c)
where three voltage gating regimes can be identified based on the effect VCMA
has on their features: a separation regime, a tuning regime, and a confining
regime.

For strong negative voltages, the effective PMA under the gate becomes large
and suppresses the excitation of localized modes in the gate region. Thus, the
modes delocalize and separate into two coupled in-phase regions, as shown in
the left column of Fig. 2(c). This _separation regime_ spans from
approximately –80 V to –10 V (or $\Delta K$ 200 kJ/m3 to 12.5 kJ/m3) and
presents mostly flat $\alpha_{\mathit{{eff}}}$ and $f_{\mathit{{NC}}}$ curves.
Because the oscillations are driven out, the NC is driven mostly by the
evanescent modes outside of it, which implies that the excited spin-wave
volume increases, promoting radiative damping Slavin and Tiberkevich (2005);
Demidov  _et al._ (2012). The volume of this mode remains approximately the
same and thus the damping is constant with voltage.

The _tuning regime_ , with gate voltage around roughly –10 V to +40 V ($\Delta
K$ 12.5 kJ/m3 to –100 kJ/m3), is where we observe the largest tunabilities.
From the middle column of Fig. 2(c), this regime shows the biggest change in
excited mode volumes, which explains the 800 MHz tunability of frequency and
five-fold increase in effective damping. These are due to the volume
delocalization of the excited mode, which pushes emitted magnons into the non-
gated region while preserving the mode’s core. The flattening of
$\alpha_{\mathit{{eff}}}$ above +15V comes from the fact that the oscillations
are limited to the gated region and we recover the intrinsic $\alpha_{0}$.
Also, in this regime, the non-linear gated mode is close with the intrinsic
mode of the system and thus drives the overall dynamics, reshaping
$f_{\mathit{{NC}}}$. Such a range lends the device a lot of versatility as a
microwave component, with the advantage of the low-power consumption a voltage
source permits.

For high positive voltages between +40 V to +80 V ($\Delta K$ –100 kJ/m3 to
–200 kJ/m3), the local out-of-plane effective field angle is smoothly lowered
with larger voltage (and smaller PMA). In this interval, the excited modes
localized within the constriction, this regime also shows a minimum and a
change in the sign of the derivative of $f_{\mathit{{NC}}}$. In this
_confining regime_ , the localized mode shifts between the bulk (where the
oscillations occur mainly in the middle of the constriction) and the edge
(where they occur mainly at the edges) Dvornik, Awad, and Åkerman (2018), as
shown in the right column of Fig. 2(c).

Figure 3: Threshold current as a function of gate voltage (change of PMA top
axis) fitted with a sigmoid logistic function as a guide to the eye.

To determine the auto-oscillation threshold current, direct currents between
10 and 400 $\mu$A were applied and analyzed for all $V_{G}$. The auto-
oscillation threshold is defined as the smallest current producing an
exponentially growing amplitude. Fig. 3 shows the auto-oscillation threshold
currents as a function of $V_{G}$. The shape of the curve resembles a sigmoid
logistic function, as shown in the fit, and it follows the behavior of the
damping shown in Fig. 2(b), as expected. The sigmoid behavior draws
similarities with an activation function in an artificial neural network,
which invites exploration of possible technological avenues for arrays of
SHNOs based on voltage-based synaptic activation. Very recent work seems to
suggest a voltage-enabled synaptic potentiation and depression that could
permit SHNO based neurons without external memory circuitry Choi  _et al._
(2022).

Finally, to investigate the effect of the gate size on the modes, simulations
and calculations of the frequency, damping, and power were repeated for a wide
range of gate widths between 10 nm and 400 nm. In Fig. 4(a), where we show
PSDs for six widths, a wider gate improves the tunability of the frequency by
affecting (either by localization or delocalization) a larger area, thus
increasing the concavity of the gated mode. For the three widest gates at the
bottom, we can lower the voltages at which the gated mode is observable before
the system switches back to its intrinsic mode. Furthermore, at w = 160 nm, we
can see mode coexistence in the 0 V to –25 V interval.

Figure 4: (a) PSD and extracted frequencies for six different gate widths,
with the bottom gates being ten times wider than the top ones. (b) Effective
damping for the gates depicted in (a). (c) Damping and frequency tunabilities
as a function of gate width. Radiative damping shows a maximum
$\Delta\alpha_{\mathit{{eff}}}$ at 150 nm.

Fig. 4(b), shows that the width of the gate plays an important role in the
radiative damping process. As the width of the gate increases, there is a
gradual change in the shape of $\alpha_{\mathit{{eff}}}$, with a sudden order
of magnitude increase at 160 nm and a subsequent reduction at higher values.
For this width value, $\alpha_{\mathit{{eff}}}$ increases with PMA up to a
point, where PMA compensates the local demagnetization (at $-10$ V), i.e. the
sharp maximum is observed at the transition from in-plane to out-of-plane type
of anisotropy.

The damping ($\Delta\alpha_{\mathit{{eff}}}$) and frequency ($\Delta f_{NC}$)
tunabilities are plotted in Fig. 4(c) as functions of the gate width. $\Delta
f_{NC}$ grows monotonically as the PMA is compensated in a bigger gated area.
However, around 150 nm, there is a large increase in the slope of the curve
before it saturates for wider gates. 150 nm corresponds to the constriction
size and it is the width at which full compensation of the demagnetization
field is achieved; wider gates lead to saturation of the compensation as the
extending the area further does not affect compensation. There is also a sharp
maximum in $\Delta\alpha_{\mathit{{eff}}}$ with a maximum at 150 nm gate
width, where a matching between the width of the gate and the size of the
mode’s core at the center of the nanoconstriction enhances the power transfer
between the gated and non-gated regions and allows for a higher voltage-driven
manipulation of the damping. For wider gates, the modes are driven out
completely, and we have two uncoupled regions, with minimal damping
tunability. The curves in Fig.4(c) could serve as a guide to design tailored
devices for applications where a trade off between damping and frequency
tunabilities might be important.

In conclusion, we find that the VCMA separates the local magnetodynamic
properties of nanoconstriction-based SHNOs into three major regimes of mode
confinement, tuning, and separation, with qualitatively different behavior. We
find that the strongest tuning happens when the gate and nano-constriction
have similar sizes, for which the damping can be varied by one order of
magnitude and the frequency by several GHz. VCMA-oriented design paired with
further developments in heterostructure fabrication techniques can hence pave
the way towards more complex SHNO architectures for next generation
unconventional spintronic computation platforms.

## Acknowledgment

This work was supported by the Swedish Research Council (VR) and the Horizon
2020 research and innovation programme (ERC Advanced Grant No. 835068
”TOPSPIN”).

## Data Availability Statement

The code and output data that support the findings of this study are available
from the corresponding author upon reasonable request.

## References

  * Liu  _et al._ (2012) L. Liu, C.-F. Pai, D. C. Ralph, and R. A. Buhrman, “Magnetic oscillations driven by the spin hall effect in 3-terminal magnetic tunnel junction devices,” Phys. Rev. Lett. 109, 186602 (2012).
  * Demidov  _et al._ (2012) V. E. Demidov, S. Urazhdin, H. Ulrichs, V. Tiberkevich, A. Slavin, D. Baither, G. Schmitz, and S. O. Demokritov, “Magnetic nano-oscillator driven by pure spin current,” Nature Mater. 11, 1028–1031 (2012).
  * Liu, Lim, and Urazhdin (2013) R. H. Liu, W. L. Lim, and S. Urazhdin, “Spectral Characteristics of the Microwave Emission by the Spin Hall Nano-Oscillator,” Phys. Rev. Lett. 110, 147601 (2013).
  * Demidov  _et al._ (2014a) V. E. Demidov, H. Ulrichs, S. V. Gurevich, S. O. Demokritov, V. S. Tiberkevich, A. N. Slavin, A. Zholud, and S. Urazhdin, “Synchronization of spin Hall nano-oscillators to external microwave signals,” Nat. Commun. 5, 3179 (2014a).
  * Demidov  _et al._ (2014b) V. E. Demidov, S. Urazhdin, A. Zholud, A. V. Sadovnikov, and S. O. Demokritov, “Nanoconstriction-based spin-Hall nano-oscillator,” Appl. Phys. Lett. 105, 172410 (2014b).
  * Duan  _et al._ (2014) Z. Duan, A. Smith, L. Yang, B. Youngblood, J. Lindner, V. E. Demidov, S. O. Demokritov, and I. N. Krivorotov, “Nanowire spin torque oscillator driven by spin orbit torques,” Nat. Commun. 5, 5616 (2014).
  * Sato  _et al._ (2019) N. Sato, K. Schultheiss, L. Körber, N. Puwenberg, T. Mühl, A. A. Awad, S. S. Arekapudi, O. Hellwig, J. Fassbender, and H. Schultheiss, “Domain Wall Based Spin-Hall Nano-Oscillators,” Physical Review Letters 123, 57204 (2019).
  * Hirsch (1999) J. E. Hirsch, “Spin Hall Effect,” Phys. Rev. Lett. 83, 1834–1837 (1999).
  * Hoffmann (2013) A. Hoffmann, “Spin Hall Effects in Metals,” IEEE Trans. Magn. 49, 5172–5193 (2013).
  * Sinova  _et al._ (2015) J. Sinova, S. O. Valenzuela, J. Wunderlich, C. H. Back, and T. Jungwirth, “Spin Hall effects,” Rev. Mod. Phys. 87, 1213–1260 (2015).
  * Slonczewski (1996) J. C. Slonczewski, “Current-driven excitation of magnetic multilayers,” J. Magn. Magn. Mater. 159, L1–L7 (1996).
  * Berger (1996) L. Berger, “Emission of spin waves by a magnetic multilayer traversed by a current,” Phys. Rev. B 54, 9353–9358 (1996).
  * Ralph and Stiles (2008) D. C. Ralph and M. D. Stiles, “Spin transfer torques,” J. Magn. Magn. Mater. 320, 1190–1216 (2008).
  * Awad  _et al._ (2017) A. A. Awad, P. Dürrenfeld, A. Houshang, M. Dvornik, E. Iacocca, R. K. Dumas, and J. Åkerman, “Long-range mutual synchronization of spin Hall nano-oscillators,” Nature Physics 13, 292–299 (2017).
  * Dürrenfeld  _et al._ (2017) P. Dürrenfeld, A. a. Awad, A. Houshang, R. K. Dumas, and J. Åkerman, “A 20 nm spin Hall nano-oscillator,” Nanoscale 9, 1285–1291 (2017).
  * Mazraati  _et al._ (2016) H. Mazraati, S. Chung, A. Houshang, M. Dvornik, L. Piazza, F. Qejvanaj, S. Jiang, T. Q. Le, J. Weissenrieder, and J. Åkerman, “Low operational current spin hall nano-oscillators based on nife/w bilayers,” Applied Physics Letters 109, 242402 (2016).
  * Zahedinejad  _et al._ (2018) M. Zahedinejad, H. Mazraati, H. Fulara, J. Yue, S. Jiang, A. A. Awad, and J. Åkerman, “CMOS compatible W/CoFeB/MgO spin Hall nano-oscillators with wide frequency tunability,” Applied Physics Letters 112, 1–5 (2018).
  * Mazraati  _et al._ (2018) H. Mazraati, S. R. Etesami, S. A. H. Banuazizi, S. Chung, A. Houshang, A. A. Awad, M. Dvornik, and J. Åkerman, “Auto-oscillating spin-wave modes of constriction-based spin hall nano-oscillators in weak in-plane fields,” Physical Review Applied 10, 054017 (2018).
  * Awad  _et al._ (2020) A. A. Awad, A. Houshang, M. Zahedinejad, R. Khymyn, and J. Åkerman, “Width dependent auto-oscillating properties of constriction based spin Hall nano-oscillators,” Applied Physics Letters 116, 232401 (2020), arXiv:2003.07178 .
  * Hache  _et al._ (2019) T. Hache, T. Weinhold, K. Schultheiss, J. Stigloher, F. Vilsmeier, C. Back, S. S. Arekapudi, O. Hellwig, J. Fassbender, and H. Schultheiss, “Combined frequency and time domain measurements on injection-locked, constriction-based spin Hall nano-oscillators,” Applied Physics Letters 114 (2019).
  * Zahedinejad  _et al._ (2020) M. Zahedinejad, A. A. Awad, S. Muralidhar, R. Khymyn, H. Fulara, H. Mazraati, M. Dvornik, and J. Åkerman, “Two-dimensional mutually synchronized spin Hall nano-oscillator arrays for neuromorphic computing,” Nature Nanotechnology 15, 47 (2020).
  * Yogendra  _et al._ (2016) K. Yogendra, D. Fan, B. Jung, and K. Roy, “Magnetic pattern recognition using injection-locked spin-torque nano-oscillators,” IEEE Transactions on Electron Devices 63, 1674–1680 (2016).
  * Romera  _et al._ (2018) M. Romera, P. Talatchian, S. Tsunegi, F. A. Araujo, V. Cros, P. Bortolotti, J. Trastoy, K. Yakushiji, A. Fukushima, H. Kubota, S. Yuasa, M. Ernoult, D. Vodenicarevic, T. Hirtzlin, N. Locatelli, D. Querlioz, and J. Grollier, “Vowel recognition with four coupled spin-torque nano-oscillators,” Nature 563, 230 (2018).
  * McGoldrick, Sun, and Liu (2022) B. C. McGoldrick, J. Z. Sun, and L. Liu, “Ising machine based on electrically coupled spin hall nano-oscillators,” Physical Review Applied 17, 014006 (2022).
  * Albertsson  _et al._ (2021) D. I. Albertsson, M. Zahedinejad, A. Houshang, R. Khymyn, J. Åkerman, and A. Rusu, “Ultrafast ising machines using spin torque nano-oscillators,” Applied Physics Letters 118, 112404 (2021).
  * Houshang  _et al._ (2022) A. Houshang, M. Zahedinejad, S. Muralidhar, J. Chȩciński, R. Khymyn, M. Rajabali, H. Fulara, A. A. Awad, M. Dvornik, and J. Åkerman, “Phase-binarized spin hall nano-oscillator arrays: Towards spin hall ising machines,” Phys. Rev. Applied 17, 014003 (2022).
  * Muralidhar  _et al._ (2022) S. Muralidhar, A. Houshang, A. Alemán, R. Khymyn, A. A. Awad, and J. Åkerman, “Optothermal control of spin hall nano-oscillators,” Applied Physics Letters 120, 262401 (2022), https://doi.org/10.1063/5.0086758 .
  * Liu  _et al._ (2017) R. H. Liu, L. Chen, S. Urazhdin, and Y. W. Du, “Controlling the spectral characteristics of a spin-current auto-oscillator with an electric field,” Phys. Rev. Applied 8, 021001 (2017).
  * Fulara  _et al._ (2020) H. Fulara, M. Zahedinejad, R. Khymyn, M. Dvornik, S. Fukami, S. Kanai, H. Ohno, and J. Åkerman, “Giant voltage-controlled modulation of spin hall nano-oscillator damping,” Nature Communications 11, 4006 (2020).
  * Zahedinejad  _et al._ (2022) M. Zahedinejad, H. Fulara, R. Khymyn, A. Houshang, M. Dvornik, S. Fukami, S. Kanai, H. Ohno, and J. Åkerman, “Memristive control of mutual spin hall nano-oscillator synchronization for neuromorphic computing,” Nat Mater. 21, 81–87 (2022), https://doi.org/10.1038/s41563-021-01153-6 .
  * Choi  _et al._ (2022) J.-G. Choi, J. Park, M.-G. Kang, D. Kim, J.-S. Rieh, K.-J. Lee, K.-J. Kim, and B.-G. Park, “Voltage-driven gigahertz frequency tuning of spin hall nano-oscillators,” Nature Communications 13, 3783 (2022).
  * (32) COMSOL, “Comsol multiphysics software,” .
  * Vansteenkiste  _et al._ (2014) A. Vansteenkiste, J. Leliaert, M. Dvornik, M. Helsen, F. Garcia Sanchez, and B. Van Waeyenberge, “The design and verification of MuMax3,” AIP Adv. 4, 107133 (2014).
  * Slavin and Tiberkevich (2005) A. Slavin and V. Tiberkevich, “Spin Wave Mode Excited by Spin-Polarized Current in a Magnetic Nanocontact is a Standing Self-Localized Wave Bullet,” Phys. Rev. Lett. 95, 237201 (2005).
  * Dvornik, Awad, and Åkerman (2018) M. Dvornik, A. A. Awad, and J. Åkerman, “Origin of magnetization auto-oscillations in constriction-based spin hall nano-oscillators,” Phys. Rev. Applied 9, 014017 (2018).