# Localization and Mapping of Sparse Geologic Features with Unpiloted Aircraft
Systems

Zhiang Chen, Sarah Bearman, J Ramón Arrowsmith, Jnaneshwar Das Authors are
affiliated with the School of Earth and Space Exploration, Arizona State
University, 781 Terrace Mall, Tempe, AZ 85287, USA

###### Abstract

Robotic mapping is attractive in many scientific applications that involve
environmental surveys. This paper presents a system for localization and
mapping of sparsely distributed surface features such as precariously balanced
rocks (PBRs), whose geometric fragility parameters can provide valuable
information on earthquake processes and landscape development. With this
geomorphologic problem as the test domain, we carry out a lawn-mower search
pattern from a high elevation using an Unpiloted Aerial Vehicle (UAV) equipped
with a flight controller, GPS module, stereo camera, and onboard computer.
Once a potential PBR target is detected by a deep neural network in real time,
we track its bounding box in the image coordinates by applying a Kalman filter
that fuses the deep learning detection with Kanade–Lucas–Tomasi (KLT)
tracking. The target is localized in world coordinates using depth filtering
where a set of 3D points are filtered by object bounding boxes from different
camera perspectives. The 3D points also provide a strong prior on target
shape, which is used for UAV path planning to closely map the target using
RGBD SLAM. After target mapping, the UAS resumes the lawn-mower search pattern
to locate and map the next target.

##  I introduction

Although topographic mapping using Unpiloted Aerial Systems (UAS) and
Structure from Motion (SfM) has recently proliferated in large-scale terrain
mapping, UAS-SfM methods are not ideal when the mapping is designed to map and
identify sparsely distributed targets of interest. This is because UAS-SfM
methods based on bundle adjustment require intensive offboard computation to
produce dense terrain maps that include both targets of interest and other
irrelevant features. As the sparsity of targets increases, significant amounts
of computation and memory storage are spent on reconstructing irrelevant
features. More importantly, UAS-SfM methods like offboard mapping decouple
robot navigation from mapping. This inhibits adaptive or active mapping [1, 2]
that can optimize flight paths to locate and map specific targets.

Target localization is a process of estimating a target’s location in 3D
space. Previously, target location was simplified by determining the target’s
geometric center, which is useful when the target’s dimensions are negligible.
However, in scenarios where the target’s dimensions matter, a representation
of the target’s geometric shape information is desired. In this work, a
target’s location is represented by a set of 3D points, which can indicate its
geometric center, orientation, and dimension. The additional information
facilitates UAV path planning for target mapping.

This work is motivated by the challenges of finding and mapping precariously
balanced rocks (PBRs) (Fig. 1) and other fragile geologic features (e.g. rock
pillars). PBRs are easily toppled by strong earthquake shaking, so their
existence can provide valuable information about earthquake history in a given
region [3, 4, 5].

Figure 1: Precariously balanced rocks: (Left) PBR in Granite Dell, Arizona.
(Right) PBR mapping simulation environment in Gazebo. In simulation, PBR (with
dark texture at the top-left in right figure) is on the foot wall of a fault
scarp.

However, model assessment in a broad statistical sense is challenging because
PRBs are sparsely distributed and difficult to identify. Our ultimate goal is
to assess the precarious features broadly, increasing the number of analyses
and accounting for their geomorphic setting [4, 6]. Large datasets of
spatially explicit PBR fragility with detailed geomorphic and geologic context
provide a valuable assessment of sensitivity to ground motions. Instead of
having a few fragilities, it should be possible to have a fragility histogram
for the fragile geologic features at a site.

Figure 2: System workflow: The system includes target tracking, target
localization, and target mapping. The motion mode switches from lawn-mower
searching during target tracking to localization mode once fine localization
is requested, and then switches to mapping mode after the target is finely
localized.

The objective of this paper is to present an efficient, real-time approach to
localize and map PBRs by taking advantage of deep neural network object
detection, sample-based state estimation, and RGBD SLAM. The system workflow
is presented in Fig. 2. We start by tracking bounding box coordinates in image
frames using a Kalman filter that fuses deep neural network detection and KLT
tracking. Then, the target is localized in world coordinates by sampling a set
of 3D points that are generated by back-projecting a polyhedral cone from the
vertices of the enlarged tracked bounding box at different camera
perspectives. Finally, we deploy RGBD SLAM for accurate target mapping. For
the convenience of tracking, the math notations are consistent in the entire
paper.

##  II target tracking

We employ a deep neural network to detect targets, and then track their
bounding boxes in image coordinates using a Kalman filter. A KLT tracker is
used to predict the new state of each bounding box, and the state variable is
updated using object detection from a well-trained deep neural network. We use
an IoU threshold to register new bounding boxes, and we deregister the
bounding boxes when they move out of the current image frame. False detections
are deregistered by examining the differential entropy of bounding box state
distributions.

The Kalman filter algorithm is presented in Algorithm 1. The state variable of
the Kalman filter is the bounding box coordinates
$\mathbf{x}=(u_{min},v_{min},1,u_{max},v_{max},1)$ in homogeneous coordinate
system, or $\mathbf{u}=(u_{min},v_{min},u_{max},v_{max})$ in Euclidean
coordinate system. Homogeneous coordinate system and Euclidean coordinate
system can be converted by matrix operations:

$\mathbf{x}_{t-1}=\mathbf{T}\mathbf{u}_{t-1}+\mathbf{a}$
$\mathbf{u}_{t}=\mathbf{M}\mathbf{x}_{t}$

where
$\mathbf{T}=\begin{bmatrix}1&0&0&0\\\ 0&1&0&0\\\ 0&0&0&0\\\ 0&0&1&0\\\
0&0&0&1\\\ 0&0&0&0\\\ \end{bmatrix}$, $\mathbf{a}=\begin{bmatrix}0\\\ 0\\\
1\\\ 0\\\ 0\\\ 1\\\ \end{bmatrix}$, $\mathbf{M}=\begin{bmatrix}1&0&0&0&0&0\\\
0&1&0&0&0&0\\\ 0&0&0&1&0&0\\\ 0&0&0&0&1&0\\\ \end{bmatrix}$ Lines 1-2 and
Lines 5-6 in Algorithm correspond the conversions between homogeneous and
Euclidean coordinates.

The predict phase uses a KLT tracker to estimate a bounding box state. We
consider a linear prediction model for bounding boxes,

$\mathbf{x}_{t}=\mathbf{F}_{t-1}\mathbf{x}_{t-1}+\mathbf{e}_{t-1}$ (1)

where $\mathbf{F}_{6\times 6}$ is state transition model, and
$\mathbf{e}\in\mathbb{R}^{6}$ is a prediction error vector:

$\mathbf{F}=\begin{bmatrix}\mathbf{S}&\mathbf{0}\\\
\mathbf{0}&\mathbf{S}\end{bmatrix},\mathbf{e}=(e,e,0,e,e,0)$

$\mathbf{S}_{3\times 3}$ is a similarity transform produced from KLT tracker
that applies Harris features in each bounding box. We use homogeneous
coordinate system in the predict phase to keep consistent with $\mathbf{F}$,
which is in 2D homogeneous coordinates.

Algorithm 1 Kalman Filter for Bounding Box Tracking
---
|  input: $\mathbf{u}_{t-1}$, $\Sigma_{t-1}$, $\mathbf{F}_{t-1}$,
$\mathbf{z}_{t}$
---
output: $\mathbf{u}_{t}$
parameter: $\mathbf{R}$, $\mathbf{Q}$
Predict:
1\. $\mathbf{x}_{t-1}=\mathbf{T}\mathbf{u}_{t-1}+\mathbf{a}$
2\. $\Omega_{t-1}=\mathbf{T}\Sigma_{t-1}+\mathbf{a}$
3\. $\mathbf{x}_{t}=\mathbf{F}_{t-1}\mathbf{x}_{t-1}$
4\. $\Omega_{t}=\mathbf{F}_{t-1}\Omega_{t-1}\mathbf{F}_{t-1}^{T}+\mathbf{R}$
5\. $\mathbf{u}_{t}=\mathbf{M}\mathbf{x}_{t}$
6\. $\Sigma_{t}=\mathbf{M}\Omega_{t}$
Update:
7\. $\mathbf{K}=\Sigma_{t}(\Sigma_{t}+\mathbf{Q})^{-1}$
8\. $\mathbf{u}_{t}=\mathbf{u}_{t}+\mathbf{K}(\mathbf{z}_{t}-\mathbf{u}_{t})$
9\. $\Sigma_{t}=(\mathbf{I}_{4\times 4}-\mathbf{K})\Sigma_{t}$

Object detection from the deep neural network is deployed to update the state
variable,

$\mathbf{z}_{t}=\mathbf{H}\mathbf{u}_{t}+\mathbf{w}_{t}$ (2)

where $\mathbf{H}_{4\times 4}$ is an identity matrix,
$\mathbf{w}\in\mathbb{R}^{4}$ is an error vector for the update model, and
$\mathbf{z}=(u_{min},v_{min},u_{max},v_{max})\in\mathbb{R}^{4}$ is bounding
box coordinates from the neural network. We obtain these coordinates from

$\mathbf{z}_{t}=f(I_{t})$ (3)

where $I$ is an input image and $f$ is a well-trained object detection neural
network. We assume both predict error $\mathbf{e}$ and update error
$\mathbf{w}$ have multi-variable Gaussian distribution:

$\mathbf{e}\sim\mathcal{N}(\mathbf{0},\mathbf{R}),\mathbf{w}\sim\mathcal{N}(\mathbf{0},\mathbf{Q})$

where $\mathbf{R}_{6\times 6}$, $\mathbf{Q}_{4\times 4}$ are the covariance
matrices:

$\mathbf{R}=\begin{bmatrix}e^{2}&0&0&0&0&0\\\ 0&e^{2}&0&0&0&0\\\
0&0&0&0&0&0\\\ 0&0&0&e^{2}&0&0\\\ 0&0&0&0&e^{2}&0\\\
0&0&0&0&0&0\end{bmatrix},\mathbf{Q}=w^{2}\mathbf{I}_{4\times 4}$

and $\mathbf{I}_{4\times 4}$ is an identity matrix. We use Euclidean
coordinate system in the update phase because an invertible covariance matrix
is needed to update Kalman gain $\mathbf{K}$, while the covariance matrix in
homogeneous coordinate system $\Omega$ is degenerate.

A new bounding box detected from the neural network will be registered if its
IoU is smaller than a threshold with any previously registered bounding boxes.
Otherwise, the new bounding box coordinates will be used to update the
registered bounding boxes that satisfy the IoU threshold. A bounding box will
be deregistered when it is moving out of the current image frame.

False detection from the deep neural network can be problematic, so we monitor
the differential entropy of each bounding-box-state distribution:

$h=\frac{k}{2}+\frac{k}{2}\ln(2\pi)+\frac{1}{2}\ln(|\Sigma|)$ (4)

where $k=4$ is the dimensionality of the state vector space. When the
differential entropy $h$ becomes greater than a threshold, it implies there
has been no update for several predictions, and the bounding box will be
deregistered. An example of tracking results is shown in Fig. 3.

Figure 3: Kalman filtering for bounding box tracking: The UAV is moving north.
The top-row images are neural network detection. The bottom-row images are the
results from the Kalman-filtering tracking.

##  III target localization

Once a target is tracked in image frames, we first randomly generate 3D points
within a four-face polyhedral cone that is back-projected from an enlarged
bounding box (Fig. 4(A)). The 3D points will be projected to different image
frames and resampled according to their positions with respect to the tracked
bounding box (Fig. 4(B)). The rationale is similar to the depth filter in [7],
but we adapt it to bounding boxes and use a set of 3D points to represent a
target location. Unlike other work that simplifies target locations as
geometric centers, 3D points offer added information such as orientation and
dimension, which can facilitate UAV motion planning for target mapping.

###  III-A Camera Model

We derive the back-projection of image points to rays on special Euclidean
group based on the models in Hartley and Zisserman’s book [8]. A general
projective camera maps a point in space $\mathbf{L}=(x,y,z,1)$ to an image
point $\mathbf{l}=(u,v,1)$ according to the mapping
$\mathbf{l}=\mathbf{P}\mathbf{L}$. $\mathbf{P}$ is the projection matrix, and
$\mathbf{P}=[K|\mathbf{0}]T_{c}^{w}$, where $K_{3\times 3}$ is the camera
intrinsic matrix, and $T_{c}^{w}\in SE(3)$ is the transformation matrix from
world frame to camera frame. We can back-project a ray from an image point,

$\mathbf{r}_{c}=S(\mu K^{-1}\mathbf{l})+\mathbf{b}$ (5)
$\mathbf{r}_{w}=T_{w}^{c}\mathbf{r}_{c}$ (6)

where $\mathbf{r}_{c}$ is a homogeneous 3D point representing a ray in camera
frame, $\mathbf{r}_{w}$ is the corresponding ray in world frame, $\mu$ is a
non-negative scalar, $S_{4\times 3}=\begin{bmatrix}I_{3\times 3}\\\
\mathbf{0}\end{bmatrix}$, $\mathbf{b}=(0,0,0,1)$, and $T_{w}^{c}\in SE(3)$ is
the transformation matrix from camera frame to world frame. $S$ and
$\mathbf{b}$ convert Euclidean coordinates to Homogeneous coordinates. A ray
in world frame is represented by a 3D points set
$\eta=\\{\mathbf{r}_{w}|\mu\geq 0\\}$.

Figure 4: 3D point generation and update: (A) 3D points are generated within a
polyhedral cone that is back-projected from an enlarged bounding box. (B) 3D
points are projected to a different image frame and resampled according to
their projections’ positions with respect to the new bounding box.

###  III-B 3D Point Generation

We have $T_{w}^{c}=T_{w}^{b}T_{b}^{c}$, where $T_{b}^{c}$ is the
transformation matrix from camera frame to robot base frame, and $T_{w}^{b}$
is the transformation matrix from the robot base frame to the world frame. We
enlarge the bounding boxes from target tracking to account for errors in the
drone state estimator. A four-face polyhedral cone,
$\\{\eta_{w1},\eta_{w2},\eta_{w3},\eta_{w4}\\}$, is generated by four 3D rays
back-projected from four corners of the enlarged bounding boxed. To randomly
initialize a 3D point in the cone, we adopt the following steps.

  1. 1.

randomly generate a direction vector from a convex combination of four
enlarged corner vectors:

$\mathbf{v}=\alpha_{1}K^{-1}\mathbf{l}_{1}+\alpha_{2}K^{-1}\mathbf{l}_{2}+\alpha_{3}K^{-1}\mathbf{l}_{3}+\alpha_{4}K^{-1}\mathbf{l}_{4}$
(7)

s.t.
$\alpha_{1},\alpha_{2},\alpha_{3},\alpha_{4}\sim\mathcal{U}$
$\alpha_{1}+\alpha_{2}+\alpha_{3}+\alpha_{4}=1$
where $\mathcal{U}$ is a standard uniform distribution, $\\{\mathbf{l}_{i}\\}$
are corners of an enlarged bounding box in image frame.

  2. 2.

randomly scale the direction vector: $\mathbf{\bar{L}}=\delta\mathbf{v}$,
where $\delta$ is a random depth. $\mathbf{\bar{L}}$ is a Homogeneous 3D point
in the cone in camera frame.

  3. 3.

convert the 3D point to world frame:
$\mathbf{L}=T_{w}^{c}(S\mathbf{\bar{L}}+\mathbf{b})$.

$\mathbf{L}$ is one randomly generated point within a cone in world frame. We
generate 1000 3D points to localize a tracked target using the matrix form of
3D point generation in Algorithm 2.

Algorithm 2 3D Point generation
---
|  input: $\\{\mathbf{l}_{1},\mathbf{l}_{2},\mathbf{l}_{3},\mathbf{l}_{4}\\}$,
$m$, $T_{w}^{c}$
---
output: $P_{w}$
parameter: $K$, $d_{m}$
1\. $A=\mathbb{U}(4,m)$
2\. $B=Diag(|A|_{m})$
3\. $\bar{A}=AB^{-1}$
4\.
$C=[K^{-1}\mathbf{l}_{1},K^{-1}\mathbf{l}_{2},K^{-1}\mathbf{l}_{3},K^{-1}\mathbf{l}_{4}]$
5\. $\delta=d_{m}\mathbb{U}(1,m)$
6\. $\bar{\delta}=Diag(\delta)$
7\. $\tilde{P}_{c}=C\bar{A}\bar{\delta}$
8\. $P_{c}=S\tilde{P}_{c}+\mathbf{b}$
9\. $P_{w}=T_{w}^{c}P_{c}$
Comments:
1). $m$ is the number of 3D points to be generated, e.g. $m=1000$.
2). $\mathbb{U}(x,y)$ is a standard uniform sampler returning a
$x\times y$ matrix.
3). $Diag(x)$ creates a diagonal matrix from a vector $x$.
4). $|\cdot|_{m}$ returns a vector of Manhattan distances of all columns.
5). $d_{m}$ is the maximum depth of 3D points along $z$ axis
in camera frame.

When a bounding box is updated in tracking, a new set of 3D points will be
generated when there is no existing 3D point in the cone back-projected from
the updated bounding box. The four-face polyhedral cone from a bounding box
can be represented by four inward half-spaces. For convenience, we use camera
frame for 3D point registration because all four hyperplanes pass through the
origin. Then we have a 3D point registration condition:

$\mathbf{N}\mathbf{\tilde{L}}>\mathbf{0}$ (8)

where $\mathbf{\tilde{L}}=(x,y,z)$ is the 3D point in camera frame,
$\mathbf{N}_{4\times 3}$ is composed of four rows of positive normal vectors
on the four half-spaces. Positive normal vectors on a half-space can be
acquired by cross product,
$\mathbf{n}_{ij}=(K^{-1}\mathbf{l}_{i})\times(K^{-1}\mathbf{l}_{j})$, where
$\mathbf{l}_{i}$, $\mathbf{l}_{j}$ are two adjacent enlarged corners in
clockwise order to ensure the normal vector direction is inwards. If there is
no existing 3D point satisfying the condition, a new set of 3D points will be
registered; otherwise, the set of registered 3D points that consist of any
points satisfying the condition will be updated using the following subsection
method.

###  III-C 3D Point update

Our 3D point update is adapted from the depth filter in SVO [7] and developed
for semantic bounding box localization. We project existing 3D points to new
image frames and resample them based on the positions between their
projections and the bounding boxes in the new frames (Algorithm 3). The
distribution $f(\mathbf{p}|\mathbf{x})$ assesses the weights of points
$\mathbf{p}$ on an image given a bounding box $\mathbf{x}$, and is formed by a
combination of a Gaussian distribution and a uniform distribution:

$f(\mathbf{p}|\mathbf{x})=w_{1}h(\mathbf{p}|\mathbf{x})+w_{2}g(\mathbf{p}|\mathbf{x})$
(9)

where $h(\mathbf{p}|\mathbf{x})$ is distribution density function of a 2D
Gaussian distribution $\mathcal{N}(\mathbf{c_{x}},\Sigma_{\mathbf{x}})$, and
$g(\mathbf{p}|\mathbf{x})$ is distribution density function of a 2D uniform
distribution $\mathcal{U}$ among an enlarged area of the bounding box
$\mathbf{x}$. Variables $w_{1}$ and $w_{2}$ are the weights of the two
distributions. For $\mathcal{N}(\mathbf{c_{x}},\Sigma_{\mathbf{x}})$,
$\mathbf{c_{x}}=(\frac{u_{min}+u_{max}}{2},\frac{v_{min}+v_{max}}{2})$,
$\Sigma_{\mathbf{x}}=\begin{bmatrix}(\frac{u_{max}-u_{min}}{2})^{2}&0\\\
0&(\frac{v_{max}-v_{min}}{2})^{2}\end{bmatrix}$
The 3D points $P_{w}$ in Algorithm 3 will be resampled according to their
weights from the joint distribution by importance sampling.

Algorithm 3 3D Point Update
---
|  input: $P_{w}$, $\mathbf{x}$, $T_{c}^{w}$
---
output: $P_{w}^{\prime}$
parameter: $\sigma$, $m$, $K$
1\. $\mathbf{a}=\sqrt{\sigma}\mathbb{N}(m,3)$
2\. $\bar{\mathbf{a}}=S\mathbf{a}+\mathbf{b}$
3\. $\mathbf{p}=[K|\mathbf{0}]T_{c}^{w}(P_{w}+\bar{\mathbf{a}})$
4\. $\mathbf{i}=f(\mathbf{p}|\mathbf{x})$
5\. $P_{w}^{\prime}=Importance\\_Sampling(P_{w},\mathbf{i})$
Comments:
1). $\mathbb{N}(x,y)$ is an isotropic standard multi-variate Gaussian
sampler returning a $x\times y$ matrix.
2). $\sqrt{\sigma}$ is the standard variance of update noise.
3). $f(\mathbf{p}|\mathbf{x})$ is a two-variable distribution composed of a
Gaussian distribution and a uniform distribution.

###  III-D Properties

Some properties are used to describe 3D points and determine if target
localization has been achieved. We apply Principle Component Analysis (PCA) to
each 3D points set. The largest eigenvalue $\lambda_{m}$ can indicate how
compact a 3D points set is. We also utilize the eigenvalues and eigenvectors
from PCA to approximate an ellipsoid for the object, which can be used for
visualization, to convey the information of target size, shape, and
orientation.

When a 3D points set has been updated, we want to measure the amount of
information gain from the update. One metric that captures relative entropy is
Kullback–Leibler divergence (KL divergence) of updated 3D points distribution
with respect to previous 3D points distribution. To compute KL divergence, we
approximate the 3D points set with a three-dimensional Gaussian distribution,
whose mean and covariance are approximated by the mean and covariance of the
3D points. We can compute the KL divergence following the formula for
multivariate Gaussian distribution:

$\displaystyle
D_{KL}(\mathcal{N}_{0}||\mathcal{N}_{1})=\frac{1}{2}(tr(\Pi_{1}^{-1}\Pi_{0})+$
(10)
$\displaystyle(\tau_{1}-\tau_{0})^{T}\Pi_{1}^{-1}(\tau_{1}-\tau_{0})-k+\ln(\frac{|\Pi_{1}|}{|\Pi_{0}|}))$

where $k=3$ is the dimension of the random variable,
$\mathcal{N}_{0}(\tau_{0},\Pi_{0})$ is the estimated Gaussian distribution of
the updated 3D points, $\mathcal{N}_{1}(\tau_{1},\Pi_{1})$ is the estimated
Gaussian distribution of the previous 3D points, and $tr(\cdot)$ is a matrix
trace function.

Similar to bounding box tracking, we also inspect differential entropy of each
3D point set $h_{p}$. With multivariate Gaussian distribution approximation as
stated above, the differential entropy shows the compactness of a 3D point
set.

Because we have no prior knowledge of targets and the UAV state estimator is
not accurate, it is challenging to find a sufficient condition for
localization. In practice, we jointly consider the necessary conditions such
as the largest eigenvalue $\lambda_{m}$, KL divergence
$D_{KL}(\mathcal{N}_{0}||\mathcal{N}_{1})$, and differential entropy $h_{p}$
to decide the status of target localization. For example, $\lambda_{m}$ should
be smaller than a threshold. When 3D points are converged, any new update will
not change the 3D point distribution, and the 3D points should be compact.
Thus we also need thresholds for $D_{KL}(\mathcal{N}_{0}||\mathcal{N}_{1})$
and $h_{p}$.

Figure 5: UAV motion system.222https://github.com/ZhiangChen/uav_motion

###  III-E Motion Planning

We need multiple, different camera perspectives with bounding boxes to
localize a target. View planning is necessary especially when there are noises
in UAV state $T_{c}^{w}$ and bounding box $\mathbf{x}$. A lawn-mower flight
pattern is initialized to search targets of interest. Once a set of 3D points
has a largest eigenvalue $\lambda_{m}$ and a differential entropy $h_{p}$ that
are smaller than certain thresholds but not small enough to accurately
localize the target due to the lack of enough camera perspectives, we switch
to a motion planning for fine localization.

We approach the motion planning problem by first considering geometric
constraints and then finding a feasible solution. We want to have only one
motion system in all target search, fine localization, and mapping. The motion
system is illustrated in Fig. 2. The trajectory generator takes keypoints from
the path generator and minimizes the snap of 10th-degree polynomials [9]. The
geometric controller [10, 11] takes a desired robot’s position and yaw heading
and outputs desired robot’s body rate and thrusts. In the target search stage,
the path generator is a hard-coded lawn-mower pattern. For fine localization,
the path generator should produce a set of keypoints, i.e. a set of robot
poses $\\{\mathbf{q}_{k}\\}$ such that the sequential controllers can generate
camera perspectives to reduce uncertainty in the target location. Because the
geometric controller is only concerned with desired position and yaw, we add
maximum acceleration and maximum velocity constraints in trajectory generator
so that UAV’s pose can be approximated to hovering. That is, UAV’s roll and
pitch angles will be negligible to affect the target projections in images.
For view planning, we simply consider hovering pose without other UAV
dynamics. The sketch of the motion planning is presented in Fig. 6. Because
the distance $d_{1}$ between $C_{e}$ and the camera is much larger than the
distance $d_{2}$ between the camera to the UAV ($d_{1}:d_{2}>10:1$), we do not
distinguish camera position or UAV position in the view planning.

We constrain the UAV positions on the horizontal plane that is parallel to
$XY$ plane in world frame and has the same height as the lawn-mower search
height $h_{s}$. There are four variables to be determined $(x,y,yaw,\gamma)$,
where $(x,y)$ is the UAV coordinates on the horizontal plane in world frame,
$yaw$ is the UAV’s heading, and $\gamma$ is the angle between the camera $z$
axis and the horizontal plane (Fig. 6). Because we use the 3D points
information from rough localization to decide a view planning and the rough
localization is not accurate, the 3D points projection in the image can
deviate from the actual target projection. To alleviate this issue, we set the
camera $z$ axis to pass through the 3D point center (mean Euclidean distance)
$C_{e}$ to keep the target projection in the image frame as much as possible.
Thus the UAV’s heading $yaw$ is correspondingly determined.

Figure 6: Path planning for fine localization: The ellipse at the bottom
represents a sideview of the ellipsoid from PCA. $C_{e}=(x_{e},y_{e},z_{e})$
is the center of the ellipsoid. The dark-gray rounded rectangular represents a
target with the center $C_{t}$. The large ellipse on the top is the circular
path with the center $C_{l}=(x_{e},y_{e},h_{s})$. $\gamma$ is the angle
between the camera $z$ axis and the horizontal plane. $P_{t}$ is the path
planning start pose; $P_{t+1}$ is an illustrated next-best-view pose.

We decide the next-best-view pose to align the camera $z$ axis to the
eigenvector with the smallest eigenvalue (smallest eigenvector) from PCA of 3D
points. That is, the image plane from the next-best-view pose is parallel to
the plane formed by the eigenvectors with the largest and the second largest
values, which are the directions with the two largest uncertainty. From such a
camera pose, the bounding box on the image plane can reduce most uncertainty
in the 3D points.

When the smallest eigenvector is parallel to the horizontal plane, the next
best view will be at infinity and $\gamma\to 0$. We relax the optimality by
fixing the angle $\gamma$ between the camera $z$ axis and the horizontal plane
and minimizing the angle between camera $z$ axis and the smallest eigenvector.
By having a fixed angle $\gamma$, the remaining two variables $(x,y)$ will be
constrained by a circle
$\mathbf{C}_{l}:(x-x_{e})^{2}+(y-y_{e})^{2}=\frac{(h_{s}-z_{e})^{2}}{\tan^{2}{(\gamma)}}$
(notations are defined in Fig. 6). Due to the ambiguity of the eigenvector
direction, we set the smallest eigenvector direction to have a non-negative
value along $z$ world axis $\upsilon_{z}$. When $\upsilon_{z}>0$, the solution
to the minimization of the angle between camera $z$ axis and the smallest
eigenvector is the intersection between the circle and the eigenvector
projection on the circle plane $\mathbf{C}_{l}$. When $\upsilon_{z}=0$, we set
the next-best-view point as the farthest point on the circle $\mathbf{C}_{l}$
to the current UAV position. We add a circular path to connect the current UAV
pose and the next-best-view pose. The start keypoint is current UAV pose. The
next keypoint is the closest point on the circle $\mathbf{C}_{l}$. Then the
UAV will follow the circle curve $\mathbf{C}_{l}$ to approach the next-best-
view pose and the heading $yaw$ is always toward the 3D points center $C_{e}$.

Figure 7: Path planning for mapping: The small yellow dots represent 3D points
of a target from fine localization. $C_{e}$ is the center of the 3D points
(mean Euclidean distance). $r_{e}$ is the radius of the cylinder. $Z_{m}$ is
the altitude of the cylinder top. $Z_{n}$ is the altitude of the cylinder
bottom. $r_{m}$ is the distance to be kept between the UAV and the cylindrical
curve. The vertical scanning $\beta$ is an angle slightly smaller than the
angle between the upper scanning ray and the lower scanning ray of the camera.
$\beta$ is similar to the concept of camera vertical field of view but is
smaller. The three illustrated circles are the planned path for mapping in
this case. $\gamma_{m}$ is the lower scanning angle
$\gamma_{m}=\gamma-\frac{\beta}{2}$. $\gamma_{n}$ is the upper scanning angle
$\gamma_{n}=\gamma+\frac{\beta}{2}$ Figure 8: Experiment I Gazebo simulation
from DEM: (A) PBR world. (B) 3DR Iris with RGBD camera. Figure 9: Localization
and mapping visualization in Rviz: (A, B) target localization. (C, D) target
mapping. The green curves represents UAV path. The vertical red ellipsoid in
(A) is formed by three eigenvectors and their corresponding eigenvalues from
PCA of 3D points, which indicate uncertainties along three orthogonal axes.

##  IV target mapping

We deploy an RGBD SLAM [12] for target mapping after fine localization. We use
the same motion system, except that the keypoints from the path generator
should include enough camera views to densely map the full-body of a target.
The 3D points from the fine localization provide occupancy information of a
target, which is exploited to plan a path for the mapping. We apply a circular
scanning for this purpose as shown in Fig. 7. We first fit the 3D points with
a vertical cylinder: the cylinder’s axis traverses the center of the 3D points
$C_{e}$, its top is at the highest 3D point, its bottom is at the lowest 3D
point, and its radius is the smallest radius that can include all 3D points.
One objective of the path planning is to provide enough camera views to
determine the occupancy certainty of the space around the cylinder.

We deploy a circular path for the mapping, which can produce 360-degree scans
of the target (Fig. 7). The circles lie on horizontal planes that are parallel
to $XY$ plane in world frame. The centers of the circles are on the axis of
the cylinder. We set the circle radius $r_{c}=r_{e}+r_{m}$ to keep some
distance between the UAV and the cylinder. The spacing and number of the
circles are determined by the camera mount angle $\gamma$ and vertical
scanning angle $\beta$ (Fig. 7) such that the target body will be fully
covered. We have $\beta$ slightly smaller than the camera vertical field of
view so that there are vertical overlaps between circular scans. The first
circle is placed at the bottom where the cylinder bottom can be just covered
by the lower scanning ray of the camera. The other circles are repeatedly
arranged upward until the cylinder top can be covered by an upper scanning
ray.

Figure 10: Experiment II Gazebo simulation from Bishop fault scarp: (Top)
Gazebo. (Medium) corresponding YOLO object detection. (Bottom) corresponding
localization and mapping visualization in Rviz. In the top row, UAV is
highlighted by red circles. Figure 11: PBR basal details from mapping: 3D map
is represented by point clouds. Aqua square dots represent camera positions.

##  V experiment

We built the target localization and mapping
system333https://github.com/ZhiangChen/target_mappingon ROS [13] and conducted
two experiments in Gazebo simulation using a gaming laptop (Dell G7: Intel
Core i7-8750H, 16GB RAM, and Nvidia GeForce GTX 1060 Max-Q). In both two
experiments, we used a 3DR Iris with an RGBD sensor ($\gamma=55\degree$)
operated by PX4 SITL and MAVROS. The UAV state estimation was maintained by an
Extended Kalman Filter fusing information from IMU in the flight controller
and GPS module. YOLO v2 [14] was finely tuned with the images collected in the
Gazebo worlds, and the real-time inference was deployed using YOLO ROS [15].
We applied RTAB-MAP [12] for RGBD mapping, and the mapping service was only
activated during the mapping stage. The vertical scanning angle $\beta$ was
$40\degree$. For the motion system, the maximum velocity was $1m/s$ and the
maximum acceleration was $1m/s^{2}$.

###  V-A Experiment I

We built a simple Gazebo world based on a digital elevation map (DEM) and
imported 3D precariously balanced rock models into the world as shown in Fig.
8 (A). Lawn-mower flight patterns from various elevations were conducted, and
images were collected from the camera on the 3D Iris (Fig. 8 (B)). We
annotated 68 images in LabelMe [16] and finely tuned YOLO v2 that was
initialized from the darknet53 model [14].

The results of fine location and mapping are shown in Fig. 9. The 3D Iris
started a lawn-mower search pattern from a relatively high elevation. As a PBR
was roughly localized, fine localization service was triggered and a circular
motion was conducted (Fig. 9 (A)). When the 3D points were converged (Fig. 9
(B)), motion planning for target mapping was activated (Fig. 9 (C)). Then the
UAV moved closer to the target and deployed RGBD SLAM (Fig. 9 (D)).

By the presented system, we can access to the geometric information at the
base of a PBR as shown in Fig. 11, which is crucial to calculate the PBR’s
fragility. Such detailed basal information, however, is not available from
UAS-SfM with a lawn-mower flight pattern.

###  V-B Experiment II

In this environment, the 3D terrain model (Fig. 1 (B)) was built from a fault
scarp near Bishop, California, which was modeled by the previous work [17]. A
3D PBR model was placed on the foot wall of the fault scarp. We used the same
vehicle and examined the presented system. As a similar process as the
Experiment I, we collected aerial images from different elevations, annotated
162 images, and finely tuned YOLO v2.

The localization and mapping processes are shown in Fig. 10. From left to
right columns, the 3D Iris first tracked the PBR in image coordinates and
generated 1000 3D points in the polyhedral cone. When the 3D points were
roughly converged in the second column, fine localization service was
requested, and the UAV started a circular motion. The fine localization
service was done shortly as shown in the third column. Although the objective
pose in localization stage is the next-best-view pose that aims to reduce the
most uncertainty in 3D points, from both experiments, the targets could be
finely localized before the UAV reached the next-best-view poses. Once the PBR
was finely localized, RGBD mapping service was triggered (the fourth column),
and the UAV moved close to the PBR and conducted circular motion for target
mapping. The mapping service was done in the last column.

##  VI contribution and future work

We present a localization and mapping system for UAVs to map sparse geologic
features such as PBRs, whose geometric fragility parameters can provide
valuable information on earthquake processes and landscape development. The
presented system leverages semantics and features from object detection to
track targets, localizes the targets from multiple different camera
perspectives using a depth filter based algorithm, and maps the targets by
RGBD SLAM. Such a system provides access to autonomous PBR mapping, which was
lacking in field geology but will be essential to geologic model assessment.

The latency in target tracking increases when there are more than 8 objects
tracked in the same image, which can cause false 3D point generation issue. We
will alleviate this problem by using our target tracking algorithm to generate
more labels offline, which will be used to retrain and improve the object
detection network. The bounding box tracking will be substituted with tracking
algorithms without using image features [18]. Additionally, in the fine
localization stage, bundle adjustment can be used to jointly improve the
accuracy of the target localization and the UAV state estimation.

Our aim is to deploy this localization and mapping system targeting actual
boulder fields with many targets and assess the quality of the final 3D target
mapping, which will lead to subsequent applications to earthquake assessment
and interpretation of landscape change and evolution.

## ACKNOWLEDGEMENTS

This work was supported in part by Southern California Earthquake Center
(SCEC) award 20129, National Science Foundation award CNS-1521617, and
National Aeronautics and Space Administration STTR award 19-1-T4.01-2855.

## References

  * [1] Andreas Bircher et al. “Receding Horizon "Next-best-view" Planner for 3D Exploration” In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2016, pp. 1462–1468
  * [2] Jeffrey Delmerico, Elias Mueggler, Julia Nitsch and Davide Scaramuzza “Active Autonomous Aerial Exploration for Ground Robot Path Planning” In _IEEE Robotics and Automation Letters_ 2.2 IEEE, 2017, pp. 664–671
  * [3] James N Brune “Precariously Balanced Rocks and Ground-motion Maps for Southern California” In _Bulletin of the Seismological Society of America_ 86.1A The Seismological Society of America, 1996, pp. 43–54
  * [4] Abdolrasool Anooshehpoor, James N Brune and Yuehua Zeng “Methodology for Obtaining Constraints on Ground Motion from Precariously Balanced Rocks” In _Bulletin of the Seismological Society of America_ 94.1 Seismological Society of America, 2004, pp. 285–303
  * [5] David E Haddad et al. “Estimating Two-dimensional Static Stabilities and Geomorphic Settings of Precariously Balanced Rocks from Unconstrained Digital Photographs” In _Geosphere_ 8.5 Geological Society of America, 2012, pp. 1042–1053
  * [6] David E Haddad et al. “Applications of Airborne and Terrestrial Laser Scanning to Paleoseismology” In _Geosphere_ 8.4 Geological Society of America, 2012, pp. 771–786
  * [7] Christian Forster, Matia Pizzoli and Davide Scaramuzza “SVO: Fast Semi-Direct Monocular Visual Odometry” In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2014
  * [8] Richard Hartley and Andrew Zisserman “Multiple View Geometry in Computer Vision” Cambridge University Press, 2003
  * [9] Charles Richter, Adam Bry and Nicholas Roy “Polynomial Trajectory Planning for Aggressive Quadrotor Flight in Dense Indoor Environments” In _Robotics Research_ Springer, 2016, pp. 649–666
  * [10] Jaeyoung Lim “mavros_controllers - Aggressive Trajectory Tracking Using Mavros for PX4 Enabled Vehicles”, 2019 DOI: 10.5281/zenodo.2652888
  * [11] Taeyoung Lee, Melvin Leok and N Harris McClamroch “Geometric Tracking Control of a Quadrotor UAV on SE (3)” In _49th IEEE Conference on Decision and Control (CDC)_ , 2010, pp. 5420–5425
  * [12] Mathieu Labbé and François Michaud “RTAB-Map as An Open-source Lidar and Visual Simultaneous Localization and Mapping Library for Large-scale and Long-term Online Operation” In _Journal of Field Robotics_ 36.2 Wiley Online Library, 2019, pp. 416–446
  * [13] Stanford Artificial Intelligence Laboratory. “Robotic Operating System”, 2018 URL: https://www.ros.org
  * [14] Joseph Redmon and Ali Farhadi “YOLO9000: Better, Faster, Stronger” In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7263–7271
  * [15] Marko Bjelonic “YOLO ROS: Real-Time Object Detection for ROS”, https://github.com/leggedrobotics/darknet_ros, 2018
  * [16] Bryan C Russell, Antonio Torralba, Kevin P Murphy and William T Freeman “LabelMe: A Database and Web-based Tool for Image Annotation” In _International journal of computer vision_ 77.1-3 Springer, 2008, pp. 157–173
  * [17] Zhiang Chen et al. “Geomorphological Analysis Using Unpiloted Aircraft Systems, Structure from Motion, and Deep Learning” In _arXiv preprint arXiv:1909.12874_ , 2019
  * [18] Erik Bochinski, Volker Eiselein and Thomas Sikora “High-Speed Tracking-by-Detection Without Using Image Information” In _International Workshop on Traffic and Street Surveillance for Safety and Security at IEEE AVSS 2017_ , 2017 URL: http://elvera.nue.tu-berlin.de/files/1517Bochinski2017.pdf


# Predicting Opioid Use Outcomes in Minoritized Communities

Abhay Goyal  Dept. of CS, Missouri S & TUSA ,  Nimay Parekh  Jivox, USAUSA ,
Lam Yin Cheung  Yale University, USAUSA ,  Koustuv Saha  University of
Illinois Urbana-Champaign, USAUSA ,  Frederick L Altice  Yale University,
USAUSA ,  Robin O’hanlon  Memorial Sloan Kettering Cancer Center, USAUSA ,
Roger Ho Chun Man  Institute for Health Innovation and Technology
(iHealthtech), NUS, SingaporeSingapore ,  Christian Poellabauer  Florida
International University, USAUSA ,  Honoria Guarino  CUNY, USAUSA ,  Pedro
Mateu Gelabert  CUNY, USAUSA and  Navin Kumar  New York University, USAUSA

(2018)

###### Abstract.

Machine learning algorithms can sometimes exacerbate health disparities based
on ethnicity, gender, and other factors. There has been limited work at
exploring potential biases within algorithms deployed on a small scale, and/or
within minoritized communities. Understanding the nature of potential biases
may improve the prediction of various health outcomes. As a case study, we
used data from a sample of 539 young adults from minoritized communities who
engaged in nonmedical use of prescription opioids and/or heroin. We addressed
the indicated issues through the following contributions: 1) Using machine
learning techniques, we predicted a range of opioid use outcomes for
participants in our dataset; 2) We assessed if algorithms trained only on a
majority sub-sample (e.g., Non-Hispanic/Latino, male), could accurately
predict opioid use outcomes for a minoritized sub-sample (e.g., Latino,
female). Results indicated that models trained on a random sample of our data
could predict a range of opioid use outcomes with high precision. However, we
noted a decrease in precision when we trained our models on data from a
majority sub-sample, and tested these models on a minoritized sub-sample. We
posit that a range of cultural factors and systemic forms of discrimination
are not captured by data from majority sub-samples. Broadly, for predictions
to be valid, models should be trained on data that includes adequate
representation of the groups of people about whom predictions will be made.
Stakeholders may utilize our findings to mitigate biases in models for
predicting opioid use outcomes within minoritized communities.

Opioid Use, Bias, Marginalization

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Impact of Computing Culture††ccs:
Recognizing and Defining Computational Problems

##  1\. Introduction

Within the healthcare space, machine learning algorithms can sometimes
exacerbate racial, ethnic, and gender disparities, among others. Many machine
learning algorithms are trained on data from majority populations, thereby
generating less accurate or reliable results for minoritized groups
(kostick2022mitigating, ). For example, in a widely used algorithm, at a given
risk score, the technique falsely concludes that Black individuals are
healthier than equally sick White individuals (obermeyer2019dissecting, ).
Thus, such large-scale algorithms can often perpetuate biases. There has been
limited work at exploring potential biases in algorithms deployed within
minoritized communities. In particular, minimal research has detailed how
biases may manifest in algorithms developed by insurance companies to predict
opioid use outcomes, or opioid overdoses among people who use opioids in urban
areas. An algorithm trained on data from white individuals may provide
incorrect estimates for Hispanic/Latino individuals, perhaps resulting in
adverse health outcomes.

Since predicting opioid use outcomes is important to improving health in
populations often neglected by larger health systems (kumar2022substance, ),
our goal is to examine how machine learning algorithms perform at determining
opioid use outcomes within minoritized communities. As a case study, we used
data from a sample of 539 young adults who engaged in nonmedical use of
prescription opioids and/or heroin (mateu2015injection, ). The prevalence and
incidence of opioid use has increased rapidly in the US in the past two
decades, which is related to concomitant increases in opioid dependence,
accidental overdose and death. We addressed the indicated issues through the
following contributions: 1) Using machine learning techniques, we predicted
opioid use outcomes for participants in our dataset; 2) We assessed if an
algorithm trained on a majority sub-sample e.g., Non-Hispanic/Latino, male,
could accurately predict opioid use outcomes for a minoritized sub-sample
e.g., Latino, female. Our analysis was conducted to replicate possible real-
world scenarios, and provide insight on how to improve broad health outcomes
via predictive modeling. For example, if an insurance company primarily caters
to Non-Hispanic/Latino individuals, models trained on data from Non-
Hispanic/Latino individuals may not predict life insurance costing accurately
for Hispanic individuals seeking treatment, and our analysis can provide
understanding into such scenarios.

##  2\. Related Work

Several papers explored the creation of frameworks to mitigate biases within
algorithms used in the healthcare space. A paper indicated that while
technical AI advances in healthcare showcased impressive performances in lab
settings, they seemed to fail when moving to clinical practice
(yildirim2021technical, ). The authors discussed three interrelated challenges
to AI in clinical practice: technical feasibility, financial viability, and
clinician acceptance. Similar work explored queer concerns in privacy,
censorship, language, online safety, and health (tomasev2021fairness, ). Most
current approaches for algorithmic fairness assumed that the target
characteristics for fairness, such as, race and legal gender, can be observed
or recorded. The authors highlighted the importance of developing new
approaches for algorithmic fairness that are different from the prevailing
assumption of observed characteristics. Another paper indicated that AI-based
applications raised not only ethical but legal and safety concerns
(kostick2022mitigating, ), where models trained on data from majority sub-
samples could generate less accurate or reliable results for minorities and
other disadvantaged groups. Research has developed algorithms for a range of
healthcare applications, such as to classify key types of gastrointestinal
tract conditions (thambawita2020extensive, ), and to classify skin lesions
(lu2022fair, ). Several studies have also used machine learning techniques to
predict substance use outcomes. One study evaluated prediction performance of
three different machine learning techniques in predicting opioid misuse among
US adolescents, using the 2015–2017 National Survey on Drug Use and Health
(NSDUH) (han2020using, ). Prediction models were developed using three
algorithms, including neural networks, distributed random forest, and gradient
Boosting. Findings suggested that machine learning techniques could be useful
for predicting adolescent opioid misuse. Another study used a random forest
classifier and NSDUH data to predict adults at risk for opioid use disorder
(OUD) (wadekar2019predicting, ). Early initiation of cannabis, prior to 18
years of age, emerged as the dominant predictor for developing OUD in adult
life. Other work used administrative claims data of 560,057 Medicare
beneficiaries (lo2019evaluation, ), and found that deep neural networks and
gradient boosting machine models outperformed other methods for identifying
opioid overdose risk.

Overall, research around whether machine learning exacerbates disparities
within healthcare was limited, especially among smaller-scale cases. Most
studies centered on developing frameworks to mitigate biases, or explored
machine learning algorithms for various healthcare use cases with limited
discussion on possible biases that may arise. We thus predicted a range of
opioid use outcomes for minoritized sub-samples, and more importantly,
determined how biases may occur in our use case.

##  3\. Data

Sociodemographic Feature |  Breakdown
---|---
Gender |  Male (N=365)
Female (N=170)
Ethnicity |  Non-Hispanic/Latino (N=383)
Hispanic/Latino (N=154)
Race |  White (N=497)
Asian (N=7)
Black/African American (N=42)
Housing status |  Not homeless (N=400)
Homeless (N=137)
Household income growing up |  Income$>$25000 (N=136)
Income$<$25000 (N=390)
Table 1. Sociodemographic breakdown. Rows do not always add to 539 as we
displayed larger subcategories.

539 participants were recruited from July 2014 - October 2015 using respondent
driven sampling, a form of chain-referral sampling designed to engage hard-to-
reach populations. Most participants (Table 1) were lower income
($<$25000/year) Non-Hispanic white men (N=365) who were not homeless were
over-represented in the data. Women (N=170), Hispanic/Latino (N=154), Asian
(N=7), Black (N=42), homeless (N=137), and high income participants (N=136)
were underrepresented. The questionnaire used the standard items for
collecting race and ethnicity as per the 1997 Office of Management and Budget
(OMB) Revisions to the Standards for the Classification of Federal Data on
Race and Ethnicity. These items are commonly used for federal data collection
purposes. We conformed to these race and ethnicity criteria as the data
collection process was federally funded. The revised OMB standards contained
five minimum categories for race: American Indian or Alaska Native, Asian,
Black or African American, Native Hawaiian or Other Pacific Islander, and
white. There were two categories for ethnicity: Hispanic or Latino and Not
Hispanic or Latino. Thus, we collected both race and ethnicity data.
Participants were asked to recruit fellow prescription opioid and/or heroin
users. Several papers have been published with this dataset and further
information on data collection and methods was contained in past work
(friedman2017sexual, ; mateu2015injection, ). Eligibility criteria included:
nonmedical use of prescription opioids and/or heroin use in the past 30 days;
current residence in NYC; aged 18–29 years-old; English-speaking; and ability
to provide informed consent. Structured interviews lasted between 90 and 120
minutes and were used to collect quantitative data on drug use, sexual
behavior, sexual and injection partnerships and networks, overdose, drug
treatment, HIV and hepatitis C knowledge and testing history.

##  4\. Methods

Features of interest.

Outcome |  Possible values
---|---
Have you injected drugs in the past 3 months? |  No (N=222), Yes (N=317)
Have you ever been in treatment for drug use? |  No (N=155), Yes (N=384)
Table 2. Opioid use outcomes of interest

Category |  Feature |  Possible values
---|---|---
Sociodemographics |  1\. Do you consider yourself to be Hispanic or Latino/a? |  No, Yes
2\. Which racial group do you consider yourself to be in? |  a. American Indian
b. Asian
c. Black/African American
d. Native Hawaiian/Pacific Islander
|  e. White
3\. What borough do you currently live in? |  a. Manhattan
b. Staten Island
c. Brooklyn
d. Bronx
e. Queens
4\. Do you consider yourself to be male, female or transgender? |  a. Male
b. Female
c. Transgender Male to Female
d. Transgender Female to Male
Sexual Violence |  1\. Have you ever had the experience while you were using drugs when someone touched or rubbed against your body in a sexual manner without your consent? |  No, Yes
2\. Have you ever felt that someone expected you to have sex with them because you were using drugs together? |  No, Yes
3\. While you were using drugs, did someone ever have sex (oral, vaginal, and/or anal) with you without your consent? |  No, Yes
Perceived Risk for HIV, HCV & STIs |  1\. How concerned are you about getting HIV/AIDS? |  a. Not At All Concerned
b. Somewhat Concerned
c .Very Concerned
d. I Have It
2\. How concerned are you about getting Hepatitis C? |  a. Not At All Concerned
b. Somewhat Concerned
c. Very Concerned
d. I have it
3\. How concerned are you about getting chlamydia? |  a.Not At All Concerned
b. Somewhat Concerned
c. Very Concerned
d. I have it

Mental Health |  1\. Have you ever been diagnosed with a psychological or emotional problem? |  No, Yes
---|---|---
2\. How many times in your life, if any, have you been treated for any psychological or emotional problem in a hospital or inpatient unit? |  No. Of Episodes
3\. Experienced serious depression? |  No, Yes

Table 3. Features of interest

Opioid use outcomes of interest. To develop a list of outcomes to predict, we
first selected content experts with a research specialization in opioid use
who assembled a list of features relevant to opioid use outcomes. Content
experts had published ¿10 peer-reviewed articles in opioid use, broadly
defined. All outcomes were derived from the quantitative survey data
collected. Content experts only retained items that had been agreed upon after
discussion with any disagreements resolved by a third content expert. Our
final list of two outcomes that we sought to predict using machine learning
are in Table 2. Feature selection. Content experts selected an initial list of
questions for feature selection. All features were derived from the survey
questions. Content experts only retained items that had been agreed upon after
discussion. Experts only selected items that did not replicate the opioid use
outcomes of interest. For example, the question In the past 30 days, did you
take prescription opioid(s) nonmedically? was not included in our initial set
of features as it was too similar to the opioid use outcomes we were trying to
predict. Any disagreements were resolved by a third content expert. This
process resulted in 327 features. Following this, we used the following
feature selection techniques to reduce the feature set: Coefficient of
variation (CV); Pairwise correlations (PC). CV: We employed the CV method to
reduce the feature space based on explained variance. This involved measuring
the ratio of standard deviation to mean for each feature and removing any
outliers that exceeded a threshold CV of two standard deviations from the mean
(21.67). Two features exceeded this threshold and were subsequently dropped.
PC: Correlated features which can negatively impact machine learning
prediction models by producing unstable solutions or obscuring the
interactions between significant features. To identify uncorrelated features,
we calculated Pearson’s correlation coefficient between all feature pairs and
removed any with an absolute correlation value above the threshold of 0.8.
This resulted in the removal of 74 features. Examples from the final list of
280 selected features were presented in Table 3. Feature names and options
were verbatim from the interviews. Missing data was imputed with a value of 0.

Models. We selected several commonly used models to predict opioid use
outcomes, as below.
Random Forest Classifier. Random forest classifier is an ensemble machine
learning method that is computationally efficient over large datasets. The
Random forest uses multiple decision trees and then finds the most predicted
class when making the prediction. We used an estimator size of 100 and depth
to 100 to allow for pruning to take place. For initial feature prediction, we
used a train size of 70% and test size of 30%.
Bagging Classifier. Bagging classifier is an ensemble method which uses other
basic methods, in our case, decision trees and bags or combines the different
results together. It then chooses the predicted label as the one which has
been predicted by most of the decision trees. In bagging classifier, we used a
70% and 30% train/test split.
Gradient Boosting Classifier. The gradient boosting classifier builds
different models sequentially and it builds the next model such that the
losses of the previous one are minimized. The loss used in the classifier is
the log-likelihood.
Adaptive Boosting Classifier. The adaptive boosting classifier (AdaBoost) is
made such that it adapts based on errors made by the previous classifier. The
output of the other learning algorithms is combined into a weighted sum that
represents the final output of the boosted classifier. We used the decision
trees as the base estimator with a depth of 100.

Feature prediction.

Feature |  Majority Sociodemographic Group |  Minority Sociodemographic Group
---|---|---
Do you consider yourself to be male, female or transgender? |  Male (N=365) |  Female (N=170)
Do you consider yourself to be Hispanic or Latino/a |  Non-Hispanic/Latino (N=383) |  Hispanic/Latino (N=154)
Table 4. Features to subset data

We first predicted each of the outcomes in Table 2 using all 280 features,
some of which were indicated in Table 3. We kept to guidelines which indicated
n-1 as the optimal feature size for uncorrelated features (hua2005optimal, ).
We then predicted the same outcomes, but trained our models on a majority sub-
sample, then tested the models on a individual minoritized sub-sample, based
on the features in Table 4. Content experts selected majority and
minoritized/minority sub-samples based on where the greatest discrepancy in
opioid use outcomes may exist between two groups, e.g., male/female. By
majority and minoritized/minority, we referred to these demographic groups’
representation within this particular sample, not necessarily their
representation in society at large. We mainly referred to the distribution of
these groups within this sample, but used the data to make a larger point
about the use of machine learning algorithms to predict health risk in a broad
range of US populations. A group’s status as majority versus minority was
context-dependent; for example, while females were a minority in this sample,
they were not a minority in the general US population. For each outcome that
we were trying to predict, we trained our model on only data from a single
sub-sample indicated in the Majority Sociodemographic Group column of Table 4,
and tested the model on data from a single sub-sample detailed in the Minority
Sociodemographic Group column of Table 4. We repeated this process with each
minoritized/minority sub-sample pair in Table 4, and outcome as per Table 2.
For example, in predicting the variable Have you injected drugs in the past 3
months?, we trained our models only on data from Non-Hispanic/Latino
participants (N=383), but tested models with data only from Hispanic/Latino
participants (N=154). We then did the same, but with male (N=365) and female
participants (N=170).

##  5\. Results

Feature | Model | Metrics
---|---|---
ROC AUC | Accuracy | F1-Score | Average Precision
Have you injected drugs in the past 3 months? | Random Forest | 0.785 | 0.802 | 0.802 | 0.775
Adaptive Boost | 0.809 | 0.820 | 0.820 | 0.799
Gradient Boost | 0.825 | 0.839 | 0.839 | 0.811
Bagging classifier | 0.757 | 0.759 | 0.759 | 0.759
Have you ever been in treatment for drug use? | Random Forest | 0.695 | 0.777 | 0.777 | 0.787
Adaptive Boost | 0.710 | 0.753 | 0.753 | 0.798
Gradient Boost | 0.747 | 0.796 | 0.796 | 0.819
Bagging classifier | 0.734 | 0.777 | 0.777 | 0.811
Table 5. Predictions for opioid use outcomes

We first used a range of models to predict opioid use outcomes as per Table 5.
While we provided several model metrics, we believed that average precision
provided the most insight as precision is the best metric to be used when the
classes are highly imbalanced (armah2014deep, ). Precision is the fraction of
instances that the model classifies as positive that are indeed correct. We
indicated in bold the highest precision for each feature we predicted.
Moreover, precision provided the proportion of positive identifications which
were accurately predicted, central to our analysis. Based on all metrics for
each model per feature, we indicated in bold the best performing model for
each feature. Overall, our models predicted the opioid use outcomes of
interest with relatively high precision, likely due to the survey questions
being specifically designed to understand opioid use outcomes and the fact
that the survey was administered to a substance-using population.

To provide more insight into our results, we used SHapley Additive
exPlanations (SHAP) to understand how each feature contributed to predicting
the opioid use outcomes. SHAP quantifies the contribution each feature brings
to the prediction made by the model. We detailed beeswarm plots (Figure 1) for
the models with the highest precision, for each opioid use outcome predicted.
Features that contributed the most to the prediction are closer to the top of
the plot. Broadly, having peers who engaged in opioid use seemed to contribute
most to the prediction of being in treatment for drug use, and injecting
drugs. Past work suggested the role of peer effects in initiating opioid use
(henneberger2021peer, ). Peer effects are when one’s behavior is affected by
the behavior of their peers. However, apart from peer effects, there were
likely other features which more accurately predict opioid use. For example,
engagement in harm reduction practices, such as using clean syringes, or
proximity to a methadone treatment center, may relate to injecting drugs, but
such features were not collected in the data.

(a) Injection drug use in the last three months  (b) Being in treatment for
drug use

Figure 1. SHAP visualizations for opioid use outcomes Feature | Majority | Minority | Model | Metrics
---|---|---|---|---
Accuracy | F1-Score | Average Precision
Have you ever injected drugs in the past 3 months? |  Male (N=369) |  Female (N=170) | Random Forest | 0.758 | 0.758 | 0.721
Adaptive Boosting | 0.788 | 0.788 | 0.754
Gradient Boosting | 0.794 | 0.794 | 0.750
Bagging classifier | 0.723 | 0.723 | 0.728
Non-Hispanic/ Latino (N=385) |  Hispanic/ Latino (N=154) | Random Forest | 0.792 | 0.792 | 0.629
Adaptive Boosting | 0.831 | 0.831 | 0.674
Gradient Boosting | 0.805 | 0.805 | 0.639
Bagging classifier | 0.798 | 0.798 | 0.628
Have you ever been in treatment for drug use? |  Male (N=369) |  Female (N=170) | Random Forest | 0.788 | 0.788 | 0.797
Adaptive Boosting | 0.782 | 0.782 | 0.812
Gradient Boosting | 0.709 | 0.8 | 0.8
Bagging classifier | 0.764 | 0.764 | 0.816
Non-Hispanic/ Latino (N=385) |  Hispanic/ Latino (N=154) | Random Forest | 0.720 | 0.720 | 0.673
Adaptive Boosting | 0.714 | 0.714 | 0.686
Gradient Boosting | 0.727 | 0.727 | 0.688
Bagging classifier | 0.694 | 0.694 | 0.667
Table 6. Predictions for opioid use outcomes by training model on data from a
majority sub-sample and testing model on data from a minority sub-sample

We then sought to determine if models trained on a majority sub-sample could
predict outcomes for a minoritized sub-sample (Table 6). We indicated in bold
the highest precision for each feature we predicted. As expected, training
models on a majority sub-sample to predict opioid use outcomes for a
minoritized sub-sample provided poorer results than compared to our analysis
in Table 5. We compared precision between our two sets of analysis (Table 5,
Table 6). Essentially, we compared the differences in precision between: 1)
training a model on a randomly selected subset of our data and testing the
same model on a randomly selected subset of our data; 2) training a model on
data from a majority sub-sample (e.g., Non-Hispanic/Latino participants), and
testing the same model on data from a minoritized sub-sample (e.g.,
Hispanic/Latino participants).

We noted a decrease in precision when predicting opioid use outcomes across
the minoritized categories of interest. Results may suggest that using data
from males to predict female opioid use, and predicting Hispanic/Latino opioid
use with data from non-Hispanics/Latinos was infeasible. The reduced precision
in predicting opioid use outcomes may be attributed to the specific patterns
of drug use among young people in our dataset. Most of the participants in our
study began using opioids during their teenage years, with prescription
opioids (POs) being the primary form of nonmedical use. However, by the time
they were included in the study (with an average age of 24.5), many had
switched to using heroin instead of POs (guarino2018young, ). This trajectory
was influenced by their birth cohort and the availability of different drugs
during their adolescence. During the time when our sample were 16-17 years
old, PO prescribing in the US was at an all-time high, and diverted POs were
easily accessible. As POs became more challenging and expensive to obtain,
those who developed opioid use disorder usually transitioned to using heroin,
which was cheaper and more potent in New York City. There were also
racial/ethnic differences in these patterns, with white individuals more
likely to use heroin and have longer opioid use histories, while other groups
were more likely to use POs only and have shorter and less severe opioid use
histories. PO use was a recent phenomenon in the peer networks of young Black
and Latino individuals at the time of data collection, but white communities
had been affected by the opioid epidemic earlier, resulting in more prolonged
exposure to opioids among white adolescents (guarino2018young, ). Furthermore,
individuals who had longer histories of opioid use and more severe opioid use
disorder, predominantly male and white individuals, were more likely to engage
in the challenging process of breaking down PO pills for injection
(guarino2018young, ). These differing behaviors across gender and
ethnic/racial categories may elucidate why models trained on data from males
did not accurately predict female injection use. Harm reduction policy
approaches are generally lacking in low-income ethnic minority neighborhoods.
Thus, datasets to predict opioid use may not have features which model harm
reduction initiatives in such communities, perhaps resulting in poor precision
to predict opioid use outcomes for Hispanic/Latino populations with models
trained on a majority population. Similarly, unstable housing is a predictor
for opioid use outcomes, but datasets may not contain data from unhoused
people, or features that aggregate the experience of being unhoused.

We suggest that researchers designing such models in diverse populations need
to be aware of cultural nuances, such as differential opioid use behaviors
across racial/ethnic groups. For example, training a model on data from one
racial/ethnic community and using it to predict injection drug use in another
community may perpetuate biases, and completely ignore racial/ethnic nuances
around opioid use. While there have been numerous attempts to predict opioid
use outcomes using a range of machine learning techniques
(tseregounis2021assessing, ), models have a high rate of false positives,
largely because opioid use outcomes such as overdose are rare in the general
population, and some important, non-person-level factors contribute strongly
to overdose - such as potency of a particular batch or dose of drugs.
Moreover, these models are rarely implemented in clinical practice. Our
results suggest that predicting opioid use outcomes within minoritized
communities may have low precision, indicating that much more work is required
before machine learning algorithms to predict such outcomes can be
productionized in healthcare systems.

##  6\. Discussion

Implications of findings. Our goal was to predict a range of opioid use
outcomes as assessed in a sample of young people who use opioids, and
determine if an algorithm trained only on a majority sub-sample could
accurately predict opioid use outcomes for a minoritized sub-sample, to inform
best practices for the use of predictive algorithms in substance-using
populations. Models were able to predict recent injection drug use and
participation in drug treatment. The presence of peers who also engaged in
opioid use appeared to play a role in predicting drug treatment and injection
drug use. However, the available data lacked comprehensive information on
other facets of opioid use, such as harm reduction. We noted a decrease in
precision when we trained our models on only data from a majority sub-sample,
and tested these models on a minoritized sub-sample. Overall, machine learning
approaches are only as precise and useful as the data they are trained on, and
to make valid and accurate predictions they must be trained on data from
people who are similar in terms of key sociodemographic characteristics as the
populations about whom predictions will be made.
Recommendations. Key to mitigating biases in models to predict health outcomes
within minoritized communities, is the inclusion of stakeholders at every
stage of the machine learning operations (MLOps) pipeline. For example,
methadone patients need to be involved in the development of models to predict
methadone dropout risk (chen2022partisan, ; chen2023categorizing, ).
Similarly, a committee of ethnic minority individuals can be involved in
auditing algorithms used to detect cardiovascular risk. Insurance companies
and other stakeholders who use machine learning to predict opioid use outcomes
need to be aware that models can exacerbate biases, and seek to improve their
predictive modelling capabilities. Insurance companies that have primarily
white individuals in their datasets should seek to augment their datasets with
individuals from minoritized backgrounds. Such practices can aid providers in
making accurate predictions if their client demographics shift, or if nonwhite
individuals seek treatment. There increasingly exist independent corporations
that audit large scale machine learning models, and such corporations need to
ensure that minoritized communities are adequately represented within the
audit committee. Similarly, governments and similar stakeholders need to
initiate guardrails and implement sound AI-centric policymaking. Interventions
to educate the public on such models is also critical. For example,
stakeholders can develop short videos on YouTube explaining how such
algorithms may reproduce biases. Including features in models that capture the
nuances of opioid use behaviors within minoritized communities is key
(oles2022characterizing, ; kumar2022online, ). Models can include features
that capture individuals’ distance from harm reduction centers, or their
engagement with harm reduction practices, such as the use of sterile syringes,
and fentanyl testing.

Limitations. Findings relied on the validity of questions collected within the
dataset, and there may have been features of interest not included in the
questionnaire. Our data may not be generalizable to other health outcomes or
populations. Questionnaire answers may be unreliable, perhaps further limiting
generalizability. Our dataset is small relative to other similar studies, and
we will collect a larger sample in future work. We were unable to conduct
balancing for our data given the small sample size. The data pertains to a
highly particular subset of people who use drugs, specifically, young adults
residing in NYC who engage in opioid use. Therefore, it remains uncertain to
what degree the outcomes can be applicable to individuals in dissimilar
geographic regions, varying age groups, or those consuming different types of
substances.

## References

  * (1) Armah, G. K., Luo, G., and Qin, K. A deep analysis of the precision formula for imbalanced class distribution.  International Journal of Machine Learning and Computing 4, 5 (2014), 417–422.
  * (2) Chen, K., Babaeianjelodar, M., Shi, Y., Janmohamed, K., Sarkar, R., Weber, I., Davidson, T., De Choudhury, M., Yadav, S., Khudabukhsh, A., et al. Partisan us news media representations of syrian refugees.  arXiv preprint arXiv:2206.09024 (2022).
  * (3) Chen, K., Feng, A., Aanegola, R., Saha, K., Wong, A., Schwitzky, Z., Lee, R. K.-W., O’Hanlon, R., De Choudhury, M., Altice, F. L., et al. Categorizing memes about the ukraine conflict.  In Computational Data and Social Networks: 11th International Conference, CSoNet 2022, Virtual Event, December 5–7, 2022, Proceedings (2023), Springer, pp. 27–38.
  * (4) Friedman, S., Mateu-Gelabert, P., Ruggles, K., Goodbody, E., Syckes, C., Jessell, L., Teubl, J., and Guarino, H. Sexual risk and transmission behaviors, partnerships and settings among young adult nonmedical opioid users in new york city.  AIDS and Behavior 21 (2017), 994–1003.
  * (5) Guarino, H., Mateu-Gelabert, P., Teubl, J., and Goodbody, E. Young adults’ opioid use trajectories: From nonmedical prescription opioid use to heroin, drug injection, drug treatment and overdose.  Addictive behaviors 86 (2018), 118–123.
  * (6) Han, D.-H., Lee, S., and Seo, D.-C. Using machine learning to predict opioid misuse among us adolescents.  Preventive medicine 130 (2020), 105886.
  * (7) Henneberger, A. K., Mushonga, D. R., and Preston, A. M. Peer influence and adolescent substance use: A systematic review of dynamic social network research.  Adolescent Research Review 6 (2021), 57–73.
  * (8) Hua, J., Xiong, Z., Lowey, J., Suh, E., and Dougherty, E. R. Optimal number of features as a function of sample size for various classification rules.  Bioinformatics 21, 8 (2005), 1509–1515.
  * (9) Kostick-Quenet, K. M., Cohen, I. G., Gerke, S., Lo, B., Antaki, J., Movahedi, F., Njah, H., Schoen, L., Estep, J. E., and Blumenthal-Barby, J. Mitigating racial bias in machine learning.  Journal of Law, Medicine & Ethics 50, 1 (2022), 92–100.
  * (10) Kumar, N., Chen, K., Shi, Y., and Altice, F. L. Online platforms’ framing around vaping.  Drug Testing and Analysis (2022).
  * (11) Kumar, N., Janmohamed, K., Nyhan, K., Martins, S. S., Cerda, M., Hasin, D., Scott, J., Frimpong, A. S., Pates, R., Ghandour, L. A., et al. Substance, use in relation to covid-19: a scoping review.  Addictive behaviors 127 (2022), 107213.
  * (12) Lo-Ciganic, W.-H., Huang, J. L., Zhang, H. H., Weiss, J. C., Wu, Y., Kwoh, C. K., Donohue, J. M., Cochran, G., Gordon, A. J., Malone, D. C., et al. Evaluation of machine-learning algorithms for predicting opioid overdose risk among medicare beneficiaries with opioid prescriptions.  JAMA network open 2, 3 (2019), e190968–e190968.
  * (13) Lu, C., Lemay, A., Chang, K., Höbel, K., and Kalpathy-Cramer, J. Fair conformal predictors for applications in medical imaging.  In Proceedings of the AAAI Conference on Artificial Intelligence (2022), vol. 36, pp. 12008–12016.
  * (14) Mateu-Gelabert, P., Guarino, H., Jessell, L., and Teper, A. Injection and sexual hiv/hcv risk behaviors associated with nonmedical use of prescription opioids among young adults in new york city.  Journal of substance abuse treatment 48, 1 (2015), 13–20.
  * (15) Obermeyer, Z., Powers, B., Vogeli, C., and Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations.  Science 366, 6464 (2019), 447–453.
  * (16) Oles, W., Alexander, M., Kumar, N., Howell, B., O’Connor, P. G., Madden, L. M., and Barry, D. T. Characterizing the social support and functioning of a low-threshold medication for opioid use disorder treatment cohort at intake.  BMC psychiatry 22, 1 (2022), 236.
  * (17) Thambawita, V., Jha, D., Hammer, H. L., Johansen, H. D., Johansen, D., Halvorsen, P., and Riegler, M. A. An extensive study on cross-dataset bias and evaluation metrics interpretation for machine learning applied to gastrointestinal tract abnormality classification.  ACM Transactions on Computing for Healthcare 1, 3 (2020), 1–29.
  * (18) Tomasev, N., McKee, K. R., Kay, J., and Mohamed, S. Fairness for unobserved characteristics: Insights from technological impacts on queer communities.  In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (2021), pp. 254–265.
  * (19) Tseregounis, I. E., and Henry, S. G. Assessing opioid overdose risk: a review of clinical prediction models utilizing patient-level data.  Translational Research 234 (2021), 74–87.
  * (20) Wadekar, A. Predicting opioid use disorder (oud) using a random forest.  In 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC) (2019), vol. 1, IEEE, pp. 960–961.
  * (21) Yildirim, N., Zimmerman, J., and Preum, S. Technical feasibility, financial viability, and clinician acceptance: On the many challenges to ai in clinical practice.  In HUMAN@ AAAI Fall Symposium (2021).