#  Classifying geospatial objects from multiview aerial imagery using semantic
meshes

David Russell
University of California, Davis
<EMAIL_ADDRESS>Ben Weinstein
University of Florida
<EMAIL_ADDRESS>David Wettergreen
Carnegie Mellon University
<EMAIL_ADDRESS>Derek Young
University of California, Davis
<EMAIL_ADDRESS>
###### Abstract

Aerial imagery is increasingly used in Earth science and natural resource
management as a complement to labor-intensive ground-based surveys. Aerial
systems can collect overlapping images that provide multiple views of each
location from different perspectives. However, most prediction approaches
(_e.g_. for tree species classification) use a single, synthesized top-down
“orthomosaic” image as input that contains little to no information about the
vertical aspects of objects and may include processing artifacts. We propose
an alternate approach that generates predictions directly on the raw images
and accurately maps these predictions into geospatial coordinates using
semantic meshes. This method—released as a user-friendly open-source
toolkit111https://github.com/open-forest-observatory/geograypher [29]—enables
analysts to use the highest quality data for predictions, capture information
about the sides of objects, and leverage multiple viewpoints of each location
for added robustness. We demonstrate the value of this approach on a new
benchmark dataset222https://osf.io/6snfq/ [30] of four forest sites in the
western U.S. that consists of drone images, photogrammetry results, predicted
tree locations, and species classification data derived from manual surveys.
We show that our proposed multiview method improves classification accuracy
from 53% to 75% relative to an orthomosaic baseline on a challenging cross-
site tree species classification task.

##  1 Introduction

(a) Orthomosaic

(b) Raw nadir

(c) Raw oblique

Figure 1:  The same tree in (a) an orthomosaic created from nadir drone images
collected from 120 m above ground level, (b) a raw nadir drone image collected
from 120 m above ground level, and (c) a raw oblique drone image collected
from 80 m above ground level. Distortions due to orthorectification are
apparent in (a).

Addressing questions in Earth science and natural resource management often
requires detailed knowledge of the locations and identities of resources—for
example, trees, birds, nest sites, and geologic structures. Traditionally,
this information is collected through time- and labor-intensive ground-based
manual surveys, which have limited spatial extent. To overcome these
constraints, modern natural resource inventory approaches have increasingly
leveraged aerial and satellite imagery to derive broad-extent, fine-scale maps
of the resources of interest [5, 15, 33, 6].

Aerial platforms can quickly collect many overlapping images of a region and
are valuable because they can be flown on demand and have high spatial
resolution. The traditional approach to analyzing the data from these
multiview surveys is to first generate an “orthomosaic”, representing a
synthesized top-down (orthographic) perspective of the scene. Orthomosaic-
based analyses have enabled mapping efforts including individual tree
detection [36], classification [22], and trait prediction [20]. Orthomosaic-
based approaches, however, have two important shortcomings. First, the
orthorectification process necessarily introduces some distortion of the scene
(Fig. 1(a)), since the raw 2D images must be warped to the orthographic
perspective and stitched together. Second, the orthomosaic only contains a
top-down view, thus foregoing any opportunity to leverage multiple (e.g.,
oblique) perspectives that likely contain substantial information to support
object detection and characterization (Fig. 1(b), Fig. 1(c)).

Using the information contained in aerial images with multiple views of each
object can substantially improve classification accuracy over an orthomosaic-
based approach [18, 16]. This benefit has been demonstrated most clearly in
the context of wetland detection and segmentation, in which researchers
achieved classification accuracy of 82%, relative to 65% for a baseline
orthoimagery approach [18]. While useful for wetland mapping and agricultural
applications [35], current approaches use a 2.5D representation of the scene
and require the 3D environment to be relatively flat and free of structures
that may create occlusions. This assumption does not hold in systems with
substantial surface relief, such as many forests.

To take advantage of multiview imaging while accommodating complex surface
relief, we adopt a 3D mesh-based approach to geometric reasoning that allows
us to accurately account for inter-object occlusions. Using this mesh
representation, we aggregate segmentation predictions from a deep learning
model from each viewpoint to generate a classified mesh, and we then use this
mesh to classify individual trees to species. The contributions of this work
are as follows:

  * •

A novel workflow for geospatial object classification using semantic meshes
and multiview imagery that yields 75% accuracy in a tree species
classification experiment, compared to 53% for an orthomosaic baseline

  * •

A field-validated dataset to promote research on multiview tree species
classification

  * •

A user-friendly open-source toolkit for generating landscape-scale geospatial
predictions with multiview imagery

##  2 Related Work

###  2.1 Multiview reasoning in natural environments

Initial efforts to apply multiview reasoning to Earth observation data focused
on detailed characterization of spectral reflectance by measuring light
reflected at multiple angles from a given surface (Koukal _et al_. [16], Liu
_et al_. [18]). More recently, focus has shifted to incorporating multi-
perspective imagery into computer vision-based semantic segmentation and
classification routines that account for spatial context around the pixel or
object of interest. Such methods have enabled improved segmentation and
reconstruction of urban scenes using multiview orthorectified multispectral
satellite imagery [24, 17]. One approach developed for wetlands involved
projecting the bounds of geospatial objects (unclassified land cover units)
onto the raw drone images, cropping out each raw view of each object as its
own image, classifying the resulting images using a computer vision algorithm,
and aggregating the multiple predictions (one per drone image the object
appeared in) to a single prediction per object using vote counting [19]. An
open-source tool called EasyIDP333https://github.com/UTokyo-FieldPhenomics-
Lab/EasyIDP [35] aims to facilitate a similar workflow for agricultural
applications by projecting the bounds of a given geospatial region onto the
aerial images. However, these approaches do not translate well to high-relief
systems (e.g. forest canopies) and/or oblique images because (a) they
represent surface relief using a 2.5D digital elevation model that cannot
capture undercut regions, and (b) they do not account for the possibility of
occluding structures.

The works of Russell _et al_. [28] and Andrada _et al_. [3] expand beyond
photogrammetry-based methods by using a drone equipped with a lidar sensor, an
inertial measurement unit, and cameras to classify and map forest fuels in a
structurally-complex scene. These approaches use deep learning applied to the
camera images to classify fuel and use the lidar to identify its 3D location
relative to the drone platform. To build a globally-coherent map from multiple
observations, these approaches use simultaneous localization and mapping
(SLAM) [11] to build a real-time estimate of the drone’s location and thereby
the structure of the world. Unfortunately, SLAM methods are susceptible to
both accumulating errors and catastrophic failures in localization. This can
result in maps that while locally consistent, do not accurately represent the
true geospatial location of objects, and they are therefore challenging to use
for natural resource management and analysis tasks.

One option for representing complex geometries is the mesh data structure that
is commonly used in computer graphics [7]. Such meshes are generally produced
using structure-from-motion photogrammetry, which uses a set of overlapping
images to predict (a) camera lens parameters, (b) the pose (position and
orientation) of the camera at each image capture, and (c) a 3D representation
of the scene constructed via triangulation of keypoints common across images.
Meshes consist of vertices (points in 3D space), connected by triangular
faces. The geometric information contained in the mesh data structure can be
augmented by adding additional “semantic” information to the faces or vertex
such as a species ID for that location. Work such as [34] generates semantic
meshes using per-image predictions by first classifying each pixel of the
input images and then associating these predictions to locations on the mesh
surface. This approach is validated on indoor and street-scale scenes, but to
our knowledge, no prior work uses semantic meshes and un-rectified multiview
image-based predictions for large-scale aerial mapping tasks.

###  2.2 Predicting tree species using remote sensing imagery

Classification of tree species using remote sensing data has been a highly
active field of research for at least two decades [12]. An initial focus on
multispectral satellite imagery has expanded in recent years to include
species prediction from aerial imagery, generally driven by computer vision
algorithms employing artificial neural networks. For example, researchers have
used fine-tuned CNNs, applied to RGB (red, green, blue) drone-derived
orthoimagery, to identify the species of Amazonian palms [13] and seven
Japanese tree species [23]. Rather than using orthomosaics, Santos _et al_.
[32] applied CNNs to un-rectified drone imagery to detect specific tree
species of concern among urban trees. However, these experiments were only
conducted at the image-level and predictions were not mapped into geospatial
coordinates, which is critical for many management applications. Approaches to
species classification include (a) first cropping out individual
(unclassified) tree crowns detected by other methods and then applying a
classification algorithm [23], (b) applying pixel-level semantic segmentation
to orthoimagery and then defining objects as clusters of pixels of the same
class [13] or (c) applying architectures such as YOLO [26] that simultaneously
detect objects and classify them [32]. Most often, species classes are
obtained through manual, direct annotation of images by experts [13, 32],
which may inflate accuracy because human annotators observe the same images as
the vision algorithm and may make similar errors. Using a more robust approach
of assigning tree species classes through field-based surveys, researchers
classified 72 species in a continental-scale survey using pixel-based
classification of crewed-aircraft hyperspectral imagery (426 reflectance
bands) [21]. Later work built on this approach to more accurately classify
species by using multi-temporal data and hierarchical computer vision-based
classification [37]. Although some tree species classification work has
leveraged imagery of the same trees from multiple time points and multiple
sensors [37, 2], no work to our knowledge has used multiple views of the same
trees from different angles.

##  3 Dataset

Name | Coordinates |  Fire year |  Imagery year |  Field data year |  Reference tree count |  Dominant species
---|---|---|---|---|---|---
Delta | 41.04, -122.53 | 2018 | 2020 | 2020, 2021 | 288 | PIPJ, PSME, PILA
Chips | 40.13, -121.11 | 2012 | 2020, 2021 | 2020 | 243 | PSME, PIPJ, ABCO
Valley | 38.82, -122.69 | 2015 | 2020 | 2020 | 64 | PIPJ, PSME, PILA
Lassic | 40.30, -123.57 | 2015 | 2021 | 2020, 2021 | 178 | CADE, ABCO, PIPJ
Table 1: Study site attributes. Coordinates are provided as latitude,
longitude. Dominant species are listed in order of decreasing abundance.

###  3.1 Dataset domain

The data used in this work was collected to study forest dynamics following
wildfires. It was collected at four locations in California, USA that
experienced major wildfires between two to nine years prior to data collection
(Tab. 1). All sites are dominated by the “California mixed-conifer” forest
type [4, 31] containing primarily sugar pine (_Pinus lambertiana_ , labeled
“PILA”); white fir (_Abies concolor_ , “ABCO”); Douglas-fir (_Pseudotsuga
menziesii_ , “PSME”); incense cedar (_Calocedrus decurrens_ , “CADE”); and
ponderosa pine (_Pinus ponderosa_) or Jeffrey pine (_Pinus jeffreyi_)
(collectively labeled “PIPJ” due to the inability to definitively distinguish
them in the field). The data consists of both drone imagery and manual field
reference observations.

###  3.2 Field reference data

The field reference data consists of manual measurements of the geospatial
(lat/lon) locations of trees as well as attributes including species and
height. In the field, tree locations were in most cases mapped based on their
distance (measured by laser rangefinder) and azimuth (measured by compass)
relative to survey centerpoints, which were located using a high-precision RTK
GPS system. In some areas with sufficient sky view, individual tree locations
were instead directly measured with the RTK GPS system. To co-register the
field-mapped trees to the drone-derived products (mesh model, orthomosaic,
canopy height model, and camera poses), we applied a manually identified x,y
shift identically to all trees measured from a given centerpoint (or located
directly using RTK GPS) such that prominent field-surveyed trees aligned
optimally with prominent drone-detected trees.

###  3.3 Drone imagery

The drone data was collected using a combination of DJI Phantom 4 Advanced and
DJI Phantom 4 Pro v2 drones, which both use an integrated 20 megapixel RGB
camera (model FC6310/FC6310s) with the same specifications. A flight planner
was used to collect images using two different missions per site: one with
parameters optimized for structure reconstruction and structure-based tree
detection, and one with parameters that were hypothesized to enhance species
classification performance. The structure reconstruction mission was executed
at 120 m altitude above ground level with a nadir-facing camera and 90% front
and side overlap. The species classification mission was executed as a “grid”
mission consisting of two perpendicular sub-missions at an altitude of 80 m
above ground level (except 90 m for the Valley site) with 80-90% front and
70-80% side overlap with a camera pitch of 20 degrees off nadir (except 25
degrees for the Lassic site).

###  3.4 Photogrammetry products

The imagery from each site was processed using Agisoft Metashape version 2.0
or 2.1 [1] using the workflow and parameterization identified by Young _et
al_. [39] as optimal for mapping conifer forests. The published workflow was
modified to use an image upscaling factor of 4 (instead of 2) during the depth
mapping/dense cloud generation step to account for internal Metashape changes
since version 1.6 that was used for the published workflow development. In the
meshing step, which is not included in the published workflow, we used the
Metashape defaults. Each site processed successfully and all or nearly all of
the cameras aligned properly. Manually selected ground control points were
used to accurately align the photogrammetry project with geospatial
coordinates.

The first output we use is the location of the cameras, which captures the
pose of each image as well as the calibrated intriniscs of the camera that are
shared across all images. Second, we use the mesh model of the scene in
geospatial coordinates. Finally, we use two geospatial raster products: the
digital surface model (DSM) represents the elevation of the top surface of the
mesh while the digital terrain model (DTM) represents the elevation of the
ground surface predicted by Metashape.

###  3.5 Tree detection

Following previous work [39], treetops were detected as local maxima within
the canopy height model (CHM) raster. The CHM was produced by subtracting the
DTM from the DSM, resampling the result to 0.25 m resolution via bilinear
interpolation, and applying a 7x7 pixel sliding window mean smooth. Trees were
detected using a variable-radius sliding window filter on the smoothed CHM,
parameterized to use a circular local maximum search window with a horizontal
radius equal to 0.11 times the height of the focal pixel. Local maxima with
height $>$ 5 m were retained. The local maximum search was performed using the
locate_trees and lmf functions in the R package lidR [27].

The horizontal extent of the crown surrounding each detected treetop was
delineated using the method of Silva _et al_. [8], parameterized to include
all contiguous pixels with a height greater than 0.1 times the treetop height,
to a maximum horizontal distance of 0.24 times the treetop height. The method
delineates intersecting crowns using a Voronoi tessellation. Crown delineation
was performed using the segment_trees and its_silva2016 functions in the R
package lidR. The resulting crown polygons were then simplified using a kernel
smoothing filter. Field labels were assigned to detected trees using the
algorithm of Young _et al_. [39] developed for this task (Sup. Sec. 7.2).

##  4 Multiview Semantic Meshes

Figure 2: Schematic of the analytical workflow, with input data (c, d, j, l)
outlined in green and output data (i) outlined in blue. At the sites used to
train the computer vision model, raw drone images (d) are collected and
processed using photogrammetry to estimate camera poses and a 3D geospatial
mesh model (a), which is then textured using geospatial ground-truth species
labels (c) obtained via field surveys. The semantic mesh tool presented here
is then used to render species labels (e) that match pixel-for-pixel with the
raw drone images. The raw images and labels are then used to train a computer
vision semantic segmentation model (f). To classify trees to species at a new
site, another set of drone images (j) is collected and segmented pixel-for-
pixel into species classes (g) using the trained model. In parallel, the
images are processed using photogrammetry to yield camera poses and a 3D mesh
model (k). The semantic mesh tool presented here is then used to project the
image-based species classes onto the mesh faces (h). The geospatial locations
of trees (l) are determined, and the mesh-based species classes are
transferred to the geospatial tree map to yield the final output, a tree map
with inferred species labels (l).

The goal of multiview mapping is to generate predictions for some task on raw
images and then convert these predictions into geospatial data that is useful
for land managers. In practice, these predictions are generated using deep
learning models applied to the images. It is uncommon for pre-trained models
to exist for a given ecology task, so the user generally has to train their
own domain-specific model. Hand labeling data for model training is laborious,
and for many ecology problems it cannot be accurately conducted without
relying on in-situ observations. Given these challenges, a significant portion
of the multiview mapping workflow is devoted to training models using relevant
geospatial ground truth data.

###  4.1 Definitions

Class IDs: We assume that in our classification problems there are a fixed
number of classes $N_{clc}$ that are known beforehand. Therefore, our set of
potential $IDs$ is represented by a set of integers, one for each class, plus
the special $null$ value used to represent background or unlabeled content.
This is represented as $IDs=\\{1...N_{cls}\\}\cup\\{null\\}$.
Geospatial labels: The ground truth data used to train machine learning models
is provided in geospatial format. Specifically, this data consists of a set of
closed 2D polygons, each with an associated ID in $IDs$. These polygons are
assumed to be non-overlapping. The set of geospatial label polygons is
represented as $L_{g}$, and $L_{g}[x,y]$ can be used to query the class ID for
a given 2D geospatial location. If the point is not contained within any of
the polygons, the $null$ class ID is returned.
Class ID images: This approach currently assumes that machine learning models
are trained on and predict per-pixel integer labels for each image they
receive, represented as a $I_{h}\times I_{w}$ array of integer class IDs with
the same number of rows and columns as the original image. If this label image
array is defined as $L_{i}$, the image can be indexed with an integer pixel
location to obtain the class for that pixel as $L_{i}[i,j]$. Any indices
outside the bound of the image return the $null$ label.
Meshes: The 3D geometry of the scene is represented by a mesh consisting of a
set of points called vertices and a set of triangular faces. The vertices $V$
are represented as an $N_{v}\times 3$ array, where each row represents a point
in 3D space. The faces $F$ are represented by an $N_{f}\times 3$ array, where
each row consists of three non-repeating integer indices into the vertex array
describing the corners of a triangular face.
Cameras: Virtual cameras are used to represent the properties of how each
image was captured. These cameras are primarily parameterized by $4\times 4$
extrinsic matrix, representing the camera’s location plus orientation, and a
$3\times 3$ intrinsic matrix, representing projection of 3D points in the
camera’s field of view into the 2D location of pixels. Both of these matrices
are estimated by photogrammetry software. A thorough review of geometric
computer vision concepts is provided by Hartley and Zisserman [14]. We
represent the set of cameras for a scene as $C$ with $N_{cam}$ elements and
define the individual cameras as $c\in C$.
Digital terrain models: The digital terrain model represents the estimated
elevation of the ground surface and is represented as $G$. The estimated
elevation can be queried for a given 2D location to obtain the elevation of
the ground at that point as $G[x,y]$. The ground surface estimate can be
produced by many photogrammetry software and in this work is assumed to be
defined for the entire study region.

###  4.2 Computing image-to-mesh correspondences

The core operation in the semantic mesh framework is computing which face on
the mesh is observed by a given pixel in a camera’s image. This involves using
the camera’s extrinsic and intrinsic matrices and the pixel indices to define
a ray originating at the camera’s location and extending in the direction of
that pixel’s observation. Then, the index of first mesh face that the ray
intersects is reported. For a mesh $M$, camera $c$, and pixel index $(i,j)$,
Eq. 1 is used to compute a face index $F_{i}$ representing which face would be
observed from that pixel. If the ray does not intersect any face, which can
occur in the case of cropped meshes or inaccurate 3D reconstruction, the
$null$ index is returned.

$\displaystyle I_{f}=pix2face(M,c,(i,j))$ (1)

There are numerous ways to implement the $pix2face$ function using modern
computer graphics libraries. In this work, we use the explicit pixel-to-face
correspondences computed in the PyTorch3D [25] differentiable rendering
pipeline.

###  4.3 Training workflow

The training workflow generates per-image training labels using a mesh $M$,
set of cameras $C$, ground plane estimate $G$, and geospatial polygon labels
$L_{g}$.

The region observed in the aerial survey is often significantly larger than
the region labeled in the field survey. For computational reasons, we define a
2D region of interest around the labels and only consider this area during the
training label generation process. Specifically, we define the $RoI$ as a
closed 2D polygon consisting of locations within a user-defined buffer radius
$r_{b}$ of any polygon in $L$. The user should set $r_{b}$ conservatively,
such that it is unlikely any cameras outside of this radius can observe
labeled regions. The mesh $M$ is updated so that only faces contained entirely
within $RoI$ are retained. Similarly, the camera set $C$ is filtered so that
only cameras located within the $RoI$ are kept.

The next step is to add class IDs to $M$ using the information contained in
$L$. The label for each vertex in the mesh vertices $V$ is obtained using Eq.
2.

$\displaystyle L_{v}$ $\displaystyle=\\{L_{g}[v_{x},v_{y}]\mid v\in V\\}$ (2)

Since the per-vertex labeling is performed using a 2D polygon, some regions of
ground below tree canopies may be labeled as the class of the canopy above. To
filter out these spurious labels, we use a simple height-above-ground filter.
The vertex elevation is compared to the estimated ground elevation at that 2D
location to obtain a height above ground $H_{g}$ for each vertex (Eq. 3). The
vertices that have a height below the user-defined threshold height $H_{t}$
are computed and the corresponding vertex labels are set to the $null$ label
(Eq. 4).

$\displaystyle H_{g}$ $\displaystyle=\\{v_{z}-G[v_{x},v_{y}]\mid v\in V\\}$
(3) $\displaystyle L_{v}[H_{g}<H_{t}]$ $\displaystyle=null$ (4)

Then each face is labeled based on the most common label for the associated
vertices (Eq. 5). The $mode$ function is defined to randomly tie-break in
situations where all the vertex labels differ. A face label is set to $null$
only if all vertices are labeled $null$.

$\displaystyle L_{f}$
$\displaystyle=\\{mode(L_{v}[f_{1}],L_{v}[f_{2}],L_{v}[f_{3}])\mid f\in F\\}$
(5)

Using the labeled mesh we generate per-image labels for each virtual camera.
For a given camera $c\in C$, we compute a labeled image $L_{i}$. For each
pixel in the image, we compute the face ID $I_{f}$ using $pix2face$ (Eq. 1).
Then we obtain the label by taking the corresponding face label,
$L_{f}[I_{f}]$. These IDs are assembled into an image of the same dimension as
the input image, and can be used as labels for the corresponding real images
for model training (Sup. Fig. 4).

###  4.4 Prediction workflow

The prediction workflow takes as input mesh $M$, a ground plane estimate $G$,
a set of cameras $C$ and predicted segmentation image $P_{img}$ for each
camera $c\in C$. In the case of per-object classification, the boundaries of
each unclassified object are defined as a set of polygons, $O_{u}$. Similar to
the training workflow (Sec. 4.3), the mesh is cropped and the camera set is
filtered to the $RoI$ around the study site.

For each camera $c\in C$, the corresponding predicted label $L_{i}\in P_{I}$
is obtained. A $N_{f}\times N_{cls}$ matrix $P_{face}$ is initialized for each
to all zeros to record the predicted class(s) per face. For each camera, for
all pixel indices $\\{(i,j)\mid i\in 1...I_{h},j\in 1...I_{w}\\}$, the
predicted class at that pixel is obtained (Eq. 6), the corresponding face
index is identified (Eq. 7), and the $P_{face}$ is updated (Eq. 8).

$\displaystyle P_{pix}$ $\displaystyle=P_{img}[i,j]$ (6) $\displaystyle
I_{face}$ $\displaystyle=pix2face(M,c,(i,j))$ (7) $\displaystyle
P_{face}[I_{face},L_{pix}]$ $\displaystyle=1$ (8)

This procedure is preformed for each camera to obtain $P^{i}_{face}\mid i\in
1...N_{cam}$ and these matrices are summed element-wise across cameras to
obtain $P^{sum}_{face}$. The most common class prediction per face is computed
using Eq. 9.

$\displaystyle P^{ID}_{face}=\underset{col}{\text{argmax}}P^{sum}_{face}$ (9)

This produces a mesh where the most commonly-predicted class per face is
identified. For tasks where distinct (unclassified) objects are already
identified, we generate a class prediction per object. For each object,
represented by a 2D polygon, we record which mesh faces have an overlapping
top-down 2D projection. Each face is assigned a weight based on the 3D area of
that face in the original mesh. The 3D area, rather than the 2D projected
area, is chosen because the near-vertical faces capture useful information
from oblique predictions. Faces that are identified as ground are then down-
weighted by a user-provided factor. Finally, using per-face weighting and
predicted classes from $P^{ID}_{face}$, the per-class predictions are summed
and the object is assigned to the class with the highest value. This process
is repeated for each unlabeled object to obtain per-object labels.

##  5 Experiments

###  5.1 Dataset splits

The goal of our experiments is to compare an orthomosaic-based approach and a
multiview approach on a challenging realistic task. We adopted a cross-site
evaluation methodology to avoid overly-optimistic results caused by training
and testing on spatially-adjacent data. Specifically, we rendered the labels
for each of the sites. Then, we used a leave-one-out strategy and train a
model using all but one site, which we then used for testing. This procedure
was repeated for each site with both the orthomosaic (“Ortho”) and multiview
(“MV”) approaches.

(a) Orthomosaic; accuracy: 53%.

(b) Multiview-HN; accuracy: 63%.

(c) Multiview-LO; accuracy: 75%.

Figure 3: Confusion matrices for the cross-site tree species classification
task, with species counts summed across the four sites and three trials.

###  5.2 Species segmentation with deep learning

We treated species classification as semantic segmentation problem where per-
pixel labels are produced. To ensure fair comparisons, we used the same deep
learning model and training regime in both the multiview and orthomosaic
workflows. Specifically, we used the SegFormer [38] architecture (MIT-B5
backbone variant) implemented in MMSegmentation [9] with the default
Cityscapes [10] pretained weights. This architecture was used for a related
low-data forest vegetation classification task in the works of Russell _et
al_. [28] and Andrada _et al_. [3] with good results. In all experiments, the
models were trained for 10,000 iterations with a fixed learning rate schedule.
To address non-determinism in model training, three distinct models were
trained on each configuration of data and each used to generated independent
predictions, which were then each evaluated.

###  5.3 Orthomosaic baseline

Our goal was to develop a realistic baseline approach based on an orthomosaic
constructed from images taken at 120 m above ground level. Since orthomosaics
are often large—frequently 10s of thousands of pixels on a side—tiled model
training is critical. In our method we used the orthomosiac at the native
resolution and cropped square chips that are the size of the smaller dimension
of the images used for photogrammetry and multiview workflow. For training, we
used a 50% overlap between neighboring chips so content at the borders of
chips was still used effectively. The classes per pixel were extracted from
the corresponding geospatial ground truth polygons and saved as label images
with the same width and height as the training crops. For each site, in
alphabetical order, 28, 43, 24, and 11, training chips were produced.

We trained models following the procedure outlined in Section 5.2, with three
independent models trained on each configuration that excluded a single site.
The same procedure to generate orthomosaic chips that was used for training
was also used for inference. For each site, the models not trained on any
labels from that site were used to generate predictions. These per-chip
predictions were then aggregated into a raster prediction for the entire
scene. Since the chips overlap, for each pixel in the region, the number of
predictions for each class was recorded. To avoid abrupt class transitions at
the borders of chips, the pixels in the exterior 25% were down-weighted using
a linear ramp ranging from 1 (full weight) at the interior to 0 (no weight) at
the very edge. The most common weighted prediction for each aggregated pixel
was reported. Finally, each tree crown polygon was assigned the class that is
most commonly predicted within the polygon boundary.

| Chips | Delta | Lassic | Valley | Aggregated
---|---|---|---|---|---
| Or. |  MV- HN |  MV- LO | Or. |  MV- HN |  MV- LO | Or. |  MV- HN |  MV- LO | Or. |  MV- HN |  MV- LO | Or. |  MV- HN |  MV- LO
Acc. | 0.54 | 0.71 | 0.82 | 0.60 | 0.71 | 0.77 | 0.29 | 0.28 | 0.52 | 0.86 | 0.92 | 1.00 | 0.53 | 0.63 | 0.75
std | 0.01 | 0.02 | 0.01 | 0.01 | 0.01 | 0.01 | 0.00 | 0.05 | 0.01 | 0.04 | 0.02 | 0.00 | 0.00 | 0.01 | 0.01
Rec. | 0.41 | 0.53 | 0.73 | 0.44 | 0.56 | 0.65 | 0.56 | 0.57 | 0.73 | 0.92 | 0.89 | 1.00 | 0.44 | 0.54 | 0.69
std | 0.05 | 0.03 | 0.00 | 0.01 | 0.01 | 0.01 | 0.02 | 0.03 | 0.04 | 0.05 | 0.01 | 0.00 | 0.00 | 0.01 | 0.01
Prec. | 0.56 | 0.61 | 0.77 | 0.50 | 0.72 | 0.73 | 0.47 | 0.50 | 0.59 | 0.81 | 0.87 | 1.00 | 0.54 | 0.63 | 0.72
std | 0.05 | 0.02 | 0.02 | 0.10 | 0.05 | 0.01 | 0.03 | 0.04 | 0.04 | 0.02 | 0.03 | 0.00 | 0.02 | 0.04 | 0.01
Table 2: Leave-one-site-out tree species inference performance. Recall (Rec.)
and precision (Prec.) are macro-averaged across species. The aggregated
results across the sites are computed by summing the confusion matrices across
all sites and then computing the summary statistics. Or.: orthomosaic; MV-HN:
High-altitude nadir multiview; MV-LO: Low-altitude oblique multiview. Best
values are bold.

###  5.4 Multiview workflow

The multiview (“MV”) workflow was performed twice for each site: once using
the high altitude (120 m) nadir images (“MV-HN”; the same images used to
construct the orthomosaic and mesh), and once using the low altitude (80-90 m)
oblique images (“MV-LO”). For each of these two runs of the workflow, the
first step was to generate training labels (pixel-wise class annotations) for
the raw images from each of the four sites. Since the ground truth polygon
data was guaranteed to be non-overlapping, it could be directly used without
any pre-processing. The region of interest was defined as any location within
50 meters of the ground truth data, to account for unlabeled trees that could
potentially occlude the labeled trees. Following the procedure described in
Section Sec. 4.3, the mesh vertices and then faces were textured from the
ground truth polygon data. Since faces are labeled if they fall within the 2D,
top-down boundary of the polygons, some mesh faces corresponding to the ground
were labeled with the class of the tree crown above them. To avoid spurious
training labels, these faces were filtered out using a simple height-based
filter. We used a digital terrain model (DTM) computed during the
photogrammetry step to estimate the ground surface and and check the height of
each face above the DTM. Faces less than 2 meters above the ground were set to
the $null$ class. Finally, the label images were rendered from the perspective
of every camera within 50 horizontal meters of the training polygons. The
sites, in alphabetical order, had 360, 453, 529, and 125 training images each
for the HN dataset and 437, 687, 495, and 143 for the LO dataset. Models were
trained using eight separate dataset configurations (four for MV-HN and four
for MV-LO), in each case using data from all but one site within the given
dataset (MV-HN or MV-LO). For each configuration, three independent models
were trained.

To obtain tree class predictions, semantic segmentation was performed for the
images at each site using the model trained on the other three sites. Then,
these predictions were aggregated onto the mesh using the procedure described
in Section 4.4. Ground faces were identified using the same procedure used in
training and ground face predictions were down-weighted by a factor of 0.01.

###  5.5 Accuracy assessment methods

For each of the four sites, one set of predictions was generated using the
orthomosaic and two using the multiview approach (one for MV-HN and one for
MV-LO). For each configuration, we generated three independent predictions
using each of the replicated models. For each set, we computed the binary
recall and binary precision per class and macro-averaged these metrics across
classes to highlight the performance on rarer classes. We also aggregated
(summed) the confusion matrices across all sites and all replications and
using these confusion matrices computed the overall accuracy, recall (macro-
averaged by class) and precision (also macro-averaged by class). In all cases,
the site that was predicted was excluded from the training data, and the
orthomosaic, MV-HN, and MV-LO datasets were always kept separate.

###  5.6 Results

The low-oblique multiview (MV-LO) approach performed better than the
orthomosaic approach in every instance of comparison (Tab. 2). On the
aggregate metrics, the MV-LO approach conferred a major improvement in
accuracy (75% vs. 53%), recall (69% vs. 44%), and precision (72% vs. 54%)
relative to the orthomosaic approach. The high-nadir multiview (MV-HN)
approach performed worse than the MV-LO approach (aggregate accuracy: 63% vs.
75%; Tab. 2), as expected given that higher-altitude images capture less fine
detail and and that nadir images do not capture clear side views that can aid
species differentiation. Nonetheless, even the MV-HN approach generally
performed substantially better than the orthomosaic approach (aggregate
accuracy: 63% vs. 53%). The MV-HN approach relied on the same images that were
used to create the orthomosaic, so its improved performance demonstrates that
classification tasks can benefit from (a) using multiple views, such as the
slightly oblique perspectives that are present at the edges even of nadir-
facing images, (b) using raw images that contain less distortion than the
orthomosaic, or (c) both.

The orthomosaic approach dramatically over-predicted the two most common
classes, PIPJ and PSME, while the distribution predicted by the multiview
approaches matched the true distribution much more closely (Fig. 3). This
pattern generally held across individual sites (Fig. 5). Species-level recall
and precision varied substantially depending on the species—likely due to
variation in training sample size and distinctiveness of appearance—but were
consistently higher for the multiview approaches (Tab. 3). With the MV-LO
approach, recall for the most common class (PIPJ) was very high (95%), and
precision was also reasonably high (81%) and higher than for the less common
classes. This was less true for the orthomosaic approach, suggesting that the
multiview approach helps to constrain the overprediction of the most common
classes.

##  6 Conclusions

This study demonstrates the value of multiview reasoning and deep learning
applied to raw drone images for geospatial prediction tasks. The improvement
in tree species classification performance that multiview reasoning yielded in
our experiments may result from a combination of several factors, including
elimination of artifacts from the orthomosaicing process, increased number and
diversity of training examples, improved robustness due to prediction
ensembling, and added information from the side views of objects. Further
research may help to understand the contributions of each of these mechanisms.

Although forest stand structure may be best reconstructed from high-altitude
(e.g., 120 m above ground level) drone imagery [39], our work suggests that to
obtain optimal species classification results, an additional imaging mission
to collect lower-altitude, oblique imagery should be performed. However, when
performing a second mission is infeasible, the multiview approach applied to
the same high-altitude nadir images used for structure estimation can still
enable increased accuracy relative to an orthomosaic-based approach.

The workflow we present is applicable to land cover mapping tasks and to
classification of existing polygons, but it cannot be naively applied to
object detection tasks. Future work will extend this framework to multiview
object detection. The large, novel forest dataset underlying our experiments
is available to support such future development. We aim for it, along with our
open-source toolbox, to enable researchers and practitioners in forestry and
other geospatial domains to leverage and continue to develop multiview
reasoning tools.

## References

  * Agisoft [2021] L L C Agisoft.  Metashape python reference.  _Release_ , 1:1–199, 2021.
  * Ahlswede et al. [2023] S. Ahlswede, C. Schulz, C. Gava, P. Helber, B. Bischke, M. Förster, F. Arias, J. Hees, B. Demir, and B. Kleinschmit.  TreeSatAI Benchmark Archive: a multi-sensor, multi-label dataset for tree species classification in remote sensing.  _Earth System Science Data_ , 15(2):681–695, 2023.
  * Andrada et al. [2023] Maria Eduarda Andrada, David Russell, Tito Arevalo-Ramirez, Winnie Kuang, George Kantor, and Francisco Yandun.  Mapping of potential fuel regions using uncrewed aerial vehicles for wildfire prevention.  _Forests_ , 14, 2023.
  * Barbour et al. [2007] Michael Barbour, Todd Keeler-Wolf, and Allan A Schoenherr.  _Terrestrial vegetation of California_.  Univ of California Press, 2007.
  * Brandt et al. [2020] M. Brandt, C.J. Tucker, A. Kariryaa, K. Rasmussen, C. Abel, J.L. Small, J. Chave, L.V. Rasmussen, P. Hiernaux, A.A. Diouf, L. Kergoat, O. Mertz, C. Igel, F. Gieseke, J. Schöning, S. Li, K.A. Melocik, J.R. Meyer, S. Sinno, E. Romero, E.N. Glennie, A. Montagu, M. Dendoncker, and R. Fensholt.  An unexpectedly large count of trees in the west african sahara and sahel, 2020.
  * Bröker et al. [2019] Koen C. A. Bröker, Rikke G. Hansen, Kathleen E. Leonard, William R. Koski, and Mads Peter Heide-Jørgensen.  A comparison of image and observer based aerial surveys of narwhal.  _Marine Mammal Science_ , 35(4):1253–1279, 2019.
  * Buss [2009] Samuel R. Buss.  3 ‐ d computer graphics: A mathematical introduction with opengl.  _School Science and Mathematics_ , 109:183–184, 2009.
  * Carlos A. Silva Andrew T. Hudak and Khosravipour [2016] Lee A Vierling E Louise Loudermilk Joseph J O’Brien J Kevin Hiers Steve B Jack Carlos Gonzalez-Benecke Heezin Lee Michael J Falkowski Carlos A. Silva Andrew T. Hudak and Anahita Khosravipour.  Imputation of Individual Longleaf Pine (Pinus palustris Mill.) Tree Attributes from Field and LiDAR Data.  _Canadian Journal of Remote Sensing_ , 42(5):554–573, 2016.
  * Contributors [2020] MMSegmentation Contributors.  MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation, 2020.
  * Cordts et al. [2016] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  _CoRR_ , abs/1604.01685, 2016.
  * Durrant-Whyte and Bailey [2006] Hugh Durrant-Whyte and Tim Bailey.  Simultaneous localisation and mapping (slam): Part i the essential algorithms, 2006.
  * Fassnacht et al. [2016] Fabian Ewald Fassnacht, Hooman Latifi, Krzysztof Stereńczak, Aneta Modzelewska, Michael Lefsky, Lars T. Waser, Christoph Straub, and Aniruddha Ghosh.  Review of studies on tree species classification from remotely sensed data.  _Remote Sensing of Environment_ , 186:64–87, 2016.
  * Ferreira et al. [2020] Matheus Pinheiro Ferreira, Danilo Roberti Alves de Almeida, Daniel de Almeida Papa, Juliano Baldez Silva Minervino, Hudson Franklin Pessoa Veras, Arthur Formighieri, Caio Alexandre Nascimento Santos, Marcio Aurélio Dantas Ferreira, Evandro Orfanó Figueiredo, and Evandro José Linhares Ferreira.  Individual tree detection and species classification of amazonian palms using uav images and deep learning.  _Forest Ecology and Management_ , 475:118397, 2020.
  * Hartley and Zisserman [2004] R. I. Hartley and A. Zisserman.  _Multiple View Geometry in Computer Vision_.  Cambridge University Press, ISBN: 0521540518, second edition, 2004.
  * Hodgson et al. [2016] Jarrod C. Hodgson, Shane M. Baylis, Rowan Mott, Ashley Herrod, and Rohan H. Clarke.  Precision wildlife monitoring using unmanned aerial vehicles.  _Scientific Reports_ , 6, 2016.
  * Koukal and Atzberger [2012] Tatjana Koukal and Clement Atzberger.  Potential of multi-angular data derived from a digital aerial frame camera for forest classification.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 5(1):30–43, 2012.
  * Leotta et al. [2019] Matthew J. Leotta, Chengjiang Long, Bastien Jacquet, Matthieu Zins, Dan Lipsa, Jie Shan, Bo Xu, Zhixin Li, Xu Zhang, Shih Fu Chang, Matthew Purri, Jia Xue, and Kristin Dana.  Urban semantic 3D reconstruction from multiview satellite imagery.  _IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops_ , 2019-June:1451–1460, 2019.
  * Liu and Abd-Elrahman [2018a] Tao Liu and Amr Abd-Elrahman.  Multi-view object-based classification of wetland land covers using unmanned aircraft system images.  _Remote Sensing of Environment_ , 216:122–138, 2018a.
  * Liu and Abd-Elrahman [2018b] Tao Liu and Amr Abd-Elrahman.  An Object-Based Image Analysis Method for Enhancing Classification of Land Covers Using Fully Convolutional Networks and Multi-View Images of Small Unmanned Aerial System.  _Remote Sensing_ , 10(3), 2018b.
  * Marconi et al. [2021] Sergio Marconi, Sarah J. Graves, Ben G. Weinstein, Stephanie Bohlman, and Ethan P. White.  Estimating individual-level plant traits at scale.  _Ecological Applications_ , 31(4):e02300, 2021.
  * Marconi et al. [2022] Sergio Marconi, Ben G. Weinstein, Sheng Zou, Stephanie A. Bohlman, Alina Zare, Aditya Singh, Dylan Stewart, Ira Harmon, Ashley Steinkraus, and Ethan P. White.  Continental-scale hyperspectral tree species classification in the united states national ecological observatory network.  _Remote Sensing of Environment_ , 282:113264, 2022.
  * Maschler et al. [2018] Julia Maschler, Clement Atzberger, and Markus Immitzer.  Individual tree crown segmentation and classification of 13 tree species using airborne hyperspectral data.  _Remote Sensing_ , 10(8), 2018.
  * Onishi and Ise [2021] Masanori Onishi and Takeshi Ise.  Explainable identification and mapping of trees using uav rgb image and deep learning.  _Scientific Reports_ , 11, 2021.
  * Purri et al. [2019] Matthew Purri, Jia Xue, Kristin Dana, Matthew Leotta, Dan Lipsa, Zhixin Li, Bo Xu, and Jie Shan.  Material Segmentation of Multi-View Satellite Imagery.  2019\.
  * Ravi et al. [2020] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  _arXiv:2007.08501_ , 2020.
  * Redmon et al. [2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection, 2016.
  * Roussel et al. [2020] Jean-Romain Roussel, David Auty, Nicholas C. Coops, Piotr Tompalski, Tristan R.H. Goodbody, Andrew Sánchez Meador, Jean-François Bourdon, Florian de Boissieu, and Alexis Achim.  lidr: An r package for analysis of airborne laser scanning (als) data.  _Remote Sensing of Environment_ , 251:112061, 2020.
  * [28] David Russell, Tito Arevalo, Chinmay Garg, Winnie Kuang, Francisco Yandun, David Wettergreen, and George Kantor.  Unmanned aerial vehicle mapping with semantic and traversability metrics for forest fire mitigation.
  * Russell et al. [2024a] David Russell, Rahul Prabhu, Avnoor Sidhu, Derek Young, and Thitupathi Addisherla.  open-forest-observatory/geograypher: Initial release, 2024a.
  * Russell et al. [2024b] David Russell, Ben Weinstein, David Wettergreen, and Derek J N Young.  Classifying geospatial objects from multiview aerial imagery using semantic meshes, 2024b.
  * Safford et al. [2017] Hugh D Safford, Jens T Stevens, et al.  _Natural range of variation for yellow pine and mixed-conifer forests in the Sierra Nevada, southern Cascades, and Modoc and Inyo National Forests, California, USA_.  United States Department of Agriculture, Forest Service, Pacific Southwest …, 2017.
  * Santos et al. [2019] Anderson Aparecido dos Santos, José Marcato Junior, Márcio Santos Araújo, David Robledo Di Martini, Everton Castelão Tetila, Henrique Lopes Siqueira, Camila Aoki, Anette Eltner, Edson Takashi Matsubara, Hemerson Pistori, Raul Queiroz Feitosa, Veraldo Liesenberg, and Wesley Nunes Gonçalves.  Assessment of cnn-based methods for individual tree detection on images captured by rgb cameras attached to uavs.  _Sensors_ , 19(16), 2019.
  * Swayze et al. [2021] Neal C. Swayze, Wade T. Tinkham, Jody C. Vogeler, and Andrew T. Hudak.  Influence of flight parameters on uas-based monitoring of tree height, diameter, and density.  _Remote Sensing of Environment_ , 263:112540, 2021.
  * Valentin et al. [2013] Julien P.C. Valentin, Sunando Sengupta, Jonathan Warrell, Ali Shahrokni, and Philip H.S. Torr.  Mesh based semantic modelling for indoor and outdoor scenes.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pages 2067–2074, 2013.
  * Wang et al. [2021] Haozhou Wang, Yulin Duan, Yun Shi, Yoichiro Kato, Seishi Ninomiya, and Wei Guo.  EasyIDP: A Python Package for Intermediate Data Processing in UAV-Based Plant Phenotyping.  _Remote Sensing_ , 13(13), 2021.
  * Weinstein et al. [2020] Ben G. Weinstein, Sergio Marconi, Mélaine Aubry-Kientz, Gregoire Vincent, Henry Senyondo, and Ethan P. White.  Deepforest: A python package for rgb deep learning tree crown delineation.  _Methods in Ecology and Evolution_ , 11:1743–1751, 2020.
  * Weinstein et al. [2023] Ben G. Weinstein, Sergio Marconi, Sarah J. Graves, Alina Zare, Aditya Singh, Stephanie A. Bohlman, Lukas Magee, Daniel J. Johnson, Phillip A. Townsend, and Ethan P. White.  Capturing long-tailed individual tree diversity using an airborne imaging and a multi-temporal hierarchical model.  _Remote Sensing in Ecology and Conservation_ , 9(5):656–670, 2023.
  * Xie et al. [2021] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  pages 1–18, 2021.
  * Young et al. [2022] Derek J.N. Young, Michael J. Koontz, and Jonah Maria Weeks.  Optimizing aerial imagery collection and processing parameters for drone-based individual tree mapping in structurally complex conifer forests.  _Methods in Ecology and Evolution_ , 13:1447–1463, 2022.

Supplementary Material

##  7 Dataset

###  7.1 Photogrammetry

  * •

Metashape version: 2.0, except 2.1 for the Lassic site and for aligning
oblique images to the nadir project. Platform: Ubuntu 22.04, AMD EPYC-Milan
CPU and NVIDIA GRID A100X-40C GPU via IU Jetstream2 cloud compute instances

  * •

GCP details: To ensure proper geospatial registration of the photogrammetry
products, we used a post-survey ground control point (GCP) workflow. We
located distinctive point features (e.g., the corner of a boulder or
intersection of two logs) in Google Earth imagery (at least 5 per site) and
recorded their geospatial coordinates, as well as the elevation from a USGS
3DEP 10 m DEM at each location. We then located those same landscape features
in at least 3 drone images each and recorded their pixel coordinates. This
process was performed for the 120 m image set for each site. Tabular GCP data
are provided within each site’s 120 m imagery mission folder, in the folder
named ‘gcps’. These data are provided in the format expected by the automate-
metashape444https://github.com/open-forest-observatory/automate-metashape
library v0.2.0.

###  7.2 Matching detected and field reference trees

Tree species labels recorded during the field surveys were assigned to the
tree crowns detected from the drone-derived data using the algorithm of Young
_et al_. [39] developed for this task. The pairwise horizontal distances
between all field-mapped tree stems and drone-mapped treetops were computed,
and candidate field-drone pairings were retained if (a) the height of the
drone tree was within $\pm 50\%$ of the height of the field tree, and (b) the
horizontal distance was less than $0.1h+1$, where $h$ is the height of the
field tree and units are in meters. Among the remaining candidate pairings,
final pairs were determined in order of increasing distance, with the
constraint that once a tree is assigned to a final pairing, it cannot be
included in any other pairings.

##  8 Supplemental figures and tables

Included below.

Figure 4: Labels rendered during the training process. The left pane shows the
label IDs colored by class, the middle shows the raw images, and the right
shows the classes overlayed on a grayscale image.

(a) Chips ortho. Acc: 54%

(b) Delta ortho. Acc: 60%

(c) Lassic ortho. Acc: 29%

(d) Valley ortho. Acc: 85%

(e) Chips MV-HN. Acc: 70%

(f) Delta MV-HN. Acc: 70%

(g) Lassic MV-HN. Acc: 28%

(h) Valley MV-HN. Acc: 92%

(i) Chips MV-LO. Acc: 82%

(j) Delta MV-LO. Acc: 77%

(k) Lassic MV-LO. Acc: 52%

(l) Valley MV-LO. Acc: 100%

Figure 5: Site-level confusion matrices for the leave-one-site-out tree species classification task, summed over all trials, for the orthomosaic (ortho.) dataset (a-d), high-nadir multiview (MV-HN) dataset (e-h), and low-oblique multiview (MV-LO) dataset (i-l). | ABCO (128) | CADE (104) | PILA (78) | PIPJ (250) | PSME (176)
---|---|---|---|---|---
| Or. |  MV- HN |  MV- LO | Or. |  MV- HN |  MV- LO | Or. |  MV- HN |  MV- LO | Or. |  MV- HN |  MV- LO | Or. |  MV- HN |  MV- LO
Rec. | 0.26 | 0.42 | 0.75 | 0.06 | 0.03 | 0.09 | 0.42 | 0.60 | 0.80 | 0.86 | 0.92 | 0.95 | 0.58 | 0.72 | 0.84
std | 0.04 | 0.09 | 0.02 | 0.01 | 0.00 | 0.02 | 0.02 | 0.04 | 0.02 | 0.00 | 0.01 | 0.01 | 0.04 | 0.02 | 0.01
Prec. | 0.57 | 0.61 | 0.70 | 0.41 | 0.51 | 0.57 | 0.63 | 0.80 | 0.85 | 0.51 | 0.66 | 0.81 | 0.56 | 0.55 | 0.67
std | 0.02 | 0.06 | 0.02 | 0.13 | 0.21 | 0.03 | 0.05 | 0.05 | 0.05 | 0.01 | 0.04 | 0.01 | 0.01 | 0.01 | 0.02
Table 3: Class-level classification performance in the leave-one-site-out
task, with values macro-averaged across sites and standard deviations taken
across runs. The number of trees per class is shown in parentheses. Species
classes are defined in Sec. 3.


# Dissecting vocabulary biases datasets through statistical testing and
automated data augmentation for artifact mitigation in Natural Language
Inference

Dat Thanh Nguyen
Department of Computer Science, University of Texas at Austin
<EMAIL_ADDRESS>
###### Abstract

In recent years, the availability of large-scale annotated datasets, such as
the Stanford Natural Language Inference and the Multi-Genre Natural Language
Inference, coupled with the advent of pre-trained language models, has
significantly contributed to the development of the natural language inference
domain. However, these crowdsourced annotated datasets often contain biases or
dataset artifacts, leading to overestimated model performance and poor
generalization. In this work, we focus on investigating dataset artifacts and
developing strategies to address these issues. Through the utilization of a
novel statistical testing procedure, we discover a significant association
between vocabulary distribution and text entailment classes, emphasizing
vocabulary as a notable source of biases. To mitigate these issues, we propose
several automatic data augmentation strategies spanning character to word
levels. By fine-tuning the ELECTRA pre-trained language model, we compare the
performance of boosted models with augmented data against their baseline
counterparts. The experiments demonstrate that the proposed approaches
effectively enhance model accuracy and reduce biases by up to 0.66% and 1.14%,
respectively.

##  1 Introduction

In the last few years, natural language inference (NLI), a sub-domain of
natural language processing (NLP), has been an active research field. In NLI,
systems are asked to determine whether a hypothesis sentence (h) can be
logically inferred (entailed) from a given premise sentence (p) MacCartney and
Manning (2008). For example, a hypothesis, "A family of three is at the beach"
would likely to be inferred from the premise of "A man, woman, and child
enjoying themselves on a beach". This task has been proved challenging due to
the complex nature of natural language, which involves logical reasoning,
lexical semantic knowledge, and variations in linguistic expression. Recent
years have witnessed significant research efforts in the field of NLI, notably
marked by the creation of benchmark datasets such as the Stanford Natural
Language Inference (SNLI) corpus Bowman et al. (2015) and the Multi-Genre
Natural Language Inference (MNLI) corpus Williams et al. (2017). These
datasets provide substantial amounts of annotated data, fostering the
development and evaluation of NLP models. Additionally, the advent of pre-
trained language models has played a key role in enhancing the efficacy of the
NLI problem.

Regardless of the usefulness of these large datasets in the training and
evaluation of NLI systems, recent works showed that they contain biases or
dataset artifacts, i.e., there are spurious patterns present in the datasets
that NLI models can exploit to achieve high performance without learning
relationships of the premise and hypothesis sentences Poliak et al. (2018);
Gururangan et al. (2018); Tsuchiya (2018). The cause of this phenomenon
possibly arises from the fact that crowdsourcing annotated datasets often
employ a few strategies to obtain different entailment conditions. For
example, negation words such as "not, non, nobody" are likely associated with
contradiction Gururangan et al. (2018). Furthermore, modifying or omitting
gender information, and neutralizing gender information by simply counting the
number of persons in the premise for neutral pairs are also common practices
during the generation of entailment hypotheses. For instance, Table 1 includes
three different examples from the SNLI dataset that simply modify the subject
word "people" to a specific gender word such as "man", or "woman" to obtain
the neutral hypothesis examples. As a result, NLI models trained on datasets
with such artifacts may tend to overestimate their performance and generalize
poorly in other settings.

In this work, we focus on investigating dataset artifacts in NLI domain and
developing strategies to mitigate these issues. Specifically, we identify
dataset artifacts in the SNLI dataset by a novel statistical testing
procedure. We show that vocabulary distribution in the hypothesis is the
source of biases. We then propose several automatic data augmentation
strategies for mitigating artifacts in this dataset. We finally employ the
hypothesis-only training approach Gururangan et al. (2018) to fine-tune the
pre-trained language model of Efficiently Learning an Encoder that Classifies
Token Replacements Accurately (ELECTRA) Clark et al. (2020) to evaluate the
efficiency of the proposed data augmentations.

Premise |  Hypothesis |  Label
---|---|---
People on a snowy hill. |  The people are women. |  Neutral
People attempting to move X-ray machine down the street. |  Men working in healthcare support service. |  Neutral
People wait for their luggage at an airport. |  Men are waiting for their luggage. |  Neutral
Table 1:  A typical strategy to neutralize the hypothesis in SNLI dataset. The
subject word "people" is replaced by a specific gender word such as "man", or
"woman" to obtain neutral examples.

##  2 Related works

###  2.1 Dataset artifact detection

Several strategies have been developed to uncover biases in NLI datasets. In
2018, a pioneering study by Poliak et al. introduced a hypothesis-only
baseline, wherein premises are excluded from the dataset for both training and
evaluation. Given the distinctive manner in which NLI baseline datasets are
generated through crowdsourcing, a related study suggests that models can
achieve satisfactory performance even when evaluated on hypothesis-only
baseline datasets Gururangan et al. (2018).

Another approach involves the creation of contrast examples, achieved by
perturbing input in various ways and assessing the model’s classification of
these examples. This methodology highlights regions where the model might make
incorrect decisions, providing an avenue for addressing potential dataset
artifacts contributing to inaccuracies Gardner et al. (2020).

An alternative technique is the checklist test, comprising a series of
assessments covering a diverse range of common behaviors that NLP models
should handle. The model is then evaluated on these trials to identify
potential artifacts Ribeiro et al. (2020).

Further investigations are dedicated to adversarial challenge sets Nie et al.
(2019); McCoy et al. (2019); Wallace et al. (2019). For instance, McCoy et al.
introduced adversarial challenges to investigate syntactic heuristics, while
Wallace et al. focused on trigger patterns causing incorrect predictions in
NLI models. In addition, statistical tests are also employed to detect
potential artifacts, such as the "competency problems" framework Gardner et
al. (2021)

###  2.2 Dataset artifact mitigation

Various strategies exist for addressing artifacts in datasets used for NLP
models. In cases where the available training data is insufficient, one
approach involves supplementing it with "adversarial datasets." These datasets
are generated by perturbing and transforming the original corpus on various
levels, providing additional sets for NLP model learning Morris et al. (2020).
Another approach that aids the model in better handling the unique challenges
present in the dataset is inoculating a model by fine-tuning it on a small,
problem-specific dataset Liu et al. (2019). Additionally, focusing on
contradiction-word bias and word-overlapping bias is an effective strategy to
address dataset artifacts, as shown by Zhou and Bansal. Specifically, the
proposed method explores both data-level and model-level debiasing methods to
enhance the robustness of models against lexical dataset biases Zhou and
Bansal (2020).

##  3 Materials and methods

###  3.1 Dataset

In this work, we employ the widely used SNLI dataset to examine and mitigate
dataset artifacts. The dataset, introduced by the Stanford NLP group, is
commonly used for evaluating the performance of machine learning models in
tasks related to textual entailment. Created using crowdsourcing, annotators
provided labels for sentence pairs. Specifically, the premise sentences are
captions from the Flickr30k corpus, and the hypotheses were generated through
crowd workers following specific rules. The original SNLI dataset contains
over 570,000 labeled sentence pairs distributed into three main subsets:
training, development (validation), and test sets, with around 550,000,
10,000, and 10,000 examples, respectively. Each sentence pair in the dataset
is labeled with one of three categories: "entailment" (the hypothesis
logically follows from the premise), "neutral" (no clear logical relationship
exists between the premise and hypothesis), or "contradiction" (the hypothesis
contradicts the premise) Bowman et al. (2015).

###  3.2 Pre-trained language model and learning optimization

Over the past few years, ELECTRA, or Efficiently Learning an Encoder that
Classifies Token Replacements Accurately, has become increasingly popular in
the field of NLP. ELECTRA was introduced by Clark et al. in 2020 as a
transformer-based model specifically tailored for efficient pre-training and
effective downstream task performance through fine-tuning. Pre-training
ELECTRA also involves replacing a portion of the input text with incorrect or
"masked" tokens, similar to other language models like BERT. However, the key
difference that makes ELECTRA more efficient is that the problem has been
reframed as a binary classification problem, where the model is tasked with
identifying whether each token in a sequence is "real" or "replaced" Clark et
al. (2020).

Numerous studies have shown that it significantly enhances performance on NLP
tasks, including NLI and question answering. ELECTRA-small is a compact
version of the original ELECTRA model that requires fewer computing resources
for both training and optimization. Despite its reduced size, the ELECTRA-
small model can deliver competitive performance to the standard ELECTRA model
in specific tasks. Consequently, it is the preferred choice for use cases with
constrained computer resources. In this work, we employ a learning
optimization approach using the pre-trained ELECTRA-small from the Huggingface
Transformers repository Wolf et al. (2020). Specifically, we fine-turn the
model with the provided API of `AutoModelForSequenceClassification` for a
three-class classification problem. In this study, all experimental models are
fine-tuned with approximately 12,800 steps, each step containing a batch size
of 256. Model checkpoints are recorded for every 500 steps. Subsequently, the
development set is used to determine the best model checkpoint to compute the
final performances based on the remaining test set.

##  4 Analysis of dataset artifacts

###  4.1 Hypothesis-only baseline model

In our study, we examine dataset artifacts through the utilization of a
hypothesis-only baseline model as described previously Poliak et al. (2018).
The approach involves excluding all premise sentences from the dataset and
exclusively employing hypothesis sentences for both training and evaluation
purposes. To implement this, we leverage the `datasets.load_dataset` function
from the datasets package to load the SNLI dataset. Subsequently, we partition
the dataset into training, developing, and testing sets and remove premise
sentences for all subsets, retaining only the hypothesis sentences. The
hypothesis-only training set is then utilized to fine-tune the ELECTRA-small
pre-trained model. Following the model’s training, we optimize the models
based on the developing set performance to select the optimal checkpoint
before assessing its performance on the hypothesis-only testing set.
Additionally, we conduct a comparative analysis between the performance of the
hypothesis-only model and a model trained on the complete SNLI dataset,
encompassing both premise and hypothesis sentences. This comparative
evaluation aims to identify the impact of premise sentence removal on the
model’s overall performance and also to establish a baseline for evaluating
our strategies for artifact mitigation later.

Models | Accuracy | Dataset
---|---|---
Premise and hypothesis | 88.32 | SNLI
Hypothesis-only | 69.37 | SNLI
Table 2:  ELECTRA small performances on the SNLI dataset. "Premise and
hypothesis" indicates the model trained with complete data, while "hypothesis-
only" indicates the model trained solely on the hypothesis. Performances are
shown in percentages.

Table 2 provides a performance comparison between two models: one trained on
both premise and hypothesis sentences and the other trained only on hypothesis
sentences. The accuracy values indicate how well each model performed on the
SNLI testing set. The accuracy of the standard premise and hypothesis model is
high, as expected, at 88.32%. Surprisingly, the model trained only on the
hypothesis also achieves relatively good performance, with 69.37% accuracy.
This is much more impressive than the expected random guess accuracy of 1/3,
indicating the existence of spurious patterns present in the datasets. As a
result, the hypothesis-only model can exploit these signals to achieve high
performance without learning the relationships between premise and hypothesis.

###  4.2 Statistical testing

Type |  Word |  Entailment |  Neutral |  Contradiction |  P-value
---|---|---|---|---|---
Expected proportion |  |  33.39 |  33.27 |  33.35 |  1.0
Main subject (noun) |  man |  31.66 |  33.21 |  35.13 |  6.0e-9
Main subject (noun) |  women |  29.92 |  31.66 |  38.41 |  4.0e-80
Main subject (noun) |  people |  41.93 |  28.96 |  29.12 |  1.0e-inf
Main subject (noun) |  men |  30.31 |  33.16 |  36.52 |  4.6e-14
Main subject (noun) |  boy |  29.51 |  34.70 |  35.79 |  1.9e-14
Main verb |  is |  34.64 |  35.94 |  29.42 |  7.3e-81
Main verb |  are |  39.16 |  33.29 |  27.55 |  1.2e-132
Main verb |  playing |  34.88 |  30.62 |  34.50 |  1.2e-17
Main verb |  wearing |  37.76 |  29.89 |  32.35 |  2.3e-36
Main verb |  sitting |  29.34 |  16.17 |  54.49 |  1.0e-inf
Table 3:  Test statistics results including label proportions corresponding to
three classes and p-values for top 10 common subject nouns and main verbs. The
first row is the expected proportions of three classes across the dataset.
Extreme values are highlighted.

Although the hypothesis-only approach is an effective means for detecting and
measuring dataset artifacts by examining model performances Poliak et al.
(2018); Tsuchiya (2018); Gururangan et al. (2018), the key disadvantage of
this approach is the lack of interpretability. In other words, the method
cannot show us why hypothesis-only models are performing so well and cannot
pinpoint biases. Motivated by the observation that hypothesis-only models can
predict textual entailment accurately without seeing the premises, we
hypothesize that the vocabulary or word choice in the hypothesis may be
associated with its labels. To address this issue, we propose a simple yet
effective statistical testing procedure to unveil biases behind hypothesis-
only models.

The proposed method firstly performs part-of-speech tagging parsing to extract
the syntactic information of the sentences. It then applies rule-based filters
to extract the most meaningful words in the sentences such as the main
subjects, or main verbs. Finally, a statistical test for the association
between the extracted words and hypothesis labels is carried out with
$\chi^{2}$ goodness of fit test given the null hypothesis is the expected
proportion of each label class.

To implement this idea, we apply the procedure to extract the main subject
(noun) and main verb for each hypothesis example in the training set and
collect the corresponding labels. Examples in which our syntactic parsing
fails to extract information are excluded from the analysis. We then perform
statistical testing on the top 5 common nouns and verbs against the expected
proportions. The top popular nouns and verbs tested are "man," "woman,"
"people," "men," "boy" and "is," "are," "playing," "wearing," "sitting"
respectively.

Results of the tests are shown in Table 3 and visualized in appendix Figure 1,
2. The first subplot being the expected proportion across the dataset and the
remaining 5 subplots corresponding to the top 5 common words for each type. In
general, there is a clear association between label classes and tested words.
For example, the word "people" likely go with "entailment" (41.93%) why other
subjects like "man, women, boy, men" tends to be accompanied with
"contradiction". Similarly, the tests for main verbs also reveal that verbs
like "wearing, are" tend to be "entailment" while "sitting" strongly
associated with "contradiction" (54.49%). These evidences indicate that
vocabulary biases toward a specific label, i.e., knowing who is doing what,
indeed carry information about the true labels of the hypothesis, and also
help us to explain why the hypothesis-only model performs much better than
expected by chance.

##  5 Proposed approaches

The identification of vocabulary biases naturally prompts us to the
exploration of data augmentation strategies. The key idea is introducing
perturbations that may aid in balancing the vocabulary distributions for the
three labels. To assess this idea, we employ several types of data
augmentations commonly utilized in NLP, such as character perturbations and
diverse word replacement methods. Given that the biases originate from the
hypotheses, our augmentation procedures are exclusively applied to this
segment of the provided examples.

Given the typically large size of NLI datasets, our objective is to develop a
fully automatic system for generating augmented data. To this end, we adopt
Natural Language Processing Augmentation or `nlpaug` package
111https://github.com/makcedward/nlpaug Ma (2019) to generate augmented text
data. `nlpaug` is a widely used Python library designed for data augmentation
in NLP. `nlpaug` offers a variety of techniques, from simple rule-based
approaches to more sophisticated methods like back translation and contextual
word embedding. In addition, various augmentation methods implemented in the
package are provided with simple and consistent API.

In detail, this study implements five distinct NLP augmentation strategies,
comprising one at the character level and four at the word level, with the aim
of enhancing model accuracy and alleviating dataset biases. For character-
level augmentation, the `nlpaug.augmenter.char.RandomCharAug` class with
`action="substitute"` is applied to generate random character replacements for
the given hypothesis. At a higher level, random word augmentation involves
replacing words randomly using sophisticated methods that identify suitable
replacements through similar embeddings, word synonym exploitation, and
matched `tf-idf` distribution. Specifically, the
`nlpaug.augmenter.word.WordEmbsAug` is employed with the `word2vec` model
Mikolov et al. (2013) as the input embedding for similar word finding. For
synonym word augmentation, the `nlpaug.augmenter.word.SynonymAug` is applied
using two different databases: `WordNet` Miller (1995) and `The Paraphrase
Database (PPDB)` Ganitkevitch et al. (2013). Regarding `tf-idf` replacements,
a model is first trained using the `nlpaug.model.word_stats.TfIdf` API,
followed by sampling using `nlpaug.augmenter.word.TfIdfAug`.

Post-augmentation, the augmented data are combined with the original datasets.
Subsequently, both baseline hypothesis-only models and complete models,
incorporating both hypotheses and premises, are trained to evaluate the
proposed approaches. Results are detailed in the following section.

##  6 Experimental results

Augmentation approaches | Premise and hypothesis |  Hypothesis-only
---|---|---
No - baseline |  88.32 |  69.37
Character |  88.74 |  69.29
Word embeding (word2vec) |  88.62 |  68.23
Word sysnonym (PPDB) |  88.98 |  69.59
Word sysnonym (wordnet) |  88.84 |  69.24
Word distribution (tf-idf) |  88.56 |  68.81
Table 4:  ELECTRA small performances on the SNLI dataset with various data
augmentation techniques. "Premise and hypothesis" indicates the model trained
with complete data, while "hypothesis-only" indicates the model trained solely
on the hypothesis. Performances are shown in percentages. The top performance
for each model type is highlighted.

The performance of ELECTRA small on the SNLI dataset with various data
augmentation techniques is presented in Table 4. In general, we anticipate an
effective solution that enhances the overall model performance while
diminishing the efficiency of the hypothesis-only model. As indicated in Table
4, all augmentation approaches result in performance improvements for the
complete models, while four out of five approaches show reductions in
performance for the hypothesis-only model.

The most effective approach for mitigating biases in the SNLI dataset is word
replacement by `word2vec` similarity, leading to a reduction in hypothesis-
only accuracy of 1.14% from 69.37% to 68.23%. Following closely is `tf-idf`
distribution sampling, which results in a reduction of 0.56%. In terms of
complete model optimization, augmentation by word synonym from `BPDP` proves
to be the most successful approach, demonstrating an improvement of 0.66% from
88.32% to 88.98%. Additionally, word synonym augmentation by `WordNet` and
character-level augmentation are also effective, achieving accuracies of
88.84% and 88.74%, respectively.

##  7 Conclusion and future works

In this study, we address a prevalent issue in NLP - dataset artifacts, with a
specific focus on the Natural Language Inference (NLI) domain. We identify
these artifacts in the SNLI dataset using a hypothesis-only baseline model by
fine-tuning the pre-trained ELECTRA model. Additionally, we introduce and
apply an innovative statistical testing procedure to uncover the sources of
these artifacts. Our investigation highlights that biases originate from the
vocabulary distribution in the hypothesis sets. To mitigate these dataset
artifacts, we propose several data augmentation techniques designed to shift
the distribution.

Through evaluations on both complete models and hypothesis-only models, we
observe that data augmentation is indeed effective in enhancing the
performance of complete models and reducing biases in their hypothesis-only
counterparts.

While our findings are promising, there are opportunities for future
improvements. The current implementation heavily relies on the `nlpagu`
library, which may not offer the highest flexibility for the NLI domain. One
limitation is that the current augmentation approach does not consider the
results of the proposed statistical tests. In the future, we believe that
augmentations targeting highly biased words could lead to improved performance
in both model accuracy and artifact mitigation.

## Ethics Statement

Not applicable.

## Acknowledgements

I would like to thank my beloved person, My Phan, for always supporting my
research.

## References

  * Bowman et al. (2015) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015.  A large annotated corpus for learning natural language inference.  _arXiv preprint arXiv:1508.05326_.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_.
  * Ganitkevitch et al. (2013) Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013.  Ppdb: The paraphrase database.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 758–764.
  * Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. 2020.  Evaluating models’ local decision boundaries via contrast sets.  _arXiv preprint arXiv:2004.02709_.
  * Gardner et al. (2021) Matt Gardner, William Merrill, Jesse Dodge, Matthew E Peters, Alexis Ross, Sameer Singh, and Noah A Smith. 2021.  Competency problems: On finding and removing artifacts in language data.  _arXiv preprint arXiv:2104.08646_.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018.  Annotation artifacts in natural language inference data.  _arXiv preprint arXiv:1803.02324_.
  * Liu et al. (2019) Nelson F Liu, Roy Schwartz, and Noah A Smith. 2019.  Inoculation by fine-tuning: A method for analyzing challenge datasets.  _arXiv preprint arXiv:1904.02668_.
  * Ma (2019) Edward Ma. 2019.  Nlp augmentation.  https://github.com/makcedward/nlpaug.
  * MacCartney and Manning (2008) Bill MacCartney and Christopher D Manning. 2008.  Modeling semantic containment and exclusion in natural language inference.  In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_ , pages 521–528.
  * McCoy et al. (2019) R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  _arXiv preprint arXiv:1902.01007_.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Miller (1995) George A Miller. 1995.  Wordnet: a lexical database for english.  _Communications of the ACM_ , 38(11):39–41.
  * Morris et al. (2020) John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020.  Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp.  _arXiv preprint arXiv:2005.05909_.
  * Nie et al. (2019) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019.  Adversarial nli: A new benchmark for natural language understanding.  _arXiv preprint arXiv:1910.14599_.
  * Poliak et al. (2018) Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018.  Hypothesis only baselines in natural language inference.  _arXiv preprint arXiv:1805.01042_.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of nlp models with checklist.  _arXiv preprint arXiv:2005.04118_.
  * Tsuchiya (2018) Masatoshi Tsuchiya. 2018.  Performance impact caused by hidden bias of training data for recognizing textual entailment.  _arXiv preprint arXiv:1804.08117_.
  * Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019.  Universal adversarial triggers for attacking and analyzing nlp.  _arXiv preprint arXiv:1908.07125_.
  * Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017.  A broad-coverage challenge corpus for sentence understanding through inference.  _arXiv preprint arXiv:1704.05426_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_ , pages 38–45.
  * Zhou and Bansal (2020) Xiang Zhou and Mohit Bansal. 2020.  Towards robustifying nli models against lexical dataset biases.  _arXiv preprint arXiv:2005.04732_.

##  Appendix A Appendix figures

Figure 1: Statistics results including label proportions corresponding to
three classes and p-values for top 5 common subject nouns. The first row is
the expected proportions of three classes across the dataset. Figure 2:
Statistics results including label proportions corresponding to three classes
and p-values for top 5 common main verbs. The first row is the expected
proportions of three classes across the dataset.