11institutetext: University of Maryland, College Park 22institutetext:  Google
Research

# PRIF: Primary Ray-based Implicit Function

Brandon Yushan Feng 11 Yinda Zhang 22 Danhang Tang 22 Ruofei Du 22 Amitabh
Varshney 11

###### Abstract

We introduce a new implicit shape representation called Primary Ray-based
Implicit Function (PRIF). In contrast to most existing approaches based on the
signed distance function (SDF) which handles spatial locations, our
representation operates on oriented rays. Specifically, PRIF is formulated to
directly produce the surface hit point of a given input ray, without the
expensive sphere-tracing operations, hence enabling efficient shape extraction
and differentiable rendering. We demonstrate that neural networks trained to
encode PRIF achieve successes in various tasks including single shape
representation, category-wise shape generation, shape completion from sparse
or noisy observations, inverse rendering for camera pose estimation, and
neural rendering with color.

##  1 Introduction

Learning an accurate and efficient geometric representation of a 3D object is
an important problem for computer graphics, computer vision, and robotics.
Recent advances in machine learning have inspired a growing trend of implicit
neural shape representations, where a neural network learns to predict the
signed distance function (SDF) for an arbitrary location in the 3D space.
Moreover, in addition to the 3D location $(x,y,z)$, the neural SDF network may
take in a latent vector that describes the object identity, thus enabling the
generative modeling of multiple objects. Such an implicit neural
representation (INR) not only produces fine-grained geometry, but also enables
a plethora of applications [28], e.g., shape completion, pose estimation, via
a differentiable rendering based optimization.

However, rendering and extracting the shape from a trained neural SDF network
are computationally expensive and often limited to watertight shapes. The
direct approach to rendering from SDF requires sphere tracing, which needs
access to the SDF values at multiple locations along each pixel ray [21]. The
indirect approach computes and stores the SDF values at predefined 3D grid
points, from which the shape can be rendered with sphere-tracing or extracted
as a polygon mesh through Marching Cubes [30]. Both cases demand a large
number of network evaluations since they require sampling SDF values at
locations far away from the surface. Moreover, the shape quality is ultimately
constrained by the converging criteria of sphere tracing or the 3D resolution
of the grid.

Figure 1: Overview. Common neural shape representation methods learn the
level-set functions implicitly describing the object geometry, such as the
signed distance functions (SDF) and occupancy functions (OF). Rendering from
these implicit networks requires either sphere tracing or rasterizing a
separately extracted mesh. Sphere tracing is inefficient since it requires
multiple network evaluations for a single ray. Rasterization induces a
separate meshing step (often through Marching Cubes), which hinders end-to-end
gradient propagation in differentiable applications, and the shape quality is
ultimately restricted by the meshing algorithm’s limitation, such as grid
resolutions and shape watertightness. Our new representation, PRIF, directly
maps each primary ray to its hit point. A network encoding PRIF is more
efficient and convenient for rendering, since it requires only one evaluation
for each ray, avoids the watertight constraint in conventional methods, and
easily enables differentiable rendering.

In this work, we present a novel geometric representation that is efficient,
accurate, and innately compatible for downstream tasks involving
reconstruction and rendering. We break away from the conventional point-wise
implicit functions and propose to encode 3D geometry into a novel ray-based
implicit function. Specifically, our representation operates on the realm of
oriented rays $r=(\textbf{p}_{r},\textbf{d}_{r})$, where
$\textbf{p}_{r}\in\mathbb{R}^{3}$ is the ray origin and
$\textbf{d}_{r}\in\mathbb{S}^{2}$ is the normalized ray direction. Unlike SDF
that only outputs the distance to the nearest but undetermined surface point,
we formulate our representation such that its output directly reveals the
surface hit point of the input ray. The rays whose surface intersection we
care about are specifically known in computer graphics as primary rays, whose
origins are the rendering viewpoint, as opposed to secondary rays that
originate from the object surface [1]. Therefore, we name our representation
as Primary Ray-based Implicit Function (PRIF). In effect, a neural network
trained to encode PRIF represents the manifestation of the object’s geometry
at any viewpoint that it is observed.

Modeling the object from the ray-based perspective, rather than 3D-point-
based, has huge implications for efficient application in any task that
involves rendering the object from a viewpoint. In Section 4, we show that
PRIF outperforms common functions such as SDF and OF, and we further
demonstrate successful applications of PRIF to various tasks using neural
networks.

Properly formulating the rays is nontrivial. While it may be intuitive to let
PRIF output the distance from the ray origin $\textbf{p}_{r}$ to the hit point
$\textbf{h}_{r}$, this formulation leads to ray aliasing - i.e., if we move
$\textbf{p}_{r}$ along the ray direction $\textbf{d}_{r}$, the distance to the
hit point $\textbf{h}_{r}$ needs to change. However, the actual surface
intersection point would not change to translation along the view direction.
Therefore, it is undesirable to have such a potential variance as it adds
unnecessary complexity to the network output.

To avoid the aliasing, we reparametrize the ray $r$ by replacing the view
position $\textbf{p}_{r}$ with perpendicular foot $\textbf{f}_{r}$ from the
coordinate origin O to $r$. Wherever we move $\textbf{p}_{r}$ along the ray
direction $\textbf{d}_{r}$, the perpendicular foot $\textbf{f}_{r}$ stays the
same. Furthermore, we can easily define the surface hit point $\textbf{h}_{r}$
by a single scalar value: its distance from $\textbf{f}_{r}$. Thus, we
formulate PRIF so that it outputs this distance for an input ray. More details
are in Section 3.

In summary, our main contributions are the followings:

  * $\bullet$

We present PRIF, a novel formulation for geometric representation using ray-
based neural networks.

  * $\bullet$

We show that PRIF outperforms common level-set-based methods in shape
representation accuracy and extraction speed.

  * $\bullet$

We demonstrate that PRIF enables generative modeling, shape completion, and
camera pose estimation.

##  2 Related Work

We discuss prior art on neural representations for 3D shapes and scene rays.

###  2.1 3D Shape Representations

####  2.1.1 Functional Representations.

Traditional 3D shape representations include polygon meshes, point clouds, and
voxels. In recent years, as deep neural networks achieve remarkable success on
various vision-related tasks, there has been a growing interest in developing
implicit neural representations (INRs) of 3D shapes. Following seminal works
[9, 36, 32] showing successful applications of neural network to encode 3D
shapes, many methods have been introduced to solve various vision and graphics
tasks using INRs of 3D shapes [38, 10, 28, 35, 56, 24, 17, 43, 40, 52, 3, 11,
29, 15, 7, 49, 42, 4, 53].

INRs usually use the multilayer perceptron (MLP) architecture to encode
geometric information of a 3D shape by learning the mapping from a given 3D
spatial point and a scalar value. Typically, the output scalar value denotes
either the occupancy at the given point, or the signed distance from the given
point to the nearest point on the shape. On one hand, networks that are
trained to encode the occupancy function [32] (OF) essentially learns the
binary classification problem, where the output equals 0 if the point is
empty, and equals 1 if occupied. Therefore, the decision boundary where the
network predictions equal to 0.5 represents the surface of the encoded shape.
On the other hand, for networks trained to encode the signed distance function
[9, 36] (SDF), the surface is represented by the decision boundary where the
network predictions equal to 0. A 3D surface determined in such fashions is
also known as an isosurface, which is a level set of a continuous 3D function.
In practice, however, obtaining 3D meshes from these isosurfaces extracted
from INRs still requires an additional meshing step often through the Marching
Cubes algorithm.

In this paper, we introduce a new shape representation which is not determined
by an isosurface. Instead, our representation encodes a 3D shape by learning
the PRIF associated with the shape. Outputs of such a function directly
correspond to points on the surface, and the shape can be extracted without an
additional meshing step that could inject inaccuracies to the final
representation.

####  2.1.2 Global v.s. Local Representations

The initial works of INRs for 3D shapes inspire many techniques to improve its
the rendering efficiency [28, 55] and representation quality [44, 48]. Among
many techniques, a common thread is the idea of spatial partitions which
manifest in two main approaches. One approach divides the surfaces of shapes
into different local patches, reducing the difficulty of globally fitting a
complex surface with a single network [50, 20, 23, 19, 5, 37]. Another
approach divides the 3D volume into small local regions (often based on the
spatial octree structure), and then train INRs to encode the geometric
information within each local region [47, 54, 8, 14, 31, 27, 34, 41].

In this paper, we only focus on global representations, where a shape is
represented by a single network without any spatial partitions. While the
aforementioned works largely focus on improving the performance of INRs based
on SDF, our main contribution is a new functional representation in place of
isosurface-based representations like SDF. Nonetheless, the idea of spatial
partitions explored in previous works has the potential to improve the
performance of our currently global representation, and we see local
specializations as a promising future direction to explore.

###  2.2 Ray-based Neural Networks

Our method is closely related to an emerging concept called Ray-based Neural
Networks. Rays are a common construct from computer graphics, and can be
denoted by a tuple consisting of a 3D point (ray origin) and a 3D direction
(ray direction). Due to their closeness to rendering 3D scenes, rays have
become a central component in the problem formulation of many recent works
using neural networks to model 3D scenes [33, 46]. Feng et al. [18] and Attal
et al. [2] have further demonstrated that for front-facing light field scenes,
MLPs can be trained to accurately map camera rays to their observed colors in
highly detailed real-world scenes, but their networks only consider rays
restricted within two parallel planes. Most recently, Sitzmann et al. [45]
successfully train a MLP to encode the observed color of unrestricted rays
with arbitrary origins and directions, and the key is to parametrize rays
using the Plucker coordinates [22].

We similarly adopt rays as input to the neural network. However, instead of
the Plucker parametrization which replaces the ray position by its moment
vector about the origin, we replace the moment vector by the perpendicular
foot between the ray and the origin. As discussed in Sec. 3, our formulation
allows the network to simply produce an affine transformation to its input.

Figure 2: Representation. (a) Signed distance at a sampling position (white)
reveals the sphere (blue dots) where its nearest surface point (blue) exists,
but that may be irrelevant when we really want to know the hit point (red)
along a specific direction. Thus, multiple samples are often required. (b) Our
new representation uses only one sample (yellow) along the ray to obtain its
surface hit point. The sampling position is the perpendicular foot between the
given ray and the coordinate system’s origin O. (c) The proposed function
takes in the ray’s direction and its sampling point, and returns the distance
from that point to the actual surface hit point. We train neural networks to
encode geometry by learning this new function, and we demonstrate the
capability of our trained networks in various tasks involving shape
representation and rendering.

##  3 Method

In this section, we introduce our new representation for 3D shapes. We first
discuss the motivation behind ray parametrization used in prior works, and
then we present our new formulation specifically designed for shape
representations.

###  3.1 Background

Sitzmann et al. [45] encode light fields by training an MLP $\Phi_{\phi}$ with
parameters $\phi$ on a set of observed rays to learn the mapping of
$r=(\textbf{p}_{r},\textbf{d}_{r})\to c_{r}$, where $c_{r}$ denotes the color
of the observed radiance. However, naively concatenating $\textbf{p}_{r}$ and
$\textbf{d}_{r}$ as input to the network is not ideal due to ray aliasing. If
we move the position of a ray $r$ along the ray direction $\textbf{d}_{r}$, we
would obtain a ray $r^{\prime}=(\textbf{p}_{r^{\prime}},\textbf{d}_{r})$ that
is really an aliased version of $r$. There is no guarantee that the trained
network would produce the same output for different aliases $r^{\prime}$ of
the ray $r$.

To resolve ray aliasing, Sitzmann et al. [45] reparametrize the ray
$r=(\textbf{p}_{r},\textbf{d}_{r})$ into Plucker coordinates as
$r=(\textbf{m}_{r},\textbf{d}_{r})$, where
$\textbf{m}_{r}=\textbf{p}_{r}\times\textbf{d}_{r}$ is also known as the
moment vector of $\textbf{p}_{r}$ about the origin O. Plucker coordinates
represent all oriented rays in space without singularity or special cases, and
they are invariant to changes in the ray position along the ray direction. To
better understand this property, consider moving the ray position to some
other point $\textbf{p}_{r}^{\prime}$ at a fixed ray direction
$\textbf{d}_{r}$. Then, for a certain $\lambda\in\mathbb{R}$,
$\textbf{p}_{r}^{\prime}=\textbf{p}_{r}-\lambda\textbf{d}_{r}$, and

$\displaystyle\textbf{p}_{r}^{\prime}\times\textbf{d}_{r}$
$\displaystyle=(\textbf{p}_{r}-\lambda\textbf{d}_{r})\times\textbf{d}_{r}$
$\displaystyle=\textbf{p}_{r}\times\textbf{d}_{r}-\lambda\mathbf{0}$
$\displaystyle=\textbf{p}_{r}\times\textbf{d}_{r}.$ (1)

Therefore, $\textbf{m}_{r}=\textbf{p}_{r}\times\textbf{d}_{r}$ is invariant to
any change of $\lambda$ along $\textbf{d}_{r}$.

###  3.2 Describing Geometry with Perpendicular Foot

Our goal is to train a neural network to encode the mapping from a ray to its
hit point on the 3D shape’s surface. Although replacing ray position with the
moment vector $\textbf{m}_{r}$ allows Sitzmann et al. [45] to train networks
to encode light fields, it is hard to geometrically relate a ray’s moment
vector to its surface hit point. We propose an alternative way to parameterize
a ray as input to the network. which has an intuitive and intrinsic
relationship to its hit point.

Specifically, we consider the perpendicular foot $\textbf{f}_{r}$ between the
ray $r$ and the coordinate system’s origin O, which may be computed by

$\textbf{f}_{r}=\textbf{d}_{r}\times(\textbf{p}_{r}\times\textbf{d}_{r}).$ (2)

Similar to $\textbf{m}_{r}$ in Plucker coordinates, $\textbf{f}_{r}$ is also
invariant to changing the ray position along the ray direction. Specifically,
let $\textbf{p}_{r}^{\prime}$ be the translated ray position defined as
before, we can then write
$\textbf{p}_{r^{\prime}}=\textbf{p}_{r}-\lambda\textbf{d}_{r}$, and

$\displaystyle\textbf{f}_{r^{\prime}}$
$\displaystyle=\textbf{d}_{r}\times(\textbf{p}_{r^{\prime}}\times\textbf{d}_{r})$
$\displaystyle=\textbf{d}_{r}\times((\textbf{p}_{r}-\lambda\textbf{d}_{r})\times\textbf{d}_{r})$
$\displaystyle=\textbf{d}_{r}\times(\textbf{p}_{r}\times\textbf{d}_{r}-\lambda\mathbf{0})$
$\displaystyle=\textbf{d}_{r}\times(\textbf{p}_{r}\times\textbf{d}_{r})$
$\displaystyle=\textbf{f}_{r}$ (3)

In other words, $\textbf{f}_{r}$ is invariant to moving $\textbf{p}_{r}$ along
the direction $\textbf{d}_{r}$.

As a result, we can represent any ray $r=(\textbf{f}_{r},\textbf{d}_{r})$, and
we can further establish the following relationship for each ray $r$:

$\textbf{h}_{r}=s_{r}\cdot\textbf{d}_{r}+\textbf{f}_{r},$ (4)

where $s_{r}\in\mathbb{R}$ denotes the signed displacement between the ray’s
hit point $\textbf{h}_{r}$ and its perpendicular foot $\textbf{f}_{r}$ w.r.t.
the world origin O.

To encode a 3D shape, we propose the mapping function
$(\textbf{f}_{r},\textbf{d}_{r})\to s_{r}$, which we call Primary Ray-based
Implicit Function (PRIF). Different than SDF or OF, which implicitly encodes
the geometry through the distance from any given point to its nearest surface
point in any direction, PRIF operates on oriented points with a specific ray
direction.

In practice, we train an MLP to learn

$\Phi(\textbf{f}_{r},\textbf{d}_{r})=s_{r}.$ (5)

With this new representation, we are able to represent the surface hit point
of a ray with a single value. In effect, our objective is equivalent to
finding a simple affine transformation
$f(\textbf{x})=\textbf{A}\textbf{x}+\textbf{b}$, with the input
$\textbf{x}=\textbf{d}_{r}$, $\textbf{A}=s_{r}\mathit{I}_{3}$, and
$\textbf{b}=\textbf{f}_{r}$. We also avoid a major limitation in previous
sphere-tracing-based methods, which is having to sample multiple points and
perform multiple network evaluations to obtain a hit point.

###  3.3 Background Mask

For 3D functions like SDF or OF, every 3D point has a well-established scalar
value denoting the distance to surface or the occupancy at that point. In
contrast, our function would take in rays that never intersect with the shape
and therefore do not even have a hit point.

To address this issue, we let the network additionally produce

$\Phi(\textbf{f}_{r},\textbf{d}_{r})=a_{r},$ (6)

where $a_{r}\in[0,1]$ denotes the probability in which the ray $r$ hits the
foreground. We compute the cross-entropy loss

$\mathcal{L}_{a}=\sum_{r}-a_{r}^{gt}\log\left(a_{r}\right)-\left(1-a_{r}^{gt}\right)\log\left(1-a_{r}\right),$
(7)

where $a_{r}^{gt}=0$ for background rays and $a_{r}^{gt}=1$ for foreground
rays.

For the signed displacement $s_{r}$, we supervise the learning by computing
its absolute difference to the ground truth as
$\mathcal{L}_{s}=\sum_{r\in\mathcal{F}}\left\lVert
s_{r}-s_{r}^{gt}\right\rVert$, given the set of foreground rays $\mathcal{F}$.
As a result, the total loss function to train our network is
$\mathcal{L}=\mathcal{L}_{a}+\mathcal{L}_{s}$ and is averaged among all rays
in a training batch

###  3.4 Outlier Points Removal

In rare cases where sharp surface discontinuities exist between two
neighboring rays, the network would likely produce continuous predictions when
interpolating between those two rays, resulting in undesirable outlier points.
Fortunately, since our network is fully differentiable, for each prediction
$s_{r}$ we can compute its gradient with respect to the changes in ray
position. We discard all predictions that satisfy the threshold:
$\left\lVert\frac{\partial
s_{r}}{\partial\textbf{p}_{r}}\right\rVert\geq\delta.$ In our experiments,
$\delta$ is set equal to 5.

##  4 Experiments

In this section, we first verify the efficacy of neural PRIF for shape
representation. Then, we show verious applications achieved by using PRIF as
the underlying neural shape representation. Note that the scope of our
experiments is to compare these functional shape representations encoded by
neural networks.

$\underbracket{\hskip 433.62pt}_{\begin{subarray}{c}\vspace{-5mm}\\\
\hbox{\pagecolor{white}Reference}\end{subarray}}$

$\underbracket{\hskip 433.62pt}_{\begin{subarray}{c}\vspace{-5mm}\\\
\hbox{\pagecolor{white}OF}\end{subarray}}$

$\underbracket{\hskip 433.62pt}_{\begin{subarray}{c}\vspace{-5mm}\\\
\hbox{\pagecolor{white}SDF}\end{subarray}}$

$\underbracket{\hskip 433.62pt}_{\begin{subarray}{c}\vspace{-5mm}\\\
\hbox{\pagecolor{white}Ours - Mesh}\end{subarray}}$

$\underbracket{\hskip 433.62pt}_{\begin{subarray}{c}\vspace{-5mm}\\\
\hbox{\pagecolor{white}Ours - Points}\end{subarray}}$

Figure 3: Single Shape. To examine the representation capability of PRIF, we
train networks with the same architecture to encode the occupancy function
(OF), signed distance function (SDF), and our proposed function (PRIF). Here,
we visualize the extracted shapes from the trained neural representations. For
OF and SDF, we follow the convention and extract the shapes by Marching Cubes.
Our method directly outputs hit points, and we also apply the point-based
Screened Poisson algorithm and present the resulting mesh for comparison.

###  4.1 Single Shape Representation

We select five models (Armadillo, Bunny, Buddha, Dragon, Lucy) from the
Stanford 3D Scanning Repository [51, 13] and train a neural network to fit the
PRIF for each 3D model. We also fit the signed distance function (SDF) and the
3D occupancy function (OF) for these models. For a fair comparison between
these functions, we adopt the same network architecture as Park et al. [36],
containing eight layers with 512 hidden dimensions and ReLU activation.

Compared to SDF and OF which are trained on individual spatial points
$(x,y,z)$, PRIF requires an inherently different strategy to generate the
training data since it takes in individual rays. In our experiments, for each
3D model, we select 50 virtual camera locations oriented towards the origin,
and we capture $200\times 200$ rays at each location. For SDF and OF, we
follow Park et al. [36] and sample $500,000$ points with more aggressive
sampling near the object surface.

For all three functions, we train the neural representation for 100 epochs
with the learning rate initialized as $10^{-4}$ and decayed to $10^{-7}$ with
a cosine annealing strategy. After training, the SDF- and OF-based shape
representation are obtained by evaluating the neural network at uniform
$256^{3}$ volume grid and extracted using Marching Cubes. On the other hand,
with our PRIF-based representation, we can evaluate the neural network at
those virtual camera rays in the training set and directly obtain a dense set
of surface points.

Method | Armadillo | Bunny | Buddha | Dragon | Lucy
---|---|---|---|---|---
SDF | 1.905$|$1.260 | 1.717$|$1.147 | 6.119$|$2.258 | 5.184$|$1.946 | 3.387$|$1.417
OF | 4.805$|$1.624 | 1.704$|$1.133 | 17.279$|$3.113 | 19.577$|$3.014 | 3.396$|$1.427
PRIF | 0.978$|$0.706 | 1.169$|$0.835 | 1.443$|$0.821 | 1.586$|$0.913 | 0.846$|$0.519
Table 1: Quantitative results on single shape representation on 3D models from
the Stanford 3D Scanning Repository. The left and right numbers represent the
mean and median CD (multiplied by $10^{-4}$). After extracting shapes from
each representation, 30,000 points are sampled for evaluation.

To evaluate the representation quality, for SDF- and OF-based representations,
we first follow conventions [36] and sample $8,192$ points on the mesh
extracted with Marching Cubes. For the point set produced by the PRIF-based
representation, we apply the point-based meshing algorithm Screened Poisson
[25] in MeshLab [12] and then sample $8,192$ points from the reconstructed
mesh. Then, we obtain $8,192$ Poisson-disk samples of the ground truth surface
points from the original 3D model. Finally, we compute the mean and median
Chamfer Distance (CD) between the ground truth point set and point sets
sampled from those three representations. In Table 1 and Fig. 3, we provide
quantitative and qualitative comparisons among the three representations. PRIF
significantly outperform SDF and OF in accurately preserving the fine details
of the 3D shapes.

$\underbracket{\hskip 433.62pt}_{\begin{subarray}{c}\vspace{-5mm}\\\
\hbox{\pagecolor{white}Reference}\end{subarray}}$

$\underbracket{\hskip 433.62pt}_{\begin{subarray}{c}\vspace{-5mm}\\\
\hbox{\pagecolor{white}OF}\end{subarray}}$

$\underbracket{\hskip 433.62pt}_{\begin{subarray}{c}\vspace{-5mm}\\\
\hbox{\pagecolor{white}SDF}\end{subarray}}$

$\underbracket{\hskip 433.62pt}_{\begin{subarray}{c}\vspace{-5mm}\\\
\hbox{\pagecolor{white}Ours - Mesh}\end{subarray}}$

$\underbracket{\hskip 433.62pt}_{\begin{subarray}{c}\vspace{-5mm}\\\
\hbox{\pagecolor{white}Ours - Points}\end{subarray}}$

Figure 4: Shape Generation. We further examine the representation power on
generative modeling over multiple shapes. After training the neural networks
to learn shapes from a category, we use auto-decoding to find latent vectors
that best represents novel shapes within this category. Here, we visualize the
extracted shapes from networks trained to encode the three different
functions. Note that Points are the immediate output of the PRIF networks.

Method | Car | Chair | Table | Plane | Lamp | Sofa
---|---|---|---|---|---|---
SDF | 2.315$|$0.495 | 2.649$|$0.407 | 7.213$|$0.441 | 2.728$|$0.170 | 32.571$|$3.475 | 6.427$|$0.218
OF | 2.820$|$0.587 | 4.589$|$0.835 | 6.427$|$1.296 | 2.999$|$0.169 | 143.377$|$145.753 | 12.672$|$0.184
PRIF | 1.961$|$0.347 | 0.982$|$0.267 | 4.532$|$0.315 | 0.389$|$0.125 | 3.276$|$0.534 | 1.236$|$0.222

Table 2: Quantitative results on generative representation on unseen 3D shapes
of six categories from ShapeNetCore. The left and right numbers represent the
mean and median CD ($\times 10^{-3}$) averaged over the test set. After
extracting shapes from each representation, 30,000 points are sampled for
evaluation.

$\underbracket{\hskip
433.62pt}_{\begin{subarray}{c}\hbox{\pagecolor{white}Noisy
Data}\end{subarray}}$

$\underbracket{\hskip
433.62pt}_{\begin{subarray}{c}\hbox{\pagecolor{white}Reconstruction}\end{subarray}}$

$\underbracket{\hskip
433.62pt}_{\begin{subarray}{c}\hbox{\pagecolor{white}Incomplete
Data}\end{subarray}}$

$\underbracket{\hskip
433.62pt}_{\begin{subarray}{c}\hbox{\pagecolor{white}Reconstruction}\end{subarray}}$

Figure 5: Noisy and Incomplete Observations. Networks trained for shape
generation can reconstruct unseen shapes in challenging scenarios where only
noisy or incomplete observations are available. Here we visualize the
observations and reconstructions as raw point cloud data.

###  4.2 Shape Generation

Having established the representation power of PRIF for single shapes, we now
examine its capability for generative shape modeling. We adopt the strategy
from Park et al. [36] and enable multi-shape generation from a single network
by concatenating a latent code for each object with the original network
input.

In line with Park et al. [36], we select five categories of 3D objects from
ShapeNetCore [6]. Within each category, we select the first 180 objects as the
training set and the next 20 as the testing set. We maintain the same network
architecture and training schedule when training neural networks to fit OF,
SDF, and PRIF. We evaluate the trained networks on unseen shapes from the test
set using auto-decoding [36] and provide quantitative and qualitative
comparisons in Table 2 and Fig. 4. The PRIF-based representation extends its
success from representing single shapes and remains effective in the
generative task.

Figure 6: Complexity Analysis. We evaluate the trained SDF network at grid
resolutions of $64^{3}$, $128^{3}$, $192^{3}$, $256^{3}$, $512^{3}$ and obtain
final meshes by Marching Cubes. We evaluate the trained PRIF network at
resolutions of $5\times 200^{2}$, $10\times 400^{2}$, $20\times 400^{2}$,
$25\times 400^{2}$, $50\times 400^{2}$ (# of Cameras $\times$ Resolution), and
we apply Screened Poisson [25] to the output hit points to obtain the final
mesh. The reconstructed mesh quality is measured by mean and median CD
($\times 10^{-5}$). The PRIF network achieves better quality with fewer
evaluations.

$\underbracket{\hskip
433.62pt}_{\begin{subarray}{c}\hbox{\pagecolor{white}Ground
Truth}\end{subarray}}$

$\underbracket{\hskip
433.62pt}_{\begin{subarray}{c}\hbox{\pagecolor{white}OF}\end{subarray}}$

$\underbracket{\hskip
433.62pt}_{\begin{subarray}{c}\hbox{\pagecolor{white}SDF}\end{subarray}}$

$\underbracket{\hskip
433.62pt}_{\begin{subarray}{c}\hbox{\pagecolor{white}Ours -
Points}\end{subarray}}$

Figure 7: Stress Testing. We test on a Tetrahedron grid that is self-
intersecting and non-watertight. We obtain the SDF and OF values with the
scanning method [26], and extract the mesh by Marching Cubes. While the level-
set representations fail as expected, our method reliably preserves the shape.

Setup | Trained | Novel
---|---|---
Proposed | 12.07$|$7.95 | 20.81$|$12.05
$\textbf{f}_{r}\Rightarrow$ $\textbf{m}_{r}$ | 12.15$|$8.05 | 28.38$|$13.53
$\textbf{f}_{r}\Rightarrow$ $\textbf{p}_{r}$ | 1.16$|$0.77 | 1924.4$|$1067.7
w/ Outliers | 12.14$|$8.06 | 21.32$|$12.05

[table]A table beside a figure

Figure 8: Ablations. Left: Impact of outlier removal described in Sec. 3.
While the outliers have little impact on the quantitative metrics since they
are scant, removing them improves visual consistency. Right: Mean and median
CD ($\times 10^{-5}$) of network prediction on trained rays and novel rays
under different setup. Notice that changing $\textbf{f}_{r}\Rightarrow$
$\textbf{p}_{r}$ causes ray aliasing discussed in Sec. 3, leading the network
to overfit on trained rays and collapse on novel rays.

###  4.3 Shape Denoising and Completion

As depth sensors have become available on mobile and AR/VR devices, there are
an array of applications for persistent geometric reconstructions [16]. Given
real-world data of sparse and noisy observations, the capability of generative
modeling could enable shape recovery from noisy and incomplete point clouds.

In Fig. 5, we demonstrate the denoising and completion ability of a trained
generative PRIF network. Specifically, after training a PRIF network on an
object category as described before, we provide the network with an unseen
object’s point cloud observations, which are either incomplete or contain
noise.

###  4.4 Analysis and Ablations

####  4.4.1 Complexity Analysis.

Shapes are commonly extracted from networks encoding SDF or OF using meshing
algorithms like Marching Cubes. This meshing step requires evaluating the
network at many 3D grid sampling points, and the grid resolution affects the
final quality of the mesh. In our case, we can extract the shapes by rendering
the scene from multiple positions and change the number of evaluations by
varying the number and resolution of virtual cameras. For a more objective
comparison against prior arts that require meshing, we further apply Screened
Poisson to mesh our output hit points. In Fig. 6, we analyze the trade-off
between the number of network evaluations and the reconstructed mesh quality
measured in mean and median CD. Evidently, PRIF produces reconstruction with a
better quality with fewer network evaluations.

$\underbracket{\hskip 433.62pt}_{\begin{subarray}{c}\vspace{-5mm}\\\
\hbox{\pagecolor{white}Initial}\end{subarray}}$

$\underbracket{\hskip 433.62pt}_{\begin{subarray}{c}\vspace{-5mm}\\\
\hbox{\pagecolor{white}Optimization Progress}\end{subarray}}$

$\underbracket{\hskip 433.62pt}_{\begin{subarray}{c}\vspace{-5mm}\\\
\hbox{\pagecolor{white}Target}\end{subarray}}$

Figure 9: Learning Camera Poses. Starting from the Initial camera pose, we
optimize the learnable pose parameters based on the difference between PRIF
output rendered at the current pose and the PRIF output rendered at an unknown
Target pose. Here, we render the PRIF output as depth images. PRIF-based
rendering successfully facilitates the differentiable optimization progress as
the camera pose gradually converges to the correct Target pose.

Figure 10: Incorporating Color Rendering. After training the PRIF network to
encode the geometry (rendered as depth images in the first row), we can
further train a second-stage network that takes in the hit point produced by
PRIF and produces its corresponding color (rendered in the second row).

####  4.4.2 Stress Testing.

Fig. 7 shows an example of encoding a self-intersecting and non-watertight
shape, which is expected to be challenging for neural networks trained to
encode conventional functional representations like SDF and OF. In contrast,
the shape is well preserved by the network trained to encode our proposed
functional representation PRIF.

####  4.4.3 Ablations.

We present ablation studies validating the effectiveness of our proposed
techniques in Fig. 8 and Table 3. The results verify that: 1) not
reparametrizing $r=(\textbf{p}_{r},\textbf{d}_{r})$ leads to severe
overfitting on the training data, and 2) replacing the Plucker moment vector
$\textbf{m}_{r}$ with our proposed $\textbf{f}_{r}$ improves the encoding
quality, possibly because the network only has to learn a simple affine
transform matrix as mentioned in Sec. 3.2.

###  4.5 Further Applications

####  4.5.1 Learning Camera Poses.

Evaluating the PRIF network on some input camera rays is essentially
performing differentiable rendering of the underlying scene represented by the
network. With a PRIF network trained on a 3D shape, given a silhouette image
at an unknown camera pose, our task is to recover that camera pose based on
the silhouette image. In this task, the weights of the PRIF network are
frozen, and the only learnable parameters are the camera pose matrix. At each
iteration, we render the scene through the PRIF network with camera rays
defined by the current estimate of the camera pose, and we adjust the estimate
based on the silhouette difference between the observed and rendered images.
In Fig. 9, we present the rendered image during the optimization steps. The
estimated camera pose gradually converges to the correct solution as the
rendered image becomes more similar to the observed image.

####  4.5.2 Neural Rendering with Color.

The PRIF network behaves as a geometry renderer since it effectively returns
the surface hit point given a viewing ray. Here we show that the PRIF network
can be further extended to rendering color at the surface. We select a 3D
model of human from the Renderpeople dataset [39] and virtually capture the
hit points and RGB colors at 25 locations surrounding the model. We then train
the PRIF network to fit the geometric shape, similar to the single shape
experiments. Finally, we extend the network to produce the observed RGB color
given the input ray while fixing its hit point prediction. Fig. 10 shows the
rendered results of geometry and appearance of the 3D model.

####  4.5.3 Limitation.

This paper is focused on 3D shape encoding and decoding through hit points of
rays from known views. PRIF can render at novel views by sampling new rays,
but to guarantee consistent novel view results would likely require further
modifications such as multi-view consistency loss or denser training views.

##  5 Conclusion

We propose PRIF, a new 3D shape representation based on the relationship
between a ray and its perpendicular foot with the origin. We demonstrate that
neural networks can successfully encode PRIF to achieve accurate shape
representations. With this new representation, we avoid multi-sample sphere
tracing and obtain the hit point with a single network evaluation. Neural
networks trained to encode PRIF inherit such advantages and can represent
shapes more accurately than common neural shape representations using the same
network architecture. We further extend the neural PRIF networks to enable
various downstream tasks including generative shape modeling, shape denoising
and completion, camera pose optimization, and color rendering. Promising
future directions include using spatial partitions to improve the network
accuracy, jointly learning the geometry and view-dependent color directly from
images, speeding up network training, and real-time inference for robotic
applications.

## References

  * [1] Akenine-Moller, T., Haines, E., Hoffman, N.: Real-Time Rendering. AK Peters/CRC Press (2019)
  * [2] Attal, B., Huang, J.B., Zollhoefer, M., Kopf, J., Kim, C.: Learning Neural Light Fields With Ray-Space Embedding Networks. ArXiv Preprint ArXiv:2112.01523 (2021). https://doi.org/10.48550/arXiv.2112.01523
  * [3] Atzmon, M., Lipman, Y.: Sal: Sign Agnostic Learning of Shapes From Raw Data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2565–2574 (2020). https://doi.org/10.1109/CVPR42600.2020.00264
  * [4] Bhatnagar, B.L., Sminchisescu, C., Theobalt, C., Pons-Moll, G.: Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction. In: European Conference on Computer Vision. pp. 311–329. Springer, Springer (2020). https://doi.org/10.1007/978-3-030-58536-5_19
  * [5] Chabra, R., Lenssen, J.E., Ilg, E., Schmidt, T., Straub, J., Lovegrove, S., Newcombe, R.: Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction. In: European Conference on Computer Vision. pp. 608–625. Springer, Springer (2020). https://doi.org/10.1007/978-3-030-58526-6_36
  * [6] Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., Yu, F.: ShapeNet: an Information-Rich 3D Model Repository. Tech. Rep. arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago (2015)
  * [7] Chen, Z., Zhang, Y., Genova, K., Fanello, S., Bouaziz, S., Häne, C., Du, R., Keskin, C., Funkhouser, T., Tang, D.: Multiresolution Deep Implicit Functions for 3D Shape Representation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13087–13096 (2021). https://doi.org/10.1109/ICCV48922.2021.01284
  * [8] Chen, Z., Tagliasacchi, A., Zhang, H.: Bsp-Net: Generating Compact Meshes Via Binary Space Partitioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 45–54 (2020). https://doi.org/10.1109/CVPR42600.2020.00012
  * [9] Chen, Z., Zhang, H.: Learning Implicit Fields for Generative Shape Modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5939–5948 (2019). https://doi.org/10.1109/CVPR.2019.00609
  * [10] Chibane, J., Alldieck, T., Pons-Moll, G.: Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 6968–6979 (2020)
  * [11] Chibane, J., Alldieck, T., Pons-Moll, G.: Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6970–6981 (2020)
  * [12] Cignoni, P., Callieri, M., Corsini, M., Dellepiane, M., Ganovelli, F., Ranzuglia, G.: Meshlab: an open-source mesh processing tool. In: Eurographics Italian Chapter Conference. vol. 2008, pp. 129–136. Salerno, Italy (2008)
  * [13] Curless, B., Levoy, M.: A Volumetric Method for Building Complex Models From Range Images. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. pp. 303–312 (1996). https://doi.org/10.1145/237170.237269
  * [14] Deng, B., Genova, K., Yazdani, S., Bouaziz, S., Hinton, G., Tagliasacchi, A.: CvxNet: Learnable Convex Decomposition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 31–44 (2020). https://doi.org/10.1109/CVPR42600.2020.00011
  * [15] Driess, D., Ha, J.S., Toussaint, M., Tedrake, R.: Learning Models As Functionals of Signed-Distance Fields for Manipulation Planning. In: Conference on Robot Learning. pp. 245–255. PMLR (2022). https://doi.org/10.48550/arXiv.2110.00792
  * [16] Du, R., Turner, E., Dzitsiuk, M., Prasso, L., Duarte, I., Dourgarian, J., Afonso, J., Pascoal, J., Gladstone, J., Cruces, N., Izadi, S., Kowdle, A., Tsotsos, K., Kim, D.: DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality. In: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. pp. 829–843. UIST, ACM (Oct 2020). https://doi.org/10.1145/3379337.3415881
  * [17] Duan, Y., Zhu, H., Wang, H., Yi, L., Nevatia, R., Guibas, L.J.: Curriculum Deepsdf. In: European Conference on Computer Vision. pp. 51–67. Springer, Springer (2020). https://doi.org/10.1007/978-3-030-58598-_4
  * [18] Feng, B.Y., Varshney, A.: SIGNET: Efficient Neural Representation for Light Fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14224–14233 (2021). https://doi.org/10.1109/ICCV48922.2021.01396
  * [19] Genova, K., Cole, F., Sud, A., Sarna, A., Funkhouser, T.: Local Deep Implicit Functions for 3D Shape. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4857–4866 (2020). https://doi.org/10.1109/CVPR42600.2020.00491
  * [20] Genova, K., Cole, F., Vlasic, D., Sarna, A., Freeman, W.T., Funkhouser, T.: Learning Shape Templates With Structured Implicit Functions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7154–7164 (2019). https://doi.org/10.1109/ICCV.2019.00725
  * [21] Hart, J.C.: Sphere Tracing: a Geometric Method for the Antialiased Ray Tracing of Implicit Surfaces. The Visual Computer 12(10), 527–545 (1996)
  * [22] Jia, Y.B.: Plucker Coordinates for Lines in the Space. https://faculty.sites.iastate.edu/jia/files/inline-files/plucker-coordinates.pdf (2020)
  * [23] Jiang, C., Sud, A., Makadia, A., Huang, J., Niessner, M., Funkhouser, T., et al.: Local Implicit Grid Representations for 3D Scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6001–6010 (2020). https://doi.org/10.1109/CVPR42600.2020.00604
  * [24] Jiang, Y., Ji, D., Han, Z., Zwicker, M.: Sdfdiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1251–1261 (2020)
  * [25] Kazhdan, M., Hoppe, H.: Screened Poisson Surface Reconstruction. ACM Transactions on Graphics (ToG) 32(3), 1–13 (2013). https://doi.org/10.1145/2487228.2487237
  * [26] Kleineberg, M., Fey, M., Weichert, F.: Adversarial generation of continuous implicit shape representations. In: Wilkie, A., Banterle, F. (eds.) 41st Annual Conference of the European Association for Computer Graphics, Eurographics 2020 - Short Papers, Norrköping, Sweden, May 25-29, 2020 [online only]. pp. 41–44. Eurographics Association (2020). https://doi.org/10.2312/egs.20201013
  * [27] Lindell, D.B., Van Veen, D., Park, J.J., Wetzstein, G.: BACON: Band-Limited Coordinate Networks for Multiscale Scene Representation. ArXiv Preprint ArXiv:2112.04645 (2021)
  * [28] Liu, S., Zhang, Y., Peng, S., Shi, B., Pollefeys, M., Cui, Z.: DIST: Rendering Deep Implicit Signed Distance Function With Differentiable Sphere Tracing. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (Jun 2020). https://doi.org/10.1109/CVPR42600.2020.00209
  * [29] Liu, S., Saito, S., Chen, W., Li, H.: Learning to Infer Implicit Surfaces Without 3D Supervision. Advances in Neural Information Processing Systems 32 (2019)
  * [30] Lorensen, W.E., Cline, H.E.: Marching Cubes: a High Resolution 3D Surface Construction Algorithm. ACM Siggraph Computer Graphics 21(4), 163–169 (1987). https://doi.org/10.1145/37402.37422
  * [31] Martel, J.N., Lindell, D.B., Lin, C.Z., Chan, E.R., Monteiro, M., Wetzstein, G.: Acorn: Adaptive Coordinate Networks for Neural Scene Representation. ArXiv Preprint ArXiv:2105.02788 (2021), https://arxiv.org/pdf/2105.02788
  * [32] Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy Networks: Learning 3D Reconstruction in Function Space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4460–4470 (2019). https://doi.org/10.1109/CVPR.2019.00459
  * [33] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J., Ramamoorthi, R., Ng, R.: NeRF: Representing Scenes As Neural Radiance Fields for View Synthesis. In: ECCV 2020: Computer Vision – ECCV 2020, pp. 405–421. Springer International Publishing (2020). https://doi.org/10.1007/978-3-030-58452-_24
  * [34] Müller, T., Evans, A., Schied, C., Keller, A.: Instant Neural Graphics Primitives With a Multiresolution Hash Encoding. ArXiv Preprint ArXiv:2201.05989 (2022)
  * [35] Niemeyer, M., Mescheder, L., Oechsle, M., Geiger, A.: Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3504–3515 (2020)
  * [36] Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019). https://doi.org/10.1109/CVPR.2019.00025
  * [37] Paschalidou, D., Katharopoulos, A., Geiger, A., Fidler, S.: Neural Parts: Learning Expressive 3D Shape Abstractions With Invertible Neural Networks. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 3203–3214 (2021). https://doi.org/10.1109/CVPR46437.2021.00322
  * [38] Peng, S., Niemeyer, M., Mescheder, L., Pollefeys, M., Geiger, A.: Convolutional Occupancy Networks. In: European Conference on Computer Vision. pp. 523–540. Springer, Springer (2020). https://doi.org/10.1007/978
  * [39] Renderpeople: renderpeople.com (2022)
  * [40] Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2304–2314 (2019). https://doi.org/10.1109/ICCV.2019.00239
  * [41] Saragadam, V., Tan, J., Balakrishnan, G., Baraniuk, R.G., Veeraraghavan, A.: MINER: Multiscale Implicit Neural Representations (2022)
  * [42] Simeonov, A., Du, Y., Tagliasacchi, A., Tenenbaum, J.B., Rodriguez, A., Agrawal, P., Sitzmann, V.: Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation. ArXiv Preprint ArXiv:2112.05124 (2021)
  * [43] Sitzmann, V., Chan, E., Tucker, R., Snavely, N., Wetzstein, G.: MetaSDF: Meta-Learning Signed Distance Functions. Advances in Neural Information Processing Systems 33, 10136–10147 (2020)
  * [44] Sitzmann, V., Martel, J., Bergman, A., Lindell, D., Wetzstein, G.: Implicit Neural Representations With Periodic Activation Functions. Advances in Neural Information Processing Systems 33, 7462–7473 (2020). https://doi.org/10.1109/WACV51458.2022.00234
  * [45] Sitzmann, V., Rezchikov, S., Freeman, B., Tenenbaum, J., Durand, F.: Light Field Networks: Neural Scene Representations With Single-Evaluation Rendering. Advances in Neural Information Processing Systems 34 (2021)
  * [46] Sitzmann, V., Zollhöfer, M., Wetzstein, G.: Scene Representation Networks: Continuous 3d-Structure-Aware Neural Scene Representations. Advances in Neural Information Processing Systems 32 (2019). https://doi.org/10.5555/3454287.3454388
  * [47] Takikawa, T., Litalien, J., Yin, K., Kreis, K., Loop, C., Nowrouzezahrai, D., Jacobson, A., McGuire, M., Fidler, S.: Neural Geometric Level of Detail: Real-Time Rendering With Implicit 3D Shapes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11358–11367 (2021)
  * [48] Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., Ng, R.: Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. Advances in Neural Information Processing Systems 33, 7537–7547 (2020)
  * [49] Tang, D., Singh, S., Chou, P.A., Hane, C., Dou, M., Fanello, S., Taylor, J., Davidson, P., Guleryuz, O.G., Zhang, Y., et al.: Deep Implicit Volume Compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1293–1303 (2020). https://doi.org/10.1109/CVPR42600.2020.00137
  * [50] Tretschk, E., Tewari, A., Golyanik, V., Zollhöfer, M., Stoll, C., Theobalt, C.: Patchnets: Patch-Based Generalizable Deep Implicit 3D Shape Representations. In: European Conference on Computer Vision. pp. 293–309. Springer, Springer (2020)
  * [51] Turk, G., Levoy, M.: Zippered Polygon Meshes From Range Images. In: Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques. pp. 311–318 (1994). https://doi.org/10.1145/192161.192241
  * [52] Xu, Q., Wang, W., Ceylan, D., Mech, R., Neumann, U.: DISN: Deep Implicit Surface Network for High-Quality Single-View 3D Reconstruction. Advances in Neural Information Processing Systems 32 (2019). https://doi.org/10.5555/3454287.3454332
  * [53] Yang, G., Belongie, S., Hariharan, B., Koltun, V.: Geometry Processing With Neural Fields. In: Thirty-Fifth Conference on Neural Information Processing Systems (2021). https://doi.org/10.1145/3186564
  * [54] Yao, S., Yang, F., Cheng, Y., Mozerov, M.G.: 3D Shapes Local Geometry Codes Learning With SDF. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2110–2117 (2021). https://doi.org/10.1109/ICCVW54120.2021.00239
  * [55] Yariv, L., Gu, J., Kasten, Y., Lipman, Y.: Volume Rendering of Neural Implicit Surfaces. Advances in Neural Information Processing Systems 34 (2021). https://doi.org/10.1109/CVPR46437.2021.01120
  * [56] Zakharov, S., Kehl, W., Bhargava, A., Gaidon, A.: Autolabeling 3D Objects With Differentiable Rendering of SDF Shape Priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12224–12233 (2020). https://doi.org/10.1109/CVPR42600.2020.01224

##  6 Supplementary Material

###  6.1 Training Setup

####  6.1.1 Optimization.

In all of our experiments, we train the MLPs in PyTorch using the Adam
optimizer with the default parameters. We supervise the network based on the
mean squared error loss between reconstructed color and ground truth color. We
adopt the cosine annealing learning rate scheduler, with initial learning rate
as $10^{-4}$ and final learning rate as $10^{-7}$. We randomly shuffled all
the training samples and divided them into batches of size equal to 1,024. To
ensure reproducibility, we set the random seed as 0.
On a machine with NVIDIA GeForce RTX 2080 TI, training 10 epochs takes around
4 minutes for the SDF and OF condition, and 13 minutes for PRIF.

####  6.1.2 Network Architecture.

We set all networks to have 10 layers with residual connections, where the 8
intermediate layers having a dimension size of $512\times 512$. We apply layer
normalization after all MLP layers except the final one. When we further
extend the network to model color or background mask, we simply invoke a
second network with the same architecture, as network compression rate or
total parameter count is not the main focus of the current work.

###  6.2 Runtime Discussions

####  6.2.1 Rendering Speed.

The formulation of PRIF inherently allows us to bypass the inefficient multi-
sample issue of the previous methods that adopt level-set functional
representations. To more quantitatively measure the speed-up, we measure the
number of network evaluations (Queries) and the actual time to render a
$512\times 512$ depth image from a trained neural representation on the same
machine with an NVIDIA GeForce RTX 2080 Ti graphics card.

Method | Steps | Queries | Time (s)
---|---|---|---
DeepSDF + Sphere Tracing | 100 | 2,453,700 | 1.01070
DeepSDF + DIST Acceleration | 100 | 382,325 | 0.30539
PRIF | 1 | 262,144 | 0.00578
Table 3: Quantitative results on the cost to render a $512\times 512$ depth
image from trained neural shape representations. Queries refer to the number
of network evaluations, and Steps refer to the maximum sphere tracing steps
specified in the actual implementation. Steps = 100 is in line with the
original DIST implementation from Liu et al. [28].

We first consider a baseline method where a trained DeepSDF network is used to
perform naive sphere tracing. We then consider an improved baseline where the
DeepSDF-based sphere tracing is sped up by various techniques proposed by Liu
et al. [28]. Both baselines are adopted from the implementation of Liu et al.,
and correspond to two conditions (trivial and recursive) in their
implementation.

####  6.2.2 Meshing Speed

The overall meshing speed depends on network inference \+ surface
reconstruction. We examine the trade-off by decoding the same shape with PRIF
and SDF trained with networks of equivalent capacity. Our method spends 1.01s
at inference, then 2.65s to obtain the triangle mesh from MeshLab (Poisson
with normal estimation). The SDF-based network (similar to DeepSDF) spends
6.97s for inference, then 1.19s for Marching Cubes. We also note that many
practical scenarios do not require the meshing process, such as camera pose
registration and neural rendering.

###  6.3 More Ablations

####  6.3.1 Comparison to Plücker Coordinate.

We use the original Plücker coordinate for the shape generation task for the
Car category. Below are the results (with mean/median Chamfer distance):

Method | Car | Chair | Table | Plane | Lamp | Sofa
---|---|---|---|---|---|---
Plain Plücker | 2.397$|$0.547 | 2.603$|$0.531 | 5.054$|$0.554 | 0.766$|$0.199 | 4.838$|$1.094 | 1.738$|$0.305
Proposed | 1.961$|$0.347 | 0.982$|$0.267 | 4.532$|$0.315 | 0.389$|$0.125 | 3.276$|$0.534 | 1.236$|$0.222

Although the plain Plücker formulation works well for single shape
representation (Section 5 Table 3), it performs much worse in the generative
task. We find the network trained with Plücker coordinates often fails to
predict correct background masks, leading to many noisy points and worse
performance overall. Compared to the Plücker moment vector, our proposed
reference points is better geometrically related to the surface hitpoint,
which is our overarching goal of shape representation and rendering.

####  6.3.2 Varying Model complexity.

We examine the effect of varying the model capacity (by changing MLP layer
width) for the generation task on shapes from the Car category.

# Param (in Millions) | 2.96 | 2.23 | 1.58 | 1.35 | SDF (2.10)
---|---|---|---|---|---
Mean$|$Median CD | 1.961$|$0.347 | 2.026$|$0.364 | 2.058$|$0.368 | 2.055$|$0.386 | 2.315$|$0.495

Results show we can decrease the number of parameters and still achieve better
performance than SDF.

####  6.3.3 Different Noise Levels.

We perform additional denoising experiments (as Fig. 5 in the paper) and
present the impact of different noise levels on the shape representation
accuracy on the Car category (measured by Chamfer distance):

Noise Level | 0.01 | 0.02 | 0.04 | 0.08 | 0.1
---|---|---|---|---|---
Mean$|$Median CD | 1.929$|$0.368 | 1.931$|$0.428 | 2.005$|$0.590 | 2.471$|$0.951 | 2.842$|$1.123

###  6.4 Additional Results

Figure 11: More Single Shape from Different Angles. Results from Rows 1 to 4
correspond to Reference, OF, SDF, PRIF - Mesh.

Figure 12: More Single Shape from Different Angles. Results from Rows 1 to 4
correspond to Reference, OF, SDF, PRIF - Mesh.

Figure 13: More Single Shape from Different Angles. Results from Rows 1 to 4
correspond to Reference, OF, SDF, PRIF - Mesh.

Figure 14: More Single Shape from Different Angles. Results from Rows 1 to 4
correspond to Reference, OF, SDF, PRIF - Mesh.

Figure 15: More Single Shape from Different Angles. Results from Rows 1 to 4
correspond to Reference, OF, SDF, PRIF - Mesh.

Figure 16: More Generative Results from Different Categories. Results from
Rows 1 to 4 correspond to Reference, OF, SDF, PRIF - Mesh.

Figure 17: More Generative Results from Different Categories. Results from
Rows 1 to 4 correspond to Reference, OF, SDF, PRIF - Mesh.

Figure 18: More Generative Results from Different Categories. Results from
Rows 1 to 4 correspond to Reference, OF, SDF, PRIF - Mesh.

Figure 19: More Generative Results from Different Categories. Results from
Rows 1 to 4 correspond to Reference, OF, SDF, PRIF - Mesh.

Figure 20: More Generative Results from Different Categories. Results from
Rows 1 to 4 correspond to Reference, OF, SDF, PRIF - Mesh.

Figure 21: More Generative Results from Different Categories. Results from
Rows 1 to 4 correspond to Reference, OF, SDF, PRIF - Mesh.

Figure 22: More Color Rendering Results. Depth (left) and Color (right)
renderings from the same view point are shown.

Figure 23: More Color Rendering. Depth (left) and Color (right) renderings
from the same view point are shown.


# Primer AI’s Systems for Acronym Identification and Disambiguation

Nicholas Egan, John Bohannon
Primer AI
San Francisco, CA
{negan<EMAIL_ADDRESS>
###### Abstract

The prevalence of ambiguous acronyms make scientific documents harder to
understand for humans and machines alike, presenting a need for models that
can automatically identify acronyms in text and disambiguate their meaning. We
introduce new methods for acronym identification and disambiguation: our
acronym identification model projects learned token embeddings onto tag
predictions, and our acronym disambiguation model finds training examples with
similar sentence embeddings as test examples. Both of our systems achieve
significant performance gains over previously suggested methods, and perform
competitively on the SDU@AAAI-21 shared task leaderboard. Our models were
trained in part on new distantly-supervised datasets for these tasks which we
call AuxAI and AuxAD. We also identified a duplication conflict issue in the
SciAD dataset, and formed a deduplicated version of SciAD that we call SciAD-
dedupe. We publicly released all three of these datasets, and hope that they
help the community make further strides in scientific document understanding.

##  1 Introduction

Writers of scientific documents frequently utilize abbreviations as tools to
make unwieldy technical terms less verbose. These abbreviations often take the
form of acronyms or initialisms, which are abbreviations formed from the first
letters of words in the term. We refer to the abbreviated form as the “short
form” or “acronym,” and we refer to the full term as the “long form” or
“expansion.” The widespread usage of these abbreviations makes writing more
convenient for scientists, but poses a challenge to machines and non-expert
humans attempting to read scientific documents. This has led to an
accumulation of scientific jargon, and a need for AI tools to manage acronyms
and their expansions.

? (?) recently released two large datasets for acronym understanding in
scientific documents: the first is for the acronym identification task (AI),
and the second is for the acronym disambiguation task (AD). The goal of
acronym identification is to extract short and long form acronyms within a
sentence, and the goal of acronym disambiguation is to determine the expansion
of a particular acronym given sentence context.

##### Contributions

In this paper, we describe our systems for the AI and AD tasks, which improve
upon the models proposed by ? and perform competitively on the task
leaderboard (?). Our AI method projects learned token embeddings from a
transformer-based language model onto tag predictions, and our AD method finds
similar training examples for testing examples. We improved the performance of
our systems through the development of distantly-supervised auxiliary
datasets, which we are releasing to the public. Finally, we identified some
issues with the SciAD dataset, and propose a remedy for these issues that we
hope will make SciAD more useful as a tool for the NLP community. Our three
datasets are publicly available on our GitHub data
repository.111https://github.com/PrimerAI/sdu-data

##  2 Datasets

###  2.1 SciAI

The SciAI dataset (?) consists of 17,560 sentences annotated for acronym
identification, where each sentence token is tagged for short form and long
form acronym boundaries in BIO format. To construct this dataset, the authors
assembled a corpus of 6,786 papers from arXiv, identified candidate sentences
in these papers that likely contained acronyms, and hired Amazon Mechanical
Turk workers to gold label the sentences. The candidate sentences were
sentences containing consecutive (or near consecutive) word sequences for
which the concatenation of the first few characters from these words could
spell out another word in the document that consists of at least 50% capital
letters. When labeling, humans were instructed to find all short form acronyms
in the sentence, even if the acronym’s long form did not appear in the
sentence.

###  2.2 Auxiliary AI Data

To build on the training data provided by SciAI, we took a distantly
supervised approach to build a more noisy dataset we call AuxAI. We started by
scraping abbreviations with their expansions from Abbreviations.com within the
Academic & Science, Computing, and Internet categories. We then searched for
these terms on arXiv, finding paper abstracts for which the short and long
form both appear. We were able to identify 6497 terms across 274,149 abstracts
this way. After finding these abstracts, we searched our abstracts for any
other abbreviations that appear with their expanded form, and labeled them as
such. Additionally, since some acronyms like USA and DNA are common enough to
stand on their own in short form without including their expanded forms, we
compiled a list of “universal acronyms” which are acronyms in the SciAI
training dataset that have no long form in their sentence for at least 2
sentences. We found 1807 such acronyms, which were all marked in our token
labels.

Tag | SciAI Train | SciAI Dev | AuxAI
---|---|---|---
O | 85.12% | 84.78% | 88.01%
B-short | 5.34% | 5.69% | 3.59%
I-short | 0.31% | 0.35% | 0.01%
B-long | 2.94% | 2.92% | 2.55%
I-long | 6.29% | 6.27% | 5.84%
Table 1: Distribution of BIO tag labels among the training and development
partitions of the SciAI dataset, as well as the AuxAI dataset.

Overall, we were able to produce a dataset of consisting of 313,914 sentences.
Table 1 shows that the tag distribution within this dataset skews slightly
more towards the O tag, suggesting that our distantly supervised approach has
imperfect recall. Another issue with this dataset is that it contains around
3800 different short form acronyms, while the SciAI training set contains
around 6500 despite being smaller, suggesting that AuxAI captures less data
diversity. During training, we experimented with subsampling AuxAI data in
such a way that the ratio between unique terms and training examples matched
that of SciAI.

###  2.3 SciAD

The SciAD dataset (?) consists of 62,441 examples annotated for acronym
disambiguation, where each example consists of a sentence, a short form within
that sentence, and the correct long form that the short form refers to, which
may or may not appear in the sentence. This dataset was split into 50,034
training examples, 6189 development examples, and 6218 test examples. To
construct this dataset, the authors used SciAI to compile a dictionary of
acronyms consisting of short forms with their possible long forms. The long
forms were normalized through a combination of edit distance and human
verification. They then used the one-sense-per-discourse assumption to infer
that if a short form is mapped to a long form in the SciAI dataset, then they
can find other short forms within the document to use as ambiguous acronyms.

#### Duplicates in SciAD

When exploring the SciAD dataset, we noticed that it contained many
duplicates: while there are 62,441 total examples in the dataset, there are
only 42,945 examples with a unique (sentence, acronym) pair. For 12,672 of the
examples, there exists at least one other example with the exact same sentence
and acronym. 45.4% of the development examples contained a duplicate in the
training data, and 45.1% of the test examples contained a duplicate in the
training data. This overlap of data between train time and test time suggests
that SciAD is a biased measure of performance on the AD task.

To make matters more interesting, duplicate examples do not always have
duplicate labels: 10.5% of our duplicated examples in the train and
development datasets contain more than one long form label. 93.1% of the
development examples that have duplicates within the training dataset share a
label with at least one of the duplicates, and 10.8% of the development
examples that have duplicates within the training dataset have a conflicting
label with at least one of the duplicates. Since the AD task asks us to find
suitable long forms using features extracted from the sentence and short form
alone based on the training data, we claim that the accuracy of any model
should be upper bounded by 93.1% on the 45.4% of the development data that
contains a duplicate in the training data.

It is plausible that some of these label conflicts among duplicates is
genuine: two different papers could write the exact same sentence yet refer to
different acronym expansions. But we suspect that human error from the
annotators is the more likely explanation for most of these cases.

In order to remedy this problem, we propose that when one measures development
and test performance, they ignore examples that also exist in the training
data. In the experiments section, we report our model performance on both this
subset of the data as well as the full dataset. Additionally, we propose
removing training examples that are duplicates of other training examples for
model training. In order to resolve conflicting labels, one can use the more
common label among the duplicates. For convenience to other researchers, we
released our deduplicated version of the SciAD dataset which we call SciAD-
dedupe.222https://github.com/PrimerAI/sdu-data

###  2.4 Auxiliary AD Data

To build on the training data provided by SciAD, we took a distantly
supervised approach to build a more noisy dataset we call AuxAD. We queried
arXiv abstracts for acronyms found in the SciAD dictionary, and collected
56,874 such abstracts. We then assumed that if a short and long form from the
dictionary both appeared within a sentence, then the short form can be
resolved to the long form. Using the one-sense-per-discourse assumption, we
found other sentences within the document that contained the short form and
assumed that these short forms also corresponded to the same long form. This
resulted in a dataset of 112,788 examples, which we release in our same data
repository.

While this AuxAD dataset contains more examples than SciAD, it is less
diverse: both datasets started with the same dictionary of 2308 terms, but the
SciAD training dataset contains 2152 unique terms and the AuxAD dataset
contains 1268 unique terms. We suspect that this is because SciAD used full
arXiv documents while AuxAD relied on arXiv abstracts, and thus had a harder
time finding certain terms from the dictionary.

##  3 Methods

###  3.1 Acronym Identification

Our model architecture consisted of using a transformer-based language model
(?) to embed the input sentence tokens, followed by a linear projection onto
logits for each BIO tag, an approach mirroring our model for Named Entity
Recognition (?). We started by joining together the word-tokenized input
sentence, and re-tokenizing the sentence with SentencePiece byte-pair encoding
(?) to get $L$ tokens. These $L$ tokens are embedded with the XLNet language
model (?) to get an $\mathbb{R}^{H}$ embedding for each token, where $H$ is
768 for XLNet-base and 1024 for XLNet-large. We run these embeddings through a
linear layer to get $T$ tag logits per token, where $T$ is the number of BIO
tags (in this case 5). We use the tag with the highest logit per token as the
predicted tag, with the label of the first byte-pair encoded token within a
word being used for the word.

##### Model Training

Training was performed on both the XLNet encoder and the linear projection
weight matrix with cross-entropy loss on output logits and an AdamW optimizer.
Training hyperparameters included:

  * •

Pretrain on AuxAI then finetune on SciAI, or just train on SciAI

  * •

The subset of AuxAI to use when finetuning

  * •

XLNet model size

  * •

Whether or not to down-weigh the $O$ tag

  * •

Learning rate

We formed an ensemble of these XLNet models trained with different
hyperparameter configurations, and averaged together their predicted logits
during inference time. After picking the highest scoring tag per token, we
cleaned up the predictions such that $I$ tags could not follow $O$ tags to get
our final predictions.

###  3.2 Acronym Disambiguation

? modeled AD as a classification problem: given a sentence and a short-form
acronym within that sentence, they used a classifier to predict the acronym’s
expansion. We instead view it as an information retrieval problem: given a
test sentence containing an acronym, we want to find the most similar training
sentence and use its label. The intuition behind this approach is that
contextual clues within a sentence can determine the subfield of research that
the paper falls into. By computing the similarity between two sentences, we
could perhaps identify if they are within the same research field based on how
much their semantics align. For instance, a sentence talking about “CNN” would
likely include either several machine learning terms or several news terms. We
can compare our sentence to several others in the training dataset, and if the
dataset is sufficiently comprehensive, we should be able to find a sentence
semantically similar to the sentence in question.

More specifically, we start by computing a sentence embedding for every
example in the datasets. To infer a label for a given test example, we compute
the cosine similarity between its embedding and the embedding of every
sentence in the training dataset, pick the training sentence with highest
cosine similarity, and use its label. We were able to squeeze out a small
performance boost by additionally checking to see if any possible expansion
for the acronym appears within the sentence itself, and using that expansion
if we find it.

Measuring the utility of this approach is complicated by the fact that the
dataset contains many duplicate sentences across testing and training
datasets, and sometimes their labels conflict. In cases where multiple
duplicate sentences were found, we used the label that was more common overall
in the training dataset.

##### Model Training

To train an embedding model for this task, we constructed datasets of sentence
pairs from SciAD and AuxAD, where the sentence pairs share a short form
acronym and are labeled as having the same long form or a different long form,
with a balanced number of positive and negative pairs. We used various
transformer-based language models as encoders, and trained these language
models as Twin Networks (also known as Siamese Networks) (?): sentence
embeddings $(e_{1},e_{2})$ were computed for sentence pair $(s_{1},s_{2})$ by
running each sentence individually through the same encoder. The cosine
similarity between the sentences was computed as

$\cos(e_{1},e_{2})=\frac{e_{1}^{T}e_{2}}{||e_{1}||\cdot||e_{2}||}$

The encoder weights were optimized through mean squared error loss for the
sentence pairs representing training examples:

$\mathcal{L}_{MSE}(\mathcal{D})=\frac{1}{n}\sum_{(s_{1},s_{2},y)\in\mathcal{D}}(y-\cos(E(s_{1}),E(s_{2})))^{2}$

where $\mathcal{D}$ is our dataset of $n$ training examples, $E$ is our
transformer embedding model, and $y$ is our desired similarity score, which
was 1 if the sentences shared a long form and 0 otherwise. Training our
encoder in this way teaches it to learn an embedding space for which sentences
containing acronyms with the same meaning will have higher cosine similarity
than sentences containing acronyms with different meanings.

##### Pretrained Models

In addition to our trained model, we also tested the other embedding methods
of SIF (?) and several pretrained models from sentence transformers (?):

  * •

XLM (?) trained for paraphrase detection

  * •

DistilRoBERTa (?) trained for paraphrase detection

  * •

DistilRoBERTa trained for information retrieval on the MS MARCO dataset (?)

  * •

DistilRoBERTa trained for Quora question similarity

Our final system was an ensemble of these models plus some trained models,
where cosine similarity scores were averaged across models in the ensemble.

##  4 Acronym Identification Experiments

###  4.1 Model Building

Our model was implemented using our existing codebase for Named Entity
Recognition, which was based on PyTorch Transformers (?). Each XLNet model
took between 10 and 60 minutes to train on a single NVIDIA V100 GPU, depending
on hyperparameters like the number of epochs and training dataset. Inference
took 3ms per example when using a batch size of 16.

###  4.2 Performance

System | F1 | Precision | Recall
---|---|---|---
Baseline | 85.46 | 93.22 | 78.90
LSTM-CRF* | 86.55 | 86.96 | 86.16
XLNet, SciAI | 92.17 | 91.62 | 92.72
XLNet, AuxAI | 66.96 | 86.10 | 54.78
XLNet, AuxAI $\rightarrow$ SciAI | 93.14 | 93.23 | 93.04
XLNet Ensemble | 93.63 | 93.99 | 93.28
Table 2: Acronym identification performance of various models on the SciAI
dataset. Results for the LSTM-CRF model were taken from ? which used the test
dataset, while the other scores are on the development dataset. Our models are
below the line, with “$\rightarrow$” denoting finetuning. All performance
metrics are macro-averaged between short and long forms.

Performance results are shown in table 2. In this table, we compare our
methods to a rule based baseline (?) and the LSTM-CRF model proposed by ?. All
scores are computed on the development dataset due to fact that test dataset
labels are not yet publicly available, except for the LSTM-CRF model where
scores are taken from their paper. It is clear that while a model trained on
just our AuxAI dataset performs poorly, pretraining on AuxAI then finetuning
on SciAI results in a measurable boost in performance. Our final ensemble
method consisted of 15 different XLNet models trained with different
hyperparameters, and achieved an F1 score of 92.60 on the test set.

##### Error Analysis

We performed a small-scale error analysis by looking at a random sample of 50
mistakes made by the ensemble on the SciAI development dataset. Of those
mistakes, 18 were genuine mistakes made by the model, 24 were errors made by
the human annotators, and 8 were too ambiguous for us to tell. Both model and
human mistakes were most commonly the result of failing to extract an acronym
that should have been extracted, representing 39 of the errors: the fact that
humans frequently missed acronyms within the data likely led to trained models
being overly conservative. 7 of the errors came from a misalignment between
the true and predicted boundaries of acronyms, and only 1 error came from
incorrectly extracting a non-acronym.

##  5 Acronym Disambiguation Experiments

###  5.1 Model Building

Our embedding models were all based on the sentence-transformers library (?),
with the exception of SIF, for which we used fastText (?). Our final ensemble
consisted of the following embedding models:

  * •

SIF

  * •

XLM paraphrase

  * •

DistilRoBERTa paraphrase

  * •

DistilRoBERTa MS Marco

  * •

DistilBERT Quora

  * •

RoBERTa SciAD

  * •

XLM paraphrase finetuned on AuxAD

The last two of these methods were models that we trained as Twin Networks.
Training our Twin Network transformer models took around 20 minutes on an
NVIDIA V100 GPU depending on what data was used and the number of training
epochs. Evaluation consisted of embedding all of the sentences in the training
and testing data, which took around 2 minutes per model on a V100, and
computing distances between training and testing data, which took around a
minute on 16 CPUs for the whole ensemble.

When predicting labels for the development dataset, we used the SciAD training
dataset for finding matches, and when predicting labels for the test dataset,
we merged together the training and development datasets from SciAD. We
experimented with using the AuxAD as well as the SciAD datasets at query time,
but found that this led to a slight decrease in performance. Only 12% of
SciAD-dedupe development examples had a closer match in AuxAD than SciAD
training, despite AuxAD being a larger dataset, which can largely be explained
by the fact that the AuxAD dataset contained fewer terms. Within the small
proportion of AuxAD examples that are used, we tend to have less accurate
predictions, with an accuracy of 87% on SciAD-dedupe versus an accuracy of 96%
that we get on the chosen AuxAD examples.

###  5.2 Performance

System | SciAD | SciAD-dedupe
---|---|---
Baseline | 59.73 | 59.97
GAD* | 81.90 | -
SIF | 88.11 | 89.13
XLM paraphrase | 89.42 | 90.89
DistilRoBERTa paraphrase | 89.20 | 90.56
DistilRoBERTa MS Marco | 88.48 | 89.78
DistilBERT Quora | 86.09 | 86.34
RoBERTa SciAD | 88.18 | 89.46
XLM paraphrase $\rightarrow$ AuxAD | 83.75 | 83.04
Ensemble | 91.22 | 93.15
Table 3: Acronym disambiguation performance of various systems on the SciAD
dataset. Results for the GAD model were taken from ? which used the test
dataset, while the other scores were computed for the development dataset.
Models below the line are the methods we tested, which represent a combination
of pretrained sentence transformers and models we trained ourselves, with
“$\rightarrow$” denoting finetuning.

Table 3 shows the macro-averaged F1 scores for each of the individual
embedding methods, the embedding ensemble, the GAD classifier proposed in ?,
and the baseline of using the most frequent expansion for an acronym.
Performance is shown for the SciAD development set, as well as the development
set of SciAD-dedupe. The exception is GAD, for which we include the
performance on the test dataset reported in ?. We can see that the ensemble
clearly outperforms the rest of the models, including every individual
embedding model it is comprised of. The individual embedding models perform
similarly, except for the finetuned XLM paraphrase model. Despite the poor
performance from this model, we found that it was valuable to include as a
member of the ensemble. Our ensemble achieved an F1 score of 91.58 on the test
dataset.

What is also interesting to see is that the systems tend to perform better on
the SciAD-dedupe than SciAD, which is counter-intuitive considering the fact
that leaking training data into the testing data should theoretically drive up
performance scores. To investigate this, we extracted the subset of the SciAD
development dataset that was duplicated from the training dataset, and
measured the performance of a system that uses the most frequent long form of
training examples with the same duplicated tokens. This method achieves an F1
score of 89.41, which is surprisingly lower than the F1 score of our models on
the deduplicated data. This subset of the development dataset that repeats
sentences from the training dataset thus seems to be quite noisy.

Figure 1: Similarity score distributions for predictions on SciAD by our
ensemble. Each score represents the cosine similarity between an example in
the development dataset and its closest example in the training dataset. The
blue distribution is for examples that were judged to be correct, and the
orange distribution is for examples that were judged to be incorrect. Both
distributions were normalized.

Figure 1 shows the distribution of cosine similarity scores between
development examples and the testing example inferred to be most similar
within the training dataset for our ensemble on SciAD. The distribution of
scores for correct examples and incorrect examples are shown and normalized
separately. We can see visually that $p(M|\neg C)>p(M|C)$ where $M$ indicates
that our development example found a perfect match in the training dataset and
$C$ indicates that our predicted expansion is correct. We can also see that if
we ignore the perfect matches, correct predictions tend to have higher
similarity scores than incorrect predictions, suggesting that our model can
trade off recall for boosts in precision by using the similarity scores as a
threshold.

##### Error Analysis

We performed a small-scale error analysis by looking at a random sample of 50
mistakes made by the ensemble on the SciAD development dataset. Of those
mistakes, 18 were genuine mistakes made by the model, and 32 were mistakes
made by human annotators. 10 of the model mistakes came from semantically
similar sentences across development and training having different labels,
which highlights a limitation of this approach. 8 of the model mistakes were
from the lack of similar training examples for a given development example,
which could be potentially fixed given a larger training corpus. 30 of the
human mistakes came from conflicting labels in duplicate examples, and the
remaining 2 of the human errors came from mistakes during canonicalization of
long forms, such as “sum capacity” and “sum capacities” both existing in the
acronym dictionary.

##  6 Conclusion

In conclusion, we have developed new neural models for acronym identification
and disambiguation. Our acronym identification model uses a transformer
followed by linear projection, and our acronym disambiguation model finds
similar examples with embeddings learned from Twin Networks. Both models
benefited from ensembling, and both models achieve significant performance
gains over the models originally proposed by ?

We introduced new datasets for acronym identification and disambiguation,
AuxAI and AuxAD, which were labeled through distant supervision. We also
identified a duplication issue in the SciAD dataset, and formed a deduplicated
version of this dataset that we call SciAD-dedupe. We released all three of
these datasets and we hope that they serve as useful tools for the NLP
community.

## References

  * [Arora, Liang, and Ma] Arora, S.; Liang, Y.; and Ma, T.  2017\.  A simple but tough-to-beat baseline for sentence embeddings.
  * [Bajaj et al.] Bajaj, P.; Campos, D.; Craswell, N.; Deng, L.; Gao, J.; Liu, X.; Majumder, R.; McNamara, A.; Mitra, B.; Nguyen, T.; et al.  2016\.  Ms marco: A human generated machine reading comprehension dataset.  arXiv preprint arXiv:1611.09268.
  * [Chicco] Chicco, D.  2021\.  Siamese Neural Networks: An Overview.  New York, NY: Springer US.  73–94.
  * [Joulin et al.] Joulin, A.; Grave, E.; Bojanowski, P.; Douze, M.; Jégou, H.; and Mikolov, T.  2016\.  Fasttext.zip: Compressing text classification models.  arXiv preprint arXiv:1612.03651.
  * [Kudo and Richardson] Kudo, T., and Richardson, J.  2018\.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 66–71.  Brussels, Belgium: Association for Computational Linguistics.
  * [Lample and Conneau] Lample, G., and Conneau, A.  2019\.  Cross-lingual language model pretraining.  Advances in Neural Information Processing Systems (NeurIPS).
  * [Primer AI] Primer AI.  2019\.  A new state of the art for named entity recognition.  https://primer.ai/blog/a-new-state-of-the-art-for-named-entity-recognition/.
  * [Reimers and Gurevych] Reimers, N., and Gurevych, I.  2019\.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.  Association for Computational Linguistics.
  * [Sanh et al.] Sanh, V.; Debut, L.; Chaumond, J.; and Wolf, T.  2020\.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.
  * [Schwartz and Hearst] Schwartz, A., and Hearst, M.  2003\.  A simple algorithm for identifying abbreviation definitions in biomedical text.  Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing 4:451–62.
  * [Vaswani et al.] Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L. u.; and Polosukhin, I.  2017\.  Attention is all you need.  In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., Advances in Neural Information Processing Systems, volume 30, 5998–6008.  Curran Associates, Inc.
  * [Veyseh et al.] Veyseh, A. P. B.; Dernoncourt, F.; Nguyen, T. H.; Chang, W.; and Celi, L. A.  2020a.  Acronym identification and disambiguation shared tasks for scientific document understanding.  In Proceedings of the AAAI-21 Workshop on Scientific Document Understanding.
  * [Veyseh et al.] Veyseh, A. P. B.; Dernoncourt, F.; Tran, Q. H.; and Nguyen, T. H.  2020b.  What Does This Acronym Mean? Introducing a New Dataset for Acronym Identification and Disambiguation.  In Proceedings of COLING.
  * [Wolf et al.] Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Scao, T. L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. M.  2020\.  Transformers: State-of-the-art natural language processing.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 38–45.  Online: Association for Computational Linguistics.
  * [Yang et al.] Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R.; and Le, Q. V.  2019\.  Xlnet: Generalized autoregressive pretraining for language understanding.  In NeurIPS.