# TriVol: Point Cloud Rendering via Triple Volumes

Tao Hu 1∗ Xiaogang Xu1∗ Ruihang Chu 1 Jiaya Jia1,2
1 The Chinese University of Hong Kong 2 SmartMore
<EMAIL_ADDRESS>
###### Abstract

Existing learning-based methods for point cloud rendering adopt various 3D
representations and feature querying mechanisms to alleviate the sparsity
problem of point clouds. However, artifacts still appear in rendered images,
due to the challenges in extracting continuous and discriminative 3D features
from point clouds. In this paper, we present a dense while lightweight 3D
representation, named TriVol, that can be combined with NeRF to render photo-
realistic images from point clouds. Our TriVol consists of triple slim
volumes, each of which is encoded from the point cloud. TriVol has two
advantages. First, it fuses respective fields at different scales and thus
extracts local and non-local features for discriminative representation.
Second, since the volume size is greatly reduced, our 3D decoder can be
efficiently inferred, allowing us to increase the resolution of the 3D space
to render more point details. Extensive experiments on different benchmarks
with varying kinds of scenes/objects demonstrate our framework’s effectiveness
compared with current approaches. Moreover, our framework has excellent
generalization ability to render a category of scenes/objects without fine-
tuning. The source code is available at https://github.com/dvlab-
research/TriVol.git.

Figure 1:  Given the colored point cloud of a category-specific scene or
object, our TriVol can render photo-realistic images.  11footnotetext: Equal
Contribution.

##  1 Introduction

Photo-realistic point cloud rendering (without hole artifacts and with clear
details) approaches can be employed for a variety of real-world applications,
_e.g_., the visualization of automatic drive [23, 6, 8, 29], digital humans
[27, 1, 19], and simulated navigation [43, 11, 5]. Traditional point cloud
renderers [36] adopt graphics-based methods, which do not require any
learning-based models. They project existing points as image pixels by
rasterization and composition. However, due to the complex surface materials
in real-world scenes and the limited precision of the 3D scanners, there are a
large number of missing points in the input point cloud, leading to vacant and
blurred image areas inevitably as illustrated in Fig. 1.

In recent years, learning-based approaches [1, 35, 45, 49, 10] have been
proposed to alleviate the rendering problem in graphics-based methods. They
use a variety of querying strategies in the point cloud to obtain continuous
3D features for rendering, _e.g_., the ball querying employed in the
PointNet++ [34] and the KNN-based querying in Point-NeRF [45]. However, if the
queried position is far away from its nearest points, the feature extraction
process will usually fail and have generalization concerns. To guarantee
accurate rendering, two groups of frameworks are further proposed. The first
group [38, 1, 35] projects the features of all points into the 2D plane and
then trains 2D neural networks, like UNet [37], to restore and sharpen the
images. However, since such a 2D operation is individual for different views,
the rendering outcomes among nearby views are inconsistent [40, 17], _i.e_.,
the appearance of the same object might be distinct under different views. To
overcome the artifact, physical-based volume rendering [24] in Neural Radiance
Fields (NeRF) [26] without 2D neural rendering is an elegant solution [40].
The second group [10, 47] applies a 3D Convolutional Network (ConvNet) to
extract a dense 3D feature volume, then acquires the 3D feature of arbitrary
point by trilinear interpolation for subsequent volume rendering.
Nevertheless, conducting such a dense 3D network is too heavy for high-
resolution 3D representation, limiting their practical application. In
summary, to the best of our knowledge, there is currently no lightweight point
cloud renderer whose results are simultaneously view-consistent and photo-
realistic.

In this paper, we propose a novel 3D representation, named TriVol, to solve
the above issues. Our TriVol is composed of three slender volumes which can be
efficiently encoded from the point cloud. Compared with dense grid voxels, the
computation on TriVol is efficient and the respective fields are enlarged,
allowing the 3D representation with high resolution and multi-scale features.
Therefore, TriVol can provide discriminative and continuous features for all
points in the 3D space. By combining TriVol with NeRF [26], the point cloud
rendering results show a significant quantitative and qualitative improvement
compared with the state-of-the-art (SOTA) methods.

In our framework, we develop an efficient encoder to transform the point cloud
into the Initial TriVol and then adopt a decoder to extract the Feature
TriVol. Although the encoder can be implemented with the conventional point-
based backbones [33, 34], we design a simple but effective grouping strategy
that does not need extra neural models. The principle is first voxelizing the
point cloud into grid voxels and then re-organizing the voxels on each of
three axes individually, which is empirically proven to be a better method. As
for the decoder, it can extract the feature representation for arbitrary 3D
points. Hence, we utilize three independent 3D ConvNet to transfer each volume
into dense feature representations. Due to the slender shape of the volumes,
the computation of the 3D ConvNet is reduced. Also, the 3D ConvNet can capture
more non-local information in the grouped axis via a larger receptive field,
and extract local features in the other two directions.

With the acquired dense Feature TriVol, the feature of any 3D points can be
queried via trilinear interpolation. By combining the queried features with
the standard NeRF [26] pipeline, the photo-realistic results are achieved.
Extensive experiments are conducted on three representative public datasets
(including both datasets of scene [9] and object [4, 12]) level, proving our
framework’s superiority over recent methods. In addition, benefiting from the
discriminative and continuous feature volume in TriVol, our framework has a
remarkable generalization ability to render unseen scenes or objects of the
same category, when there is no further fine-tuning process.

In conclusion, our contributions are three-fold.

  * •

We propose a dense yet efficient 3D representation called TriVol for point
cloud rendering. It is formed by three slim feature volumes and can be
efficiently transformed from the point cloud.

  * •

We propose an effective encoder-decoder framework for TriVol representation to
achieve photo-realistic and view-consistent rendering from the point cloud.

  * •

Extensive experiments are conducted on various benchmarks, showing the
advantages of our framework over current works in terms of rendering quality.

##  2 Related Works

###  2.1 3D Representation

3D representation is very important when analyzing and understanding 3D
objects or scenes. There are several important 3D representations, including
point cloud, dense voxels [10], sparse voxels [7], Multi-Plane Images (MPI),
triple-plane (triplane) [40, 3, 28, 13], multi-plane [41, 25, 39], and NeRF
[26], designed from different tasks. A point cloud is a set of discrete data
points in space representing a 3D object or scene. Each point location has a
coordinate value and could further contain the color. The point cloud is an
efficient 3D representation that is usually captured from a real-world scanner
or obtained via Multi-view Stereo (MVS) [14]. Each voxel in the dense voxels
represents a value on a regular grid in the 3D space. By using interpolation,
the continuous 3D features of all 3D positions can be obtained. The sparse
voxels [15, 16, 7] are the compressive representation of the dense voxels
since only parts of the voxels have valid values. The triplane representation
[40, 3] is also the simplification of dense voxels, obtained by projecting the
3D voxels to three orthogonal planes. The MPI [41, 25, 10] represents the
target scene as a set of RGB and alpha planes within a reference view frustum.
Moreover, NeRF [26] is a recently proposed implicit 3D representation, which
can represent the feature of any input 3D coordinate with an MLP. The MLP maps
the continuous input 3D coordinate to the geometry and appearance of the scene
at that location. We propose TriVol as a new 3D representation of the point
cloud and demonstrate its advantages in point cloud rendering.

###  2.2 Point-based Rendering

Point cloud rendering can be implemented with graphics- and learning-based
approaches. The points are projected to the 2D plane via rasterization and
composition in the graphics-based algorithms [36]. The learning-based methods
[19, 20, 7, 45] design various strategies to compensate the missing
information from the sparse point cloud. For example, ME [7] first conducts
sparse ConvNet to extract features for existing points, then computes the
features of arbitrary 3D points by ball querying in the local space.
Obviously, most of the points in the whole 3D space have no meaningful
features. Point-NeRF [45] makes use of multi-view images to enhance the
features of the input point cloud, formulating the sparse 3D feature volume
and then querying any point features by $K$ Nearest Neighbors (KNN) [34].
Also, quite a few learning-based methods [44, 32, 31, 48, 38, 1, 35] project
the point cloud onto the 2D plane and utilize the 2D networks to recover the
hole artifacts caused by the point cloud’s discrete property. For instance,
NPBG [1] renders the point cloud with learned neural features in multiple
scales and sets a 2D UNet for refinement. Furthermore, several approaches
construct 3D feature volumes for rendering [10], _e.g_., NPCR [10] uses 3D
ConvNet to obtain 3D volumes from point clouds and produce multiple depth
layers to synthesize novel views.

Figure 2: Overview of the proposed TriVol for point cloud rendering. (a)
Encoding: Input point cloud is encoded to our Initial TriVol along $x$, $y$,
and $z$ axis; (b) Decoding: Each volume is decoded to dense feature volume via
a unique 3D UNet; (c) Feature Querying: Any point’s feature is queried by the
trilinear interpolation in the Feature TriVol; (d) Rendering: We combine the
queried point feature with NeRF to render the final image.

##  3 Approach

We aim to train a category-specific point renderer $\mathcal{R}$ to directly
generate photo-realistic images $I$ (the image height and width are denoted as
$H$ and $W$) from the colored point cloud $P$, given camera parameters
(intrinsic parameter $K$ and extrinsic parameters $R$ and $t$). When rendering
novel point clouds of the same category, no fine-tuning process is required.
The rendering process can be represented as

$I=\mathcal{R}(P|R,t,K),$ (1)

where $P$ is usually obtained from MVS [14, 2], LiDAR scanners [9], or sampled
from synthesized mesh models. In this section, we first encode the point cloud
as the proposed TriVol, then utilize three 3D UNet to decode it into the
feature representation. Finally, we combine NeRF [26] by querying point
features from TriVol at sampled locations to render the final images. An
overview of our method is illustrated in Fig. 2.

###  3.1 TriVol Representation

Grid Voxels. To begin with, we voxelize sparse point cloud $P$ into grid
voxels $V$ with shape $\mathbb{R}^{C\times S\times S\times S}$, where $S$ is
the resolution of each axis, and $C$ is the number of feature channels. Since
$V$ is a sparse tensor, directly querying within $V$ will only get meaningless
values for most of the 3D locations, leading to the vacant areas in the
rendered images. Therefore, the critical step is transforming the sparse $V$
into a dense and discriminative representation. One approach is employing a 3D
encoder-decoder, _e.g_., 3D UNet. Nevertheless, such a scheme is not efficient
to represent a high-resolution space and render fine-grained details. The
reason is two-fold: 1) conducting 3D ConvNet on $V$ requires a lot of
computations and memory resources, leading to a small value of $S$ [10]; 2)
the sparsity of $V$ impedes the feature propagation since regular 3D ConvNet
only has a small kernel size and receptive field.

From grid voxels to TriVol. To overcome above two issues of $V$, we propose
TriVol, including ${V_{x},V_{y},V_{z}}$, as a novel 3D representation. As
illustrated in Fig. 2, each item in TriVol is a thin volume whose resolution
of one axis is obviously smaller than $S$, and the others are the same as $S$.
As a consequence, the number of total voxels is reduced for lightweight
computations. Note that our TriVol is different from triple-plane
representations that are employed in existing works, _e.g_., ConvOnet [30] and
EpiGRAF [40]. Their point features are projected to three standard planes,
thus much 3D information might be lost [30].

####  3.1.1 Encoder for Initial TriVol

We first encode the input point cloud into the Initial TriVol
($\\{\bar{V}_{x},\bar{V}_{y},\bar{V}_{z}\\}$). This process can be completed
by existing point-cloud-based networks, such as PointNet[33], PointNet++ [34],
and Dense/Sparse ConvNet [7, 15]. Nevertheless, these networks bring an
additional and heavy computation burden. Instead, we design an efficient
strategy without an explicit learning model.

The main step in our encoder is the x-grouping, y-grouping, and z-grouping
along different axes. The procedure can be denoted as
$\\{\bar{V}_{x},\bar{V}_{y},\bar{V}_{z}\\}=E(V)$, where
$E=\\{E_{x},E_{y},E_{z}\\}$. Specifically, to obtain the slim volumes of the
$x$-axis, we first divide $V$ into $G\times S\times S$ groups along the
$x$-axis, thus each group contains $N=S/G$ voxels. Then we concatenate all $N$
voxels in each group as one new feature voxel to obtain
$\bar{V}_{x}\in\mathbb{R}^{(C\cdot N)\times G\times S\times S}$, where $C\cdot
N$ is the number of feature channels for each new voxel. $\bar{V}_{y}$ and
$\bar{V}_{z}$ are encoded by the similar grouping method but along $y$ and $z$
axis, respectively. Therefore, $E$ can be formulated as

$\displaystyle\bar{V}_{x}=E_{x}(V)\in\mathbb{R}^{(C\cdot N)\times G\times
S\times S}$ (2) $\displaystyle\bar{V}_{y}=E_{y}(V)\in\mathbb{R}^{(C\cdot
N)\times S\times G\times S}$
$\displaystyle\bar{V}_{z}=E_{z}(V)\in\mathbb{R}^{(C\cdot N)\times S\times
S\times G}$

Our encoder is simple and introduces two benefits. Firstly, we can set the
different sizes of $G$ and $S$ to balance the performance and computation.
When $G\ll S$, huge computing resources are not required, compared with grid
voxels $V$. Thus, we can increase the resolution $S$ to model more point cloud
details. Secondly, since the voxels in the same group share the identical
receptive field, the receptive field on the grouped axis is amplified $N$
times, allowing the subsequent decoder to capture global features on the
corresponding axis without using a large kernel size. For instance, in volume
$\bar{V}_{x}$, we can extract non-local features on the $x$-axis and local
features on the $y$ and $z$ axis.

####  3.1.2 Decoder for Feature TriVol

After obtaining the Initial TriVol $\\{\bar{V}_{x},\bar{V}_{y},\bar{V}_{z}\\}$
from the encoder, we utilize 3D ConvNet to decode them as the Feature TriVol
$\\{V_{x},V_{y},V_{z}\\}$. Our Feature TriVol decoder consists of three 3D
UNet [37] modules $D=\\{D_{x},D_{y},D_{z}\\}$. Each 3D UNet can acquire non-
local features on the grouped axis with the amplifying receptive field and can
extract local features on the ungrouped two axes that preserve the standard
local receptive field. The decoding procedure can be represented as

$V_{x}=D_{x}(\bar{V}_{x}),\;V_{y}=D_{y}(\bar{V}_{y}),\;V_{z}=D_{z}(\bar{V}_{z}),$
(3)

where $V_{x}\in\mathbb{R}^{F\times G\times S\times S}$,
$V_{y}\in\mathbb{R}^{F\times S\times G\times S}$, and
$V_{z}\in\mathbb{R}^{F\times S\times S\times G}$, and $F$ denotes channel
number of Feature TriVol. Although three 3D UNets are required, the small
number of $G$ still makes it possible for us to set a large resolution $S$
without increasing computing resources (verified in Sec. 4), resulting in
realistic images with rich details.

###  3.2 TriVol Rendering

The encoder and decoder modules have transformed the sparse point cloud into a
dense and continuous Feature TriVol. Therefore, the feature of any 3D location
can be directly queried by the trilinear interpolation in
$\\{V_{x},V_{y},V_{z}\\}$. Finally, the rendering images can be obtained from
the point cloud by following feature querying and volume rendering pipeline of
NeRF [26].

####  3.2.1 Feature Querying

The feature querying consists of point sampling along the casting ray and
feature interpolation in the TriVol.

Point sampling. Given the camera parameters $\\{R,t,K\\}$, we can calculate a
random ray with camera center $\textbf{r}_{o}\in\mathbb{R}^{3}$ and normalized
direction $\textbf{r}_{d}\in\mathbb{R}^{3}$, we adopt the same coarse-to-fine
sampling strategy as NeRF [26] to collect the queried points
$\textbf{x}\in\mathbb{R}^{3}$ along the ray, as

$\textbf{x}=\textbf{r}_{o}+z\cdot\textbf{r}_{d},\quad z\in[z_{n},z_{f}],$ (4)

where $z_{n}$, $z_{f}$ are the near and far depths of the ray.

Querying. For Feature TriVol $\\{V_{x},V_{y},V_{z}\\}$ and a queried location
x, we first utilize trilinear interpolation to calculate 3 feature vectors:
$V_{x}(\textbf{x})\in\mathbb{R}^{F}$, $V_{y}(\textbf{x})\in\mathbb{R}^{F}$,
and $V_{z}(\textbf{x})\in\mathbb{R}^{F}$ as shown in Fig. 2. Then, we
concatenate them as the final feature $F(\textbf{x})$ for location x, as

$F(\textbf{x})=V_{x}(\textbf{x})\oplus V_{y}(\textbf{x})\oplus
V_{z}(\textbf{x}),$ (5)

where $\oplus$ is the concatenation operation.

####  3.2.2 Volume Rendering

Implicit mapping. For the queried feature of all points on the ray, we set a
Multi-Layer Perceptron (MLP) as an implicit function $g$ to map interpolated
feature $F(\textbf{x})$ to their densities $\sigma\in\mathbb{R}_{+}$ and
colors $c\in\mathbb{R}^{3}$, with the view direction $\textbf{r}_{d}$ as the
condition, as

$\sigma,\textbf{c}=g(F(\textbf{x}),\textbf{r}_{d}).$ (6)

Methods 3D Representation Feature Extraction Feature Query Rendering NPBG++
[35] Point Cloud 2D UNet - Graphics & CNN Dense 3D ConvNet Grid Voxels 3D UNet
Trilinear Interpolation NeRF Sparse 3D ConvNet Sparse Voxels 3D Sparse UNet
Ball Query NeRF Point-NeRF [46] Point Cloud MLP KNN NeRF Ours Triple Volumes
(TriVol) 3D UNet Trilinear Interpolation NeRF

Table 1: Compare different point cloud renderers in 3D representation, feature
extraction, feature query, and rendering strategy.

Rendering. The final color of each pixel $\hat{\textbf{c}}$ can be computed by
accumulating the radiance on the ray through the pixel using volume density
[24]. Suppose there are $M$ points on the ray, such a volume rendering can be
described as

$\displaystyle\begin{aligned}
&\hat{\textbf{c}}=\sum_{i=1}^{M}T_{i}\alpha_{i}\textbf{c}_{i},\\\
&\alpha_{i}=1-\textbf{ exp}(-\sigma_{i}\delta_{i}),\\\
&T_{i}=\textbf{exp}(-\sum_{j=1}^{i-1}\sigma_{j}\delta_{j}),\end{aligned}$ (7)

where $T_{i}$ represents volume transmittance, $\delta_{j}$ is the distance
between neighbor samples along the ray r.

###  3.3 Loss Function

Our model is only supervised by the rendering loss, which is calculated by the
mean square error between rendered colors and ground-truth colors, as

$\displaystyle\\{\hat{\textbf{c}}_{1},...,\hat{\textbf{c}}_{H\times
W}\\}=\mathcal{R}(P|R,t,K),$ (8)
$\displaystyle\mathcal{R}=\\{D_{x},D_{y},D_{z},g\\},$
$\displaystyle\mathcal{L}=\lVert\hat{\textbf{c}}_{i}-\bar{\textbf{c}}_{i}\rVert_{2}^{2},$

where $I=\\{\hat{\textbf{c}}_{1},...,\hat{\textbf{c}}_{H\times W}\\}$ are the
rendered colors, $\\{\bar{\textbf{c}}_{1},...,\bar{\textbf{c}}_{H\times W}\\}$
are the ground-truth colors.

ScanNet [9] ShapeNet [4] Google Scanned Objects [12] PSNR ($\uparrow$) SSIM
($\uparrow$) LPIPS ($\downarrow$) PSNR ($\uparrow$) SSIM ($\uparrow$) LPIPS
($\downarrow$) PSNR ($\uparrow$) SSIM ($\uparrow$) LPIPS ($\downarrow$)
Graphics Renderer [36] 13.62 0.528 0.779 19.24 0.814 0.182 23.14 0.829 0.153
Sparse-ConvNet [7] 15.27 0.646 0.602 22.16 0.836 0.159 25.36 0.868 0.128 NPCR
[10] 16.22 0.659 0.574 23.41 0.855 0.136 27.73 0.892 0.107 ConvONet [30] 16.43
0.665 0.584 24.25 0.848 0.122 28.17 0.917 0.093 ADOP [38] 16.83 0.699 0.577
24.96 0.857 0.129 29.06 0.922 0.089 NPBG++ [35] 16.81 0.671 0.585 25.32 0.874
0.120 29.42 0.929 0.081 Point-NeRF [46] 17.53 0.685 0.517 25.73 0.897 0.107
29.84 0.938 0.069 Voxels-128 17.65 0.694 0.538 25.53 0.872 0.116 29.41 0.926
0.078 Ours 18.56 0.734 0.473 27.22 0.927 0.084 31.24 0.961 0.045

Table 2: Quantitative comparison for point cloud rendering accuracy between
ours and the state-of-the-art methods as well as baselines on the ScanNet [9],
ShapeNet [4], and Google Scanned Objects [12] datasets.

##  4 Experiments

###  4.1 Datasets

We evaluate the effectiveness of our framework with the TriVol representation
on object-level and scene-level datasets. For the object level, we use both
synthesized and real-world scanned datasets, including ShapeNet [4] and Google
Scanned Objects (GSO) [12]. ShapeNet is a richly-annotated and large-scale 3D
synthesized dataset. There are about 51,300 unique 3D textured mesh models,
and we choose the common Car category for evaluation. GSO dataset contains
over 1000 high-quality 3D-scanned household items, and we perform experiments
on the category of shoe. The point clouds in object-level datasets can be
obtained by 3D mesh sampling [21], and ground-truth rendered images are
generated by Blender [18] under random camera poses. For the scene level, we
conduct the evaluation on the ScanNet [9] dataset, which contains over 1500
indoor scenes. Each scene is constructed from an RGBD camera. We split the
first 1,200 scenes as a training set and the rest as a testing set.

###  4.2 Implementation Details

We set the volume resolution $S$ to 256, and the number of groups $G$ to 16.
The decoders ($D_{x}$, $D_{y}$, and $D_{z}$) do not share the weights. The
number of layers in NeRF’s MLP $g$ is 4. For each iteration, we randomly
sample $1024$ rays from one viewpoint. The number of points for each ray is
$64$ for both coarse and fine sampling. The resolutions $H\times W$ of
rendered images on the ScanNet [9] dataset are $512\times 640$, and the other
datasets are $256\times 256$. AdamW [22] is adopted as the optimizer, where
the learning rate is initialized as $0.001$ and will decay to $0.0001$ after
100 epochs. We train all the models using four RTX 3090 GPUs. For more
details, please refer to the supplementary file.

###  4.3 Baselines

Besides the comparison with existing point cloud rendering methods, _e.g_.,
NPBG++ [35], ADOP [38] and Point-NeRF [46], several important baselines are
also combined with NeRF [26] to demonstrate the effectiveness of our
framework. They can be described as follows:

  * •

Voxels-128: This baseline generates dense voxels with 3D dense UNet. However,
due to its constrained efficiency and the memory limitation of the computation
resource, the voxel resolution is set as $128^{3}$.

  * •

Sparse ConvNet: MinkowskiEngine [7] is a popular sparse convolution library.
We make use of its sparse 3D UNet, _i.e_., MinkUnet34C [7], as the baseline.
Its feature querying is performed by a ball query.

  * •

ConvOnet [30]: ConvOnet converts the point cloud features into a triple-plane
representation for 3D reconstruction. We replace its occupancy prediction with
NeRF module to achieve point cloud rendering.

Note that Voxels-128 has the same channel number as TriVol. One alternative
baseline is the dense voxels with high resolution, e.g., 256, and reduced
channel number for efficiency. However, the minor channel number will impede
the formulation of discriminative feature volumes, and such an alternative
baseline has been proved to have terrible rendering results empirically. The
differences between our framework and these baselines are summarized in Tab.
1. We adopt the metrics of PSNR, SSIM [42], and LPIPS [50] for evaluation.

###  4.4 Evaluate Scene-Level Rendering

In this experiment, we compare ours with baselines, and SOTA methods [10, 46,
35] on the ScanNet dataset [9]. For each scene, we sample 100k points on the
provided textured mesh as the colored point cloud. Quantitative and
qualitative results are presented in Tab. 2 and Fig. 3, respectively.

The point clouds at the scene level usually have missing points and parts,
which causes hole artifacts in graphics-based rendering images, as shown in
the left column of Fig. 3. The traditional voxel-based method could not
generate high-quality images due to its low-resolution representation. There
are many artifacts in NPBG++ [35], since the 2D-CNN-based rendering only
employs the limited 2D information, not 3D context, to remove the discrete
issues, leading to imprecise and unrealistic results when there is no fine-
tuning stage. Moreover, the view-inconsistent shortcomings of 2D-CNN-based
approaches will be demonstrated in the supplementary file. Furthermore, Point-
NeRF [46] could not recover the missing parts. The reason is that the KNN
strategy will usually fail if the nearest neighbor points are far from the
queried points [34]. Ours performs the best in complementing the missing areas
and enhancing local details, thus could render photo-realistic images.
Moreover, as shown in Tab. 2, our framework performed better than others by
large margins on all metrics. The performance of our method demonstrates the
better generalization ability of TriVol compared with the SOTA methods and
baselines.

###  4.5 Evaluate Object-Level Rendering

We also conduct the evaluation of rendering at the object level. Experiments
are conducted on the ShapeNet Car dataset and the Google Scanned Object Shoe
dataset. We uniformly sample 100k points for each 3D model. Different from the
scene level, most of the point cloud at the object level is relatively dense.
Therefore, all methods can achieve higher metrics than the results at the
scene level. The quantitative results are shown in Tab. 2, our framework still
has the best performance over others, showing the discriminative 3D
representation of TriVol. Moreover, the qualitative evaluations are displayed
in Figs. 4 and 5. Although there is no fine-tuning process, our method
produces rendering results with richer and more precise details than
baselines. For example, as shown in Fig. 4, the car wheel structures could be
accurately rendered with our approach, and are blurry at baselines’ results.
The realistic rendering performance demonstrates the effectiveness of our
framework in general category-specific point cloud rendering. Please refer to
the supplementary file for more object-level comparisons within various
categories.

Figure 3: Qualitative point cloud rendering comparison between ours and SOTA
methods and baselines on the ScanNet dataset. Figure 4: Qualitative point
cloud rendering comparison between ours and SOTA methods and baselines on the
ShapeNet dataset. Figure 5: Qualitative point cloud rendering comparison
between ours and SOTA methods and baselines on the GSO dataset.

###  4.6 Ablation Study

In the subsequent experiments, we study the influence of different modules and
parameters in our method.

####  4.6.1 The Effect of Encoder

In this experiment, we explore the performance of different encoders in
transforming a point cloud to our Initial TriVol. The baselines are the
encoders containing the point-based networks, including PointNet [46],
PointNet++ [34], and Sparse ConvNet [7]. After the point feature extraction in
baselines, we pool the point features to our TriVol’s shape. The results are
shown in Tab. 3. We note that our grouping strategy does not require
additional memory while performing the best, showing the strength in employing
the grouping mechanism in the TriVol encoder.

Encoder Memory (GB) PSNR ($\uparrow$) SSIM ($\uparrow$) LPIPS ($\downarrow$)
PointNet [33] 4.78 18.03 0.698 0.521 PointNet++ [34] 5.23 18.26 0.707 0.505
Sparse ConvNet [7] 2.15 18.50 0.719 0.494 Grouping (Ours) 0 18.56 0.734 0.473

Table 3: The effect of different encoders on the ScanNet Dataset. The decoder
of these methods are all the same three 3D UNet.

####  4.6.2 The Effect of TriVol Resolution

One advantage of our method is that high-resolution TriVol can be obtained
with lightweight computation through dense 3D ConvNets. In this experiment, we
study the influence of TriVol resolution, including the group size $G$ and
volume resolution $S$. The experimental results are shown in Tab. 4, where
“Number” indicates the number of volumes in each 3D representation. Thus, the
top two rows are the baselines of dense voxels with the resolution of
$128^{3}$ and $256^{3}$, respectively. The dense voxel with resolution
$128^{3}$ requires a tremendous amount of computation resources in terms of
GPU Memory and FLOPS. The voxels of $256^{3}$ even return an Out-Of-Memory
(OOM) issue. Moreover, we noted that the rendering accuracy with these dense
voxels is not satisfying enough. On the other hand, the bottom four rows in
Tab. 4 explore the effect of our framework with different combinations between
group sizes and volume resolutions. Note that different combinations’ memory
costs are lower than the dense voxel with $128^{3}$ resolution. The
superiority in the efficiency of TriVol can also be observed from the metric
of FLOPS. Moreover, TriVol has an much better accuracy in rendering metrics.
Therefore, we finally adopt $G=16$, $S=256$ to balance the computing resources
and accuracy, which has been indicated in Sec. 4.2.

Number G S GPU Memory ($GB,\downarrow$) FLOPS ($G,\downarrow$) PSNR
($\uparrow$) 1 - 1283 13.83 167.91 25.53 1 - 2563 OOM - - 3 8 642 2.11 9.34
22.35 3 16 1282 3.13 74.69 24.84 3 16 2562 8.47 103.48 27.22 3 32 2562 9.89
197.55 27.38

Table 4: The effect of TriVol Resolution on the ShapeNet Dataset.

####  4.6.3 The Effect of Point Number

In the real-world application, the testing scenes or objects might contain a
visibly different point number from the training data, leading to performance
degradation. Therefore, in this experiment, we set different point numbers for
the training and evaluation data, as shown in Tab. 5. When the point number
during the training and testing are the same, similar to Point-NeRF [46],
increasing the input point number can improve the rendering accuracy. Our
method achieves the best performance with different numbers of input points.
When the number of testing points is far less than the train points, our
approach shows a more robust performance compared with Point-NeRF [46] since
TriVol is a general discriminative and continuous 3D representation regardless
of the point cloud’s sparse/dense degree. Note that although the point number
is changed, TriVol’s resolution is fixed in this experiment. Hence, our TriVol
is more robust than Point-NeRF [46] when the points’ sparsity changes.

Methods Training Points Testing Points PSNR ($\uparrow$) Point-NeRF [46] 10k
10k 23.83 100k 10k 22.75 100k 100k 25.73 TriVol (Ours) 10k 10k 25.31 100k 10k
27.01 100k 100k 27.22

Table 5: The effect of the number of points during training and testing on the
ShapeNet dataset.

##  5 Conclusion

In this paper, we analyze the limitations of present point-based rendering
methods, including the hole artifacts for graphics-based methods, the view
inconsistency for the 2D neural-based approaches, and the inefficiency of
existing 3D neural-based strategies. We propose a novel rendering framework by
designing a lightweight and continuous 3D representation TriVol. The Initial
TriVol is obtained from the point cloud with a simple axis grouping mechanism,
and the Feature TriVol is inferred by efficient computation in 3D UNets. The
feature querying for all 3D locations can be completed via the trilinear
interpolation in TriVol. The TriVol-based framework can represent high-
resolution volumes and adaptive capture both local and non-local information
for NeRF-based volume rendering. Extensive experiments are conducted on both
scene- and object-level benchmarks, showing our framework can be utilized in
rendering images with clear details and without holes artifacts.

Limitation and future work. Although high-quality rendering images were
obtained with TriVol from point clouds, it is still very challenging for our
method to render the scene where an extremely large number of points are
missing. This is technically a 3D scene synthesis task and requires a pre-
trained 3D generator trained on a large-scale dataset to synthesize more
missing points. We plan to solve it in future work based on the proposed
TriVol representation.

## Acknowledgments

This work is partially supported by Shenzhen Science and Technology Program
KQTD20210811090149095.

## References

  * [1] Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky.  Neural point-based graphics.  In ECCV, 2020.
  * [2] Dan Cernea.  OpenMVS: Multi-view stereo reconstruction library.  City, 2020.
  * [3] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In CVPR, 2022.
  * [4] Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  Shapenet: An information-rich 3d model repository.  arxiv, 2015.
  * [5] Gaurav Chaurasia, Sylvain Duchene, Olga Sorkine-Hornung, and George Drettakis.  Depth synthesis and local warps for plausible image-based navigation.  ACM TOG, 2013.
  * [6] Yun Chen, Frieda Rong, Shivam Duggal, Shenlong Wang, Xinchen Yan, Sivabalan Manivasagam, Shangjie Xue, Ersin Yumer, and Raquel Urtasun.  Geosim: Realistic video simulation via geometry-aware composition for self-driving.  In CVPR, 2021.
  * [7] Christopher B. Choy, JunYoung Gwak, and Silvio Savarese.  4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks.  In CVPR, 2019.
  * [8] Tiago Cortinhal, Fatih Kurnaz, and Eren Erdal Aksoy.  Semantics-aware multi-modal domain translation: From lidar point clouds to panoramic color images.  In ICCV, 2021.
  * [9] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  ScanNet: Richly-annotated 3D reconstructions of indoor scenes.  In CVPR, 2017.
  * [10] Peng Dai, Yinda Zhang, Zhuwen Li, Shuaicheng Liu, and Bing Zeng.  Neural point cloud rendering via multi-plane projection.  In CVPR, 2020.
  * [11] Alexandre Devaux and Mathieu Brédif.  Realtime projective multi-texturing of pointclouds and meshes for a realistic street-view web navigation.  In Proceedings of the 21st International Conference on Web3D Technology, 2016.
  * [12] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B. McHugh, and Vincent Vanhoucke.  Google scanned objects: A high-quality dataset of 3d scanned household items.  In ICRA, 2022.
  * [13] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler.  Get3d: A generative model of high quality 3d textured shapes learned from images.  In NeurIPS, 2022.
  * [14] Michael Goesele, Brian Curless, and Steven M Seitz.  Multi-view stereo revisited.  In CVPR, 2006.
  * [15] Ben Graham.  Sparse 3d convolutional neural networks.  In BMVC, 2015.
  * [16] Benjamin Graham and Laurens van der Maaten.  Submanifold sparse convolutional networks.  arxiv, 2017.
  * [17] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt.  StyleNeRF: A style-based 3d-aware generator for high-resolution image synthesis.  In ICLR, 2021.
  * [18] Roland Hess.  Blender Foundations: The Essential Guide to Learning Blender 2.6.  Focal Press, 2010.
  * [19] Georgios Kopanas, Julien Philip, Thomas Leimkühler, and George Drettakis.  Point-based neural rendering with per-view optimization.  In Comput. Graph. Forum, 2021.
  * [20] Christoph Lassner and Michael Zollhofer.  Pulsar: Efficient sphere-based neural rendering.  In CVPR, 2021.
  * [21] Davi Lazzarotto and Touradj Ebrahimi.  Sampling color and geometry point clouds from shapenet dataset.  arxiv, 2022.
  * [22] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [23] Arun Mallya, Ting-Chun Wang, Karan Sapra, and Ming-Yu Liu.  World-consistent video-to-video synthesis.  In ECCV, 2020.
  * [24] Nelson Max.  Optical models for direct volume rendering.  IEEE TVCG, 1995.
  * [25] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  ACM TOG, 2019.
  * [26] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [27] Phong Nguyen-Ha, Nikolaos Sarafianos, Christoph Lassner, Janne Heikkilä, and Tony Tung.  Free-viewpoint rgb-d human performance capture and rendering.  In ECCV, 2022.
  * [28] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada.  Unsupervised learning of efficient geometry-aware neural articulated representations.  In ECCV, 2022.
  * [29] Julian Ost, Issam Laradji, Alejandro Newell, Yuval Bahat, and Felix Heide.  Neural point light fields.  In CVPR, 2022.
  * [30] Songyou Peng, Michael Niemeyer, Lars M. Mescheder, Marc Pollefeys, and Andreas Geiger.  Convolutional occupancy networks.  In ECCV, 2020.
  * [31] Sergey Prokudin, Michael J Black, and Javier Romero.  Smplpix: Neural avatars from 3d human models.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021.
  * [32] Albert Pumarola, Stefan Popov, Francesc Moreno-Noguer, and Vittorio Ferrari.  C-flow: Conditional generative flow models for images and 3d point clouds.  In CVPR, 2020.
  * [33] Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas.  PointNet: Deep learning on point sets for 3d classification and segmentation.  In CVPR, 2017.
  * [34] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas.  PointNet++: Deep hierarchical feature learning on point sets in a metric space.  In NeurIPS, 2017.
  * [35] Ruslan Rakhimov, Andrei-Timotei Ardelean, Victor Lempitsky, and Evgeny Burnaev.  NPBG++: Accelerating neural point-based graphics.  In CVPR, 2022.
  * [36] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3D deep learning with PyTorch3D.  arxiv, 2020.
  * [37] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional networks for biomedical image segmentation.  In Nassir Navab, Joachim Hornegger, William M. Wells III, and Alejandro F. Frangi, editors, MICCAI, 2015.
  * [38] Darius Rückert, Linus Franke, and Marc Stamminger.  ADOP: approximate differentiable one-pixel point rendering.  ACM TOG, 2022.
  * [39] Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang.  3d photography using context-aware layered depth inpainting.  In CVPR, 2020.
  * [40] Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka.  Epigraf: Rethinking training of 3d gans.  arxiv, 2022.
  * [41] Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely.  Pushing the boundaries of view extrapolation with multiplane images.  In CVPR, 2019.
  * [42] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE TIP, 2004.
  * [43] Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra.  Embodied question answering in photorealistic environments with point cloud perception.  In CVPR, 2019.
  * [44] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson.  Synsin: End-to-end view synthesis from a single image.  In CVPR, 2020.
  * [45] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann.  Point-NeRF: Point-based neural radiance fields.  In CVPR, 2022.
  * [46] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann.  Point-NeRF: Point-based neural radiance fields.  In CVPR, 2022.
  * [47] Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, and Bolei Zhou.  3d-aware image synthesis via learning structural and textural representations.  In CVPR, 2022.
  * [48] Ilya Zakharkin, Kirill Mazur, Artur Grigorev, and Victor Lempitsky.  Point-based modeling of human clothing.  In ICCV, 2021.
  * [49] Qiang Zhang, Seung-Hwan Baek, Szymon Rusinkiewicz, and Felix Heide.  Differentiable point-based radiance fields for efficient view synthesis.  arXiv, 2022.
  * [50] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.


# We’re Calling an Intervention:
Taking a Closer Look at Language Model Adaptation
to Different Types of Linguistic Variation

Aarohi Srivastava & David Chiang
Department of Computer Science and Engineering
University of Notre Dame
Notre Dame, IN, USA
<EMAIL_ADDRESS>
###### Abstract

We present a suite of interventions and experiments that allow us to
understand language model adaptation to text with linguistic variation (e.g.,
nonstandard or dialectal text). Our interventions address several features of
linguistic variation, resulting in character, subword, and word-level changes.
Applying our interventions during language model adaptation with varying size
and nature of training data, we gain important insights into what makes
linguistic variation particularly difficult for language models to deal with.
For instance, on text with character-level variation, performance improves
with even a few training examples but approaches a plateau, suggesting that
more data is not the solution. In contrast, on text with variation involving
new words or meanings, far more data is needed, but it leads to a massive
breakthrough in performance. Our findings inform future work on dialectal NLP
and making language models more robust to linguistic variation overall. We
make the code for our interventions, which can be applied to any English text
data, publicly available.

##  1 Introduction

Linguistic variation is all around us. Whether a user adopts a regional
dialect, follows different spelling conventions, or uses culturally-specific
vocabulary, encountering linguistic variation in most day-to-day NLP use cases
is inevitable. Yet, recent work continues to find large gaps in performance
between standard and nonstandard text and speech (Kantharuban et al., 2023;
Faisal et al., 2024), and efforts dedicated to reducing this gap are typically
case-, variety-, or task-dependent (Held et al., 2023; Joshi et al., 2024).
Whether deploying a large-scale system for diverse users, or working towards
language model personalization, it is important to understand the challenges
one might face when trying to adapt a language model to different varieties.
This is tricky, though – it is difficult to tease apart the interactions
between the complex and intertwined features that comprise linguistic
variation, and the blackbox nature of language models makes it even more
difficult to do so.

One particular challenge relates to how text is represented by language models
(Kumar et al., 2020; Soper et al., 2021). Language models like BERT (Devlin et
al., 2019) and GPT (Radford et al., 2019) are pre-trained to assign meaning to
subword units. Frequently seen words often correspond to a single token, while
infrequently seen (or misspelled) words often correspond to several shorter-
length (or single character) tokens. We present a few such examples below,
where $\stackrel{{\scriptstyle\text{tok}}}{{\leadsto}}$ denotes tokenization
(what the model receives as input):

coffee $\stackrel{{\scriptstyle\text{tok}}}{{\leadsto}}$ coffee

cofee $\stackrel{{\scriptstyle\text{tok}}}{{\leadsto}}$ co, fe, e

airfryer $\stackrel{{\scriptstyle\text{tok}}}{{\leadsto}}$ air, f, ry, er

googling $\stackrel{{\scriptstyle\text{tok}}}{{\leadsto}}$ go, og, ling

When dealing with linguistic variation, we will encounter infrequent strings
far more often, which inherently means we are dealing with longer sequences of
tokens comprised of shorter subwords. Shorter subwords (e.g., fe, og) can
appear in many more contexts and take on different meanings each time.
Moreover, when dealing with lexical or semantic variation, frequent words may
take on a new meaning, which the model may not have knowledge of, or a word
may be used in a new context. In either case, the model will not already have
the relevant knowledge related to the word. These types of issues result in
difficulty for a BERT-like model to assign appropriate contextual meaning to
inputs exhibiting linguistic variation.

To this end, we develop a set of experiments that isolate data-related factors
that can play a role in language model adaptation (e.g., type, amount, and
composition of training data), and we assemble a suite of ten interventions to
synthetically induce different forms of linguistic variation (e.g.,
orthographic, morphosyntactic, lexicosemantic) in controlled settings. We will
make the code for these interventions publicly available. Through our
experiments, we evaluate and stress-test BERT’s ability to adapt to various
types of non-standard text under different conditions.

Our suite of interventions and experiments is designed in such way that
intertwined aspects of linguistic variation found in naturally-occurring
dialect data are isolated and separately tested. Though dialectal NLP work is
often difficult due to the lack of available data, our synthetic interventions
can be directly applied to any type and amount of English text data. The
results of our experiments inform important questions about what is needed to
facilitate language model adaptation to text with linguistic variation and
provide key insights about the most challenging aspects:

  * •

The language models tested demonstrate extremely low understanding capability
out-of-the-box for all types of linguistic variation tested.

  * •

Access to more data is vital when dealing with lexical and semantic variation
(e.g., Spanish varieties, Italian dialects). Multilingual models are also more
helpful in such cases. In contrast, the amount of data is not as important for
varieties exhibiting more spelling variation (e.g., Germanic languages and
varieties), and robustness to such variation will likely require another
solution besides more data. Monolingual models are more helpful than
multilingual models in these cases.

  * •

As with any adaptation task, data composition and quality are very important.
When adapting to language varieties, it is particularly crucial that the
necessary patterns are clear from the data, that is, standard and nonstandard
variants should not coexist in the data.

##  2 Related Work

As larger and larger language models with newfound capabilities continue to
emerge, the NLP community also continues to find that dealing with linguistic
variation (e.g., dialects, language varieties, and noisy text) remains a
challenge. For instance, Kantharuban et al. (2023) and Faisal et al. (2024)
conduct numerous experiments on several tasks involving language varieties
from every language family, finding large gaps in performance across the board
when evaluating on the standard versus a nonstandard variety of a language.
Perhaps an even more well-documented issue is the issue of noisy text.
Language models are not robust at all to noisy text, plummeting in performance
when faced with seemingly simple matters like misspellings, typos, and
grammatical errors, even though it is a natural issue to expect in almost any
use case (Kumar et al., 2020; Yin et al., 2020; Aspillaga et al., 2020).
Approaches to adapting or making a model more robust to linguistic variation,
though successful in their use cases, are often highly dependent on the
language variety itself and the type of task, typically requiring availability
of data for each dialect in question (Held et al., 2023) and exposing
differences in performance on a task-to-task and variety-to-variety basis
(Srivastava & Chiang, 2023). As such, we need to take a step back and gain a
better understanding of what makes this type of adaptation difficult and what
is needed to facilitate the required learning.

Past work has investigated the question of what makes cross-lingual transfer
difficult (Philippy et al., 2023). For instance, Wu et al. (2022) explore
three high-level changes (embedding space disruption, changes in tokenization,
and changes in word order), and transform GLUE datasets to induce each of
these changes. They find that language models can learn to handle changes in
tokenization and word order, but are not as equipped to recover from embedding
space disruption (e.g., learning new alignments). Deshpande et al. (2022) and
Jain et al. (2022) come to similar conclusions through their experiments;
Deshpande et al. (2022) find that subword overlap and token embedding
alignment are vital for successful knowledge transfer, and Jain et al. (2022)
suggest the need for word alignment and dictionary matching techniques.

The issues of embedding space disruption and token alignment are also
extremely relevant to linguistic variation, as both relate to learning new
alignments or mappings of token embeddings. The findings from past work inform
the decisions made in selecting and designing our interventions and
experiments in such a way that the model has to overcome embedding and token
alignment issues to accomplish the task.

##  3 Linguistic Variation

Linguistic variation is a natural response to the productive nature of
language. It can be observed at all levels of linguistic knowledge and
reasoning (Haber, 1976; Geeraerts et al., 1994): phonological and orthographic
variation (i.e., sounds and spellings), morphological and syntactic variation
(i.e., word and sentence structure), lexical and semantic variation (new word
senses and meanings), and variation in style and tone, often related to
language-external sociopolitical factors. Style and tone variation is
typically comprised of a mixture of features from the first three categories,
described in more detail below.

Drawing on documented features of linguistic variation, we devised a suite of
10 interventions that each pick apart a different aspect of linguistic
variation. We apply these interventions to all text used in our experiments.
These 10 interventions fall into four categories: orthographic variation,
subword boundary manipulation, morphosyntactic variation, and lexicosemantic
variation. All of our data and experiments are in English, but the algorithmic
nature of many of our interventions can be applied to other languages.

###  3.1 Orthographic Variation

Orthographic variation, or spelling differences, can arise for several
reasons, including phonological influences (e.g., accent, sound change) and
new methods of writing (e.g., social media) (Condorelli & Rutkowska, 2023). We
include two interventions under this category; the first is a reflection of
phonological variation, and the second is more of a stress test:

  1. 1.

IPA: Letters corresponding to consonants on the IPA chart that form a minimal
feature pair (voiced vs. unvoiced) are swapped. For example, _p_ and _b_ ,
which differ only in the voicing feature but share the same place and manner
of articulation, are swapped. Changes to consonants are a hallmark of modern
English orthographic variation, particularly observed on social media
(Eisenstein, 2015; Ilbury, 2020).

boots $\stackrel{{\scriptstyle\text{IPA}}}{{\leadsto}}$ poodz
$\stackrel{{\scriptstyle\text{tok}}}{{\leadsto}}$ p, ood, z

  2. 2.

Shift: A Caesar cipher is applied to the letters of the alphabet:
$a\rightarrow b,b\rightarrow c,...,z\rightarrow a$. This intervention is the
most extreme form of orthographic change, in which every alphabetic symbol is
renamed. It roughly approximates a situation where a language variety uses a
different alphabet than the standard variety.

boots $\stackrel{{\scriptstyle\text{Shift}}}{{\leadsto}}$ cpput
$\stackrel{{\scriptstyle\text{tok}}}{{\leadsto}}$ c, pp, ut

###  3.2 Subword Boundaries

As demonstrated above, linguistic variation often results in changes to
subword token boundaries, which triggers a domino effect and ultimately
results in lower quality contextual embeddings assigned to nonstandard text by
the model (Kumar et al., 2020; Soper et al., 2021). Because of this, we
include three interventions that overtly manipulate subword boundaries, two of
which preserve the original spelling of the word and only split tokens.

  1. 1.

Reg: Subword regularization using the MaxMatch Dropout (Hiraoka, 2022) method
for BERT’s WordPiece algorithm is applied with a dropout of 0.5. This means
50% of the subword vocabulary is randomly dropped with each tokenization call.

boots $\stackrel{{\scriptstyle\text{Reg}}}{{\leadsto}}$ boot, s

illustrate $\stackrel{{\scriptstyle\text{Reg}}}{{\leadsto}}$ ill, us, trate

  2. 2.

Char: Subword regularization using the MaxMatch Dropout method for WordPiece
is applied with a dropout of 1, meaning each letter of a word is its own
token.

boots $\stackrel{{\scriptstyle\text{Char}}}{{\leadsto}}$ b, o, o, t, s
$\stackrel{{\scriptstyle\text{tok}}}{{\leadsto}}$ b, o, o, t, s

  3. 3.

Pig: Words are converted to Pig Latin, in which the word-initial consonant(s)
is moved to the end, and a suffix (_ay_ or _yay_) is added. For example, _pig
latin_ becomes _igpay atinlay_.

boots $\stackrel{{\scriptstyle\text{Pig}}}{{\leadsto}}$ ootsbay
$\stackrel{{\scriptstyle\text{tok}}}{{\leadsto}}$ o, ots, bay

###  3.3 Morphosyntactic Variation

Morphosyntactic variation, often highly stigmatized, relates to grammatical
acceptability – some utterances may be grammatical in one variety and not in
the standard, or vice versa (Neef, 2009; Zanuttini & Horn, 2014).

  1. 1.

$-$End: Using MorphyNet (Batsuren et al., 2021), inflectional endings are
dropped from words that have them. Inflectional endings are morphemes with
grammatical functions that typically change grammatical features like part of
speech and number.

boots $\stackrel{{\scriptstyle\text{-End}}}{{\leadsto}}$ boot
$\stackrel{{\scriptstyle\text{tok}}}{{\leadsto}}$ boot

  2. 2.

Multi: Sentences are modified to exhibit common morphosyntactic features of
English dialects. We use Multi-VALUE (Ziems et al., 2023), a tool that
systematically modifies text reflecting morphosyntactic features collected
from several English dialects, including African American, Appalachian, and
Indian English.111A full list of English varieties supported by Multi-VALUE
can be found here: https://github.com/SALT-NLP/multi-value.

###  3.4 Lexicosemantic Variation

Word-level variation can be just that – a new word introduced to refer to a
specific, perhaps novel, referent, or to be used in a new context. But,
lexical variation can often introduce semantic variation; the typical process
being that a word’s senses expand and eventually shift to the new meaning,
through specialization, generalization, or subjectification (Kakharova, 2021;
Geeraerts et al., 2024).

  1. 1.

Affix: Using MorphyNet (Batsuren et al., 2021), derivational prefixes and
suffixes are mapped cyclically. For instance, _non-_ becomes _ab-_ , _ab-_
becomes _pre-_ , etc. _Nonsense_ is now _absense_ , _absence_ is now
_presence_ , and so on. In this way, we tinker with part of a word but change
its whole meaning. Derivational affixes are used to change the meaning of a
word, which may or may not also change its part of speech. Unlike inflectional
endings, which are a functional category, derivational affixes are productive
and have much more room for linguistic variation.

nonsense $\stackrel{{\scriptstyle\text{Affix}}}{{\leadsto}}$ absense
$\stackrel{{\scriptstyle\text{tok}}}{{\leadsto}}$ a, bs, ense

  2. 2.

Hyp: Exemplifying generalization, through which a word’s range of reference
broadens, words with hyponyms found in WordNet (Princeton University, 2010)
are replaced by their hyponym.

boot $\stackrel{{\scriptstyle\text{Hyp}}}{{\leadsto}}$ buskin
$\stackrel{{\scriptstyle\text{tok}}}{{\leadsto}}$ bus, kin

  3. 3.

Ant: Exemplifying subjectification (specifically, ameliorization, through
which a word’s meaning becomes more positive, and pejorization, through which
a word’s meaning becomes more negative) words with antonyms found in WordNet
(Princeton University, 2010) are replaced by their antonym.

nice $\stackrel{{\scriptstyle\text{Ant}}}{{\leadsto}}$ nasty

Inherently, all the interventions are new mappings for the model to learn;
however, the differences in how they impact the model’s tokenization and
understanding can make learning these mappings during fine-tuning more or less
difficult.

##  4 Data Preparation

Figure 1: A sketch of our train/test pipeline.

To compare the model’s ability to adapt to each synthetic variety, we fine-
tune and test it with the mask-filling objective on text with the appropriate
intervention applied. We provide a sketch of our train/test pipeline in Figure
1. All of our data is sourced from Wikicorpus to reduce external effects of
choice of data. We reserve half the Wikicorpus articles for fine-tuning and
half for testing. These are separated into sentences using the sentence
tokenizer from NLTK (Loper & Bird, 2002). In our experiments, a _word_ is any
string that is an element of NLTK word tokenizer’s output when applied to a
sentence, and that satisfies Python’s isalpha function. Sentences with 0 or 1
words are eliminated.

When fine-tuning, we vary the amount of data used. Each of these splits is a
fixed set of sentences sampled from the fine-tuning articles. We cover three
orders of magnitude: small (264), medium (2,641), and large (26,415). We
report statistics on sentence length for each of the splits to demonstrate
that they are comparable in this regard, where sentence length is measured in
terms of number of _words_ , in Table 1.

Learning during fine-tuning is typically task-oriented, and as a result, it
can be difficult to try to adapt a model to both a task and a new variety at
once. As a result, we select mask-filling as the fine-tuning task, which is
also the pre-training objective of the models used, meaning fine-tuning
resources can be dedicated to learning about the new variety. Our masking
policy during fine-tuning is as follows. One _word_ per sentence is randomly
selected to be masked. We practice whole-word masking, meaning the model could
be asked to fill one or more consecutive mask tokens, dependent on the number
of subword tokens that comprise the selected masked word.

In addition to varying the data size and the intervention to be applied, we
also vary the composition of the fine-tuning data. It could be _mixed_ ,
meaning the intervention is only applied to half the sentences, or _full_ ,
meaning the intervention is applied to all sentences. When an intervention is
applied to a sentence, aside from our syntactic intervention which operates at
the sentence level, all other interventions are applied to each _word_.

Depending on the word selected to be masked, it is possible the intervention
does not actually change that word. While this is natural for fine-tuning, to
make sure the comparison during testing is fair, we mandate that the word
masked at test time is actually modified by all 10 interventions. To arrive at
a satisfactory test set, we begin with 10000 sentences randomly sampled from
the testing articles, and only keep the sentences where there exists a word
that is indeed modified by all 10 interventions, resulting in 931 sentences.
For each sentence, the same word is masked in all interventions/tests for
consistency.

Size | Sentences | Words
---|---|---
|  | Average | 25th %ile | 50th %ile | 75th %ile
S | 264 | 19.9 | 10 | 18 | 26
M | 2641 | 18.5 | 10 | 16 | 24
L | 26415 | 18.4 | 10 | 16 | 24
Table 1:  Sentence and word count statistics for each data split.

##  5 Experiments

There are four axes of variation in our experiments: _amount_ of fine-tuning
data (small, medium, large), _composition_ of fine-tuning data (intervention
is applied to 50% (mixed) or 100% (full) of sentences), _intervention_ to be
applied (10 total), and pre-trained model (monolingual or multilingual). We
use BERT-base-cased222https://huggingface.co/bert-base-cased (BERT) as the
monolingual English model, and BERT-base-multilingual-
cased333https://huggingface.co/bert-base-multilingual-cased (mBERT) as the
multilingual model. We do not include character-level models like
CharacterBERT (El Boukkouri et al., 2020) and CANINE (Clark et al., 2022) in
our experiments; unlike BERT and mBERT, they cannot be used out-of-the-box for
the mask-filling task and would not yield comparable results. For each
possible combination of the four axes of variation, a pre-trained model is
fine-tuned on the corresponding data with the masked language modeling
objective. As described in the previous section, one word per sentence is
masked (whole word masking), and the objective during fine-tuning is to fill
the masked token(s) with the tokens of the original word.

We use the Low-Rank Adaptation (LoRA) method (Hu et al., 2021) of parameter-
efficient fine-tuning, which adapts the attention weights of each Transformer
encoder layer and freezes all other parameters. We add a trainable language
modeling head before fine-tuning, which maps the output of the final encoder
layer to predictions over the vocabulary for which token fills the mask. We
follow Hu et al.’s guidelines for hyperparameter choice, using the AdamW
optimizer (Loshchilov & Hutter, 2018) with a linear scheduler, LoRA rank of 8,
and LoRA scaling factor $\alpha$ of 8. We use learning rate $7\cdot 10^{-4}$
when fine-tuning on the small data amount and $5\cdot 10^{-4}$ for the medium
and large data amounts.

We measure performance with three metrics: from most to least strict, exact
match accuracy, 1-best accuracy, and 5-best accuracy. Exact match accuracy
measures for how many masked words each token of the word is predicted
correctly, divided by the test set size. The 1-best accuracy metric measures
the total number of masked tokens filled correctly (by the top probability
prediction) divided by the total number of masked tokens in the test set.
Similarly, 5-best accuracy measures the total number of masked tokens whose
top five predictions include the correct answer, divided by the total number
of masked tokens in the test set. The 1-best accuracy metric provides the best
summary of the results; we include results using the other two metrics in the
appendix.

##  6 Results

The main 1-best results for our experiments are found in Table 2, and the
normalized scores are reported in Table 3. Additional results using the other
metrics are included in Table 4-6. We reiterate the categories of
interventions for readability:

  * •

Orthographic Variation: IPA, Shift

  * •

Subword Boundary Variation: Reg, Char, Pig

  * •

Morphosyntactic Variation: $-$End, Multi

  * •

Lexicosemantic Variation: Affix, Hyp, Ant

See Section 3 for detailed information on these interventions and types of
linguistic variation.

###  6.1 Baselines: We need an intervention!

We include a baseline row for each model, denoted by data amount 0, in which
the pre-trained model, without any additional fine-tuning, is tested on each
of the interventions. We also include a baseline intervention _None_ , in
which the models are evaluated on the original text without any intervention
applied.

The baseline performance achieved by mBERT is much lower than that of BERT,
indicating that the monolingual BERT model is better out-of-the-box at the
mask-filling task on Wikipedia text. In addition, the baseline column where
the models are fine-tuned and evaluated on unmodified text reveals that fine-
tuning on additional data without applying any intervention is actually
counter-productive; this is likely because the model is already trained on the
same task with a dramatically larger pre-training dataset. Fine-tuning is used
to adapt a model, which is appropriate for the interventions; however, when
fine-tuning on a much smaller sample drawn from the same corpus with no
intervention applied, we actually have no need to adapt the model. As a
result, if we do still fine-tune the model in this case, the model will start
“adapting to” patterns specific to the fine-tuning data, decreasing overall
generalizability. Because of these differences, we also include a table with
relative performance – the scores for each of the 10 baselines in Table 2
(full data composition) are normalized by the corresponding mask-filling
performance with no intervention, and expressed as percentages in Table 3.

Out-of-the-box performance (data amount 0) is best when there is no
intervention, but very low across the interventions. Out-of-the-box
performance is exceptionally low for the three lexicosemantic interventions
(Affix, Hyp, Ant), which require learning new word meanings/senses. These low
scores demonstrate the need for new methods to strengthen BERT-like models’
performance on text with linguistic variation.

###  6.2 Mixed vs. Full Composition: What is needed to learn new mappings?

The knowledge transfer that takes place when adapting a model to a new variety
of the language is akin to learning how to map elements of the standard
variety to the new one. One of the varied parameters of our experiments is
whether the fine-tuning data is mixed (intervention is only applied to 50% of
sentences) or full (intervention is applied to all sentences). Sometimes, we
might expect model learning to benefit from seeing both standard and
nonstandard versions of text during fine-tuning, while other times, this only
makes learning the appropriate patterns more difficult.

Our results indicate that the latter holds true when dealing with linguistic
variation; average performance (not including baseline scores) is about 4.2
points, or 36%, higher with the full composition (Table 2) rather than the
mixed composition (Table 4). Performance with full composition is also higher
when comparing averages for each intervention category, so the mixed
composition does not provide an advantage for any type of variation tested.

Because of this difference in performance, we refer only to the full
composition results when discussing and analyzing the remaining factors below.

###  6.3 Data Size: When is more data needed?

Across our experiments, particularly looking at the normalized performance
relative to the baseline intervention scores (Table 3), more fine-tuning data
helps. When adapting to the synthetic varieties with subword boundary
manipulation, performance improves gradually as more data is added. The
surface-level appearance can vary substantially word-to-word, particularly in
terms of the new token sequence passed to the model, and exposure to more
examples keeps adding to the model’s inventory of mappings.

In contrast, for the orthographic and morphosyntactic interventions, simply
fine-tuning, even on the small or medium amounts, makes all the difference.
These interventions seem to give way to clearer mappings that the model need
only be exposed to. (The Multi intervention is an exception which we discuss
below.)

Strikingly, there is an apparent breakthrough effect when fine-tuning with the
largest data size for the three lexicosemantic interventions. While the
performance attained is comparatively low for the small and medium data sizes,
there is a massive jump when the large data size is used for fine-tuning. This
is most clearly observed in the exact-match performance (Table 5) for BERT and
the relative performance for mBERT, which nears or exceeds 100% in the three
lexicosemantic tasks. These tasks require relearning the usage of words in new
contexts; mapping word-level information (i.e., new spellings) but also
contextual meaning evidently requires much more data.

###  6.4 Monolingual vs. Multilingual: Which type of knowledge is more
helpful?

For the most part, the average performance (not including baseline
performance) is extremely close between BERT and mBERT. Notably, mBERT
provides a substantial advantage for adapting to lexicosemantic variation,
providing a 20% boost in absolute scores (see Table 2) and a 74% improvement
in relative scores (see Table 3). Because mBERT is trained on several
languages, it is likely not as rigid in terms of relating words to specific
meanings/contexts, providing an advantage here.

###  6.5 Implications for Dialects and Language Varieties

Multi-VALUE is the only intervention that operates at the sentence level
rather than word-by-word, manipulating morphosyntactic features of a sentence
on the basis of 189+ documented linguistic features of English dialects (Ziems
et al., 2023). As a result, any mapping to be learned will be unclear with so
many interacting variables at play. The model’s inability to improve with
fine-tuning alone on text manipulated by Multi-VALUE serves as a testament to
the challenges one will face when directly fine-tuning on dialect text.

Consequently, the results convey the importance of gaining an improved
understanding of what is needed to facilitate learning different types of
linguistic mappings. To start, as demonstrated in all results, fine-tuning
performance rarely comes close to the performance when the intervention is
_none_. Even with larger data size, it is much harder for BERT-like models to
learn character-level information than word-level information.

While more data is not always the key, it is the key when dealing with
lexicosemantic variation, suggesting that more data may be needed when
adapting to such varieties (e.g., Spanish). Indeed, the new affix and antonym
substitution interventions are the only ones in which mask-filling performance
reaches or exceeds the corresponding mBERT baseline.

Finally, the issues of data composition or quality are pertinent in any
adaptation context. Though linguistic variation is an ongoing process in which
standard and nonstandard variants often coexist, the difference in performance
between the mixed and full data compositions suggests that simply fine-tuning
on dialectal or new-domain data may not be sufficient; perhaps it is helpful
to keep only those examples for fine-tuning that most strongly convey the
desired mappings.

|  | Interventions
---|---|---
Model | Data | None | IPA | Shift | Reg | Char | Pig |  $-$End | Multi | Affix | Hyp | Ant
BERT | 0 | 58.8 | 2.6 | 2.3 | 7.0 | 5.4 | 9.3 | 1.5 | 11.3 | 0.1 | 0.1 | 0.2
| S | 52.4 | 3.4 | 5.6 | 9.0 | 12.2 | 15.3 | 32.3 | 9.1 | 0.3 | 2.0 | 2.6
| M | 48.6 | 15.3 | 8.7 | 12.7 | 16.1 | 14.8 | 28.9 | 12.1 | 14.5 | 9.5 | 19.1
| L | 47.7 | 18.8 | 9.5 | 11.6 | 26.5 | 23.1 | 37.6 | 12.5 | 35.7 | 29.6 | 29.6
mBERT | 0 | 42.0 | 2.6 | 2.2 | 5.1 | 6.4 | 13.8 | 2.6 | 11.7 | 1.7 | 1.7 | 1.1
| S | 38.8 | 10.4 | 4.4 | 4.7 | 11.6 | 13.6 | 23.1 | 6.8 | 1.7 | 5.2 | 7.1
| M | 31.6 | 12.1 | 3.7 | 12.1 | 20.5 | 16.7 | 30.5 | 12.6 | 18.8 | 17.6 | 19.7
| L | 34.4 | 13.2 | 5.2 | 11.1 | 29.7 | 21.3 | 30.0 | 11.9 | 41.7 | 28.9 | 33.6

Table 2:
1-best accuracy results for all experiments with the full data composition.

|  | Interventions
---|---|---
Model | Data | None | IPA | Shift | Reg | Char | Pig |  $-$End | Multi | Affix | Hyp | Ant
BERT | 0 | 100.0 | 4.4 | 3.8 | 11.9 | 9.2 | 15.7 | 2.6 | 19.2 | 0.2 | 0.1 | 0.3
| S | 100.0 | 6.5 | 10.8 | 17.2 | 23.3 | 29.2 | 61.7 | 17.4 | 0.5 | 3.8 | 4.9
| M | 100.0 | 31.5 | 18.0 | 26.1 | 33.1 | 30.5 | 59.4 | 24.9 | 29.8 | 19.6 | 39.3
| L | 100.0 | 39.4 | 19.9 | 24.4 | 55.4 | 48.3 | 78.8 | 26.1 | 74.9 | 62.0 | 62.1
mBERT | 0 | 100.0 | 6.1 | 5.3 | 12.1 | 15.1 | 32.8 | 6.2 | 27.9 | 4.1 | 4.0 | 2.5
| S | 100.0 | 26.9 | 11.4 | 12.1 | 30.0 | 35.2 | 59.6 | 17.5 | 4.3 | 13.3 | 18.3
| M | 100.0 | 38.4 | 11.6 | 38.1 | 64.9 | 52.9 | 96.4 | 40.0 | 59.5 | 55.5 | 62.4
| L | 100.0 | 38.4 | 15.0 | 32.2 | 86.2 | 61.8 | 87.0 | 34.5 | 120.9 | 83.9 | 97.6

Table 3:  Relative performance normalized by the baseline (_None_), as
percentages.

##  7 Conclusion

We present a new suite of interventions that synthetically induce different
features of linguistic variation and can be used for language model analysis,
data augmentation, and more. In particular, our experiments using these text
interventions provide key insights into the challenges of adapting to
nonstandard/dialectal text, as well as suggestions for what can be done during
adaptation to best facilitate the required learning.

We find that using more data during fine-tuning is helpful overall, but the
composition of this data is very important. In particular, all examples in the
data should follow the nonstandard conventions rather than the standard
conventions so that the model can clearly learn the necessary patterns. This
might mean that data found in the wild needs a quality check – linguistic
variation is a process, meaning that standard and nonstandard variants will
coexist in naturally occurring data, but this may not be conducive to model
adaptation.

While more data is helpful, the importance of having more data is very
different for each category of variation. In particular, character-level
changes are a trouble spot for models; exposure alone is helpful to improve
performance, but more data does not mean that performance will continue to
improve. So fine-tuning on in-variety text alone is not enough to adapt to
orthographic/phonological variation, particularly those in which several
characters are modified (i.e., several mappings need to be learned
simultaneously). Some other solution besides adding more data will be needed
for robustness to this kind of variation.

In contrast, we find that the amount of data is extremely important when
dealing with lexical and semantic variation (i.e., words and meanings).
Indeed, we find a breakthrough effect where meaningful knowledge is not
acquired until the largest of our three data amounts is used during fine-
tuning, when performance jumps dramatically (particularly for multilingual
BERT). This information is pertinent to developing NLP techniques for
varieties and dialects in which lexical variation is high, such as Spanish and
Italian varieties.

We hope that our findings, with our publicly available suite of interventions,
will provide important guidelines and resources informing future work in
language modeling and serve as a call to action to make NLP models more
inclusive of linguistic variation.

## Acknowledgements

This material is based upon work supported by the National Science Foundation
under Grant No. IIS-2125948.

## References

  * Aspillaga et al. (2020) Carlos Aspillaga, Andrés Carvallo, and Vladimir Araujo.  Stress test evaluation of Transformer-based models in natural language understanding tasks.  In _Proc. LREC_ , pp. 1882–1894, 2020.  URL https://aclanthology.org/2020.lrec-1.232.
  * Batsuren et al. (2021) Khuyagbaatar Batsuren, Gábor Bella, and Fausto Giunchiglia.  Morphynet: a large multilingual database of derivational and inflectional morphology.  In _Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pp. 39–48, 2021.  URL https://aclanthology.org/2021.sigmorphon-1.5/.
  * Clark et al. (2022) Jonathan H Clark, Dan Garrette, Iulia Turc, and John Wieting.  Canine: Pre-training an efficient tokenization-free encoder for language representation.  _Transactions of the Association for Computational Linguistics_ , 10:73–91, 2022.  URL https://aclanthology.org/2022.tacl-1.5.pdf.
  * Condorelli & Rutkowska (2023) Marco Condorelli and Hanna Rutkowska.  _The Cambridge Handbook of Historical Orthography_.  Cambridge University Press, 2023.
  * Deshpande et al. (2022) Ameet Deshpande, Partha Talukdar, and Karthik Narasimhan.  When is BERT multilingual? isolating crucial ingredients for cross-lingual transfer.  In _Proc. NAACL HLT_ , pp. 3610–3623, 2022.  URL https://aclanthology.org/2022.naacl-main.264/.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proc. NAACL-HLT_ , pp. 4171–4186, 2019.  URL https://aclanthology.org/N19-1423.pdf.
  * Eisenstein (2015) Jacob Eisenstein.  Systematic patterning in phonologically-motivated orthographic variation.  _Journal of Sociolinguistics_ , 19(2):161–188, 2015.
  * El Boukkouri et al. (2020) Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun’ichi Tsujii.  Characterbert: Reconciling elmo and bert for word-level open-vocabulary representations from characters.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pp. 6903–6915, 2020.  URL https://aclanthology.org/2020.coling-main.609/.
  * Faisal et al. (2024) Fahim Faisal, Orevaoghene Ahia, Aarohi Srivastava, Kabir Ahuja, David Chiang, Yulia Tsvetkov, and Antonios Anastasopoulos.  DIALECTBENCH: A NLP benchmark for dialects, varieties, and closely-related languages, 2024.  URL https://arxiv.org/abs/2403.11009.  arXiv:2403.11009.
  * Geeraerts et al. (1994) Dirk Geeraerts, Stefan Grondelaers, and Peter Bakema.  _The structure of lexical variation: Meaning, naming, and context_.  Number 5. Walter de Gruyter, 1994.
  * Geeraerts et al. (2024) Dirk Geeraerts, Dirk Speelman, Kris Heylen, Mariana Montes, Stefano De Pascale, Karlien Franco, and Michael Lang.  _Lexical Variation and Change: A Distributional Semantic Approach_.  Oxford University Press, 2024.
  * Haber (1976) Lyn R. Haber.  Leaped and leapt: A theoretical account of linguistic variation.  _Foundations of Language_ , pp. 211–238, 1976.
  * Held et al. (2023) William Held, Caleb Ziems, and Diyi Yang.  TADA: Task agnostic dialect adapters for English.  In _Findings of the ACL_ , pp. 813–824, 2023.  URL https://aclanthology.org/2023.findings-acl.51/.
  * Hiraoka (2022) Tatsuya Hiraoka.  MaxMatch-Dropout: Subword regularization for WordPiece.  In _Proc. ICLR_ , pp. 4864–4872, 2022.  URL https://aclanthology.org/2022.coling-1.430/.
  * Hu et al. (2021) Edward J. Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  LoRA: Low-rank adaptation of large language models.  In _Proc. ICLR_ , 2021.  URL https://openreview.net/forum?id=nZeVKeeFYf9.
  * Ilbury (2020) Christian Ilbury.  “Sassy Queens”: Stylistic orthographic variation in Twitter and the enregisterment of AAVE.  _Journal of Sociolinguistics_ , 24(2):245–264, 2020.
  * Jain et al. (2022) Neel Jain, John Kirchenbauer, Jonas Geiping, and Tom Goldstein.  How to do a vocab swap? a study of embedding replacement for pre-trained transformers.  2022\.
  * Joshi et al. (2024) Aditya Joshi, Raj Dabre, Diptesh Kanojia, Zhuang Li, Haolan Zhan, Gholamreza Haffari, and Doris Dippold.  Natural language processing for dialects of a language: A survey, 2024.  URL https://arxiv.org/abs/2401.05632.  arXiv:2401.05632.
  * Kakharova (2021) Nilufar Nuridinovna Kakharova.  On the nature of linguistic variation and its types.  _Asian Journal of Research in Social Sciences and Humanities_ , 11(11):507–510, 2021.
  * Kantharuban et al. (2023) Anjali Kantharuban, Ivan Vulić, and Anna Korhonen.  Quantifying the dialect gap and its correlates across languages.  In _Findings of the ACL: EMNLP_ , pp. 7226–7245, 2023.  URL https://aclanthology.org/2023.findings-emnlp.481.pdf.
  * Kumar et al. (2020) Ankit Kumar, Piyush Makhija, and Anuj Gupta.  Noisy text data: Achilles’ heel of BERT.  In _Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)_ , pp. 16–21, 2020.  URL https://aclanthology.org/2020.wnut-1.3.pdf.
  * Loper & Bird (2002) Edward Loper and Steven Bird.  NLTK: the natural language toolkit.  In _Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics_ , pp. 63–70, 2002.  URL https://aclanthology.org/W02-0109/.
  * Loshchilov & Hutter (2018) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2018.  URL https://openreview.net/forum?id=Bkg6RiCqY7.
  * Neef (2009) Martin Neef.  Morphological variation: A declarative approach.  In _Describing and Modeling Variation in Grammar_ , pp. 117–133. Mouton de Gruyter Berlin, 2009.
  * Philippy et al. (2023) Fred Philippy, Siwen Guo, and Shohreh Haddadan.  Towards a common understanding of contributing factors for cross-lingual transfer in multilingual language models: A review.  In _Proc. ACL_ , 2023.  URL https://aclanthology.org/2023.acl-long.323.pdf.
  * Princeton University (2010) Princeton University.  About WordNet, 2010.  URL https://wordnet.princeton.edu.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners, 2019.  URL https://github.com/openai/gpt-2.
  * Soper et al. (2021) Elizabeth Soper, Stanley Fujimoto, and Yen-Yun Yu.  BART for post-correction of OCR newspaper text.  In _Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021)_ , pp. 284–290, 2021.  URL https://aclanthology.org/2021.wnut-1.31/.
  * Srivastava & Chiang (2023) Aarohi Srivastava and David Chiang.  Fine-tuning BERT with character-level noise for zero-shot transfer to dialects and closely-related languages.  In _Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023)_ , pp. 152–162, 2023.  URL https://aclanthology.org/2023.vardial-1.16.
  * Wu et al. (2022) Zhengxuan Wu, Isabel Papadimitriou, and Alex Tamkin.  Oolong: Investigating what makes crosslingual transfer hard with controlled studies, 2022.  URL https://arxiv.org/abs/2202.12312.  arXiv:2202.12312.
  * Yin et al. (2020) Fan Yin, Quanyu Long, Tao Meng, and Kai-Wei Chang.  On the robustness of language encoders against grammatical errors.  In _Proc. ACL_ , 2020.  URL https://aclanthology.org/2020.acl-main.310/.
  * Zanuttini & Horn (2014) Raffaella Zanuttini and Laurence Horn.  _Micro-syntactic Variation in North American English_.  Oxford University Press, 2014.
  * Ziems et al. (2023) Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang.  Multi-VALUE: A framework for cross-dialectal English NLP.  In _Proc. ACL_ , 2023.  URL https://aclanthology.org/2023.acl-long.44/.

##  Appendix A Appendix

|  | Interventions
---|---|---
Model | Data | None | IPA | Shift | Reg | Char | Pig |  $-$End | Multi | Affix | Hyp | Ant
BERT | 0 | 58.8 | 2.6 | 2.3 | 6.9 | 5.4 | 9.3 | 1.5 | 11.3 | 0.1 | 0.1 | 0.2
| S | 52.4 | 2.4 | 5.9 | 11.0 | 11.7 | 17.4 | 18.5 | 9.0 | 0.8 | 0.6 | 0.8
| M | 48.6 | 7.5 | 8.3 | 12.5 | 13.1 | 16.3 | 26.7 | 10.6 | 10.6 | 8.3 | 3.0
| L | 47.7 | 14.4 | 8.3 | 12.0 | 19.9 | 20.3 | 15.1 | 11.6 | 25.5 | 17.0 | 19.7
mBERT | 0 | 42.0 | 2.6 | 2.2 | 5.1 | 6.4 | 13.8 | 2.6 | 11.7 | 1.7 | 1.7 | 1.1
| S | 38.8 | 10.5 | 1.5 | 7.4 | 12.3 | 10.1 | 5.2 | 7.7 | 4.2 | 1.3 | 6.8
| M | 31.6 | 9.3 | 4.8 | 12.8 | 14.1 | 16.2 | 7.6 | 8.8 | 24.0 | 9.7 | 8.5
| L | 34.4 | 14.7 | 6.8 | 13.0 | 16.4 | 16.9 | 11.9 | 12.3 | 29.5 | 25.3 | 22.8

Table 4:
1-best accuracy results for all experiments with the mixed data composition.

|  | Interventions
---|---|---
Model | Data | None | IPA | Shift | Reg | Char | Pig | -End | Multi | Affix | Hyp | Ant
BERT | 0 | 62.0 | 0.1 | 0.0 | 12.1 | 0.0 | 0.0 | 1.6 | 12.6 | 0.1 | 0.0 | 0.2
| S | 55.1 | 0.3 | 0.0 | 9.9 | 0.5 | 0.0 | 34.2 | 11.6 | 0.1 | 1.8 | 3.2
| M | 51.3 | 2.4 | 2.3 | 12.5 | 1.2 | 0.2 | 30.7 | 14.9 | 3.2 | 10.1 | 18.1
| L | 50.3 | 9.8 | 2.2 | 10.2 | 7.3 | 7.5 | 39.0 | 15.4 | 25.2 | 31.2 | 31.5
mBERT | 0 | 44.8 | 1.2 | 0.9 | 6.8 | 0.9 | 1.4 | 2.9 | 11.8 | 0.5 | 1.7 | 0.8
| S | 40.7 | 1.2 | 0.4 | 1.8 | 0.8 | 0.9 | 25.5 | 7.5 | 0.4 | 7.0 | 10.4
| M | 33.6 | 2.7 | 0.2 | 9.9 | 8.1 | 2.8 | 33.3 | 14.8 | 9.0 | 13.5 | 17.1
| L | 37.0 | 5.3 | 0.8 | 8.5 | 8.9 | 7.7 | 32.2 | 14.2 | 31.0 | 27.7 | 30.5

Table 5:
Exact match accuracy results for all experiments with the full data
composition.

|  | Interventions
---|---|---
Model | Data | None | IPA | Shift | Reg | Char | Pig | -End | Multi | Affix | Hyp | Ant
BERT | 0 | 77.0 | 10.5 | 10.0 | 13.9 | 26.4 | 23.9 | 21.5 | 21.6 | 2.2 | 1.2 | 0.5
| S | 74.3 | 17.1 | 17.3 | 23.5 | 46.6 | 33.1 | 47.1 | 19.4 | 2.0 | 6.9 | 8.5
| M | 74.9 | 29.3 | 28.3 | 27.9 | 55.6 | 40.2 | 46.6 | 24.8 | 31.7 | 23.8 | 31.0
| L | 62.6 | 35.2 | 33.5 | 27.6 | 60.0 | 51.8 | 55.8 | 22.4 | 52.6 | 45.2 | 45.9
mBERT | 0 | 64.1 | 6.9 | 8.6 | 14.3 | 25.3 | 27.4 | 21.5 | 22.6 | 4.1 | 2.6 | 2.3
| S | 63.5 | 21.2 | 14.3 | 15.4 | 44.9 | 32.8 | 36.9 | 15.9 | 5.5 | 11.2 | 16.1
| M | 61.6 | 24.9 | 14.4 | 29.3 | 55.2 | 35.2 | 46.9 | 23.3 | 31.2 | 32.5 | 36.6
| L | 49.0 | 25.2 | 19.3 | 23.7 | 62.4 | 44.7 | 43.7 | 20.3 | 56.8 | 44.7 | 47.7

Table 6:
5-best accuracy results for all experiments with the full data composition.