# T2FPV: Constructing High-Fidelity First-Person View Datasets From Real-World
Pedestrian Trajectories

Benjamin Stoler1, Meghdeep Jana2, Soonmin Hwang3, and Jean Oh3 1Computer
Science Dept., Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, PA
<EMAIL_ADDRESS>Mechanical Engineering Dept., Carnegie Mellon University,
5000 Forbes Ave, Pittsburgh, PA<EMAIL_ADDRESS>Institute,
Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh,
<EMAIL_ADDRESS>
###### Abstract

Predicting pedestrian motion is essential for developing socially-aware robots
that interact in a crowded environment. While the natural visual perspective
for a social interaction setting is an egocentric view, the majority of
existing work in trajectory prediction has been investigated purely in the
top-down trajectory space. To support first-person view trajectory prediction
research, we present T2FPV, a method for constructing high-fidelity first-
person view datasets given a real-world, top-down trajectory dataset; we
showcase our approach on the ETH/UCY pedestrian dataset to generate the
egocentric visual data of all interacting pedestrians. We report that the
bird’s-eye view assumption used in the original ETH/UCY dataset, i.e., an
agent can observe everyone in the scene with perfect information, does not
hold in the first-person views; only a fraction of agents are fully visible
during each 20-timestep scene used commonly in existing work. We evaluate
existing trajectory prediction approaches under varying levels of realistic
perception—displacement errors suffer a $356\%$ increase compared to the top-
down, perfect information setting. To promote research in first-person view
trajectory prediction, we release our T2FPV-ETH dataset and software
tools444https://github.com/cmubig/T2FPV.

##  I INTRODUCTION

As more and more autonomous robots are anticipated to interact with people in
shared environments, trajectory prediction in robotics including navigation
among human crowds [2, 3, 4, 5, 6] and unmanned aerial vehicles [7] has become
increasingly popular in the research community, as well as among various
industry and military stakeholders. In particular, predicting pedestrian
motion is essential for developing socially-aware robots that interact in a
crowded environment. Existing state-of-the-art (SOTA) trajectory prediction
algorithms leverage datasets such as the ETH/UCY pedestrian dataset that
provide full trajectory information of all pedestrians in a bird’s-eye view
(BEV) scene [1]. However, bird’s-eye view is an unrealistic view for agents
navigating in the real-world; agents generally rely on egocentric, first-
person view (FPV) sensing for these tasks. A realistic setting also includes
limited field-of-view (FOV), occlusions, and changes in perspective and
orientation of the ego-agent.

Whereas it is relatively convenient to collect top-down data using an over-
head camera, creating a first-person view counterpart is far more challenging
due to several reasons. For instance, all participants in the scene would need
to wear a camera sensor to record their egocentric views, as well as a
location-recording sensor to establish their ground truth locations.
Furthermore, such a setting is subject to psychological issues such as the
observer (or Hawthorne) effect [8], where people’s behaviors in these
experiments may not be entirely representative of a natural social
interaction.

Existing real-world first-person view pedestrian datasets, such as [4],
generally do not include other agents’ ground truth world locations in the
scenes. Other synthetic first-person view datasets [5, 9] share a similar
intuition as ours—however, their synthesized images’ quality is low fidelity,
and they do not consider realistic perception. Our approach utilizes
SEANavBench [10], a flexible high-fidelity simulation environment. In
SEANavBench, to acquire first-person view of a scene, a new agent can be added
to an existing scene to observe pre-recorded pedestrians and navigate among
them. However, because the agents are part of a recorded scene, there is no
ground truth available for how they should react to the agent’s actions.

In this context, we propose Trajectory-to-First-Person Views (T2FPV), a method
for constructing a first-person view version of data from a trajectory-only
dataset by simulating the agents in high fidelity. Each agent follows their
recorded trajectory with a simulated camera attached to them. To showcase our
approach, we construct the T2FPV-ETH dataset based on the ETH/UCY trajectory
dataset [1] as shown in Fig. LABEL:fig:teaser.

We evaluate existing trajectory prediction approaches using a top-performing
detection and tracking algorithm to show that the displacement errors on
trajectory prediction increase by a large margin. As shown in Fig. 3, this is
an increase of more than 1 meter (a $356\%$ increase) compared to the top-
down, perfect information setting.

Our main contributions are: 1) we propose a method for creating an egocentric
view for each agent given a set of trajectories; 2) we generate the T2FPV-ETH
dataset, a first-person view dataset that corresponds to the ETH/UCY dataset;
3) we perform experiments with increasingly realistic perception, and report
the degraded performance of a top-performing trajectory prediction approach;
4) we show that layer normalization combined with social pooling generally
improves trajectory prediction performance, especially in the first-person
view setting; and 5) to promote research in first-person view trajectory
prediction, we release our dataset and software tools.

##  II RELATED WORK

Real-World First-Person Datasets. Various large-scale datasets provide video
footage from an ego agent’s perspective. [11] is a dataset of egocentric video
stream along with pose, acceleration and orientation information. [12] is a
large-scale first-person view video dataset, with over 3500 hours of footage
collected from various sources around the world. However, these datasets only
provide the single perspective of an ego agent in each scenario, and lack the
ground truth pose information and perspective of all other agents in the
scene. Egocentric Basketball Motion Planning [13] provides a wearable camera
perspective from multiple people in the scene. Nonetheless, these datasets are
not focused on social navigation. They feature many instances of the ego agent
walking by themselves or performing an unrelated task (such as carpentry,
basketball, etc.) that inherently have different social contexts than
navigating in public.

Synthetic Pedestrian Datasets and Simulation. Several recent works have
generated synthetic data in simulations based on a corresponding real-world
dataset. FvTraj [5] uses Unity to render FPV images from ground truth
trajectory data [14], but these rendered images consist only of a flat ground
plane, with no corresponding environment modeled. DeepSocNav [9] generates ego
view depth images from ETH/UCY, with a low-fidelity environment model.
However, they do not include the images from RGB cameras, which are far more
common than depth sensors. Furthermore, DeepSocNav and FvTraj do not release
any generated images or the in-house simulators. [15] and [10] are relatively
high-fidelity simulation environments with scene constructions of ETH/UCY,
built in Unreal Engine [16] and Unity [14] respectively, but both lack first-
person views. Additionally, none of these synthetic datasets account for
partial trajectories or realistic detection and tracking as a consequence of
occlusions and limited FOV.

Trajectory Prediction. Recent work on trajectory prediction has mostly focused
on top-down trajectory datasets such as ETH/UCY [1], SDD [17], and inD [18].
[2] uses LSTMs to jointly predict trajectories of all agents, incorporating
pooled hidden-state information from neighbors as a social cue. Some
approaches, such as AC-VRNN [19], use generative models within a VRNN [20],
with social interactions incorporated via attentive hidden state refinement.
[3] predicts socially plausible futures by training in an adversarial manner
against a recurrent discriminator. Several works also leverage top-down images
explicitly, whether in an RGB form or with added semantic segmentation [6, 21,
22]. SGNet [23] generates coarse step-wise goals to assist trajectory
prediction in a sequential manner. [24] incorporates agent dynamics and
environment information and forecasts using a graph-structured recurrent
model. [4] utilizes FPV to model and predict the trajectory directly in pixel-
space. [25] creates a spatial visual distribution of objects from FPV, and
applies perception and ego-agent trajectory planning in a 2.5D coordinate
system. Additionally, these FPV methods perform trajectory prediction on
either a single target or only the ego agent themselves.

##  III PROBLEM FORMULATION

A trajectory prediction problem using complete information is defined as
follows: for $N$ pedestrians in a scene, we denote the position of each agent
$i$ in the $xy$ ground-plane at time-step $t$ as $(x_{i}^{t},y_{i}^{t})$.
Given the observed track histories
$\\{(x_{i}^{t},y_{i}^{t})|t=1,2,...,T_{obs}\\}$, the task is to predict the
future paths $\\{(x_{i}^{t},y_{i}^{t})|t=T_{obs+1},T_{obs+2},...,T_{pred}\\}$
for all agents $i$ in a given scene.

In this paper, we introduce a trajectory prediction task where each agent is
to predict the trajectories of all agents in their view only using their
egocentric information. Formally, let $\phi_{i,t}$ denote agent $i$’s FPV
image at time step $t$ and $\Psi(i)$ denote the set of agents that are within
agent $i$’s field of view. Then, agent $j\in\Psi(i)$ if agent $i$’s views
$\\{\phi_{i,t}|t=t^{\prime},t^{\prime}+1,...,t^{\prime}+k\\}$ contain at least
some $P$ pixels associated with agent $j$, for some $k$ time steps. For each
agent $i$ in $1,...,N$, the FPV trajectory prediction task is to predict the
future trajectories of agent $i$ and all agents within agent $i$’s FOV, given
FPV observations, $\\{\phi_{i,t}|t=1,...T_{obs}\\}$, as well as their ego
track history.

##  IV TRAJECTORIES TO FIRST-PERSON VIEW

We describe how we construct first-person view data from a trajectory dataset,
using the ETH/UCY dataset as an example.

(a) Groundtruth

(b) Prediction

(c) Top-down view (green: groundtruth, blue: prediction)

Figure 2: Example of our detection and tracking results on Zara1; pedestrians
are assigned colors based on their tracked ID.

###  IV-A Video and Annotation Generation

Our approach for creating FPV datasets from real-world trajectory datasets
begins with generating videos and ground-truth annotations. We use the
SEANavBench [10] simulation environment as a starting point for our
simulation. SEANavBench consists of high-fidelity pre-modeled scenes for each
location within ETH/UCY. We leave these scenes as unchanged as possible, for
consistency with prior works using SEANavBench.

As in [5], we enforce a number of assumptions when rendering these tracks. For
instance, we orient each pedestrian’s gaze with the direction they are
traveling in, with spherical linear interpolation for smoother angle changes.
Additionally, we mount a camera on each pedestrian at a fixed height of $1.6m$
from their base, and assign the following physical characteristics to the
camera: $18mm$ focal length, $36\times 24mm$ sensor, and zero lens shift for
the principal point. When rendered at our $640\times 480$ resolution, this
results in a vertical FOV of approximately $67\degree$.

Using the above assumptions, we then render the first-person videos for every
person following their track from the original dataset, as well as output an
annotation for each agent at every frame. The videos consist of the RGB
render, as well as an instance segmentation render, as shown in Fig.
LABEL:fig:teaser, where each object in the scene has been given a unique
color. The annotations consist of the agent’s ID, pose information, and a list
of what other agents can be seen in the camera’s view, i.e., the poses of all
visible agents in both camera and world reference frame. This detection list
is generated by utilizing the aforementioned segmentation mask to determine
agent visibility.

###  IV-B Detection and Tracking

To assess the performance of SOTA trajectory prediction methods under a
realistic setting, we employed an off-the-shelf object detector and tracker to
produce the observations required for trajectory prediction. We used a 3D
object detector [26] which is SOTA among recent image-only methods which do
not require depth information [27], and a simple but effective probabilistic
tracker [28]. We made changes to both approaches to produce reasonable
detection and tracking results.

In DD3D [26], we set the parameters of feature map assignment to use
thresholds that fit our ground truths appropriately. We also only used
instances that are “visible” (as defined in Section IV-C2), which helps to
filter out heavily occluded instances. For the tracker [28], we changed the
matching metric to use BEV IoU (Intersection-over-Union in top-down view) from
Mahalanobis distance [29] to associate detections to tracks. We also applied
the Kalman filter only to each instance’s 3D location and orientation and used
state and observation noise covariances calculated from our ground truth data.

Following the common evaluation procedure as in the ETH/UCY trajectory
prediction task, we trained one model for each of the five folds, using the
other four folds as the training and validation sets respectively. We then
produced tracking results on all ego videos from each fold’s test-set.

###  IV-C Data Loading

We explain several variations of data loading used in our dataset. Let us
define a scene as a sequence of time steps where the agents in the environment
have contiguous, full information provided over the durations of observation
$T_{obs}$ and prediction $T_{pred}$, i.e., each 20-step scene in a sliding-
window approach with a single step stride. Let a tracklet refer to the portion
of the trajectory of an agent that is present in a scene.

We use the pre-processing as popularized in Social GAN [3], which consists of
agent tracks in world coordinates taken at 2.5 FPS. For mediated perception
approaches where vision-based approaches are used to pre-process FPV data into
trajectories, we provide two versions of imperfect trajectories, one with
synthetic noise and the other with vision-based detection and tracking.

Our dataset variations are defined as follows:

####  IV-C1 Bird’s-Eye View (BEV)

Following the data loading process from Social GAN [3], BEV scenes are
constructed as follows: in a potential scene, agents whose tracks start late,
end early, or have other missing intermediate data points are thrown out.
These scenes are then filtered to ensure that only scenes with at least two
agents’ tracklets are included in the BEV set. When loading a batch, indexing
is done at this scene-level granularity. In this set, agents’ visibility is
ignored. Thus, two assumptions are used in BEV: 1) agents that are present
partially during a scene are ignored, and 2) agents that are fully present
during a scene are perfectly visible to every agent even if they are
completely occluded to some agents.

####  IV-C2 First-Person View Ground Truth (FPV-GT)

In transitioning from BEV to FPV, given a scene with $N$ agents, we now
construct $N$ variations of the same scene, i.e., from each agent’s
perspective.

To take realistic constraints and limitations such as occlusions and limited
FOV, we redesign the scene as follows: First, we relax the full-duration
assumption from BEV and include all of the partially observed agents that
appear in at least $k$ of the first $T_{obs}$ time steps Second, using the
ground truth information as defined in Section IV-A, we only consider agents
that are truly visible in the first-person view, with at least $P$ pixels
visible. The number of tracklets in T2FPV-ETH, as seen in Table I, is based on
$k=3$ and $P=100$. By addressing the two limitations of BEV, we establish FPV
ground truth by emulating what a “perfect” first-person view detector and
tracker would produce.

####  IV-C3 FPV-Noisy

To examine what slightly noisy detection and tracking would cause, we also
provide a supplementary dataset by adding a small amount of synthetic noise
and other randomization to the ground truth visible agents, while leaving the
ego agent track untouched. The sources of noise are defined as follows: 1)
Each tracklet has a $1\%$ chance of being dropped; 2) Each visible bounding
box has a $10\%$ chance of being dropped from the tracklet; 3) At each
timestep in a detection sequence, there is a $2\%$ chance of the detected
agent being “lost”, i.e., assigned a new ID for subsequent frames; and 4) Each
visible agent location in meters has Gaussian noise with $\mu=0$ and
$\sigma=0.05m$ applied. Note that the corresponding ground truth tracklets in
the prediction phase ($t>T_{obs}$) have no such noise applied, even if the
observation was fully dropped.

####  IV-C4 FPV-Det

This version of the dataset is processed identically to FPV-GT, except in the
observation phase, the actual detection and tracking results from Section IV-B
is substituted in. Note again, that the prediction ground truth is still
unchanged from the original FPV-GT version.

###  IV-D Statistics

TABLE I: Total scenes and tracklets in each test-set. Fold | Scenes | Tracklets
---|---|---
BEV |  FPV-GT |  FPV-Noisy |  FPV-Det
ETH | 70 |  181 |  985 |  917 |  469
Hotel | 301 |  1.05k |  3.62k |  3.47k |  2.11k
Univ | 947 |  24.3k |  366k |  346k |  159k
Zara1 | 602 |  2.25k |  10.1k |  9.85k |  8.91k
Zara2 | 921 |  5.83k |  30.5k |  29.8k |  22.6k

####  IV-D1 BEV vs. FPV-GT

Table I provides a high-level overview of the number of scenes and tracklets
in each version of the T2FPV-ETH dataset, as created in Section IV-C. We note
that this table demonstrates a data augmentation effect when using all
perspectives in a BEV scene; a single ground-truth track is often observed by
multiple other agents at once.

####  IV-D2 FPV-Det

TABLE II: Detection and tracking performance. Fold | Detection | Tracking
---|---|---
$AP_{2D}$ |  $AP_{BEV}$ |  AMOTA |  AMOTP
ETH |  96.50 |  44.10 |  0.384 |  1.262
Hotel |  94.24 |  42.56 |  0.361 |  1.325
Univ |  90.65 |  67.56 |  0.318 |  1.465
Zara1 |  97.29 |  90.22 |  0.709 |  0.610
Zara2 |  94.67 |  73.78 |  0.517 |  1.000

Figure 3: Effect of different perception types on SGNet, A-VRNN, VRNN, and our
variations. As perception quality becomes more realistic, performance on all
metrics becomes significantly worse.

We measure the detection and tracking performances of the SOTA methods we
employed in Table II. For detection performance, we measure the standard
average precision ($AP_{2D}$) in 2D image space and observe that it performs
well. Also, we measure the localization quality of detected objects in 3D
space by calculating IoU-based average precision in the top-down view
($AP_{BEV}$). Both metrics use the same IoU threshold of 0.5. The $AP_{BEV}$
performance is worse than $AP_{2D}$, which shows the challenge of image-based
3D detection. For tracking, we adopt two popular metrics from [30], Average
Multi-Object Tracking Accuracy (AMOTA) and Precision (AMOTP). AMOTA combines
false positives, missed targets, and identity switches, and AMOTP measures the
misalignment between prediction and ground truth. Although “Univ” shows the
worst performance because of the pedestrian density (Table. I), the detector
and tracker perform reasonably well in most cases, as shown qualitatively in
Fig. 2.

##  V TRAJECTORY PREDICTION METHODS

###  V-A Baseline Approach Study

We implemented several representative approaches on the ETH/UCY trajectory
prediction task as baselines to examine task performance. We selected these
algorithms as they appear to stand out along several key techniques common in
human trajectory prediction, such as variational prediction, social awareness,
and goal conditioning. This approach led us to select VRNN [20], A-VRNN [19],
and SGNet [23] as our initial algorithms to examine.

Note that A-VRNN is an ablation of AC-VRNN, which adds in goal conditioning in
a somewhat similar manner to SGNet, but ultimately performs worse in reported
results; hence, we did not include it in our study. Additionally, while there
are performers on the ETH/UCY benchmark leaderboard which report performing
better than SGNet, as discussed in Section II, they generally rely on
additional input modalities beyond the scope of our task. Additional
approaches would certainly be interesting to incorporate and study as well,
but we leave this as future work extending beyond the scope of this study.

###  V-B Our Approach

To improve training stability, we employ layer normalization (LN) [31] after
each hidden layer within the Multi-Layer Perceptron (MLP) blocks. This was
motivated by works purporting its success in recurrent architectures,
especially in recent Transformer-based approaches [32].

We also re-examined the methods of social awareness used, beyond GAT sub-
networks [33]. This technique was especially important when trying to improve
SGNet [23], since their approach has no method of social awareness built in.
In particular, besides graph-based social methods, other works have attempted
pooling approaches [2, 3]. We thus created a version of social pooling that
incorporates lessons learned from GAT and hidden state-refinement approaches
[33]: use a similarity adjacency matrix, based on closeness to other people in
the scene at each timestep, to perform a weighted average pooling over others’
hidden states to refine each agent’s own hidden state.

##  VI EXPERIMENTS

TABLE III: ADE/FDE (mAP) for each fold, averaged over test-sets in Section IV-C. Algorithm | ETH | Hotel | Univ | Zara1 | Zara2 | Avg
---|---|---|---|---|---|---
A-VRNN | 1.31/2.37 (0.58) | 0.91/1.60 (0.63) | 1.09/1.74 (0.69) | 0.70/1.21 (0.82) | 0.83/1.38 (0.77) | 0.97/1.66 (0.70)
A-VRNN + LN | 1.23/2.11 (0.58) | 0.77/1.30 (0.68) | 0.95/1.46 (0.73) | 0.61/1.00 (0.84) | 0.69/1.06 (0.80) | 0.85/1.38 (0.73)
SGNet CVAE | 1.00/1.93 (0.67) | 0.51/0.80 (0.75) | 0.80/1.16 (0.76) | 0.49/0.75 (0.88) | 0.56/0.79 (0.83) | 0.67/1.08 (0.78)
SGNet CVAE + LN |  0.98/1.92 (0.69) |  0.49/0.78 (0.76) |  0.79/1.14 (0.77) |  0.47/0.71 (0.88) |  0.55/0.78 (0.84) |  0.66/1.07 (0.78)
SGNet CVAE + SocPool | 1.03/1.97 (0.69) | 0.51/0.79 (0.76) | 0.87/1.28 (0.75) | 0.48/0.74 (0.88) | 0.57/0.80 (0.83) | 0.69/1.12 (0.78)
SGNet CVAE + SocPool + LN |  0.98/1.76 (0.67) | 0.50/0.80 (0.76) | 0.80/1.16 (0.77) |  0.47/0.71 (0.88) | 0.56/0.79 (0.83) |  0.66/1.04 (0.78)
SGNet CVAE GAT | 1.07/2.02 (0.67) | 0.49/0.78 (0.76) | 0.82/1.18 (0.76) | 0.48/0.74 (0.88) | 0.58/0.82 (0.83) | 0.69/1.11 (0.78)
SGNet CVAE GAT + LN | 1.04/1.91 (0.66) | 0.50/0.79 (0.76) | 0.82/1.18 (0.76) | 0.48/0.73 (0.88) | 0.58/0.82 (0.83) | 0.68/1.09 (0.78)
VRNN | 1.39/2.29 (0.53) | 0.88/1.47 (0.64) | 1.03/1.64 (0.71) | 0.67/1.13 (0.83) | 0.77/1.24 (0.78) | 0.95/1.55 (0.70)
VRNN + LN | 1.21/2.15 (0.60) | 0.71/1.18 (0.70) | 0.96/1.47 (0.73) | 0.61/1.01 (0.83) | 0.70/1.06 (0.80) | 0.84/1.37 (0.73)
VRNN + SocPool | 1.41/2.34 (0.53) | 1.26/2.15 (0.55) | 1.06/1.67 (0.70) | 0.72/1.34 (0.82) | 0.73/1.15 (0.80) | 1.04/1.73 (0.68)
VRNN + SocPool + LN | 1.19/2.11 (0.60) | 0.69/1.11 (0.70) | 0.96/1.46 (0.73) | 0.60/0.97 (0.84) | 0.70/1.06 (0.80) | 0.83/1.34 (0.73)

###  VI-A Evaluation Procedure

As in Social GAN, we evaluate trajectory predictions using a leave-one-out
approach. For each of the five folds, models are trained and validated on data
from four of them at a time. Then, the best model according to validation
performance is tested on the entirety of the held-out fifth fold.

To examine how important perception quality is in trajectory prediction
accuracy, we utilize a transfer setting, where models are trained using the
ground truth scene groupings as in prior works (BEV), but then additionally
tested on the three FPV settings (FPV-GT, FPV-Noisy, and FPV-Det), as
described in Section IV-C.

###  VI-B Metrics

In the field of trajectory prediction, and especially for ETH/UCY, the most
commonly used metrics are Average Displacement Error (ADE) and Final
Displacement Error (FDE). These metrics can be easily computed on a per-agent
basis, for ground truth future track
$\\{(x_{i}^{t},y_{i}^{t})|t=T_{obs+1},T_{obs+2},...,T_{pred}\\}$ and predicted
future track
$\\{(\hat{x_{i}}^{t},\hat{y_{i}}^{t})|t=T_{obs+1},T_{obs+2},...,T_{pred}\\}$,
for each agent $i$ in the scene. The L2-distance at each time $t$ is taken
between $(x_{i}^{t},y_{i}^{t})$ and $(\hat{x}_{i}^{t},\hat{y}_{i}^{t})$; ADE
is the average of these distances, while FDE is the final distance, at time
$T_{pred}$.

However, since we consider detection and tracking to be a core part of the
challenge posed by this dataset, our metrics must extend beyond ADE/FDE and
incorporate a sense of precision and recall regarding which agents should have
been observed and predicted in the first place. This is made even more
challenging by the fact that there is no guarantee of alignment of agent IDs
between the observed tracks and ground truth tracks, even if the precision and
recall is perfect. To account for these issues, we use a variant of mean
Average Precision [34] (mAP) for trajectory prediction, using ADE as a stand-
in for both confidence and match quality.

Note that all approaches we consider utilize a “best-of-$K$” sample prediction
strategy. This accounts for the fact that there are multiple socially valid
predictions for agents in a scene, so feasible (but ultimately incorrect)
predictions should be punished less. As a result, ADE and FDE are computed in
a $K$-to-one manner, for each sample, and are reduced to their minimum value
before being passed to downstream mAP computation. We note that there are
other metrics which could be utilized, including collision rate, social
comfort level, path complexity, and many more [35]. We chose to focus on the
core metrics of the tasks at hand, but suggest that future work in applying
these metrics could provide helpful new insight.

##  VII RESULTS

The ADE, FDE and mAP performance is shown for each fold in Table III and for
each dataset variation in Fig. 3—note that the $x$-axis is categorical, and
points are connected only for improved visibility. In general, the FPV-GT
version of the dataset proves to be a moderately hard setting, with an average
ADE increase for the best performing SGNET variant of 14.8% for ADE. We
hypothesize that this is mostly due to models having to predict partially
observed detections, based on FOV and occlusion filtering. With the addition
of a small amount of noise in FPV-Noisy, as well as actual FPV-Det detection
and tracking results, the drop in performance is quite significant, i.e.,
92.9% and 356% increase in ADE relative to BEV respectively on the top
performing approach. All algorithms tested demonstrated similar behavior when
evaluated on the different test-sets.

Regarding our proposed additions to existing approaches, applying LN seems to
help near uniformly compared to each model’s non-LN counterpart. We suspect
this could be due to the trajectory prediction methods being highly sensitive
to hyper-parameter selection during training, and LN helps to “smooth” these
dynamics [31]. Additionally, incorporating spatially-weighted average pooling
(social pooling) seems to improve performance as well, in most cases more so
than GAT sub-networks. One possible explanation for this is the added
complexity of training GATs, due to their increase in resulting model size.
Ultimately, we were able to noticeably improve upon our SOTA baseline
selection of SGNet by incorporating these techniques. The improvement is most
significant in FDE, indicating stronger long-term prediction capabilities.
Note that mAP seems to be affected only marginally by algorithm variations, as
they all share the same detection and tracking module. Approaches which
utilize an end-to-end, rather than mediated, perception approach may exhibit
more dramatic changes.

##  VIII LIMITATIONS

Although SEANavBench is a high-fidelity environment, we do note that further
effort in improving its realism could be useful. Realism could be enhanced not
just by increasing the 3D-modeling asset and animation qualities, but also by
further improving alignment between the reproduced scenery and the original
locations. In particular, since we believe that performing Sim2Real transfer
experiments on our tasks would be an interesting future direction, the
resulting Sim2Real gap might be improved by increasing T2FPV quality.

Furthermore, we believe that significantly better results could be achieved by
using models which leverage first-person view more explicitly, such as by
incorporating image features, e.g. ResNet embeddings[36]. It could also be a
fruitful direction to evaluate models which are trained directly on FPV-GT or
FPV-Det, rather than our transfer setting.

##  IX CONCLUSION

In existing work, pedestrian trajectory prediction has been mainly studied
under a complete information assumption. In this paper, we introduce a first-
person view trajectory prediction problem where agents would need to make
predictions based on partial, imprecise information. To promote this research
direction, we present T2FPV, a method for generating high-fidelity egocentric
views for pedestrian navigation by leveraging existing real-world trajectory
datasets. Our experiments show that top-performing baselines suffer a
substantial decrease in performance in our proposed first-person view setting
using a SOTA detection and tracking approach—ADE relative to performance when
tested in the complete information, bird’s-eye view setting increases by 356%.
We also report that using layer normalization and social pooling generally
improves the prediction performance, especially in the first-person view
setting. Our constructed T2FPV-ETH dataset provides a benchmark for trajectory
prediction and pedestrian detection and tracking tasks in a more natural and
realistic setting. We argue that this is an important direction to move toward
in enabling robots to navigate in the real world. To promote further research
in first-person view trajectory prediction, we release our dataset and
software tools to the public.

## References

  * [1] A. Ess, B. Leibe, K. Schindler, , and L. van Gool, “A mobile vision system for robust multi-person tracking,” in _IEEE/CVF Computer Vision and Pattern Recognition (CVPR)_ , 2008.
  * [2] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese, “Social lstm: Human trajectory prediction in crowded spaces,” in _IEEE/CVF Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [3] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi, “Social GAN: socially acceptable trajectories with generative adversarial networks,” _CoRR_ , 2018.
  * [4] T. Yagi, K. Mangalam, R. Yonetani, and Y. Sato, “Future person localization in first-person videos,” _CoRR_ , 2017.
  * [5] H. Bi, R. Zhang, T. Mao, Z. Deng, and Z. Wang, “How can i see my future? fvtraj: Using first-person view for pedestrian trajectory prediction,” in _European Conference on Computer Vision (ECCV)_ , 2020.
  * [6] J. Yue, D. Manocha, and H. Wang, “Human trajectory prediction via neural social physics,” 2022.
  * [7] J. Patrikar, B. Moon, J. Oh, and S. Scherer, “Predicting like a pilot: Dataset and method to predict socially-aware aircraft trajectories in non-towered terminal airspace,” 2021.
  * [8] B. Irfan, J. Kennedy, S. Lemaignan, F. Papadopoulos, E. Senft, and T. Belpaeme, “Social psychology and human-robot interaction: An uneasy marriage,” 2018.
  * [9] J. P. de Vicente and A. Soto, “Deepsocnav: Social navigation by imitating human behaviors,” _CoRR_ , 2021.
  * [10] N. Tsoi, M. Hussein, O. Fugikawa, J. D. Zhao, and M. Vázquez, “An approach to deploy interactive robotic simulators on the web for hri experiments: Results in social robot navigation,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2021.
  * [11] A. A. E. Krishna Kumar Singh, Kayvon Fatahalian, “Krishnacam: Using a longitudinal, single-person, egocentric dataset for scene understanding tasks,” in _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2016.
  * [12] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane, T. Do, M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q. Fu, C. Fuegen, A. Gebreselasie, C. Gonzalez, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolar, S. Kottur, A. Kumar, F. Landini, C. Li, Y. Li, Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. R. Puentes, M. Ramazanova, L. Sari, K. Somasundaram, A. Southerland, Y. Sugano, R. Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Y. Zhu, P. Arbelaez, D. Crandall, D. Damen, G. M. Farinella, B. Ghanem, V. K. Ithapu, C. V. Jawahar, H. Joo, K. Kitani, H. Li, R. Newcombe, A. Oliva, H. S. Park, J. M. Rehg, Y. Sato, J. Shi, M. Z. Shou, A. Torralba, L. Torresani, M. Yan, and J. Malik, “Ego4d: Around the World in 3,000 Hours of Egocentric Video,” in _IEEE/CVF Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [13] G. Bertasius, A. Chan, and J. Shi, “Egocentric basketball motion planning from a single first-person image,” in _IEEE/CVF Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * [14] A. Juliani, V. Berges, E. Vckay, Y. Gao, H. Henry, M. Mattar, and D. Lange, “Unity: A general platform for intelligent agents,” _CoRR_ , 2018.
  * [15] J. Liang, L. Jiang, K. P. Murphy, T. Yu, and A. G. Hauptmann, “The garden of forking paths: Towards multi-future trajectory prediction,” _CoRR_ , 2019\.
  * [16] Epic Games, “Unreal engine.” [Online]. Available: https://www.unrealengine.com
  * [17] A. Robicquet, A. Sadeghian, A. Alahi, and S. Savarese, “Learning social etiquette: Human trajectory understanding in crowded scenes,” in _European Conference on Computer Vision (ECCV)_ , 2016.
  * [18] J. Bock, R. Krajewski, T. Moers, S. Runde, L. Vater, and L. Eckstein, “The ind dataset: A drone dataset of naturalistic road user trajectories at german intersections,” _CoRR_ , 2019.
  * [19] A. Bertugli, S. Calderara, P. Coscia, L. Ballan, and R. Cucchiara, “Ac-vrnn: Attentive conditional-vrnn for multi-future trajectory prediction,” _Computer Vision and Image Understanding_ , 2021.
  * [20] J. Chung, K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio, “A recurrent latent variable model for sequential data,” _CoRR_ , 2015.
  * [21] K. Mangalam, Y. An, H. Girase, and J. Malik, “From goals, waypoints & paths to long term human trajectory forecasting,” _CoRR_ , 2020.
  * [22] C. Wong, B. Xia, Z. Hong, Q. Peng, and X. You, “View vertically: A hierarchical network for trajectory prediction via fourier spectrums,” _arXiv preprint arXiv:2110.07288_ , 2021.
  * [23] K. Li, N. Y. Wang, Y. Yang, and G. Wang, “Sgnet: A super-class guided network for image classification and object detection,” _CoRR_ , 2021.
  * [24] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone, “Trajectron++: Multi-agent generative trajectory forecasting with heterogeneous data for control,” _CoRR_ , 2020.
  * [25] H. S. Park, J.-J. Hwang, Y. Niu, and J. Shi, “Egocentric future localization,” June 2016.
  * [26] D. Park, R. Ambrus, V. Guizilini, J. Li, and A. Gaidon, “Is pseudo-lidar needed for monocular 3d object detection?” in _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * [27] X. Ma, W. Ouyang, A. Simonelli, and E. Ricci, “3d object detection from images for autonomous driving: a survey,” _arXiv preprint arXiv:2202.02980_ , 2022\.
  * [28] H. kuang Chiu, A. Prioletti, J. Li, and J. Bohg, “Probabilistic 3d multi-object tracking for autonomous driving,” _arXiv, vol. abs/2001.05673_ , 2020.
  * [29] _Mahalanobis Distance_ , 2008.
  * [30] X. Weng, J. Wang, D. Held, and K. Kitani, “3d multi-object tracking: A baseline and new evaluation metrics,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020.
  * [31] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” 2016.
  * [32] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _CoRR_ , 2017\.
  * [33] P. Zhang, W. Ouyang, P. Zhang, J. Xue, and N. Zheng, “SR-LSTM: state refinement for LSTM towards pedestrian trajectory prediction,” _CoRR_ , 2019.
  * [34] L. LIU and M. T. ÖZSU, Eds., _Mean Average Precision_ , 2009.
  * [35] C. I. Mavrogiannis, F. Baldini, A. Wang, D. Zhao, P. Trautman, A. Steinfeld, and J. Oh, “Core challenges of social robot navigation: A survey,” _CoRR_ , 2021.
  * [36] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” _CoRR_ , 2015.


# Do language models learn typicality judgments from text?

Kanishka Misra1<EMAIL_ADDRESS>
Allyson Ettinger2<EMAIL_ADDRESS>
Julia Taylor Rayz1<EMAIL_ADDRESS>
1Department of Computer and Information Technology, Purdue University, IN, USA
2Department of Linguistics, University of Chicago, IL, USA

###### Abstract

Building on research arguing for the possibility of conceptual and categorical
knowledge acquisition through statistics contained in language, we evaluate
predictive language models (LMs)—informed solely by textual input—on a
prevalent phenomenon in cognitive science: typicality. Inspired by experiments
that involve language processing and show robust typicality effects in humans,
we propose two tests for LMs. Our first test targets whether typicality
modulates LM probabilities in assigning taxonomic category memberships to
items. The second test investigates sensitivities to typicality in LMs’
probabilities when extending new information about items to their categories.
Both tests show modest—but not completely absent—correspondence between LMs
and humans, suggesting that text-based exposure alone is insufficient to
acquire typicality knowledge.

Keywords: typicality; neural networks; language models; conceptual knowledge
representation

Disclaimer: This is a preprint of an accepted manuscript, which will be
published in Proceedings of the 43rd Annual Conference of the Cognitive
Science Society. (Vienna; held online). The final version is available online
at: url TBD.

## Introduction

Perhaps one of the most robust findings in the study of human categorical
knowledge is the phenomenon of typicality, the observation that certain
members of a category are considered to be more representative of the category
than others (?, ?). As observed in the seminal work of ? (?), native English
speakers rate robins and canaries as more typical birds than penguins and
emus, chairs and sofas as more typical furniture than clocks and vases, etc.
Typicality differences in stimuli strongly predict response times in taxonomic
sentence verification tasks (?, ?, ?) and category production (?, ?). In the
context of learning, typical items facilitate faster concept acquisition than
do atypical items (?, ?). Typicality also prominently affects category-based
induction (?, ?, ?): that is, subjects more readily extend new information
about typical—as opposed to atypical—items to the entire category. In summary,
typicality is a salient and impactful phenomenon in the study of human
category knowledge.

There is a growing body of research on the view that words or language in
general act as distributional cues to categorical knowledge, as opposed to
mappings onto concepts—that statistics contained in language can, to an
extent, inform about the world (?, ?). This view is supported by recent
evidence in natural language processing (NLP) and cognitive science that shows
encouraging signs of computational models learning world (?, ?), categorical
(?, ?), and conceptual (?, ?) knowledge while relying solely on text-based
input. Though these works investigated knowledge of categories (through word
prediction-based categorization prompts such as “A robin is a  .”), they do
not consider any distinction between central and peripheral members of
categories.

Expanding on the aforementioned promising results related to conceptual and
categorical knowledge, we ask the question: “How much do the statistical
associations contained in text reflect typicality effects in categories?” To
this end, we present a case-study on language models (LMs) that are pre-
trained on massive amounts of text and learn representations that are
optimized to reflect the statistics of the language used in textual-form. We
investigate whether the phenomenon of typicality emerges as a result of LM
pre-training. Our tests are grounded in the psychological study of concepts
and categories, and are inspired by prior human experiments that show clear
sensitivities to typicality in processing of textual stimuli. First, we build
on prior work analyzing conceptual and categorical knowledge in LMs, and test
whether typicality effects modulate LM judgments of taxonomic sentence
verification (“a robin is a bird”) as they do in humans (?, ?, ?).
Complementing this simple and direct test of taxonomic category membership, we
add a layer of complexity, and investigate the manifestation of typicality
effects in LMs on the basis of how they extend new information about items
(“robins can dax”) to all members of a category (“all birds can dax”),
inspired by tests targeting psychological strength of inductive arguments (?,
?). Though the human experiments that inspire our tests do not explicitly
target typicality as a phenomenon, typicality effects still robustly modulate
human behavior on them. Hence, we examine whether LMs show comparable
typicality effects on stimuli similar to those used in the above experiments.

We find non-trivially positive but modest sensitivities of LMs to typicality
effects in both our experiments. We also find LMs, on average, to be less
extreme in their sensitivities to atypical and typical items as compared to
humans. This suggests that the word prediction capacities of LMs that are
optimized to reflect the statistics that are contained in textual corpora are
moderately influenced by typicality effects in assessing strength of simple
taxonomic verification as well as more complex inductive inferences about
categories. Our results reflect the difficulty of acquiring human-like
category knowledge without extra-linguistic input, at least with the current
computational models of language processing.

## Materials and Methods

### Models Studied

We conduct our analyses on pre-trained LMs based on the transformer
architecture (?, ?). Our choice of LMs is motivated by recent evidence that
shows qualitative alignment of category knowledge (“a robin is a bird”, “a
bear has fur, has claws.”) in pre-trained LMs (?, ?, ?). Although we focus on
a particular type of pre-trained LMs (transformers) in this paper, the tests
we propose can be applied to any LM. We investigate two broad classes of
transformer-based pre-trained LMs: (1) Incremental LMs, trained
autoregressively (left to right) to predict one word at a time, when
conditioned on exclusively the left context; and (2) Masked LMs, that access
context of the word to be predicted bidirectionally, e.g., the models are
optimized to predict correct completions (airplane or bird) to sentences such
as “the [mask] flew away,” where [mask] represents the hidden word. We apply
our tests on GPT (?, ?) and GPT2 (?, ?) as our Incremental LMs, and ALBERT (?,
?), ELECTRA (?, ?), BERT, (?, ?) and RoBERTa (?, ?) as our Masked LMs. In
addition, we use compressed versions of the above models (?, ?): distilGPT2,
distilBERT-base, and distilRoBERTa-base. All transformer-based pre-trained LMs
were accessed using the transformers library (?, ?).

Finally, we also used a 5-gram language model with kneyser-ney smoothing,
trained using the KenLM toolkit (?, ?), as a baseline model that lacks the
kind of representational learning mechanisms that empower the above models.
This model is trained on the Dec, 2020 dump of English
Wikipedia.111https://dumps.wikimedia.org/enwiki/20201220/ The performance of
the 5-gram model represents the extent to which our tests can be approximated
simply by memorizing sequences of up to 5 words in length.

### Data and Stimuli

#### Item typicality data

For both experiments, we use as our primary source the list of 565 item-
typicality ratings compiled by ? (?) across 10 different categories. In the
original human experiments, 209 native speakers of English were tasked to rate
the “goodness of example” for various items of each given category, on a scale
of 1 (most typical) to 7 (least typical). The statistics of the items and
categories is presented in Table 1. It should be noted that the experiments we
base our tests on involve sensitivities to typicality measured using different
quantities (response times and raw typicality ratings), but make none or only
a small subset of results available. Therefore, we use the ? (?) ratings as
the common “ground-truth” typicality ratings for our experiments.

#### Stimuli Setup

Because the models we investigate are sentence processors, and because all of
our tests involve propositions about items and categories expressed as
sentences, we rely on using sentence stimuli in our experiments. Every
stimulus consists of two components: (1) condition, which is a noun
phrase/sentence consisting of the item (robins, sparrows, eagles, etc.); and
(2) predicted material, which consists of the super-ordinate category (bird).
The exact linguistic format in which it appears depends on the experiments —
we use single words as the predicted material in our Taxonomic Sentence
Verification experiment while for our Category-based induction experiment we
use a full sentence as our predicted material. In evaluating typicality
measurements of various items for a given category, the predicted material
remains constant, while the condition changes depending on the item. Table 2
shows examples of stimuli we use in each of our experiments.

Table 1: Number of items (N) per category (?, ?). Category | N | Category | N
---|---|---|---
furniture | 60 | vegetable | 56
tool | 60 | clothing | 55
toy | 60 | bird | 54
weapon | 60 | fruit | 51
sport | 59 | vehicle | 50

### Measures

Following precedent set by previous work evaluating conceptual knowledge in
pre-trained LMs, we use the models’ probability estimates as our main variable
of interest. Specifically, we focus on the log-probability of the word or
statement represented in the predicted material, given the condition, $\log
p_{\textit{LM}}(\textrm{predicted}\mid\textrm{condition})$, i.e., we are
measuring the effect on the probability of the predicted part (held constant
for a given category) due to the item mentioned in the condition. Our reason
for separating the item from the predicted material is two-fold: (1) it avoids
skewed measurements due to the choice of determiner (_a_ vs _an_) that
precedes the item in the condition (a model might assign higher value to
$p(\textit{ostrich}\mid\textit{an})$ simply due to a component that is
sensitive to determiner prefixes), or when the model does not include the item
word in its vocabulary,222E.g., RoBERTa segments the word ostrich into ostr
and ich, and during estimation, the probability of ich given that it is
preceded by ostr is anomalously high, skewing the overall sequence
probability. and (2) it aids in factoring out the role played by the frequency
of the item in the condition – the model can prefer an item over the other
simply due to its frequency in the training corpus. While it is
straightforward to compute our conditional probability measure for incremental
LMs by using the chain-rule, we rely on recent work by ? (?) to approximate
sequence log-probabilities in Masked LMs by summing the conditional log-
probabilities of all words in the stimuli.

Figure 1: (A) Spearman correlation ($\rho$) measured between LM log-
probabilities assigned to word completion in taxonomic stimuli (experiment 1)
and typicality ratings from ? (?). Models from the same family are arranged in
an increasing order of total number of parameters. (B) Scaled typicality
scores from LMs (log-probabilities on taxonomic stimuli) and Humans (raw
ratings) between low and high typicality category members.

## Experiments

### 1\. Taxonomic Sentence Verification

#### Phenomenon

Typicality effects in the sentence verification paradigm were introduced by ?
(?) and ? (?). Subjects were tasked with verifying the truth of sentences
expressing taxonomic propositions, such as “An $X$ is a $Y$”—where $X$ and $Y$
are the item and category, respectively. The subjects consistently responded
faster to verifying the truth of propositions where $X$ was a typical member
of $Y$ than when it was an atypical one.

#### Linking Phenomenon to LMs

We draw on the aforementioned findings and investigate whether typicality is
able to account for difference in the word probabilities to complete taxonomic
sentences by our tested LMs. Linking our hypothesis to the original experiment
requires a simplifying assumption that an LM’s sequence log-probability is
proportional to its plausibility for a sequence. That is, we assume and expect
a semantically sound LM to show overall high probability scores for
semantically plausible propositions, which in this case, are simple taxonomic
propositions333However, we acknowledge that this might not always be the case.
For instance, LMs are largely insensitive to negation and semantic role-
reversal (?, ?).. Therefore, LMs that are more sensitive to typicality effects
should show greater magnitudes for the measure $\log
p_{\textit{LM}}(Y\mid\textit{An }X\textit{ is a})$ when $X$ is a more typical
member of $Y$.

Table 2: Examples of stimuli used in our experiments. Our measures take the form: $\log p(\textrm{predicted}\mid{\color[rgb]{0,0.3125,0.51171875}\textrm{condition}})$ Experiment |  Stimulus
---|---
Taxonomic Sentence Verification |  ${\color[rgb]{0,0.3125,0.51171875}\underbrace{\textrm{A robin is a}}_{\textrm{condition}}}\underbrace{\textrm{bird}}_{\textrm{predicted}}$.
Category-based Induction |  ${\color[rgb]{0,0.3125,0.51171875}\underbrace{\textrm{Saws can dax.}}_{\textrm{condition}}}\underbrace{\textrm{All tools can dax.}}_{\textrm{predicted}}$

#### Experiment

We follow ? (?) and ? (?) and construct sentences expressing taxonomic
propositions using items from the ? (?) data, i.e., “An $X$ is a $Y$,”
amounting to 565 unique propositions. We test for typicality effects by
measuring the Spearman correlation ($\rho$) of the sequence log-probability
$\log p_{\textit{LM}}(Y\mid\textit{An }X\textit{ is a})$ with the human
typicality ratings for items, as collected by ? (?). This correlation measure
reflects the extent to which the predictive estimates of an LM reflect
typicality information—or information that underlies it—to assess taxonomic
verification in sentences. Additionally, we perform a median split on the ?
(?) ratings by the items’ typicality ratings, per category, leaving us with
two sets of typical and atypical ratings. We then compute the average log-
probabilities assigned to items in each set and compare them to the average
ratings elicited by humans. All scores in this analysis are re-scaled to be
between 0 and 1.

#### Results

Figures 1A and 1B show results from our correlation and typicality-effect
comparisons. Non-trivially positive but modest correlations between LM log-
probabilities and human typicality ratings ($\rho\in$ [0.24, 0.41], $p<$ .001)
suggest that LMs’ judgments of taxonomic propositions are moderately
reflective of typicality effects. Though all LMs assign greater probability
scores to category items with high—as compared to low—typicality (see Figure
1B), they are consistently less extreme as compared to humans ($p<$ .001
across all models). Correlation of 5-gram LM log-probabilities, though weakest
in magnitude, are highly competitive with certain smaller yet highly
expressive LMs (ALBERT-b, ALBERT-xl, distilGPT2, and distilRoBERTa). This
suggests that a substantial portion of the observed correspondence between
model and human typicality judgments can be attributed to fairly simpler
sequential statistical effects in word prediction (e.g. memorizing n-grams).
Interestingly, with the exception of the ALBERT family of Masked LMs, models
with greater number of parameters tend to show greater correspondence with
humans on taxonomic judgments ($\rho=0.82,p<.001$), suggesting that the
information needed to distinguish typical vs. atypical category members during
taxonomic attribution requires greater model expressivity.

### 2\. Category-based Induction

#### Phenomenon

Typicality of items plays a salient role in making inductive inferences about
categories (?, ?), i.e., when informed about a member $m$ of a category $c$
having a novel property $\gamma$, people are more likely to extend the
presence of $\gamma$ to all members of $c$ when $m$ is typical or central to
$c$. This was more robustly illustrated by ? (?) in their study exploring the
psychological strength of categorical inductive arguments. An argument is a
finite set of sentences of the form
$\mathcal{P}_{1},\mathcal{P}_{2},...,\mathcal{P}_{n}/\mathcal{C}$, where
$\mathcal{P}_{1},\mathcal{P}_{2},...,\mathcal{P}_{n}$ are the argument’s
premises and $\mathcal{C}$ is its conclusion. In categorical arguments,
$\mathcal{P}$ and $\mathcal{C}$ take the form “All members of cat have
property $\gamma$,” where cat is a natural category such as car or sofa, and
the property $\gamma$ remains constant across $\mathcal{P}$ and $\mathcal{C}$.
Such arguments can be visualized by separating the premises and conclusions by
a horizontal line, like in (1) and (2). The psychological strength of
inductive arguments, for a subject $\mathcal{S}$, is the degree to which
$\mathcal{S}$’s belief in $\mathcal{P}$ strengthens their belief in
$\mathcal{C}$ (?, ?).

Robins have property $\gamma$.
---
All birds have property $\gamma$.
(1) Penguins have property $\gamma$.
---
All birds have property $\gamma$.
(2)

Unlike deductive arguments, which involve logical reasoning, inductive
arguments such as (1) and (2) involve probabilistic reasoning, i.e., there is
an epistemic uncertainty whether the conclusions follow from the given premise
(for a detailed review, see ?, ?). A caveat in the Osherson et al. experiments
is that the property space $\Gamma=\\{\gamma_{1},...,\gamma_{n}\\}$ only
includes properties that are unfamiliar to $\mathcal{S}$, such that the
influence of prior knowledge about the properties on the induction process is
minimal. Such properties are also known as blank predicates — for instance, ?
(?) use properties such as love onions, have sesamoid bones, etc.

Typicality effects are one of the 13 phenomena examined by ? (?).
Specifically, for single-premise arguments where the category of the
conclusion subsumes that of the premise, subjects were more likely to believe
in the conclusion when the category of the premise was a more typical member
of the category in the conclusion, i.e., the argument strength of (1) was
found to be greater than that of (2) since robins are more typical birds as
compared to penguins.

#### Linking Phenomenon to LMs

The ? (?) study explicitly targets the degree to which uncertain statements
such as “all birds love onions” are judged in light of new information about a
subordinate category such as robins. Analogously, we are interested in
assessing whether sophisticated LMs show similar behavior in assigning
probabilities to conclusions when conditioned on premises whose categories
vary based on their typicality. If LMs show sensitivity to the typicality of
items in this setting, i.e., their log-probability is greater for conclusions
with typical versus atypical premise, then we take this as the extent to which
typicality—or the factors that underlie it—modulates inductive inference in
LMs. We formulate an approximation of inductive argument strength ($AS$) in an
LM as the probability it assigns to the conclusion when conditioned on a given
premise. For instance $AS(\textit{robin},\textit{bird})$ for the property
“love onions” is given by:

$\displaystyle\log p_{\textit{LM}}(\textit{``All birds love
onions.''}\mid\textit{``Robins love onions.''})$

The premise and conclusion sentences naturally fit within our stimulus setup
discussed earlier — the premise sentence is the condition, and the conclusion
sentence the predicted material.

Figure 2: (A) Spearman correlation ($\rho$) measured between average $AS$
scores and human typicality ratings compiled by ? (?). Models from the same
family are arranged in an increasing order of total number of parameters. (B)
Scaled typicality scores from LMs ($AS$ values) and Humans (raw ratings)
between low and high typicality category members.

#### Experiment

For our items and categories we use the same stimuli from the previous
experiment. Since ? do not make all of their blank predicates available, we
construct synthetic properties using nonce words such as dax, wugs, feps,
vorpal, etc., such that these words do not occur in the vocabulary444Due to
their tokenization mechanism, the LMs we study are always able to encode any
text through ‘word pieces’ instead of relying on <unk> tokens. of the models,
conforming to the blank predicate condition applied by ? (?) and ? (?). We
create between 15 to 30 properties555The choice of properties depends largely
on the class of word the items belong to, such that syntactic constraints are
met. For instance, if dax is a verb, it would be ungrammatical to have “can
dax” as a property of sports, which can be better paired with properties such
as “involve” and “require”. The entire unique set of synthetic properties and
our construction method is made available in our supplementary materials. for
all items in each category, resulting in a total of 12,180 premise-conclusion
pairs across 10 categories. An example of the stimuli we use for our category-
based induction task is shown in Table 2. We calculate the $AS$ metric for
each premise-conclusion pair with each of our tested LMs.

Conditioning our LMs as we do here has two potential confounds: (1) Premise
Order Sensitivity (POS): A model might estimate high probabilities for words
in the conclusion simply because it is relying on lexical cues in its premise
(?, ?), instead of processing the premise compositionally and making
inferences about items possessing a property. We account for this confound by
also computing the LMs’ average probability for the conclusion sentence when
prefixed by a shuffled version of the premise (10 times, with random seeds).
(2) Taxonomic Sensitivity (TS): LMs might tend to repeat the property phrase
mentioned in the predicted material with high probability when prefixed by a
sentence containing it, i.e., repeating “can dax” in the conclusion when
already conditioned on the same phrase in the premise, confounding the degree
to which the conclusion is generated using the taxonomic relationship between
the premise and the conclusion categories. To account for this tendency, we
compute the LMs’ probabilities for conclusions consisting of a different
category with the exact same property as the original (for instance, “All
fruits are slithy” given “Sofas are slithy”).

We find that a substantial amount of variance in our original $AS$ scores is
in fact captured by both these confounds (overall $r^{2}$ = 0.43,
$\beta_{\textsc{ts}}=0.68,\beta_{\textsc{pos}}=-0.04,p<.0001$ in both cases).
We regress these relationships out from our $AS$ scores by first fitting a
multiple regression model to predict $AS$ using our confounds, and then
subtracting the relationship with TS and POS as follows:

$\displaystyle AS$
$\displaystyle=\beta_{0}+\beta_{1}\textrm{TS}+\beta_{2}\textrm{POS}+\epsilon$
$\displaystyle AS^{\prime}$
$\displaystyle=AS-\beta_{1}\textrm{TS}-\beta_{2}\textrm{POS}$
$\displaystyle=\beta_{0}+\epsilon$ (Adjusted $AS$)

Using the adjusted $AS$ scores in each LM, we compute the score of generating
the conclusion (scaled between 0 and 1) for each category, item, and synthetic
property, and average them to get the model’s overall score for extending new
information about an item to its category. As was the case in our taxonomic
verification judgement, we compute the correlation between our normalized
adjusted $AS$ scores and the human typicality ratings from ? (?), and compare
average $AS$ scores (across all blank properties that we used in this
experiment) and human typicality ratings assigned to low and high-typicality
items. Note that since the 5-gram LM predicts word probabilities by
conditioning only up to four preceding tokens, which is far fewer than the
number of tokens in our stimuli, it shows constant $AS$ values in this
experiment.

#### Results

Figure 2 summarizes results from our induction experiments. When LMs extend
information about an item to its category, they are moderately but positively
influenced by its typicality ($\rho\in$ [0.27, 0.45], $p<$ .001). This
influence is above and beyond their usual predilection towards repeating
sequences and being lexically sensitive to items present in the premise (?,
?). Deviating from results in the previous experiment, we observe Incremental
LMs to show stronger correspondence with human ratings as compared to Masked
LMs of comparable size, suggesting that they are slightly more sensitive to
the typicality of the premise item in generating the conclusion. Unlike the
previous experiment, we notice almost no effect of model size (in terms of
parameters) on the results, suggesting that while making typicality-sensitive
attribution of items to their super-ordinate categories is generally improved
by scaling up the overall expressiveness of the model, the factors that
underlie typicality effects in category-based induction are likely independent
of the number of parameters of an LM.

## General Discussion and Conclusion

Extensive research in the field of cognitive science has highlighted the
prevalent role played by typicality in studies of categories—that certain
items (chair) are considered to be better representatives of a category
(furniture) than others (vase). Motivated by recent evidence showing pre-
trained LMs to capture patterns exhibiting conceptual and categorical
knowledge, we presented two experiments targeting sensitivities to typicality
in LMs. The first experiment targets typicality directly, in its role played
in associating items to their taxonomic categories (“football is a sport”).
Our second experiment complements this by instead assessing the extent to
which the “knowledge” of category typicality is used to extend information
about items (“football involves blicking”) to their respective categories
(“all sports involve blicking”). We investigate typicality effects in LMs by
evaluating their log-probabilities in response to stimuli as measures of (1)
taxonomic verification and (2) inductive argument strength (when conditioned
on a premise). For each test, we made the simplifying assumption that the
likelihood assigned by the LM to the sentence stimuli corresponds to the
variables of interest—strength of category membership in the first experiment,
and argument strength in the second. Overall, the pre-trained LMs showed
positive but modest correlations with human typicality ratings in both
experiments, and were, on average, far less extreme in distinguishing between
typical and atypical items than humans. We also observed that a considerable
amount of sensitivity to typicality effects can be attributed to the
mechanisms available to simpler LMs (5-gram), relative to the sophisticated
pre-trained LMs that we studied here, suggesting that the representational
mechanisms in most models that are optimized to reflect the statistics in
training corpora only account for a minimal gain over correspondence that is
afforded by simpler sequential statistics. Results on pre-trained LMs suggests
that the statistical associations that inform their word probabilities are
modestly sensitive to human-elicited typicality ratings in (1) attributing
items to their category members, as well as (2) making complex inductive
inferences about categories when conditioned on new information about the
items. While our taxonomic sentence verification experiments showed typicality
correspondence to increase with model size, this was not the case in our
induction experiments, suggesting that extending new information about items
to their categories in a manner that is positively modulated by typicality
effects does not scale with an increase in parameters. We leave fine-grained
exploration of specific language modelling factors affecting typicality
correspondence for future work.

LMs are trained by exclusively relying on distributional evidence to inform
their word predictions. In our experiments, we find that while the
aforementioned word prediction capacities show qualitatively similar patterns
of associating concepts with human-produced property norms (?, ?), they show
weak agreement with the typicality effects that are robustly elicited in
humans (?, ?, and references therein). This suggests that solely relying on
text is insufficient for exhibiting quantitatively similar categorical
knowledge to that in humans, and highlights the limitations of using word-
prediction capacities from state-of-the-art pre-trained LMs as mechanisms to
model semantic cognition. This is in line with work in knowledge acquisition
through text, which suggests large textual corpora to lack real world
grounding, in that these corpora represent language use but distort general
knowledge about the world (?, ?). Even though text data contain encyclopedic
knowledge, they miss out on the more perceptual or semi-perceptual features
that can be learned through visual input, and that have been found to better
align with human ratings of typicality, albeit on non-taxonomic categories (?,
?). Another line of work supporting the lack of typicality signal in textual
corpora is that of ? (?). These authors analyze parent-child interactions
using models that are similar to—but less-sophisticated than—pre-trained LMs,
and find them to negatively align with typicality ratings on adjective-noun
compounds. The authors conclude from their findings that much of what children
hear (corresponding to language use by the parent) is atypical, as opposed to
typical information about noun concepts (specifically with respect to the
adjectives that modify them). While our results also shed light on the
difficulty of acquiring knowledge about typical members of categories, they do
suggest the presence of some typicality effects, by contrast to the findings
of ? (?)—raising the possibility that associations in text that impact
typicality of adjective-noun compounds could be independent of, or even run in
opposition to, those that impact taxonomic categories. At the same time,
considering that we do see non-zero correspondence with human typicality
ratings, our results also suggest that textual corpora are not fully devoid of
associations that may align with empirical phenomena underlying typicality
effects. Taking this as inspiration, future work on modeling of typicality
through text will likely require models to correct for the distorted frequency
of atypical items mentioned in text, and potentially also include features
informed from a more grounded source of knowledge. One promising way of doing
so could be to let LMs and their representations adapt to texts represented as
more explicit sources of concept and categorical knowledge (?, ?) —
potentially in the form of statements such as a robin has wings. Explicitly
encoding features into LMs could possibly make them compliant with feature-
based hypotheses of typicality (?, ?) and inductive reasoning (?, ?), and
better facilitate research into other key facets of semantic cognition (?, ?)
in models that learn through text.

#### Acknowledgments

We thank the three anonymous reviewers and the meta-reviewer for their
comments and feedback. This research has benefited from fruitful discussions
with the members of the AKRaNLU lab at Purdue University, and the CompLing lab
at the University of Chicago. We also thank the Department of CIT at Purdue
University for providing hardware resources for computation.

#### Reproducibility

To facilitate further research into manifestation of typicality in language
processing models, we make our code and supplementary materials available at:
https://github.com/kanishkamisra/typicalityprobing

## References

  * Bergey, Morris, YurovskyBergey et al. Bergey, C., Morris, B. C., Yurovsky, D.  (2020).  Children hear more about what is atypical than what is typical.  In _CogSci 2020_ (pp. 501–507).  Cognitive Science Society.
  * Bhatia RichieBhatia Richie Bhatia, S., Richie, R.  (2020).  Transformer networks of human concept knowledge.
  * Clark, Luong, Le, ManningClark et al. Clark, K., Luong, M.-T., Le, Q. V., Manning, C. D.  (2020).  ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators.  In _International Conference on Learning Representations._
  * Devlin, Chang, Lee, ToutanovaDevlin et al. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.  (2019).  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of NAACL-HLT 2019_ (pp. 4171–4186).  Minneapolis, Minnesota: Association for Computational Linguistics.
  * EttingerEttinger Ettinger, A.  (2020).  What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models.  _Transactions of the Association for Computational Linguistics_ , _8_ , 34–48.
  * Feeney HeitFeeney Heit Feeney, A. E., Heit, E. E.  (2007).  _Inductive reasoning: Experimental, developmental, and computational approaches._ Cambridge University Press.
  * Gordon Van DurmeGordon Van Durme Gordon, J., Van Durme, B.  (2013).  Reporting bias and knowledge acquisition.  In _Proceedings of the 2013 workshop on automated knowledge base construction_ (pp. 25–30).
  * HeafieldHeafield Heafield, K.  (2011).  Kenlm: Faster and smaller language model queries.  In _Proceedings of WMT_ (pp. 187–197).
  * Lake, Zaremba, Fergus, GureckisLake et al. Lake, B. M., Zaremba, W., Fergus, R., Gureckis, T. M.  (2015).  Deep neural networks predict category typicality ratings for images.  In _CogSci 2015_ (pp. 1243–1248).  Cognitive Science Society.
  * Lan et al.Lan et al. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.  (2019).  Albert: A lite bert for self-supervised learning of language representations.  In _International conference on learning representations._
  * Liu et al.Liu et al. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., … Stoyanov, V.  (2019).  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _arXiv preprint arXiv:1907.11692_.
  * Lupyan LewisLupyan Lewis Lupyan, G., Lewis, M.  (2019).  From words-as-mappings to words-as-cues: The role of language in semantic knowledge.  _Language, Cognition and Neuroscience_ , _34_(10), 1319–1337.
  * Misra, Ettinger, RayzMisra et al. Misra, K., Ettinger, A., Rayz, J.  (2020).  Exploring BERT’s sensitivity to lexical cues using tests from semantic priming.  In _Proceedings of EMNLP 2020: Findings_ (pp. 4625–4635).
  * MurphyMurphy Murphy, G.  (2002).  _The Big Book of Concepts_.  MIT press.
  * Osherson, Smith, Wilkie, Lopez, ShafirOsherson et al. Osherson, D. N., Smith, E. E., Wilkie, O., Lopez, A., Shafir, E.  (1990).  Category-based induction.  _Psychological review_ , _97_(2), 185.
  * Petroni et al.Petroni et al. Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., Miller, A.  (2019).  Language models as knowledge bases?  In _Proceedings of the EMNLP-IJCNLP 2019_ (pp. 2463–2473).
  * Radford, Narasimhan, Salimans, SutskeverRadford et al. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.  (2018).  Improving language understanding by generative pre-training.
  * Radford et al.Radford et al. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.  (2019).  Language models are unsupervised multitask learners.  _OpenAI blog_ , _1_(8), 9.
  * RipsRips Rips, L. J.  (1975).  Inductive judgments about natural categories.  _Journal of verbal learning and verbal behavior_ , _14_(6), 665–681.
  * Rips, Shoben, SmithRips et al. Rips, L. J., Shoben, E. J., Smith, E. E.  (1973).  Semantic distance and the verification of semantic relations.  _Journal of verbal learning and verbal behavior_ , _12_(1), 1–20.
  * Rogers McClellandRogers McClelland Rogers, T. T., McClelland, J. L.  (2004).  _Semantic cognition: A parallel distributed processing approach_.  MIT press.
  * RoschRosch Rosch, E.  (1973).  On the internal structure of perceptual and semantic categories.  In _Cognitive development and acquisition of language_ (pp. 111–144).  Elsevier.
  * RoschRosch Rosch, E.  (1975).  Cognitive representations of semantic categories.  _Journal of experimental psychology: General_ , _104_(3), 192.
  * Rosch, Simpson, MillerRosch et al. Rosch, E., Simpson, C., Miller, R. S.  (1976).  Structural bases of typicality effects.  _Journal of Experimental Psychology: Human perception and performance_ , _2_(4), 491.
  * Sanh, Debut, Chaumond, WolfSanh et al. Sanh, V., Debut, L., Chaumond, J., Wolf, T.  (2019).  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * SlomanSloman Sloman, S. A.  (1993).  Feature-based induction.  _Cognitive psychology_ , _25_(2), 231–280.
  * Vaswani et al.Vaswani et al. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., … Polosukhin, I.  (2017).  Attention is All you Need.  In _NeurIPS 2017_ (pp. 5998–6008).
  * Wang ChoWang Cho Wang, A., Cho, K.  (2019).  BERT has a mouth, and it must speak: BERT as a Markov random field language model.  In _Proceedings of the workshop on methods for optimizing and evaluating neural language generation_ (pp. 30–36).  Minneapolis, Minnesota: Association for Computational Linguistics.
  * Weir, Poliak, Van DurmeWeir et al. Weir, N., Poliak, A., Van Durme, B.  (2020).  Probing neural language models for human tacit assumptions.  In _CogSci 2020_ (pp. 377–383).  Cognitive Science Society.
  * Wolf et al.Wolf et al. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., … Rush, A.  (2020).  Transformers: State-of-the-art natural language processing.  In _Proceedings of EMNLP 2020: Demos_ (pp. 38–45).  Online: Association for Computational Linguistics.