1183 Research  Shared and collaborative augmented reality experience on mobile
devices, powered by high-precision localisation in city-scale 3D maps.

# Collaborative Augmented Reality on Smartphones
via Life-long City-scale Maps

Lukas Platinsky  Michal Szabados  Filip Hlasek  Ross Hemsley  Luca Del Pero
Andrej Pancik  Bryan Baum  Hugo Grimmett  Peter Ondruska This work was
conducted at Blue Vision Labs (now part of Lyft Level 5). To contact authors
please email<EMAIL_ADDRESS>
###### Abstract

In this paper we present the first published end-to-end production computer-
vision system for powering city-scale shared augmented reality experiences on
mobile devices. In doing so we propose a new formulation for an experience-
based mapping framework as an effective solution to the key issues of city-
scale SLAM scalability, robustness, map updates and all-time all-weather
performance required by a production system. Furthermore, we propose an
effective way of synchronising SLAM systems to deliver seamless real-time
localisation of multiple edge devices at the same time. All this in the
presence of network latency and bandwidth limitations. The resulting system is
deployed and tested at scale in San Francisco where it delivers AR experiences
in a mapped area of several hundred kilometers. To foster further development
of this area we offer the data set to the public, constituting the largest of
this kind to date.

###### keywords:

Computer vision, Augmented reality, Structure from motion, Large-scale SLAM

Computing methodologiesArtificial intelligenceComputer visionTracking and
Reconstruction; Computing methodologiesMixed / augmented Reality

Figure 1: The system diagram for the proposed system. A large-scale 3D map is
computed from a vehicle-based image collection and used to localise camera-
equipped edge devices for AR experiences over a mobile network.

##  1 Introduction

Most of the currently available production Augmented Reality (AR) today are
single-user experiences running at a small scale. For example, in games such
as Pokemon Go, a user can see AR content on their edge device (e.g. smartphone
or AR glasses), but a second user cannot see the same content in the same
place on their own device. This fundamental limitation is due to the each
device’s pose being estimated in its own coordinate systems different to each
other, by the simultaneous localisation and mapping (SLAM) system running on-
board. Additionally, these are often limited to small scale (i.e. one room),
constant visual conditions, and do not support large-scale applications such
as outdoor AR navigation.

The properties of SLAM systems are well known [32, 11, 34], but building
production-grade, city-size, and multi-agent solutions poses unique challenges
due to the high requirements for system robustness, scalability, multi-weather
and time-of-the day performance, and map updates combination of which was
never explored together.

In this work we present the first published end-to-end production system that
allows city-scale robust localisation of multiple devices in the same shared
coordinate system for building shared and collaborative augmented reality
experiences (as shown in Figure Collaborative Augmented Reality on Smartphones
via Life-long City-scale Maps, and the accompanying video 111Video of the
deployed system is at https://youtu.be/tXwVg2S9wuY.). In doing so we propose
and exploit a new experience-based mapping framework for constructing large-
scale all-time all-weather maps, keeping them perpetually updated and
supporting localisation of edge devices over a mobile network. In particular,
the contributions of this paper are five-fold:

  1. 1.

The end-to-end design of a production-grade system for augmented reality at
scale.

  2. 2.

An experience-based mapping framework, which is an effective solution to the
key issues of all-time all-weather city-scale SLAM scalability.

  3. 3.

A novel way of localising multiple edge devices in the same coordinate space,
in real time over a mobile network with latency and limited bandwidth.

  4. 4.

A comprehensive evaluation of map-building and AR localisation performance on
a city-scale dataset.

  5. 5.

A new city-scale dataset 222To request access to the dataset please visit
www.bluevisionlabs.org.

##  2 Related work

Some previous work tackles global localisation without using a map of the
environment, for example by combining GPS and inertial sensors (e.g. [30]), or
GPS and visual odometry (e.g., in [15]). However, in our tests neither of
these options met the accuracy requirements for providing immersive real-time
AR experiences due to large GPS location error, particularly in urban settings
with tall buildings (urban canyons). To obtain the required accuracy, we have
used a visual mapping and localisation approach providing a more accurate
localisation signal than GPS. This approach requires constructing a large-
scale global map of the environment before the AR experiences can happen.

Much work has been done on building mapping and localisation algorithms. Many
of the classical systems [7, 25, 8] have been achieving impressive results
with single-agent real-time systems performing mapping and localisation at the
same time (SLAM). There are also several impressive solutions that
specifically provide the ability to build a map first and then use it to
localise allowing also for multi-agent solutions [20, 31, 10]. These systems,
however, typically operate in a limited geographic area due to the memory
constraints and algorithmic complexity of the underlying structure-from-motion
(SfM) problem. Their map-building part is often limited to a single machine
and cannot be directly used with the amounts of data processed by the solution
proposed in this paper.

Building visual maps at a scale needed for the described use cases has been
addressed in [2, 14, 39]. Compared to these approaches, our method is designed
to optimally exploit the multi-experience nature of the data collection
leading to approximately linear complexity of map-building process, fast map
updates, and extreme robustness to outliers and SfM failures arising from
difficult visual conditions. Unlike the previous solutions, the proposed
system is significantly more robust to the variances in quality of the input
data, allowing for crowd-sourced approach for data collection. Moreover, we do
not stop at the map-building stage and provide a large-scale multi-agent
localization framework as well.

Image-based location retrieval is a topic with a lot of interest and it was
explored, for example, in [12, 27, 28, 33, 9, 6, 13, 16, 18, 37]. Many of the
approaches focus on limited geographic areas or on a limited set of lighting
conditions. Several impressive works focusing on providing localisation of
mobile devices presented in [35, 21, 36] suffer from similar vices, allowing
for experiences only in significantly limited areas and we show several orders
of magnitude improvement in the size of the area supported in our experiments.
We present a multi-agent system where many devices continuously interact and
localise within a single, unbounded map under varying visual conditions.

Vision systems degrade in changing visual conditions due to increasing
perceptual difference (i.e. day vs night) between the map and localisation
images. Robust, all time-time, all-weather performance is key for production
systems and was considered in [17, 3, 29] by building robust localisation
feature representations. In contrast, we take inspiration from experience-
based navigation framework [5] where robust multi-weather performance is
achieved by repeatedly collecting data along a route in varied conditions and
building one multi-weather map. By doing so the map covers all possible visual
conditions resulting into robust performance. This setup is uniquely
beneficial for building productions systems, where more data can be collected
as required to achieve the desired performance of the system. In particular,
in this work we focus on the ability to combine these experiences, collected
at scale by a fleet of vehicles, robustly fuse them to form a single map, and
efficiently update the map when more data are available.

Finally, unlike all previous works that addressed only parts of the technical
problem, this is the first work that ties all the components together and at
scale into one production-grade AR system. We acknowledge imperfections in the
current algorithms and our collected data, and rather than attempting to find
a silver bullet for each part of the system, we acknowledge the imperfections
and build a redundant and robust solution around it.

We are also releasing a new large-scale mapping and localisation dataset from
San Francisco, comprising millions of frames and videos. It is the largest
publicly-available dataset of this kind, 5x larger than the next largest
similar dataset [4].

Main difference - many orders of magnitude more. Focus is not on solving each
individual part perfectly. We accept imperfect nature of algorithms and design
the whole end-to-end approach with redundancy and solution focused on
manipulating several orders of magnitude larger data than the previous known
approaches.

##  3 System overview

We propose a computer vision system that enables collaborative augmented
reality experiences. At its core lies a map-based localisation system outlined
in Figure 1 that can track the global pose of an edge device (e.g. smartphone
or AR glasses) $p^{\mathcal{M}}_{t}$ over time in the shared coordinate space
of the 3D map $\mathcal{M}$ of the environment. This leverages the approximate
location $p^{gps}_{t}$ given by the device GPS, and the image history of what
it sees through its camera $I_{1\dots t}$.

The map is constructed from a large set of individual data collection logs
called _experiences_ $\\{\mathcal{I}^{1}\dots\mathcal{I}^{N}\\}$. Each
experience comprises a set of images $\mathcal{I}^{i}=\\{I^{i}_{1}\dots
I^{i}_{O}\\}$ and GPS points collected during a single session at particular
time of day / weather condition by a vehicle equipped with multiple cameras
and an INS system. It can range in length from several meters to up to a few
kilometers. The ability to build maps from very large number of experiences is
the key to achieving high performance and robustness (see Sec. 4). We use a
fleet of cars to collect experiences that are used in map-building to
efficiently control and improve the performance of the resulting system.

Figure 2: The city-scale map built for supporting AR experiences. The subset
of the map we evaluated AR performance (middle) consists of 4249 submaps and
was computed on a cluster of 500 CPUs over period of 91 hours.

The map contains billions of landmarks, and due to its size cannot be stored
on the edge device. Instead, the device runs a visual-inertial odometry (VIO)
system locally. This system is aligned with the map through the local-to-
global coordinate transform $T_{vo\rightarrow\mathcal{M}}$, recovered by
estimating the pose with respect to the map of a subset of the individual
images captured by the device. Due to map size and the need to keep it
perpetual updates, this takes place in the cloud. In Section 5 we describe how
we dealt with network latency and limited network bandwidth.

This localisation system is then augmented with persistent content storage and
real-time pose sharing, enabling powerful collaborative AR experiences. We
describe details of these in Section 6. In what follows we present the core
architecture of the system, while we discuss smaller implementations details
in 6.

##  4 Large scale experience-based mapping

In this section we present our system for building a large scale map
$\mathcal{M}$ from the set of collected experiences
$\mathcal{I}_{1}\dots\mathcal{I}_{N}$. It is designed to achieve three key
goals:

  1. 1.

_Localisation performance_ : reliable all-time, all-weather localisation

  2. 2.

_Map robustness_ : ensure map quality over a large geographic area in the face
of SfM failures

  3. 3.

_Map freshness_ : efficient map updates by adding more data when needed.

The algorithm consists of four components: (1) data splitting, (2) submap
building, (3) submap verification, and (4) submap fusion with map update. We
now provide an overview before presenting each component in detail.

The algorithm starts with an empty map $\mathcal{M}$ and then iteratively adds
new experiences to it. For each experience, we group all its images into
smaller overlapping subsets (data splitting), and then run SfM on each subset
independently, producing submaps (submap building). This is the most expensive
part of the algorithm, but we parallelise it on a cluster of CPUs. Therefore,
there is no limit to the size of input experiences. Next, SfM failures are
identified, and these ‘broken’ submaps are discarded. The remaining submaps
are fused into a single, global map. The entire map can be built from scratch
in approximately linear time complexity with the amount of data, or
alternatively, experiences can be added one-by-one. The resulting map
$\mathcal{M}$ comprises $K$ geo-aligned submaps $\mathcal{M}_{1\dots K}$, each
containing the camera poses $p$ of the input images, and a set of 3D
localisation landmarks $x$ such that $\mathcal{M}_{i}=\\{p_{1\dots
M},x_{1\dots L}\\}$.

1\. Data splitting. We bound the computational complexity of submap building
(step 2) by grouping the images in an experience (which can be potentially
numerous) into consistently-sized subsets of images. The first image in the
experience is assigned to a subset, which is grown in size until it contains
1,000 images (we enforce subsets to have a circular shape, with a minimum
radius of 20m). This is done using a greedy line-sweep algorithm over the 2D
GPS positions of each image (see Figure 3). In areas with large image density,
we randomly sub-sample images, to fit into the 1,000 limit. These values were
chosen empirically to allow submaps to cover enough geographical area (making
the grouping robust to GPS errors, which are severe in urban canyons), yet be
small enough for efficient submap building. Lastly, for each submap in an
experience we add images randomly chosen from neighbouring submaps, which is
key for submap fusion (step 3). To enable the fusion of submaps across
experiences, we also add randomly-selected images from the same geographic
location but different experiences. While this increases the computation
required in step 2 (most images are in multiple submaps), this greatly
simplifies the fusion step.

Figure 3: The line-sweep algorithm that splits data collected during one
mapping session into submaps of size up to 1,000 frames.

2\. Submap building. Next, incremental SfM [23] is run on each subset
independently, relying on a standard pipeline of feature extraction, matching,
track creation and bundle adjustment. This recovers the poses of the images
and the 3D positions of landmarks, by minimising the re-projection error:

$\mathcal{C}(x,p)=\sum_{i,j}(z_{i,j}-\pi(x_{i},p_{j}))^{2}+\lambda\sum_{i}(p_{i}-p_{i}^{gps})^{2}$
(1)

where $z_{i,j}$ is the position of the $i$-th feature in the $j$-th image, and
$\pi(x_{i},p_{j})$ is re-projection of landmark $x_{i}$ using pose $p_{j}$.
The last term is a GPS prior, weighted by $\lambda$. Thanks to the small size
of the subsets, each submap can be built on a single CPU. Examples of submaps
are shown in Figure 4.

3\. Submap verification. Due to challenging conditions such as occlusions,
moving objects, or camera failures, SfM can output incorrect or warped 3D maps
as shown in Figure 4. To detect this we check whether the reconstructed poses
agree with the relative orientations of consecutive images as measured by the
INS. Submaps are also discarded if the reconstructed gravity vectors deviate
too much from the INS deviate measurement, or if the positions of consecutive
images violate the kinematic constraints of a simple vehicle motion model.
Discarded submaps are not used for map fusion or update.

Figure 4: Examples of submaps the map is composed of. On the _left_ a submap
where SfM succeeded, and on the _right_ one where structure-from-motion failed
due to difficult visual conditions. Failed submaps are discarded prior to
submap fusion.  Figure 5: Detail of the geo-aligned map shown on top of
satellite picture. Blue: triangulated landmarks, green: camera poses, red: GPS
priors.

4\. Submap fusion & map update. Lastly, new submaps and existing submaps in
$\mathcal{M}$ are fused together rigidly. For each of $K$ submaps, we compute
a 7DoF transformation $T_{k}$ that minimises the displacement between images
that occur in multiple submaps (i.e., in the coordinate system of the fused
map, the positions of the same image in different submaps must coincide):

$\mathcal{C}(T)=\sum_{i}\sum_{k,l}(T_{k}p^{k}_{i}-T_{l}p^{l}_{i})^{2}+\lambda\sum_{i,k}(T_{k}p^{k}_{i}-p_{i}^{gps})^{2}$
(2)

where $p_{i}^{k}$ is the pose of image $I_{i}$ in submap $\mathcal{M}_{k}$ and
$k,l$ iterate over pairs of submaps containing $I_{i}$.

The result is a geo-aligned map (Fig. 2). Figure 5 shows the map’s alignment
with geo-coordinates due to the combination of geometric and GPS terms.

Despite its simplicity, the overall mapping framework is remarkably robust. As
further discussed in Section 6, it robustly builds highly accurate 3D maps
even if the SfM has only a 89% success rate at the individual submap level
(step 2). Moreover, both map completeness and localisation performance improve
as new experiences are fused with the map, filling any low-density regions.

Removing data from the map, which might be desirable in the case of
environmental change or manually detected map errors, is equally simple. The
affected submaps are simply removed and we rerun the fusion steps on the
remaining submaps. Note that we do not perform any global bundle-adjustment to
refine positions of landmark after step 4, unlike [40]. We found that this
only marginally improves map accuracy, and thus does not warrant the
additional computational cost for the AR use case.

##  5 Real-time SLAM synchronisation

Figure 6: The process of continuously localising edge device in the map over
the network. Recently captured images are transmitted over the network to
determine their global position and compute optimal offset of the local
visual-odometry frame of reference.

In this section we describe how to continuously localise an edge device in the
large-scale multi-experience map we described in the previous section.

Given the map is stored in the cloud direct image-based localisation
impossible. Instead, each device runs a visual-inertial odometry system that
continuously tracks its own pose $p^{vo}_{t}$ over time in a local coordinate
frame and simultaneously estimates a global-to-local coordinate transform
$T_{vo\rightarrow\mathcal{M}}$. This is achieved by (1) sending recently
captured keyframes over the mobile network to the cloud service containing the
map, (2) the cloud service determining the edge device’s global pose by
performing image-based localisation, and (3) sending the global localisation
result back to the edge device, as shown in Figure 6. The optimal transform
$T^{*}_{vo\rightarrow\mathcal{M}}$ is given by minimising

$\mathcal{C}(T)=\sum_{I_{t}}\tau^{-t}\|p_{\mathcal{M}}(I_{t})-Tp^{vo}_{t}\|_{\delta},$
(3)

over the history of transmitted images $I_{t}$ where $p_{\mathcal{M}}(I_{t})$
is the result of the localisation of image $I_{t}$ in map $\mathcal{M}$,
$p^{vo}_{t}$ is the local pose of the image and $\tau$ is a time decay factor
that favours recent measurements. We use robust Huber loss to mitigate outlier
localisation. The current real-time pose is then given as

$p^{\mathcal{M}}_{t}=T^{*}_{vo\rightarrow\mathcal{M}}p^{vo}_{t}.$ (4)

Recall that the map is composed of submaps $\mathcal{M}_{1\dots K}$. In order
to handle a very large map, the submaps are stored together based on GPS
proximity, and the image is queried against all submaps within a GPS radius.
The query result with the largest number of inlier feature matches is then
used. This process typically takes approximately one second per image, and is
independent of the size of the map.

The accuracy of visual localisation performance depends on a number of factors
and the localisation error is approximately given by

$\epsilon=\epsilon_{map}+\epsilon_{loc}+\epsilon_{vo}\frac{\Delta t}{r_{loc}}$
(5)

where $\epsilon_{map},\epsilon_{loc},\epsilon_{vo}$ is respectively the error
of the map, localisation and odometry drift over localisation request time
interval $\Delta t$ and $r_{loc}$ is the localisation success rate %. As
discussed in Section 6 this is usually in order of few centimeters.

Figure 7: The dataset we collected in San Francisco and used to evaluate the
system. Blue: data collected by vehicles to build the map, Red: data collected
by pedestrians to test localisation performance. The area with both vehicle
and pedestrian coverage is released to the public.

##  6 Experiments

In this section we evaluate the proposed system and outline how it can be used
to AR usecases. In particular, we are interested in (a) the experience-based
mapping system’s ability to produce large-scale maps, and (b) the edge-device
localisation performance for multi-user, production AR.

Stage | # CPUs | Per submap | Total
---|---|---|---
Data splitting | 1 | - | 1 hour
Feature detection | 300 | 1 hour | 18 hours
Feature matching | 300 | 1.5 hours | 27 hours
SfM | 500 | 4 hours | 44 hours
Submap fusion | 1 | - | 1 hour
Total |  |  | 91 hours
Table 1: The map-building compute time broken down by each stage, for a map
consisting of 7 million frames split into 4249 submaps.

###  6.1 Dataset

We have collected two datasets to build a map and evaluate the system. We
report the performance of the system on the subset of the data where these two
datasets overlap, and release it to the public as a part of this work. The
used dataset is several orders of magnitude larger than data used for
experimentation by similar end-to-end multi-agent mapping and localisation
systems (e.g., [31]), rendering any state-of-the-art comparisons infeasible.

Figure 8: Samples from the dataset. Top: a view from 8 cameras mounted on the
vehicle. Bottom: data seen by a mobile phone held by pedestrian during
localisation.

Map-building dataset. To build the map of the environment we collected 27M
images in San Francisco using a fleet of vehicles carrying camera and GPS-
equipped devices over a period of around 3 months. Subset of 7M (equivalent of
roughly 1,000 miles driven) is being released and used for evaluation.

Each vehicle used for the 7M dataset was equipped by 8 rolling-shutter cameras
in rosette configuration as shown in Figure 8. Each camera has a $70^{\circ}$
Field of View (FOV) providing a $360^{\circ}$ view around the vehicle with
approximately $25^{\circ}$ overlap between neighbouring cameras. The vehicles
traversed the city, covering each street multiple times and thus creating
several experiences per street. Experiences come from different times of the
day to cover different lighting conditions and improve the robustness of the
resulting system. Most of the dataset was collected under sunny or overcast
conditions. It contains images, IMU data at 100Hz, and GPS at 1Hz. The cameras
are rigidly mounted but slightly different for different collect vehicles as
this simplifies data collection operations. Exact camera intrinsics and
extrinsics are determined as part of the structure-from-motion process and the
cameras are not synchronised to further decrease the operational complexity.

AR localisation dataset. In order to evaluate the map’s performance for
pedestrian AR use cases, we collected 1009 short video snippets of handheld
data together with device’s visual-inertial odometry log modelling user using
the system. A human operator would stand close to the road (within 2m of the
curb) and slowly looks around, to cover roughly a $180^{\circ}$ FOV, and then
walks roughly 20m along the sidewalk. We have chosen this evaluation because
for any AR experience to give a good user experience, the edge device must
quickly re-localize against the map, and with minimal user interaction.

Area |  Attempted |  Reconstructed |  Yield
---|---|---|---
1+ passes |  3000 miles |  1950 miles |  66%
4+ passes |  1517 miles |  1384 miles |  91%
Table 2: Map yield by number of passes. Map yield is defined as a percentage of the streets where we have successfully reconstructed batches against where we attempted to construct the map. |
---|---
Figure 9: Map coverage as a function of more experiences from the area. Left:
top-down view of a challenging area of the city after a single data
collection, right: the same area after adding more data to the map. Figure 10:
A histogram of metric error of localised pose, compared with a ground-truth
from a high-accuracy sensor, over a 20m distance.

###  6.2 Map building performance

We processed data collected by the mapping vehicles and build the map shown in
Figure 2. We split the data in the evaluated area into 4249 submaps, with an
average diameter of 50m, and 20m overlap between neighbouring submaps. The
success rate of building submaps was 89%. Failures were caused by difficult
visual lighting conditions (strong sun), large numbers of moving objects, or
operational issues with data collection hardware. We also deliberately set a
high threshold for the automated and manual checks on the submaps, resulting
in many being discarded.

Implementation details. For our mapping pipeline, we use SIFT features [19]
and OpenMVG’s incremental SfM implementation [23] when building submaps. We do
not perform any form of map simplification, since our submaps comfortably fit
into our servers’ memory, though this would be an interesting optimisation. We
separately experimented with masking out dynamic agents in the images, but we
do not use it for these experiment as it did not bring any improvement (parked
cars can provide good features in map building and thanks to combining map
localization and odometry proved enough to reliably filter outliers in
localization). We do not perform further bundle adjustment for fusing the
submaps as it would be infeasible given the overall map size, and this choice
proved to acheive enough localisation accuracy to support AR applications (our
goal). In our production system, before deploying a new map we perform a
light-weight manual inspection in a web viewer to detect obvious errors
(typically larger geometric disagreements between submaps in the same region
which are easy to spot). After removing these submaps we rerun the submap
fusion step, and deploy.

Computation time. The computation took more than 35,000 hours distributed over
500 CPUs and produced a 700 GB map. For run-time and parallelisation of the
different steps, see Table 1. Most of the time is spent in the feature
matching and bundle adjustment steps, which run in quadratic and cubic time in
the number of images in a submap respectively. A map update on this map
consisting of building one or multiple additional submaps and fusing them in
takes approximately 6 hours.

Figure 11: Examples of different augmented reality experiences we built
powered by the proposed localisation of edge devices held by pedestrians or
equipped on vehicles: Shared and persistent AR content visible by all users at
the same location, avatar-augmented device pose, AR navigation and
collaborative AR games.

The impact of experiences on map coverage. Figure 9 qualitatively shows
improvement of map coverage as we add experiences. With an average of only one
mapping experience per location, the map displays missing areas and would be
unfit for localisation on those streets. As seen in Table 2, this
significantly improves as more experiences are added, despite high failure
rate of individual submaps.

###  6.3 Localisation performance

We measured the performance of edge device localisation in the map. Figure 12
shows typical localisation performance for a pedestrian’s smartphone as they
walk down the road, as compared with GPS. The visual localisation system is
significantly more accurate than GPS localisation. We used 1,009 recordings
from the AR localisation dataset described above. The key localisation metric
used is _localisation rate_ , defined as the fraction of images able to
localise. We complement it with two accuracy metrics: localisation consistency
and localisation accuracy.

Implementation details. For the server-side localization of single images we
used SIFT features [19] and the localization tool from OpenMVG [24]. We
modified the approximate nearest neighbour lookup implementation to reduce the
size of the data and speed of loading it into memory to fit our submap size.
This was necessary to guarantee server responses within around 1 second at a
stable 4G connection. For odometry, we experimented with VINS [26] and ORBSLAM
[25], but we ended up using a commercial version (ARKit [1]), as it performed
best for our handheld AR use case and provided the most robust experience on
the tested smartphones.

Localisation rate. We evaluated 37,727 localisation requests that originated
from the challenging AR dataset recordings simulating a real use of the system
across the city. Out of those, 73% were successfully localised. As these poses
can be computed from erroneous matches we have also calculated that after
further filtering of outlier localizations in the system, 61% of the requests
produce a coherent result when combined with the visual-inertial odometry
running on the client. This rate is enough to provide a smooth user
experience.

Figure 12: A comparison of typical GPS localisation accuracy of (red) and the
proposed visual localisation (blue) for a pedestrian holding a smartphone
while walking down the street. Visual localisation is significantly more
accurate.

Localisation consistency. We measured the positional discrepancy of the
server-side localisation service against the visual odometry running on the
device before fusing them to be 30cm. We have tuned the system to bias heavily
towards false positives in server-side localizations. The synchronisation
algorithm running on the device can pick out the correct relocalisations and
discard the high-variance noise.

Localisation accuracy. To evaluate positional accuracy against a ground truth
we collected a separate dataset comprising 2h of data. It was obtained by
attaching an edge device to a self-driving-capable vehicle with high-
performance odometry, and driving it around the city. We repeatedly computed
the difference between raw localised poses after the device traveled distance
of 20m and compared it against the groundtruth odometry. Figure 10, shows
result of this relative error [38] comparison. The mean error is 42cm (2%) and
median error is 18cm (1%). This accuracy is sufficient for most AR usecases.

The impact of experiences on localisation performance. Similarly to [5], in
Table 3 we observe increased localisation performance as data collected from
varying times of day and weather are combined. To measure the impact on
localisation performance we collected mapping data and corresponding edge
device localisation recordings in one area of the map at different times of
day. We evaluate the localisation rate as a function of the amount of data
used for map-building.

Computation time in Figure 4 summarises timing of individual stages of a
single localisation request. On average the localisation requests take 970ms
on 4G network and 1700ms on 3G network.

Mapping collections in an area |  Localization rate
---|---
1 day-time experience |  69.5%
4 day-time experiences |  81.5%
6 day and night experiences |  96.6%
7 day and night experiences |  97.3%
Table 3: Localisation performance as a function of more experiences from the area. With more observations the localisation rate across different lighting conditions increases. Stage |  Timing (ms)
---|---
Image transfer |  200 (4G) – 1,000 (3G)
Feature detection |  250
Feature matching |  500
Triangulation |  20
Table 4: Timing of different stages of localising one image from the cloud
device in the map stored in the cloud.

###  6.4 AR system

Our system can be run in parallel on multiple edge devices jointly localising
in the same coordinate space of the map $\mathcal{M}$. In order to allow users
to create and interact with shared persistent content and with each other, we
augmented the base localisation system with two services:

Persistent content storage allows users to create and store the coordinates of
AR content in a shared database, making it immediately visible to other users.

Real-time pose sharing exchanges current poses between active devices. This is
important for interaction and allows you to, say, show an avatar on top of
players’ current locations, or play collaborative games.

Figure 11 and the accompanied video showcase example AR experiences we built
using these methods. Thanks to the shared coordinate system, all users see
content at exactly the same place and can interact with each other.

###  6.5 Limitations and failure cases

We observed the system works robustly under usual lightning conditions. Here
we summarise the most significant causes we observed to be limiting its
performance:

Multiple passes required for mapping. Our system achieve robustness by
exploiting redundancies in the data - this was a deliberate decision. We
embraced the stochastic nature of the individual methods in the overall system
(each step can fail on subsets of the data), but redundancy in data allows us
to handle local algorithmic failures (e.g. while building a submap from an
experience with SfM can fail due poor illumination, occlusions, etc., a
successful submap from another experience in the same area can fill the gap).
The drawback is that we typically require a minimum amount of data in every
area: note how mapping performance degrades as we go from 4 passes to 1 pass
(this is the same as an experience) in Table 2, with localisation rate showing
a similar trend in Table 3. However, this is not a problem in our setup due to
the scalability of our algorithms, and the simplicity of our sensor
installation and data collection processes which allows us to collect large
amounts of data efficiently.

Mobile network connection. One of the biggest limitations we observed in
practice is the requirement of a stable mobile data connection during its use.
We observed common sudden drops in mobile bandwidth when testing in crowded
areas of the city. This has a large noticeable impact on initalisation of the
system and also on the positional error accumulated due to dead-reckoning
drift of odometry. This can be solved by doing the map relocalisation on the
edge device. While possible, it would require optimising the algorithm’s
runtime, using quicker (most likely binary rather than float) feature
descriptors, and simplification of the map to reduce its size to serve it to
the device, which will also need to cache parts of the map. This optimisation
is left as future work for the next versions of the system.

Viewpoint change too large. While the system is resilient to localization from
some novel viewpoints, the second most common issue comes from inability of
the system to match images due to occlusion, or when there is a large
difference in viewpoint. The latter is mostly noticeable in areas where there
is a larger gap between the sidewalk (where the edge device is held) and the
street (where the mapping collections happens). Thanks to the submap
implementation and focus on data redundancy, one way to tackle this is to also
use data collected from the sidewalks with a smaller hand-held camera rig for
mapping (additionally to our current car mapping collect). This works well,
however, pedestrian data collection can become a bottleneck when scaling to
larger areas. An alternative is to explore methods like ASIFT [22] known to
increase the robustness to viewpoint changes.

Other failure cases. Figure 13 displays some of the failure cases mentioned
above, such as obstructed sidewalks, and a few more failures due to excessive
vertical tilting, reflections, and repetitive textures. These can prevent or
slow down an initial localisation in some areas (e.g. where recent changes
happened due to constructions, or the user is pointing their phone towards the
sky), but we observed that their impact on the overall experience is mitigated
by the VIO system.

Figure 13: Typical localisation failure cases. These include obstructed
sidewalks, user pointing the phone upwards, repetitive structures and
reflections.

##  7 Conclusions

We have presented the first published end-to-end production system for
building collaborative augmented reality experiences at city-scale. The
presented system can be extended in many ways, for example, by automatically
updating the map when detecting changes in the environment, potentially with
data transmitted by the edge devices. We believe that , together with the
released dataset, this work will enable the community to address these and
other challenges, and accelerate the development of AR systems towards
seamless and ubiquitous experiences.

###### Acknowledgements.

This work was done thanks to the hard work and support of the Blue Vision Labs
team: Ben Haines, Bryon Shannon, David Evans, Ed Dingley, Fady Kalo, Filippo
Brizzi, Gabriel Nica, Gabriele Angeletti, George Thomas, Giacomo Dabisias,
Guido Zuidhof, Haoyue Zhu, Ivan Katanic, James Close, Jesse DaSilva, Karim
Shaban, Long Chen, Lorenzo Peppoloni, Lucas Chatham, Matej Hamas, Mimosa
Nguyen, Miranda Aperghis, Owen Parker, Robert Kesten, Sina Nickdel, Stepan
Simsa, Suraj Mannakunnel Surendran, Yerzhan Utkelbayev.

## References

  * [1] ARKit.  https://developer.apple.com/augmented-reality/arkit/.
  * [2] S. Agarwal, Y. Furukawa, N. Snavely, I. Simon, B. Curless, S. M. Seitz, and R. Szeliski.  Building rome in a day.  Commun. ACM, 54(10):105–112, Oct. 2011. doi: 10 . 1145/2001269 . 2001293
  * [3] R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, and J. Sivic.  Netvlad: Cnn architecture for weakly supervised place recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5297–5307, 2016.
  * [4] D. Chen, G. Baatz, K. Koeser, S. Tsai, R. Vedantham, T. Pylvanainen, K. Roimela, X. Chen, J. Bach, M. Pollefeys, B. Girod, and R. Grzeszczuk.  City-scale landmark identification on mobile devices.  In IEEE International Conference on Computer Vision and Pattern Recognition, 2011.
  * [5] W. Churchill and P. Newman.  Experience-based Navigation for Long-term Localisation.  The International Journal of Robotics Research (IJRR), 2013.
  * [6] M. J. Cummins and P. M. Newman.  Fab-map: Appearance-based place recognition and mapping using a learned visual vocabulary model.  In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 3–10, 2010.
  * [7] A. J. Davison, I. D. Reid, N. D. Molton, and O. Stasse.  Monoslam: Real-time single camera slam.  IEEE transactions on pattern analysis and machine intelligence, 29(6):1052–1067, 2007.
  * [8] J. Engel, T. Schöps, and D. Cremers.  Lsd-slam: Large-scale direct monocular slam.  In European conference on computer vision, pp. 834–849. Springer, 2014.
  * [9] Y. Feng, L. Fan, and Y. Wu.  Fast localization in large-scale environments using supervised indexing of binary features.  IEEE Transactions on Image Processing, 25(1):343–358, 2015.
  * [10] C. Forster, S. Lynen, L. Kneip, and D. Scaramuzza.  Collaborative monocular slam with multiple micro aerial vehicles.  In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3962–3970, 2013.
  * [11] J. Fuentes-Pacheco, J. Ruiz-Ascencio, and J. M. Rendón-Mancha.  Visual simultaneous localization and mapping: a survey.  Artificial Intelligence Review, 43(1):55–81, 2015.
  * [12] A. Irschara, C. Zach, J.-M. Frahm, and H. Bischof.  From structure-from-motion point clouds to fast location recognition.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2599–2606. IEEE, 2009.
  * [13] A. Kendall, M. Grimes, and R. Cipolla.  Posenet: A convolutional network for real-time 6-dof camera relocalization.  In Proceedings of the IEEE international conference on computer vision, pp. 2938–2946, 2015.
  * [14] B. Klingner, D. Martin, and J. Roseborough.  Street view motion-from-structure-from-motion.  In Proceedings of the IEEE International Conference on Computer Vision, pp. 953–960, 2013.
  * [15] D. Larnaout, V. Gay-Bellile, S. Bourgeois, and M. Dhome.  Vehicle 6-dof localization based on slam constrained by gps and digital elevation model information.  In 2013 IEEE International Conference on Image Processing, pp. 2504–2508, 2013.
  * [16] Y. Li, N. Snavely, D. Huttenlocher, and P. Fua.  Worldwide pose estimation using 3d point clouds.  In European conference on computer vision, pp. 15–29. Springer, 2012.
  * [17] C. Linegar, W. Churchill, and P. Newman.  Made to measure: Bespoke landmarks for 24-hour, all-weather localisation with a camera.  In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 787–794. IEEE, 2016.
  * [18] L. Liu, H. Li, and Y. Dai.  Efficient global 2d-3d matching for camera localization in a large-scale 3d map.  In Proceedings of the IEEE International Conference on Computer Vision, pp. 2372–2381, 2017.
  * [19] D. G. Lowe.  Distinctive image features from scale-invariant keypoints.  Int. J. Comput. Vision, 60(2):91–110, Nov. 2004. doi: 10 . 1023/B:VISI . 0000029664 . 99615 . 94
  * [20] S. Lynen, T. Sattler, M. Bosse, J. A. Hesch, M. Pollefeys, and R. Siegwart.  Get out of my lab: Large-scale, real-time visual-inertial localization.  In Robotics: Science and Systems, vol. 1, 2015.
  * [21] S. Middelberg, T. Sattler, O. Untzelmann, and L. Kobbelt.  Scalable 6-dof localization on mobile devices.  In D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, eds., Computer Vision – ECCV 2014, pp. 268–283. Springer International Publishing, Cham, 2014.
  * [22] J.-M. Morel and G. Yu.  Asift: A new framework for fully affine invariant image comparison.  SIAM J. Img. Sci., 2(2):438–469, Apr. 2009. doi: 10 . 1137/080732730
  * [23] P. Moulon, P. Monasse, and R. Marlet.  Adaptive structure from motion with a contrario model estimation.  In Asian Conference on Computer Vision, pp. 257–270. Springer, 2012\.
  * [24] P. Moulon, P. Monasse, R. Perrot, and R. Marlet.  Openmvg: Open multiple view geometry.  In International Workshop on Reproducible Research in Pattern Recognition, pp. 60–74. Springer, 2016.
  * [25] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós.  ORB-SLAM: a versatile and accurate monocular SLAM system.  IEEE Transactions on Robotics, 31(5):1147–1163, 2015. doi: 10 . 1109/TRO . 2015 . 2463671
  * [26] T. Qin, P. Li, and S. Shen.  Vins-mono: A robust and versatile monocular visual-inertial state estimator.  IEEE Transactions on Robotics, 34(4):1004–1020, 2018.
  * [27] T. Sattler, B. Leibe, and L. Kobbelt.  Fast image-based localization using direct 2d-to-3d matching.  In 2011 International Conference on Computer Vision, pp. 667–674. IEEE, 2011.
  * [28] T. Sattler, B. Leibe, and L. Kobbelt.  Improving image-based localization by active correspondence search.  In European conference on computer vision, pp. 752–765. Springer, 2012.
  * [29] T. Sattler, W. Maddern, C. Toft, A. Torii, L. Hammarstrand, E. Stenborg, D. Safari, M. Okutomi, M. Pollefeys, J. Sivic, et al.  Benchmarking 6dof outdoor visual localization in changing conditions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8601–8610, 2018.
  * [30] G. Schall, D. Wagner, G. Reitmayr, E. Taichmann, M. Wieser, D. Schmalstieg, and B. Hofmann-Wellenhof.  Global pose estimation using multi-sensor fusion for outdoor augmented reality.  In 2009 8th IEEE International Symposium on Mixed and Augmented Reality, pp. 153–162, 2009.
  * [31] T. Schneider, M. T. Dymczyk, M. Fehr, K. Egger, S. Lynen, I. Gilitschenski, and R. Siegwart.  maplab: An open framework for research in visual-inertial mapping and localization.  IEEE Robotics and Automation Letters, 2018. doi: 10 . 1109/LRA . 2018 . 2800113
  * [32] H. Strasdat, J. M. Montiel, and A. J. Davison.  Visual slam: why filter?  Image and Vision Computing, 30(2):65–77, 2012.
  * [33] L. Svärm, O. Enqvist, F. Kahl, and M. Oskarsson.  City-scale localization for cameras with known vertical direction.  IEEE transactions on pattern analysis and machine intelligence, 39(7):1455–1461, 2016.
  * [34] T. Taketomi, H. Uchiyama, and S. Ikeda.  Visual slam algorithms: a survey from 2010 to 2016.  IPSJ Transactions on Computer Vision and Applications, 9(1):16, 2017\.
  * [35] J. Ventura, C. Arth, G. Reitmayr, and D. Schmalstieg.  Global localization from monocular SLAM on a mobile phone.  IEEE Transactions on Visualization and Computer Graphics, 20(4):531–539, apr 2014. doi: 10 . 1109/tvcg . 2014 . 27
  * [36] J. Ventura and T. Höllerer.  Wide-area scene mapping for mobile visual tracking.  In 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 3–12, 2012.
  * [37] B. Zeisl, T. Sattler, and M. Pollefeys.  Camera pose voting for large-scale image-based localization.  In The IEEE International Conference on Computer Vision (ICCV), December 2015.
  * [38] Z. Zhang and D. Scaramuzza.  A tutorial on quantitative trajectory evaluation for visual (-inertial) odometry.  In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7244–7251. IEEE, 2018.
  * [39] S. Zhu, T. Shen, L. Zhou, R. Zhang, J. Wang, and T. Fang.  Parallel structure from motion from local increment to global averaging.  2017\.
  * [40] S. Zhu, R. Zhang, L. Zhou, T. Shen, T. Fang, P. Tan, and L. Quan.  Very large-scale global sfm by distributed motion averaging.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4568–4577, 2018.


# Evaluating Visual Number Discrimination in Deep Neural Networks

Ivana Kajić<EMAIL_ADDRESS>
DeepMind, Montréal, QC, Canada
Aida Nematzadeh<EMAIL_ADDRESS>
DeepMind, London, United Kingdom

###### Abstract

The ability to discriminate between large and small quantities is a core
aspect of basic numerical competence in both humans and animals. In this work,
we examine the extent to which the state-of-the-art neural networks designed
for vision exhibit this basic ability. Motivated by studies in animal and
infant numerical cognition, we use the numerical bisection procedure to test
number discrimination in different families of neural architectures. Our
results suggest that vision-specific inductive biases are helpful in
numerosity discrimination, as models with such biases have lowest test errors
on the task, and often have psychometric curves that qualitatively resemble
those of humans and animals performing the task. However, even the strongest
models, as measured on standard metrics of performance, fail to discriminate
quantities in transfer experiments with differing training and testing
conditions, indicating that such inductive biases might not be sufficient.

## Basic Numerical Competence

The ability to represent abstract numbers and compare numerical quantities is
a basic numerical competence observed in both animals and humans (?, ?). It
helps animals in foraging, navigation, hunting, and reproduction (?, ?), and
is also correlated with the later mathematical ability in prelinguistic
infants (?, ?, ?). While such a skill is shared across species and is
independent of explicit feedback or formal education (?, ?, ?), the degree to
which more advanced numerical skills, such as counting and symbolic
representation of number, are present across species remains a debated topic
(?, ?, ?, ?, ?).

To investigate number representation and processing, different neural networks
have been used as cognitive models of various numerical skills such as
magnitude comparison (?, ?, ?, ?), subitizing (?, ?) and counting (?, ?, ?).
Neural networks are able to encode exact magnitudes (?, ?) and develop basic
numerical abilities such as numerosity comparison (?, ?).

While such networks have been used successfully to explain different phenomena
in numerical cognition, their architecture is often designed for a task
targeting specific cognitive function. In contrast to such specialized
networks, in recent years we have witnessed a radical improvement in both the
performance, and the quality of representations learned by deep neural
networks that are trained end-to-end across vision (?, ?, ?), language (?, ?,
?, ?), and multimodal (?, ?, ?, ?) domains.

Here, we investigate whether state-of-the-art models designed for visual
processing, also referred to as vision encoders, can exhibit basic numerical
competence as observed in humans and animals. Specifically, we evaluate
_number discrimination_ in vision encoders, defined as the ability to make
broad relative numerical judgements such as many versus few, which is
imprecise and not as advanced as counting, but within the normal ability of
many animals (?, ?). We draw inspiration from studies in animal and child
cognition and use a simple discrimination paradigm known as the _bisection
task_ to examine if recent vision encoders can learn to discriminate stimuli
on the basis of number.

We consider three vision encoders with varying degrees of explicit inductive
biases: ResNet (?, ?), ViT (?, ?), and Swin (?, ?), as well a simple,
comparatively small, multi-layer perception (MLP) not designed for vision
tasks as a baseline. Across all conditions, Swin and ResNet with image-
specific inductive biases are the most successful models in number
discrimination; moreover, Swin matches the empirical data from humans and
animals in more conditions than ResNet suggesting that its additional
hierarchical bias results in a better abstract number representation. Even the
strongest models, however, often fail in conditions that test for the transfer
of numerical skill to a new condition; for example, when models are trained on
a stimulus with solid shapes but tested on a stimulus where shapes are not
filled. Although models fail in such transfer conditions, we find that they do
learn structured number representations, forming clusters that are ordered
based on the number identity. This suggests that, unlike humans and animals
whose numerical skills generalize across different ecological contexts, vision
encoders might require additional modeling innovations or a greater quantity
and variety of data to use their learned knowledge in new situations.

## The Numerical Bisection Task

The numerical bisection task is used to assess perception of numerical
quantitites in both animals and humans. First, a participant is trained to
discriminate small and large sample numerosities by associating them with
different responses (labels such as _few_ and _many_). For example, ? (?)
train pigeons to respond to images with 1 or 2 shapes by pecking to the left
(corresponding to _few_), and to the right for images with 6 or 7 shapes
(corresponding to _many_). In ? (?), children learn to pick a green cup for 2
drumbeats, or a blue cup for 8 drumbeats in one experiment, and raise a red
glove on their left hand after 2 drumbeats and a yellow glove on their right
hand after 8 drumbeats in another experiment. The numerosities used for
training (_i.e.,_ 1, 2, or 8) are often referred to as _anchor numerosities_.

Then, to probe number discrimination, participants are subsequently tested on
intermediate numbers that are _not_ seen during training (_e.g.,_ 3 in the
previous experiment). A participant is more likely to select the response
associated with the larger anchor value (_e.g.,_ _many_), resulting in an
s-shaped psychometric curve. Such s-shaped psychometric curves have been used
to characterize basic numerical competence in rats (?, ?), pigeons (?, ?, ?),
rhesus macaques (?, ?), as well as children and adults (?, ?, ?, ?).
Qualitatively, psychometric curves documented in the literature have the
following characteristics: (1) the initial segment with smaller numerosities
is mostly labeled with _few_ , (2) intermediate segment with a gradually
increasing slope reflecting an increase in _many_ responses, (3) final segment
with the largest numerosities mostly labeled with _many_. Although these
properties characterize the majority of psychometric responses documented in
the literature, between- and within-subject variability has been observed
depending on the task and numerosity ranges (?, ?).

### Experimental Stimuli

Figure 1:  Sample visual stimuli used in the numerical bisection task. Rows
are different numerosities and columns different stimuli types. "(C)" denotes
"contours", as opposed to shapes with solid white background.

We automatically generate images with black background and white circles
varying the number of circles from 1 to 7. Similar to ? (?), we use images
with 1, 2, 6, or 7 circles as anchor numerosities for training. When designing
stimuli, previous work has identified and controlled for potential perceptual
confounds such as the size of the constituent elements (_i.e.,_ , circles in
our case), total white area, or total perimeter (?, ?, ?, ?); processing these
non-numerical features—which may be a confound in the observed numerical
discrimination behavior—can develop independently of number processing, as has
indeed been observed in children’s developmental trajectory (?, ?). To control
for such potential confounds, we generate six different stimulus categories
shown in Figure 1:

  1. 1.

_Vary Size_. This is our most general setting, where for each image, we draw
circles with radii drawn randomly from a set of 3 values ($r=\\{10,35,55\\}$).

  2. 2.

_Constant Size_. We control for the size of circles—all circles have the same
radius ($r=20$); this enables us to examine whether models can discriminate
numbers better when circles are identical compared to varied in size.

  3. 3.

_Constant Area_. In the previous condition (_Constant Size_), the white area
(covered by circles) increases as the number of circles increases. We control
for this potential confound by fixing the total white area to be constant
across stimuli. This results in smaller circles in images that depicts larger
numbers.

  4. 4.

_Constant Area (contour)_. We also examine if solid shape background has an
impact on models’ behavior; we consider a condition the same as _Constant
Area_ , but using contours instead of shapes with white background.

  5. 5.

_Constant Circumference_. While the total area is controlled for in the
_Constant Size_ condition, the total circumference of circles increases with
numerosity. Here, we control for the total circumference by keeping it
constant across stimuli.

  6. 6.

_Constant Circumference (contour)_. It is the same as _Constant Circumference_
, but using contours instead of shapes with white background.

We generate the stimuli on-the-fly for both the training and testing and store
them in-memory to be used during training and testing, with 100 images
generated for each numerosity category of one stimulus type, resulting in
overall $400$ images for training, and $1100$ images for testing for each
stimulus type. The images are of dimensionality expected by models, _i.e.,_
$224\times 224\times 3$.

## Experimental Setup

In this section, we examine a few recent families of deep neural networks
designed for computer vision (henceforth, vision encoders); all these models
have achieved impressive results on computer vision tasks (such as image
classification), but differ with respect to inductive biases that their
architecture encode. We first describe these models briefly, and then discuss
the details of our experimental setup.

#### Models.

We consider three types of vision encoders: ResNet (?, ?), ViT (?, ?), and
Swin (?, ?). The ResNet model includes a stack of convolutional neural network
(CNN) blocks that process images using convolution kernels. These kernels
introduce an explicit _locality_ bias—pixels (or features depending on the
layer) that are close spatially are combined; as a result, a model with CNN
blocks typically learns to encode low-level features (such as edges) in its
first layers, and more high-level ones (such as parts) in its last layers.

Both ViT and Swin use Transformer blocks (?, ?) consisting of feed-forward
layers and a _self-attention_ mechanism; self-attention introduces a weaker
and less explicit _locality_ bias (compared to CNNs) as a model can learn to
group neighboring image patches.111Self-attention is designed for sequential
data such as language; thus, it is less suitable for modelling the two-
dimensional spatial relations among image patches. Swin builds on ViT and
introduces an explicit _hierarchical_ bias by modifying how self-attention is
applied across different layers; more specifically, local image patches are
merged at at various stages as the depth of the model increases, resulting in
a hierarchical representation.

We use specific variants of ResNet, ViT, and Swin encoders: the ResNet-50
variant with 25.6M parameters (?, ?), ViT-B (?, ?) with 86M parameters, and
“tiny” Swin, Swin-T (?, ?), with 29M parameters. We picked the smallest ViT
and Swin variants, and a ResNet model that has a similar number of parameters
to Swin.

Finally, as a simple baseline, we consider a generic feed-forward multi-layer
perceptron (MLP) that does not include any inductive biases such as
convolutions or attention which are known to be helpful for processing of
real-world images. We use an MLP consisting of 2 hidden layers with 256 units
each, separated by ReLU non-linearities, and a final linear layer with 2
units. With 0.13M parameters and no “bells and whistles”, this makes it a
substantially smaller, yet less computationally inexpensive baseline model.

#### Training.

For each stimulus type (_e.g.,_ _Constant Area_), we train ResNet, ViT, Swin
and MLP models on data generated for that stimulus, _i.e.,_ images and their
labels (_few_ and _many_). More specifically, we add a classification head to
these models, to predict the label _few_ for images with 1 and 2 circles, and
_many_ for images with 6 and 7 circles, where labels are encoded as one-hot
vectors. All models are trained with a cross-entropy loss and L2
regularization. To get an estimate of variability in model responses for each
stimulus category, we train 10 networks by choosing a different seed that
randomly initializes network weights.

We perform a hyper-parameter search on the batch size, number of steps,
learning rate, and optimizer type to find combinations where training loss has
converged on the validation set, and where a network is achieving close to
100% accuracy on the training set. Accuracy is defined as a percentage of
correctly classified labels.222 We find that the batch size of 16, and $5000$
steps worked well for all models, although losses in some models (_e.g.,_ Swin
and ResNet) converged much faster. We use the Adam optimizer (?, ?) for MLP,
ViT, and Swin models, with learning rates of 1e-04, 5e-04, and 5e-05,
respectively. We use the SGD optimizer with a learning rate of 1e-2 for
ResNet. The models are trained using a NVidia Tesla V100 GPU.

#### Testing.

We test the models on new images of anchor numerosities (_i.e.,_ _not_ seen
during training), as well as images of novel interpolated numerosities: 3, 4
and 5. We use 100 images for each numerosity category and each stimulus type.

## Experimental Results

In Experiment 1, we investigate models’ behavior when trained and tested on
the same stimulus type. In Experiment 2, we investigate transfer of the number
discrimination skill by testing models on images from a stimulus category that
is not used in training (_i.e.,_ train on _Constant Size_ , and test on
_Constant Area_).

### Experiment 1: Number Discrimination

In this experiment, we test number discrimination using images from the same
stimulus category that is used during training (_e.g.,_ train on _Constant
Size_ , and test on _Constant Size_). We evaluate models based on the accuracy
of the stimulus test set, and the quantitative and qualitative characteristics
of psychometric curves in relation to the empirical data.

| Few (1, 2) | Many (6, 7) | Total Error (Few+Many)
---|---|---|---
| ResNet | ViT | Swin | MLP | ResNet | ViT | Swin | MLP | ResNet | ViT | Swin | MLP
Vary Size | 3.0 | 11.8 | 0.2 | 17.4 | 3.6 | 11.9 | 0.6 | 7.8 | 3.3 | 11.8 | 0.4 | 12.6
Const. Size | 0.0 | 35.5 | 0.0 | 32.1 | 0.0 | 0.3 | 0.1 | 5.0 | 0.0 | 17.9 | 0.0 | 18.6
Const. Area | 1.1 | 33.0 | 0.0 | 7.5 | 0.8 | 7.2 | 0.1 | 2.8 | 0.9 | 20.1 | 0.1 | 5.1
Const. Area (C) | 0.4 | 8.0 | 0.0 | 63.2 | 1.5 | 12.8 | 0.3 | 10.1 | 0.9 | 10.4 | 0.1 | 36.6
Const. Circ. | 0.6 | 0.4 | 0.0 | 3.5 | 0.0 | 3.5 | 0.0 | 51.8 | 0.3 | 1.9 | 0.0 | 27.6
Const. Circ. (C) | 8.7 | 40.8 | 0.7 | 31.1 | 9.9 | 35.0 | 12.8 | 44.6 | 9.3 | 37.9 | 6.8 | 37.9

Table 1: Error rates (%) in classifying anchor numerosities as either “few” or
“many” on respective test sets. Highest error rates for each stimulus type and
each anchor numerosity are highlighted. Highest total error rates across
stimuli for each model are underlined.

#### Performance on seen numbers.

We first examine the performance of the four architectures when tested on
novel images of the anchor numerosities (seen during training): 1, 2, 6, 7. An
error occurs when an image with a small numerosity (_i.e.,_ 1 or 2 circles) is
classified as _many_ , or when an image with a large numerosity (_i.e.,_ 6 or
7) is classified as _few_. Average error rates for each network and each
stimulus type are shown in Table 1. Overall, we observe that ResNet and Swin,
with image-specific inductive biases, have smallest mean error rates of less
than 1% in 6/12 and 11/12 conditions, respectively. ViT has mean error rates
that are in some cases comparable to or even exceed errors of the MLP
baseline. When averaged across all 4 numerosities, we find that highest error
rates are consistently observed with the _Constant Circumference (contour)_
stimulus category (See Table 1, column “Total error”); suggesting that this
combination of visual features represented the most challenging dataset for
number abstraction. Meanwhile, no such consistent pattern exists for datasets
resulting in smallest errors—the smallest error for ResNet is observed with
_Constant Size_ , for ViT and Swin with _Constant Circumference_ , and for the
MLP with _Constant Area_. This observation is not surprising given that these
models encode different inductive biases.

#### Performance on new numbers.

Next, we examine how different models perform on numbers not seen at the
training time (_i.e.,_ 3, 4, and 5). We plot the psychometric curves for
selected stimuli, showing percentages of _many_ responses across numerosities
for models trained on that stimuli in Figure 2. We selected these stimulus
categories as representative of the easiest (_Constant Size_ , _Constant
Circumference_) and hardest (_Constant Circumference (contour)_) conditions
based on the average error rates in Table 1. Different from Table 1, each
value on the y-axis represents a proportion of _many_ responses for a certain
numerosity (x-axis).

Overall, some curves in Fig. 2 exhibit characteristics of typical psychometric
functions as discussed in Sec. Experimental Setup—specifically, for small
numerosities 1, 2, and sometimes 3, we observe a slowly accelerating initial
segments, followed by gradual increase with intermediate numbers, and a slowly
decelerating final segment for larger numerosities (6, 7). Examples of such
curve profiles are Swin and ResNet responses to _Constant Size_ , and
_Constant Circumference_ stimulus categories. However, there are also curves
that have atypical flat shapes indicative of failure to learn this task,
_i.e.,_ those not found in the literature. Out of all 24 curves we analyzed, 4
in total exhibit such a shape with 3 shown in Fig. 2 (_i.e.,_ MLP with
_Constant Circumference_ and MLP with _Constant Circumference (contour)_ , ViT
with _Constant Circumference (contour)_). Such curves either do not start at,
or do not end at expected values indicating lack of sensitivity to number
categories, some of which is also evident based on large error values in Table
1.

Figure 2:  Psychometric functions for each model trained and tested on one
stimulus type on the numerical bisection task. Vertical bars are 95%
bootstrapped CIs.

### Experiment 2: Transfer to Novel Stimuli

Figure 3:  Psychometric curves in transfer experiments. Bars are standard
errors of the mean. Models are represented across rows, and selected training
datasets in columns. Shades of blue indicate the test stimulus, orange curves
denote train and test on the same stimulus type.

The previous experiment demonstrates that Swin and ResNet architectures to a
great degree appear to be able to differentiate number of items in an image
when trained and tested on the same stimulus type (_e.g.,_ constant total area
or circumference). To understand whether our models have indeed developed a
notion of a number category as opposed to learning a given stimuli, we draw a
parallel with research in animal cognition and examine if the models “ _base
their behaviour on the numerosity of a set, independent of its other
attributes_ ” (?, ?). In other words, if models learn an abstract
representation of a number category, we would expect this representation to be
agnostic to perceptual features of the stimulus. To test this, we examine
models in a cross-stimulus transfer setting: we train a model on one set of
stimuli, but test it on other types of stimuli (_i.e.,_ train on _Constant
Size_ , test on _Constant Circumference_). The test stimuli are _out of
distibution (OOD)_ with respect to the model’s training distribution. Compared
to the _in distribution_ setting where training and test are drawn from the
same distribution, the OOD setting is known to be challenging for neural
networks (?, ?).333The degree to which a stimulus category is OOD depends on
the target distribution, as some stimulus categories are correlated across
different dimensions; _e.g.,_ , total cumulative white area increases on
average with numerosity for both _Constant Size_ and _Vary Size_.

Figures 3A) and 3B) show a selected subset of psychometric curves evaluated
using such a cross-stimulus protocol, for models trained on _Vary Size_ and
_Constant Circumference_. Orange curves denote cases where a model is trained
and tested on the same stimulus category (_i.e.,_ the protocol from Exp. 1),
and are included for reference, while blue curves are obtained when models are
tested on datasets different from the ones they are trained on. We report
results on _Vary Size_ and _Constant Circumference_ since they exhibit the
most successful (_Vary Size_) and least successful (_Constant Circumference_)
transfer cases, as defined by the expected qualititative characteristics of
psychometric curves in Sec. The Numerical Bisection Task. Even for the best
matching condition _Vary Size_ (Fig. 3A), we observe a number of _transfer
failures_ , where a trained model shows poor transfer of number discrimination
ability to a novel stimulus category, revealing a failure to abstract the
number category.

An interesting case of transfer failures is evident in _all-or-none_
responses, where models unanimously assign either _few_ or _many_ response to
all numerosities. This has been observed across all models, and is
particularly prominent with models trained on _Constant Circumference_ (Fig. 3
B). In some cases, most notably with MLP and to a smaller extent with ViT, we
also observe flatter curves with smaller slopes, resulting from frequent
misclassification of _many_ responses as _few_ and vice versa. Finally, we
also observe a new response pattern, an _inverted_ psychometric curve, where
small numerosities are overwhelmingly assigned label _many_ , and the opposite
for large numerosities. Fig. 3B) showcases that this pattern is consistent
across models trained on _Constant Circumference_. We conjecture that this is
due to models latching onto total white area during training, which is
inversely correlated with numerosity in _Constant Circumference_.

Next, we consider an easier case of transfer, and examine if exposing models
to more a diverse set of stimuli (as opposed to one type of stimulus) can help
in learning a better representation of number categories; we train models on
all but one stimulus type, and evaluate them on the hold-out stimulus type.
Instead of 100 images per number category, a model is seeing 500 images per
number category (_i.e.,_ 100 images for a number for each of the 5 training
stimulus types). As shown in Fig. 3C), we see that increasing data variability
results in more curves that resemble the expected s-shaped curve, especially
for ResNet and Swin. However, even then, models failed to generalize to
_Constant Circumference (contour)_ , confirming again the difficulty of this
stimulus category. Overall, we find that Swin and ResNet produce
representations that better match observed empirical data even in the more
challenging transfer setting. We also observe that the training models on a
variety of stimulus types help in generalising to new stimulus.

Finally, we examine if learned number representations form meaningful
clusters; to answer this question, we do a forward pass on images from a given
stimulus category for two models with lowest error rates (_i.e.,_ ResNet and
Swin). For each image, we extract embeddings from the last dense layer of the
model, prior to the 2-unit classification head. We use PCA followed by the
t-SNE (?, ?) dimensionality reduction method to project high-dimensional
embedding vectors ($2048$ for ResNet and 768 for Swin) into 2D space. In Fig.
4A) we show one selected example of such a projection, where individual points
have been color-coded based on the numerosity of the stimulus image. First, we
observe that embeddings cluster in groups based on number, with a greater
cluster overlap for subsequent numbers. Second, we observe an ordering of
clusters based on numerosities. This type of pattern is observed more often
with embeddings from Swin, compared to ResNet embeddings which generally
result in less discernible clusters (with the exception of clusters for
numerosities 1 and 2). Interestingly, based on visual inspection of the data,
we do not find that more distinct projections suggest better performance on
the task. For example, while clusters in Fig. 4A) seem to be discernible based
on number, the model performs poorly when tested on _Constant Size_ , possibly
because the classifier does not discriminate based on the dimensions that are
discriminable in the embeddings.

Figure 4: A) t-SNE projections of _Constant Size_ image embeddings for Swin
trained on _Constant Area_. B) Numerosities of image pairs for which there was
no significant difference in the percentage of “many” responses.

## Discriminability of Small vs. Large Numbers

Empirical data from humans and animals shows that in the numerical bisection
task, it is consistently more difficult to distinguish larger numerosities
from each other, compared to smaller ones (?, ?, ?). This observation is
likely to be related to a more general finding in numerical cognition that
small numbers are processed differently than larger numbers (?, ?, ?).

We examine whether similar observations can be made for our models’ responses,
using a measure of discriminability for different pairs of numbers. As an
example, for a given model and a pair of numbers (such as 5 and 6), we
statistically test if the mean percentage of _many_ responses for images of
one number (5) is the same as that of images of the other number (6).
Intuitively, the two numbers are harder to discriminate if their mean
percentage of _many_ responses are the same. We consider models with smallest
test error rates in Experiment 1 (_i.e.,_ ResNet and Swin). For a given model
and each stimulus type, we consider all possible number pairs (_i.e.,_ all
points on average psychometric curves) and perform the Tukey’s HSD test for
multiple pair-wise comparison of means (with family-wise error rate
FWER=$.05$). This approach is based on the similar statistical tests used with
pigeon responses in ? (?).

In Fig. 4B), we show the breakdown of 18 cases (out of total 228 comparisons)
where we failed to reject the null hypothesis across number pairs;
intuitively, the models find it difficult to discriminate between these number
pairs.444From this plot we excluded 24 comparisons for number pairs (1, 2) and
(6, 7) since means within these pairs are the same by the design of networks’
training objective. The figure shows that pairs at the higher end of the
numerical range, such as (5, 7) and (5, 6) are frequently indistinguishable
which is in contrast to the pairs on the lower end of the numerical scale. We
conclude that similar to the empirical data, Swin and ResNet better
distinguish numerosities at the lower end of the number range compared to
those of the higher end of number ranges. Moreover, this effect is stronger
among Swin responses compared to ResNet, suggesting that number
representations learned by Swin are more discernible.

## Discussion

Number discrimination is a core aspect of basic numerical competence in humans
and animals. We investigate if recent, state-of-the-art neural networks used
in computer vision exhibit the capacity of discriminating between small and
large quantities. We evaluate these models on the numerical bisection task
where models learn to categorize numerosity of sets of items, and we
investigate their performance on novel stimuli and novel numerosities.

We find that ResNet and Swin, the two models with vision-specific inductive
biases, achieve the smallest errors when categorizing novel stimuli as _few_
or _many_. Psychometric curves of models trained on a wide range of stimuli,
as well as those of models trained and tested on the same type of stimuli,
often resemble the response curves of animals and humans on the same task. In
addition, Swin responses are more discernible for smaller numbers compared to
larger numbers, and its internal representations are structured in a way that
reflects number category and order. Swin’s predecessor, ViT, which is also a
transformer-based model, albeit with a weak image-specific inductive bias, has
errors on the task that are comparable to or even higher than the basic,
substantially smaller MLP baseline. This is surprising considering that
performance of ViT is within a few percentage points of Swin performance on
different computer vision benchmarks (?, ?).

Finally, when controlled for perceptual attributes (_e.g.,_ keeping the total
white area constant during training, but varying area during testing), most of
these models show poor transfer of the number discrimination skill. This might
mean that models latch onto features that, while correlated with number, are
considered non-numerical in the literature (?, ?, ?). When analyzing the
internal representations of number in Swin, we find that often, despite poor
transfer, number representations are structured in an interpretable way. In
other words, although these representations could in theory support number
discrimination, we do not observe this in practice. One possible reason for
poor transfer might be that the models are trained in a limited data regime,
in contrast to humans and animals whose numerical cognition develops gradually
in a rich environmental context, and who might be biologically predisposed to
represent and process numerical quantities (?, ?). Future work should explore
whether pretraining models on larger and more diverse sets of images would
result in a more transferable skill. Finally, we only investigate one specific
task—the numerical bisection, and it remains to be explored whether our
findings generalize across other perceptual domains.

### Acknowledgments

The authors would like to thank Stephanie Chan for detailed feedback on this
manuscript as well as other colleagues at DeepMind for feedback and
discussions that helped improve this work.

## References

  * Alayrac et al.Alayrac et al. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., … Simonyan, K.  (2022).  Flamingo: a visual language model for few-shot learning.  In A. H. Oh, A. Agarwal, D. Belgrave, K. Cho (Eds.), _Advances in neural information processing systems._
  * Almeida, Arantes, MachadoAlmeida et al. Almeida, A., Arantes, J., Machado, A.  (2007, November).  Numerosity discrimination in preschool children.  _J. Exp. Anal. Behav._ , _88_(3), 339–354.
  * Anobile, Cicchini, BurrAnobile et al. Anobile, G., Cicchini, G. M., Burr, D. C.  (2016).  Number as a primary perceptual attribute: A review.  _Perception_ , _45_(1-2), 5–31.
  * Brown et al.Brown et al. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., … others  (2020).  Language models are few-shot learners.  _Advances in neural information processing systems_ , _33_ , 1877–1901.
  * Creatore, Sabathiel, SolstadCreatore et al. Creatore, C., Sabathiel, S., Solstad, T.  (2021).  Learning exact enumeration and approximate estimation in deep neural network models.  _Cognition_ , _215_ , 104815.  doi: https://doi.org/10.1016/j.cognition.2021.104815
  * Davis MemmottDavis Memmott Davis, H., Memmott, J.  (1982, November).  Counting behavior in animals: A critical evaluation.  _Psychol. Bull._ , _92_(3), 547–571.
  * DehaeneDehaene Dehaene, S.  (1997).  _The number sense: How the mind creates mathematics_ [Book].  Oxford University Press New York.
  * Dehaene ChangeuxDehaene Changeux Dehaene, S., Changeux, J.-P.  (1993).  Development of elementary numerical abilities: A neuronal model.  _Journal of cognitive neuroscience_ , _5_(4), 390–407.
  * Dehaene, Dehaene-Lambertz, CohenDehaene et al. Dehaene, S., Dehaene-Lambertz, G., Cohen, L.  (1998).  Abstract representations of numbers in the animal and human brain.  _Trends in neurosciences_ , _21_(8), 355–361.
  * Devlin, Chang, Lee, ToutanovaDevlin et al. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.  (2018).  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dosovitskiy et al.Dosovitskiy et al. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., … others  (2020).  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International conference on learning representations._
  * Droit-Volet, Clément, FayolDroit-Volet et al. Droit-Volet, S., Clément, A., Fayol, M.  (2003, January).  Time and number discrimination in a bisection task with a sequence of stimuli: a developmental approach.  _J. Exp. Child Psychol._ , _84_(1), 63–76.
  * Emmerton, Lohmann, NiemannEmmerton et al. Emmerton, J., Lohmann, A., Niemann, J.  (1997, June).  Pigeons’ serial ordering of numerosity with visual arrays.  _Anim. Learn. Behav._ , _25_(2), 234–244.
  * Fang, Zhou, Chen, McClellandFang et al. Fang, M., Zhou, Z., Chen, S., McClelland, J.  (2018).  Can a recurrent neural network learn to count things?  In _Cogsci._
  * Gallistel GelmanGallistel Gelman Gallistel, C. R., Gelman, R.  (1992).  Preverbal and verbal counting and computation.  _Cognition_ , _44_(1-2), 43–74.
  * Geirhos et al.Geirhos et al. Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., Wichmann, F. A.  (2020).  Shortcut learning in deep neural networks.  _Nature Machine Intelligence_ , _2_(11), 665–673.
  * Gilmore, McCarthy, SpelkeGilmore et al. Gilmore, C. K., McCarthy, S. E., Spelke, E. S.  (2007).  Symbolic arithmetic knowledge without instruction.  _Nature_ , _447_(7144), 589–591.
  * Halberda, Mazzocco, FeigensonHalberda et al. Halberda, J., Mazzocco, M. M., Feigenson, L.  (2008).  Individual differences in non-verbal number acuity correlate with maths achievement.  _Nature_ , _455_(7213), 665–668.
  * He, Zhang, Ren, SunHe et al. He, K., Zhang, X., Ren, S., Sun, J.  (2016).  Deep residual learning for image recognition.  In _Proceedings of the ieee conference on computer vision and pattern recognition_ (pp. 770–778).
  * Honig StewartHonig Stewart Honig, W. K., Stewart, K. E.  (1989, June).  Discrimination of relative numerosity by pigeons.  _Anim. Learn. Behav._ , _17_(2), 134–146.
  * Jordan BrannonJordan Brannon Jordan, K. E., Brannon, E. M.  (2006).  Weber’s law influences numerical representations in rhesus macaques (macaca mulatta).  _Animal cognition_ , _9_(3), 159–172.
  * Kingma BaKingma Ba Kingma, D. P., Ba, J.  (2014).  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Liu et al.Liu et al. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., … Guo, B.  (2021).  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the ieee/cvf international conference on computer vision_ (pp. 10012–10022).
  * Lu, Batra, Parikh, LeeLu et al. Lu, J., Batra, D., Parikh, D., Lee, S.  (2019).  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _Advances in neural information processing systems_ , _32_.
  * Meck ChurchMeck Church Meck, W. H., Church, R. M.  (1983, July).  A mode control model of counting and timing processes.  _J. Exp. Psychol. Anim. Behav. Process._ , _9_(3), 320–334.
  * NiederNieder Nieder, A.  (2020).  The adaptive value of numerical competence.  _Trends in Ecology & Evolution_, _35_(7), 605–617.
  * OdicOdic Odic, D.  (2018).  Children’s intuitive sense of number develops independently of their perception of area, density, length, and time.  _Developmental Science_ , _21_(2), e12533.
  * O’Shaughnessy, Gibson, PiantadosiO’Shaughnessy et al. O’Shaughnessy, D. M., Gibson, E., Piantadosi, S. T.  (2021).  The cultural origins of symbolic number.  _Psychological review_.
  * Peterson SimonPeterson Simon Peterson, S. A., Simon, T. J.  (2000).  Computational evidence for the subitizing phenomenon as an emergent property of the human cognitive architecture.  _Cognitive Science_ , _24_(1), 93–122.
  * Radford et al.Radford et al. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., … others  (2021).  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ (pp. 8748–8763).
  * Revkin, Piazza, Izard, Cohen, DehaeneRevkin et al. Revkin, S. K., Piazza, M., Izard, V., Cohen, L., Dehaene, S.  (2008).  Does subitizing reflect numerical estimation?  _Psychological science_ , _19_(6), 607–614.
  * Rodriguez, Wiles, ElmanRodriguez et al. Rodriguez, P., Wiles, J., Elman, J. L.  (1999).  A recurrent neural network that learns to count.  _Connection Science_ , _11_(1), 5–40.
  * Simonyan ZissermanSimonyan Zisserman Simonyan, K., Zisserman, A.  (2014).  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_.
  * Testolin, Dolfi, Rochus, ZorziTestolin et al. Testolin, A., Dolfi, S., Rochus, M., Zorzi, M.  (2020, June).  Visual sense of number vs. sense of magnitude in humans and machines.  _Sci. Rep._ , _10_(1), 10045.
  * Van der Maaten HintonVan der Maaten Hinton Van der Maaten, L., Hinton, G.  (2008).  Visualizing data using t-sne.  _Journal of machine learning research_ , _9_(11).
  * Vaswani et al.Vaswani et al. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., … Polosukhin, I.  (2017).  Attention is all you need.  _Advances in neural information processing systems_ , _30_.
  * Verguts FiasVerguts Fias Verguts, T., Fias, W.  (2004).  Representation of number in animals and humans: A neural model.  _Journal of cognitive neuroscience_ , _16_(9), 1493–1504.
  * Zorzi ButterworthZorzi Butterworth Zorzi, M., Butterworth, B.  (1999).  A computational model of number comparison.  _Proceedings of the Twenty First Annual Conference of the Cognitive Science Society_ , 772–777.