# Seeing past words: Testing the cross-modal capabilities of
pretrained V&L models

Letitia Parcalabescu1 Albert Gatt2 Anette Frank1 Iacer Calixto3,4

1Heidelberg University, Department of Computational Linguistics
2University of Malta, Institute of Linguistics and Language Technology
3New York University 4ILLC, University of Amsterdam
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We investigate the ability of general-purpose pretrained vision and language
(V&L) models to perform reasoning in two tasks that require multimodal
integration: (1) discriminating a correct image-sentence pair from an
incorrect one, and (2) counting entities in an image. We evaluate three
pretrained V&L models on these tasks: ViLBERT, ViLBERT 12-in-1 and LXMERT, in
zero-shot and finetuned settings. Our results show that models solve task (1)
very well, as expected, since all models use task (1) for pretraining.
However, none of the pretrained V&L models are able to adequately solve task
(2), our counting probe, and they cannot generalise to out-of-distribution
quantities. Our investigations suggest that pretrained V&L representations are
less successful than expected at integrating the two modalities. We propose a
number of explanations for these findings: LXMERT’s results on the image-
sentence alignment task (and to a lesser extent those obtained by ViLBERT
12-in-1) indicate that the model may exhibit catastrophic forgetting. As for
our results on the counting probe, we find evidence that all models are
impacted by dataset bias, and also fail to individuate entities in the visual
input.

##  1 Introduction

Recently, many vision and language (V&L) models have been proposed that
combine images and text using self-supervised pretraining Lu et al. (2019);
Tan and Bansal (2019); Li et al. (2019); Chen et al. (2020); Li et al. (2020);
Su et al. (2020). All these models are designed as general-purpose models for
grounding language in vision and follow the pretrain-and-finetune paradigm,
i.e. they are pretrained on large amounts of image-caption pairs111Sometimes
models are also pretrained on other image-text datasets, e.g., visual question
answering data. and are then finetuned on the task(s) of interest. Such V&L
models have obtained state-of-the-art performance across a number of different
V&L tasks, e.g. visual question answering (VQA); visual commonsense reasoning;
grounding referring expressions; and image retrieval, among others.

Pretrained V&L models use a combination of masked multimodal modelling – i.e.,
masking out words and object bounding boxes from the input and predicting them
– and image-sentence alignment, i.e., predicting whether an image-sentence
pair is correctly aligned or not. Such models hold the promise of partially
addressing the ‘meaning gap’ in unimodal pretrained language models such as
BERT Devlin et al. (2019) by directly connecting language to visual
representations Bender and Koller (2020); Bisk et al. (2020).

In this paper, we investigate how well pretrained V&L models integrate and
reason upon textual and visual representations. To do that, we directly probe
these models with tasks that require this integration to be adequately solved.
One technique that has proved to be a useful diagnostic tool to interpret the
performance of V&L models is foiling, whereby a model needs to distinguish
between correct and mismatched text-image pairs Shekhar et al. (2017).
Inspired by the foiling approach, we propose two tasks: (1) discriminating a
correctly aligned image-sentence pair from an incorrectly aligned one, and (2)
counting entities in the image.

V&L models are commonly pretrained on task (1), and should not have many
difficulties detecting incorrect image-sentence pairs. Matching image-sentence
pairs has featured in previous work on foiling, albeit in the context of
specific tasks with relatively small datasets Shekhar et al. (2017).

Counting, our task (2), nicely puts together visual and textual reasoning. It
requires the detection of object instances in the visual input, mapping these
instances to categories, as well as properly aligning such instances to
references in the textual input. There are many recent works investigating
model architectures specifically designed for counting, which is known to be a
hard problem for V&L models Zhang et al. (2018); Acharya et al. (2019); Trott
et al. (2018); Chattopadhyay et al. (2017). Some related V&L work has also
investigated generalised quantifiers (such as most) in a V&L context, but this
work has generally exploited synthetic datasets Sorodoc et al. (2018);
Pezzelle and Fernández (2019); Testoni et al. (2019). Here, we task the model
to judge whether an unambiguous question or statement about the number of
entities visible in an image is correct or not. While this is similar to prior
work on counting, here we focus on general-purpose pretrained V&L models
instead of specialised models to handle the counting task.

We use three publicly available, representative V&L models in our
investigation: LXMERT222github.com/huggingface/transformers Tan and Bansal
(2019), ViLBERT and ViLBERT 12-in-1333github.com/facebookresearch/vilbert-
multi-task (Lu et al., 2019, 2020). ViLBERT and ViLBERT 12-in-1 use the same
BERT-based model architecture, which incorporates two separate visual and
linguistic streams that interact through multiple co-attention transformer
layers. ViLBERT is trained using self-supervised learning on image-caption
pairs, while ViLBERT 12-in-1 is further finetuned on 12 different tasks using
multi-task learning. LXMERT is also a dual-stream architecture and combines
textual and visual transformer-based encoders with cross-modal layers.
However, LXMERT is pretrained not only on image-caption pairs but also
directly on the visual question answering (VQA) task using multiple VQA
datasets.

With these three models we are able to construct different experimental
settings for probing: while all models are trained on image-sentence
alignment, only ViLBERT is not directly trained on VQA; hence the model can be
probed on our counting task in a zero-shot setting. LXMERT, by contrast, is
directly pretrained on VQA datasets (also including how many questions, the
focus in our counting probe). Hence, it was exposed to examples where
answering a question correctly would presumably hinge on adequately performing
the subtasks noted above, namely detection and categorisation of instances in
an image and alignment of these to text. Finally, ViLBERT 12-in-1 is also
finetuned on many V&L tasks, including VQA instances which have numerical
answers, hence requiring counting abilities. Hence, we believe it serves as a
solid baseline and should be well equipped to detect foiled probing instances.
However, to our surprise, we find that none of these models perform
particularly well in our probing experiments.

Our main contributions are: i) We show that all three pretrained V&L models
perform image-sentence alignment well, as expected given that they are
pretrained on this task; ii) We build a counting probe, a task which can only
be resolved by a V&L model if it can adequately perform cross-modal grounding;
iii) We find that all models perform similarly to the random baseline when
directly applying the image-sentence alignment head to perform counting
without finetuning; iv) We find that ViLBERT, ViLBERT 12-in-1 and LXMERT seem
to exploit dataset bias and only partially solve our counting probe even when
finetuned, and cannot generalise to out-of-distribution quantities.444We will
make our dataset available upon publication.

##  2 V&L Models

The pretrained V&L models we use are ViLBERT (Lu et al., 2019), ViLBERT
12-in-1 (Lu et al., 2020), and LXMERT (Tan and Bansal, 2019). ViLBERT is
pretrained on Google Conceptual Captions (Sharma et al., 2018) on (i)
multimodal masked prediction, i.e., masking objects and words and predicting
them; and (ii) image-sentence alignment, i.e., determining whether a text
corresponds to an image or not. LXMERT uses losses (i) and (ii) and in
addition is pretrained on multiple VQA datasets, as well as object labelling.
Finally, ViLBERT 12-in-1 starts from a pretrained ViLBERT model checkpoint and
is additionally finetuned on 12 different tasks, once again including VQA.

###  2.1 Evaluation

Our experiments use image-sentence alignment and counting probes. In both
cases, models are exposed to either correct or foiled image-text pairs. We
evaluate pretrained V&L models on our probes using accuracy ($acc$), which is
the overall accuracy on all classes; precision ($p_{c}$), which measures how
well the models identify the correct examples; and foil precision ($p_{f}$),
which measures how well a model identifies foiled instances:

$\begin{split}acc&=\frac{P+N}{P+N+\tilde{P}+\tilde{N}},\\\
p_{c}&=P/(P+\tilde{P}),\\\ p_{f}&=N/(N+\tilde{N}),\end{split}$

where $P$ and $N$ are the number of true positives and true negatives, and
$\tilde{P}$ and $\tilde{N}$ are the number of false positives and false
negatives, respectively.

We also evaluate our models using a pairwise ranking accuracy $acc_{r}$
computed using the image-sentence alignment score $\phi$ that the model
assigns to correct and foiled image-text pairs. Given an image ($i$) paired
with a correct ($c$) versus a foil ($f$) text, if the score of the
positive/correct pair is greater than that of the foiled pair, the prediction
is considered successful.

$\begin{split}acc_{r}&=\frac{\sum_{(i,c)\in C}\sum_{f\in
F}s(i,c,f)}{|C|+|F|},\\\ s(i,c,f)&=\begin{cases}1,&\text{if
}\phi(i,f)\leq\phi(i,c),\\\ 0,&\text{otherwise,}\end{cases}\end{split}$

where $C$ is the set of correct image-caption pairs ($i,c$), and $F$ is the
set of foils for the pair ($i,c$).

##  3 Image-Sentence Alignment Probe

In this set of experiments, we probe whether pretrained V&L models can
distinguish correct image-sentence pairs from foiled ones. While all models
under consideration have been pretrained on this task, results are not usually
reported for pretraining. Our aim is to explicitly establish their
capabilities on a fundamental V&L task that we would expect them to perform
well at, before probing them on the more challenging counting task.

###  3.1 Data

To probe our models on the image-sentence alignment task, we construct
evaluation sets using 5000 images each from the MSCOCO Lin et al. (2014) and
Google Conceptual Captions (GCC; Sharma et al., 2018) validation splits MSCOCO
images are collected from Flickr and its captions are crowdsourced. GCC’s
images are obtained from the web with captions harvested from online alt-text
enabled sources, and therefore contain more noise and variability. ViLBERT is
pretrained on GCC image-caption pairs; LXMERT is pretrained on five datasets
including MSCOCO, but not GCC. For both datasets we select one correct caption
for each image, and create foils by pairing the image to one random caption
from the remaining 4999 images. All models are tested on the same data.

###  3.2 Experiments

In these experiments, we probe pretrained V&L models without any additional
fine-tuning. Table 1 reports the results of applying the models’ pretrained
image-sentence alignment prediction head to image-caption pairs from MSCOCO
and GCC. We also highlight which models can be considered ‘zero-shot’ in this
setting: GCC is used to pretrain the two ViLBERT models, while MSCOCO is used
when pretraining LXMERT.555We believe that LXMERT’s official documentation
(https://huggingface.co/transformers/model_doc/lxmert.html) incorrectly
describes the image-sentence alignment logits (correct – index 0 and foiled –
index 1), and we use the inverse of what is said in the documentation.
However, we reached out to the authors and will update this paper if our
interpretation is wrong.

| Model | ZS? | $acc$ | $p_{c}$ | $p_{f}$
---|---|---|---|---|---
| Random |  | 50.0 | 50.0 | 50.0
COCO | ViLBERT | ✓ | 97.4 | 98.0 | 96.8
ViLBERT 12-in-1 | ✓ | 96.4 | 93.4 | 99.4
LXMERT | ✗ | 78.3 | 57.0 | 99.5
GCC | ViLBERT | ✗ | 96.8 | 96.7 | 96.9
ViLBERT 12-in-1 | ✗ | 84.9 | 73.1 | 96.7
LXMERT | ✓ | 61.4 | 24.9 | 97.9
Table 1: Image-sentence alignment results on our COCO and Google CC validation
sets. ’ZS?’ indicates whether the model is applied zero-shot, i.e., the model
was never trained on examples from MSCOCO/GCC. We report the overall accuracy
$acc$, precision on correct examples $p_{c}$, and precision on foiled examples
$p_{f}$.

ViLBERT performs very well on both datasets and achieves $96$–$97$ $acc$
overall. It predicts both correct and foiled examples well, as shown by
$96$–$98$ $p_{c}$ and $\sim 96$ $p_{f}$. When using ViLBERT 12-in-1, results
on GCC are considerably worse compared to ViLBERT. This is surprising, since
ViLBERT 12-in-1 was trained using more tasks and considerably more data than
ViLBERT. Finally, LXMERT performs worst overall among all three models.

These results suggest that LXMERT (and to a lesser extent, ViLBERT 12-in-1)
may be exhibiting catastrophic forgetting, a well-studied problem in neural
networks Robins (1995) which has received attention in NLP Kirkpatrick et al.
(2017); Yogatama et al. (2019) as well as in V&L tasks in particular Greco et
al. (2019): LXMERT is finetuned on visual question answering in the last 10
epochs of pretraining, and ViLBERT 12-in-1 is finetuned on 12 different tasks.
This finetuning may be responsible for the worse results observed, resulting
in a downgrading of performance on the task the models were originally
pretrained on.

In summary, all models solve the image-sentence alignment probe well (as
expected) but show notable differences in performance; we conjecture
catastrophic forgetting may be impacting the finetuning procedure of each
model differently.

##  4 Counting Probe

In our second task we probe pretrained V&L models on their ability to count,
i.e., to correctly predict the number of entities visible in an image, given
the image itself and either a corresponding question coupled with a numerical
answer, or a declarative statement about the number of entities of a specific
kind derived from the question-answer pair (see Figure 1).

###  4.1 Data

We collect our counting probe data from Visual7W (Zhu et al., 2016), a VQA
dataset with diverse question types including how many questions, where a
correct answer requires the model to count the number of entities of a certain
type in an image.

Figure 1:  How many question from Visual7W dataset.

####  4.1.1 Data Formats

The data is originally in question-answering format, where the answer is a
number. We experiment with two alternative formats.

##### Q+A format

We concatenate the original question with the separator token [SEP] and each
answer (correct and foil), e.g. the example in Fig. 1 becomes “How many
magnets are on the bottom of the fridge? [SEP] 2/3/4/5”.

##### Declarative format

ViLBERT is never pretrained on questions and answers with a separator token
[SEP]. We therefore create a version of the counting data where we transform
the question and answer into a declarative statement using simple
templates.666See Appendix A.1 for a detailed explanation on how we create and
use these templates. For instance, we create the following statements for the
example shown in Fig. 1: “There are 2/3/4/5 magnets on the bottom of the
fridge.”

We remove examples where we could not convert the question-answer pair into a
declarative statement following the templates described in Appendix A.1.

Split |  | #Train | #Valid | #Test
---|---|---|---|---
Standard | Correct | $6,001$ | $2,439$ | $3,622$
Foiled | $17,896$ | $7,283$ | $10,800$
Total | $23,897$ | $9,722$ | $14,422$
Hard | Correct | $1,567$ | $1,130$ | $1,352$
Foiled | $4,672$ | $3,378$ | $4,040$
Total | $6,239$ | $4,508$ | $5,392$
Interpolated | Correct | $3,303$ | $1,331$ | $2,013$
Foiled | $9,840$ | $3,969$ | $5,998$
Total | $13,143$ | $5,300$ | $8,011$
Table 2: Counting data statistics.

####  4.1.2 Data Splits

First, we ensure that our counting data has no leaks with respect to Visual7W
by always placing instances from a given Visual7W split into the same split in
our counting dataset.777 I.e., V7W train $\rightarrow$ counting train, V7W
valid $\rightarrow$ counting valid, V7W test $\rightarrow$ counting test.

We create three splits for our counting dataset: standard, hard, and
interpolated. In the standard split we directly use the data as is, i.e., we
include all examples of how many questions in the train, dev and test splits
in Visual7W, excluding examples that cannot be transformed into a declarative
statement with our templates. The distribution of numerals in the standard
split is highly skewed and answers such as “1” or “2” are by far the most
common (see the outer circles in Fig. 2). We mitigate this by also introducing
a hard split (depicted in the inner circles in Fig. 2), in which high-
frequency classes are capped at $k=200$ for train, dev and test sets, and any
training examples where the answer is a number greater than 20 are removed.
Finally, in the interpolated setting we split the original data so that only
examples correctly answered by even numbers are added to the training set, and
only examples correctly answered by odd numbers are added to validation and
test sets.

Figure 2: Percentage of numerals in the counting data. Outer circle: standard
split, inner circle: hard split.

In Table 2 we show the counting data statistics. We note that: the hard split
has considerably fewer examples than the other two splits, due to the capping
at $k=200$ examples per answer type; furthermore, the interpolated split also
has fewer examples than the standard split because we discard all examples
with odd answers from its training set and all examples with even answers from
its validation and test sets. The hard split is more balanced with regards to
the number of classes, whereas quantities in the _standard_ split follow a
more natural distribution, where numerals like “one”, “two” or “three” are
more common than large quantities or mentions of empty sets (Figure 2). The
less skewed distribution in the _hard_ split would be expected to be harder,
since we artificially lower the relative frequency of frequent answers.

###  4.2 Experiments

We conduct a number of experiments where counting capabilities are probed in
different ways, via _image-sentence alignment_ (Section 4.2.1), _masked
language modelling_ (MLM; Section 4.2.2), and _visual question answering_
(Section 4.2.3).

It is important to note that there is a difference between the three models in
terms of their prior exposure to the VQA task in general, and to questions
involving counting in particular. Specifically, while ViLBERT was exclusively
pretrained on GCC, LXMERT’s pretraining involved VQA in the final ten epochs,
and this included the Visual7W training set. In the case of ViLBERT 12-in-1,
VQA was also one of the tasks on which it was finetuned, and again this
included Visual7W. In the experiments reported below, we distinguish between a
no finetuning and a finetuned scenario. In the former, we present results on
models which were not directly finetuned on our counting training set,
irrespective of whether they were exposed to Visual7W during pretraining (as
in the case of LXMERT) or training (as in the case of ViLBERT 12-in-1). In the
finetuned scenario, we finetune each model using three different random seeds
and report mean and standard deviation for all metrics.

Split | Format | $acc$ | $p_{c}$ | $p_{f}$ | $acc_{r}$
---|---|---|---|---|---
Random baseline | 50.0 | 50.0 | 50.0 | 50.0
ViLBERT
std. | Q+A | 37.8 | 74.3 | 25.5 | 49.0
decl. | 37.6 | 77.9 | 24.1 | 57.0
hard | Q+A | 38.6 | 73.1 | 27.1 | 51.9
decl. | 38.0 | 75.3 | 25.5 | 55.9
LXMERT
std. | Q+A | 50.4 | 35.0 | 65.8 | 56.2
decl. | 53.8 | 40.9 | 66.6 | 71.3
hard | Q+A | 50.4 | 36.9 | 63.9 | 58.5
decl. | 51.6 | 40.8 | 62.4 | 63.4
ViLBERT 12-in-1
std. | Q+A | 43.3 | 80.2 | 30.9 | 77.3
decl. | 62.4 | 73.7 | 58.7 | 75.4
hard | Q+A | 46.9 | 67.1 | 40.1 | 70.3
decl. | 61.3 | 70.0 | 58.3 | 72.6
Table 3: Counting: test results for models without fine-tuning on our counting
dataset, including ViLBERT “zero-shot”. We report overall accuracy $acc$,
precision on correct examples $p_{c}$, precision on foiled examples $p_{f}$,
and pairwise accuracy $acc_{r}$. Splits: standard (std.) and hard. Formats:
Q+A and declarative (decl).

####  4.2.1 Counting as Image-Sentence Alignment

Split | Format | $acc$ | $p_{c}$ | $p_{f}$ | $acc_{r}$ |  | $acc$ | $p_{c}$ | $p_{f}$ | $acc_{r}$
---|---|---|---|---|---|---|---|---|---|---
Random baseline | 50.0 | 50.0 | 50.0 | 50.0 |  | 50.0 | 50.0 | 50.0 | 50.0
|  | ViLBERT |  | ViLBERT 12-in-1
std. | Q+A | 74.4 (0.2) | 49.3 (0.3) | 88.9 (0.3) | 78.2 (0.6) |  | 81.1 (0.3) | 60.3 (0.7) | 90.2 (0.2) | 83.5 (0.1)
decl. | 71.7 (3.6) | 46.3 (4.1) | 88.6 (1.1) | 76.7 (2.7) |  | 81.1 (0.1) | 60.6 (0.1) | 89.7 (0.1) | 83.3 (0.1)
hard | Q+A | 56.7 (22.2) | 16.9 (11.9) | 75.4 (0.6) | 56.2 (0.7) |  | 64.3 (4.2) | 38.7 (3.3) | 86.0 (2.7) | 69.8 (2.6)
decl. | 54.0 (21.4) | 38.6 (14.4) | 52.3 (37.0) | 57.5 (0.9) |  | 71.9 (1.9) | 46.4 (1.9) | 89.4 (0.8) | 77.5 (0.6)
interp. | Q+A | 48.0 (0.2) | 0.2 (0.1) | 65.6 (0.1) | 12.8 (0.3) |  | 52.5 (0.5) | 0.1 (0.1) | 67.6 (0.2) | 11.4 (1.4)
decl. | 49.1 (0.6) | 0.3 (0.2) | 66.2 (0.3) | 17.9 (0.8) |  | 52.7 (0.3) | 0.0 (0.0) | 67.7 (0.1) | 13.5 (2.8)

Table 4: Counting: test results for models fine-tuned on our counting training
data. We report mean (std) over three runs: overall accuracy $acc$, precision
on correct examples $p_{c}$, precision on foiled examples $p_{f}$, and
pairwise accuracy $acc_{r}$. Splits: standard (std.), hard, and interpolated
(interp.). Formats: Q+A and declarative (decl).

In this setup, we use the counting dataset containing images and sentences
with a correct or foiled numeral (in both Q+A and declarative format), and
frame the counting task as an image-sentence alignment problem. We use the
pretrained V&L models’ image-sentence alignment head either to predict whether
the sentence matches the image or not (i.e., in a per-example comparison
evaluated with $acc$, $p_{c}$, $p_{f}$), or to score correct and foiled pairs
(i.e., in a pairwise comparison evaluated with $acc_{r}$). See Section 2.1 for
details on how we compute these metrics. In Tables 3 and 4, we report our main
results without and with additional finetuning on the counting training data,
respectively.

##### No Finetuning

As Table 3 shows, both ViLBERT and LXMERT perform below or close to the random
baseline according to accuracy, and tend to slightly improve upon the random
baseline according to pairwise accuracy. We note that ViLBERT identifies
correct image-sentence pairs relatively well ($73$–$79$ $p_{c}$), while
failing on foils ($24$–$27$ $p_{f}$). This trend is also visible in ViLBERT
12-in-1; however all scores tend to improve when compared to ViLBERT
(especially precision on foiled examples). Roughly, we can rank models
according to their performance from worse to best: ViLBERT, LXMERT, ViLBERT
12-in-1. ViLBERT 12-in-1 is the only model that performs above chance level
according to standard accuracy when applied without direct finetuning
(declarative format, standard and hard splits). From these initial results, it
seems that whereas ViLBERT 12-in-1 is able to identify correct image-sentence
pairs well (i.e., up to $\sim 80$ $p_{c}$), its most important gains come from
improved precision on foiled examples (up to $58$ $p_{f}$).

Overall, results when performing pairwise scoring ($acc_{r}$) agree with the
general trends in per-example results. We can clearly observe that ViLBERT
performs close to chance, LXMERT is somewhat better ($56$–$71$), and ViLBERT
12-in-1 performs best ($70$–$77$). All models perform better when evaluated
using pairwise accuracy compared to per-example accuracy. For example, ViLBERT
12-in-1 performs below chance level in per-example metrics ($43$ $acc$) but
still show good pairwise accuracy ($77$ $acc_{r}$), i.e., $acc_{r}$ is a more
generous metric than standard accuracy $acc$.

##### With Finetuning

In Table 4 we note that when models are directly finetuned on counting
training data, results tend to improve (except for the interpolated data
split, which we discuss separately further below). ViLBERT improves overall
accuracy ($acc$) on the standard split considerably to about $71$–$74$, but it
still struggles on the hard and interpolated splits. Results on the hard split
are not good and have very high variance, which could be due to the model
overfitting on the small amount of counting training data. When finetuning
ViLBERT 12-in-1 further on counting data, results also improve compared to the
‘no finetuning‘ setting. As expected, ViLBERT 12-in-1 clearly outperforms
ViLBERT on both standard and hard splits according to all metrics evaluated:
on per-example metrics ($acc$, $p_{c}$, $p_{f}$) and also according to a
pairwise ranking comparison ($acc_{r}$).

Pairwise results show little variation across the Q+A and declarative data
formats, and exhibit more consistent differences between splits, i.e.,
interpolated $\leq$ hard $\leq$ standard. Both ViLBERT and ViLBERT 12-in-1
yield satisfying results of $76$–$83$ $acc_{r}$ in the standard and $56$–$77$
in the hard split.

##### Interpolated

Finally, in our interpolated split, we train on examples where correct answers
are even numbers and test on examples where correct answers are odd numbers.
By doing that, we gain a glimpse into whether models are really learning to
count, in which case interpolating even/odd numbers should be a relatively
simple task. We first note that models fail badly when finetuned and evaluated
on interpolated data, achieving per-example accuracies between $48$–$61$ while
still failing almost completely at identifying correct matches, as illustrated
by precision $p_{c}$ close to zero. Failure in the interpolated split is more
clearly seen by inspecting pairwise accuracies, which are in the range
$11$–$18$ and well below the random baseline of $50$. Although ViLBERT 12-in-1
achieves reasonable results on the standard and hard splits, it still fails
completely on the interpolated split. This is in stark contrast to recent
findings with text-only pretrained language models, which have a good grasp of
numeracy and perform well when interpolating quantities Wallace et al. (2019).

####  4.2.2 Counting as Masked Language Modelling

In this experiment, we set the image-sentence-alignment head aside and employ
the MLM capacity of LXMERT to further test its pretrained visual-linguistic
representations on counting.

We mask the numeral in the declarative statements of our counting dataset and
use LXMERT to predict the [MASK] token (see Figure 3). The model assigns
probabilities to all words in its English vocabulary, comprising more than 30k
words. We remove vocabulary items that are neither numerals nor denote
numerical quantities,888We include the indefinite article ‘a’ and the negation
‘no’ in our definition of a vocabulary item that denotes a numerical quantity,
since they are interpretable as indicating ‘one’ and ‘zero’ respectively. sort
the remaining items in order of descending probability, and obtain a list of
all numerical quantities LXMERT predicts for the masked token ordered by
likelihood. In that list, we count the rank of the correct numeral in any
formulation (_e.g._ “1”, “one”, “a”) and compute Recall@$k$ and mean rank
(MR). We report the results in Table 5, where we also show aggregate results
per answer.

Figure 3: Two examples of applying masked language modelling on the counting dataset with LXMERT. Num. | Recall@1 | Recall@2 | MR
---|---|---|---
| std. | hard | std. | hard | std. | hard
overall | 54.5 | 33.3 | 70.5 | 46.9 | 6.7 | 12.8
zero | 59.3 | 59.9 | 63.8 | 64.1 | 2.9 | 2.8
one | 86.1 | 86.3 | 89.5 | 89.7 | 1.6 | 1.5
two | 81.3 | 79.3 | 91.7 | 90.3 | 1.2 | 1.3
three | 14.4 | 14.1 | 83.8 | 84.2 | 2.0 | 2.0
four | 6.9 | 6.5 | 18.9 | 20.6 | 3.0 | 3.0
five | 0.3 | 0.3 | 1.0 | 1.0 | 5.1 | 5.1
six | 0.0 | 0.0 | 0.0 | 0.0 | 6.8 | 6.8
7 | 0.0 | 0.0 | 0.0 | 0.0 | 10.9 | 10.9
8 | 0.0 | 0.0 | 0.0 | 0.0 | 10.0 | 10.0
9 | 0.0 | 0.0 | 0.0 | 0.0 | 12.7 | 12.7
10 | 0.0 | 0.0 | 0.0 | 0.0 | 17.7 | 17.7
11 | 0.0 | 0.0 | 0.0 | 0.0 | 20.3 | 20.3
12 | 0.0 | 0.0 | 0.0 | 0.0 | 15.1 | 15.1
13 | 0.0 | 0.0 | 0.0 | 0.0 | 25.0 | 25.0
… | … | … | … | … | … | …
Table 5: Masked language modelling with LXMERT. We report Recall@$k$ and the
mean rank (MR) of the predicted numeral on our counting dataset’s test split.

The results of the _overall_ Recall@$k$ and MR show clear differences between
the standard and hard splits, whereas Recall@$k$ and MR _per numeral_ exhibit
very consistent results for the same splits. Recall that in the hard split
numerals are more evenly distributed, whereas in the standard split the
frequencies of different answers follow a more Zipfian distribution (Figure
2). This shows that the model has a strong preference for the numerals “2”,
followed by “1” and “0”. The result suggests a model whose performance is
largely determined by the statistical bias in the training data, rather than
the specifics of the visual input in relation to the text.

The model’s bias towards frequent numerals can also be observed when the MLM
predictions are wrong. While Table 5 reports metrics depicting how often the
model is _correct_ , we further analyse the cases where the model’s prediction
is _wrong_. Our analysis shows that among image-sentence pairs in the hard
split where the model prediction is wrong, the model predicts: “two” 51% of
the time, “no” 12% of the time (which we count as “0”), followed by “three” in
12% of cases, “four” (5%) and “five” (1%).

Good performance on low quantities is unsurprising since they appear more
frequently in V&L datasets (Goyal et al., 2017). The very poor performance on
under-represented quantities suggests both a lack of generalisation of the V&L
model, as well as limitations in its visual backbone, i.e., Faster-RCNN (Ren
et al., 2015).

Answer | VQA accuracy | Answer | VQA accuracy
---|---|---|---
| std. | hard |  | std. | hard
overall | $\bm{49.1}$ | $\bm{37.8}$ | seven | 4.0 | 2.6
zero | 94.4 | 93.1 | eight | 2.0 | 3.9
one | 70.3 | 65.0 | nine | 7.1 | 5.0
two | 56.8 | 58.3 | ten | 3.0 | 4.7
three | 26.6 | 24.4 | eleven | 0.0 | 0.0
four | 23.7 | 19.2 | twelve | 23.8 | 29.0
five | 13.3 | 13.5 | thirteen | 0.0 | 0.0
six | 12.4 | 11.5 | … | … | …

Table 6: Overall (in bold) and per-answer accuracy of LXMERT further fine-
tuned on the VQA task (Antol et al., 2015) on the standard and hard counting
splits.

####  4.2.3 Counting as VQA

Finally, we frame the counting probe in its standard setting as a visual
question answering problem, without employing any foils. We use the publicly
available LXMERT model that after pretraining has been further finetuned on
the VQA v2.0 dataset (Goyal et al., 2017). The test setting is the same as for
the original VQA task: the model receives questions (“How many …?”) from our
counting dataset as input and has to predict the most likely answer from a
list of 3,129 possible answers. All answers in our dataset are contained in
the model’s answer list. We report detailed results in Table 6.

The model achieves an overall 49.1 accuracy on the standard split and 37.8 on
the hard split. The detailed accuracies _per numeral_ show the same trend as
Table 5. Differences in overall performance between the standard (Zipfian) and
the hard (more balanced) splits are predominantly due to the different
proportion of quantities in the splits. Once again, this reveals the model’s
lack of generalisation coupled with a surplus of bias exploitation potential:
since quantities like “one” or “two” are highly frequent, the model relies on
these numerals as a “safe bet” when predicting under uncertainty.

A notable difference between using LXMERT’s MLM head without direct finetuning
on our counting training data (as in Table 5), and using LXMERT’s VQA head
further finetuned on VQA v2.0 (Table 6) is seen for the numeral “zero”: the
model’s capacity of predicting “zero” is enhanced by the finetuning process,
to the detriment of the frequent quantities “one” and “two”. Fine-tuning also
seems to improve prediction quality for numerals “four” to “six”, and also for
“12”.999“12” or “a dozen” is a frequent answer in VQA v2.0. Counting further
than that is a challenge for LXMERT.

##  5 Related Work

The pretrain-and-finetune paradigm101010This paradigm was originally proposed
for text-only language models; it is robust to domain shifts and leads to
state-of-the-art results across different NLU and NLG tasks Devlin et al.
(2019); Wang et al. (2019); Lewis et al. (2020). has become the de facto
standard when building models for vision and language tasks. It generally
consists of large scale pretraining using self-supervised learning followed by
usually smaller-scale finetuning in one or more specific tasks. A core idea is
that pretraining a model on large and diverse data should lead to robust
multimodal representations across vision and language, so that the model can
be easily finetuned to efficiently perform different tasks.

##### Pretrained Vision & Language Models

Recently, many pretrained V&L models have been proposed that combine images
and text using BERT-like architectures, such as ViLBERT Lu et al. (2019,
2020), LXMERT Tan and Bansal (2019), VisualBERT Li et al. (2019), UNITER Chen
et al. (2020), Unicoder-VL Li et al. (2020), VL-BERT Su et al. (2020), among
others. All these models use a BERT-like architecture to encode images and
text, are motivated as being general-purpose models for vision and language,
and follow the pretrain-and-finetune paradigm. They can be classified into
single- or dual-stream architectures: single-stream models concatenate words
to object bounding box features and encode this sequence using a single
transformer stack; dual-stream models have separate transformer stacks for
visual and textual inputs, and later include a set of layers that fuse these
into multimodal features (i.e., co-attention layers). ViLBERT, ViLBERT
12-in-1, and LXMERT, i.e. the models we use with in this work, are all dual-
stream models.

Bugliarello et al. (2020) propose a framework that unifies multiple single-
and dual-stream V&L model architectures, and find that these models perform
comparably under similar conditions. Ilharco et al. (2020) probe text-only
language models for how well they encode visual representations and find that
contextual models such as BERT do it reasonably well, while still lagging
behind humans. Finally, Cao et al. (2020) propose the VALUE benchmark, which
uses many V&L tasks to probe V&L models for different V&L capabilities. We
corroborate some of Cao et al. (2020)’s findings, such as the dominant role of
textual features compared to visual features in the model’s predictions.

##### Vision & Language Models for Counting

Counting is known to be a hard task for V&L models, and has been studied
extensively with different approaches in the literature Seguí et al. (2015);
Chattopadhyay et al. (2017); Trott et al. (2018); Zhang et al. (2018); Acharya
et al. (2019).

Chattopadhyay et al. (2017) investigates different strategies for counting:
(1) detecting objects of a particular class and adding the number of detected
objects; (2) directly predicting the count from a global feature vector (i.e.,
regression); (3) subitising, which is similar to (2) but where one first
splits the image into regions, solves the problem for the sub-images, and
aggregates the partial counts together; and (4) ensembling, where predictions
by single models are expected to be noisy, but on average should converge to
the correct count.

Trott et al. (2018) create the HowMany-QA counting dataset, and also propose
the Interpretable RL Counter (IRLC) model which is trained with reinforcement
learning and consists of a sequential decision process that solves the problem
by iteratively selecting objects to include into a pool until it decides to
stop adding objects. The count is given by directly reporting the size of the
pool. Acharya et al. (2019) propose TallyQA, a large counting dataset designed
to have not only simple questions (such as How many giraffes?) but also
questions that are harder to answer because they involve additional properties
(e.g. How many giraffes are sitting down?). It includes 188k simple and 60k
complex questions in its training set. HowMany-QA and TallyQA include examples
with answers of numbers up to 20 (inclusive) and numbers up to 15 (inclusive),
respectively.

Finally, Zhang et al. (2018) argues that using an attention bottleneck leads
to poor counting capabilities (see Section 3; Zhang et al., 2018). They devise
an elaborate architecture that includes an additional branch in the model
especially designed to compute counts that breaks the attention bottleneck.
The modified architecture maintains the same accuracy on the original task,
while also leading to considerable improvements on counting capability.

##### Counting and the attention bottleneck

Zhang et al. (2018) raises the issue of counting and the “attention
bottleneck”, which is further discussed in Acharya et al. (2019). The
“attention bottleneck” issue can be generally described as afflicting model
architectures where the image pipeline has the general form “image
$\rightarrow$ CNN $\rightarrow$ convolutional feature maps $\rightarrow$
attention bottleneck $\rightarrow$ prediction”. In other words, there is a
bottleneck (i.e., an attention mechanism) between input and prediction layer.

None of the model architectures we experiment with in this paper suffer from
the attention bottleneck. This issue is also not directly applicable to any of
the pretrained V&L models mentioned earlier in this Section. ViLBERT, ViLBERT
12-in-1, and LXMERT have two multi-layer transformer stacks to encode image
and text, respectively. In all these models, the outputs of modality-specific
encoders are processed in multiple co-attention layers that combine visual and
textual representations. When finetuning the model on a target task, a
prediction head is commonly trained from scratch and uses the output of the
last co-attention layer as input. The issue just discussed above therefore
does not directly apply to these model architectures, since none of them have
an attention bottleneck.

For a guiding example and more details on how the attention bottleneck impacts
a model’s capability to count, see Appendix C and refer to Zhang et al. (2018,
Section 3).

##  6 Conclusions and Future Work

Our results raise concerns about heavyweight V&L models. We probed three
pretrained V&L models on (1) image-sentence alignment and (2) counting: two
tasks that require joint understanding of image, text and their
correspondence. Our results show strong image-text alignment capabilities for
all three models we evaluated, as expected. We highlight that LXMERT (and to a
lesser extent, ViLBERT 12-in-1) may be suffering from catastrophic forgetting.
We observed sub-optimal counting capabilities in all models investigated, even
after finetuning on counting data. In these models, there is limited evidence
of grounding of symbols in visual data after pretraining; all models exploit
biases in the data and seem to lack the capability to individuate entities in
the visual input, a prerequisite for counting.

We encourage researchers i) to report the performance on pretraining tasks on
training and validation splits, ii) to work towards effective pretraining, and
iii) to test for catastrophic forgetting during finetuning. Our investigations
suggest that the high computational and environmental cost of current
pretraining practices outweigh the benefits of reusing such models, while
leaving the prospect of lightweight and green AI as a distant goal.

## Acknowledgments

IC has received funding from the European Union’s Horizon 2020 research and
innovation programme under the Marie Skłodowska-Curie grant agreement No
838188. AG is supported by the European Union’s Horizon 2020 research and
innovation Programme under the Marie Skłodowska-Curie grant agreement No
860621. This collaboration was facilitated by the Multi3Generation COST Action
CA18231.

## References

  * Acharya et al. (2019) Manoj Acharya, Kushal Kafle, and Christopher Kanan. 2019.  Tallyqa: Answering complex counting questions.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):8076–8084.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015.  Vqa: Visual question answering.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2425–2433.
  * Bender and Koller (2020) Emily M Bender and Alexander Koller. 2020.  Climbing towards NLU : On Meaning , Form , and Understanding in the Age of Data.  In _Proceedings ofthe 58th Annual Meeting ofthe Association for Computational Linguistics (ACL’20)_ , pages 5185–5198.
  * Bisk et al. (2020) Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020.  Experience grounds language.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8718–8735, Online. Association for Computational Linguistics.
  * Bugliarello et al. (2020) Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott. 2020\.  Multimodal pretraining unmasked: Unifying the vision and language berts.  _arXiv preprint arXiv:2011.15124_.
  * Cao et al. (2020) Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. 2020.  Behind the scene: Revealing the secrets of pre-trained vision-and-language models.  _arXiv preprint arXiv:2005.07310_.
  * Chattopadhyay et al. (2017) P. Chattopadhyay, R. Vedantam, R. R. Selvaraju, D. Batra, and D. Parikh. 2017.  Counting everyday objects in everyday scenes.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4428–4437.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Uniter: Universal image-text representation learning.  In _ECCV_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017\.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6904–6913.
  * Greco et al. (2019) Claudio Greco, Barbara Plank, Raquel Fernández, and Raffaella Bernardi. 2019\.  Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering.  In _Proceedings ofthe 57th Annual Meeting ofthe Association for Computational Linguistics (ACL’19)_ , pages 3601–3605, Florence, Italy. Association for Computational Linguistics.
  * Ilharco et al. (2020) Gabriel Ilharco, Rowan Zellers, Ali Farhadi, and Hannaneh Hajishirzi. 2020.  Probing text models for common ground with visual representations.  _arXiv preprint arXiv:2005.00619_.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences_ , 114(13):3521–3526.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2020) Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 11336–11344. AAAI Press.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019\.  Visualbert: A simple and performant baseline for vision and language.  In _Arxiv_.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _Computer Vision – ECCV 2014_ , pages 740–755, Cham. Springer International Publishing.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _Advances in Neural Information Processing Systems_ , pages 13–23.
  * Lu et al. (2020) Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020.  12-in-1: Multi-task vision and language representation learning.  In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Pezzelle and Fernández (2019) Sandro Pezzelle and Raquel Fernández. 2019.  Is the red square big? MALeViC: Modeling adjectives leveraging visual contexts.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2865–2876, Hong Kong, China. Association for Computational Linguistics.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In _Advances in neural information processing systems_ , pages 91–99.
  * Robins (1995) Anthony Robins. 1995.  Catastrophic Forgetting, Rehearsal and Pseudorehearsal.  _Connection Science_ , 7(2):123–146.
  * Seguí et al. (2015) Santi Seguí, Oriol Pujol, and Jordi Vitria. 2015.  Learning to count with deep object features.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pages 90–96.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of ACL_.
  * Shekhar et al. (2017) Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. 2017.  FOIL it! find one mismatch between image and language caption.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 255–265, Vancouver, Canada. Association for Computational Linguistics.
  * Sorodoc et al. (2018) Ionut Sorodoc, Sandro Pezzelle, Aurélie Herbelot, Mariella Dimiccoli, and Raffaella Bernardi. 2018.  Learning quantification from images: A structured neural architecture.  _Natural Language Engineering_ , 24(3):363–392.
  * Su et al. (2020) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020\.  Vl-bert: Pre-training of generic visual-linguistic representations.  In _International Conference on Learning Representations_.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  LXMERT: Learning cross-modality encoder representations from transformers.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5100–5111, Hong Kong, China. Association for Computational Linguistics.
  * Testoni et al. (2019) Alberto Testoni, Sandro Pezzelle, and Raffaella Bernardi. 2019.  Quantifiers in a Multimodal World: Hallucinating Vision with Language and Sound.  In _Proceedings ofthe Workshop on Cognitive Modeling and Computational Linguistics_ , pages 105–116, Minneapolis, MN. Association for Computational Linguistics.
  * Trott et al. (2018) Alexander Trott, Caiming Xiong, and Richard Socher. 2018.  Interpretable counting for visual question answering.  In _International Conference on Learning Representations_.
  * Wallace et al. (2019) Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019.  Do NLP Models Know Numbers? Probing Numeracy in Embeddings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5307–5315, Hong Kong, China. Association for Computational Linguistics.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Advances in Neural Information Processing Systems_ , volume 32, pages 3266–3280. Curran Associates, Inc.
  * Yogatama et al. (2019) Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. 2019.  Learning and evaluating general linguistic intelligence.  _arXiv preprint arXiv:1901.11373_.
  * Zhang et al. (2018) Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett. 2018.  Learning to count objects in natural images for visual question answering.  In _International Conference on Learning Representations_.
  * Zhu et al. (2016) Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016.  Visual7W: Grounded Question Answering in Images.  In _IEEE Conference on Computer Vision and Pattern Recognition_.

##  Appendix A Appendix

###  A.1 Question-to-statement template

We create a few simple templates to convert <question, answer> pairs into a
declarative statement. We denote the answer as A, and by definition it is
always a number. Other capitalised letters (e.g., B, C, etc.) denote entire
sets of words that are either copied over to the declarative sentence or
removed according to the template. If a set of words is optional in the
template, it is enclosed in brackets, e.g. [D]. A template is selected if
there is substring match between the template’s key and the question. We
denote negation by $\sim$. We process templates in order so that if a template
matches, it “consumes” the QA pair and produces a declarative sentence. If no
template matches, the QA pair is ignored and not added to our counting
dataset.

##### “are there”

How many B are there [C]? $\rightarrow$ There are A B [C]. E.g.: “How many
black cats are there in the picture?” $\rightarrow$ “There are A black cats in
the picture.”

##### “can you see”

How many B can you see [C]? $\rightarrow$ You see A B [C]. E.g.: “How many
elephants can you see?” $\rightarrow$ “You see A elephants.”

##### “do you see”

How many B do you see [C]? $\rightarrow$ There are A B [C]. E.g.: “How many
people do you see by the tree?” $\rightarrow$ “There are A people by the
tree.”

##### “are”

How many B are C? $\rightarrow$ There are A B C. E.g.: “How many glasses are
on the table?” $\rightarrow$ “There are A glasses on the table.”

##### “can”

How many B can C? $\rightarrow$ A B can C. E.g.: “How many surcoats can be
found in the storage?” $\rightarrow$ “A surcoats can be found in the storage.”

##### “do” and “have”

How many B do C have [D]? $\rightarrow$ C have A B [D]. E.g.: “How many
headphones do the people have?” $\rightarrow$ “The people have A headphones.”

##### “does” and “have”

How many B does C have D? $\rightarrow$ C has A B C. E.g.: “How many holes
does he have in his pants?” $\rightarrow$ “He has A holes in his pants.”

##### “have”

How many B have C? $\rightarrow$ A B have C. E.g.: “How many bottles have blue
caps?” $\rightarrow$ “A bottles have blue caps.”

#####  $\sim$ “is” and $\sim$ “will” and $\sim$ “does” and $\sim$ “has”

How many B? $\rightarrow$ There are A B. E.g.: “How many cars in the picture?”
$\rightarrow$ “There are A cars in the picture.”

####  A.1.1 Plurals

Finally, after applying the above mentioned templates we check if the original
answer to the question is the number 1. When that is the case, we convert all
sentences starting with “There are” by There are B. $\rightarrow$ There is B.
We also transform the following words: “people” $\rightarrow$ “person”, “men”
$\rightarrow$ “man”, “women” $\rightarrow$ “woman”, and also remove the final
“s” of words up to the fourth word in the declarative sentence (all words but
“has”).

##  Appendix B Counting Data

In Table 7 we show the percentage of occurrence of numerals (these numbers are
complementary to the visualisation in Figure 2). These percentages show that
quantities in the _standard_ split follow a more natural distribution, where
numerals like “one”, “two” or “three” are more common than large quantities or
mentions of empty sets.

Numeral | Percentage (%)
---|---
| Train | Valid | Test
| std. | hard | std. | hard | std. | hard
zero | 9 | 11 | 7 | 15 | 7 | 12
one | 30 | 11 | 26 | 14 | 27 | 12
two | 25 | 9 | 32 | 15 | 19 | 13
three | 14 | 11 | 13 | 13 | 16 | 12
four | 8 | 9 | 9 | 15 | 7 | 12
five | 5 | 12 | 4 | 10 | 5 | 13
six | 3 | 11 | 3 | 6 | 3 | 9
7 | 1 | 5 | 1 | 2 | 1 | 4
8-10 | 3 | 11 | 2 | 5 | 3 | 8
10-20 | 2 | 7 | 2 | 4 | 2 | 5
21+ | 0 | 0 | 0 | 1 | 0 | 1
Table 7: Percentage of numerals in the counting data.

##  Appendix C Counting and the attention bottleneck

The attention bottleneck takes place when there is an image encoder model and
there is a bottleneck between the model input and the layer that makes the
predictions of interest. This situation can be exemplified where the image
pipeline has the general form “image $\rightarrow$ CNN $\rightarrow$
convolutional feature maps $\rightarrow$ attention bottleneck $\rightarrow$
prediction”. We now use an idealised example meant to illustrate the attention
bottleneck issue, similar to the one used in Zhang et al. (2018). The goal is
just to clarify when the issue should arise and in what conditions.

Imagine there is a cat prediction model, and we present it with an image with
a single cat. After a number of CNN layers, the model computes a convolutional
feature map $\bm{c}_{\text{i,j}}$. In the attention bottleneck, a perfectly
trained model will assign probability close to 1 to the “cat” feature vector,
e.g., say feature map $\bm{c}_{\text{4,7}}$, and $0$ elsewhere, and the
attention output will roughly be $1\cdot\bm{c}_{\text{4,7}}+0\cdot\sum_{i\neq
4,j\neq 7}c_{i,j}$.

We can now think of an idealised scenario where we create an identical copy of
the cat image and paste it side-by-side with the original image (i.e., so that
there are two cats), or we can think of another image which depicts two
identically looking cats. When encoding any such image, each “cat” feature
vector should get $\sim 0.5$ probability in the attention layer, again
assuming an idealised and perfectly trained model. By design of the attention
mechanism, the attention output will consist of the two sets of “cat” features
multiplied by $\sim 0.5$ each and summed together. Therefore the attention
output for the image with two cats would virtually be the same as the output
for the image with a single cat. If the model only has access to these
features, i.e., the attention mechanism is a bottleneck, it becomes very hard
for the model to count, which would require being able to differentiate
between the number of cats in the input image by definition.


# On the Asymptotics of a Prime Spin Relation

Christine McMeekin

###### Abstract.

For certain cyclic totally real number fields, we give formulas for the
density of primes that satisfy a given spin relation.

###### Contents

  1. 1 Introduction
  2. 2 The Spin of Prime Ideals
  3. 3 Some Class Field Theory
  4. 4 An Equidistribution Lemma
  5. 5 Property Star and the Starlight Invariant
  6. 6 Density Theorems

##  1\. Introduction

Let $K=K(n,\ell)$ denote a totally real number field that is cyclic over
$\mathbb{Q}$ with odd prime degree $n$ such that the class number of $K$ is
odd, 2 is inert, and every totally positive unit is a square. Let $\ell$
denote the conductor of $K$ and let $\mathcal{O}_{K}$ denote the ring of
integers of $K$.

Let $\sigma\in\operatorname{Gal}(K/\mathbb{Q})$, $\sigma\neq 1$. Given an odd
principal ideal $\mathfrak{a}$, following [FIMR13], we define the spin of
$\mathfrak{a}$ (with respect to $\sigma$) to be

$\text{spin}(\mathfrak{a},\sigma)=\left(\frac{\alpha}{\mathfrak{a}^{\sigma}}\right)$

where $\mathfrak{a}=(\alpha)$, $\alpha$ is totally positive, and
$\left(\frac{\cdot}{\cdot}\right)$ denotes the quadratic residue symbol in
$K$.

The main results of this paper give a formula for the density of rational
primes that exhibit the spin relation

$\operatorname{spin}(\mathfrak{p},\sigma)=\operatorname{spin}(\mathfrak{p},\sigma^{-1})\quad\text{for
all }\sigma\neq 1\in\operatorname{Gal}(K/\mathbb{Q})$

where $\mathfrak{p}$ is a prime of $K$ above $p$. The formula is given in
terms of $n=[K:\mathbb{Q}]$ and $m_{K}$, a computable and bounded invariant of
the number field $K$. Define

$\mathbf{M}_{4}:=(\mathcal{O}_{K}/4\mathcal{O}_{K})^{\times}/\left((\mathcal{O}_{K}/4\mathcal{O}_{K})^{\times}\right)^{2}$

where $\mathcal{O}_{K}$ denotes the ring of integers of $K$. We define the
Starlight invariant of the number field $K$ (denoted $m_{K}$) to be the number
of non-trivial $\operatorname{Gal}(K/\mathbb{Q})$-orbits of $\mathbf{M}_{4}$
with representative $\alpha\in\mathcal{O}_{K}$ such that the Hilbert symbol
$(\alpha,\alpha^{\sigma})_{2}=1$ for all non-trivial
$\sigma\in\operatorname{Gal}(K/\mathbb{Q})$.

The main results of this paper are motivated by the following conjecture,
which gives a computable and bounded formula for the density of rational
primes with constant spin equal to $1$.

######  Conjecture 1.1.

Fix $K:=K(n,\ell)$. The density of rational primes $p$ such that
$\operatorname{spin}(\mathfrak{p},\sigma)=1$ for all non-trivial
$\sigma\in\operatorname{Gal}(K/\mathbb{Q})$ is given by

$C_{K}:=\frac{2^{n-1}(n-1)+m_{K}n+1}{\left(\sqrt{2}\right)^{3n-1}n}.$

Restricting to rational primes that split completely in $K/\mathbb{Q}$, the
corresponding conditional density is given by

$C_{K,S}:=\frac{m_{K}n+1}{\left(\sqrt{2}\right)^{3n-1}}.$

The reasoning behind Conjecture 1.1 is as follows. We break up the density
$C_{K}$ into a product of two densities as though one might break up a
probability into a product of conditional probabilities. Then $C_{K}$ is the
product of two densities; the first is $D_{K}$, the density of rational primes
satisfying the spin relation given in Theorem 1.2, and the second is the
conditional density of rational primes $p$ with
$\operatorname{spin}(\mathfrak{p},\sigma)=1$ for all non-trivial
$\sigma\in\operatorname{Gal}(K/\mathbb{Q})$, assuming that $p$ satisfies the
spin relation. Conjecture 1.1 then asserts that

$C_{K}=D_{K}\left(\frac{1}{2}\right)^{\left(\frac{n-1}{2}\right)}.$

Note that the condition that $p$ satisfies the spin relation is a Cebotarev
condition so by Theorems 1.1 and 1.2 in [FIMR13], if
$\operatorname{spin}(\mathfrak{p},\sigma)$ and
$\operatorname{spin}(\mathfrak{p},\tau)$ are independent for
$\sigma,\tau\in\operatorname{Gal}(K/\mathbb{Q})$ with
$\sigma\neq\tau,\tau^{-1}$, then we arrive at Conjecture 1.1.

A corollary of Conjecture 1.1 is a family of number fields
$\\{F_{K}(p)\\}_{p}$ depending on $p$ such that $p$ is always ramified in
$F_{K}(p)/\mathbb{Q}$ and the density of rational primes that split as
completely as possible in $F_{K}(p)/\mathbb{Q}$ (given the ramification) is

$\frac{C_{K,S}}{n}.$

######  Theorem 1.2.

Let $K:=K(n,\ell)$. The density of rational primes $p$ that satisfy the spin
relation

$\operatorname{spin}(\mathfrak{p},\sigma)=\operatorname{spin}(\mathfrak{p},\sigma^{-1})\quad\text{for
all }\sigma\neq 1\in\operatorname{Gal}(K/\mathbb{Q})$

where $\mathfrak{p}$ is a prime of $K$ above $p$ is given by

$D_{K}=\frac{2^{n-1}(n-1)+m_{K}n+1}{2^{n}n}.$

######  Theorem 1.3.

Let $K:=K(n,\ell)$. Then

$0<\frac{2^{n-1}(n-1)+1}{2^{n}n}\leq D_{K}\leq\frac{1}{2}.$

Table 1 gives examples of computed Starlight invariants for cyclic number
fields of degree $n$ over $\mathbb{Q}$ and conductor $\ell$ for the given $n$
and $\ell$ values and it gives the corresponding density of primes satisfying
the spin relation. These values of $m_{K}$ were computed using magma [BCP97];
the code can be found in Appendix $B$ of [McM18].

$n$ | 3 | 5 | 7 | 11 | 13 | 17 | 19
---|---|---|---|---|---|---|---
$\ell$ | 7 | 11 | 43 | 23 | 53 | 103 | 191
$m_{K}$ | 1 | 1 | 3 | 3 | 5 | 17 | 27
$D_{K}$ | $\frac{1}{2}$ | $\frac{7}{16}$ | $\frac{29}{64}$ | $\frac{467}{1024}$ | $\frac{1893}{4096}$ | $\frac{30849}{65536}$ | $\frac{124187}{262144}$
Table 1. Computed Starlight invariants and densities of the prime spin
relation using Theorem 1.2.

We remark that we can simplify the restrictions on $K$ in the cubic case. For
$n=3$, the assumption that the class number of $K$ is odd is sufficient to
imply that every totally positive unit is a square due to results of Armitage
and Fröhlich [AF67].

######  Theorem 1.4.

[AF67] Let $K$ be a cyclic cubic number field with odd class number. Then
every totally positive unit is a square.

###### Proof.

Let $U:=\mathcal{O}_{K}^{\times}$ denote the group of units, $U_{T}$ the
totally positive units, and $U^{2}$ the square units. Observe $U^{2}\subseteq
U_{T}\subseteq U$. Then we have a surjective homomorphism

$\phi:\frac{U}{U^{2}}\to\frac{U}{U_{T}}.$

If none of the nontrivial class representatives of $U/U^{2}$ are totally
positive then $\phi$ is injective. By Theorem $V$ in [AF67], all signatures
are represented by units. Square units are always totally positive and there
are 8 signatures and 8 classes of units mod squares, so each class of
$U/U^{2}$ must have a different signature. Therefore $U_{T}$$=$$U^{2}$. ∎

##  2\. The Spin of Prime Ideals

Let $K:=K(n,\ell)$ and let $h(K)$ denote the class number of $K$.

######  Definition 2.1.

[FIMR13] Let $\sigma\neq 1\in\operatorname{Gal}(K/\mathbb{Q})$. Given an odd
principal ideal $\mathfrak{a}$, we define the spin of $\mathfrak{a}$ (with
respect to $\sigma$) to be

$\text{spin}(\mathfrak{a},\sigma)=\left(\frac{\alpha}{\mathfrak{a}^{\sigma}}\right)$

where $\mathfrak{a}^{h(K)}=(\alpha)$, $\alpha$ is totally positive, and
$\left(\frac{\cdot}{\cdot}\right)$ denotes the quadratic residue symbol in
$K$.

Spin is well-defined; since every totally positive unit is a square, the
choice of totally positive generator $\alpha$ does not affect the quadratic
residue and Lemma 3.1 asserts the existence of a totally positive generator.

Lemma 11.1 in [FIMR13] states that the product
$\text{spin}(\mathfrak{p},\sigma)\text{spin}(\mathfrak{p},\sigma^{-1})$ is a
product of Hilbert symbols at places dividing 2. We restate this more
explicitly in Lemma 2.2.

For a place $v$ of $K$, let $K_{(v)}$ denote the completion of $K$ at $v$. For
$a,b\in K$ co-prime to $v$, the Hilbert Symbol is defined such that
$(a,b)_{v}:=1$ if the equation $ax^{2}+by^{2}=z^{2}$ has a solution $x,y,z\in
K_{(v)}$ where at least one of $x$, $y$, or $z$ is nonzero and $(a,b)_{v}:=-1$
otherwise.

######  Lemma 2.2.

[FIMR13] Let $K:=K(n,\ell)$. Let $\alpha$ be a totally positive generator of
the odd prime ideal $\mathfrak{p}\subseteq\mathcal{O}_{K}$. Then

$\text{spin}(\mathfrak{p},\sigma)\text{spin}(\mathfrak{p},\sigma^{-1})=\prod_{v|2}(\alpha,\alpha^{\sigma})_{v}.$

In particular, if $\alpha\equiv 1\bmod 4$ then
$\prod_{v|2}(\alpha,\alpha^{\sigma})_{v}=1$. Since 2 is inert in
$K/\mathbb{Q}$,

$\text{spin}(\mathfrak{p},\sigma)\text{spin}(\mathfrak{p},\sigma^{-1})=(\alpha,\alpha^{\sigma})_{2}.$

###### Proof.

See Lemma 11.1 in [FIMR13] or use the standard fact of Hilbert symbols that
$\prod_{v}(\alpha,\alpha^{\sigma})_{v}=1$. ∎

##  3\. Some Class Field Theory

We now diverge momentarily from the spin of prime ideals to discuss some class
field theory in the case when every totally positive unit is a square. We say
a modulus is narrow whenever it is divisible by all infinite places. We say a
modulus is wide whenever it is not divisible by any infinite place. We say a
ray class group or ray class field is narrow or wide whenever its defining
modulus is narrow or wide respectively. Let $U:=\mathcal{O}_{K}^{\times}$
denote the group of units of $K$, let $U_{T}$ denote the totally positive
units, and let $U^{2}$ denote the square units. The following lemma is an
exercise in class field theory.

######  Lemma 3.1.

Let $K$ be a totally real number field. The following are equivalent.

  1. (1)

$U_{T}=U^{2}$.

  2. (2)

The narrow and wide Hilbert class groups of $K$ coincide.

  3. (3)

Every principal ideal of $K$ has a totally positive generator.

###### Proof.

To show the equivalence of (1) and (2), apply Theorem V.1.7 in [Mil13] using
the modulus given by the product of all infinite places. Statements (3) and
(2) are equivalent by the definitions of narrow and wide class groups. ∎

######  Definition 3.2.

Let $K:=K(n,\ell)$. For $q$ a power of 2, we define the group

$\mathbf{M}_{q}:=\left(\mathcal{O}_{K}/q\mathcal{O}_{K}\right)^{\times}/\left(\left(\mathcal{O}_{K}/q\mathcal{O}_{K}\right)^{\times}\right)^{2}.$

The Galois group $\operatorname{Gal}(K/\mathbb{Q})$ acts on $\mathbf{M}_{q}$
in the natural way.

We will primarily be interested in $\mathbf{M}_{4}$. We will see in Lemma 3.5
that $\mathbf{M}_{q}$ is canonically isomorphic to a quotient of the narrow
ray class group over $K$ of conductor $q$ modulo squares.

######  Lemma 3.3.

[Mun] Let $K$ be a cyclic number field of odd degree $n$ over $\mathbb{Q}$
such that $2$ is inert in $K$. Then as vector spaces

$\mathbf{M}_{4}\cong(\mathbb{Z}/2)^{n}.$

Furthermore, the invariants of the action of
$\operatorname{Gal}(K/\mathbb{Q})$ are exactly $\pm 1\in\mathbf{M}_{4}$.

###### Proof.

This proof is due to Sam Mundy [Mun]. Consider the exact sequence

(1) $0\to
1+2(\mathcal{O}_{K}/4)\to(\mathcal{O}_{K}/4)^{\times}\to(\mathcal{O}_{K}/2)^{\times}\to
1.$

Note that $\mathcal{O}_{K}/2\cong\mathbb{F}_{2^{n}}$ because $K$ is cyclic of
odd degree and 2 is inert in $K$. Also,
$G\cong\operatorname{Gal}(\mathbb{F}_{2^{n}}/\mathbb{F}_{2})$.

Viewing $\mathbb{F}_{2^{n}}$ as an additive group with Galois action by
$G\cong\operatorname{Gal}(\mathbb{F}_{2^{n}}/\mathbb{F}_{2})$, there is an
isomorphism of Galois modules given by

$\psi:\mathbb{F}_{2^{n}}\cong\mathcal{O}_{K}/2\to 1+2(\mathcal{O}_{K}/4)$
$\psi:x\mapsto 1+2x.$

This map is easily seen to be a Galois equivariant homomorphism. Injectivity
and surjectivity follow from considering 2-adic expansions of elements in
$\mathcal{O}_{K}/4$. Since $\psi$ is an isomorphism we can rewrite the exact
sequence of Galois modules in equation 1 as

(2)
$0\to\mathbb{F}_{2^{n}}\to(\mathcal{O}_{K}/4)^{\times}\to\mathbb{F}_{2^{n}}^{\times}\to
1.$

Next consider the diagram of exact sequences below.

${0}$${\mathbb{F}_{2^{n}}}$${(\mathcal{O}_{K}/4)^{\times}}$${\mathbb{F}_{2^{n}}^{\times}}$${1}$${0}$${\mathbb{F}_{2^{n}}}$${(\mathcal{O}_{K}/4)^{\times}}$${\mathbb{F}_{2^{n}}^{\times}}$${1}$$2(\cdot)$$(\cdot)^{2}$$(\cdot)^{2}$

The first vertical map is multiplication by 2, which is the zero map. The next
two vertical maps are squaring. The third vertical map is an isomorphism
because $\mathbb{F}_{2^{n}}^{\times}$ is cyclic of odd order. Recall that

$\mathbf{M}_{4}:=\left(\mathcal{O}_{K}/4\mathcal{O}_{K}\right)^{\times}/\textrm{squares}.$

Then we apply the snake lemma to the diagram below.

${1}$${0}$${\mathbb{F}_{2^{n}}}$${(\mathcal{O}_{K}/4)^{\times}}$${\mathbb{F}_{2^{n}}^{\times}}$${1}$${0}$${\mathbb{F}_{2^{n}}}$${(\mathcal{O}_{K}/4)^{\times}}$${\mathbb{F}_{2^{n}}^{\times}}$${1}$${\mathbb{F}_{2^{n}}}$${\mathbf{M}_{4}}$${1}$$0$$(\cdot)^{2}$$(\cdot)^{2}$

The snake lemma gives us the exact sequence of $G$-modules

$0\to\mathbb{F}_{2^{n}}\to\mathbf{M}_{4}\to 1.$

Therefore $\mathbf{M}_{4}\cong\mathbb{F}_{2^{n}}$ as $G$-modules. The
invariants of $\mathbb{F}_{2^{n}}$ are $\mathbb{F}_{2}$. Tracing through the
isomorphism we see that this corresponds to the invariants $\\{\pm 1\\}$ in
$\mathbf{M}_{4}$. ∎

Let $\mathbb{M}_{q,G}$ denote the set of
$\operatorname{Gal}(K/\mathbb{Q})$-orbits of $\mathbf{M}_{q}$ for $q$ a power
of 2. Recall that we say a modulus of $K$ is narrow whenever
$\mathfrak{m}_{\infty}$ divides the modulus where $\mathfrak{m}_{\infty}$ is
the product of all infinite places of $K$. Letting $\mathfrak{m}$ denote a
narrow modulus with finite part $\mathfrak{m}_{0}$, let
$J_{K}^{\mathfrak{m}}=J_{K}^{\mathfrak{m}_{0}}$ denote the group of fractional
ideals of $K$ prime to $\mathfrak{m}_{0}$ and let
$P_{K}^{\mathfrak{m}}=P_{K}^{\mathfrak{m}_{0}}$ denote the subgroup of
$J_{K}^{\mathfrak{m}}$ formed by the principal ideals with generator
$\alpha\in K^{\times}$ such that
$\operatorname{ord}_{2}(q)\leq\operatorname{ord}_{2}(\alpha)$ and
${\alpha}\succ 0$. We let
$\mathscr{P}_{K}^{\mathfrak{m}}=\mathscr{P}_{K}^{\mathfrak{m}_{0}}$ denote the
set of prime ideals of $\mathcal{O}_{K}$ co-prime to $\mathfrak{m}_{0}$ so
that $J_{K}^{\mathfrak{m}}$ is generated by $\mathscr{P}_{K}^{\mathfrak{m}}$.

######  Definition 3.4.

Let $K:=K(n,\ell)$. Let $q\geq 4$ be a power of 2.

  1. (1)

Define the map

$\displaystyle\mathbf{r}_{0}:$
$\displaystyle\mathscr{P}_{K}^{2}\to\mathbf{M}_{q}$
$\displaystyle\mathfrak{p}\mapsto\alpha$

where $\alpha\in\mathcal{O}_{K}$ is a totally positive generator for the
principal ideal $\mathfrak{p}^{\text{$h(K)$}}$.

  2. (2)

Define the map

$\displaystyle\mathbf{r}:$ $\displaystyle\mathscr{P}_{\mathbb{Q}}^{2}\to\text{
$\mathbb{M}_{q,G}$ }$ $\displaystyle p\mapsto[\mathbf{r}_{0}(\mathfrak{p})]$

where $\mathfrak{p}$ is any prime in $K$ above $p$. Here $[\alpha]$ denotes
the $\operatorname{Gal}(K/\mathbb{Q})$-orbit of $\alpha\in\mathbf{M}_{4}$
considered in $\mathbb{M}_{q,G}$.

The map $\mathbf{r}_{0}$ is well-defined out of $\mathscr{P}_{K}^{2}$; recall
that by Lemma 3.1, $U_{T}=U^{2}$ is equivalent to the coincidence of the
narrow and wide Hilbert class groups so $U_{T}=U^{2}$ if and only if all
principal ideals have a totally positive generator. Since squares are trivial
in $\mathbf{M}_{q}$ by definition and $U_{T}=U^{2}$, the map $\mathbf{r}_{0}$
is well-defined.

The map $\mathbf{r}$ is well-defined out of $\mathscr{P}_{\mathbb{Q}}^{2}$
because $\mathbb{M}_{q,G}$ is the quotient of $\mathbf{M}_{q}$ by the
$\operatorname{Gal}(K/\mathbb{Q})$-action so different choices of primes
$\mathfrak{p}$ of $K$ above $p$ give the same result;
$\mathbf{r}_{0}(\mathfrak{p}^{\sigma})=\mathbf{r}_{0}(\mathfrak{p})^{\sigma}$
for $\sigma\in\operatorname{Gal}(K/\mathbb{Q})$ and $\mathfrak{p}$ an odd
prime of $K$.

Since $J_{K}^{q}$ is generated by $\mathscr{P}_{K}^{q}=\mathscr{P}_{K}^{2}$,
the map $\mathbf{r}_{0}$ induces a homomorphism

${\varphi}_{0}:J_{K}^{q}\to\mathbf{M}_{q}.$

######  Lemma 3.5.

Let $K:=K(n,\ell)$. The homomorphism $\varphi_{0}:J_{K}^{q}\to\mathbf{M}_{q}$
induces a canonical surjective homomorphism

$\varphi:{\operatorname{n\mathcal{C}l}}^{q}_{K}\to\mathbf{M}_{q}.$

###### Proof.

We first show the induced homomorphism is well-defined. By Proposition V.1.6
in [Mil13], every element of ${\operatorname{n\mathcal{C}l}}^{q}$ is
represented by an integral ideal. Let $\mathfrak{a}$ and $\mathfrak{b}$ be two
integral ideals representing the same element of
${\operatorname{n\mathcal{C}l}}^{q}$. Then by Proposition V.1.6 in [Mil13],
there exist nonzero $a,b\in\mathcal{O}_{K}$ such that

$\displaystyle b\mathfrak{a}=a\mathfrak{b},$ $\displaystyle a\equiv b\equiv
1\bmod q,\quad\text{and}$ $\displaystyle{ab}\succ 0.$

Since $\varphi_{0}:J_{K}^{q}\to\mathbf{M}_{q}$ is a homomorphism,

$\varphi_{0}(b\mathcal{O}_{K})\varphi_{0}(\mathfrak{a})=\varphi_{0}(a\mathcal{O}_{K})\varphi_{0}(\mathfrak{b}).$

Noting that $h(K)$ is odd and squares are trivial in $\mathbf{M}_{q}$ by
definition, $\varphi_{0}$ maps any principal integral ideal $(\alpha)$ to the
class in $\mathbf{M}_{q}$ containing the representative
$\alpha\in\mathcal{O}_{K}$ where $\alpha$ is a totally positive generator.

Since $U_{T}=U^{2}$, every principal ideal of $\mathcal{O}_{K}$ has a totally
positive generator so there exists a unit $u\in\mathcal{O}_{K}^{\times}$ such
that ${ua}\succ 0$ and $\varphi_{0}(a)=ua$. Since ${ab}\succ 0$, then
${u^{-1}b}\succ 0$ so $\varphi_{0}(b)=u^{-1}b$. We know that $a\equiv b\equiv
1\bmod q$. Since squares are trivial in $\mathbf{M}_{q}$ by the definition of
$\mathbf{M}_{q}$, this implies

$\displaystyle u^{2}a\equiv b\quad\text{ in }\mathbf{M}_{q}$
$\displaystyle\implies$ $\displaystyle ua\equiv u^{-1}b\quad\text{ in
}\mathbf{M}_{q}$ $\displaystyle\implies$
$\displaystyle\varphi_{0}(a\mathcal{O}_{K})=\varphi_{0}(b\mathcal{O}_{K})$
$\displaystyle\implies$
$\displaystyle\varphi_{0}(\mathfrak{a})=\varphi_{0}(\mathfrak{b}).$

Therefore the homomorphism $\varphi_{0}$ induces a well-defined homomorphism
from ${\operatorname{n\mathcal{C}l}}^{q}_{K}$.

We now show the homomorphism is a canonical surjective homomorphism. Let
$\mathfrak{m}$ be the narrow modulus with finite part $q$. Let

$\displaystyle K_{\mathfrak{m}}:=\\{a\in
K^{\times}:\operatorname{ord}_{2}(a)=0\\},$ $\displaystyle
K_{\mathfrak{m},1}:=\\{a\in
K^{\times}:\operatorname{ord}_{2}(a-1)\geq\operatorname{ord}_{2}(q),{a}\succ
0\\},$ $\displaystyle U_{\mathfrak{m},1}:=K_{\mathfrak{m},1}\cap U.$

Let $X\in\mathbf{M}_{q}$. Consider the exact sequence from Theorem V.1.7 in
[Mil13];

$1\to U/U_{\mathfrak{m},1}\to
K_{\mathfrak{m}}/K_{\mathfrak{m},1}\to{\operatorname{n\mathcal{C}l}}^{\mathfrak{m}}_{K}\to
C\to 1$

and the canonical isomorphism

(3)
$K_{\mathfrak{m}}/K_{\mathfrak{m},1}\cong\left(\pm\right)^{n}\times\left(\mathcal{O}_{K}/q\right)^{\times}.$

Consider only the 2-part of each group. Then since $h(K)$ is odd, we have the
short exact sequence

$1\to(U/U_{\mathfrak{m},1})[2^{\infty}]\to(K_{\mathfrak{m}}/K_{\mathfrak{m},1})[2^{\infty}]\to({\operatorname{n\mathcal{C}l}}^{\mathfrak{m}}_{K})[2^{\infty}]\to
1.$

Note that since squaring sends all signatures to the trivial signature, the
canonical isomorphism in equation 3 induces a canonical isomorphism on the
2-part modulo squares;

$(K_{\mathfrak{m}}/K_{\mathfrak{m},1})[2^{\infty}]/(K_{\mathfrak{m}}/K_{\mathfrak{m},1})[2^{\infty}]^{2}\cong(\pm)^{n}\times\mathbf{M}_{q}.$

Consider the squaring map and apply the snake lemma to get the following
commutative diagram of exact sequences;

${1}$${(U/U_{\mathfrak{m},1})[2^{\infty}]}$${(K_{\mathfrak{m}}/K_{\mathfrak{m},1})[2^{\infty}]}$${({\operatorname{n\mathcal{C}l}}^{q}_{K})[2^{\infty}]}$${1}$${1}$${(U/U_{\mathfrak{m},1})[2^{\infty}]}$${(K_{\mathfrak{m}}/K_{\mathfrak{m},1})[2^{\infty}]}$${({\operatorname{n\mathcal{C}l}}^{q}_{K})[2^{\infty}]}$${1}$${U/U^{2}}$${(\pm)^{n}\times\mathbf{M}_{q}}$${{\operatorname{n\mathcal{C}l}}^{q}_{K}/({\operatorname{n\mathcal{C}l}}^{q}_{K})^{2}}$${1}$${1}$${1}$${1}$$\psi$$(\cdot)^{2}$$(\cdot)^{2}$$(\cdot)^{2}$

Then $\psi$ induces an isomorphism

$\psi:\faktor{((\pm)^{n}\times\mathbf{M}_{q})}{\operatorname{image}(U/U^{2})}\longrightarrow\faktor{{\operatorname{n\mathcal{C}l}}^{q}_{K}}{({\operatorname{n\mathcal{C}l}}^{q}_{K})^{2}}.$

Tracing through the definitions of the maps, $\varphi\circ\psi$ is surjective
(it is essentially the identity). Therefore $\varphi$ is surjective.

∎

##  4\. An Equidistribution Lemma

######  Definition 4.1.

Let $S$ be a set of primes and let $R\subseteq S$. If the limit exists, then
the restricted density of $R$ (restricted to $S$) is defined as

$d(R|S):=\lim_{N\to\infty}\frac{\\#R_{N}}{\\#S_{N}}$

where $S_{N}$ and $R_{N}$ denote the set of primes in $S$ and $R$ respectively
of absolute norm less than $N\in\mathbb{Z}_{+}$.

Recall that $\mathscr{P}_{\mathbb{Q}}^{m}$ denotes the set of rational primes
not dividing $m$ and $\mathscr{P}_{K}^{\mathfrak{m}}$ denotes the set of
primes of $K$ not dividing $\mathfrak{m}$. Letting $\mathfrak{p}$ be a prime
of $K$ above a rational prime $p$, denote the corresponding inertia degree
$f_{K/\mathbb{Q}}(p)=f_{K/\mathbb{Q}}(\mathfrak{p})$ (well-defined because $K$
is Galois over $\mathbb{Q}$). That is,

$f_{K/\mathbb{Q}}(p)=f_{K/\mathbb{Q}}(\mathfrak{p})=\frac{\\#D}{\\#E}$

where $D$ is the decomposition group of $\mathfrak{p}$ for the extension
$K/\mathbb{Q}$ and $E$ is the inertia group.

######  Definition 4.2.

Let $K=K(n,\ell)$. Define the following sets of rational primes.

$S:=\\{p\in\mathscr{P}_{\mathbb{Q}}^{2\ell}:f_{K/\mathbb{Q}}(p)=1\\}\quad\text{and}\quad
I:=\\{p\in\mathscr{P}_{\mathbb{Q}}^{2\ell}:f_{K/\mathbb{Q}}(p)=n\\}.$

Define the following sets of primes of $K$.

$S^{\prime}:=\\{\mathfrak{p}\in\mathscr{P}_{K}^{2\ell}:f_{K/\mathbb{Q}}(\mathfrak{p})=1\\}\quad\text{and}\quad
I^{\prime}:=\\{\mathfrak{p}\in\mathscr{P}_{K}^{2\ell}:f_{K/\mathbb{Q}}(\mathfrak{p})=n\\}.$

That is, $S\subseteq\mathscr{P}_{\mathbb{Q}}^{2\ell}$ is the set of odd
rational primes that split completely in $K/\mathbb{Q}$ and
$I\subseteq\mathscr{P}_{\mathbb{Q}}^{2\ell}$ is the set of odd rational primes
that are inert in $K/\mathbb{Q}$. Furthermore, $S^{\prime}$ is the set of
primes of $K$ laying above the primes in $S$ and $I^{\prime}$ is the set of
primes of $K$ laying above the primes in $I$.

Since $K/\mathbb{Q}$ is cyclic of prime degree $n$, then
$f_{K/\mathbb{Q}}(p)=1$ or $n$ for all $p\in\mathscr{P}_{\mathbb{Q}}^{2\ell}$
so in this case, $\mathscr{P}_{\mathbb{Q}}^{2\ell}$ is the disjoint union of
$S$ and $I$. The next Lemma asserts that for $K:=K(n,\ell)$, the primes are
equidistributed in $\mathbf{M}_{4}$ via the map $\mathbf{r}_{0}$.

Although the equidistribution generalizes to $\mathbf{M}_{q}$, note that the
number of elements of $\mathbf{M}_{8}$ for example is different than the
number of elements of $\mathbf{M}_{4}$ so the generalized statement would need
to be adjusted accordingly.

######  Lemma 4.3.

Let $K:=K(n,\ell)$.

  1. (1)

For any $\alpha\in\mathbf{M}_{4}$, the density of
$\mathfrak{p}\in\mathscr{P}_{K}^{2\ell}$ such that
$\varphi(\mathfrak{p})=\alpha$ is $\frac{1}{2^{n}}$. That is,

$d(\mathbf{r}_{0}^{-1}(\alpha)|\mathscr{P}_{K}^{2\ell})=\frac{1}{\\#\mathbf{M}_{4}}=\frac{1}{2^{n}}.$

  2. (2)

Furthermore, the density does not change when we restrict to primes of $K$
that split completely in $K/\mathbb{Q}$. That is,

$d(\mathbf{r}_{0}^{-1}(\alpha)\cap
S^{\prime}|S^{\prime})=\frac{1}{\\#\mathbf{M}_{4}}=\frac{1}{2^{n}}.$

###### Proof.

Recall that $\operatorname{nR}^{4}=\operatorname{nR}^{4}_{K}$ denotes the
narrow ray class field over $K$ of conductor $4\mathfrak{m}_{\infty}$. Let
$G:=\operatorname{Gal}(\operatorname{nR}^{4}/K)$. Define $H\leq G$ to be

$H:=\operatorname{Art}(\ker(\varphi))$

where $\operatorname{Art}$ denotes the Artin isomorphism. In other words, we
define $H$ by the following commutative diagram of exact sequences

${1}$${\ker(\varphi)}$${{\operatorname{n\mathcal{C}l}}^{4}}$${\mathbf{M}_{4}}$${1}$${1}$${H}$${G}$${\mathbf{M}_{4}}$${1}$$\varphi$$\operatorname{Art}$$\operatorname{Art}$id.

where surjectivity of $\varphi$ is proven in Lemma 3.5. Let $L$ be the fixed
field of $H$ so that $\operatorname{Gal}(L/K)\cong G/H$.

${\operatorname{nR}^{4}}$${L}$${K}$$G$$H$

This induces a canonical isomorphism

$\mathbf{M}_{4}\cong G/H\cong\operatorname{Gal}(L/K).$

For $\alpha\in\mathbf{M}_{4}$, define $P(\alpha)$ to be the set of odd
unramified prime ideals of $K$ which map to $\alpha$ via $\varphi$. Let
$\sigma\in G/H$ corresponding to $\alpha$. Then

$P(\alpha)=\\{\mathfrak{p}\in\mathscr{P}_{K}^{2\ell}:\operatorname{Art}_{L|K}(\mathfrak{p})=\sigma\\}$

where $\mathscr{P}_{K}^{2\ell}$ is the set of odd unramified prime ideals of
$K$ and $\operatorname{Art}_{L|K}$ denotes the Artin map for the extension
$L|K$.

Theorem 4 in [Ser81] asserts Cebotarev’s Density Theorem for natural density,
(or see [Neu99] Theorem VII.13.4 for a simpler proof using Dirichlet density).
By the special case of Cebotarev’s Density Theorem in which $L/K$ is cyclic,
$P(\alpha)$ has a density and it is given by

$\frac{1}{\\#\operatorname{Gal}(L/K)}=\frac{1}{\\#\mathbf{M}_{4}}.$

The first asserted equality of part (1) is proved. The second equality of part
(1) is true by Lemma 3.3.

To prove part (2), observe that

$d(\mathbf{r}_{0}^{-1}(\alpha)|\mathscr{P}_{K}^{2\ell})=d(\mathbf{r}_{0}^{-1}(\alpha)\cap
S^{\prime}|S^{\prime})d(S^{\prime}|\mathscr{P}_{K}^{2\ell})+d(\mathbf{r}_{0}^{-1}(\alpha)\cap
I^{\prime}|I^{\prime})d(I^{\prime}|\mathscr{P}_{K}^{2\ell}).$

Since $d(S^{\prime}|\mathscr{P}_{K}^{2\ell})=1$,
$d(I^{\prime}|\mathscr{P}_{K}^{2\ell})=0$, and $0\leq
d(\mathbf{r}_{0}^{-1}(\alpha)\cap I^{\prime}|I^{\prime})\leq 1$,

$d(\mathbf{r}_{0}^{-1}(\alpha)|\mathscr{P}_{K}^{2\ell})=d(\mathbf{r}_{0}^{-1}(\alpha)\cap
S^{\prime}|S^{\prime}).$

∎

##  5\. Property Star and the Starlight Invariant

Let $K:=K(n,\ell)$. Recall that $\mathbb{M}_{4,G}$ denotes the set of
$\operatorname{Gal}(K/\mathbb{Q})$-orbits of $\mathbf{M}_{4}$ and recall the
statement of Lemma 2.2, which motivates the following.

######  Theorem 5.1.

Let $K:=K(n,\ell)$. Let $\alpha\in\mathcal{O}_{K}$ denote a representative of
$[\alpha]\in$ $\mathbb{M}_{4,G}$. Define the map

$\displaystyle\text{$\star$}:$ $\displaystyle\text{ $\mathbb{M}_{4,G}$
}\to\\{\pm 1\\}$ $\displaystyle[\alpha]\mapsto\left\\{\begin{array}[]{l
l}1&\quad\text{if }(\alpha,\alpha^{\sigma})_{2}=1\text{ for all non-trivial
}\sigma\in\operatorname{Gal}(K/\mathbb{Q})\\\ -1&\quad\text{otherwise.}\\\
\end{array}\right.$

Then $\star$ is a well-defined map.

Note that by Lemma 2.2, a rational prime $p$ satisfies the spin relation

$\operatorname{spin}(\mathfrak{p},\sigma)=\operatorname{spin}(\mathfrak{p},\sigma^{-1})\quad\text{for
all }\sigma\neq 1\in\operatorname{Gal}(K/\mathbb{Q}),$

where $\mathfrak{p}$ is a prime of $K$ above $p$ exactly when
$\star\circ\mathbf{r}(p)=1$ where $\mathbf{r}$ is as defined in Definition
3.4.

###### Proof.

We will show that $\star$ is well-defined out of $\mathbf{M}_{4}$. Then
because $\star$ is a property of the full Galois orbit, $\star$ is well-
defined out of $\mathbb{M}_{4,G}$.

Let $\alpha,\beta\in\mathcal{O}_{K}$ be two representatives of the same class
in $\mathbf{M}_{4}$ so

$\alpha\equiv\beta\gamma^{2}\bmod 4\mathcal{O}_{K}\quad\text{for some
$\gamma\in\mathcal{O}_{K}$.}$

If $\alpha\equiv\beta\gamma^{2}\bmod 8\mathcal{O}_{K}$ then we can apply Lemma
2.3 from [FIMR13] to see that
$(\alpha,\alpha^{\sigma})_{2}=(\beta,\beta^{\sigma})_{2}$ for all
$\sigma\in\operatorname{Gal}(K/\mathbb{Q})$. Therefore, we may assume

$\alpha\equiv 5\beta\gamma^{2}\bmod 8\mathcal{O}_{K}.$

Suppose $(\alpha,\alpha^{\sigma})_{2}=1$. Then by Lemma 2.3 in [FIMR13], since
$\alpha\equiv 5\beta\gamma^{2}\bmod 8\mathcal{O}_{K}$,

$\displaystyle\left(5\beta\gamma^{2},\left(5\beta\gamma^{2}\right)^{\sigma}\right)_{2}=1$
$\displaystyle\implies$
$\displaystyle\left(5\beta,\left(5\beta\right)^{\sigma}\right)_{2}=1\quad\text{by
a property of Hilbert symbols.}$

Using bimultiplicativity of the Hilbert symbol,

$\left(5\beta,\left(5\beta\right)^{\sigma}\right)_{2}=(5,5)_{2}(\beta,5)_{2}(5,\beta^{\sigma})_{2}(\beta,\beta^{\sigma})_{2}.$

Notice that since 2 is inert in $K/\mathbb{Q}$ and since 5 is invariant under
the action of $\operatorname{Gal}(K/\mathbb{Q})$, applying the Galois action
to the quadratic form for $(\beta,5)_{2}$ yields the form for
$(5,\beta^{\sigma})_{2}$ so the cross terms cancel one another. Therefore

$\left(5\beta,\left(5\beta\right)^{\sigma}\right)_{2}=(5,5)_{2}(\beta,\beta^{\sigma})_{2}.$

Since $5\times 2^{2}+5\times 1^{2}=5^{2}$, $(5,5)_{2}=1$. Therefore

$\left(5\beta,\left(5\beta\right)^{\sigma}\right)_{2}=(\beta,\beta^{\sigma})_{2}$

so

$(\alpha,\alpha^{\sigma})_{2}=1\implies(\beta,\beta^{\sigma})_{2}=1.$

Therefore $\star$ is a well-defined map from $\mathbf{M}_{4}$.

We now prove that if $\alpha,\beta\in\mathbf{M}_{4}$ are the in same Galois
orbit, then $\star(\alpha)=\star(\beta)$. Let
$\tau\in\operatorname{Gal}(K/\mathbb{Q})$ such that $\alpha^{\tau}=\beta$ for
$\alpha,\beta\in\mathbf{M}_{4}$.

Suppose $(\alpha,\alpha^{\sigma})_{2}=1$ for all $\sigma\neq 1$ in
$\operatorname{Gal}(K/\mathbb{Q})$. Then in $K_{(2)}$, the completion of $K$
at $2\mathcal{O}_{K}$, there is a nontrivial solution $x,y,z$ to

$\alpha x^{2}+\alpha^{\sigma}y^{2}=z^{2}.$

Applying the action of $\tau$ yields a nontrivial solution to

$\beta x^{2}+\beta^{\sigma}y^{2}=z^{2}$

so $(\beta,\beta^{\sigma})_{2}=1$ for all $\sigma\neq 1$. ∎

Recall that by Lemma 3.3, the elements of $\mathbf{M}_{4}$ that are invariant
under the $\operatorname{Gal}(K/\mathbb{Q})$-action are exactly $\pm 1$. The
following lemma fully describes $\star$ on these invariants.

######  Lemma 5.2.

Let $K:=K(n,\ell)$.

  1. (1)

$\star(1)=1$.

  2. (2)

$\star(-1)=-1$.

###### Proof.

Observe that $(1,1)_{2}=1$ because $x^{2}+y^{2}=z^{2}$ has the solution
$(x,y,z)=(1,0,1)$.

If $(-1,-1)_{2}=1$, there would be a non-trivial solution to
$x^{2}+y^{2}+z^{2}\equiv 0\bmod 4$. Since there is no such solution,
$(-1,-1)_{2}=-1$. ∎

######  Definition 5.3.

Let $K:=K(n,\ell)$. Define the starlight invariant, $m_{K}$ to be the number
of $\operatorname{Gal}(K/\mathbb{Q})$-orbits $X$ of $\mathbf{M}_{4}$ of non-
trivial size such that $\star(X)=1$. That is, for $\sigma$ a generator of
$\operatorname{Gal}(K/\mathbb{Q})$,

$m_{K}:=\\#\\{X\in\mathbb{M}_{4,G}:\\#X=n\text{ and }\star(X)=1\\}.$

######  Remark 5.4.

By Lemma 5.2, it is equivalent to define the starlight invariant of $K$, as

$m_{K}=\\#\ker(\star)-1.$

Here $\star$ refers to the map $\star:\mathbb{M}_{4,G}\to\pm 1$ given in
Theorem 5.1.

We now define

$\star:\mathscr{P}_{K}^{2}\to\\{\pm
1\\}\quad\text{and}\quad\star:\mathscr{P}_{\mathbb{Q}}^{2}\to\\{\pm 1\\}$

to be the composition of $\star$ as defined in Theorem 5.1 with
$\mathbf{r}_{0}$ and $\mathbf{r}$ respectively as defined in Definition 3.4.

######  Definition 5.5.

Let $p\in\mathscr{P}_{\mathbb{Q}}^{2}$ and let
$\mathfrak{p}\in\mathscr{P}_{K}^{2}$. Define
$\star(\mathfrak{p}):=\star\circ\mathbf{r}_{0}$ and
$\star(p):=\star\circ\mathbf{r}$, the composition of the maps $\mathbf{r}_{0}$
and $\mathbf{r}$ respectively with the map $\star$ from Definition 3.4.

We say that a prime $\mathfrak{p}\in\mathscr{P}_{K}^{2}$ (respectively
$p\in\mathscr{P}_{\mathbb{Q}}^{2}$) has property $\star$ or that $\star$ is
true for $\mathfrak{p}$ (respectively $p$) whenever $\star(\mathfrak{p})=1$
(respectively $\star(p)=1$).

Theorem 6.2 and Theorem 1.2 give formulas in terms of $n$ and $m_{K}$ for the
density of rational primes (assumed to split completely in Theorem 6.2) that
satisfy property $\star$.

##  6\. Density Theorems

We first state and prove Theorem 6.2 which gives a formula describing the
restricted density of rational primes that satisfy the spin relation, the
restriction being to primes that split completely in $K/\mathbb{Q}$. Handling
the inert case separately, we then apply Theorem 6.2 to obtain Theorem 1.2
which gives a formula for the overall density of rational primes that satisfy
the given spin relation. Lastly, we prove Theorems 6.5 and 1.3 which give
bounds on the densities given in Theorems 6.2 and 1.2 respectively.

Recall the definitions of $S$, $S^{\prime}$, $I$, and $I^{\prime}$ from
Definition 4.2.

######  Definition 6.1.

Let $K:=K(n,\ell)$. Define the following sets of rational primes.

$\displaystyle B$
$\displaystyle:=\\{p\in\mathscr{P}_{\mathbb{Q}}^{2\ell}:\star(p)=1\\}$
$\displaystyle R$ $\displaystyle:=B\cap S.$

Note that by Lemma 2.2, $B$ is exactly the set of rational primes in
$p\in\mathscr{P}_{\mathbb{Q}}^{2\ell}$ such that

$\operatorname{spin}(\mathfrak{p},\sigma)=\operatorname{spin}(\mathfrak{p},\sigma^{-1})\quad\text{for
all }\sigma\in\operatorname{Gal}(K/\mathbb{Q})$

where $\mathfrak{p}$ is a prime of $K$ above $p$.

Recall from Definition 4.1 that $d(R|S)$ denotes the restricted density of
primes $p\in R$ restricted to $S$.

######  Theorem 6.2.

Let $K:=K(n,\ell)$. Then

$d(R|S)=\frac{1+\text{$m_{K}$}n}{2^{n}}.$

###### Proof.

Let $N\in\mathbb{Z}_{+}$. Let $R_{N}$ and $S_{N}$ denote the sets of primes in
$R$ and $S$ respectively of norm less than $N$. We will show that

(4)
$\lim_{N\to\infty}\frac{\\#R_{N}}{\\#S_{N}}=\frac{\\#\\{X\in\mathbf{M}_{4}:\star(X)=1\\}}{\\#\mathbf{M}_{4}}=\frac{1+m_{K}n}{2^{n}}.$

Let $R^{\prime}_{N}\subseteq\mathscr{P}_{K}^{2\ell}$ denote the set of primes
of $K$ that lay above rational primes in
$R_{N}\subseteq\mathscr{P}_{\mathbb{Q}}^{2\ell}$ and define $S_{N}^{\prime}$
similarly with respect to $S_{N}\subseteq\mathscr{P}_{\mathbb{Q}}^{2\ell}$.
Let $\mathbf{r}_{0,N}$ denote the restriction of $\mathbf{r}_{0}$ to
$S^{\prime}_{N}\subseteq\mathscr{P}_{K}^{2\ell}$. Since we have restricted to
primes that split completely in $K/\mathbb{Q}$,

$\frac{\\#R_{N}}{\\#S_{N}}=\frac{\\#R^{\prime}_{N}}{\\#S^{\prime}_{N}}$

and

$R^{\prime}_{N}=\bigcup_{\star(\alpha)=1}\mathbf{r}_{0,N}^{-1}(\alpha)$

where the above is a disjoint union over elements $\alpha\in\mathbf{M}_{4}$
such that $\star(\alpha)=1$. Therefore

$\frac{\\#R^{\prime}_{N}}{\\#S^{\prime}_{N}}=\frac{1}{\\#S^{\prime}_{N}}\sum_{\star(\alpha)=1}\\#\mathbf{r}_{0,N}^{-1}(\alpha).$

By Lemma 4.3, this implies

$\displaystyle\lim_{N\to\infty}\frac{\\#R^{\prime}_{N}}{\\#S^{\prime}_{N}}$
$\displaystyle=\sum_{\star(\alpha)=1}\lim_{N\to\infty}\frac{\\#\mathbf{r}_{0,N}^{-1}(X)}{\\#S^{\prime}_{N}}$
$\displaystyle=\sum_{\star(\alpha)=1}\frac{1}{2^{n}}$
$\displaystyle=\frac{\\#\\{\alpha\in\mathbf{M}_{4}:\star(\alpha)=1\\}}{2^{n}}.$

This proves the first equality in equation 4. Let $\sigma$ be a generator of
$\operatorname{Gal}(K/\mathbb{Q})$. By Lemma 3.3, the elements of
$\alpha\in\mathbf{M}_{4}$ such that $\alpha^{\sigma}=\alpha$ are $\alpha=\pm
1$ and we know that $\star(1)=1$ and $\star(-1)=-1$ by Lemma 5.2. Recalling
that
$m_{K}=\\#\\{[\alpha]\in\mathbb{M}_{4,G}:\alpha^{\sigma}\neq\alpha,\star(\alpha)=1\\}$,
this implies

$\\#\\{\alpha\in\mathbf{M}_{4}:\star(\alpha)=1\\}=m_{K}n+1.$

since $n=[K:\mathbb{Q}]$ is prime so Galois orbits $X\in\mathbb{M}_{4,G}$ such
that $X^{\sigma}\neq X$ each contain $n$ elements. ∎

We now state an extended version of Lemma 4.3 which handles the inert case
allowing us to give a formula in Theorem 1.2 for
$d(B|\mathscr{P}_{\mathbb{Q}}^{2\ell})$, the overall density of rational
primes that satisfy $\star$.

######  Lemma 6.3.

Let $K:=K(n,\ell)$.

  1. (1)

For any $\alpha\in\mathbf{M}_{4}$, the density of
$\mathfrak{p}\in\mathscr{P}_{K}^{2\ell}$ such that
$\varphi(\mathfrak{p})=\alpha$ is $\frac{1}{2^{n}}$. That is,

$d(\mathbf{r}_{0}^{-1}(\alpha)|\mathscr{P}_{K}^{2\ell})=\frac{1}{\\#\mathbf{M}_{4}}=\frac{1}{2^{n}}.$

  2. (2)

Restricting to primes of $K$ that split completely in $K/\mathbb{Q}$,

$d(\mathbf{r}_{0}^{-1}(\alpha)\cap
S^{\prime}|S^{\prime})=\frac{1}{\\#\mathbf{M}_{4}}=\frac{1}{2^{n}}.$

  3. (3)

Restricting to inert primes of $K$,

$d(\mathbf{r}_{0}^{-1}(\alpha)\cap
I^{\prime}|I^{\prime})=\left\\{\begin{array}[]{cc}\frac{1}{2}&\text{if
}\alpha=\pm 1\\\ 0&\text{otherwise.}\end{array}\right.$

###### Proof.

Part (a) and part (b) were proven in Lemma 4.3.

If $\alpha\neq\pm 1$ (for $\alpha\in\mathbf{M}_{4}$) then
$\mathbf{r}_{0}^{-1}(\alpha)\cap I^{\prime}=\emptyset$ since $\pm 1$ are the
only invariants of the $\operatorname{Gal}(K/\mathbb{Q})$-action on
$\mathbf{M}_{4}$ by Lemma 3.3. Therefore $d(\mathbf{r}_{0}^{-1}(\alpha)\cap
I^{\prime}|I^{\prime})=0$ if $\alpha\neq\pm 1$.

Now fix $s=\pm 1$. Then

$\mathbf{r}_{0}^{-1}(s)\cap I^{\prime}=\left\\{\mathfrak{p}\in
I^{\prime}:\left(\frac{\alpha}{4}\right)_{K}=s\right\\}$

where $\left(\frac{\alpha}{4}\right)_{K}$ denotes the quadratic residue symbol
in $\mathcal{O}_{K}$ for $\alpha\in\mathcal{O}_{K}$ a totally positive
generator of $\mathfrak{p}^{h(K)}$. This is a congruence condition so it is
routine to show that

$d(\mathbf{r}_{0}^{-1}(s)\cap I^{\prime}|I^{\prime})=\frac{1}{2}.$

by the cyclic case of Cebotarev’s Density Theorem. Theorem 4 in [Ser81]
asserts Cebotarev’s Density Theorem for natural density, or see [Neu99]
Theorem VII.13.4 for an simpler proof using Dirichlet density. ∎

We now prove the main results.

See 1.2

###### Proof.

Recall Definition 6.1. Note that by Lemma 2.2, $B$ is the set of rational
primes not dividing $2\ell$ that satisfy the given spin relation. Therefore

$D_{K}=d(B|\mathscr{P}_{\mathbb{Q}}^{2\ell}).$

Let $N\in\mathbb{Z}_{+}$. Let $I_{N}$ and $S_{N}$ denote the sets of
(rational) primes in $I$ and $S$ respectively with positive generator less
than $N$. Let $I^{\prime}_{N}\subseteq\mathscr{P}_{K}^{2\ell}$ denote the set
of primes of $K$ which lay above rational primes in
$I_{N}\subseteq\mathscr{P}_{\mathbb{Q}}^{2\ell}$ and define $S_{N}^{\prime}$
similarly with respect to $S_{N}\subseteq\mathscr{P}_{\mathbb{Q}}^{2\ell}$.
Note that while $S^{\prime}_{N}=\\{\mathfrak{p}\in
S^{\prime}:\operatorname{Norm}_{K/\mathbb{Q}}(\mathfrak{p})<N\\}$,

$I^{\prime}_{N}=\\{\mathfrak{p}\in
I^{\prime}:\operatorname{Norm}_{K/\mathbb{Q}}(\mathfrak{p})<N^{n}\\}.$

Since we have restricted to primes that are inert in $K/\mathbb{Q}$,

$\frac{\\#B\cap I_{N}}{\\#I_{N}}=\frac{\\#B^{\prime}\cap
I^{\prime}_{N}}{\\#I^{\prime}_{N}}$

where
$B^{\prime}:=\\{\mathfrak{p}\in\mathscr{P}_{K}^{2\ell}:\star(\mathfrak{p})=1\\}=\\{\mathfrak{p}\in\mathscr{P}_{K}^{2\ell}:\mathfrak{p}\text{
lays above some }p\in B\\}.$

Let $\mathbf{r}_{0,N}$ denote the restriction of $\mathbf{r}_{0}$ to
$I^{\prime}_{N}\subseteq\mathscr{P}_{K}^{2\ell}$. Observe that
$\mathfrak{p}\in I^{\prime}$ implies $\mathfrak{p}^{\sigma}=\mathfrak{p}$ so
$\mathbf{r}_{0}(\mathfrak{p})=\pm 1$ for all $\mathfrak{p}\in I^{\prime}$ by
Lemma 3.3. Lemma 5.2 states that $\star(1)=1$ and $\star(-1)=-1$. Therefore

$B^{\prime}\cap I^{\prime}_{N}=\mathbf{r}_{0}^{-1}(1)\cap I_{N}^{\prime}.$

Therefore $d(B^{\prime}\cap I^{\prime}|I^{\prime})=\frac{1}{2}$ by part (c) of
Lemma 6.3. Then since $\frac{\\#B\cap I_{N}}{\\#I_{N}}=\frac{\\#B^{\prime}\cap
I^{\prime}_{N}}{\\#I^{\prime}_{N}}$, we have proven that

(5) $d(B\cap I|I)=\frac{1}{2}.$

Note that since $K/\mathbb{Q}$ is cyclic, $\mathscr{P}_{\mathbb{Q}}^{2\ell}$
is the disjoint union of $S$ and $I$.

Cebotarev’s Density Theorem is true for natural density by Theorem 4 in
[Ser81]. (Theorem VII.13.4 in [Neu99] gives a simpler proof using Dirichlet
density.)

By Cebotarev’s Density Theorem,
$d(S|\mathscr{P}_{\mathbb{Q}}^{2\ell})=\frac{1}{n}$ and
$d(I|\mathscr{P}_{\mathbb{Q}}^{2\ell})=\frac{n-1}{n}$. Therefore

$\displaystyle d(B|\mathscr{P}_{\mathbb{Q}}^{2\ell})$
$\displaystyle=\lim_{N\to\infty}\frac{\\#B_{N}}{\\#\mathscr{P}_{\mathbb{Q},N}^{2\ell}}=\lim_{N\to\infty}\left(\frac{\\#B\cap
I_{N}}{\\#I_{N}}\frac{\\#I_{N}}{\\#\mathscr{P}_{\mathbb{Q},N}^{2\ell}}+\frac{\\#B\cap
S_{N}}{\\#S_{N}}\frac{\\#S_{N}}{\\#\mathscr{P}_{\mathbb{Q},N}^{2\ell}}\right)\quad$
$\displaystyle=\left(\frac{1}{2}\right)\left(\frac{n-1}{n}\right)+\left(\frac{m_{K}n+1}{2^{n}}\right)\left(\frac{1}{n}\right)\quad\text{by
Theorem \ref{thm:littledensity}}$
$\displaystyle=\frac{2^{n-1}(n-1)+m_{K}n+1}{2^{n}n}.$

∎

######  Lemma 6.4.

Let $K:=K(n,\ell)$. For all $\alpha\in\mathbf{M}_{4}$, if $\star(\alpha)=1$
then $\star(-\alpha)=-1$.

###### Proof.

By Lemma 5.2, $(-1,-1)_{2}=-1$.

Next note that $(a,b)_{2}=(a^{\sigma},b^{\sigma})_{2}$ for all
$\sigma\in\operatorname{Gal}(K/\mathbb{Q})$ since $2$ is inert in $K$.

Assume $\star(\alpha)=1$. Then $(\alpha,\alpha^{\sigma})_{2}=1$ for all
nontrivial $\sigma\in\operatorname{Gal}(K/\mathbb{Q})$. Let
$\sigma\in\operatorname{Gal}(K/\mathbb{Q})$ be nontrivial. By
bimultiplicativity of Hilbert symbols,

$\displaystyle(-\alpha,-\alpha^{\sigma})_{2}$
$\displaystyle=(-\alpha,-1)_{2}(-\alpha,\alpha^{\sigma})_{2}$
$\displaystyle=(-1,-1)_{2}(\alpha,-1)_{2}(-1,\alpha^{\sigma})_{2}(\alpha,\alpha^{\sigma})_{2}.$

Next observe $(\alpha,-1)_{2}=(-1,\alpha)_{2}=(-1,\alpha^{\sigma})_{2}$, the
second equality coming from the Galois-invariance shown earlier in this proof.
Therefore $(\alpha,-1)_{2}(-1,\alpha^{\sigma})_{2}=1$. Then since
$(\alpha,\alpha^{\sigma})_{2}=1$ and $(-1,-1)_{2}=-1$, we get that

$(-\alpha,-\alpha^{\sigma})_{2}=-1.$

Therefore $\star(-\alpha)=-1$. ∎

Recall the Definitions 4.2 and 6.1 defining $S$ and $R$.

######  Theorem 6.5.

Let $K:=K(n,\ell)$.

$\frac{1}{2^{n}}\leq d(R|S)\leq\frac{1}{2}.$

###### Proof.

By Theorem 6.2,

$d(R|S)=\frac{1+m_{K}n}{2^{n}}=\frac{\\#\\{\alpha\in\mathbf{M}_{4}:\star(\alpha)=1\\}}{2^{n}}.$

Lemma 6.4 implies the upper bound; $\alpha\neq-\alpha$ in $\mathbf{M}_{4}$
because $-1$ is not a square modulo $4\mathcal{O}_{K}$.

The lower bound is true because $\star(1)=1$ by Lemma 5.2 so

$\\#\\{\alpha\in\mathbf{M}_{4}:\star(\alpha)=1\\}\geq 1.$

∎

Theorem 1.3 is a Corollary of Theorem 6.5 obtained from the fact that

$D_{K}=\frac{n-1}{2n}+\left(\frac{1}{n}\right)d(R|S)$

as in the proof of Theorem 1.2.

See 1.3

## Acknowlegements

This research was conducted at Cornell University under the supervision of
Ravi Ramakrishna. The paper was finalized at the Max Planck Institute for
Mathematics. In addition, the author would like to thank Brian Hwuang and
Christian Maire for their helpful mathematical discussions.

## References

  * [AF67] J. V. Armitage and A. Fröhlich.  Classnumbers and unit signatures.  Mathematika, 14:94–98, 1967.
  * [BCP97] Wieb Bosma, John Cannon, and Catherine Playoust.  The Magma algebra system. I. The user language.  J. Symbolic Comput., 24(3-4):235–265, 1997.  Computational algebra and number theory (London, 1993).
  * [FIMR13] J. B. Friedlander, H. Iwaniec, B. Mazur, and K. Rubin.  The spin of prime ideals.  Invent. Math., 193(3):697–749, 2013.
  * [McM18] Christine McMeekin.  A Density of Ramified Primes.  ProQuest LLC, Ann Arbor, MI, 2018.  Thesis (Ph.D.)–Cornell University, (pending publication).
  * [Mil13] J.S. Milne.  Class field theory (v4.02), 2013.  Available at www.jmilne.org/math/.
  * [Mun] Sam Mundy.  Conversation.
  * [Neu99] Jürgen Neukirch.  Algebraic number theory, volume 322 of Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences].  Springer-Verlag, Berlin, 1999.  Translated from the 1992 German original and with a note by Norbert Schappacher, With a foreword by G. Harder.
  * [Ser81] Jean-Pierre Serre.  Quelques applications du théorème de densité de Chebotarev.  Inst. Hautes Études Sci. Publ. Math., (54):323–401, 1981.