mO [#1]Paola

11institutetext: Department of Computer Science, Rice University, USA
11email: {zilin, pc51<EMAIL_ADDRESS>22institutetext: Microsoft STCA,
China
22email: {migon, xingyaozhang<EMAIL_ADDRESS>
# Grounding Language Models for Visual Entity Recognition

Zilin Xiao 11 Ming Gong 22 Paola Cascante-Bonilla 11
Xingyao Zhang 22 Jie Wu 22 Vicente Ordonez 11

###### Abstract

We introduce AutoVER, an Autoregressive model for Visual Entity Recognition.
Our model extends an autoregressive Multimodal Large Language Model by
employing retrieval augmented constrained generation. It mitigates low
performance on out-of-domain entities while excelling in queries that require
visually-situated reasoning. Our method learns to distinguish similar entities
within a vast label space by contrastively training on hard negative pairs in
parallel with a sequence-to-sequence objective without an external retriever.
During inference, a list of retrieved candidate answers explicitly guides
language generation by removing invalid decoding paths. The proposed method
achieves significant improvements across different dataset splits in the
recently proposed Oven-Wiki benchmark. Accuracy on the Entity seen split rises
from 32.7% to 61.5%. It also demonstrates superior performance on the unseen
and query splits by a substantial double-digit margin.

###### Keywords:

Language Model Visual Entity Recognition

##  1 Introduction

Multimodal Large Language Models (Multimodal LLMs) have demonstrated superior
performance on a variety of vision-and-language tasks such as visual question
answering, image captioning, zero-shot image classification, among others [46,
33]. Their abilities can be transferred with few-shot tuning [59, 2] or even
learning in context without parameter updates [4, 62]. Given their remarkable
generalization abilities and prior knowledge based on large scale pretraining
we consider there is still great potential for using them on tasks that
require knowledge grounding.

A recently proposed task requiring knowledge grounding is the Open-domain
Visual Entity Recognition (Oven-Wiki) [23] task. In this task, given an input
image and a question about the image, the goal is to answer with a very
specific entity from Wikipedia. For instance, the answers could be a specific
model of airplane, _e.g_. ATR 42, or British Aerospace 146. This is a very
challenging task where Multimodal LLM-based solutions are prone to
hallucinations, or producing answers that are not at the right level of
granularity. Importantly, visual entity recognition requires recognizing
entities that never appear in the training data (unseen split). Moreover, a
portion of the Oven-Wiki benchmark (query split) extends the task beyond
recognition, requiring non-trivial reasoning to resolve the query question.

Figure 1: A representative query-entity pair from Oven-Wiki. We briefly
illustrate the model inference process and compare predictions from PaLI-17B
(in red) and AutoVER-7B (in green) which obtains the correct answer: ATR 42.
AutoVER retrieves entity candidates without an external retriever (step 1),
dynamically constructs a prefix tree (trie) (step 2), and performs decoding-
time augmentation to guide autoregressive generation (step 3).

Visual Entity Recognition (VER) presents unique challenges compared to general
Visual Question Answering (VQA) [1]. The first challenge lies in the answer
label space comprising over 6 million Wikipedia entities, which is prohibitive
for classifier-based VQA models [49] as many entities considered in this
ontology are visually very similar. The second issue involves generation-based
VQA solutions [33, 14], where hallucinations can lead to generated text that
is not grounded to the entity space. Lastly, existing VQA approaches fail to
consider the visual information of candidates [56, 35]. Entity images from the
knowledge base also play a significant role when disambiguating between
entities with similar identifiers but different visual appearances. These
methods also struggle with out-of-domain generalization and multi-hop
reasoning, challenges which are precisely covered by the unseen and query
split of the Oven-Wiki dataset.

In this work, we introduce an Autoregressive Visual Entity Recognizer, the
first approach that enables multimodal language models to perform accurate
visual entity recognition over a massive knowledge base, abbreviated as
AutoVER. AutoVER addresses entity recognition by reformulating the problem as
a sequence-to-sequence generation problem, as depicted in Fig. 1. The query
image is translated into the token embedding space using a learnable
projection layer, in a manner akin to treating images as a foreign language
[37, 61]. This allows us to utilize pre-trained multi-modal language models
and enormously improves performance on the query split which requires
reasoning.

To take advantage of visual clues on the entity side and enhance the
generalization capability of the model on unseen entities, we propose a
unified and compact retrieval-augmented generation (RAG) framework upon
AutoVER. Specifically, a special token <ret> is added to the vocabulary, whose
last-layer hidden states serve as the representation for the query side. A
lightweight two-layer Transformer is responsible for fusing visual features
from entity images and textual features from entity descriptions and producing
a representation for the entities, allowing contrastive learning with query-
entity positive pairs in parallel with language modeling. Unlike other RAG
systems that directly use retrieved items as context [48, 64] or infuse them
into the model’s intermediate hidden states [32, 20, 13], AutoVER dynamically
constructs a prefix tree (trie) from the retrieved entity identifiers, and
then generates entity identifiers by leveraging a methodology that constrains
the next possible tokens and eliminates invalid options based on the trie,
thus ensuring the generated text can always be grounded in retrieved
candidates. To alleviate the entity granularity problem, two hard negative
sampling strategies are proposed in our contrastive-generative framework to
encourage the model to maximally distinguish between similar entities.

In summary, AutoVER offers several key advantages over baselines and prior
work on VQA in the realm of visual entity recognition: (i) we refine the
recognition process by integrating contrastive training into an MLLM. (ii) our
proposed retrieval-augmented constrained decoding framework guarantees correct
grounded entity prediction and enhances prediction over unseen entities; (iii)
we show that leveraging pre-trained visual models and knowledge graphs for
hard negative mining significantly strengthens our contrastive-generative
framework in fine-grained entity recognition; (iv) experimental results show
that AutoVER consistently outperforms fine-tuned CLIP and PaLI variants on all
Oven-Wiki splits. For instance, AutoVER-7B achieves a 61.5% accuracy on the
Oven-Wiki entity seen split over PaLI-17B’s 30.6%, and 21.7% on entity unseen
over PaLI-17B’s 12.4%. Additionally, we evaluate AutoVER alongside public
multimodal LLMs on entity-related questions from the A-OKVQA, demonstrating
that our model can effectively zero-shot transfer to general out-of-domain VQA
datasets. Furthermore, ablation studies confirm the effectiveness of the
retrieve-generate framework we introduce for visual entity recognition. Code
will be released at https://github.com/MrZilinXiao/AutoVER.

##  2 Related Work

Visual Entity Recognition (VER) [23] is an emerging task for assessing the
ability of a model to perform multi-modal knowledge grounding. It can be
regarded as a multi-choice variant of Visual Question Answering (VQA) [1] with
the key distinction that the choice set will comprise all entities in a
specific knowledge base. Earlier solutions view this problem as an image-text-
to-image-text retrieval task, where CLIP-based models [47] are fine-tuned and
the top-scored answer in inference is treated as the model prediction.
Alternatively, others rely on fine-tuning generative language models such as
PaLI [9] in an attempt to match the generated text with candidate entity
identifiers using sparse matching approaches such as BM25 [52]. Entity Linking
(EL) has been a long-standing text-only counterpart to VER, which entails
locating mentions in the document and disambiguating them against a set of
candidate entities. Similar to prior solutions in VER, a bi-encoder was used
to retrieve and link each potential mention to an external knowledge base. Our
method bears the closest resemblance with the recent generative EL paradigm
[6, 40, 67, 68] which also reduces the knowledge-grounding problem into a
sequence-to-sequence generation task. However, we stand out with several
distinct advantages, including the seamless integration of retrieval
capabilities into the language model and the usage of explicit guidance for
sequence generation.

Multi-modal Large Language Models (MLLMs) are motivated by the remarkable
reasoning abilities of Large Language Models (LLMs), enabling researchers to
explore transferring them into the vision and other domains. BLIP [34]
proposes to bootstrap the multi-modal understanding ability into a unified
semantic space. Flamingo [2] applies a cross-attention architecture to fuse
visual contexts, thus bringing visual in-context learning. LLaVA [37] builds
upon LLaMA [58] and uses instruction-tuning to align visual features to the
language space. The common practice of prompting MLLMs to produce proxy
representations for downstream usage is to expand the vocabulary of language
models. Such expansion augments the MLLM functionality beyond language
generation. FROMAGe [27], GILL [26], and LISA [30] achieve satisfactory
results over image-text-to-text retrieval, image generation and reasoning
segmentation by inserting a retrieval token, a generation token and a
segmentation token into MLLMs respectively. Our approach, built upon the
latest MLLM model architecture, presents a compelling alternative to the bi-
encoder shallow dot-product interaction or encoder-decoder sparse surface form
matching employed in previous works on visual entity recognition.

Retrieval-Augmented Language Models (RALMs) represent a class of NLP solutions
focusing on knowledge-intensive tasks. RALMs typically condition a language
model on relevant documents from a grounding corpus during generation, thereby
enhancing the performance in knowledge-intensive language understanding tasks.
Lewis _et al_. [32] jointly fine-tune a retriever with an encoder-decoder
model, enabling the community to explore the RALM paradigm in language
understanding. Guu _et al_. [19] train a bi-directional variant and also
demonstrate superior performance. Ram _et al_. [48] propose to do RALM in-
context without any further training of the LM. Apart from retrieved items in
context, Févry _et al_. [13] are the first to integrate retrieved entity
supervision by injecting intermediate representations. While augmented context
helps in mitigating the well-known issue of hallucination, it does not ensure
a faithful prediction with respect to an external knowledge base. Our
retrieve-generate framework is distinct from existing retrieval-augmented
methods. AutoVER reduces the candidate entity set from millions to hundreds
for generative language models through retrieval and dynamically constrains
language model generation using a prefix tree. This framework coincides with
the design philosophy of agents [18] in Interactive NLP [63], where an agent
interacts with the dynamic environment by beam-searching all available
options.

##  3 Methodology

###  3.1 Problem Definition

Open-domain Visual Entity Recognition (VER) can be viewed as a multimodal
knowledge grounding task, in which the model is required to process an image-
text pair input $x=(\mathbf{Q}_{\text{im}},\mathbf{Q}_{\text{t}})$ and predict
an entity $e$. $\mathbf{Q}_{\text{t}}$ describes the specific intent that
prompts the model to ground some entity $e$ in the image
$\mathbf{Q}_{\text{im}}$ to a label space $\mathcal{E}$. Each entity
$e\in\mathcal{E}$ is a member of the knowledge base
$\mathcal{K}=\\{(e,\mathbf{E}_{\text{im}},\mathbf{E}_{\text{desc}})\mid
e\in\mathcal{E}\\}$ where $\mathbf{E}_{\text{desc}}$ are the text description
and $\mathbf{E}_{\text{im}}$ is a set of relevant images of the entity. To
circumvent trivial solutions for some query questions, all image-text pairs
are annotated so the question cannot be correctly answered without the image.

Considering that Oven-Wiki is the only dataset currently available for VER, we
highlight the dataset splits within Oven-Wiki to underscore the challenges of
the VER task. The dataset is divided into train, validation, test, and human-
validation sets, with each of the first three containing both an entity split
and a query split. The Entity split focuses on image recognition across
diverse visual entities, while the Query split pertains to Visual Question
Answering with visually-situated natural language queries. Each split is
further divided into seen and unseen subsets to assess the generalization
capabilities of models beyond the training domain.

###  3.2 Model Overview

Figure 2: Joint training of in-batch contrastive learning and language
modeling in AutoVER. For each training quadruple consisting of an entity
image, an entity description, a query image and a query question, a
lightweight Transformer encoder produces the fused entity representation
$E_{i}$ (left half). A special retrieval token prompts the multimodal language
model to generate the query representation $Q_{i}$. The query-to-entity
contrastive training ($\mathrm{L}_{\text{query2ent}}$) encourages the correct
retrieval of entities given the query pair, and the language modeling
($\mathrm{L}_{\text{LM}}$) helps the successful entity grounding.

AutoVER consists of two main modules: a multi-modal language model $f_{\phi}$
initialized with pre-trained weights $\phi$, and a multi-modal entity encoder
$F_{\varphi}$ that fuses the entity image and textual description. To
integrate the retrieval functionality into $f_{\phi}$, we insert a special
token <ret> into the vocabulary of $f_{\phi}$ with its corresponding token
embedding denoted as $\mathbf{H}_{\texttt{<ret>}}$.

We illustrate the training process in Fig. 2. For a training sample
$(\mathbf{Q}_{\text{im}},\mathbf{Q}_{\text{t}},$
$\mathbf{E}_{\text{im}},\mathbf{E}_{\text{desc}})$, we first use a frozen pre-
trained CLIP visual encoder $g_{\text{vision}}$ and a learnable projection
$\mathbf{W}_{q}$ to extract query image features $\mathbf{H}_{\text{im}}$ and
the embedded query text representations $\mathbf{H}_{\text{t}}$ as follow:

$\displaystyle\mathbf{H}_{\text{im}}=\mathbf{W}_{q}\cdot
g_{\text{vision}}\left(\mathbf{Q}_{\text{im}}\right)$
$\displaystyle\mathbf{H}_{\text{t}}=\mathbf{W}_{\text{embed}}\cdot\mathbf{Q}_{\text{t}}$

Where $\mathbf{W}_{\text{embed}}$ is the token embedding layer associated with
$f_{\phi}$, thus, treating the query image features as a foreign language. We
concatenate the embedded query text representations and the query image
features with the retrieval token embedding to organize the input instruction
to model
$\mathbf{H}=[\mathbf{H}_{\text{im}},\mathbf{H}_{\text{t}},\mathbf{H}_{\texttt{<ret>}}]$.
The last-layer hidden states of <ret> token undergo dimension-matching
projection and L2 normalization to a hyperspherical space, serving as the
representation on the query side denoted as $Q$. The use of a causal attention
mask in $f_{\phi}$ allows this representation to incorporate the multi-modal
query without leaking any label information.

On the entity encoder side, a frozen CLIP visual encoder $g_{\text{vision}}$
transforms entity image $\mathbf{E}_{\text{im}}$ into grid image features
$\mathbf{Z}_{\text{im}}$, and the frozen CLIP text encoder $g_{\text{text}}$
encodes the entity identifier and description into text features
$\mathbf{Z}_{\text{text}}$. A two-layer Transformer encoder handles the fusion
of two modalities by a fusion token as a soft prompt [31], whose design was
also adopted in [25]. The final hidden states of the fusion token are also
subjected to dimension matching and L2 normalization, providing the entity
representation $E$.

In-Batch Contrastive Training. Given normalized representations from both the
query and entity side, $Q$ and $E$, we formally introduce our in-batch
contrastive training method specifically designed for query-to-entity
retrieval. While image-text retrieval has been popular in learning joint
vision and language representations [24, 47], few studies have delved into
image-text-to-image-text retrieval, which happens to be the focus of query-to-
entity retrieval. We adopt contrastive learning [10] and the InfoNCE loss [43]
used in previous image-text retrieval work. Starting with computing the cosine
similarity for a pair of representations,

$\operatorname{sim}(Q,E)=\frac{Q\cdot E}{\|Q\|\|E\|},$

we minimize the InfoNCE loss for query-to-entity on a mini-batch consisting of
$N$ query-entity pairs $(Q_{i},E_{i})$. Each corresponding pair is considered
a positive pair and others are treated as negatives. $\tau$ is a learnable
temperature parameter. Our formulation is as follows:

$\displaystyle\mathcal{L}_{\mathrm{query}2\mathrm{ent}}=-\frac{1}{N}\sum_{i=1}^{N}\left(\log\frac{\exp\left(\operatorname{sim}\left(Q_{i},E_{i}\right)/\tau\right)}{\sum_{j=1}^{N}\exp\left(\operatorname{sim}\left(Q_{i},E_{j}\right)/\tau\right)}\right).$

Note that we do not train for the reversed objective, _i.e_. entity-to-query,
as it does not align with our retrieval-augmented intuition.

In contrast to image-text retrieval which offers a unique correspondence
between images and texts, query-to-entity mapping is subjective, _i.e_.
multiple queries in the training data are associated with the same entity.
This requires a particular sampling strategy during training by ensuring that
a batch does not include the same entity, referred to as a “conflicting
batch”. To preserve the integrity of the training data distribution, we
implement a rejection sampling [16] method as an alternative to standard
random sampling, resorting to resampling upon encountering conflicting
batches.

Language Modeling.  With organized input instruction $\mathbf{H}$, we optimize
the following cross-entropy loss, which is known as the next token prediction
loss in causal language modeling:

$\mathcal{L}_{\mathrm{LM}}=-\frac{1}{N-n}\sum_{i=n}^{N}\log
P\left(y_{i}\mid\mathbf{H},\ldots,y_{i-1}\right),$

where $n$ is the length of $\mathbf{H}$, $N$ denotes the length of the
concatenation of $\mathbf{H}$ and the expected output, _i.e_. entity text
identifier (Bugatti Veyron in Figure 2). Note that we do not backward the next
token prediction loss on the input sequence, but only on the target sequence.

The final training loss is a linear combination of the language modeling loss
and in-batch contrastive loss, denoted as follows:

$\mathcal{L}=\mathcal{L}_{\mathrm{LM}}+\lambda_{r}\cdot\mathcal{L}_{\mathrm{query}2\mathrm{ent}},$

where $\lambda_{r}$ is an empirically determined trade-off hyperparameter.

Figure 3: Retrieval-augmented constrained decoding illustration of AutoVER
inference process. The representation $Q$ will query a pre-cached entity
database constructed using the multimodal entity encoder, and get the top-$k$
candidate entities. A prefix-constrained tree is dynamically built based on
retrieved entity identifiers and guides the language model to autoregressively
generate the next token, thereby ensuring the successful grounding of
generated content.

###  3.3 Hard-negative Mining

To alleviate the entity granularity problem, two hard negative sampling
approaches are proposed in our contrastive-generative framework, _i.e_.
vision-hard and kb-hard. Both approaches create entity groups used for
constructing an in-batch sampler that prefers sampling similar entities in
contrastive training, yet they differ in how they generate these similar
groups. vision-hard relies on a pre-trained ViT [12] image classifier to
identify visually similar entities based on shared prediction classes. In
contrast, kb-hard uses the category hierarchy of Wikidata as an external
knowledge source, considering entities that share a parent node in the
category hierarchy as knowledge-similar entities. We refer readers to Appendix
0.B for details about the selection of hard negative groups.

###  3.4 Retrieval-augmented Constrained Decoding

We illustrate the model inference in Fig. 3. After training, all entity
candidates are cached using the multi-modal entity encoder $F_{\varphi}$ to
construct an entity vector database $\mathcal{V}\in\mathbb{R}^{n\times d}$ for
efficient retrieval. Given an evaluation sample
$\left(\mathbf{Q}_{\text{im}},\mathbf{Q}_{\text{t}}\right)$, we first forward
the MLLM for the normalized representation of the <ret> token, and query
database $\mathcal{V}$ using top-$k$ similarity search to get $k$ entity
candidates. The model dynamically generates a prefix tree (trie) that covers
$k$ entity candidates, and the trie will explicitly guide entity identifier
generation by eliminating impossible decoding paths when autoregressively
generating tokens. The retrieval process takes into consideration the image of
entities and improves the model towards out-of-domain entities. Moreover, the
retrieval-augmented constrained decoding guarantees the grounding of the
generated content to the knowledge base, alleviating the issue of
hallucinations.

##  4 Experiments

###  4.1 Settings

Metrics. We follow the standard setup in [23], which uses accuracy and
harmonic mean to evaluate model performance on different data splits. The
harmonic mean of each split will equally weigh the importance of the seen and
unseen subsets and penalize models that show a weakness in either aspect.
Finally, we report the overall harmonic mean on the Entity and Query splits as
the final metric for the validation and test sets.

Data Pre-processing and Models. We pre-process all query and entity images by
resizing them into 336 $\times$ 336 pixels with padding to keep an identical
aspect ratio. Entity descriptions are truncated to 77 tokens to fit the
context window size of CLIP by cutting off sentences to the maximum available
ones. Since query texts are typically short, no truncation is needed for query
texts. Encoders on the entity side and visual encoder in MLLM are pre-trained
CLIP-ViTL/14-336px. MLLM is initialized with the vicuna-7b-v1.5 or
vicuna-13b-v1.5 checkpoint and corresponding visual projectors pre-aligned on
a 558k subset of LAION [54], CC [7] and SBU [45] curated by [36].

Training and Evaluation. We train on the entity train and query train splits
of Oven-Wiki, which consists of nearly 5 million query-entity pairs. We refer
to Appendix 0.A for hyper-parameter choices and training details. We set
$\lambda_{r}$ to 1 for all training settings. Due to a limited compute budget,
we select 10% and 50% of the training data through weighted sampling according
to entity occurrence frequency to facilitate ablation studies and final
results reporting. For the same reason, we do not run a hyperparameter search.
In evaluation, the number of retrieved entities $k$ is set to 300 according to
empirical trials on the validation split.

Table 1:  Comparison among models on the Oven-Wiki validation set. We report
accuracies for the seen and unseen subsets and the harmonic mean of each split
(hm). Metrics of CLIP and PaLI variants are from [23]. For each subset, bold
indicates the best metric.
|  |  | Entity Split | Query Split | Overall
---|---|---|---|---|---
Category |  | Method | seen | unseen | hm | seen | unseen | hm | hm
Discriminative |  | $\text{CLIP}_{\texttt{ViTL14}}$ |  05.4 |  05.3 |  05.4 |  00.8 |  01.4 |  01.0 |  01.7
| $\text{CLIP Fusion}_{\texttt{ViTL14}}$ | 32.7 |  04.3 |  07.7 | 33.4 |  02.2 |  04.2 |  05.4
| $\text{CLIP2CLIP}_{\texttt{ViTL14}}$ | 12.6 | 10.1 | 11.2 |  04.1 |  02.1 |  02.8 |  04.4
Generative |  | PaLI-3B | 21.6 | 6.6 | 10.1 | 33.2 | 14.7 | 20.4 | 13.5
| PaLI-17B | 30.6 | 12.4 | 17.6 | 44.2 | 22.4 | 29.8 | 22.1
Zero-shot |  | $\text{{BLIP-2}}_{\texttt{Flan-T5-XXL}}$ | 8.6 | 3.4 | 4.9 | 24.6 | 17.7 | 20.6 | 7.9
| GPT-4V | 29.8 | 19.3 | 23.4 | 56.5 | 52.7 | 54.5 | 32.9
Ours |  | AutoVER-7B | 61.5 | 21.7 | 32.1 | 69.0 | 31.4 | 43.2 | 36.8
| AutoVER-13B | 63.6 | 24.5 | 35.6 | 68.6 | 32.3 | 43.9 | 39.2

###  4.2 Main Results

Table 2:  Results of methods on the Oven-Wiki test set and human evaluation set. Human evaluation results from [23] are highlighted in gray.  | Entity Split | Query Split | Overall | Human Eval
---|---|---|---|---
Method | seen | unseen | seen | unseen | hm | seen | unseen | hm
Human + Search | - | - | - | - | - | 76.1 | 79.3 | 77.7
$\text{CLIP}_{\texttt{ViTL14}}$ |  05.6 |  04.9 |  01.3 |  02.0 |  02.4 |  04.6 |  06.0 |  05.2
$\text{CLIP Fusion}_{\texttt{ViTL14}}$ | 33.6 |  04.8 | 25.8 |  01.4 |  04.1 | 18.0 |  02.9 |  05.0
$\text{CLIP2CLIP}_{\texttt{ViTL14}}$ | 12.6 | 10.5 |  03.8 |  03.2 |  05.3 | 14.0 | 11.1 | 12.4
PaLI-3B | 19.1 | 6.0 | 27.4 | 12.0 | 11.8 | 30.5 | 15.8 | 20.8
PaLI-17B | 28.3 | 11.2 | 36.2 | 21.7 | 20.2 | 40.3 | 26.0 | 31.6
AutoVER-7B | 62.8 | 16.0 | 63.7 | 31.9 | 31.8 | 64.7 | 39.9 | 49.4
AutoVER-13B | 65.0 | 18.6 | 65.7 | 32.0 | 34.6 | 68.4 | 44.2 | 53.7

Results on the validation set are presented in Tab. 1. Our experimental
results demonstrate a consistent improvement in all data splits and subsets on
Oven-Wiki. Specifically, we observe a double accuracy improvement in the
entity seen subsets of both Oven-Wiki validation and test splits. This
indicates that AutoVER can more effectively utilize its parameter capacity,
achieving stronger in-domain visual entity recognition capabilities with fewer
model parameters. The improvement on the query seen subsets is slightly lower
compared with the entity seen improvement but still significant with +24.8%
relative accuracy difference. We attribute this to the inherent reasoning and
visual localization capabilities within the MLLM, a proficiency that has been
extensively demonstrated across various benchmarks targeting MLLMs [41, 8].

On unseen subsets, accuracy is severely impacted by queries whose answers
involve out-of-domain entities, particularly in the Entity split.
Nevertheless, AutoVER still outperforms the largest model, PaLI-17B, which can
be attributed to the retrieval-augmented framework eliminating many improbable
options for the decision-making of the MLLM.

We report results on test and human evaluation set in Tab. 2. Similarly,
AutoVER-13B achieves the best performance on subsets and splits of the test
set. However, it must be acknowledged that there remains a gap between our
results and those of Human + Search in the human validation set, particularly
in the human unseen subset, where the difference reaches 35.1%.

In addition, we report the zero-shot visual entity recognition abilities of
the largest BLIP-2 [33] checkpoint and GPT-4V111Due to the limited budget, we
only conduct experiments on 10% of Entity Split${}_{(\texttt{Val})}$ and 50%
of Query Split${}_{(\texttt{Val})}$. Also, the detail of the image is set to
low to minimize prompt token consumption. [44] on the validation set in Tab.
1, whereas no fine-tuning is performed on the training dataset. For BLIP-2,
zero-shot generated outputs are grounded to Wikipedia using BM25, following
generative baseline methods. As for GPT-4V, its elaborate generation, a result
of instruction tuning, impedes effective evaluation with BM25. Hence, we adopt
a partial matching evaluation strategy, treating the presence of the entity
identifier in the generated text as a true positive. While BLIP-2 performance
is anticipated, as its lack of fine-tuning hinders the acknowledgment of
entity grounding intents, GPT-4V has surprisingly exceptional performance on
the query split. Nevertheless, GPT-4V poor performance on the entity split
still leaves it trailing behind AutoVER. Furthermore, the opacity of its
training data prevents us from determining if GPT-4V excellent zero-shot
performance is attributable to possible data leakage.

Table 3: Model accuracies on the A-OKVQA-Ent under multi-choice evaluation strategy. ∗ denotes unusual results due to failure to follow multiple-choice prompts. Bold indicates the best metric under the same setting.  Supervision | Method | seen | unseen | Overall
---|---|---|---|---
Zero-shot |  OpenFlamingo-9B∗ | 5.9 | 9.0 | 6.9
$\text{InstructBLIP}_{\texttt{vicuna-7B}}$ | 53.7 | 49.4 | 52.4
LLaVA-v1-7B∗ | 13.0 | 10.3 | 12.1
AutoVER-7B (Ours) | 67.7 | 52.5 | 62.8
Fine-tuned | LLaVA-v1.5-7B | 72.4 | 73.7 | 72.8
Table 4: Model hit rates on the A-OKVQA-Ent under entity match evaluation strategy.  Supervision | Method | seen | unseen | Overall
---|---|---|---|---
Zero-shot | OpenFlamingo-9B | 39.8 | 30.8 | 36.8
$\text{InstructBLIP}_{\texttt{vicuna-7B}}$ | 38.8 | 34.0 | 37.2
LLaVA-v1-7B | 45.7 | 36.5 | 42.7
AutoVER-7B (Ours) | 61.3 | 42.3 | 55.0
Fine-tuned | LLaVA-v1.5-7B | 47.8 | 42.9 | 46.2

###  4.3 Zero-shot Generalization Results

Although Oven-Wiki has already incorporated 14 classic
datasets222ImageNet21k-P [53, 51], iNaturalist2017 [60], Cars196 [28], SUN397
[66], Food101 [5], Sports100 [15], Aircraft [38], Oxford Flower [42], Google
Landmarks v2 [65], VQA v2 [17], Visual7W [69], Visual Genome [29], OK-VQA
[39], Text-VQA [57] from image recognition and visual question answering,
along with extensive annotation efforts that ground answers to the knowledge
base, we still hope to test the generalization capability of our method using
additional out-of-domain datasets. As such, we manually curated a subset from
the A-OKVQA [55] validation dataset. The subset does not overlap with any
dataset sources in Oven-Wiki and covers all question-answer pairs in the
A-OKVQA validation split whose answer can be grounded to entities in the
Wikipedia knowledge base. We name the subset A-OKVQA-Ent to emphasize this
distinct property. Following the design of Oven-Wiki, we divide the subset
into seen and unseen splits depending on whether the answer entity is in the
Oven-Wiki train set. This subset includes 478 entries out of 1145 in the
A-OKVQA validation set, of which 322 are identified as seen, while the
remaining 156 are classified as unseen.

To adhere to the evaluation setting of A-OKVQA, we adopted two evaluation
approaches – multi-choice and entity match. In multi-choice setting, AutoVER
constructed a prefix tree with four available options to guide generation,
while for other models, options are numbered in the prompt and either the
correct option number or an exact entity identifier match in the response is
considered a true prediction. In entity match setting, a partial match of the
entity text identifier within the generated response qualifies a correct
prediction. To ensure equitable comparison, baselines included are other
generative models comparable in size with AutoVER-7B, specifically LLaVA [37]
and its improved v1.5 version [36], OpenFlamingo [3], and InstructBLIP [11].
While we expect that all models have not seen the A-OKVQA dataset so that we
can assess the zero-shot generalization ability, it is important to
acknowledge that LLaVA-v1.5 has been fine-tuned with 50k A-OKVQA multi-choice
instructions, potentially giving it an edge.

We present results on A-OKVQA-Ent in Tab. 3 and Tab. 4, and refer readers to
Appendix 0.E for qualitative analysis on this dataset. Our method still
outperforms baseline models across all evaluation settings, with the exception
of multiple-choice LLaVA-v1.5 which has been tuned with in-domain
instructions. Intriguingly, we find that LLaVA-v1.5 in the entity match
evaluation setting still falls short of AutoVER-7B, although it has seen in-
domain samples. We also observed that LLaVA-v1 and OpenFlamingo, with
inadequate instruction tuning, find it challenging to comply with the
multiple-choice setting, and instead generate content unrelated to the given
options, leading to their significantly lower metrics. Those findings reveal
that generative language models struggle to adapt well to out-of-domain
instructions and that unrestricted generation is prone to hallucinations in
tasks requiring precise recognition. This further emphasizes the advantages of
retrieval-assisted decoding-time augmentation for such tasks.

###  4.4 Ablation Study and Discussion

We discuss ablation studies on AutoVER-7B covering the effectiveness of our
retrieve-generate framework, constrained decoding, and hard negative mining
methods in Tab. 5. Ablations are conducted with 10% of training data, as such
the metrics in ablations differ from those from the main results. We name
models in ablations AutoVER-7B-0.1 to indicate such discrepancy.

Retrieval Augmentation (R). We first focus on the absence of retrieval
augmentation in AutoVER. Without retrieval, the model performs constrained
decoding over a prefix tree composed of all Wikipedia entity identifiers
instead of a set of retrieved candidates. We notice a slight performance gain
on the seen subset, likely due to the gold entity not being covered in
retrieval. Conversely, the integrated retrieval design is found to
significantly improve the performance of AutoVER on the unseen subset from
non-viable to viable. Retrieval-augmented constrained decoding narrows down
the decision-making scope of the language model, allowing AutoVER to generate
entity identifiers that are never trained in the model.

Table 5: Ablation study of AutoVER-7B-0.1 on Oven-Wiki Entity Split${}_{(\texttt{Val})}$. Performance of original AutoVER-7B-0.1 is reported in the blue background cell. R:  whether to enable retrieval augmentation. CD:  revert to greedy decoding if disabled. LoRA:  parameter-efficient tuning variant of AutoVER-7B-0.1. CLS:  classifier-based variant is tuned to produce the entity prediction.  R | CD | LoRA | CLS | seen | unseen | hm
---|---|---|---|---|---|---
$\checkmark$ | $\checkmark$ |  |  | 48.9 | 19.0 | 27.4
| $\checkmark$ |  |  | 50.7 | 0.6 | 1.2
|  |  |  | 46.8 | 0.6 | 1.2
|  | $\checkmark$ |  | 43.5 | 2.8 | 5.3
|  |  | $\checkmark$ | 12.8 | 0.1 | 0.2

Constrained Decoding (CD). Upon the excluding of retrieval augmentation, we
proceed to disable the entire constrained decoding mechanism, reverting
AutoVER to greedily decode the next token, where grounding to an external
knowledge base is no longer assured. The consequent decrease in seen accuracy
demonstrates that constrained decoding effectively alleviates hallucinations
of the model in predicting entities. Unfortunately, since neither checkpoints
of encoder-decoder variants from [23] in Table 1 and 2 nor PaLI pre-trained
weights are publicly available, we are unable to assess whether our proposed
decoding-time augmenting methods can bring universal improvement on generative
baselines.

Parameter-Efficient Tuning (LoRA). Our ablations extend to the impact of
parameter-efficient tuning methods on AutoVER. Specifically, we configure a
low-rank adapter (LoRA) [22] with a rank of 128 and alpha set at 256 and train
this LoRA-variant of AutoVER. The results reveal that the efficacy of LoRA
falls short of expectation and explicitly harms the recognition performance on
the seen subset. We leave more specialized efficient tuning methods for the
generative VER framework as future work.

Autoregressive Model or Classifier (CLS). One may attribute the success of
AutoVER to the large model size of its underlying MLLM and accompanied
expressive capacity instead of our proposed design. In response to that, we
devise a [CLS] variant of AutoVER. It follows the classical design of treating
a decoder-only LM as a classifier. Specifically, we introduce a new [CLS]
token into the MLLM vocabulary and append this token at the end of each query
image-question pair. The corresponding last-layer hidden states are then fine-
tuned for classifying the query into the 20,549 existing entities in Oven-
Wiki. We observed a significant performance degradation in this setting,
particularly in the unseen subset. This highlights the ineffectiveness of
classifier-based VQA methods in the realm of visual entity recognition,
indicating that our model performance stems from more than just its size.

Figure 4: Illustration of selected query image-question pairs, retrieved
candidates and AutoVER-7B decisions. AutoVER adeptly captures slight
variations in the query text and retrieves entirely different entity
candidates, which forms the basis for the generative decisions from the
language model.

###  4.5 Case Study

We illustrate the retrieved entity candidates and model decisions on four
representative image-question pairs. We note that for the same query image,
questions with different intents lead to retrieval representations with
semantic relevance to the intent, laying a solid foundation for the subsequent
constrained generation. For instance, in the upper part of Fig. 4, the
question asking for the specific model of airplane prompts the model to
retrieve various kinds of airplanes. With a slight modification from “model”
to “manufacturer” in the query question, the model adapts to retrieve famous
airplane manufacturer brands instead of plane models. Likewise, the model is
also proficient in managing queries that demand visual localization and
reasoning abilities, as depicted in the lower part of Fig. 4. We refer to
Appendix 0.C for the error analysis and Appendix 0.D for more case study with
comparison against GPT-4V.

##  5 Conclusion

We present AutoVER, a compact retrieval-augmented generation framework
specifically designed for visual entity recognition. Utilizing a novel
constrained decoding technique, this approach effectively overcomes the
challenges of low performance in recognizing out-of-domain entities while
demonstrating remarkable proficiency in questions that require visually-
situated reasoning. AutoVER marks a significant advancement in visual entity
recognition by doubling accuracy on nearly all challenging subsets of
benchmarks.

## References

  * [1] Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C.L., Parikh, D., Batra, D.: Vqa: Visual question answering. International Journal of Computer Vision (2015). https://doi.org/10.1007/s11263-016-0966-6
  * [2] Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35, 23716–23736 (2022)
  * [3] Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., Jitsev, J., Kornblith, S., Koh, P.W., Ilharco, G., Wortsman, M., Schmidt, L.: Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv: 2308.01390 (2023)
  * [4] Bar, A., Gandelsman, Y., Darrell, T., Globerson, A., Efros, A.: Visual prompting via image inpainting. Advances in Neural Information Processing Systems 35, 25005–25017 (2022)
  * [5] Bossard, L., Guillaumin, M., Van Gool, L.: Food-101 – mining discriminative components with random forests. In: ECCV (2014)
  * [6] Cao, N.D., Izacard, G., Riedel, S., Petroni, F.: Autoregressive entity retrieval. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net (2021), https://openreview.net/forum?id=5k8F6UU39V
  * [7] Changpinyo, S., Sharma, P., Ding, N., Soricut, R.: Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. Computer Vision and Pattern Recognition (2021). https://doi.org/10.1109/CVPR46437.2021.00356
  * [8] Chen, L., Li, B., Shen, S., Yang, J., Li, C., Keutzer, K., Darrell, T., Liu, Z.: Large language models are visual reasoning coordinators. arXiv preprint arXiv: 2310.15166 (2023)
  * [9] Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D.M., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., Kolesnikov, A., Puigcerver, J., Ding, N., Rong, K., Akbari, H., Mishra, G., Xue, L., Thapliyal, A.V., Bradbury, J., Kuo, W., Seyedhosseini, M., Jia, C., Ayan, B.K., Riquelme, C., Steiner, A., Angelova, A., Zhai, X., Houlsby, N., Soricut, R.: Pali: A jointly-scaled multilingual language-image model. International Conference on Learning Representations (2022). https://doi.org/10.48550/arXiv.2209.06794
  * [10] Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05). vol. 1, pp. 539–546 vol. 1 (2005). https://doi.org/10.1109/CVPR.2005.202
  * [11] Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning. NEURIPS (2023)
  * [12] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net (2021), https://openreview.net/forum?id=YicbFdNTTy
  * [13] Févry, T., Baldini Soares, L., FitzGerald, N., Choi, E., Kwiatkowski, T.: Entities as experts: Sparse memory access with entity supervision. In: Webber, B., Cohn, T., He, Y., Liu, Y. (eds.) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 4937–4951. Association for Computational Linguistics, Online (Nov 2020). https://doi.org/10.18653/v1/2020.emnlp-main.400, https://aclanthology.org/2020.emnlp-main.400
  * [14] Gao, F., Ping, Q., Thattai, G., Reganti, A., Wu, Y.N., Natarajan, P.: Transform-retrieve-generate: Natural language-centric outside-knowledge visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5067–5077 (2022)
  * [15] Gerry: Sports100: 100 sports image classification. https://www.kaggle.com/datasets/gpiosenka/sports-classification/metadata (2021)
  * [16] Gilks, W.R., Wild, P.: Adaptive rejection sampling for gibbs sampling. Journal of the Royal Statistical Society: Series C (Applied Statistics) 41(2), 337–348 (1992)
  * [17] Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In: CVPR (2017)
  * [18] Gu, Y., Deng, X., Su, Y.: Don’t generate, discriminate: A proposal for grounding language models to real-world environments. Annual Meeting of the Association for Computational Linguistics (2022). https://doi.org/10.48550/arXiv.2212.09736
  * [19] Guu, K., Lee, K., Tung, Z., Pasupat, P., Chang, M.W.: Realm: retrieval-augmented language model pre-training. ICML (2020), https://dl.acm.org/doi/abs/10.5555/3524938.3525306
  * [20] Guu, K., Lee, K., Tung, Z., Pasupat, P., Chang, M.: Retrieval augmented language model pre-training. In: III, H.D., Singh, A. (eds.) Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 3929–3938. PMLR (13–18 Jul 2020), https://proceedings.mlr.press/v119/guu20a.html
  * [21] He, Z., Zhong, Z., Cai, T., Lee, J.D., He, D.: Rest: Retrieval-based speculative decoding. arXiv preprint arXiv: 2311.08252 (2023)
  * [22] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. In: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net (2022), https://openreview.net/forum?id=nZeVKeeFYf9
  * [23] Hu, H., Luan, Y., Chen, Y., Khandelwal, U., Joshi, M., Lee, K., Toutanova, K., Chang, M.W.: Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. ICCV (2023)
  * [24] Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q.V., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning (2021)
  * [25] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment anything. arXiv preprint arXiv: 2304.02643 (2023)
  * [26] Koh, J.Y., Fried, D., Salakhutdinov, R.: Generating images with multimodal language models. arXiv preprint arXiv: 2305.17216 (2023)
  * [27] Koh, J.Y., Salakhutdinov, R., Fried, D.: Grounding language models to images for multimodal inputs and outputs. International Conference on Machine Learning (2023)
  * [28] Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for fine-grained categorization. In: 3dRR-13 (2013)
  * [29] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision 123(1), 32–73 (2017)
  * [30] Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., Jia, J.: Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv: 2308.00692 (2023)
  * [31] Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. In: Moens, M.F., Huang, X., Specia, L., Yih, S.W.t. (eds.) Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 3045–3059. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.243, https://aclanthology.org/2021.emnlp-main.243
  * [32] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-augmented generation for knowledge-intensive nlp tasks. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 9459–9474. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf
  * [33] Li, J., Li, D., Savarese, S., Hoi, S.C.H.: BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (eds.) International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA. Proceedings of Machine Learning Research, vol. 202, pp. 19730–19742. PMLR (2023), https://proceedings.mlr.press/v202/li23q.html
  * [34] Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. International Conference on Machine Learning (2022)
  * [35] Lin, W., Chen, J., Mei, J., Coca, A., Byrne, B.: Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering. In: H.Larochelle, M.Ranzato, R.Hadsell, M.F.Balcan, H.Lin (eds.) Advances in Neural Information Processing Systems. Curran Associates, Inc. (2023), https://proceedings.neurips.cc/paper_files/paper/2023/hash/47393e8594c82ce8fd83adc672cf9872-Abstract-Conference.html
  * [36] Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. arXiv preprint arXiv: 2310.03744 (2023)
  * [37] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: NeurIPS (2023)
  * [38] Maji, S., Kannala, J., Rahtu, E., Blaschko, M., Vedaldi, A.: Fine-grained visual classification of aircraft. Tech. rep. (2013)
  * [39] Marino, K., Rastegari, M., Farhadi, A., Mottaghi, R.: Ok-vqa: A visual question answering benchmark requiring external knowledge. In: Proceedings of the IEEE/cvf conference on computer vision and pattern recognition. pp. 3195–3204 (2019)
  * [40] Mrini, K., Nie, S., Gu, J., Wang, S., Sanjabi, M., Firooz, H.: Detection, disambiguation, re-ranking: Autoregressive entity linking as a multi-task problem. In: Muresan, S., Nakov, P., Villavicencio, A. (eds.) Findings of the Association for Computational Linguistics: ACL 2022. pp. 1972–1983. Association for Computational Linguistics, Dublin, Ireland (May 2022). https://doi.org/10.18653/v1/2022.findings-acl.156, https://aclanthology.org/2022.findings-acl.156
  * [41] Naveed, H., Khan, A.U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., Mian, A.: A comprehensive overview of large language models. arXiv preprint arXiv: 2307.06435 (2023)
  * [42] Nilsback, M.E., Zisserman, A.: Automated flower classification over a large number of classes. In: 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. IEEE (2008)
  * [43] van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.03748 (2018)
  * [44] OpenAI: GPT-4V(ision) System Card. https://cdn.openai.com/papers/GPTV_System_Card.pdf (Sep 2023)
  * [45] Ordonez, V., Kulkarni, G., Berg, T.L.: Im2text: Describing images using 1 million captioned photographs. In: Neural Information Processing Systems (NIPS) (2011)
  * [46] Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., Wei, F.: Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv: 2306.14824 (2023)
  * [47] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event. Proceedings of Machine Learning Research, vol. 139, pp. 8748–8763. PMLR (2021), http://proceedings.mlr.press/v139/radford21a.html
  * [48] Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., Shoham, Y.: In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics (2023). https://doi.org/10.1162/tacl_a_00605
  * [49] Ramakrishnan, S., Agrawal, A., Lee, S.: Overcoming language priors in visual question answering with adversarial regularization. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 31. Curran Associates, Inc. (2018), https://proceedings.neurips.cc/paper_files/paper/2018/file/67d96d458abdef21792e6d8e590244e7-Paper.pdf
  * [50] Rasley, J., Rajbhandari, S., Ruwase, O., He, Y.: Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. p. 3505–3506. KDD ’20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3394486.3406703, https://doi.org/10.1145/3394486.3406703
  * [51] Ridnik, T., Ben-Baruch, E., Noy, A., Zelnik-Manor, L.: Imagenet-21k pretraining for the masses. In: NeurIPS (2021)
  * [52] Robertson, S., Zaragoza, H., et al.: The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval 3(4), 333–389 (2009)
  * [53] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. IJCV (2015)
  * [54] Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.: Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems 35, 25278–25294 (2022)
  * [55] Schwenk, D., Khandelwal, A., Clark, C., Marino, K., Mottaghi, R.: A-OKVQA: A benchmark for visual question answering using world knowledge. In: ECCV (8). Lecture Notes in Computer Science, vol. 13668, pp. 146–162. Springer (2022)
  * [56] Shao, Z., Yu, Z., Wang, M., Yu, J.: Prompting large language models with answer heuristics for knowledge-based visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 14974–14983 (June 2023)
  * [57] Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In: CVPR (2019)
  * [58] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., Lample, G.: Llama: Open and efficient foundation language models. ARXIV (2023)
  * [59] Tsimpoukelli, M., Menick, J.L., Cabi, S., Eslami, S., Vinyals, O., Hill, F.: Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems 34, 200–212 (2021)
  * [60] Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., Belongie, S.: The inaturalist species classification and detection dataset. In: CVPR (2018)
  * [61] Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O.K., Singhal, S., Som, S., et al.: Image as a foreign language: Beit pretraining for vision and vision-language tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19175–19186 (2023)
  * [62] Wang, X., Wang, W., Cao, Y., Shen, C., Huang, T.: Images speak in images: A generalist painter for in-context visual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 6830–6839 (June 2023)
  * [63] Wang, Z., Zhang, G., Yang, K., Shi, N., Zhou, W., Hao, S., Xiong, G., Li, Y., Sim, M.Y., Chen, X., Zhu, Q., Yang, Z., Nik, A., Liu, Q., Lin, C., Wang, S., Liu, R., Chen, W., Xu, K., Liu, D., Guo, Y., Fu, J.: Interactive natural language processing. CoRR abs/2305.13246 (2023). https://doi.org/10.48550/arXiv.2305.13246, https://doi.org/10.48550/arXiv.2305.13246
  * [64] Wang, Z., Araki, J., Jiang, Z., Parvez, M.R., Neubig, G.: Learning to filter context for retrieval-augmented generation. arXiv preprint arXiv: 2311.08377 (2023)
  * [65] Weyand, T., Araujo, A., Cao, B., Sim, J.: Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In: CVPR (2020)
  * [66] Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: Large-scale scene recognition from abbey to zoo. In: CVPR (2010)
  * [67] Xiao, Z., Gong, M., Wu, J., Zhang, X., Shou, L., Pei, J., Jiang, D.: Instructed language models with retrievers are powerful entity linkers. arXiv preprint arXiv: 2311.03250 (2023)
  * [68] Xiao, Z., Shou, L., Zhang, X., Wu, J., Gong, M., Pei, J., Jiang, D.: Coherent entity disambiguation via modeling topic and categorical dependency. arXiv preprint arXiv: 2311.03253 (2023)
  * [69] Zhu, Y., Groth, O., Bernstein, M., Fei-Fei, L.: Visual7w: Grounded question answering in images. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2016)

##  Appendix 0.A Experimental Details

Table 6: Hyperparameter settings for training of AutoVER. Hyperparameter | Value
---|---
learning rate | 2e-5
weight_decay | 0
batch size per device | 8
effective batch size | 256
learning rate strategy | Cosine
optimizer | AdamW
gradient clipping | Disabled

We present the hyperparameter choice in Table 6. As the effectiveness of
contrastive learning only reveals in a large batch size setting, we conduct
all pieces of training in a batch size of 256 on 32 V100-SXM2-32GB GPUs.
Utilizing DeepSpeed [50] for distributed training management, we train
AutoVER-7B for 2 days and AutoVER-13B for 6.5 days with 2.5 million training
samples for one epoch.

##  Appendix 0.B Hard Negative Groups Construction Details

We first illustrate the process of constructing kb-hard group construction.
Specifically, we consider three types of relations in Wikidata for category
hierarchy construction, “instance of (P31)”, “parent taxon (P171)” and
“subclass of (P279)”. To obtain category hierarchy labels, we execute the
SPARQL statement shown on the left of Figure 6 on Wikidata knowledge graph. To
exclude generic categories, we limit the depth of traversal on the knowledge
graph to 3, and manually filter generic categories333[“Wikidata metaclass”,
“second-order class”, “taxon”, “classification scheme”, “metaclass”, “third-
order class”, “human”, “direct anatomical metaclass”, “organisms known by a
particular common name”, “Wikimedia disambiguation page”, “variable-order
class”, “concept”, “food”, “artificial physical object”, “artificial object”,
“physical object”, “physical substance”, “product”, “object”, “artificial
physical structure”, “architectural structure”, “equipment”, “tool”, “physical
location”, “Wikimedia list article”]. On the right of Figure 6, we visualize
the 3-hop category hierarchy map for the entity ATR 42. Finally, we collect
entities that share any parent nodes in the category hierarchy as knowledge-
similar entities.

We use ViTL/16-224px444https://huggingface.co/google/vit-large-patch16-224
fine-tuned on ImageNet-1k as the visual classifier in vision-hard. While we
collect the entities that share the same prediction label as vision-similar
entities, we filter those entities with its image top-1 confidence score lower
than 0.4, those entities without an infobox image, and those prediction labels
with less than 10 entities. In Figure 7, we pick some entity images that share
the image prediction “Alps” and “Goose”.

##  Appendix 0.C Error Analysis

Figure 5: Error analysis of two query image-question pairs. Left:  two query
image-question pairs. Right:  some of retrieved candidates. While the correct
entity answer is retrieved indicated in green font, the MLLM decides to
produce the wrong entity identifier depicted in red font.

We present two typical types of errors in Figure 5. Although the model is able
to retrieve the correct entity, the MLLM predicts a too specific (but actually
correct as it is indeed a great blue heron) entity in the upper sample. This
is due to the misunderstood of intent by the MLLM, as the query does not ask
for species but category. The lower sample has the MLLM make a wrong
prediction on “Wetsuit” instead of “Diving suit”.

Figure 6: kb-hard group construction illustration. Left: The SPARQL query
template that used for constructing the category hierarchy of entities. We
only consider category labels within 3 hops to exclude general categories like
“object”, “food”, etc. : Right: Visualized 3-hop category hierarchy map for
the entity ATR 42. Code highlight and category visualization are from Wikidata
Graph Builder. Figure 7: Example vision-hard groups constructed with
predictions from ViTL/16-224px.

##  Appendix 0.D More Case Study

We present 7 cases for side-by-side comparison between GPT-4V and AutoVER-7B
in Figure 9. They include situations where AutoVER-7B has an advantage (first
4 rows), GPT-4V outperforms (row 5), both make accurate predictions (row 6)
and both commit mistakes (row 7). We also conclude the reasons behind GPT-4V’s
failures, such as its inability to ground accurately to entities, refusal to
predict with insufficient clues, hallucinatory responses, struggles with
disambiguating challenging entities and lack of specific domain knowledge.

##  Appendix 0.E Qualitative Analysis on A-OKVQA-ENT Dataset

We present a qualitative analysis in Fig. 8 for baselines reported in Sec.
4.3. For all baselines under multi-choice settings except AutoVER, we append
an explicit instruction555“Please strictly follow the options you are given.
Only output the option number or the option itself.” that is omitted in the
figure. Still, LLaVA-v1-7B and OpenFlamingo-9B fail to follow the
instructions, leading to their extremely low evaluation results. Our AutoVER
does not need any prompt engineering as it is guided by a prefix tree which
never allows it to generate answers out of scope. We highlight the keywords
where the model exhibits strong hallucinations in the analysis.

##  Appendix 0.F Current Limitations

Although AutoVER demonstrates superior performance using fewer trainable
parameters compared to PaLI-17B, the accuracy on the unseen subset remains
suboptimal, particularly when compared with the almost identical human
performance on the seen and unseen subsets. We expect future work that focuses
on exploring visual entity recognition for emerging entities that never show
in the training set, and even closing the gap between model and human
performance. In addition, the autoregressive nature makes it challenging to
predict through a single cheap vector retrieval like a bi-encoder. Instead, it
requires multiple forward passes on a billion-size language model. We believe
that efficient inference such as in [21] could be an interesting direction
toward real-time autoregressive visual recognition systems.

Figure 8: Qualitative analysis on AOKVQA-ENT dataset. For each triplet of
image, query and choices, the upper row illustrates the input prompt and
response from different baselines under multi-choice evaluation strategy,
while the lower row is under entity match. Background color in red indicates
an incorrect answer under the specific evaluation setting and that in green
denotes a correct response. Bold highlights the model is heavily
hallucinating.  Figure 9: Case study on comparison between the GPT-4V response
and AutoVER-7B decision. Bold indicates the entity keyword. In the GPT-4V
response, red indicates a false entity keyword and green indicates a correct
one. In retrieved candidates and model decisions, red indicates a false entity
prediction and green denotes a correct prediction. We also include the top-$k$
position of the GPT-4V predicted entity for reference. The answer for the last
query is Matteuccia.


McGaugh The Local Group as seen from the outside looking in

17 2023 10.1017/xxxxx

Proceedings of IAU Symposium 379

# Local Group Galaxies from an External Perspective

McGaugh S.S.1 1 Case Western Reserve University, 10900 Euclid Ave., Cleveland,
OH, USA

###### Abstract

I discuss Local Group galaxies from the perspective of external galaxies that
define benchmark scaling relations. Making use of this information leads to a
model for the Milky Way that includes bumps and wiggles due to spiral arms.
This model reconciles the terminal velocities observed in the interstellar
medium with the rotation curve derived from stars, correctly predicts the
gradual decline of the outer rotation curve, and extrapolates well out to 50
kpc. Rotationally supported Local Group galaxies are in excellent agreement
with the baryonic Tully-Fisher relation. Pressure supported dwarfs that are
the most likely to be in dynamical equilibrium also align with this relation.
Local Group galaxies thus appear to be normal members of the low redshift
galaxy population. There is, however, a serious tension between the dynamical
masses of the Milky Way and M31 and those expected from the stellar mass–halo
mass relation of abundance matching.

###### keywords:

Local Group (929), Milky Way Galaxy (1054), Dwarf galaxies (416), Galaxy
masses (607)

††editors: P. Bonifacio, M.-R. Cioni, F. Hammer, M. Pawlowski, and S. Taibi,
eds.

##  1 Introduction

Local Group galaxies are accessible to detailed study with large samples of
individual stars. This information is complemented by information from
external galaxies, which obey strong kinematic scaling relations (McGaugh et
al., 2020; Lelli, 2022). These include the Baryonic Tully-Fisher Relation
(BTFR: Schombert et al., 2020), the Central Density Relation (Lelli et al.,
2016), and the Radial Acceleration Relation (RAR: McGaugh et al., 2016; Lelli
et al., 2017). Local Group galaxies prove to be completely normal members of
the general galaxy population by these standards.

##  2 The Milky Way

The Milky Way is of obvious interest as our home galaxy, yet our place within
it is a source of constant frustration: we cannot see it from ‘above’ as we do
external spirals. We nevertheless infer that it is a barred spiral thanks to
decades of work employing a broad spectrum of methods (Bland-Hawthorn &
Gerhard, 2016). In external galaxies, spiral features cause bumps and wiggles
in the light profile that correspond to features in the rotation curve
(Sancisi, 2004). This correspondence is quantified by the RAR.

The RAR can be applied to the Milky Way to build a mass model for the stellar
disk (McGaugh, 2016). The terminal velocities observed by McClure-Griffiths &
Dickey (2007) and McClure-Griffiths & Dickey (2016) were fit in detail between
$3<R<8$ kpc. The resulting model is shown in Fig. 1, updated to the distance
scale (McGaugh, 2018) indicated by the GRAVITY Collaboration (2018).

Figure 1: The rotation curve of the Milky Way. Light gray points are the
terminal velocity data (McClure-Griffiths & Dickey, 2007, 2016) to which the
model was fit (McGaugh, 2018, green line). The model was not fit to the data
of Portail et al. (2017) (dark gray squares at $R<2.2$ kpc) or Eilers et al.
(2019) (black squares), but is nicely consistent with them (McGaugh, 2019). It
also extrapolates well out to 50 kpc as traced by K giants (red squares) and
BHB stars (blue circles) (Bird et al., 2022) and by globular clusters (Watkins
et al., 2019, orange triangles).

Fig. 2 shows the surface density corresponding to the rotation curve seen in
Fig. 1. The model has bumps and wiggles in the profile that are reminiscent of
those seen in external spirals. Indeed, these kinematically inferred features
correspond well to spiral arms that are known from traditional tracers (Fig.
2).

The model (McGaugh, 2018) correctly predicted the slope
$dV/dR=-1.7\;\mathrm{km}\,\mathrm{s}^{-1}\,\mathrm{kpc}^{-1}$ of the outer
rotation curve subsequently found by Eilers et al. (2019). It extrapolates
well to data for globular clusters (Watkins et al., 2019) and halo stars (Bird
et al., 2022) at still larger radii, and is nicely consistent with the inner
mass distribution inferred by Portail et al. (2017) without that region having
been specifically fit. This is a testament to how tightly the mass
distribution is coupled to that of the light: the latter is predictive of the
former (McGaugh, 2020).

Table 1: Mass Estimates

Object | Model | $M_{*}$ | $M_{\mathrm{200}}$ | $c$ | $V_{\mathrm{200}}$ | Ref.
---|---|---|---|---|---|---
|  | $10^{10}\;\mathrm{M}_{\odot}$ | $10^{10}\;\mathrm{M}_{\odot}$ |  | $\mathrm{km}\,\mathrm{s}^{-1}$ |
Milky Way | NFW from RAR |  16.2 | 139 |  18.1 | 162 | McGaugh (2018)
M31 | NFW+SSP | 13.5 | 160 | 10.1 | 169 | Chemin et al. (2009)
M31+MW | Straight sum | 19.7 | 299 | … | … | McGaugh & van Dokkum (2021)

The stellar rotation curve derived by Eilers et al. (2019) is formally
discrepant from the terminal velocities (McClure-Griffiths & Dickey, 2007) at
high significance. The model reconciles the two through numerical evaluation
of the tracer term $\partial\ln\nu/\partial\ln r$ in the Jeans equation
(McGaugh, 2019). The apparent discrepancy is an artifact of the usual
assumption of an exponential disk, which does not account for the bumps and
wiggles in $\nu(r)$. Galactic data have reached a level of precision that
these can no longer be neglected.

Figure 2: Spiral structure (left) and surface density profile (right) of the
Milky Way. The bumps and wiggles inferred kinematically correspond well to the
features known from tracers like HII regions (red points) and Giant Molecular
Clouds (blue points) (McGaugh, 2016).

We can query the model for the halo mass it implies (Table 1). To be specific,
I quote $M_{200}$ for the standardization it provides, and the estimate I
obtain is consistent with the range found by others (Wang et al., 2020).
However, we should bear in mind that this mass is a rather notional quantity
that involves an extrapolation to a radius that is not directly constrained by
data using a model (Navarro et al., 1997) that persistently provides a poor
depiction of observational reality.

A more robust quantity is the enclosed mass $M(R<50\;\mathrm{kpc})=44\times
10^{10}\;\mathrm{M}_{\odot}$. This agrees well with halo star (Bird et al.,
2022) and globular cluster (Watkins et al., 2019) data. Going farther out is
complicated by the perturbing influence of the LMC. Indeed, if we take
seriously the downturn in the Gaia-derived rotation curve at $R\approx
19\;\mathrm{kpc}$ (Wang et al., 2023), a much lower mass is obtained (Sylos
Labini et al., 2023). It is difficult to envision a model that is consistent
with both this downturn and the data for halo stars.

Even in the absence of these considerations, the total mass of the Milky Way
remains a fraught issue. Weak lensing observations of external galaxies
(Brouwer et al., 2021) indicate that rotation curves persist in being
approximately flat to hundreds of kpc, reaching beyond the expected virial
radii of bright spirals. This is not consistent with an NFW profile, but it is
consistent with the RAR, which is what works well above. These are not the
same.

##  3 The Local Group

The Milky Way falls on the BTFR defined by external galaxies (Fig. 3). Indeed,
all of the rotationally supported Local Group galaxies do so, within the
uncertainties: there is no indication that they are abnormal in this regard.
This provides both a constraint — nothing particularly peculiar is permitted —
and an opportunity: if we can calibrate the dynamical masses of Local Group
galaxies, we can infer those of more distant galaxies.

###  3.1 Dwarf Spheroidal Galaxies and Tully-Fisher

The BTFR may further serve as a method to unite the mass scales of pressure
supported early type dwarf galaxies with their rotationally supported late
type brethren. Typically we measure only a single, bulk velocity dispersion
$\sigma_{*}$ for dwarf spheroidals. The dwarf spheroidals of the Local Group
fall into two populations in the BTFR plane (McGaugh & Wolf, 2010): the
$L$-$\sigma_{*}$ data for classical dwarfs parallels the BTFR, while for the
ultrafaint dwarfs $\sigma_{*}$ is uncorrelated with $L$, seeming to hit a
floor. As a consequence of their relatively high velocity dispersions, the
latter appear to be very dense and hence safe from tidal influences (Pace et
al., 2022). This may be an illusion of the assumption of dynamical
equilibrium, as it seems unlikely that these tiny dwarfs can survive
unmolested deep in the potential well of the Milky Way while the Milky Way
itself is being strongly perturbed by the LMC. Excepting the (primarily
ultrafaint) dwarfs that may be subject to tidal influence by the criterion of
Bellazzini et al. (1996), the remaining dwarfs of the Local Group fall nicely
on the BTFR defined by rotating galaxies given a simple transformation
$M_{*}=\Upsilon_{*}L_{V}$ and $V_{f}=\beta_{c}\sigma_{*}$ with
$\log\beta_{c}=0.25\log\Upsilon_{*}+0.226$ (McGaugh et al., 2021). This works
out to $\beta_{c}=2$ for
$\Upsilon_{*}=2\;\mathrm{M}_{\odot}/\mathrm{L}_{\odot}$. This quantifies the
circular velocity that is equivalent to the line of sight velocity dispersion
in an average sense. The strong regularity seen in Fig. 3 is surely teaching
us something about the dynamical masses of galaxies.

Figure 3: The baryonic Tully–Fisher relation of Local Group galaxies (blue
circles) follows that defined by external galaxies (grey points Schombert et
al., 2020). This extrapolates to pressure supported dwarfs (squares) when
$V_{f}=2\sigma_{*}$ (McGaugh et al., 2021). Red squares are satellites of M31,
orange squares are satellites of the Milky Way; green squares are unaffiliated
(McConnachie et al., 2021).

###  3.2 Kinematic Masses and Abundance Matching

A common method for relating the observed stellar masses of galaxies with
their host dark matter halos is Abundance Matching (AM: Behroozi et al., 2013;
Moster et al., 2013; Kravtsov et al., 2018; Mowla et al., 2019). Indeed, this
has become an essential ingredient for understanding galaxies in $\Lambda$CDM.
The Local Group provides an ideal opportunity to test these relations with
independent kinematic data.

Typical AM relations are well calibrated around the knee of the luminosity
function, but not to the very low masses of dwarf spheroidals (Bullock &
Boylan-Kolchin, 2017). The agreement between kinematic mass estimates and AM
masses is reasonably good for intermediate mass galaxies like M33. The two
diverge dramatically for the brightest galaxies of the Local Group, M31 and
the Milky Way (Table 1 and Fig. 4)

Figure 4: Stellar mass–halo mass relations in the context of the Local Group
(McGaugh & van Dokkum, 2021). Filled points are from rotation curve fits to
the Milky Way (McGaugh, 2018), M31 (Chemin et al., 2009), and their sum (Table
1). The open circle adopts for the summed total mass that estimated by Shaya
et al. (2017). The blue band shows range of Milky Way mass estimates (Bland-
Hawthorn & Gerhard, 2016; Wang et al., 2020). Lines are the stellar mass–halo
mass relations from abundance matching Behroozi et al. (2013); Moster et al.
(2013); Kravtsov et al. (2018); Mowla et al. (2019). The yellow box shows the
timing mass of the Local Group (van der Marel et al., 2012). There is a
pronounced discrepancy between dynamical and abundance matching estimates of
the halo masses of the Local Group and its brightest members.

Dynamical studies typically suggest that the total mass of the Milky Way is in
the range of 0.5 — $2\times 10^{12}\;\mathrm{M}_{\odot}$ (Bland-Hawthorn &
Gerhard, 2016; Wang et al., 2020). A similar range holds for Andromeda (Chemin
et al., 2009; Kafle et al., 2018). AM predicts considerably higher masses:
$\sim 3\times 10^{12}\;\mathrm{M}_{\odot}$ for the Milky Way and
$>10^{13}\;\mathrm{M}_{\odot}$ for M31. There is some room to hedge this for
the Milky Way by adopting a stellar mass on the low end of the plausible
range, but even then the halo mass is abnormally low. The result for Andromeda
differs by an order of magnitude. There is of course some scatter in the
stellar mass–halo mass relation from AM, but it is not sufficient to explain
this stark discrepancy.

According to AM, a galaxy the mass of M31 should be the central object of a
much larger group. Indeed, for its stellar mass, the Milky Way should be the
central object of the Local Group. Andromeda should not be here. Despite its
normalcy in other respects, the Local Group has far too many stars for its
mass from the perspective of abundance matching.

It is not clear what the resolution of this issue might be. There could be a
morphological type dependence that is not accounted for in the AM relations
considered here. There might be more mass at large radii than kinematic
tracers have so far revealed, though this is limited by the timing argument
(van der Marel et al., 2012; Peñarrubia et al., 2014; Shaya et al., 2017). The
discrepancy appears to be real, and would hardly be the first indication of a
problem with the dark matter paradigm in the dynamics of galaxies (Sanders &
McGaugh, 2002; Famaey & McGaugh, 2012; McGaugh, 2020; Banik & Zhao, 2022).

## References

  * Banik & Zhao (2022) Banik, I. & Zhao, H. 2022, Symmetry, 14, 1331
  * Behroozi et al. (2013) Behroozi, P. S., Wechsler, R. H., & Conroy, C. 2013, ApJ, 762, L31
  * Bellazzini et al. (1996) Bellazzini, M., Fusi Pecci, F., & Ferraro, F. R. 1996, MNRAS, 278, 947
  * Bird et al. (2022) Bird, S. A., Xue, X.-X., Liu, C., et al. 2022, MNRAS, 516, 731
  * Bland-Hawthorn & Gerhard (2016) Bland-Hawthorn, J. & Gerhard, O. 2016, ARA&A, 54, 529
  * Brouwer et al. (2021) Brouwer, M. M., Oman, K. A., Valentijn, E. A., et al. 2021, A&A, 650, A113
  * Bullock & Boylan-Kolchin (2017) Bullock, J. S. & Boylan-Kolchin, M. 2017, ARA&A, 55, 343
  * Chemin et al. (2009) Chemin, L., Carignan, C., & Foster, T. 2009, ApJ, 705, 1395
  * Eilers et al. (2019) Eilers, A.-C., Hogg, D. W., Rix, H.-W., et al. 2019, ApJ, 871, 120
  * Famaey & McGaugh (2012) Famaey, B. & McGaugh, S. S. 2012, Living Reviews in Relativity, 15, 10
  * GRAVITY Collaboration (2018) GRAVITY Collaboration, Abuter, R., Amorim, A., et al. 2018, A&A, 615, L15
  * Kafle et al. (2018) Kafle, P. R., Sharma, S., Lewis, G. F., et al. 2018, MNRAS, 475, 4043
  * Kravtsov et al. (2018) Kravtsov, A. V., Vikhlinin, A. A., & Meshcheryakov, A. V. 2018, Astronomy Letters, 44, 8
  * Lelli et al. (2016) Lelli, F., McGaugh, S. S., Schombert, J. M., et al. 2016, ApJ, 827, L19
  * Lelli et al. (2017) Lelli, F., McGaugh, S. S., Schombert, J. M., et al. 2017, ApJ, 836, 152
  * Lelli (2022) Lelli, F. 2022, Nature Astronomy, 6, 35
  * McClure-Griffiths & Dickey (2007) McClure-Griffiths, N. M. & Dickey, J. M. 2007, ApJ, 671, 427
  * McClure-Griffiths & Dickey (2016) McClure-Griffiths, N. M. & Dickey, J. M. 2016, ApJ, 831, 124
  * McConnachie et al. (2021) McConnachie, A. W., Higgs, C. R., Thomas, G. F., et al. 2021, MNRAS, 501, 2363
  * McGaugh (2016) McGaugh, S. S. 2016, Astrophys. J., 816, 42
  * McGaugh (2018) McGaugh, S. S. 2018, Research Notes of the American Astronomical Society, 2, 156
  * McGaugh (2019) McGaugh, S. S. 2019, Astrophys. J., 885, 87
  * McGaugh (2020) McGaugh, S. 2020, Galaxies, 8, 35
  * McGaugh et al. (2020) McGaugh, S., Lelli, F., Li, P., et al. 2020, Galactic Dynamics in the Era of Large Surveys, 353, 144
  * McGaugh et al. (2016) McGaugh, S. S., Lelli, F., & Schombert, J. M. 2016, Phys. Rev. Lett., 117, 201101
  * McGaugh et al. (2021) McGaugh, S. S., Lelli, F., Schombert, J. M., et al. 2021, Astron. J., 162, 202
  * McGaugh & van Dokkum (2021) McGaugh, S. S. & van Dokkum, P. 2021, Research Notes of the American Astronomical Society, 5, 23
  * McGaugh & Wolf (2010) McGaugh, S. S. & Wolf, J. 2010, ApJ, 722, 248
  * Milgrom (1983) Milgrom, M. 1983, ApJ, 270, 371
  * Moster et al. (2013) Moster, B. P., Naab, T., & White, S. D. M. 2013, MNRAS, 428, 3121
  * Mowla et al. (2019) Mowla, L., van der Wel, A., van Dokkum, P., et al. 2019, ApJ, 872, L13
  * Navarro et al. (1997) Navarro, J. F., Frenk, C. S., & White, S. D. M. 1997, ApJ, 490, 493
  * Pace et al. (2022) Pace, A. B., Erkal, D., & Li, T. S. 2022, ApJ, 940, 136
  * Peñarrubia et al. (2014) Peñarrubia, J., Ma, Y.-Z., Walker, M. G., et al. 2014, MNRAS, 443, 2204
  * Portail et al. (2017) Portail, M., Gerhard, O., Wegg, C., et al. 2017, MNRAS, 465, 1621
  * Sancisi (2004) Sancisi, R. 2004, Dark Matter in Galaxies, 220, 233
  * Sanders & McGaugh (2002) Sanders, R. H. & McGaugh, S. S. 2002, ARA&A, 40, 263
  * Schombert et al. (2020) Schombert, J., McGaugh, S., & Lelli, F. 2020, AJ, 160, 71
  * Shaya et al. (2017) Shaya, E. J., Tully, R. B., Hoffman, Y., et al. 2017, ApJ, 850, 207
  * Sylos Labini et al. (2023) Sylos Labini, F., Chrobáková, Ž., Capuzzo-Dolcetta, R., et al. 2023, ApJ, 945, 3
  * van der Marel et al. (2012) van der Marel, R. P., Fardal, M., Besla, G., et al. 2012, ApJ, 753, 8
  * Wang et al. (2020) Wang, W., Han, J., Cautun, M., et al. 2020, Science China Physics, Mechanics, and Astronomy, 63, 109801
  * Wang et al. (2023) Wang, H.-F., Chrobáková, Ž., López-Corredoira, M., et al. 2023, ApJ, 942, 12
  * Watkins et al. (2019) Watkins, L. L., van der Marel, R. P., Sohn, S. T., et al. 2019, ApJ, 873, 118

MüllerWhat is the reason for these strong regularities?

McGaughThis behavior was predicted by MOND (Milgrom, 1983).