# The Pyramid of Captions

Delong Chen  HKUSTClear Water BayHong Kong<EMAIL_ADDRESS>,
Samuel Cahyawijaya  HKUSTClear Water BayHong Kong<EMAIL_ADDRESS>,  Etsuko Ishii  HKUSTClear Water BayHong Kong<EMAIL_ADDRESS>,  Ho Shu
Chan  HKUSTClear Water BayHong Kong<EMAIL_ADDRESS>,  Yejin Bang
HKUSTClear Water BayHong Kong<EMAIL_ADDRESS>and  Pascale Fung
HKUSTClear Water BayHong Kong<EMAIL_ADDRESS>
###### Abstract.

We introduce a formal information-theoretic framework for image captioning by
regarding it as a representation learning task. Our framework defines three
key objectives: task sufficiency, minimal redundancy, and human
interpretability. Building upon this foundation, we propose a novel Pyramid of
Captions (PoCa) method, which constructs caption pyramids by generating
localized captions for zoomed-in image patches and integrating them with
global caption information using large language models. This approach
leverages intuition that the detailed examination of local patches can reduce
error risks and address inaccuracies in global captions, either by correcting
the hallucination or adding missing details. Based on our theoretical
framework, we formalize this intuition and provide formal proof demonstrating
the effectiveness of PoCa under certain assumptions. Empirical tests with
various image captioning models and large language models show that PoCa
consistently yields more informative and semantically aligned captions,
maintaining brevity and interpretability.

Image Captioning, Visual Question Answering, Large Vision Language Models

††copyright: none††conference: Make sure to enter the correct conference title
from your rights confirmation email; under review; 2024††ccs: Computing
methodologies Computer vision††ccs: Computing methodologies Natural language
generation

##  1\. Introduction

Image captioning translates the informational content of the images into
natural language, serving pivotal roles in multimedia applications ranging
from assisting visually impaired individuals (gurari2020captioning, ;
rane2021image, ; guo2022offline, ; ganesan2022deep, ) to facilitating content-
based image retrieval (gudivada1995content, ; srihari1995automatic, ;
datta2005content, ; da2006content, ; jain2015content, ; li2021recent, ). Over
the last decade, the field of image captioning has witnessed substantial
progress, primarily driven by advancements in deep neural nets and the
availability of large-scale high-quality image-text datasets.

Despite these achievements, several fundamental questions remain unanswered
–What makes for a good image caption? Which properties should it possess, and
how can we measure them? Some existing models can generate captions closely
resembling single-sentence human annotations (wang2022ofa, ; wang2022git, ;
hu2023promptcap, ; dai2024instructblip, ), but these may not be adequate for
use cases where more comprehensive coverage of fine-grained visual information
is required. Conversely, recent Large Vision Language Models (LVLMs)
(li2022blip, ; alayrac2022flamingo, ; wang2022ofa, ; liu2023visual, ;
li2023blip, ; dai2024instructblip, ) have demonstrated the ability to generate
coherent, multi-paragraph image descriptions (urbanek2023picture, ;
cho2023davidsonian, ; liu2023visual, ; wang2024asm, ). Yet, longer captions
can sometimes be less accurate, hallucinate content, or putting excessive
emphasis on irrelevant details while omitting important information.

Recognizing the absence of a universal standard for ideal captions, this work
first aims to establish well-defined principles for image captioning that
address varying task requirements. We introduce an information-theoretic
framework based on semantic communication (Peyrard2019Simple, ;
Zhong2017Theory, ) and the information bottleneck principle
(Tishby2000information, ; ShwartzZiv2017Opening, ; Tsai2021Self, ), and define
an objective for image captioning models that balances task sufficiency,
minimal redundancy, and human interpretability. Varying the weighting
coefficients of these terms suits different preferences over image captions
(e.g., comprehensiveness, succinctness, readability), providing a rigorous
foundation for subsequent analysis and comparisons.

Moreover, the research focus has been shifting toward generating more detailed
captions, which emphasize detailed information coverage over brevity
(krause2017hierarchical, ; liu2023visual, ; chen2023sharegpt4v, ;
urbanek2023picture, ; cho2023davidsonian, ; wang2024asm, ). However, despite
this shift, current models struggle to capture intricate visual details due to
the predominance of datasets geared toward shorter captions. To address this
challenge, we propose the Pyramid of Captions (PoCa), a simple yet effective
image captioning approach that employs a hierarchical method to generate
content-rich captions by exploiting the complementary nature of local and
global visual cues and recursively merging local global information. PoCa
utilizes state-of-the-art Large Language Models (LLMs) to fuse the information
in local and global captions, resulting in improved captions that are both
semantically precise and contextually coherent. Leveraging our previously
established theoretical framework, we provide formal proof demonstrating that
each local-global aggregation operation in PoCa is guaranteed to improve
caption quality.

Through systematic empirical evaluations, we demonstrate that PoCa effectively
generates accurate and informative detailed image captions. As measured by
caption-based VQA inference with text-only LLMs on the VQA-v2 dataset
(GoyalKSBP17, ), PoCa captions cover more semantic information that is useful
for answering questions. We also conduct an evaluation using the image
paragraph captioning dataset (krause2017hierarchical, ), and show that PoCa
brings performance improvement on both reference-based and reference-free
metrics. Our PoCa method is generally applicable to both long-form and short-
form captions and different models. It works well with a wide range of LVLMs,
ranging from 1.7B MobileVLM-v2 (chu2024mobilevlm, ) to the state-of-the-art
InternVL (chen2023internvl, ) with 40B parameters, also with different scales
of LLMs for caption merging, ranging from LlaMa2-7B (touvron2023llama, ) to
ChatGPT-3.5 (bang-etal-2023-multitask, ) level Mixtral-8x7B (jiang2024mixtral,
). In summary, the contributions of this work are as follows:

  * •

We propose an information-theoretic framework for image captioning that
defines the objectives of task sufficiency, minimal redundancy, and human
interpretability;

  * •

We introduce the Pyramid of Captions (PoCa) which leverages a multi-scale
hierarchical approach for generating semantically rich captions;

  * •

We prove that PoCa generates higher-quality captions, both theoretically and
empirically.

##  2\. Theoretical Framework

Figure 1. Overview of our formulation. An image $\tilde{X}$ in data space
$\mathcal{D}$ is generated from some latent variable $X$ in a latent semantic
space $\mathcal{S}$. $\tilde{X}$ is captioned by $f_{\theta}$ producing a
caption $\tilde{Y}$ which can be mapped back to the original latent space as
$Y$. The semantic-level error $Z=Y-X$ measures the difference between the
source semantics $X$ and received semantics $Y$.

In this section, we provide a theoretical framework for image captioning as
depicted in Figure 1. First, we formulate the task of image captioning by
applying the concept of semantic units (Peyrard2019Simple, ; Zhong2017Theory,
). We suppose that an image is an observation of a latent variable in a
semantic space characterized by semantic units. An image captioning model will
generate a caption for the image, and the caption can be mapped back to the
latent semantic space and compared with the source semantic latent variable.

Based on this framework, we then introduce our proposed objectives inspired by
the information bottleneck principle (Tishby2000information, ;
ShwartzZiv2017Opening, ; Tsai2021Self, ) for feature representation learning
(Tsai2021Self, ). In our framework, we consider that the overall image
captioning objective composed of a “task sufficiency” term, a “minimal
redundancy” term, and a “human interpretability” term.

###  2.1. Formulation of Image Captioning

We assume that meaning arises from a set of independent and discrete units
called semantic units in a semantic space, and images and captions are
observations of some latent variables in this semantic space
(Peyrard2019Simple, ; Zhong2017Theory, ). Following (Peyrard2019Simple, ;
Zhong2017Theory, ), we define a semantic unit and the semantic space as:

###### Definition 0 (Semantic Units and Semantic Space).

A semantic unit represents an atomic piece of information. The set of all
possible semantic units is denoted by $\Omega=\\{w_{i}\\}_{i=1}^{n}$. A
semantic space is an $n$-dimensional space $\mathcal{S}\in[0,1]^{n}$ where the
value of the $i$-th dimension of represents the probability of the presence of
the corresponding semantic unit $p(w_{i})$.

The $\Omega$ encompasses a wide range of semantic units, similar to how a
vocabulary contains diverse words. Each point in $\mathcal{S}$ corresponds to
a specific combination of semantic units, corresponding to different meanings.
Adopting the semantic space and its probabilistic interpretation in
(Peyrard2019Simple, ; Shao2022Theory, ; Niu2024Mathematical, ), we can apply
the classical information theory (Shannon1948mathematical, ) to operate at the
semantic level, and make information in images and captions to be comparable.

Let a random variable $X\in\mathcal{S}$ and $Y\in\mathcal{S}$ represent
semantic information in an image and a caption, where $X_{i}=p(w_{i})$ and
$Y_{i}=p(w_{i})$ represent the likelihood of a semantic unit $w_{i}$ observed
in an image $\tilde{X}\in\mathcal{D_{\mathrm{image}}}$ or a caption
$\tilde{Y}\in\mathcal{D_{\mathrm{caption}}}$. These latent variables $X$ and
$Y$ encode all the information within real images
$\tilde{X}\in\mathcal{D_{\mathrm{image}}}$ and textual captions
$\tilde{Y}\in\mathcal{D_{\mathrm{caption}}}$, both low-level and high-level
semantics. Then, image captioning can be framed as follows:

###### Definition 0 (Image Captioning).

An image captioning model $f$ parameterized by $\theta$ operates in the
observed data spaces
$f_{\theta}:\mathcal{D_{\mathrm{image}}}\rightarrow\mathcal{D_{\mathrm{caption}}}$,
it translates an image into a caption, i.e.,
$\tilde{Y}=f_{\theta}(\tilde{X})$. $\tilde{X}$ is generated from some source
latent variable $\tilde{X}=g(X)$ while $\tilde{Y}$ can be converted back to
the latent semantic space $Y=h(\tilde{Y})$. Let $Z=Y-X\in[-1,1]^{n}$ denote
the error between the source and recovered semantics caused by parameters
$\theta$.

Here, $Z$ can be associated with which kind of error $\tilde{Y}$ has; negative
$Z$ indicates that the caption misses some contents of the image (omission),
and positive $Z$ indicates that the caption includes something that is not in
the image (hallucination).

###  2.2. Objectives of Image Captioning

The image captioning process can be compared to a communication system
(Shannon1948mathematical, ) where information source $X$ is converted to
signal $\tilde{X}$ by a lossless source encoder $g$, transmitted through a
noisy channel $f_{\theta}$, and the received signal $\tilde{Y}$ is losslessly
decoded by $h$, giving the final received information $Y$. From this
communication system perspective, one might say that the optimal $\theta^{*}$
could just be the one that minimizes the error $||Z||$. However, this
requirement is unrealistic as it would result in extremely long captions that
losslessly encode both high-level semantic information and all the low-level
irrelevant information.111 Consider an image with $224\times 224$ resolution
in 8-bit RGB color space that has $224\times 224\times 3\times 8=1,204,224$
bits. Given the estimated entropy of English is 0.6 to 1.3 bits per character
(Shannon1951Prediction, ), and around 6.5 characters per word on average, the
lossless caption of the image need to be at least 142k words–longer than a
typical PhD thesis.

Therefore, we apply the information bottleneck principle
(Tishby2000information, ; ShwartzZiv2017Opening, ; Tsai2021Self, ) to evaluate
the system. Information bottleneck is a generalization of rate-distortion
theory for lossy data compression, it requires a representation (which in our
case is $\tilde{Y}$) to have maximal mutual information with some information
$T$ that is required fulfill the task requirements (i.e., high task
sufficiency), while having minimal mutual information with the input $X$
(minimal redundancy). The desired minimal sufficient representation can be
given as $\tilde{Y}^{*}=\arg\max I(\tilde{Y};T)-\beta I(X;\tilde{Y})$ where
$\beta$ is a Lagrange multiplier. If the captioning model is deterministic
(given $X$, it always produces the same $\tilde{Y}$) then we have
$H(\tilde{Y}|X)=0$. Since $I(X;\tilde{Y})=H(\tilde{Y})-H(\tilde{Y}|X)$, the
minimal sufficient representation can be written as:

(1) $\tilde{Y}^{*}=\arg\underset{\tilde{Y}}{\max}\ I(\tilde{Y};T)-\beta
H(\tilde{Y}).$

The second term will penalize the captioning model when it generates an over-
length caption and the value of $\beta$ controls the penalty strength.
Combined with the first term, the objective requires the model to preserve as
much useful information as possible for the task, while keeping the captions
as succinct as possible.

Next, we give a formal definition of the task sufficiency objective of image
captioning with importance of semantic units (Peyrard2019Simple, ).

###### Definition 0 (Task Sufficiency).

For given $X$, let a latent variable $T\in\mathcal{S}$ represent the task-
relevant information in $X$, and let an importance variable $A\in[0,1]^{n}$
denote the importance scores of different semantic units. The $A$ is derived
from $X$ by an underlying mapping, thus being dependent on $X$. The $T$ is
produced by a point-wise product between $A$ and $X$, thus $T=A\odot X$. For
generated image captions $\tilde{Y}=f_{\theta}(\tilde{X})$, the task
sufficiency objective is:

(2) $J_{\mathrm{suf}}(\theta)=I(\tilde{Y};A\odot X)$

In the importance variable, $A_{i}=1$ means the semantic units $w_{i}$ are
very important in the image, while $A_{i}=0$ means $w_{i}$ is irrelevant. It
behaves similarly to the attention mechanism (Bahdanau2015Neural, ), which
also produces a heatmap between zero to one according to the given input. Note
that here the $A$ is not binary, the continual nature of $A$ gives a good
property to $J_{\mathrm{suf}}(\theta)$: when the “budget” is limited (as there
are also other objectives to optimize), more semantic units with higher
importance score will be retained while less important ones will be discarded.

###### Definition 0 (Minimal Redundancy).

The minimal redundancy objective encourages the image captioning model
$f_{\theta}$ to eliminate irrelevant information, it is given by measuring the
entropy of generated captions:

(3) $J_{\mathrm{min}}(\theta)=-H(\tilde{Y}).$

Combining $J_{\mathrm{suf}}(\theta)$ and $J_{\mathrm{min}}(\theta)$ ensures
the captions are minimal sufficient representations of images. However, there
is no guarantee that the generated captions can be understood by humans.
Therefore, we need to measure the human interpretability of the captions using
a third objective, which is the distributional similarity between
$Y\mathrm{data}$ and natural language.

###### Definition 0 (Human Interpretability).

Let $P_{\tilde{Y}}$ denote the probabilistic distribution of model-generated
captions over $\mathcal{D_{\mathrm{caption}}}$, and let $P_{\mathrm{lang}}$
denote the distribution of human interpretable natural language. Given a
certain statistical divergence measurement $D$, the human interpretability
objective is:

(4) $J_{\mathrm{int}}(\theta)=-D(P_{\tilde{Y}}||P_{\mathrm{lang}}).$

The overall objective of image captioning is a weighted combination of task
sufficiency, minimal redundancy, and human interpretability:

(5) $J(\theta)=J_{\mathrm{suf}}(\theta)-\beta J_{\mathrm{min}}(\theta)-\gamma
J_{\mathrm{int}}(\theta),$

where $\beta>0$ and $\gamma>0$ are weighting coefficients. Here, one of the
factors that the coefficient $\beta$ in Eq. 5 controls is the length of
generated captions. If we prefer more detailed, comprehensive captions, we
have smaller $\beta$ and larger $\gamma$.

##  3\. The Pyramid of Captions

Figure 2. Global caption tends to only capture the broad overview of the whole
image. Local caption alleviates this limitation by providing a fine-grained
information which can further complement the global caption.

###  3.1. Methodology

In this section, we introduce the Pyramid of Captions (PoCa) method, which
merges local and global information recursively. The key intuition behind the
PoCa method is that we can have a more accurate and detailed caption by
ensembling multiple captions, as shown in Figure 2. We propose to split an
image into multiple local patches generate local captions for each patch, and
fuse the captions to obtain a higher-quality caption for the global image.

Formally, let $\sigma_{\mathrm{split}}$ be a function operating in
$\mathcal{D}_{\mathrm{image}}$ that represents the splitting function, which
splits an image into a set of local patches:

(6)
$\sigma_{\mathrm{split}}(\tilde{X})=\left\\{\tilde{X}^{[j]}\right\\}_{j=1}^{m}.$

We apply an image captioning model $f_{\theta}$ to the local patches and
obtain a set of local captions:
$\\{\tilde{Y}^{[j]}=f_{\theta}(\tilde{X}^{[j]})\\}_{j=1}^{m}$, and also
generate a caption for the global image: $\tilde{Y}=f_{\theta}(\tilde{X})$.
The local and global information will be fused by a merging function
$\sigma_{\mathrm{merge}}$ operating in $\mathcal{D}_{\mathrm{caption}}$:

(7)
$\tilde{Y}_{\mathrm{merged}}=\sigma_{\mathrm{merge}}\left(\tilde{Y};\\{\tilde{Y}^{[j]}\\}_{j=1}^{m}\right).$

We adopt text-only LLMs as the merging function $\sigma_{\mathrm{merge}}$.
Table 1 provides an example of merging using an LLM, where we instruct it to
generate a merged caption that incorporates both local and global information.

As the PoCa method is hierarchical, we can extend it to be more layers. We can
recursively split a patch into sub-patches and merge captions for each sub-
patches to represent the patch.

Prompt for Merging Caption Pyramid System Message: Input: $\quad\bullet\quad$
You will receive a global caption describing an image. $\quad\bullet\quad$
Additionally, you will have access to local captions generated for specific
patches within the image. $\quad\bullet\quad$ Both global and local captions
may contain noise or errors. Task Objective: $\quad\bullet\quad$ Your goal is
to create a merged global caption that combines relevant information from both
sources. $\quad\bullet\quad$ The merged caption should be no longer than the
original ones. $\quad\bullet\quad$ You only give the merged caption as output,
without any additional information. $\quad\bullet\quad$ Do NOT give any
explaination or notes on how you generate this caption. Guidelines:
$\quad\bullet\quad$ Combine Information: Extract key details from both global
and local captions. $\quad\bullet\quad$ Filter Noise: Remove non-sense
content, inaccuracies, and irrelevant information. $\quad\bullet\quad$
Prioritize Visual Details: Highlight essential visual elements instead of
feeling or atmosphere $\quad\bullet\quad$ Be Concise: Use as few words as
possible while maintaining coherence and clarity. $\quad\bullet\quad$ Ensure
Coherence: Arrange the merged information logically. Remember, your output
should be a high-quality caption that is concise, informative, and coherent!
User: ### Global Caption: {global caption} ### Top-left: {top-left caption}
### Bottom-left: {bottom-left caption} ### Top-right: {top-right caption} ###
Bottom-left: {bottom-left caption} Assistant Generation Prefix: Here’s the
merged caption:

Table 1. An Example implementation of the merging function
$\sigma_{\mathrm{merge}}$ based on prompting text-only LLMs.

###  3.2. PoCa Gives Better Captions, Provably

In this section, we provide an analysis of under what condition a single
local-global merging operation in PoCa can be guaranteed to improve the
caption quality.

First, we assume that there is some function $\varphi$ to quantify the error
$Z$ by $X$ in a deterministic manner, and that function is concave. The
deterministic assumption of $\varphi$ simplifies the analysis by ignoring
errors caused by factors other than the input semantics $X$, such as the
randomness in sampling-based autoregressive generation. In other words, we
assume that $f_{\theta}$ always generates the same caption and makes the same
error for the same input. The concavity of $\varphi$ implies that it generates
a larger volume of error when $p(w_{i})$ is far from zero and one. This means
that the captioning model is more likely to make mistakes when there is high
uncertainty about the presence or absence of a semantic unit in the image.

######  Assumption 1 (Uncertainty-aware content-dependent error).

The error $Z$ produced by the image captioning model $f_{\theta}$ is dependent
on the input semantics $X$. Therefore, it can be expressed as a deterministic
function of $X$:

(8) $||Z_{i}||=\varphi(X_{i}),$

where $\varphi$ is a concave function and $1\leq i\leq n$.

Next, we introduce our assumptions on the image splitting function
$\sigma_{\mathrm{split}}$ and caption merging function
$\sigma_{\mathrm{merge}}$. These assumptions simplify the relationship between
local and global semantics by assuming linear combinations. In practice, the
relationship may be more complex and depend on factors such as the spatial
arrangement of the local patches and the presence of objects spanning multiple
patches.

######  Assumption 2 (Local-global relationship of image semantics).

The $\sigma_{\mathrm{split}}$ splits image into local patches. The latent
semantic variables corresponding to local patches satisfy the following
relationship with global semantics:

(9) $X=\sum_{j}^{m}\alpha_{j}X^{[j]},$

where the weights $\alpha_{j}$ satisfying $\sum_{j}^{m}\alpha_{j}=1$.

######  Assumption 3 (Local-global aggregation of caption semantics).

The function $\sigma_{\mathrm{merge}}$ merges the global and global captions.
The latent semantic variable corresponding to the merged caption is a weighted
combination of the global and local semantics:

(10) $Y_{\mathrm{merged}}=\eta Y+(1-\eta)\sum_{j}^{m}\alpha_{j}Y^{[j]},$

where $\eta\in(0,1)$ is a weighting coefficient.

We now present a theorem regarding $Z_{\mathrm{merged}}=Y_{\mathrm{merged}}-X$
(the proof can be found in the Appendix):

###### Theorem 1 (PoCa method reduces semantic error).

Under Assumptions 1-3, the PoCa method is guaranteed to have the smaller error
$Z_{\mathrm{merged}}$ than $Z$, i.e.,

(11) $||Z_{\mathrm{merged}}||\leq||Z||.$

Since $A$ is non negative, Theorem 1 implies non-decreasing task sufficiency;
if merging does not increase redundancy or decrease interpretability, then the
overall quality of caption becomes better. This also aligns with the findings
in (shi2024we, ) that smaller scale models combined can be as effective as a
larger scale model. Additionally, it worth noting that our assumptions require
linear combinations of semantics, while it may not hold in practice for images
with more complex structure of semantics.

##  4\. Experiments

###  4.1. Implementation Details

####  4.1.1. Image Captioning Models

We employ three groups of Large Vision Language Models (LVLMs) as the image
captioning models: LLaVA-1.5 (DBLP:journals/corr/abs-2310-03744, ), MobileVLM
v2 (chu2024mobilevlm, ), and InternVL (chen2023internvl, ), among them:

  * •

LLaVA-1.5 series is a popular LVLM with two variants: LLaVA-1.5-7B and
LLaVA-1.5-13B, which adopt Vicuna-7B and Vicuna-13B as their Language Models
(LLMs), respectively.

  * •

MobileVLM v2 is a family of efficient LVLMs with smaller scales, and we
utilize its MobileVLM-v2-1.7B and MobileVLM-v2-3B models.

  * •

InternVL is one of the top-performing publicly available LVLMs. We use its
InternVL-Chat-Chinese-V1-2-Plus model, which is based on the Yi-34B LLM and
has a total of 40.1B parameters.

All inference is performed in FP16 precision on a single NVIDIA A800 GPU. We
employ two types of prompts for short-form single-sentence image captioning
and long-form detailed image captioning: "Provide a one-sentence caption for
the provided image" and "Describe this image in detail". All generation
parameters are set to the default values provided by the source repository.

####  4.1.2. Caption Pyramids

For the caption merging function $\sigma_{\mathrm{merge}}$, we adopt and
compare a variety of Large Language Models (LLMs) as its implementation,
including the Gemma family (2B and 7B versions), LLaMA2 family (7B chat and
13B chat), Qwen-1.5-7B Chat, Mistral 7B, and a mixture-of-expert model Mixtral
8x7B. The Mixtral 8x7B model has capabilities similar to ChatGPT-3.5 and is
one of the top-performing open-source LLMs. All inference is performed in FP16
precision, except for the large Mixtral 8x7B model, for which we use 8-bit
quantization to fit it into a single NVIDIA A800 GPU. The Mixtral 8x7B is used
as the default LLM for caption merging.

We employ the prompt shown in Table 1 for caption merging, where the
"Assistant Generation Prefix" is injected after the instructions to control
the model output format. All generation parameters are set to the default
values provided by the source repository. For splitting function
$\sigma_{\mathrm{merge}}$, we adopt the most straightforward implementation by
splitting input image into four equal-sized patches.

###  4.2. PoCa Improves Task Sufficiency

####  4.2.1. VQA-based Evaluation

In this section, we conduct quantitative evaluations to study whether the PoCa
method is able to improve the task sufficiency term in the objective
(Definition 3). We adopt the VQA-v2 (GoyalKSBP17, ) dataset, which is built
upon the MS-COCO (ChenFLVGDZ15, ) dataset and contains multiple questions per
image.

We employ a text-only LLM to answer the questions in VQA-v2 based on the
generated captions, using the instruction shown in Table 2. The questions
serve as a proxy for the importance score $A$ in Definition 3, and the
accuracy of the LLM-generated correct answers becomes an estimation of the
task sufficiency term. We use LLaMA2-Chat-13B for VQA inference; a detailed
analysis of different LLMs for VQA evaluation will be provided in Section
4.3.3.

Prompt for LLM-based VQA Evaluation System Message: You will be given a
caption of an image, and your task is to try to answer the question based on
the caption. If the relevant information is not present in the caption, try
your best to guess the answer. You shouldn’t provide any rationale or
explaination in your response, just give the answer only. The answer can be a
number, a single word or a short phrase, plese make your response as short,
simple and clear as possible. User: Image Caption: {image caption} Question:
{question} Assistant Generation Prefix: The most possible answer is:

Table 2. Prompt for LLM-based VQA Evaluation.

Figure 3. VQA accuracy using captions generated by different image captioning
models and the proposed PoCa method. The PoCa method consistently improves the
VQA performance across all models and caption types.

Figure 3 provides the evaluation results on 5,000 questions in the VQA-v2
validation split. The human annotation for short captions represents the
accuracy of a single-sentence caption drawn from the MS-COCO annotation, while
the human annotation for detailed captions refers to five MS-COCO caption
annotations concatenated with the prefix "The following are several captions
of this image written by different people: " added to the front.

As can be seen, our proposed PoCa method (green) consistently yields
performance gains across all three examined LVLMs. The scale of improvement
ranges from 0.48% (MobileVLM-v2 detailed captions) to 2.10% (LLaVA-1.5 short
captions). Interestingly, we find that detailed captions do not necessarily
correlate with better information coverage, as the detailed captions generated
by MobileVLM-v2 underperform the single-sentence captions generated by
InternVL. The comparison also shows that human annotations may not be optimal
for certain scenarios, since several groups of LVLM-generated captions can
yield higher VQA accuracy compared to that of human annotators.

####  4.2.2. Image Paragraph Captioning

The image paragraph captioning dataset contains human-annotated single-
paragraph descriptions for Visual Genome images. We use its testing split,
which consists of 2,492 samples. We employ both the reference-free metric
CLIPScore (HesselHFBC21, ) and the reference-based metric METEOR (BanerjeeL05,
) to evaluate the quality of the captions.

CLIPScore measures the similarity between the image and text features
extracted by the CLIP model (we use the standard OpenAI pretrained ViT-
Base-32). The underlying assumption is that CLIP encoders are capable of
extracting semantic information and can represent the importance score $A$;
thus, a higher CLIPScore correlates with higher task sufficiency.

The reference-based metric METEOR is widely adopted for evaluations in image
captioning and natural language generation (e.g., machine translation). It
measures the word-level similarity between model-generated captions and human-
generated captions. The underlying assumption is that human annotations
optimize the task sufficiency objective, so if a model behaves similarly to
human annotations, it achieves high task sufficiency.

The results are shown in Table 3, where we also list the performance of
previous fully-supervised models and few-shot models. Once again, our PoCa
method provides task sufficiency improvement according to both the reference-
free metric CLIPScore and the reference-based metric METEOR across all three
families of LVLMs. These results further demonstrate the effectiveness of the
PoCa method in enhancing the quality and informative content of the generated
captions.

Table 3. Evaluation results on the image paragraph captioning dataset using
CLIPScore and METEOR.

Image Captioning Model | CLIPScore | METEOR
---|---|---
Fully Supervised Models | Regions-Hierarchical (krause2017hierarchical, ) | - | 15.95
RTT-GAN (LiangHZGX17, ) | - | 17.12
HSGED (YangGZ020, ) | - | 18.33
SCST (Melas-KyriaziRH18, ) | - | 17.86
CAE-LSTM (WangPYTM19, ) | - | 18.82
Few-shot & Zero-shot Models | BLIP-2 | 3-shot | - | 10.8
OPT-IML | 3-shot | - | 9.5
Naïve Ensemble | 3-shot | - | 9.8
BLIP-2 | VLIS (ChungY23, ) | - | 14.6
MobileVLM-v2-1.7B | Default | 80.05 | 13.95
PoCa | 81.80 | 16.39
MobileVLM-v2-3B | Default | 79.02 | 8.99
PoCa | 81.34 | 13.28
LLaVA-1.5-7B | Default | 81.68 | 28.11
PoCa | 81.80 | 28.79
LLaVA-1.5-13B | Default | 82.16 | 28.44
PoCa | 82.47 | 28.97
InternVL | Default | 84.65 | 29.32
PoCa | 85.52 | 29.84

###  4.3. Ablation Study and Further Analysis

####  4.3.1. Caption Merging Strategies

In this section, we compare the effectiveness of different implementations of
the merging function $\sigma_{\mathrm{merge}}$. First, we compare various LLMs
introduced in Section 4.1.2. As shown in Table 4, compared to the global
caption baseline, every LLM yields performance improvement, except for the
smallest Gemma-2B-IT model. We also provide an ablation on prompting, where we
replace the default prompt shown in Table 1 with a naive prompt of "merge
these captions". This ablation results in a slight decrease in accuracy and a
significant increase in caption length, which further violates the minimal
redundancy objective.

Additionally, we compare two parameter-free merging strategies based on simply
concatenating local-only captions (representing $\eta=0$ in Assumption 3) or
local-global captions, with positional encoding as in the "User" field in
Table 4. The results show that local captions alone cannot provide sufficient
information, while adding the global caption brings significant improvement.
However, these two concatenation-based methods generate excessively long
captions, demonstrating the necessity of LLM-based information fusion and
length compression.

Table 4. Comparison of different caption merging strategies on the VQA-v2
validation set.

Merging Function | Params | Accuracy | Length
---|---|---|---
Global Caption Baseline | 0 | 57.68 | 50.75
Gemma-2B-IT | 2B | 57.44 | 107.14
Gemma-7B-IT | 7B | 58.74 | 178.79
Mistral 7B Instruct-v0.2 | 7B | 58.92 | 136.12
LLaMA2-7B Chat | 7B | 58.64 | 199.34
LLaMA2-7B Chat (Naive Prompt) | 7B | 58.60 | 239.02
Qwen-1.5-7B-Chat | 7B | 58.64 | 130.93
LLaMA2 13B Chat | 13B | 59.06 | 154.67
Mixtral 8x7B Instruct-v0.1 | 46.7B | 59.78 | 219.22
Local Captions Concatenation | 0 | 55.66 | 265.63
Global Local Concatenation | 0 | 59.12 | 337.38

####  4.3.2. PoCa Does Not Sacrifices Minimal Redundancy

Previous evaluations primarily focused on the task sufficiency of image
captions. However, as our objective also contains other terms, it is crucial
to evaluate whether the performance gain brought by PoCa is achieved by
significantly sacrificing other objectives. In Table 5, we present the length
statistics for captions used in the previous task sufficiency comparison. We
calculate the average number of words in default captions and PoCa captions
and note their differences in the “$\pm\Delta$” column.

The results show that PoCa does not exhibit a significant trend of either
increasing or decreasing the length of captions. Among the six comparisons,
PoCa compresses the length in four cases and extends the length in two cases.
This empirically demonstrates that using LLMs as $\sigma_{\mathrm{merge}}$ in
the PoCa model does not significantly violate the minimal redundancy objective
$H(\tilde{Y}_{\mathrm{merged}})$.

Table 5. Comparison of caption lengths between default captions and PoCa
captions on VQAv2 and Img2P datasets.

LVLM | VQAv2 | Img2P
---|---|---
Default | PoCA | $\pm\Delta$ | Default | PoCA | $\pm\Delta$
MobileVLM-v2-1.7B | 54.1 | 78.2 | +24.1 | 61.6 | 47.0 | -14.6
LLaVA-1.5-7B | 82.7 | 74.7 | -8.0 | 93.2 | 133.4 | +40.2
InternVL | 158.3 | 93.4 | -65.0 | 177.4 | 176.2 | -1.2

####  4.3.3. Analysis of VQA-based Caption Evaluation

In this paper, we adopt the methodology of employing text-only LLMs for VQA
inference with image captioning input to evaluate caption quality. This
section provides a detailed analysis of this approach. Using the instruction
given in Table 2, we prompt different LLMs to generate answers and evaluate
the accuracy based on exact matching and Natural Language Inference (NLI)
based evaluation. The NLI evaluation classifies a pair of statements, "The
answer to this question is {ground truth}" and "The answer to this question is
{generated answer}", into entailment, neutral, and contradiction, where
entailment outputs are regarded as successful. Compared to exact matching, NLI
evaluation measures the correctness of answers at a semantic level and can
tolerate low-level differences.

As shown in Table 2, we find that different LLMs behave very differently in
terms of answer length, and many LLMs fail to keep the answer succinct as
instructed. Since the ground truth answers are mostly one word or a short
phrase, this results in significantly reduced exact matching accuracy, while
the actual semantic similarity is much higher, as measured by the NLI
accuracy. We also observe an increasing trend in NLI accuracy when comparing
different scales of LLMs, despite the largest Mixtral 8x7B Instruct-v0.1
producing lower NLI accuracy. We found that this outlier is caused by the
over-conservative nature of the Mixtral 8x7B Instruct-v0.1 model, which
frequently refuses to answer questions with responses such as “cannot
determine” and “not sure”. Finally, we add an ablation by instructing the LLM
to guess the answer without caption input using the prompt: "You will be given
a question regarding an image, and your task is to try to infer the most
possible answer". The resulting performance, noted as “LLaMA2 7B Chat (No
Caption)”, is much lower when measured by both exact matching and NLI
accuracy.

Table 6. Comparison of different LLMs for VQA-based caption evaluation.

LLM |  | Answer
---
Length
| Match
---
Accuracy
| NLI
---
Accuracy
Gemma-2B-IT | 33.50 | 5.20 | 55.44
Gemma-7B-IT | 38.00 | 0.00 | 54.44
Mistral 7B Instruct-v0.2 | 28.90 | 2.30 | 63.30
LLaMA2 7B Chat | 6.10 | 57.44 | 67.14
LLaMA2 7B Chat (No Caption) | 4.30 | 41.34 | 44.76
Qwen-1.5-7B-Chat | 7.90 | 56.72 | 69.06
LLaMA2 13B Chat | 5.30 | 60.24 | 69.14
Mixtral 8x7B Instruct-v0.1 | 24.40 | 8.38 | 64.86
Ground Truth Answer | 4.70 | - | -

##  5\. Related Work

###  5.1. Image Captioning

Image captioning lies at the intersection of computer vision and natural
language processing, requiring both accurate visual recognition and coherent
language generation abilities. In Section 2.2 we formally defined the
objectives for this task, but it is worth noting that current efforts do not
explicitly optimize that objective. The primary challenge is the difficulty of
back-propagation through the discrete textual space
$\mathcal{D}_{\mathrm{caption}}$, and efforts addressing this challenge
involve adopting reinforcement learning (abs-2205-12630, ; Feng00L19a, ) or
aligning continuous latent spaces with language spaces (LiuYA23, ;
YuCWKMHREB0MH023, ). However, these approaches suffer from training
instability and less satisfactory language coherence.

Most current methods rely on a surrogate methodology, where they use human
annotators $f_{\mathrm{human}}$ to write captions and train models to imitate
those captions. The underlying assumption is that human-written captions
optimize the objective $J(\mathrm{human})$, which is achieved by providing
instructions to crowd-sourced caption annotators. For example, the
instructions for MS COCO Caption annotation (ChenFLVGDZ15, ) include “Describe
all the important parts of the scene” and “Do not describe unimportant
details”, which are respectively connected to the task sufficiency term and
minimal redundancy term in our objective.

An important trend in the image captioning field is the increasing focus on
the comprehensiveness of image captions. As mentioned earlier, this represents
a decreased length penalty (smaller weight $\beta$ for the minimal redundancy
term) and more emphasis on the task sufficiency term. In recent years, there
has been an increasing number of high-quality detailed captioning datasets for
this target, such as human-annotated image paragraph captioning
(krause2017hierarchical, ), Densely Captioned Images (DCI)
(urbanek2023picture, ), and pseudo-labeled datasets, including LLaVA-Detailed-
Captions (liu2023visual, ), ShareGPT4V (chen2023sharegpt4v, ), AS-1B
(wang2024asm, ), etc. However, detailed caption annotation is much more
expensive than previous single-sentence annotation, while automated caption
labeling exhibits a high risk of hallucination.

###  5.2. Vision-Language Learning in the Era of Large Language Models

Various methods have been explored for enabling vision-language learning in
LLMs. One line of work focusing on vision-language alignment during
pretraining (dosovitskiy2020image, ; chen2020uniter, ; zhang2021vinvl, ;
wang2022simvlm, ; wang2022ofa, ; yu2022coca, ; wang2022git, ;
radford2021learning, ; li2022blip, ; li2023blip, ), allowing the model to
jointly learn a shared vision-language latent space. The other line of work,
improve the vision-language training efficiency by aligning the vision
representation into the language space of LLMs by only training the visual
encoder module or a vision-language projection matrix
(tsimpoukelli2021multimodal, ; alayrac2022flamingo, ; liu2023visual, ;
sun2023emu, ; bai2023qwen, ; zhang2024llavar, ). These two lines of works
enable vision-language alignment, enabling various joint vision and language
modalities prompting methods such as robot manipulation prompting
(jiang2023vima, ; wake2023gpt, ) and multimodal in-context learning
(zhao2024mmicl, ; zhang2024vicl, ).

Unlike the other two directions, another line of work exploits the reasoning
and planning ability of LLMs allowing zero-shot multimodal vision-language
inference by extracting the information from the vision modality into a
textual description and performing inference through frozen LLMs
(yang2022pica, ; hu2022promptcap, ; lin2022revive, ; gui2022kat, ). Recent
works in this direction showcase remarkable VQA ability through answer
heuristics generation (shao2023vllm, ) and enabling object tagging and image
editing through visual programming (gupta2023visual, ). Inspired by this line
of work, our work introduces a zero-shot hierarchical image captioning
approach that relies on the reasoning ability of LLMs to aggregate information
from local and global captions.

##  6\. Conclusion

Our work presents a novel information-theoretic framework that provides well-
defined principles for image captioning covering task sufficiency, minimal
redundancy, and human interpretability. By leveraging the theoretical
framework, we propose Pyramid of Captions (PoCa), a novel image captioning
approach that employs a hierarchical method to generate content-rich captions
by exploiting the complementary nature of local and global visual cues.
Through theoretical proofs and empirical evaluations, we demonstrate that PoCa
consistently enhances the quality of image captions, making them more
informative, semantically accurate, and contextually coherent while
maintaining brevity and interpretability. PoCa consistently improves the
captioning quality by up to +2.3% CLIPScore and +4.29% METEOR score on the
image paragraph captioning dataset. The effectiveness of PoCA-generated
captions is further validated through VQA-v2 dataset, improving up to +2.10%
accuracy score compared to the baseline captions. Our work contributes to the
ongoing efforts to improve image captioning quality and provides a rigorous
foundation for future research and applications.

## Limitations

While the PoCa method has demonstrated effectiveness in improving image
caption quality, there are several limitations that are worth discussing.

Assumptions on Image Semantics. The Assumption 2 made in this work could be
sometimes strong and unrealistic, especially for the naive patch splitting
function. The linear combination assumption may not hold well for images with
more complex structures. This issue could be particularly problematic when
objects or important semantic elements span across multiple local patches. In
future work, employing more advanced splitting functions, object detection or
semantic segmentation, could help alleviate this limitation and better capture
the semantic structure of the image.

Assumptions on Caption Semantics. Similarly, the assumption about the local-
global aggregation of caption semantics (Assumption 3) may not always be well
satisfied by the LLM used for caption merging, particularly when the LLM is
not sufficiently powerful. Weaker LLMs may struggle to effectively combine the
local and global caption semantics in the desired manner. Further
investigation into the impact of LLM choice on the fulfillment of this
assumption would be valuable.

Depth of the Caption Pyramid. In the experiments, this work has demonstrated
the benefits of a single level of local-global splitting and merging. However,
the potential of deeper caption pyramids has not been fully explored. As the
pyramid grows deeper, there could be a distribution shift for the input image
patches, leading to more errors in the generated captions. Investigating the
performance of merging functions for noisier captions is an important
direction for future research.

VQA Evaluation. While the VQA-based evaluation provides a useful measure of
caption quality in terms of task sufficiency, it has limitations. The
questions used for evaluation may not comprehensively cover all of the
important semantic units, resulting a sub-optimal estimation of the importance
score $A$. In addition, due to resource constraints, we use a 5,000 question
subset from the full VQAv2 dataset. To test its reliability, we run default
caption generation with 5 models, together with human annotated caption,
resulting in a total of (5+1)$\times$2=12 data points combining short and long
captioierns. The Pearson correlation coefficient between 5k subset accuracy
and full dataset accuracy is 0.8519 – although already quite high, it still
introduce some degree of noise for model performance evaluation.

Computational Efficiency. Our implementation of PoCa involves more inferences
to generate captions and prompting LLM for fusing the local and global
captions. These multiple inference steps and the use of large models can lead
to increased computational costs. This computational overhead may be a
concern, especially in resource-constrained environments or when processing a
large number of images. One potential solution is to finetune an image
captioning model on the captions generated by PoCa. By doing so, the knowledge
captured by PoCa can be distilled into the finetuned model, allowing for a
single inference pass during deployment, while still benefiting from the
enhanced caption quality achieved by PoCa. Similar approach of knowledge
distillation has been adopted in other literature, such as
(chen2023sharegpt4v, ; FanKIKT23, ; ChenLDW24, ).

## Broader Impact

The proposed PoCa method has the potential to positively impact various
applications, such as improving accessibility for visually impaired
individuals and enhancing information retrieval. However, it is crucial to
consider and address potential ethical implications, including biases
inherited from LLMs, privacy concerns related to revealing sensitive
information, and the environmental impact of training and deploying large-
scale models. Responsible deployment, clear communication of capabilities and
limitations, and ongoing efforts to mitigate biases and ensure fairness are
essential for developing image captioning systems that are not only
informative and accurate but also socially responsible and inclusive.

## References

  * [1] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  Advances in neural information processing systems, 35:23716–23736, 2022.
  * [2] D. Bahdanau, K. Cho, and Y. Bengio.  Neural machine translation by jointly learning to align and translate.  In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [3] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou.  Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.  2023\.
  * [4] S. Banerjee and A. Lavie.  METEOR: an automatic metric for MT evaluation with improved correlation with human judgments.  In J. Goldstein, A. Lavie, C. Lin, and C. R. Voss, editors, Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005, pages 65–72. Association for Computational Linguistics, 2005.
  * [5] Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, Q. V. Do, Y. Xu, and P. Fung.  A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity.  In J. C. Park, Y. Arase, B. Hu, W. Lu, D. Wijaya, A. Purwarianti, and A. A. Krisnadhi, editors, Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675–718, Nusa Dua, Bali, Nov. 2023. Association for Computational Linguistics.
  * [6] D. Chen, J. Liu, W. Dai, and B. Wang.  Visual instruction tuning with polite flamingo.  In M. J. Wooldridge, J. G. Dy, and S. Natarajan, editors, Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 17745–17753. AAAI Press, 2024.
  * [7] L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin.  Sharegpt4v: Improving large multi-modal models with better captions.  arXiv preprint arXiv:2311.12793, 2023.
  * [8] X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick.  Microsoft COCO captions: Data collection and evaluation server.  CoRR, abs/1504.00325, 2015.
  * [9] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu.  Uniter: Universal image-text representation learning.  In European conference on computer vision, pages 104–120. Springer, 2020.
  * [10] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, Z. Muyan, Q. Zhang, X. Zhu, L. Lu, et al.  Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.  arXiv preprint arXiv:2312.14238, 2023.
  * [11] J. Cho, Y. Hu, J. M. Baldridge, R. Garg, P. Anderson, R. Krishna, M. Bansal, J. Pont-Tuset, and S. Wang.  Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation.  In The Twelfth International Conference on Learning Representations, 2023.
  * [12] X. Chu, L. Qiao, X. Zhang, S. Xu, F. Wei, Y. Yang, X. Sun, Y. Hu, X. Lin, B. Zhang, and C. Shen.  Mobilevlm v2: Faster and stronger baseline for vision language model, 2024.
  * [13] J. Chung and Y. Yu.  VLIS: unimodal language models guide multimodal language generation.  In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 700–721. Association for Computational Linguistics, 2023.
  * [14] R. da Silva Torres and A. X. Falcao.  Content-based image retrieval: theory and applications.  RITA, 13(2):161–185, 2006.
  * [15] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  Advances in Neural Information Processing Systems, 36, 2024.
  * [16] R. Datta, J. Li, and J. Z. Wang.  Content-based image retrieval: approaches and trends of the new age.  In Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval, pages 253–262, 2005.
  * [17] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2020.
  * [18] L. Fan, D. Krishnan, P. Isola, D. Katabi, and Y. Tian.  Improving CLIP training with language rewrites.  In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
  * [19] Y. Feng, L. Ma, W. Liu, and J. Luo.  Unsupervised image captioning.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 4125–4134. Computer Vision Foundation / IEEE, 2019.
  * [20] J. Ganesan, A. T. Azar, S. Alsenan, N. A. Kamal, B. Qureshi, and A. E. Hassanien.  Deep learning reader for visually impaired.  Electronics, 11(20):3335, 2022.
  * [21] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh.  Making the V in VQA matter: Elevating the role of image understanding in visual question answering.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6325–6334. IEEE Computer Society, 2017.
  * [22] V. N. Gudivada and V. V. Raghavan.  Content based image retrieval systems.  Computer, 28(9):18–22, 1995.
  * [23] L. Gui, B. Wang, Q. Huang, A. G. Hauptmann, Y. Bisk, and J. Gao.  Kat: A knowledge augmented transformer for vision-and-language.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 956–968, 2022.
  * [24] Y. Guo, Y. Chen, Y. Xie, X. Ban, and M. S. Obaidat.  An offline assistance tool for visually impaired people based on image captioning.  In 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 969–976. IEEE, 2022.
  * [25] T. Gupta and A. Kembhavi.  Visual programming: Compositional visual reasoning without training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953–14962, 2023.
  * [26] D. Gurari, Y. Zhao, M. Zhang, and N. Bhattacharya.  Captioning images taken by people who are blind.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVII 16, pages 417–434. Springer, 2020.
  * [27] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi.  Clipscore: A reference-free evaluation metric for image captioning.  In M. Moens, X. Huang, L. Specia, and S. W. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 7514–7528. Association for Computational Linguistics, 2021.
  * [28] Y. Hu, H. Hua, Z. Yang, W. Shi, N. A. Smith, and J. Luo.  Promptcap: Prompt-guided task-aware image captioning.  arXiv preprint arXiv:2211.09699, 2022.
  * [29] Y. Hu, H. Hua, Z. Yang, W. Shi, N. A. Smith, and J. Luo.  Promptcap: Prompt-guided image captioning for vqa with gpt-3.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2963–2975, 2023.
  * [30] S. Jain, K. Pulaparthi, and C. Fulara.  Content based image retrieval.  Int. J. Adv. Eng. Glob. Technol, 3:1251–1258, 2015.
  * [31] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed.  Mixtral of experts, 2024.
  * [32] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan.  Vima: robot manipulation with multimodal prompts.  In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
  * [33] J. Krause, J. Johnson, R. Krishna, and L. Fei-Fei.  A hierarchical approach for generating descriptive image paragraphs.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 317–325, 2017.
  * [34] J. Li, D. Li, S. Savarese, and S. Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In International conference on machine learning, pages 19730–19742. PMLR, 2023.
  * [35] J. Li, D. Li, C. Xiong, and S. Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International conference on machine learning, pages 12888–12900. PMLR, 2022.
  * [36] X. Li, J. Yang, and J. Ma.  Recent developments of content-based image retrieval (cbir).  Neurocomputing, 452:675–689, 2021.
  * [37] X. Liang, Z. Hu, H. Zhang, C. Gan, and E. P. Xing.  Recurrent topic-transition GAN for visual paragraph generation.  In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 3382–3391. IEEE Computer Society, 2017.
  * [38] Y. Lin, Y. Xie, D. Chen, Y. Xu, C. Zhu, and L. Yuan.  Revive: Regional visual representation matters in knowledge-based visual question answering.  Advances in Neural Information Processing Systems, 35:10560–10571, 2022.
  * [39] H. Liu, C. Li, Y. Li, and Y. J. Lee.  Improved baselines with visual instruction tuning.  CoRR, abs/2310.03744, 2023.
  * [40] H. Liu, C. Li, Q. Wu, and Y. J. Lee.  Visual instruction tuning.  In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  * [41] H. Liu, W. Yan, and P. Abbeel.  Language quantized autoencoders: Towards unsupervised text-image alignment.  In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
  * [42] L. Melas-Kyriazi, A. M. Rush, and G. Han.  Training for diversity in image paragraph captioning.  In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 757–761. Association for Computational Linguistics, 2018.
  * [43] K. Niu and P. Zhang.  A mathematical theory of semantic communication.  CoRR, abs/2401.13387, 2024.
  * [44] M. Peyrard.  A simple theoretical model of importance for summarization.  In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 1059–1073. Association for Computational Linguistics, 2019.
  * [45] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [46] C. Rane, A. Lashkare, A. Karande, and Y. Rao.  Image captioning based smart navigation system for visually impaired.  In 2021 International Conference on Communication information and Computing Technology (ICCICT), pages 1–5. IEEE, 2021.
  * [47] C. E. Shannon.  A mathematical theory of communication.  Bell Syst. Tech. J., 27(3):379–423, 1948.
  * [48] C. E. Shannon.  Prediction and entropy of printed english.  The Bell System Technical Journal, 30(1):50–64, 1951.
  * [49] Y. Shao, Q. Cao, and D. Gündüz.  A theory of semantic communication.  CoRR, abs/2212.01485, 2022.
  * [50] Z. Shao, Z. Yu, M. Wang, and J. Yu.  Prompting large language models with answer heuristics for knowledge-based visual question answering.  In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14974–14983, 2023.
  * [51] B. Shi, Z. Wu, M. Mao, X. Wang, and T. Darrell.  When do we not need larger vision models?  arXiv preprint arXiv:2403.13043, 2024.
  * [52] R. Shwartz-Ziv and N. Tishby.  Opening the black box of deep neural networks via information.  CoRR, abs/1703.00810, 2017.
  * [53] R. K. Srihari.  Automatic indexing and content-based retrieval of captioned images.  Computer, 28(9):49–56, 1995.
  * [54] Q. Sun, Q. Yu, Y. Cui, F. Zhang, X. Zhang, Y. Wang, H. Gao, J. Liu, T. Huang, and X. Wang.  Emu: Generative pretraining in multimodality.  In The Twelfth International Conference on Learning Representations, 2023.
  * [55] N. Tishby, F. C. N. Pereira, and W. Bialek.  The information bottleneck method.  CoRR, physics/0004057, 2000.
  * [56] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * [57] Y. H. Tsai, Y. Wu, R. Salakhutdinov, and L. Morency.  Self-supervised learning from a multi-view perspective.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * [58] M. Tsimpoukelli, J. Menick, S. Cabi, S. M. A. Eslami, O. Vinyals, and F. Hill.  Multimodal few-shot learning with frozen language models.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * [59] J. Urbanek, F. Bordes, P. Astolfi, M. Williamson, V. Sharma, and A. Romero-Soriano.  A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions.  arXiv preprint arXiv:2312.08578, 2023.
  * [60] N. Wake, A. Kanehira, K. Sasabuchi, J. Takamatsu, and K. Ikeuchi.  Gpt-4v (ision) for robotics: Multimodal task planning from human demonstration.  arXiv preprint arXiv:2311.12015, 2023.
  * [61] J. Wang, Y. Pan, T. Yao, J. Tang, and T. Mei.  Convolutional auto-encoding of sentence topics for image paragraph generation.  In S. Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 940–946. ijcai.org, 2019.
  * [62] J. Wang, Z. Yang, X. Hu, L. Li, K. Lin, Z. Gan, Z. Liu, C. Liu, and L. Wang.  Git: A generative image-to-text transformer for vision and language.  Transactions on Machine Learning Research, 2022.
  * [63] P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  In International Conference on Machine Learning, pages 23318–23340. PMLR, 2022.
  * [64] W. Wang, M. Shi, Q. Li, W. Wang, Z. Huang, L. Xing, Z. Chen, H. Li, X. Zhu, Z. Cao, Y. Chen, T. Lu, J. Dai, and Y. Qiao.  The all-seeing project: Towards panoptic visual recognition and understanding of the open world.  In The Twelfth International Conference on Learning Representations, 2024.
  * [65] Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao.  SimVLM: Simple visual language model pretraining with weak supervision.  In International Conference on Learning Representations, 2022.
  * [66] X. Yang, C. Gao, H. Zhang, and J. Cai.  Hierarchical scene graph encoder-decoder for image paragraph captioning.  In C. W. Chen, R. Cucchiara, X. Hua, G. Qi, E. Ricci, Z. Zhang, and R. Zimmermann, editors, MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020, pages 4181–4189. ACM, 2020.
  * [67] Z. Yang, Z. Gan, J. Wang, X. Hu, Y. Lu, Z. Liu, and L. Wang.  An empirical study of gpt-3 for few-shot knowledge-based vqa.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3081–3089, 2022.
  * [68] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu.  Coca: Contrastive captioners are image-text foundation models.  Transactions on Machine Learning Research, 2022.
  * [69] L. Yu, Y. Cheng, Z. Wang, V. Kumar, W. Macherey, Y. Huang, D. A. Ross, I. Essa, Y. Bisk, M. Yang, K. P. Murphy, A. G. Hauptmann, and L. Jiang.  SPAE: semantic pyramid autoencoder for multimodal generation with frozen llms.  In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
  * [70] Y. Yu, J. Chung, H. Yun, J. Hessel, J. S. Park, X. Lu, P. Ammanabrolu, R. Zellers, R. L. Bras, G. Kim, and Y. Choi.  Multimodal knowledge alignment with reinforcement learning.  CoRR, abs/2205.12630, 2022.
  * [71] P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, and J. Gao.  Vinvl: Revisiting visual representations in vision-language models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5579–5588, 2021.
  * [72] Y. Zhang, R. Zhang, J. Gu, Y. Zhou, N. Lipka, D. Yang, and T. Sun.  Enhanced visual instruction tuning for text-rich image understanding, 2024.
  * [73] Y. Zhang, K. Zhou, and Z. Liu.  What makes good examples for visual in-context learning?  In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 17773–17794. Curran Associates, Inc., 2023.
  * [74] H. Zhao, Z. Cai, S. Si, X. Ma, K. An, L. Chen, Z. Liu, S. Wang, W. Han, and B. Chang.  MMICL: Empowering vision-language model with multi-modal in-context learning.  In The Twelfth International Conference on Learning Representations, 2024.
  * [75] Y. Zhong.  A theory of semantic information.  Proceedings, 1(3), 2017.

## Appendix

##  Appendix A Proof for Theorem 1

###### Theorem 1 (PoCa method reduces semantic error).

Under Assumptions 1-3, the PoCa method is guaranteed to have a smaller error
$Z_{\mathrm{merged}}=Y_{\mathrm{merged}}-X$ than $Z$, i.e.,

(12) $||Z_{\mathrm{merged}}||\leq||Z||.$

###### Proof.

First, we express the error of the $i$-th semantic unit in merged caption
$Z_{\mathrm{merged},i}$ as the difference between the merged caption semantics
$Y_{\mathrm{merged},i}$ and the source semantics $X_{i}$. Using Assumption 3
and 2, $Z_{\mathrm{merged},i}$ can be expressed as:

(13) $\displaystyle Z_{\mathrm{merged},i}$
$\displaystyle=Y_{\mathrm{merged},i}-X_{i}$ (14) $\displaystyle=\eta
Y_{i}+(1-\eta)\sum_{j}^{m}\alpha_{j}Y^{[j]}_{i}-X_{i}$ (15)
$\displaystyle=\eta(X_{i}+Z_{i})+(1-\eta)\sum_{j}^{m}\alpha_{j}(X^{[j]}_{i}+Z^{[j]}_{i})-X_{i}$
(16) $\displaystyle=\eta Z_{i}+(1-\eta)\sum_{j}^{m}\alpha_{j}Z^{[j]}_{i}.$

Here, (15) is derived from (14) is based on the decomposition of the global
and local caption semantics into there corresponding source semantics and
errors, i.e., $Y_{i}=X_{i}+Z_{i}$ and $Y^{[j]}_{i}=X^{[j]}_{i}+Z^{[j]}_{i}$.

Next, we define $\Delta_{\mathrm{PoC},i}$ as the gap between the norm of the
global caption error and the norm of the merged caption error for the $i$-th
semantic unit. Using the triangle inequality and Assumption 1, we derive a
lower bound for $\Delta_{\mathrm{PoC},i}$:

(17) $\displaystyle\Delta_{\mathrm{PoC},i}$
$\displaystyle=||Z_{i}||-||Z_{\mathrm{merged},i}||$ (18)
$\displaystyle=||Z_{i}||-||\eta
Z_{i}+(1-\eta)\sum_{j}^{m}\alpha_{j}Z^{[j]}_{i}||$ (19)
$\displaystyle\geq||Z_{i}||-(\eta||Z_{i}||+(1-\eta)\sum_{j}^{m}\alpha_{j}||Z^{[j]}_{i}||)$
(20) $\displaystyle=(1-\eta)(||Z_{i}||-\sum_{j}^{m}\alpha_{j}||Z^{[j]}_{i}||)$
(21)
$\displaystyle=(1-\eta)(\varphi(X_{i})-\sum_{j}^{m}\alpha_{j}\varphi(X^{[j]}_{i})).$

To yield (19) from (18), we apply the triangle inequality, which states that
for any two vectors $a$ and $b$, $||a+b||\leq||a||+||b||$. Finally, we apply
Assumption 1 to obtain (21), which states $||Z_{i}||=\varphi(X_{i})$ and
$||Z^{[j]}_{i}||=\varphi(X^{[j]}_{i})$.

By Assumption 1, $\varphi$ is a concave function. Applying Jensen’s inequality
and Assumption 2, we have:

(22)
$\varphi(X_{i})=\varphi(\sum_{j}^{m}\alpha_{j}X^{[j]}_{i})\geq\sum_{j}^{m}\alpha_{j}\varphi(X^{[j]}_{i}).$

This inequality implies that the error of the global caption is always greater
than or equal to the weighted average of the errors of the local captions.
Intuitively, this means that the PoCa method, which combines information from
both global and local captions, is expected to have a lower error than using
only the global caption.

Combining this result with the lower bound for $\Delta_{\mathrm{PoC},i}$
derived earlier, we can conclude that $\Delta_{\mathrm{PoC},i}$ is non-
negative for all $i$:

(23)
$\Delta_{\mathrm{PoC},i}\geq(1-\eta)(\varphi(X_{i})-\sum_{j}^{m}\alpha_{j}\varphi(X^{[j]}_{i}))\geq
0.$

The first inequality follows directly from the lower bound for
$\Delta_{\mathrm{PoC},i}$ derived earlier. The second inequality follows from
the Jensen’s inequality result, which states that
$\varphi(X_{i})\geq\sum_{j}^{m}\alpha_{j}\varphi(X^{[j]}_{i})$. Since
$1-\eta>0$ (as $\eta\in(0,1)$ by Assumption 3), the product of $(1-\eta)$ and
a non-negative term
$(\varphi(X_{i})-\sum_{j}^{m}\alpha_{j}\varphi(X^{[j]}_{i}))$ should also be
non-negative, thus proving that $\Delta_{\mathrm{PoC},i}\geq 0$ for all $i$.
Therefore, we have:

(24) $||Z_{\mathrm{merged}}||\leq||Z||.$

This completes the proof, demonstrating that under the given assumptions, the
PoCa method is guaranteed to reduce the semantic error compared to using only
the global caption.

∎

##  Appendix B PoCa Examples

We show some example of PoCa caption merging, where the images are sourced
from the test split of the image paragraph captioning dataset. Captions are
generated by LLaVA-1.5-7B model and merged by Mixtral-8x7B LLM. It shows that
local captions successfully captured some errors that are hallucinated or
omitted by the global caption, while PoCa successfully aggregated such
information (marked by green) and recompile it into a human-interpretable
caption.

|  |  Global Caption The image features a street corner with a street sign hanging above the
---
street. The sign is written in both English and Chinese, indicating that the
location is in an Asian
neighborhood. The street sign is positioned above a restaurant, which is
likely the ”Asian Pearl
Restaurant.” There are several people in the scene, with some standing near
the restaurant and
others walking around the area. A car is parked on the street, and a traffic
light is visible in the
background. The overall atmosphere of the scene suggests a bustling urban
environment with a mix of
cultures.
|  |  Top-left: The image features a large, ornate building with a green roof. The building has
---
a prominent green roof, which stands out against the sky. The sky is a pale
yellow, creating a
contrasting backdrop for the building. The building’s architecture and the
green roof make it an
eye-catching structure in the scene.
|  |  Bottom-left: The image features a brightly colored building with a yellow and red facade,
---
likely a Chinese restaurant. The building is adorned with a large sign that
reads ”Bright Pearl.”
The sign is positioned above the entrance, making it easily noticeable. In
addition to the main
building, there are two smaller buildings visible in the scene, one on the
left side and the other
on the right side. The overall atmosphere of the scene is vibrant and
inviting.
|  |  Top-right: The image features a street sign with a blue background and white lettering,
---
indicating the name of the street. The sign is attached to a green pole, which
is located near a
traffic light. The traffic light is positioned on the right side of the pole,
and it is currently
displaying a red light. The scene appears to be set in a city, with the street
sign and traffic
light being essential elements of urban infrastructure.
|  |  Bottom-right: The image features a street sign in a foreign language, likely Chinese,
---
warning people about the consequences of dumping. The sign is placed on a
pole, and it is positioned
near a building. The sign is written in both English and Chinese, making it
accessible to a wider
audience. In the background, there is a car parked on the street, and a person
can be seen standing
nearby. The scene appears to be set in an urban environment, with the street
sign serving as a
reminder for people to properly dispose of their waste.
|  |  PoCa Merged Caption: At a bustling Asian neighborhood street corner, an ornate building
---
with a green roof and a yellow-red Chinese restaurant named ”Bright Pearl”
stand out. A street sign
in English and Chinese indicates the location, while a nearby sign warns
against dumping. A car,
traffic light, and people add to the urban atmosphere. The likely ”Asian Pearl
Restaurant” is
positioned near the street sign.
|  Human Annotation: A picture of a Asian town. There are many signs in an
Asian language. Te
---
sky above the town is gray. There is a large green pole to the right with
signs mounted to it. There
are multiple story buildings along the streets. There is a restaurant with the
word pearl in it.
Therre are blue and white signs on the pole.

|  |  Global Caption The image features a group of four people dressed in old-fashioned
---
clothing, standing together in a train station. They are posing for a picture
in front of a train,
which is located on the left side of the scene. The people are dressed in
various styles, with some
wearing ties and hats. There are two handbags visible in the scene, one near
the center and the
other towards the right side. The overall atmosphere of the image suggests a
vintage or historical
setting.
|  |  Top-left: The image features a man standing in front of a train, which is prominently
---
displayed in the scene. The train is positioned towards the right side of the
image, and the man is
standing closer to the left side. The man appears to be posing for a picture
in front of the train.
In the background, there is a building with a clock on its side, adding to the
overall atmosphere of
the scene. Additionally, there are two other people in the image, one located
near the left edge and
the other towards the right side.
|  |  Bottom-left: The image features a train station with a train on the tracks. There are
---
several people in the scene, some of them standing near the train, while
others are scattered
throughout the station. A man is standing close to the train, possibly waiting
for it to depart. In
addition to the people, there are a few handbags visible in the scene, with
one near the center of
the image and another closer to the right side. The presence of these handbags
suggests that the
people in the station might be travelers or commuters.
|  |  Top-right: The image features a group of three people dressed in top hats and
---
Victorian-style clothing, standing together in a large room. They appear to be
posing for a picture,
possibly in a train station or a similar setting. The people are positioned in
the center of the
room, with one person on the left, another in the middle, and the third on the
right side. The room
has a few chairs placed around the area, with one chair located near the left
side of the room,
another in the middle, and a third chair on the right side. The overall
atmosphere of the scene
suggests a historical or themed event.
|  |  Bottom-right: The image features a man and a woman standing next to each other, both
---
dressed in old-fashioned clothing. The woman is wearing a green dress, while
the man is dressed in a
suit with a tie. They appear to be posing for a picture, possibly in a
historical setting. In the
scene, there are also two handbags visible, one near the woman and the other
closer to the man. The
overall atmosphere of the image suggests a vintage or historical theme.
|  |  PoCa Merged Caption: A group of four people in old-fashioned clothing, including two in
---
top hats, pose for a picture in a train station with a vintage atmosphere. A
train is visible on the
left, and two handbags are spotted near the center and right. Two other people
are scattered
throughout the scene, with one man standing close to the train.
|  Human Annotation: Four people wearing old-fashioned clothing stand posing
in front of the
---
camera. They are in a large building with a bright ceiling, and there is a
black train behind them.
There are two men on either side of two women. The man on the far left has a
black suit, black
shoes, and a black top hat. The woman to his right is wearing a white, pink,
and green dress, with a
white hat, and a broach pinned to her chest. She is holding a white satchel in
her white-gloved
hands. The woman next to her is also wearing white gloves and holding a white
satchel. This woman
has a green dress on, with lace on the trim. The man next to her is wearing a
checkered vest
underneath a dark long coat and dark pants. He has a black top hat.

|  |  Global Caption The image features a person holding a pair of scissors with a leafy green
---
plant in their hand. The scissors are positioned in the middle of the plant,
ready to cut it. The
person is likely preparing to trim the plant or remove a leaf. The scene takes
place outdoors, with
the person standing in a garden or an outdoor area.
|  |  Top-left: The image features a close-up view of a patch of grass on a dirt ground. The
---
grass appears to be green and healthy, with a few small rocks scattered around
the area. The dirt
ground is visible in the foreground, while the grassy patch extends into the
background. The scene
captures the essence of a natural environment, with the grass and rocks
creating a sense of depth
and texture.
|  |  Bottom-left: The image features a close-up of a green plant with a few leaves. The plant
---
is situated in a dirt field, and its leaves are visible in the foreground. The
plant appears to be
growing in a dirt-covered area, possibly a garden or a natural setting. The
focus of the image is on
the plant and its leaves, creating a sense of depth and detail.
|  |  Top-right: The image features a close-up of a pair of scissors with a leafy green plant in
---
the foreground. The scissors are positioned in the middle of the scene, with
the leafy plant
surrounding them. The plant appears to be a freshly cut herb, possibly mint,
and is placed on a dirt
surface. The scissors are likely being used to trim the plant or prepare it
for use in a recipe.
|  |  Bottom-right: The image features a pair of green scissors with a green handle, placed on a
---
pile of fresh herbs. The scissors are open, ready to be used for cutting the
herbs. The herbs are
scattered around the scissors, with some located closer to the scissors and
others further away. The
scene suggests that the person using the scissors is preparing to cut the
herbs for cooking or other
purposes.
|  |  PoCa Merged Caption: A person holds scissors with a leafy green plant, likely preparing to
---
trim it in an outdoor setting. The scissors, situated in the middle of the
plant, are positioned on
a pile of fresh herbs. The plant, possibly a type of mint, appears healthy and
green, surrounded by
small rocks and dirt.
|  Human Annotation: There are a pair of scissors sitting on top of a plant.
The handle on
---
the scissors is colored green. The other part of the scissor is metal. The
leaves of the plant or a
nice healthy green color.

|  |  Global Caption The image depicts a group of people gathered around a dining table in a
---
restaurant. They are enjoying a meal together, with several pizzas placed on
the table. There are
multiple cups and bottles scattered around the table, indicating that they are
drinking beverages as
well. The people are seated on various chairs placed around the table. Some of
them are engaged in
conversation, while others are focused on their meal. The atmosphere appears
to be relaxed and
social, as the group shares a meal and spends time together.
|  |  Top-left: The image depicts a group of people gathered in a room, possibly a restaurant or
---
a bar. There are at least nine people visible in the scene, with some sitting
at a dining table and
others standing around the room. The room features a spiral staircase, which
adds a unique
architectural element to the space. A few chairs are placed around the dining
table, and a cup can
be seen on the table. There is also a bowl and a spoon placed nearby. The
atmosphere appears to be
lively and social, with people enjoying their time together.
|  |  Bottom-left: The image features a group of people sitting around a dining table, enjoying
---
a meal together. There are at least five people visible in the scene, with
some sitting closer to
the table and others further away. The table is set with various food items,
including a pizza, a
bowl, and a sandwich. There are also multiple cups and a cell phone placed on
the table. A bottle
can be seen on the table as well. The dining table occupies a significant
portion of the image, with
the people sitting around it, creating a warm and inviting atmosphere.
|  |  Top-right: The image features a man with a long beard sitting at a dining table in a
---
restaurant. He is looking at the camera, possibly posing for a picture. The
man is wearing a blue
shirt and appears to be enjoying his time at the restaurant. The dining area
is furnished with
multiple chairs and dining tables. There are two chairs visible in the scene,
one near the man and
another further away. Two dining tables can be seen, one in the foreground and
another in the
background. In the background, there is a TV mounted on the wall, likely
providing entertainment for
the restaurant’s guests.
|  |  Bottom-right: The image features a person sitting at a dining table with a plate of pizza
---
in front of them. The table is set with a glass of beer, a cup, and a pitcher
of water. There are
also two additional cups on the table. The person is holding a fork, ready to
enjoy their meal. The
dining table is surrounded by chairs, with one chair on the left side and
another on the right side.
The scene appears to be a casual dining experience, with the person enjoying
their pizza and beer.
|  |  PoCa Merged Caption: A group of people, including a man with a beard, enjoy a meal in a
---
restaurant with various pizzas, sandwiches, and beverages on a table. Nine
people are gathered in a
room with a spiral staircase and chairs around the dining table. The
atmosphere is lively and
social. (169 characters)
|  Human Annotation: three men are talking. they all have bears on their face.
there is a
---
blue shirt on the man. the shirt has flowers on it.


stated. Once these are fixed, we find complete agreement.

The result for the purely quantum pieces in (3.3) also agrees with the result
quoted in ref. [27] on recognising that there the divergence can be recovered
by setting $\ln(1/\mu_{R})=1/2\varepsilon$. This mapping is also the one to
use to compare the level one divergence with (3.10).

The purely background terms in (3.3) agree with ref. [1] on recognising that
their $\varepsilon=8\pi^{2}(d-4)$, their definition of Ricci curvature is
minus ours, cf. below (2.5), and that their action is defined to be the
opposite sign from the usually defined Euclidean action, cf. (2.4). Their
normalisation of the scalar curvature term is also non-standard but this is
repaired by mapping $g_{\mu\nu}\mapsto\sqrt{2}\kappa g_{\mu\nu}$ and has no
effect on the one-loop result, since it is a curvature-squared action.

In the famous paper [3], this result is reproduced but the value quoted is
half that of (3.3). To see this one should note that it is Minkowski signature
and their $\varepsilon=4-d$ i.e. is twice ours. (There is also an accidental
extra factor of $1/\varepsilon$ in their quoted equation.) They also quote a
value for some two-loop double-pole divergences. The one point of comparison
is the result (3.24) for the $\bar{R}^{\mu\nu}\square\bar{R}_{\mu\nu}$
counterterm. Using these translations we see that their result is again half
of what we find.

## References

  * [1] Gerard ’t Hooft and M. J. G. Veltman.  One loop divergencies in the theory of gravitation.  Ann. Inst. H. Poincare Phys. Theor., A20:69–94, 1974.
  * [2] Marc H. Goroff and Augusto Sagnotti.  Quantum Gravity at Two Loops.  Phys. Lett., B160:81–86, 1985.
  * [3] Marc H. Goroff and Augusto Sagnotti.  The Ultraviolet Behavior of Einstein Gravity.  Nucl. Phys., B266:709–736, 1986.
  * [4] Anton E. M. van de Ven.  Two loop quantum gravity.  Nucl. Phys., B378:309–366, 1992.
  * [5] Joaquim Gomis and Steven Weinberg.  Are nonrenormalizable gauge theories renormalizable?  Nucl. Phys. B, 469:473–487, 1996, hep-th/9510087.
  * [6] John F. Donoghue.  General relativity as an effective field theory: The leading quantum corrections.  Phys. Rev., D50:3874–3888, 1994, gr-qc/9405057.
  * [7] C. P. Burgess.  Quantum gravity in everyday life: General relativity as an effective field theory.  Living Rev. Rel., 7:5–56, 2004, gr-qc/0311082.
  * [8] J Gasser and H Leutwyler.  Chiral perturbation theory to one loop.  Annals of Physics, 158(1):142–210, 11 1984.
  * [9] Steven Weinberg.  Phenomenological lagrangians.  Physica A, 96(1-2):327–340, 1979.
  * [10] G Colangelo.  Double chiral logs in the $\pi\pi$ scattering amplitude.  Physics Letters B, 350(1):85–91, 5 1995.
  * [11] J Bijnens, G Colangelo, and G Ecker.  Double chiral logs.  Physics Letters B, 441(1-4):437–446, 11 1998.
  * [12] J Bijnens, G Colangelo, and G Ecker.  Renormalization of chiral perturbation theory to order $p^{6}$.  Annals Phys, 280:100–139, 1999.
  * [13] Matthias Buchler and Gilberto Colangelo.  Renormalization group equations for effective field theories.  Eur. Phys. J. C, 32:427–442, 2003, hep-ph/0309049.
  * [14] John A. Dixon.  Field redefinition and renormalization in gauge theories.  Nucl. Phys. B, 99:420–424, 1975.
  * [15] B. L. Voronov and I. V. Tyutin.  Formulation of gauge theories of general form. I.  Theor. Math. Phys., 50:218–225, 1982.
  * [16] B. L. Voronov, P. M. Lavrov, and I. V. Tyutin.  Canonical Transformations and the Gauge Dependence in General Theories. (In Russian).  Yad. Fiz., 36:498–508, 1982.
  * [17] P. M. Lavrov and I. V. Tyutin.  Effective Action in General Gauge Theories. (In Russian).  Yad. Fiz., 41:1658–1666, 1985.
  * [18] Damiano Anselmi.  More on the subtraction algorithm.  Class. Quant. Grav., 12:319–350, 1995, hep-th/9407023.
  * [19] Jean Zinn-Justin.  Renormalization of Gauge Theories.  Lect. Notes Phys., 37:1–39, 1975.
  * [20] Jean Zinn-Justin.  Quantum field theory and critical phenomena.  Int. Ser. Monogr. Phys., 113:1–1054, 2002.
  * [21] I. A. Batalin and G. A. Vilkovisky.  Gauge Algebra and Quantization.  Phys. Lett., 102B:27–31, 1981.  [,463(1981)].
  * [22] I. A. Batalin and G. A. Vilkovisky.  Quantization of Gauge Theories with Linearly Dependent Generators.  Phys. Rev., D28:2567–2582, 1983.  [Erratum: Phys. Rev.D30,508(1984)].
  * [23] I. A. Batalin and G. A. Vilkovisky.  Closure of the Gauge Algebra, Generalized Lie Equations and Feynman Rules.  Nucl. Phys., B234:106–124, 1984.
  * [24] Joaquim Gomis, Jordi Paris, and Stuart Samuel.  Antibracket, antifields and gauge theory quantization.  Phys. Rept., 259:1–145, 1995, hep-th/9412228.
  * [25] Sergey N. Solodukhin.  Renormalization group equations and the recurrence pole relations in pure quantum gravity.  Nuclear Physics B, 962:115246, 1 2021.
  * [26] D. M. Capper.  A general gauge graviton loop calculation.  J. Phys. A, 13:199, 1980.
  * [27] Matthew Kellett, Alex Mitchell, and Tim R. Morris.  The continuum limit of quantum gravity at second order in perturbation theory.  Class. Quant. Grav., 38(11):115006, 2021, 2006.16682.
  * [28] M. K. Chase.  Absence of Leading Divergences in Two Loop Quantum Gravity.  Nucl. Phys. B, 203:434–444, 1982.
  * [29] G.W. Gibbons, S.W. Hawking, and M.J. Perry.  Path Integrals and the Indefiniteness of the Gravitational Action.  Nucl.Phys., B138:141, 1978.
  * [30] C. Becchi, A. Rouet, and R. Stora.  The Abelian Higgs-Kibble Model. Unitarity of the S Operator.  Phys. Lett., B52:344, 1974.
  * [31] C. Becchi, A. Rouet, and R. Stora.  Renormalization of the Abelian Higgs-Kibble Model.  Commun. Math. Phys., 42:127–162, 1975.
  * [32] C. Becchi, A. Rouet, and R. Stora.  Renormalization of Gauge Theories.  Annals Phys., 98:287–321, 1976.
  * [33] I. V. Tyutin.  Gauge Invariance in Field Theory and Statistical Physics in Operator Formalism.  1975, 0812.0580.
  * [34] Tim R. Morris.  Quantum gravity, renormalizability and diffeomorphism invariance.  SciPost Phys., 5:040, 2018, 1806.02206.
  * [35] Alex Mitchell and Tim R. Morris.  The continuum limit of quantum gravity at first order in perturbation theory.  JHEP, 06:138, 2020, 2004.06475.
  * [36] Jean-Louis Koszul.  Sur un type d’algebrés differéntielles en rapport avec la transgression.  In Colloque de Topologie, Bruxelles -or-, volume 78 of Bull. Soc. Math. France, pages 5 (73–81), 1950.
  * [37] Armand Borel.  Sur la cohomologie des espaces fibrés principaux et des espaces homogenes de groupes de lie compacts.  Ann. Math., 57:115–207, 1953.
  * [38] John Tate.  Homology of noetherian rings and local rings.  Illinois J. Math., 1:14–27, 1957.
  * [39] Jean M. L. Fisch and Marc Henneaux.  Homological Perturbation Theory and the Algebraic Structure of the Antifield - Antibracket Formalism for Gauge Theories.  Commun. Math. Phys., 128:627, 1990.
  * [40] Yuji Igarashi, Katsumi Itoh, and Tim R. Morris.  BRST in the Exact RG.  PTEP, 2019(10):103B01, 2019, 1904.08231.
  * [41] Glenn Barnich, Friedemann Brandt, and Marc Henneaux.  General solution of the Wess-Zumino consistency condition for Einstein gravity.  Phys. Rev. D, 51:1435–1439, 1995, hep-th/9409104.
  * [42] L.F. Abbott.  The Background Field Method Beyond One Loop.  Nucl.Phys., B185:189, 1981.
  * [43] D Kazakov.  A generalization of the renormalization-group equations for quantum-field theories of arbitrary form.  Theoretical and Mathematical Physics, 75(1):440–442, 4 1988.
  * [44] D Kazakov.  RG equations and high energy behaviour in non-renormalizable theories.  Physics Letters B, 797:134801, 10 2019.
  * [45] D Kazakov, R Iakhibbaev, and D Tolkachev.  Leading all-loop quantum contribution to the effective potential in general scalar field theory.  JHEP, 04:128, 2023, 2209.08019.
  * [46] D. Friedan.  Nonlinear Models in $2+\varepsilon$ Dimensions.  Phys. Rev. Lett., 45:1057, 1980.
  * [47] Daniel Harry Friedan.  Nonlinear Models in Two + Epsilon Dimensions.  Annals Phys., 163:318, 1985.
  * [48] Luis Alvarez-Gaume, Daniel Z. Freedman, and Sunil Mukhi.  The Background Field Method and the Ultraviolet Structure of the Supersymmetric Nonlinear Sigma Model.  Annals Phys., 134:85, 1981.
  * [49] Gerard ’t Hooft and M. J. G. Veltman.  Diagrammar.  NATO Sci. Ser. B, 4:177–322, 1974.
  * [50] William E. Caswell and A. D. Kennedy.  A Simple Approach to Renormalization Theory.  Phys. Rev. D, 25:392, 1982.