# Retrieval-augmented Image Captioning

Rita Ramos† Desmond Elliott⋆ Bruno Martins†
†INESC-ID, Instituto Superior Técnico, University of Lisbon
⋆Department of Computer Science, University of Copenhagen
<EMAIL_ADDRESS>
###### Abstract

Inspired by retrieval-augmented language generation and pretrained Vision and
Language (V&L) encoders, we present a new approach to image captioning that
generates sentences given the input image and a set of captions retrieved from
a datastore, as opposed to the image alone. The encoder in our model jointly
processes the image and retrieved captions using a pretrained V&L BERT, while
the decoder attends to the multimodal encoder representations, benefiting from
the extra textual evidence from the retrieved captions. Experimental results
on the COCO dataset show that image captioning can be effectively formulated
from this new perspective. Our model, named extra, benefits from using
captions retrieved from the training dataset, and it can also benefit from
using an external dataset without the need for retraining. Ablation studies
show that retrieving a sufficient number of captions (e.g., k=5) can improve
captioning quality. Our work contributes towards using pretrained V&L encoders
for generative tasks, instead of standard classification tasks.

##  1 Introduction

Image captioning is the task of automatically generating a short textual
description for a given image. The standard approach involves the use of
encoder-decoder neural models, combining a visual encoder with a language
generation decoder (see Hossain et al. (2019) for a survey). In early studies,
the encoder was typically a Convolutional Neural Network model (CNN)
pretrained on the ImageNet classification dataset Russakovsky et al. (2015) or
a pretrained Faster-RCNN object detector Ren et al. (2015), whereas the
decoder was commonly an LSTM Hochreiter and Schmidhuber (1997) together with
an attention mechanism Bahdanau et al. (2014). More recently, Transformer
based models have been achieving state-of-the-art results on a variety of
language processing Vaswani et al. (2017); Devlin et al. (2018); Radford et
al. (2019) and computer vision tasks Dosovitskiy et al. (2020). Accordingly,
state-of-the art image captioning models have replaced the conventional CNN-
LSTM approach with encoder-decoder Transformers Liu et al. (2021). Still, in
both cases, the encoder only attains visual representations, whereas richer
features could be captured from image–text interactions if the encoder had
access to useful textual context related to the input image (e.g., sentences
associated to similar images).

In this paper, we present a new type of image captioning model that uses a
pretrained V&L BERT (Tan and Bansal, 2019; Li et al., 2019; Bugliarello et
al., 2020, inter-alia) to encode both the input image and captions retrieved
from similar images. This model generates captions conditioned on
representations that consider linguistic information beyond the image alone.
Moreover, specifically using the retrieved captions as textual contexts rather
than other alternatives (e.g., image tags or object names) can aid guiding the
language generation process, since the model is now provided with well-formed
sentences that are semantically similar to what the predicted caption should
resemble.

In experiments on the COCO dataset Chen et al. (2015), the proposed model is
competitive against state of the art methods. In a series of ablation
experiments, we find that the model improves when encoding multiple retrieved
captions, and that it could reach better performance if it was able to
retrieve better captions from the datastore. In experiments on the smaller
Flickr30K dataset, we show that allowing the model to retrieve captions from
the larger COCO dataset can improve performance without needing to retrain the
model.

We hope that our work inspires the adoption of pretrained V&L encoders for a
broader range of generative multimodal tasks. There have been several recent
studies proposing V&L BERTs to learn generic multi-modal representations with
large amounts of paired image and text data, which can then be fine-tuned to
downstream tasks. However, these pretrained models have mostly been applied to
classification tasks and have seen limited use for image captioning, a task
which typically only considers single-input images, as opposed to image-text
pairs, as proposed in this work.

##  2 Model

We present a model that captions images, given both the image and a set of $k$
captions retrieved from similar images using a retrieval system. This approach
belongs to the class of retrieval-augmented language generation models Weston
et al. (2018); Izacard and Grave (2020). In our model, the image and the
retrieved captions are jointly encoded using a pretrained V&L encoder to
capture cross-modal representations in the combined input data. We denote our
model as extra: Encoder with Cross-modal representations Through Retrieval
Augmentation. It consists of three components, namely an encoder, a retrieval
system, and a decoder.

Figure 1: Illustration of the extra model. Given an input image, extra
retrieves captions from a datastore and encodes both the input image and the
retrieved captions using a pretrained vision-and-language encoder. The decoder
attends over both the visual and linguistic outputs, improving the quality of
the generated caption.

###  2.1 Encoder

The encoder in extra is LXMERT111The exploration of other encoders is left for
future work. Tan and Bansal (2019), a pretrained vision-and-language
Transformer that jointly encodes a visual input V and a linguistic input L.
The visual input is represented as $N$=36 regions-of-interest
V=$\\{v_{1},...,v_{N}\\}$ extracted from the image using the Faster-RCNN
object detector, pretrained Anderson et al. (2018) on the Visual Genome
dataset Krishna et al. (2016). A sentence in the linguistic input is tokenized
into $M$ sub-words using the BERT tokenizer Devlin et al. (2018), starting
with a special classification token CLS and ending with a special delimiter
token SEP. We extended LXMERT to encode $k$ sentences by concatenating the
tokenized sentences into a single input, each separated by the delimiter
token:
L=$\\{\texttt{CLS},w_{1}^{L_{1}}$…$w_{M}^{L_{1}},\texttt{SEP}$…$w_{1}^{L_{k}}$…$w_{M}^{L_{k}},\texttt{SEP}\\}$.
The sentences are obtained from a datastore via a retrieval system, as
explained in Section 2.2.

The encoder produces a sequence of cross-modal representations of image and
the text, which are the inputs to the decoder, described in Section 2.3.

###  2.2 Image–Text Retrieval and Datastore

The retrieval system builds on the Facebook AI Similarity Search (FAISS)
nearest-neighbour search library Johnson et al. (2017). FAISS allows for the
indexing of high-dimensional vectors, i.e., a datastore D, and it offers the
ability to quickly search through the datastore given a similarity measure S,
e.g., Euclidean distance or cosine similarity.

Given an input image V, the retrieval system finds L, the set of $k$ captions
retrieved from the datastore, which extra encodes together with the image. The
datastore consists of captions associated with images in a dataset222This can
either be the training set or an external dataset.. Each caption in the
datastore, and the query input image, are represented using vectors extracted
from CLIP Radford et al. (2021), allowing image–text search by projecting
images and text to a shared latent space. Using FAISS, the input image can
then be compared against the vectors333This comparison can be pre-computed for
efficiency. from D to search over the corresponding $k$ nearest-neighbours
captions.

###  2.3 Decoder

The decoder is a conditional auto-regressive language model based on GPT-2
Radford et al. (2019) with additional cross-attention layers to the encoder.
The Transformer layers in the decoder already contain a masked multi-head
self-attention sublayer, which self-attends to the previous words. We add
cross-attention layers Vaswani et al. (2017) subsequent to the masked self-
attention sublayers, so the decoder can attend to the encoder outputs.

The decoder predicts a caption $y_{1}\dots y_{M}$ token-by-token, conditioned
on the previous tokens and the outputs of the V&L encoder. The model’s
parameters $\theta$ are trained by minimizing the sum of the negative log-
likelihood of predicting the ground truth token at each time-step, using the
standard cross-entropy loss:

$\displaystyle L_{\theta}=-\sum_{i=1}^{M}\log
P_{\theta}(y_{i}|y_{<i},\textbf{V},\textbf{L}).$ (1)

We can also fine-tune the model with Self-Critical Sequence Training Rennie et
al. (2017).

##  3 Experimental Protocol

| Cross-Entropy Optimization | CIDEr Optimization
---|---|---
| B4 | METEOR | CIDEr | SPICE | B4 | METEOR | CIDEr | SPICE
Encoder-Decoder models |  |  |  |  |
Up-Down | 36.2 | 27.0 | 113.5 | 20.3 | 36.3 | 27.7 | 120.1 | 21.4
CaMEL${}_{\text{Faster R-CNN}}$ | 36.1 | 28.0 | 114.8 | 20.8 | - | - | - | -
GCN-LSTM | 36.8 | 27.9 | 116.3 | 20.9 | 38.2 | 28.5 | 127.6 | 22.0
VL-T5 | 34.5 | 28.7 | 116.5 | 21.9 | - | - | - | -
AoANet | 37.2 | 28.4 | 119.8 | 21.3 | 38.9 | 29.2 | 129.8 | 22.4
CPTR | - | - | - | - | 40.0 | 29.1 | 129.4 | -
extra ($k=5$) | 37.5 | 28.5 | 120.9 | 21.7 | 36.4 | 28.2 | 131.1 | 21.3
CaMEL${}_{\text{CLIP-RN50$\times$16}}$ | 38.8 | 29.4 | 125.0 | 22.2 | 41.3 | 30.2 | 140.6 | 23.9
V&L BERT models |  |  |  |  |  |  |  |
VLP | 36.5 | 28.4 | 116.9 | 21.2 | 39.5 | 29.3 | 129.3 | 23.2
OSCARB | 36.5 | 30.3 | 123.7 | 23.1 | 40.5 | 29.7 | 137.6 | 22.8
VinVLB | 38.2 | 30.3 | 129.3 | 23.6 | 40.9 | 30.9 | 140.4 | 25.1
Table 1: Results on the Karpathy COCO test split. extra ($k=5$) is competitive
against encoder-decoder models. We present results with cross-entropy training
and after Self-Critical Sequence Training using the CIDEr metric.

###  3.1 Datasets and Metrics

We evaluate our model on the COCO dataset Chen et al. (2015), using the
standard Karpathy splits of 113287 images for training, 5000 for validation,
and 5000 for testing, with 5 captions per image.

Standard metrics were used to evaluate caption generation, namely BLEU-4 (B4)
Papineni et al. (2002), METEOR Denkowski and Lavie (2014), CIDEr Vedantam et
al. (2015), and SPICE Anderson et al. (2016), using the MS COCO caption
evaluation package444https://github.com/tylin/coco-caption.

###  3.2 Implementation and Training Details

The implementation555https://github.com/RitaRamo/extra of extra uses the
HuggingFace Transformers library Wolf et al. (2020). The encoder is LXMERT Tan
and Bansal (2019), a 14-layer V&L model pretrained on 9 million image–sentence
pairs across a variety of datasets and tasks. Following Liu et al. (2021), the
decoder is a 4-layer randomly initialized GPT-2-style Transformer network with
12 attention heads and additional cross-attention layers. The retrieval
systems uses FAISS with a flat index (_IndexFlatIP_) without any training. The
corresponding datastore D consists of all the captions associated to the
113287 images in the COCO training set. For caption retrieval, the captions in
the datastore and the input image (i.e., the query) are both represented with
features extracted from the CLIP-ResNet50$\times$4 pretrained model. Using the
cosine similarity for comparison, a total of $k=5$ captions are retrieved to
be jointly encoded with the input image by extra. Notice that CLIP-
ResNet50$\times$4 features are only used for retrieval, while the extra
encoder, i.e. the pretrained LXMERT, requires Faster-RCNN features, and thus
it cannot use CLIP visual features.

extra is trained in two stages using a single NVIDIA V100S 32GB GPU. In the
first stage, extra is trained end-to-end with the cross-entropy loss, using a
batch size of 64 and the AdamW optimizer Loshchilov and Hutter (2017) with a
learning rate of $3e-5$. The encoder is trained with a linear warmup for the
first epoch to prevent gradients from the randomly initialized decoder from
harming the pretrained encoder. The model was trained with early stopping:
training ends if there is no improvement after 5 consecutive epochs on the
validation set over the BLEU-4 metric. In the second stage, extra is fine-
tuned with Self-Critical Sequence Training Rennie et al. (2017) with CIDEr
optimization and greedy search decoding as a baseline, using a batch size of
55, a learning rate of 3e-5, and a frozen encoder. Captions are decoded using
beam search with a beam size of 3.

##  4 Results

Table 1 shows the performance of extra compared to strong encoder-decoder
models. We compare against the widely-used Up-Down Anderson et al. (2018) and
AoANet models Huang et al. (2019), both using a Faster-RCNN image encoder; the
GCN-LSTM model Yao et al. (2018) with a Graph Convolutional Network (GCN)
encoder; the CPTR model Liu et al. (2021) employing a ViT Transformer encoder
Dosovitskiy et al. (2020); the VL-T5 Transformer model Cho et al. (2021) with
a vision and language encoder; and the recent CaMEL model Barraco et al.
(2022) with the CLIP-RN50×16 encoder. Our model is also compared with state-
of-art models that do not use the encoder-decoder paradigm but instead unify
the Transformer encoder and decoder into a single model, namely the VLP Zhou
et al. (2020), OSCAR-base Li et al. (2020), and the VinVL-base Zhang et al.
(2021) models. We note that these are general purpose V&L models, not
specifically designed for image captioning.

Overall, extra is competitive to state-of-the art captioning models. It
outperforms captioning models with vision encoders, and VL-T5, which, like
EXTRA, uses a V&L encoder, but with object tags as linguistic inputs rather
than retrieved captions. Although extra does not outperform the state of the
art captioning model, CaMEL, that uses a dual decoder, it outperforms the
variant of CaMEL that uses the same Faster-RCNN features. extra also competes
with general purpose V&L BERT models. Notice that our approach can be adapted
to other V&L encoders besides LXMERT (e.g., OSCAR, VinVL, etc.), or to more
powerful decoders (e.g., as in CaMEL). Likewise, other models could benefit
from retrieval-augmentation with captions.

###  4.1 Ablation Studies

We conducted a series of ablation studies in the Karpathy COCO validation
split to better understand what contributes to the success of extra.

#### Varying the Number of Retrieved Captions:

We start by studying the importance of training with multiple retrieved
captions, training with k=1 and k=3 captions to explore the effect of
retrieving fewer captions. Table 2 reports the result of this experiment,
showing that performance degrades when retrieving less captions.

| B4 | CIDEr
---|---|---
$k=1$ | 36.7 | 118.0
$k=3$ | 37.4 | 119.1
$k=5$ | 38.3 | 121.2
Table 2: The effect of training and evaluating using different numbers of
retrieved captions. Performance reported after training with cross-entropy
optimization.

#### Encoding Irrelevant Captions:

We also studied the performance of extra when it encodes textual input that is
not expected to be useful. We conduct two experiments where extra is trained
with textual input that is either an empty caption or a randomly chosen
caption.

  * •

Empty Caption: encode the image with an empty sentence: L={CLS, SEP};

  * •

Random Caption: encode the image with a random caption from the datastore.

Table 3 shows the result of this experiment. extra outperforms both variants,
further showing that the generation process is improved by encoding the image
together with relevant textual context from nearest-neighbour captions.
Although having an inferior performance, both models reach reasonable results
compared to other models in the literature (see Table 1), showing that LXMERT
can be used as a strong encoder for image captioning without providing
relevant input image-text pairs.

| B4 | CIDEr
---|---|---
Empty caption | 37.8 | 119.1
Random caption | 37.1 | 117.7
extra | 38.3 | 121.2
Table 3: The effect of training and evaluating with captions that are not
expected to be useful.

#### Encoding Irrelevant Images:

We tested ablating the visual input (i.e., setting the visual features to
zero). Training on “blacked out“ input images achieves 102.1 in CIDEr, which
is substantially lower than training with the actual input images, as seen in
Table 4. This further shows that extra uses the visual input, and does not
just rely on the retrieved information.

| B4 | CIDEr
---|---|---
Blacked out image | 32.1 | 102.1
extra | 38.3 | 121.2
Table 4: The effect of training and evaluating with “blacked out“ input
images.

#### Changing the Retrieval System and Datastore:

We then studied the effect of changing the retrieval system and the
representations in the datastore. Recall that extra relies on captions
obtained by Image–Text retrieval, where the datastore contains the captions
from the COCO training set, represented as vectors extracted from CLIP. We
conducted experiments with Image–Image and Image–Text retrieval to understand
which performs better:

  * •

Image–Image Retrieval: the datastore consists of all the images in the
training data. The representation of the input image is compared against those
in the datastore to find the $k$ nearest-neighbour images, and, subsequently,
to obtain the $k$ captions associated to those images. Specifically, one
reference caption is retrieved from each of the top-$k$ nearest-neighbour
images.

  * •

Image–Text Retrieval: the datastore consists of all the captions associated to
the images in the training data. The representation of the input image is
compared against the captions to directly find the top-$k$ captions.

For Image–Image retrieval, the input image and the images in D are represented
with _Faster R-CNN_ features, after global average pooling the embeddings of
the 36 region-of-interest vectors. For Image–Text Retrieval, the input image
and the caption vectors should already belong to a shared semantic space. We
use the pretrained CLIP model because it satisfies this criteria and thus
allows for direct image–text comparison. We considered two variants of CLIP
based on their visual backbone: _ViT_ or _ResNet50x4_ 666Regarding the
comparison measure S, the Euclidean distance and cosine similarity were used
respectively for Image-to-Image retrieval and Image-to-Text retrieval.. The
results of this experiment are reported in Table 5.

| B4 | CIDEr
---|---|---
Image–Image (Faster R-CNN) | 36.8 | 117.1
Image–Text (CLIP ViT) | 38.1 | 120.3
Image–Text (CLIP ResNet) | 38.3 | 121.2
Table 5: The effect of training and evaluating extra with different retrieval
systems. $k=5$ in both settings.

extra performs worse when it uses Image–Image retrieval in comparison to
retrieving captions directly with Image–Text retrieval. The best performance
is obtained with the ResNet-variant of the CLIP encoder. We also assess the
performance of directly using only one of the retrieved captions, with the
results shown in Figure 2. In this figure, we can visualize the expected CIDEr
score of the first retrieved captions and observe that some of them do not
sufficiently describe the image, or are mismatches, with a CIDEr of zero. We
also observe that the CIDEr score can change significantly depending on the
retrieval system. A larger number of mismatch captions are retrieved with
Image-to-Image retrieval. This suggests that the retrieval system and the
datastore can largely impact a retrieval-augmented image captioning model,
hence they should be carefully considered.

Figure 2: Histogram of the CIDEr scores for the nearest caption ($k=1$)
retrieved with Image-Image and Image-Text retrieval. This shows the
evaluations scores of using only the retrieved captions.

#### Oracle Performance:

Given that the retrieval system and datastore affect the performance of extra,
we also study whether extra could continue to improve if it could retrieve
better captions. After training extra with the $k=5$ retrieved captions, we
simulate an oracle retrieval system during inference, by allowing the actual
reference captions to be encoded by extra. Table 6 reports on experiments in
the validation data with respect to replacing one of the $k$ retrieved
captions with one of the reference captions, as well as replacing all with the
5 references associated to the input images. These experiments bring a 1.8 and
8.3 point increase in CIDEr score, respectively, showing the potential for
extra to improve by retrieving captions that better match the input image.

| B4 | CIDEr
---|---|---
$k=5$ retrieved captions | 38.3 | 121.2
$k=4$ and $1$ reference | 39.0 | 123.0
$k=0$ and $5$ references | 40.9 | 129.5
Table 6: Simulation of an oracle experiment, where extra can “retrieve”
reference captions of an image instead of retrieving all 5 captions from the
datastore.

##  5 Discussion

###  5.1 Vision First and Language Later

How does extra use the encoded image and retrieved captions? We quantify this
by estimating the behaviour of the cross-modal attention heads at each layer
in the decoder. Specifically, we compute the average of the cross-modal
attention across either the number of image regions or the sub-words in the
encoder, at each time-step of generating a caption and across each of the 12
attention heads.

Figure 3 shows that across the layers, the decoder’s attention shifts to the
textual outputs. In Layer 1, the model attends both to the visual and textual
representations, but the model hardly pays attention to the visual outputs by
Layer 4, relying more on the textual information from the retrieved captions.
This behaviour further shows that the semantics of the nearest captions can
aid guiding the language generation process. We performed an identical
calculation for the variants of extra that encoded an empty or a random
caption, finding in this case the opposite behaviour: the model learned to
ignore the textual embeddings provided by the encoder (see Appendix A).

Figure 3: The average cross-attention from the decoder to the outputs of the
encoder in respect to the visual $V$ and textual $L$ outputs. Values from COCO
validation.

###  5.2 Retrieve Enough Captions to Overcome Retrieval Mistakes

We note that training with an empty set of captions was better than encoding a
single $k=1$ and $k=3$ retrieved captions, observing Tables 2 and Table 3.
Thus, retrieval augmentation aids to improve caption quality when a sufficient
number ($k=5$) is considered. This further shows that retrieving enough
captions can be crucial for success. For this, we hypothesise that retrieving
more captions makes the model more robust in the presence of mismatches from
certain captions, as shown for instance in the second example in Figure 4.

###  5.3 Hot-swapping the Datastore

Besides taking advantage of similar training examples, we study whether extra
works with external image–caption collections without needing to retrain the
model. For this experiment, extra was first trained and evaluated in a small
dataset, and then the retrieval datastore was augmented with a larger dataset.
The considered datasets were Flickr30k and COCO, respectively. While Flickr30k
only contains 30k images, COCO contains 113K, each paired with five sentences.
Table 7 reports the results of these experiment. extra got a better
performance considering a larger external dataset than just using the current
training set, showing the potential for extra to adapt the retrieval
datastore.

Retrieval Datastore | B4 | CIDEr
---|---|---
Flickr30k | 28.8 | 59.6
\+ COCO | 29.5 | 59.9
Table 7: Performance of extra on the Flickr30k validation set. The model is
trained on the Flickr30K dataset with the Flickr30K datastore. The datastore
for inference is either the Flickr30K training set or combined with the COCO
training set.

###  5.4 Qualitative Examples

Figure 4 shows examples of captions generated by extra, given the input image
and the $k=5$ retrieved captions. extra benefits from textual evidence from
nearest-neighbour captions, even though sometimes the retrieved information
can be misleading, as depicted in the last example. More examples are provided
in Appendix B.

Figure 4: Examples of captions generated by extra conditioned on the input
image and retrieved captions.

##  6 Related Work

#### Image Captioning:

The task of image captioning is usually addressed by one of these three main
approaches: templates, retrieval, and encoder-decoder methods. Early
approaches involved template-based methods that consisted of filling blanks of
predefined captions through object detection Farhadi et al. (2010); Kulkarni
et al. (2013); Elliott and de Vries (2015). Retrieval-based methods instead
search over a dataset for the most similar image and fetch the corresponding
caption Hodosh et al. (2013); Ordonez et al. (2011). Currently, the most
common approach is the encoder-decoder framework Xu et al. (2015); Hossain et
al. (2019). The encoder typically used a pretrained CNN Vinyals et al. (2016)
or a Faster R-CNN Anderson et al. (2018), encoding the image into a grid of
image features or object proposal image regions. The decoder was usually a
LSTM with an attention mechanism Xu et al. (2015) to dynamically focus on
different parts of the encoded image during the prediction of each word.

Recently, Transformer-based models like BERT Devlin et al. (2018) have become
a more popular choice than LSTMs models, outperforming recurrent architectures
in different natural language processing (NLP) tasks Vaswani et al. (2017);
Qiu et al. (2020). Transformers can capture long-range dependencies with self-
attention layers and they can process each word of a sentence in parallel,
reducing training time. After the successful application in NLP, vision
Transformers like ViT Dosovitskiy et al. (2020) are also starting to become
the model of choice in the field of computer vision in place of CNNs. In
similar fashion, most recent captioning studies use the Transformer
arquitecture Herdade et al. (2019); Cornia et al. (2020); Liu et al. (2021),
employing a vision Transformer as encoder together with an autoregressive
language Transformer as decoder. Similarly to these models, this work proposes
a encoder-decoder Transformer model for the task of image captioning. However,
unlike them, the proposed model incorporates a pretrained V&L BERT to exploit
cross-modal representations, encoding images along with textual context. Also
differently from previous work, this approach explores retrieval-augmented
generation, i.e., combining neural encoder-decoder methods with traditional
retrieval-based methods.

#### V&L BERTs:

Previous studies have proposed pretrained Vision and Language (V&L) BERTs to
learn generic cross-modal representations of images and text, that can later
be used for a variety of downstream V&L tasks Bugliarello et al. (2020).
Examples include LXMERT Tan and Bansal (2019), VL-BERT Su et al. (2019),
Visual BERT Li et al. (2019), OSCAR Li et al. (2020), or UNITER Chen et al.
(2020), which were applied to VQA and other V&L classification tasks. Given
that these models are encoder-only Transformers, only few of them have been
applied to generation tasks such as image captioning. In such cases, the
generation is made from left to right by encoding the input image and using
the textual input elements with uni-directional attention masks, i.e.,
starting with a CLS token with the rest of the tokens masked, then considering
the CLS token with the predicted word (replaced by the corresponding mask
token) and the remaining ones still masked, and so on Li et al. (2020); Zhou
et al. (2020).

The use of pretrained V&L BERTs, as encoders in the standard encoder-decoder
captioning framework, remains largely unexplored. The task of image captioning
typically just considers single-input images, and not image-text pairs to be
encoded. In our work, a pretrained V&L encoder is used with a decoder for
image captioning, by leveraging not just the images as input but also
retrieved captions.

Besides pretrained V&L encoders, pretrained V&L encoder-decoder models have
recently been proposed to tackle classification and generation tasks, such as
VL-T5 Cho et al. (2021). Their captioning approach is similar to the present
paper, but VL-T5 uses object tags as textual inputs, whereas extra is
conditioned on retrieved captions.

#### Retrieval-augmented Generation:

The proposed approach is also similar to some studies on language generation
that predict the output conditioned on retrieved examples Weston et al.
(2018); Gu et al. (2018); Khandelwal et al. (2019); Lewis et al. (2020). For
instance, this work relates to Weston et al. (2018), in which a sequence-to-
sequence LSTM model, for dialog generation, encodes the current input
concatenated with the nearest retrieved response. Similarly, Izacard and Grave
(2020) used an encoder-decoder Transformer conditioned on retrieved passages
for open domain question answering. Retrieval-augmented generation is gaining
traction in NLP but has only been explored for image captioning by few studies
Wang et al. (2020); Fei (2021); Ramos et al. (2021); Sarto et al. (2022);
Ramos et al. (2022). Concurrent work proposed Transformer-based captioning
models augmented with retrieval as well Sarto et al. (2022); Ramos et al.
(2022). However, differently from these previous studies, we encode the
retrieved captions by exploiting cross-modal representations with a V&L
encoder.

##  7 Conclusions

We propose extra, a retrieval-augmented image captioning model that improves
performance by exploiting cross-modal representations of the input image
together with captions retrieved from a datastore. extra make uses of a
pretrained V&L BERT, instead of an image-only encoder, combined with a
language decoder. To generate a caption, the decoder attends to the cross-
modal encoder features, containing information from image regions and also
textual evidence from the retrieved captions. Image captioning is therefore
addressed as language generation conditioned on vision and language inputs,
instead of vision only. To evaluate this model, extra was assessed against
strong encoder-decoder models in the area, and ablation studies were also
conducted. The experiments conducted on the COCO dataset confirmed the
effectiveness of the proposed captioning approach.

For future work, we plan to explore the utility of extra in out-of-domain and
in few-shot learning settings, since the retrieval component can be easily
modified to include external datastores, without the need to retrain the whole
model. We also plan to explore how this approach can be adapted to other
powerful vision and language encoders besides LXMERT. Finally, we will explore
methods that allow us to jointly train the retrieval mechanism with the full
model in order to retrieve captions that are more similar to the input image.

## Limitations

Previous work has shown that generative models suffer from biases inherent to
the data they are trained on Weidinger et al. (2021); Thoppilan et al. (2022).
Likewise, our EXTRA model can suffer from biases present in the COCO image
captioning dataset Chen et al. (2015). Particularly, it has been shown that
there is significant gender imbalance in COCO, and that captioning models can
exhibit gender bias amplification (e.g., they are likely to generate the word
“woman” in kitchen scenarios, and the word “man” in snowboarding scenes)
Hendricks et al. (2018); Zhao et al. (2017).

However, differently from most captioning models, EXTRA is a retrieval-
augmented captioning model, and thus it has the potential to make predictions
beyond the training data, by relying on information from an external
datastore. Still, the datastore knowledge might also have inherent bias, as
mentioned by previous studies on retrieval-augmented generation Lewis et al.
(2020). In the paper, we show examples of such limitations wherein mismatched
retrieved captions can bias the model towards incorrect predictions (see the
results and appendix sections).

As a way to mitigate these limitations, we recommend analyzing the
corresponding nearest captions when using EXTRA, since the retrieved captions
can give useful insight of the bias involved in the generation process. EXTRA
can provide interpretability through textual descriptions, whereas most
captioning models only provide explanations as visual attention maps.

EXTRA also has the downside of focusing on an English-centric dataset.
Captioning datasets are primarily available in English, and most image
captioning models are trained on COCO or other english-centric datasets. To
avoid hindered research on image captioning, it is important to consider
multilingual captioning datasets that contain both language-diverse captions
and geographically-diverse visual concepts Thapliyal et al. (2022).

## Acknowledgements

This research was supported by the Portuguese Recovery and Resilience Plan
(RRP) through project C645008882-00000055 (Responsible.AI), and also through
Fundação para a Ciência e Tecnologia (FCT), namely through the Ph.D.
scholarship with reference 2020.06106.BD, as well as through the INESC-ID
multi-annual funding from the PIDDAC programme with reference UIDB/50021/2020.

## References

  * Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.  Spice: Semantic propositional image caption evaluation.  In _Proceedings of the European Conference on Computer Vision_ , pages 382–398. Springer.
  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proceedings of the Computer Vision and Pattern Recognition Conference_ , pages 6077–6086.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_.
  * Barraco et al. (2022) Manuele Barraco, Matteo Stefanini, Marcella Cornia, Silvia Cascianelli, Lorenzo Baraldi, and Rita Cucchiara. 2022.  Camel: Mean teacher learning for image captioning.  _arXiv preprint arXiv:2202.10492_.
  * Bugliarello et al. (2020) Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott. 2020\.  Multimodal pretraining unmasked: Unifying the vision and language berts.  _arXiv preprint arXiv:2011.15124_.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  Uniter: Universal image-text representation learning.  In _Proceedings of the European Conference on Computer Vision_ , pages 104–120.
  * Cho et al. (2021) Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021.  Unifying vision-and-language tasks via text generation.  _arXiv preprint arXiv:2102.02779_.
  * Cornia et al. (2020) Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020.  Meshed-memory transformer for image captioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10578–10587.
  * Denkowski and Lavie (2014) Michael Denkowski and Alon Lavie. 2014.  Meteor universal: Language specific translation evaluation for any target language.  In _Proceedings of the Workshop on Statistical Machine Translation_ , pages 376–380.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Elliott and de Vries (2015) Desmond Elliott and Arjen de Vries. 2015.  Describing images using inferred visual dependency representations.  In _Proceedings of the Joint Conference of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing_ , pages 42–52.
  * Farhadi et al. (2010) Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010.  Every picture tells a story: Generating sentences from images.  In _Proceedings of the European Conference on Computer Vision_ , pages 15–29. Springer.
  * Fei (2021) Zhengcong Fei. 2021.  Memory-augmented image captioning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 2–9.
  * Gu et al. (2018) Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. 2018.  Search engine guided neural machine translation.  In _Proceedings of the Conference of the Association for the Advancement of Artificial Intelligence_ , pages 5133–5140.
  * Hendricks et al. (2018) Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. 2018.  Women also snowboard: Overcoming bias in captioning models.  In _Proceedings of the European Conference on Computer Vision_ , pages 771–787.
  * Herdade et al. (2019) Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. 2019.  Image captioning: Transforming objects into words.  _arXiv preprint arXiv:1906.05963_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Hodosh et al. (2013) Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013.  Framing image description as a ranking task: Data, models and evaluation metrics.  _Journal of Artificial Intelligence Research_ , 47:853–899.
  * Hossain et al. (2019) MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga. 2019.  A comprehensive survey of deep learning for image captioning.  _ACM Computing Surveys_ , 51(6):1–36.
  * Huang et al. (2019) Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei. 2019.  Attention on attention for image captioning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4634–4643.
  * Izacard and Grave (2020) Gautier Izacard and Edouard Grave. 2020.  Leveraging passage retrieval with generative models for open domain question answering.  _arXiv preprint arXiv:2007.01282_.
  * Johnson et al. (2017) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017.  Billion-scale similarity search with gpus.  _arXiv preprint arXiv:1702.08734_.
  * Khandelwal et al. (2019) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019\.  Generalization through memorization: Nearest neighbor language models.  _arXiv preprint arXiv:1911.00172_.
  * Krishna et al. (2016) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2016\.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _arXiv preprint arXiv:1602.07332_.
  * Kulkarni et al. (2013) Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. 2013.  Babytalk: Understanding and generating simple image descriptions.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 35(12):2891–2903.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Proceedings of the Conference on Neural Information Processing Systems_ , 33:9459–9474.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019\.  Visualbert: A simple and performant baseline for vision and language.  _arXiv preprint arXiv:1908.03557_.
  * Li et al. (2020) Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020.  OSCAR: Object-semantics aligned pre-training for vision-language tasks.  In _Proceedings of the European Conference on Computer Vision_ , pages 121–137.
  * Liu et al. (2021) Wei Liu, Sihan Chen, Longteng Guo, Xinxin Zhu, and Jing Liu. 2021.  Cptr: Full transformer network for image captioning.  _arXiv preprint arXiv:2101.10804_.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Ordonez et al. (2011) Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011.  Im2text: Describing images using 1 million captioned photographs.  _Proceedings of the Conference on Neural Information Processing Systems_ , 24:1143–1151.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Qiu et al. (2020) Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020\.  Pre-trained models for natural language processing: A survey.  _Science China Technological Sciences_ , pages 1–26.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of the International Conference on Machine Learning_ , pages 8748–8763.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Ramos et al. (2022) Rita Ramos, Bruno Martins, Desmond Elliott, and Yova Kementchedjhieva. 2022.  SmallCap: Lightweight image captioning prompted with retrieval augmentation.  _arXiv preprint arXiv:2209.15323_.
  * Ramos et al. (2021) Rita Parada Ramos, Patrícia Pereira, Helena Moniz, Joao Paulo Carvalho, and Bruno Martins. 2021.  Retrieval augmentation for deep neural networks.  _Proceedings of the International Joint Conference on Neural Networks_.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Proceedings of the Conference on Neural Information Processing Systems_ , 28:91–99.
  * Rennie et al. (2017) Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017.  Self-critical sequence training for image captioning.  In _Proceedings of the Computer Vision and Pattern Recognition Conference_ , pages 7008–7024.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015\.  Imagenet large scale visual recognition challenge.  _International Journal of Computer Vision_ , 115(3):211–252.
  * Sarto et al. (2022) Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2022.  Retrieval-augmented transformer for image captioning.  _arXiv preprint arXiv:2207.13162_.
  * Su et al. (2019) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019\.  Vl-bert: Pre-training of generic visual-linguistic representations.  _arXiv preprint arXiv:1908.08530_.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  Lxmert: Learning cross-modality encoder representations from transformers.  _arXiv preprint arXiv:1908.07490_.
  * Thapliyal et al. (2022) Ashish V Thapliyal, Jordi Pont-Tuset, Xi Chen, and Radu Soricut. 2022.  Crossmodal-3600: A massively multilingual multimodal evaluation dataset.  _arXiv preprint arXiv:2205.12522_.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the Conference on Neural Information Processing Systems_ , pages 5998–6008.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015.  CIDEr: Consensus-based image description evaluation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4566–4575.
  * Vinyals et al. (2016) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2016.  Show and tell: Lessons learned from the 2015 mscoco image captioning challenge.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 39(4):652–663.
  * Wang et al. (2020) Li Wang, Zechen Bai, Yonghua Zhang, and Hongtao Lu. 2020.  Show, recall, and tell: Image captioning with recall mechanism.  _Proceedings of the Conference of the Association for the Advancement of Artificial Intelligence_ , 34(07):12176–12183.
  * Weidinger et al. (2021) Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021\.  Ethical and social risks of harm from language models.  _arXiv preprint arXiv:2112.04359_.
  * Weston et al. (2018) Jason Weston, Emily Dinan, and Alexander H Miller. 2018.  Retrieve and refine: Improved sequence generation models for dialogue.  _arXiv preprint arXiv:1808.04776_.
  * Wolf et al. (2020) Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of EMNLP: System Demonstrations_ , pages 38–45.
  * Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015.  Show, attend and tell: Neural image caption generation with visual attention.  In _Proceedings of the International conference on Machine Learning_ , pages 2048–2057.
  * Yao et al. (2018) Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. 2018.  Exploring visual relationship for image captioning.  In _Proceedings of the European Conference on Computer Vision_ , pages 684–699.
  * Zhang et al. (2021) Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021.  VinVL: Making visual representations matter in vision-language models.  _Proceedings of the Computer Vision and Pattern Recognition Conference_.
  * Zhao et al. (2017) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017\.  Men also like shopping: Reducing gender bias amplification using corpus-level constraints.  _arXiv preprint arXiv:1707.09457_.
  * Zhou et al. (2020) Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020.  Unified vision-language pre-training for image captioning and vqa.  In _Proceedings of the Conference of the Association for the Advancement of Artificial Intelligence_ , volume 34, pages 13041–13049.

##  Appendix A Cross-Attention

In Section 5.1, we quantified how much attention EXTRA pays to the encoded
image and retrieved captions. We also quantify this for the two other variants
of extra which encode irrelevant captions, using either an empty or a random
caption. Figures 5 and 6 show the average cross-attention weights from the
decoder to the outputs of the encoder in respect to the visual V and textual L
outputs, respectively for the empty and random caption encoding. Contrary to
the findings presented in Section 5.1, regarding the encoding of retrieved
captions, in this scenario the two variants pay more attention to the visual
outputs instead.

Figure 5: Cross-attention for the variant of EXTRA that that encodes an empty
caption. Figure 6: Cross-attention for the variant of EXTRA that that encodes
a random caption.

For details on how we calculated the corresponding attention weights, we
present the corresponding formula. Specifically, we calculated the average of
the cross-modal attention $C$ across either the number of image regions or the
sub-words in the encoder at each of $T$ time-step of generating a caption and
across each of the $H=12$ attention heads. This calculation happens
independently for each of the $L=4$ layers in the decoder:

$\displaystyle A(C^{L},V)$
$\displaystyle=\frac{1}{H}\sum_{j=1}^{H}\frac{1}{T}\sum_{t=1}^{T}\sum_{i=1}^{|V|}\alpha_{j,t\rightarrow
i}^{L}.$ (2) $\displaystyle A(C^{L},T)$ $\displaystyle=1-A(C^{L},V).$ (3)

##  Appendix B More Examples

Figure 7 shows additional examples of the captions generated by extra
considering the retrieved captions, against the other two variants: encoding
an empty and random caption instead. For the first image, the two variants
fail to recognize that the image shows kids playing basketball (perhaps given
the small size of the ball), whereas extra was able to identify it by having
that information in the retrieved captions. In the second image777The person
was blurred for privacy concerns., the two variants produced the error of
generating sandwich while extra correctly mentioned hot-dog, similar to the
retrieved captions. extra considers the semantics from the nearest captions
retrieved during generation, sometimes even copying an entire sentence, as
shown in Figure 9.

Figure 7: Examples of generated captions by extra and the other two variants
(empty and random caption). Better image captions are obtained from generating
with retrieval augmentation.

Figure 8 shows examples where the retrieved captions mislead the model. We
note however that extra is also able to succeed, despite the mismatch from
retrieved captions, as seen in Figure 10.

Figure 8: Qualitative results in which the retrieved captions are not that
related to the input image. Figure 9: Examples of generated captions for which
extra copied from the retrieved captions. Figure 10: Examples where extra is
able to succeeded even with mismatches from retrieved captions.


# An excess chemical potential for binary hard-sphere mixtures from integral
equation theory

Banzragch Tsednee  Tsogbayar Tsednee  Tsookhuu Khinayat  Institute of Physics
and Technology, Mongolian Academy of Sciences, Peace Ave 54B, 13330
Ulaanbaatar, Mongolia

###### Abstract

We solve the site-site Ornstein-Zernike equation using the Percus-Yevick
closure for binary hard-sphere mixture. We calculate an excess chemical
potential for the mixture’s diameter ratios of 0.3, 0.5, 0.6 and 0.9, and at
packing fraction of 0.49 using the analytical expression. Our numerical
results are in good agreement with those in the literature.

###### pacs:

Valid PACS appear here

##  I Introduction

In statistical thermodynamics a direct calculation of chemical potential is a
difficult task than computing other quantities, such as an internal energy,
the pressure, and the correlation functions determining the structures of the
physical systems. Various theoretical methods have been extensively employed
to calculate the chemical potential, such as, the Widom’s test particle
insertion method Widom63  or its modifications Adams74 ; Attard93  and the
scaled-particle Monte-Carlo (SP-MC) method Barosova96 ; Borowko00 ; Martin18 .

In theoretical studies for liquid an integral equation (IE) theory has been
widely used due to its low computational cost and a fact that the theory is a
thermodynamically complete, when combined with an appropriate closure equation
Ornstein14 ; Percus58 ; Chandler72 ; Hirata81 ; Perkyns92 ; Kovalenko99 ;
Luchko16 ; Lebowitz58 ; Tsogbayar19 . From the IE approach, calculating the
chemical potential is almost straightforward employing an analytical
expression based on the correlation functions. Therefore, in this work our
purpose is to calculate the chemical potential for binary hard-sphere mixtures
using the IE approach with the Percus-Yevick closure [8]. We will also compare
our numerical findings with the SP-MC results obtained by Barošová et. al.,
Barosova96 .

The paper consists of four sections. The first section presents the
Introduction, while the second and the third sections discuss a brief
description of theoretical formulation and numerical results, respectively.
Then a conclusion follows.

##  II Integral equation theory

For the atomic mixture system, the site-site Ornstein-Zernike equation (SSOZ)
Chandler72 ; Hirata81 ; Perkyns92 ; Kovalenko99  which establishes a relation
between the total correlation function $h_{ij}$ and the direct correlation
function $c_{ij}$ in the $\mathbf{k}$-space can be written in the form

$\tilde{h}_{ij}=\tilde{c}_{ij}+\sum_{i^{\prime}}\rho_{i^{\prime}}\tilde{c}_{ii^{\prime}}\tilde{h}_{i^{\prime}j},\quad(i,i^{\prime},j=1,2),$
(1)

where $\rho_{i}=x_{i}\rho$ is a number density, and $x_{i}$ is the mole
fraction of component $i$, and $\rho=\sum_{i}\rho_{i}$ is the total number
density. The tildes in Eq. (1) denote a Fourier transformation. Since we do
not know both correlation functions in Eq. (1), we cannot solve this equation
directly. Therefore, to solve this equation, we need an auxiliary equation
which we call a closure relation, and then they can be solved self-
consistently. A general closure for the mixture may be given in the form

$h_{ij}=\exp[-\beta u_{ij}+h_{ij}-c_{ij}+B_{ij}]-1.$ (2)

Here $u_{ij}$ is an interaction potential, which in this work is given by

$u_{ij}(r)=\begin{cases}r<\sigma_{ij}\\\ r\geq\sigma_{ij},\end{cases}$ (3)

where $\sigma_{ij}=\frac{1}{2}(\sigma_{i}+\sigma_{j})$, and $\sigma_{i}$ is a
diameter of component $i$, and $\beta=1/k_{\mathrm{B}}T$, $k_{\mathrm{B}}$ is
a Bolztmann’s constant and $T$ is the temperature for the system. .

The Percus-Yevick (PY) bridge function Percus58  which we will use in our
calculation has a form

$B_{ij}=\ln(1+\gamma_{ij})-\gamma_{ij},$ (4)

where $\gamma_{ij}=h_{ij}-c_{ij}$ is the indirect correlation function. We
note that in this PY approximation a solution for an IE for the hard-sphere
systems can be found in an analytical form Lebowitz58 . Once we have the
correlation functions describing the structure of the mixture, we can compute
an excess chemical potential for component $i$ using a following expression

$\beta\mu^{ex}_{i}=\sum_{j}\rho_{j}\int\Big{[}\Big{(}\frac{1}{2}h^{2}_{ij}-c_{ij}-\frac{1}{2}h_{ij}c_{ij}\Big{)}+\Big{(}1+\frac{2}{3}h_{ij}\Big{)}B_{ij}\Big{]}d\mathbf{r}.$
(5)

Note that a derivation of this expression can be performed similarly as it was
obtained for the pure Lennard-Jones fluid by Tsogbayar and Luchko Tsogbayar19
. Moreover, the expression can be used for any bridge functions since it does
not require the explicit forms of them.

##  III Results and Discussion

In our calculation we chose a component 1 as a reference particle which is a
larger component of the mixture, that is $(\sigma_{1}>\sigma_{2})$, and a
packing fraction $\eta$ is given as

$\eta=\frac{\pi}{6}\rho(x_{1}\sigma^{3}_{1}+x_{2}\sigma^{3}_{2}),$

where $x_{2}=1-x_{1}$ is the mole fraction for component 2. We solve an Eq.
(1) numerically using the Picard iteration method. We performed numerical
calculations for four different values of diameter ratio
$\sigma_{2}/\sigma_{1}:0.3,0.5,0.6,0.9$ and at $\eta=0.49$. This value of the
packing fraction indicates that our calculation can be done in the stable
liquid region since the value is slightly lower that the freezing packing
fraction for pure hard spheres Barosova96 . We do our calculation as function
of the mole fraction $x_{1}$ of the larger component. In all calculations a
number of grid points is $N=2^{15}$, and a length interval is
$L=32\sigma_{1}$. The convergence criterion for a successive two iterations of
the indirect correlation function is set to $10^{-6}\sigma_{1}$.

The columns 2 and 3 in Table I show our numerical results of an excess
chemical potential $\beta\mu^{ex}_{1}$, for various values of the mole
fraction $x_{1}$ and at $\eta=0.49$. We placed results of the SP-MC method
Barosova96  in the columns 3 and 5. When $\sigma_{2}/\sigma_{1}=0.3$, and for
the smaller values of $x_{1}$ our calculated values show small discrepancies,
however they reduce with an increase of the mole fraction of the larger
component. For $\sigma_{2}/\sigma_{1}=0.9$, our values placed in column 4 are
close to the SP-MC data shown in the last column. We note that the number of
particles used in the conventional canonical ensemble Monte-Carlo method of
Ref. Barosova96  is $1728$, however, since our IE method is an implicit
method, it is independent on number of particles involved in the system, that
is, we do not care about number of particles in the IE formulation.

Table 1:  A comparison of our calculated values for the excess chemical potential $\beta\mu^{ex}_{1}$ with those obtained by the SP-MC method Barosova96  for $\sigma_{2}/\sigma_{1}=0.3$, and $0.9$ and at $\eta=0.49$. | $\sigma_{2}/\sigma_{1}=0.3$ |  | $\sigma_{2}/\sigma_{1}=0.9$
---|---|---|---
$x_{1}$ |  $\beta\mu^{ex}_{1}$ (Eq. 5) |  Ref. Barosova96  |  |  |  $\beta\mu^{ex}_{2}$ (Eq. 5) |  Ref. Barosova96
$0.0625$ | $63.44$ | $66.75$ |  |  | 19.14 |
0.125 | 39.29 | 41.21 |  |  | 18.83 |
0.25 | 26.02 | 27.03 |  |  | 18.23 | 18.71
0.5 | 19.17 | 19.67 |  |  | 17.22 | 17.61
0.75 | 16.89 | 17.20 |  |  | 16.40 | 16.70
0.95 | 15.88 |  |  |  | 15.83 |

In Table II we present a comparison of our calculated values for
$\beta\mu^{ex}_{2}$ with available values obtained by the SP-MC method
Barosova96  for $\sigma_{2}/\sigma_{1}=0.3$ and $0.9$, and at $\eta=0.49$. All
our findings in this table are close to an accurate SP-MC values Barosova96 .

Table 2:  The same as in Table I, but for an excess chemical potential $\beta\mu^{ex}_{2}$. | $\sigma_{2}/\sigma_{1}=0.3$ |  | $\sigma_{2}/\sigma_{1}=0.9$
---|---|---|---
$x_{1}$ |  $\beta\mu^{ex}_{1}$ (Eq. 5) |  Ref. Barosova96  |  |  |  $\beta\mu^{ex}_{2}$ (Eq. 5) |  Ref. Barosova96
$0.0625$ | $5.10$ | $5.146$ |  |  | 15.34 |
0.125 | 3.76 | 3.766 |  |  | 15.08 |
0.25 | 2.97 | 2.957 |  |  | 14.62 | 14.97
0.5 | 2.55 | 2.519 |  |  | 13.82 | 14.11
0.75 | 2.40 | 2.368 |  |  | 13.17 | 13.42
0.95 | 2.33 |  |  |  | 12.73 |

Once we compute the excess chemical potential, we may obtain directly the
logarithm of the activity coefficient using a following equation Barosova96 :

$\ln\gamma_{i}=\beta\mu^{ex}_{i}(x_{i},\eta)-\beta\mu^{ex}_{i}(1,\eta).$ (6)

In Fig. 1 we show a logarithm of the activity coefficient calculated with an
Eq. 6 for the four different diameter ratios as function of $x_{1}$ at
constant $\eta=0.49$ for the smaller component. Blue, red, green and black
curves correspond to the diameter ratios of 0.3, 0.5 0.6 and 0.9,
respectively. Looking at these curves, the logarithm of the activity
coefficient decreases very quickly with $x_{1}$, especially for the smaller
diameter ratios in which the spheres’ diameters are more asymmetric, and for
$\sigma_{2}/\sigma_{1}=0.9$, its feature looks almost a linear (black). The
crosses are the SP-MC values taken from Ref. Barosova96 . The curves and
crosses follow each other.

Figure 1:  The logarithm of the activity coefficient as a function of the mole
fraction $x_{1}$ for the smaller component for $\sigma_{2}/\sigma_{1}=0.3$
(blue), 0.5 (red), 0.6 (green) and 0.9 (black) and at $\eta=0.49$. The crosses
are results taken from Ref. Barosova96 .

Fig. 2 presents same plots as shown in Fig. 1, but for the larger component.
For this case, the logarithm of the activity coefficient is negative. It
decreases relatively quickly for smaller diameter ratios as the mole fraction
of the larger component increases, which is indeed a similar pattern appeared
in Fig. 1. Crosses showing the SP-MC values Barosova96  follow our obtained
curves.

Figure 2:  The same plots as shown in Fig. 1, but for the larger component.
Crosses are taken from Ref. Barosova96 .

Red curves $(\sigma_{2}/\sigma_{1}=0.5)$ in Fig. 1 and Fig. 2 are not compared
with any data. However, looking at other obtained curves, we can note that
they can be as accurate as others.

##  IV Conclusion

In this work we have solved the site-site Ornstein-Zernike equation for binary
hard-sphere mixture for four different diameter ratios and at $\eta=0.49$ in
the PY approximation. The excess chemical potential for two components as a
function of the mole fraction $x_{1}$ of the larger component has been
computed with an analytical expression containing the correlation and bridge
functions, which in general can be used in an evaluation of the excess
chemical potential from the IE approach with any forms of bridge function.
Moreover, the logarithms of the activity coefficients for the components as a
function of the mole fraction $x_{1}$ have been obtained and compared with
accurate SP-MC data. It has been shown that our obtained results agree well
with those data.

## References

## References

  * (1) B. Widom, J. Chem. Phys. 39, 2808 (1963)
  * (2) D. J. Adams, Mol. Phys. 28, 1241 (1974)
  * (3) P. Attard, J. Chem. Phys. 98, 2225 (1993)
  * (4) M. Barošová, M. Malijevský, S. Labík, and W. R. Smith, Mol. Phys. 87, 423 (1996)
  * (5) M. Borówko, R. Zagórski and M. Malijevský, J. Chem. Phys. 112, 2315 (2000)
  * (6) S. C. Martin, B. B. Laird, R. Roth and H. Hansen-Goos, J. Chem. Phys. 149 084701 (2018)
  * (7) L. S. Ornstein and F. Zernike, Proc. Acad. Sci. Amsterdam 17, 793 (1914)
  * (8) J. K. Percus and G. J. Yevick, Phys. Rev. 110, 1 (1958)
  * (9) D. Chandler and H. C. Andersen, J. Chem. Phys. 57, 1930 (1972)
  * (10) F. Hirata and P. J. Rossky, Chem. Phys. Lett. 83, 329 (1981)
  * (11) J. Perkyns and B. M. Pettitt, J. Chem. Phys. 97, 7656 (1992)
  * (12) A. Kovalenko and F. Hirata, J. Chem. Phys. 110, 10095 (1999)
  * (13) T. Luchko, N. Blinov, G. C. Limon, K. P. Joyce and A. Kovalenko, J. Comput. Aided Mol. Des. 30, 1115 (2016)
  * (14) J. K. Lebowitz, Phys. Rev. 110 1 (1958).
  * (15) Ts. Tsogbayar and T. Luchko, Phys. Rev. E 99 032130 (2019)