# Controllable Dense Captioner with Multimodal Embedding Bridging

Yuzhong Zhao1 Yue Liu1 Zonghao Guo1 Weijia Wu2
Chen Gong3 Fang Wan1 Qixiang Ye1
1University of Chinese Academy of Sciences 2Zhejiang University 3University of
Virginia Corresponding Author.

###### Abstract

In this paper, we propose a controllable dense captioner (ControlCap), which
accommodates user’s intention to dense captioning by introducing linguistic
guidance. ControlCap is defined as a multimodal embedding bridging
architecture, which comprises multimodal embedding generation (MEG) module and
bi-directional embedding bridging (BEB) module. While MEG module represents
objects/regions by combining embeddings of detailed information with context-
aware ones, it also endows ControlCap the adaptability to specialized controls
by utilizing them as linguistic guidance. BEB module aligns the linguistic
guidance with visual embeddings through borrowing/returning features from/to
the visual domain and gathering such features to predict text descriptions.
Experiments on Visual Genome and VG-COCO datasets show that ControlCap
respectively outperforms the state-of-the-art methods by 1.5% and 3.7% (mAP).
Last but not least, with the capability of converting region-category pairs to
region-text pairs, ControlCap is able to act as a powerful data engine for
dense captioning. Code is available at https://github.com/callsys/ControlCap.

##  1 Introduction

Figure 1: Comparison of ControlCap with conventional methods. Given extra
linguistic guidance ($e.g.$, fine-grained label, scene text), ControlCap is
capable of generating specialized and adaptable captions in a controllable
interaction fashion.

Dense captioning [20] is a challenging task, which requires detecting and
describing objects within an image, at the same time, for a complete
understanding of the image. To create a captioning model that not only
accommodates specific user intentions but also is adaptive to varying
scenarios is even challenging.

Conventional methods [20, 41, 42, 39] lacked the flexibility for interactions
and the adaptability to specialized scenarios. A typical pipeline, Fig.
1(upper), simply input an image, extract features and generate general
captions without considering the user’s intention.

In this study, we attempt to conquer the limitations of conventional methods
by breaking through the following two bottlenecks: 1) Specialization. The
captioning model is extended to accept user-specified controls/guidance so
that it can produce “expected” outputs. Specifically, the model is responsive
to controls by fine-grained object detection results (“Lamborghini”) or scene
text recognition results (“FAFACHL”), Fig. 1(lower). 2) Adaptability. The
model is designed to be adaptable to complex image contents, particularly
regions and objects, to ensure its applicability in real-world scenarios, Fig.
1(lower).

We propose a controllable dense captioner (ControlCap), by borrowing
inspiration from multimodal large language models (MLLMs) [23, 24, 29],
Datasetdm [40], SAM [21], and controllable text generation models [4], which
incorporate the capacity to handle various specialized controls. ControlCap
comprises a multimodal embedding generation (MEG) module and a bi-directional
embedding bridging (BEB) module. The MEG module encodes the visual and
linguistic embeddings for generating and controlling text descriptions,
respectively. For the visual embeddings, to balance the detail and contextual
information of regions without increasing computation overhead, MEG module
employs two parallel and efficient components. One captures detailed and
context-free features. The other captures contextual but less detailed
features, which are then merged as the object embeddings within the visual
feature space. For the linguistic embeddings, MEG module then adapts to
specified instructions by treating specialized controls as linguistic guidance
and uniformly encoding it in the language feature space. Given the potential
for significant variance in control information, it also incorporates a
dropout strategy to enable switching between linguistic guidance to generate
controllable embeddings.

The diversity of specialized scenarios causes the difficulty of aligning the
controllable embeddings with the object embeddings, which hinders the model’s
adaptability. To solve this, BEB module integrates the controllable embeddings
with the object embeddings in a bi-directional way. The bridging from language
to vision enhances the adaptability of visual features to the language,
thereby alleviating the alignment issue, bridging from vision to language
improves the accuracy of linguistic guidance. Additionally, ControlCap holds
the capability of transforming region-category pairs to region-text pairs,
demonstrating the potential to serve as a powerful data engine for dense
captioning.

The contributions of this study are summarized as follows:

  * •

We propose a controllable dense captioner, enabling dense captioning to
accommodate the user’s intention by introducing linguistic guidance.

  * •

We design a multimodel embedding generation module and a bi-directional
embedding bridging module, providing a simple-yet-effective way to bridge the
representation gap between linguistic guidance with visual regions/objects
caused by the specialized scenarios.

  * •

The proposed approach significantly outperforms existing methods on commonly
used benchmarks ($i.e.$, Visual Genome (VG) [22] and VG-COCO [35]).

##  2 Related Work

Figure 2: Diagram of the proposed controllable dense captioner (ControlCap).
ControlCap comprises a multimodal embedding generation (MEG) module and a bi-
directional embedding bridging (BEB) module. MEG module, which consists of
contextual visual embedding and conditional entity embedding, is introduced to
enforce MLLM’s capacity for object-aware understanding and specialized
controls. BEB module is designed to increase the model’s adaptability for
specialized controls and scenarios.

Multimodal Large Language Model. To harness the zero-shot and reasoning
capabilities of large language models (LLMs) [45, 6, 37, 1, 3], there is a
trend towards fusing vision-and-language models with LLMs to produce
multimodal large language models (MLLMs). Benefit from powerful foundation
models [11, 9, 45, 6] and huge amount of vision language data corpus, MLLMs
have achieved unprecedented performance on few-shot learning [2], visual
question answering (VQA) [23, 24, 7, 29] , image captioning [23, 24, 7, 29],
referring expression generation [5, 46, 43, 32] and phrase grounding [43, 32].

Most MLLMs share a similar composition, $i.e.$, a visual foundation model
($e.g.$, ViT [11]) to extract the visual representation of the image, a LLM
($e.g.$, Flan-T5 [6], llama [37], vicuna [48]) for language comprehension, and
an alignment network ($e.g.$, Q-former [24], Projection [29]) to map the
visual representation to the language feature space. Dense captioning is
closely linked to MLLMs and is a specialized task related to referring
expression generation, where models are required to generate precise captions
for specific regions within an image. With the connection between dense
captioning and MLLMs, we believe that the former can not only benefit from the
latter but also can contribute to them: 1) Pretrained MLLMs are strong
initialization for dense captioning. They have been trained on vast amounts of
vision and language data, showcasing their capacity in tasks like image
captioning across various benchmarks. 2) Reliable dense captioning models can
serve as a powerful data engine for MLLMs. MLLMs [32, 47, 43, 5] have either
directly incorporated dense captioning datasets ($e.g.$, Visual Genome [22])
as part of their training data or use SOTA perception models to generate them
($e.g.$, GRIT [32]). Considering labeling data for dense captioning requires
great human efforts, generating data through existing models offers a
promising solution.

Dense Captioning. This aims to detect and generate detailed descriptions for
dense object regions in an image. As a pioneered method, FCLN [20] uses a
localization network to locate regions and a recurrent network to generate
captions. JIVC [41] argues that visual concepts are associated with each
other. Based on the Faster R-CNN [15] detector, it fuses image context feature
with RoI (Regions of Interest) features and inference the location and caption
of objects with two LSTM [18]. COCG [26] takes a further step to fuse context
features of objects in the image with RoI features. CAG-Net [42] introduces
the features of neighboring regions and global images into the target region,
then generates captions for the target.

In the era of transformers, TDC [35] proposed an end-to-end transformer-based
architecture, which fully mines the relation between objects in the image and
decodes the caption by the transformer decoder. GRiT [39] views object
categories as short captions and proposes to train object detectors and
captioning models in a unified framework. CapDet [31] proposed to unify dense
captioning and open-world detection in a pretraining framework. CapDet first
unified the formulation of object category with dense caption by expanding the
category to its long text definition, then aligning the object text embedding
with RoI embeddings. Finally, CapDet finetunes a transformer-based dense
captioning head on Visual Genome [22] to achieve dense captioning ability.

Despite of progress in the past decade, the performance of dense captioning
remains unsatisfactory. An intuitive solution is using large-scale datasets
with weak labels ($e.g.$, Image-text pairs [33], region-category pairs [34])
to enhance the performance, which however remains challenged by text-object
misalignment. From the data perspective, ControlCap has two contributions: 1)
Exploiting the capacity of MLLMs trained on millions of image-text pairs. 2)
Converting large-scale object detection datasets to dense captioning datasets
in a way of self-improvement.

Controllable Text Generation. Controllable text generation (CTG) [44] is an
emerging direction within the field of natural language generation (NLG). Its
primary objective is to exert control over the text generation process by
incorporating additional conditions. There are various tasks involving CTG,
including attribute-based generation [8, 27, 4], dialogue generation [36],
storytelling [12], debiasing [30], and format control [25]. One task closely
related to our work is lexicon-controlled text generation, which is a subset
of attribute-based generation. In this task, the goal is to generate text that
is centered around a specified keyword, with the expectation that the keyword
will be present in the generated text. Existing methods have attempted to
achieve lexicon control through techniques like fine-tuning [4], post-
processing [8], and diffusion [27]. In the field of image captioning,
LaNAR[10] excels in providing image captions with specified levels of detail,
achieved by managing the length of generated captions. PromptCap[19] leverages
natural language prompts to direct the description of visual entities in the
generated captions. Different from previous methods that aim to produce fluent
text that meets certain conditions or control the generated captions in the
image-level, ControlCap is specifically designed to produce specialized
captions for designated regions, guided by region-level linguistic guidance.

##  3 The Proposed Approach

###  3.1 Overview

For a conventional dense captioning pipeline, regions in images are encoded to
visual embeddings, which are then translated to text descriptions through a
language model. To achieve controllable dense captioning, ControlCap defines a
multimodal embedding generation (MEG) module (Sec. 3.2) to not only generate
object embeddings of regions but also learn controllable embeddings within a
language feature space. These embeddings facilitate generating text
descriptions in a controllable fashion. A bi-directional embedding bridging
(BEB) module (Sec. 3.3) is further introduced to handle misalignment caused by
the variety of specialized controls.

As shown in Fig. 2, ControlCap is built upon a pretrained MLLM which consists
of a Vision Transformer (ViT), an Alignment Network and a Large Language Model
(LLM), which are used to construct the multimodal embedding generation module
and bi-directional embedding bridging module. Each image with regions of
interest ($e.g.$, the region of “FAFACHL”) is first fed to the MEG module,
which consists of two parallel branches, Fig. 2. The first branch extracts
object embeddings of regions using the ViT. The second branch, meanwhile,
encodes the linguistic guidance to generate the controllable embedding for
each region. For various types of linguistic guidance, it’s challenging to
learn a unified controllable embedding aligned with the object embedding.
Therefore, these two embeddings are further fed to the BEB module. This module
compensates for the gaps brought by different linguistic guidance through bi-
directional alignment and fusion between linguistic and visual features to
enhance the object embedding and the controllable embedding. Finally, the
object embedding is projected to a language feature space by the Alignment
Network and is concatenated with the controllable embedding to predict text
descriptions using the LLM.

During training, the goal of ControlCap is to maximize
$P(\mathcal{X}|\mathcal{I},\mathcal{B},\mathcal{C})=\prod_{i=1}^{n}p(x_{i}|x_{<i},\mathcal{I},\mathcal{B},\mathcal{C})$,
where $\mathcal{X}=[x_{1},x_{2},\cdots,x_{n}]$ denotes the generated text
sequence. $\mathcal{I},\mathcal{B},\mathcal{C}$ are the input image, the
bounding box region, and the linguistic guidance. During inference, by giving
an image $\mathcal{I}$ and a bounding box $\mathcal{B}$, ControlCap generates
region descriptions under linguistic guidance $\mathcal{C}$ (upper right of
Fig. 2) or without guidance (bottom right of Fig. 2). The trained ControlCap
is used to convert the region-category pairs to region-text pairs for data
engine application (Sec. 3.4).

###  3.2 Multimodal Embedding Generation

Figure 3: Contextual visual embedding sub-branch in multimodal embedding
generation (MEG) module.

The MEG model is introduced to enforce MLLM’s capacity for object-aware
understanding and specialized controls, which consists of contextual visual
embedding and conditional entity embedding.

Contextual Visual Embedding: Different from the methods [24, 29] generating
captions for entire images, ControlCap generates a precise description for
each image region $\mathcal{B}$. To extract context-aware object embedding for
description generation, an intuitive way, similar to object detection, is to
extract features from the entire image and then use RoI-align [17] to acquire
region/object embedding. However, this method introduces a conflict between
the detail-rich object embedding and the computational overhead brought by
image resolution. To efficiently extract context-aware and detailed features,
we extract two parallel and specialized features and then merge them, Fig. 3.

First, the image $\mathcal{I}$ is scaled to a low resolution one and is fed to
the ViT [33] to be encoded as a global feature map, upon which an RoI-align
module [17] with $H\times W$ output size is used to extract the RoI feature
$X_{c}$. $X_{c}$ is coarse yet context-aware for speeding up the computation.
We then crop a region clip according to the location of $\mathcal{B}$, and
resize it to the same size and feed it to the ViT to extract detail-rich
object embedding $X_{r}$. To couple the context information, we concatenate
$X_{r}$ and $X_{c}$ across channel dimension, followed by passing through two
learnable MLP layer $f$, so that we extract the merged object embedding for
region $\mathcal{B}$.

Figure 4: Conditional entity embedding sub-branch in multimodal embedding
generation (MEG) module.

Conditional Entity Embedding: To control an LLM to generate specialized
descriptions about an image region, we design a module to extract controllable
embedding based on a noun entity, Fig. 4. Given the ground-truth description
of an image region, an off-the-shelf algorithm [14] is employed to capture the
noun proposals, $e.g.$, “car” and “street”. We utilize the tokenizer and word
embedding layer of the pretrained LLM to encode the text embeddings
$X_{l}\in\mathbb{R}^{N\times D}$ of a randomly chosen noun proposal, where $N$
is the number of tokens in the noun proposal and $D$ is the dimension of word
embedding layer.

To increase the model’s adaptability, it is required to operate with various
types of controls or without any input controls. We experimentally set a
probability $p$ to drop the noun strings by replacing them with empty strings
during the training phase, Fig. 4. To guarantee generalized controllable
ability with these empty strings, a memory unit is developed using a 1D
learnable parameter $\theta\in\mathbb{R}^{D}$. By adding $\theta$ with each
token embedding of the text embedding $X_{l}$, the controllable embedding for
the corresponding object embedding is generated, which is further used to
control LLM to generate specialized descriptions about image regions.

###  3.3 Bi-directional Embedding Bridging

Figure 5: Bi-directional embedding bridging (BEB) module. The object embedding
and the controllable embedding are obtained from the multimodal embedding
generation (MEG) module.

With the MEG module, ControlCap can generate captions for a given image region
under linguistic guidance. However, due to the diversity of specialized
scenarios, there exists significant misalignment between the controllable
embedding and the object embedding, which hinders the model’s adaptability.

The bi-directional embedding bridging module (BEB), Fig. 5, aims to solve the
misalignment issues. BEB is composed of three types of layers, $i.e.$, adapter
layers, cross-attention layers [38] and feed-forward layers [38]. Adapter
layers are single linear layers that aim to map the object embedding or the
controllable embedding to a low-dimensional latent space, Fig. 5(left), or map
them back to the original feature space, Fig. 5(right).

Object and controllable embeddings are first mapped to the same latent space
by two adapter layers. Features from the controllable embedding are then
transmitted to the object embedding by a cross-attention layer and a feed-
forward layer, which uses controllable embedding as Key and Value and object
embedding as Query (Fig. 5 upper). Meanwhile, features from the object
embedding are transmitted to the controllable embedding by a cross-attention
layer and a feed-forward layer, which use controllable embedding as Query and
object embedding as Key and Value, Fig. 5. Finally, the feature-enhanced
object embedding and controllable embedding are mapped back to their original
feature space and fused with the original embeddings through residual
connections [16].

Method | GT localization | mAP (%)
---|---|---
FCLN [20] | ✗ | 5.39
JIVC [41] | ✗ | 9.31
ImgG [26] | ✗ | 9.25
COCD [26] | ✗ | 9.36
COCG [26] | ✗ | 9.82
CAG-Net [42] | ✗ | 10.51
TDC+ROCSU [35] | ✗ | 11.49
GRiT [39] | ✗ | 15.52
ControlCap (Ours) | ✗ | 17.64
FCLN [20] | ✓ | 27.03
JIVC [41] | ✓ | 33.58
CAG-Net [42] | ✓ | 36.29
GRiT [39] | ✓ | 39.99
BLIP2 [24] | ✓ | 37.67
ControlCap (Ours) | ✓ | 42.49
ControlCap $\dagger$ (Ours) | ✓ | 65.81
Table 1: Performance comparison of the proposed approach with the state-of-
the-art methods on the VG V1.0 dataset. $\dagger$ denotes that the first noun
in the caption is given as the linguistic guidance, which is detailed in
“Evaluation Metrics” in Sec. 4.

###  3.4 Controllable Inference

With a ControlCap trained on a dense captioning dataset ($i.e.$, VG [22]), we
can perform controllable inference in specialized scenarios. Before inference,
the object categories ($e.g.$ object detection results) and/or text guidance
($e.g.$ text recognition results) are uniformly fed to the multimodal
embedding generation module as linguistic guidance. According to the guidance,
ControlCap produces text predictions for specialized scenarios. By setting the
linguistic control to be “empty”, ControlCap transforms into a conventional
dense captioning method.

Method | GT localization | mAP (%)
---|---|---
FCLN [20] | ✗ | 5.16
JIVC [41] | ✗ | 9.96
ImgG [26] | ✗ | 9.68
COCD [26] | ✗ | 9.75
COCG [26] | ✗ | 10.39
CAG-Net [42] | ✗ | 10.51
TDC [35] | ✗ | 11.90
GRiT [39] | ✗ | 16.44
CapDet [31] | ✗ | 15.44
ControlCap (Ours) | ✗ | 17.96
GRiT [39] | ✓ | 40.34
BLIP2 [24] | ✓ | 37.88
ControlCap (Ours) | ✓ | 42.86
ControlCap $\dagger$ (Ours) | ✓ | 66.12
Table 2: Performance comparison of the proposed approach with the state-of-
the-art methods on the VG V1.2 dataset.

##  4 Experiment

Implementation Details. ControlCap is implemented upon the BLIP2 [24]
framework, where ViT, LLM and Alignment Network are respectively implemented
using EVA [13], Flan-T5${}_{\text{XL}}$ [6] and Q-former [24], Fig. 2. The
models are trained using 8 NVIDIA A800 GPUs, with the Adam optimizer where the
batch size is set to 128. Without otherwise specified, all models are trained
by 5 epochs and the initial learning rate is set to $1\times 10^{-4}$ with a
cosine learning rate decay. During inference, the beam size of the LLM is set
to 5 and a single caption is generated for each region.

Figure 6: Composition of model parameters. Compared to BLIP2 [24], ControlCap
introduces only 0.18% parameters while outperforms it on VG by 4.98% mAP.

Datasets. ControlCap is trained using detection data ($i.e.$ Object365 [34])
and caption data ($i.e.$, Visual Genome (VG) [22]). Object365 [34] is a large-
scale dataset that consists of 630,000 images and 365 categories of object
annotations. Visual Genome (VG) dataset is a finely labeled dataset with dense
annotations of objects, attributes and relationships. Following [35], we
conduct experiments on VG [22] V1.0, VG V1.2 and VG-COCO [35]. VG has 77,398
images for training, 5000 images for validation and 5000 images for testing.
VG-COCO [35] is the intersection of VG V1.2 and MS COCO [28], which has 38,080
images for training, 2489 for validation and 2476 for testing.

Evaluation Metrics. We follow the setting of [20] to evaluate ControlCap on VG
and VG-COCO. Mean Average Precision (mAP) [20] is adopted as the evaluation
metric. mAP is calculated across a range of thresholds for both localization
and language accuracy, $i.e.$, the intersection over union (IOU) thresholds
(0.3, 0.4, 0.5, 0.6, 0.7) are used for localization and the METEOR score’
thresholds (0, 0.05, 0.1, 0.15, 0.2, 0.25) is adopted for evaluating the
language generation. Since ControlCap lacks the capability to perform object
detection, we utilize a GRiT [39] model trained on the same dataset ($i.e.$,
Visual Genome) to acquire object locations (“GT localization” without “✓” in
Tabs. 1, 2, and 3). To evaluate the object-aware captioning performance
without being affected by the localization performance, we evaluate the model
when ground-truth bounding boxes are given during inference (“GT localization”
with “✓” in Tabs. 1, 2, and 3). Different from previous models, ControlCap is
capable of generating captions given linguistic guidance. To evaluate such
ability, the first noun in the caption is given as the linguistic guidance
during inference (“w guidance” in Tabs. 4 and 6). For example, for the caption
“a black car is parked beside the street”, we provide the word “car” to the
model as linguistic guidance.

###  4.1 Performance

Method | GT localization | mAP (%)
---|---|---
Max Pooling [26] | ✗ | 7.86
COCD [26] | ✗ | 7.92
COCG [26] | ✗ | 8.90
COCG+LocSiz [26] | ✗ | 8.76
TDC+ROCSU [35] | ✗ | 11.9
CapDet [31] | ✗ | 13.98
ControlCap (Ours) | ✗ | 17.68
BLIP2 [24] | ✓ | 36.94
ControlCap (Ours) | ✓ | 41.50
ControlCap $\dagger$ (Ours) | ✓ | 65.55
Table 3: Performance comparison of the proposed approach with the state-of-
the-art methods on the VG-COCO dataset.

Comparison with SOTA Methods. In Tabs. 1 2 3, the performance of ControlCap is
compared with the state-of-the-art (SOTA) methods. ControlCap respectively
achieves 17.64%, 17.96% and 17.68% mAPs on VG V1.0, VG V1.2, and VG-COCO,
outperforming the SOTA methods by significant margins. When ground-truth
bounding boxes are given, ControlCap respectively achieves 42.49%, 42.86% and
41.50% mAPs on VG V1.0, VG V1.2, and VG-COCO, outperforming the finetuned
BLIP2 [24] by 4.98% while introducing only 0.18% parameters (Fig. 6). Last but
not least, when linguistic guidance is given, ControlCap respectively achieves
65.81% mAP and 66.12% mAP on V1.0 and V1.2, which implies that linguistic
guidance can greatly boost the performance of dense captioning.

Qualitative Visualizations. Fig. 7 compares the captioning results of BLIP2
and ControlCap. In the first column and the second row, without context
information, BLIP2 falsely identifies the white surfboard as “part of a
floor”. By introducing context information, ControlCap precisely identifies it
as “part of a board” (2-th column and 2-th row). Due to the scarcity of
training data, ControlCap falsely identifies the red car as “the water is red”
(2-th column and 1-th row). By pretraining with Dense Object365 (Generated by
ControlCap, detailed in “Data Engine” in Sec. 4.2), ControlCap correctly
identifies it as “the car is red” (3-th column and 1-th row).

When linguistic guidance is provided, ControlCap is capable of generating
specialized captions. For example, in the second row of Fig. 7, when there is
no guidance, BLIP2 identifies the people as “a crowd of people watching”,
which is imprecise. When “spectators” is given, ControlCap identifies the
people as “spectators watching the surfers”, which provides more valid
information.

Fig. 8 demonstrates ControlCap’s generalization capability, $e.g.$, generating
specialized captions under diverse linguistic guidance, such as ImageNet with
fine-grained category labels, Object365 with abundant region-category pairs,
and ICDAR2015 with scene text. The ability implies that ControlCap can either
be combined with various datasets to generate domain-specific data or be
combined with specialist models ($e.g.$, classifier, detector, and text
spotter) to form a specialized captioning model.

Table 4: Ablation studies of the components in ControlCap. | CVE | CEE | BEB | mAP (%)
---|---|---|---|---
w/o guidance | w guidance
1 | ✗ | ✗ | ✗ | 37.88 | -
2 | ✓ | ✗ | ✗ | 42.35 | -
3 | ✓ | ✓ | ✗ | 41.98 | 65.11
4 | ✓ | ✓ | ✓ | 42.42 | 65.79
Table 5: Ablation studies of contextual visual embedding. “Region” and “RoI” respectively denotes $X_{r}$ and $X_{c}$ in Sec. 3.2. | Region | RoI | mAP (%)
---|---|---|---
1 | ✓ | ✗ | 37.88
2 | ✗ | ✓ | 35.85
3 | ✓ | ✓ | 42.35
Figure 7: Qualitative comparison of BLIP2 and ControlCap. “with pretraining”
means the model is pretrained on Dense Object365, while “with guidance” means
the caption is generated guided by the underlined word. Figure 8: Qualitative
analysis of the control capabilities of ControlCap. By combining pretrained
ControlCap with datasets that either contains fine-grained category labels
($e.g.$, ImageNet used for classification, ICDAR2015 used for scene text
spotting) or abundant samples ($e.g.$, Object365 used for object detection),
specialized captions can be generated.

###  4.2 Ablation Studies

Baseline. The baseline model is BLIP2 [24]. When training ControlCap, we
finetune the Q-former in BLIP2 on the region-caption pairs cropped from VG or
VG-COCO. The performance of BLIP2 on VG and VG-COCO are shown in line 1 in
Tab. 1 2 3 4. It achieves 37.88% mAP on VG V1.2.

Multimodal Embedding Generation By adding the contextual visual embedding (CVE
in Tab. 4), a performance gain of 4.47% (42.35% $vs.$ 37.88%) can be achieved
in mAP (Line 1-2 in Tab. 4), while dropping the detail-rich region features or
the context-aware RoI features all hurt the performance (Line 1-2 in Tab. 5).
The results imply that fusing the region features of detailed information and
the context-aware RoI features can boost the performance of dense captioning.

By adding conditional entity embedding (CEE in Tab. 4), the model gains the
ability to generate dense captions under linguistic guidance (Line 3 in Tab.
4). However, the performance of ControlCap in mAP drops to 41.98%, suffering
from the diversity of specialized scenarios. We set the dropout ratio $p$ in
CEE to 0.8 based on the results in Fig. 10 (results are obtained after adding
BEB).

Figure 9: Visualizations of attention maps from the bi-directional embedding
bridging module. The words at the bottom left of attention maps are the
linguistic guidance. The generated captions are shown in the last column.
Figure 10: mAP (%) on VG upon dropout ratio $p$.

Bi-directional Embedding Bridging. The module has two branches. On the one
hand, the controllable embedding is enhanced by information from the object
embedding (Top$\rightarrow$Down branch in Fig. 5). On the other hand, the
object embedding is enhanced by information from the controllable embedding
(Down$\rightarrow$Top branch in Fig. 5). By adding the Down$\rightarrow$Top
branch, the mAP of ControlCap improves both with and without guidance (Line
1-2 in Tab. 6). We further visualize the cross attention maps from the cross-
attention layer in the Down$\rightarrow$Top branch in Fig. 9. The activated
regions are highly correlated to the generated captions, demonstrating that
BEB module can guide the object embeddings to focus on the area directed by
the linguistic guidance. By adding both the two branches, the performance of
ControlCap in mAP further improves (Line 3 in Tab. 6). The results imply that
aligning the linguistic guidance with visual embeddings can increase the
model’s adaptability to specialized control.

Data Engine. To verify the feasibility of ControlCap as a data synthesis
engine, we extend Object365 into a new dense captioning dataset, which we
refer to as Dense Object365, which is generated by a pretrained ControlCap
using category labels as linguistic guidance. The data statistic of Dense
Object365 is shown in Tab. 7, which is 3x larger than and shares a similar
mean caption length with VG. Besides, semantic information from Object365 is
mostly kept as 95.18% category labels appear in the generated caption, Tab. 8.
Finally, we use Dense Object365 to pretrain ControlCap, Tab. 9. When we
directly use the combination of VG and Object365 as the pretraining dataset,
mAP on VG drops to a very low level, suffering from the inconsistency of data
type (Line 1-2). When the model is finetuned on VG, the performance gain is
negligible (Line 3). By replacing Object365 with Dense Object365, a large
10.57% performance gain can be achieved, which demonstrates that the
inconsistency of data type can be greatly alleviated (Line 2 and line 4). When
the model is finetuned on VG, a 0.44% performance gain can be achieved, which
implies that samples generated by ControlCap can effectively boost the
performance of the SOTA dense captioner (Line 5).

Table 6: Evaluation of bi-directional embedding bridging module. Top$\rightarrow$Down denotes that controllable embedding is enhanced by information from object embedding and Down$\rightarrow$Top denotes that the object embedding is enhanced by controllable embedding. | Down$\rightarrow$Top | Top$\rightarrow$Down | mAP (%)
---|---|---|---
w/o guidance | w guidance
1 | ✗ | ✗ | 41.98 | 65.11
2 | ✓ | ✗ | 42.26 | 65.78
3 | ✓ | ✓ | 42.42 | 65.79
Table 7: Comparison between Dense Object365 with existing datasets. “MTNum.” denotes the mean token numbers of captions and “Num.” the number of region-text pairs. Dataset | MTNum. | Num. | Instance level
---|---|---|---
VG | 5.34 | 3,682,892 | ✗
Object365 | 1.19 | 9,612,183 | ✓
Dense Object365 | 4.46 | 9,612,183 | ✓
Table 8: Control accuracy, which is defined as the proportion of successfully controlled captions to all captions. A successfully controlled caption is supposed to contain the word used to control. | ImageNet | Object365 | ICDAR2015
---|---|---|---
Acc (%) | 86.13 | 95.18 | 80.93
Table 9: Ablation studies of the pretraining strategy. | Pretraining | Finetuning | mAP (%)
---|---|---|---
1 | - | VG | 42.42
2 | VG, Object365 | - | 28.90
3 | VG, Object365 | VG | 42.30
4 | VG, Dense Object365 | - | 39.47
5 | VG, Dense Object365 | VG | 42.86

##  5 Conclusion

We proposed ControlCap, a new dense captioning paradigm with expanded capacity
to accommodate specialized scenarios and user intention. We designed
ControlCap upon a multimodal embedding generation (MEG) module and a bi-
directional embedding bridging (BEB) module, which facilitates object-aware
understanding and addresses misalignment issues caused by various types of
embeddings. ControlCap is validated to be effective for specialized and
precise dense captioning. It sets a solid baseline for the challenging dense
captioning task from the perspective of user control.

## References

  * Cha [2022] Introducing chatgpt. https://openai.com/blog/chatgpt.  2022\.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan.  Flamingo: a visual language model for few-shot learning.  In _NeurIPS_ , 2022.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In _NeurIPS_ , 2020.
  * Carlsson et al. [2022] Fredrik Carlsson, Joey Öhman, Fangyu Liu, Severine Verlinden, Joakim Nivre, and Magnus Sahlgren.  Fine-grained controllable text generation using non-residual prompting.  In _ACL_ , pages 6837–6857, 2022.
  * Chen et al. [2023] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao.  Shikra: Unleashing multimodal llm’s referential dialogue magic.  _arXiv preprint arXiv:2306.15195_ , 2023.
  * Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Dai et al. [2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _arXiv preprint arXiv:2305.06500_ , 2023.
  * Dathathri et al. [2020] Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu.  Plug and play language models: A simple approach to controlled text generation.  In _ICLR_ , 2020.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _NAACL_ , pages 4171–4186, 2019.
  * Ding et al. [2024] Ning Ding, Chaorui Deng, Mingkui Tan, Qing Du, Zhiwei Ge, and Qi Wu.  Image captioning with controllable and adaptive length levels.  _IEEE TPAMI_ , pages 764–779, 2024.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , 2021.
  * Fan et al. [2018] Angela Fan, Mike Lewis, and Yann N. Dauphin.  Hierarchical neural story generation.  In _ACL_ , pages 889–898, 2018.
  * Fang et al. [2023] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva: Exploring the limits of masked visual representation learning at scale.  In _IEEE CVPR_ , pages 19358–19369, 2023.
  * GAGE [1994] P GAGE.  A new algorithm for data compression.  _The C Users Journal_ , pages 23–38, 1994.
  * Girshick [2015] Ross Girshick.  Fast r-cnn.  In _IEEE ICCV_ , pages 1440–1448, 2015.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _IEEE CVPR_ , pages 770–778, 2016.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _IEEE ICCV_ , pages 2961–2969, 2017.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , pages 1735–1780, 1997.
  * Hu et al. [2023] Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A. Smith, and Jiebo Luo.  Promptcap: Prompt-guided image captioning for vqa with gpt-3.  In _IEEE ICCV_ , pages 2963–2975, 2023.
  * Johnson et al. [2016] Justin Johnson, Andrej Karpathy, and Li Fei-Fei.  Densecap: Fully convolutional localization networks for dense captioning.  In _IEEE CVPR_ , pages 4565–4574, 2016.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick.  Segment anything.  In _IEEE ICCV_ , pages 4015–4026, 2023.
  * Krishna et al. [2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _IJCV_ , pages 32–73, 2017.
  * Li et al. [2022a] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi.  BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _ICML_ , pages 12888–12900, 2022a.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi.  BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.  In _ICML_ , pages 19730–19742, 2023.
  * Li et al. [2020] Piji Li, Haisong Zhang, Xiaojiang Liu, and Shuming Shi.  Rigid formats controlled text generation.  In _ACL_ , 2020.
  * Li et al. [2019] Xiangyang Li, Shuqiang Jiang, and Jungong Han.  Learning object context for dense captioning.  In _AAAI_ , pages 8650–8657, 2019.
  * Li et al. [2022b] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto.  Diffusion-lm improves controllable text generation.  In _NeurIPS_ , 2022b.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _ECCV_ , pages 740–755, 2014.
  * Liu et al. [2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023.
  * Liu et al. [2021] Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi.  Mitigating political bias in language models through reinforced calibration.  In _AAAI_ , pages 14857–14866, 2021.
  * Long et al. [2023] Yanxin Long, Youpeng Wen, Jianhua Han, Hang Xu, Pengzhen Ren, Wei Zhang, Shen Zhao, and Xiaodan Liang.  Capdet: Unifying dense captioning and open-world detection pretraining.  In _IEEE CVPR_ , pages 15233–15243, 2023.
  * Peng et al. [2023] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei.  Kosmos-2: Grounding multimodal large language models to the world.  _arXiv preprint arXiv:2306.14824_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _ICML_ , pages 8748–8763, 2021.
  * Shao et al. [2019] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.  Objects365: A large-scale, high-quality dataset for object detection.  In _IEEE ICCV_ , pages 8430–8439, 2019.
  * Shao et al. [2022] Zhuang Shao, Jungong Han, Demetris Marnerides, and Kurt Debattista.  Region-object relation-aware dense captioning via transformer.  _IEEE TNNLS_ , 2022.
  * Song et al. [2021] Haoyu Song, Yan Wang, Kaiyan Zhang, Wei-Nan Zhang, and Ting Liu.  Bob: BERT over BERT for training persona-based dialogue models from limited personalized data.  In _ACL_ , pages 167–177, 2021.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _NeurIPS_ , 2017.
  * Wu et al. [2022] Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang.  Grit: A generative region-to-text transformer for object understanding.  _arXiv preprint arXiv:2212.00280_ , 2022.
  * Wu et al. [2023] Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen.  Datasetdm: Synthesizing data with perception annotations using diffusion models.  _NeurIPS_ , 2023.
  * Yang et al. [2017] Linjie Yang, Kevin Tang, Jianchao Yang, and Li-Jia Li.  Dense captioning with joint inference and visual context.  In _IEEE CVPR_ , pages 2193–2202, 2017.
  * Yin et al. [2019] Guojun Yin, Lu Sheng, Bin Liu, Nenghai Yu, Xiaogang Wang, and Jing Shao.  Context and attribute grounded dense captioning.  In _IEEE CVPR_ , pages 6241–6250, 2019.
  * You et al. [2023] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang.  Ferret: Refer and ground anything anywhere at any granularity.  _arXiv preprint arXiv:2310.07704_ , 2023.
  * Zhang et al. [2022a] Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song.  A survey of controllable text generation using transformer-based pre-trained language models.  _ACM Computing Surveys_ , 2022a.
  * Zhang et al. [2022b] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  OPT: open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022b.
  * Zhang et al. [2023] Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo.  Gpt4roi: Instruction tuning large language model on region-of-interest.  _arXiv preprint arXiv:2307.03601_ , 2023.
  * Zhao et al. [2023] Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, et al.  Chatspot: Bootstrapping multimodal llms via precise referring instruction tuning.  _arXiv preprint arXiv:2307.09474_ , 2023.
  * Zheng et al. [2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.

Supplementary Material

Table 10: Detailed architecture parameters for different module. $N$ is the number of tokens of the noun proposal in the Conditional Entity Embedding. Vision Transformer (ViT) | Value
---|---
type of module |  ViT-g EVA [13]
size of the input image | (224, 224)
size of the region clip | (224, 224)
size of output visual feature | (16, 16, 1408)
Contextual Visual Embedding (CVE) | Value
output size of RoI-align module | (16, 16)
dimension of the region feature | (16, 16, 1408)
dimension of the RoI feature | (16, 16, 1408)
dimension of the object embedding | (16, 16, 1408)
Conditional Entity Embedding (CEE) | Value
dropout ratio | 0.8
type of tokenizer | WordPiece
dimension of memory | (2048, )
dimension of the controllable embedding |  ($N$, 2048)
Bi-directional Embedding Bridging (BEB) | Value
dimension of the latent space | (128, )
number of heads in the cross-attention layer | (8, )
Alignment Network | Value
type of module |  Q-former [24]
Large Language Model (LLM) | Value
type of module |  Flan-T5${}_{\text{XL}}$ [6]
Table 11: Detailed hyperparameters during training and inference. Training | Value
---|---
batch size | 128
learning policy | cosine annealing
initial learning rate |  1$e$-4
minimum learning rate | 0
weight decay ratio | 0.05
warmup steps | 5000
Inference | Value
number of beams | 5

##  6 Detailed Experimental Settings

Settings of modules. The detailed architecture parameters for different
modules of ControlCap are shown in Tab. 10.

Settings of training and inference. The detailed hyperparameters during
training and inference are shown in Tab. 11.

Combining with existing detectors. Since ControlCap lacks the capability to
perform object detection, we utilize a GRiT [39] model trained on the same
dataset to acquire object locations. During the inference stage, we keep the
bounding box and object score predicted by GRiT, and then replace the
predicted caption by ControlCap to get the final result.

Model Pretraining. ControlCap is pretrained on the combination of VG [22] and
Dense Object365 for 1 epoch following the setting in Tab. 11. Then, ControlCap
is finetuned on VG for 5 epochs. More epochs can lead to model overfitting,
resulting in a decline in performance.

##  7 Additional Experimental Results

Table 12: Ablation studies of the pretrained Q-former [24]. Experiments are conducted on VG V1.2 and the ground truth bounding boxes are given. | Method | pretrained Q-former | mAP (%)
---|---|---|---
1 |  BLIP2 [26] | ✓ | 37.88
2 | ControlCap | ✗ | 40.93
3 | ControlCap | ✓ | 42.42

Pretrained Q-former. Q-former has a transformer encoder structure. BLIP2 [24]
pretrains it using massive image-text pairs. It employs three types of losses,
namely Image-Text Contrastive Learning, Image-grounded Text Generation, and
Image-Text Matching, which ensures that the features extracted by Q-former is
effectively aligned to the language feature space. As shown in line 2 in Tab.
12, when equipped with Q-former without pretraining, ControlCap outperforms
BLIP2 3.05% in mAP. When equipped with pretrained Q-former, ControlCap
outperforms BLIP2 4.54% in mAP (Line 3 in Tab. 12), demonstrating that a
better alignment network can boost the performance of ControlCap.

| BLEU-4 | METEOR | CIDEr | CLIP score
---|---|---|---|---
BLIP2 [23] | 11.1 | 16.0 | 141.2 | 28.5
GRiT [39] | 10.8 | 16.3 | 136.9 | 28.1
ControlCap | 12.0 | 17.5 | 157.2 | 28.5
Table 13: Caption quality analysis on the VG V1.2 dataset. | ImageNet-1K | Object365 | ICDAR2015
---|---|---|---
GPT-4v | (80 / 8 / 12) | (54 / 5 / 41) | (73 / 6 / 21)
Table 14: GPT4-v evaluation under different scenes. GPT-4v is employed to
mimic human preferences for captions generated by ControlCap.
$(N_{1}/N_{2}/N_{3})$ indicates the frequency with which GPT-4v assesses that
(controlled caption is better / uncontrolled caption is better / both captions
are of equal quality).

Caption quality analysis is presented in Tab. 13. ControlCap significantly
outperforms BLIP2 and GRiT when using BLEU-4, METEOR and CIDEr as caption
quality analysis metrics. For the CLIP score which is computed between the
image regions111Bounding box region cropped from the entire image. and the
predicted dense captions, it would be worse when the captions contain
contextual information ($e.g.$ object outside the image regions). Therefore it
is not suitable for evaluating region-level captioning methods, especially for
those (GRiT and ControlCap) with the ability to capture contextual
information. Under this condition, ControlCap is comparable to BLIP2 and
outperforms GRiT.

GPT-4v evaluation is presented in Tab. 14. We utilized GPT-4v as an objective
and impartial human study agent. We provide GPT-4v 100 images for each
scenario with white rectangular borders and use prompt “Caption1: {cap1},
Caption2: {cap2}. Please compare the professionalism and accuracy of the two
captions based on the white rectangular region in the pictures. Choose from
the following three options: 1. Caption1 is better. 2. Caption2 is better. 3.
They are equally good.” {cap1} and {cap2} are tested captions. {cap1} is
generated by ControlCap with linguistic guidance ($e.g.$, text or category).
{cap2} is generated by ControlCap without control. It can seen that the
quality of captions with user control is significantly better than that
without user control in various scenarios.

##  8 Additional Visualization Results

Figure 11: Cases that ControlCap fails to control. The words at the bottom
left of the figures are the linguistic guidance. Figure 12: Visualizations of
attention maps from the bi-directional embedding bridging module. The words at
the bottom left of attention maps are the linguistic guidance. The generated
captions are shown in the last column.

Extra results. We provide additional visualization results of Fig. 8 and Fig.
9 in the main document. The results are shown in Fig. 12, Fig. 13, Fig. 14,
Fig. 15, Fig. 16, and Fig. 17 respectively.

Failure cases of ControlCap. Fig. 11 shows some common cases that ControlCap
fails to control. We summarize the cases into two types: 1) ControlCap may
modifies the original linguistic, and then generate captions. In Fig. 11
(left), ControlCap modifies the original linguistic “canned” into “cans”, and
then generate caption “four cans of beer”. 2) ControlCap may ignores special
symbols. In Fig. 11 (right), ControlCap ignores symbol “&” in the original
linguistic “j&d”.

Figure 13: Qualitative analysis of the control capabilities of ControlCap on
ImageNet. The caption is generated guided by the underlined word. Figure 14:
Qualitative analysis of the control capabilities of ControlCap on ICDAR2015.
The caption is generated guided by the underlined word. Figure 15: Qualitative
analysis of the control capabilities of ControlCap on Object365. The caption
is generated guided by the underlined word. Figure 16: Qualitative analysis of
the control capabilities of ControlCap on Object365. The caption is generated
guided by the underlined word. Figure 17: Qualitative analysis of the control
capabilities of ControlCap on Object365. The caption is generated guided by
the underlined word.


Chemical] Department of Chemical Engineering, Massachusetts Institute of
Technology, 25 Ames St, Cambridge, Massachusetts 02142 USA Chemical]
Department of Chemical Engineering, Massachusetts Institute of Technology, 25
Ames St, Cambridge, Massachusetts 02142 USA [Math] Department of Mathematics,
Massachusetts Institute of Technology, 182 Memorial Drive, Cambridge,
Massachusetts 02142 USA

# Continuum theory of electrostatic correlations at charged surfaces

J. Pedro de Souza  [  Martin Z. Bazant  [<EMAIL_ADDRESS>
###### Abstract

The standard model for diffuse charge phenomena in colloid science,
electrokinetics and biology is the Poisson-Boltzmann mean-field theory, which
breaks down for multivalent ions and large surface charge densities due to
electrostatic correlations. In this paper, we formulate a predictive continuum
theory of correlated electrolytes based on two extensions of the Bazant-
Storey-Kornyshev (BSK) framework: (i) a physical boundary condition enforcing
continuity of the Maxwell stress at a charged interface, which upholds the
Contact Theorem for dilute primitive-model electrolytes, and (ii) scaling
relationships for the correlation length, for a one-component plasma at a
charged plane and around a cylinder, as well as a dilute z:1 electrolyte
screening a planar surface. In these cases, the theory accurately reproduces
Monte Carlo simulation results from weak to strong coupling, and extensions
are possible for more complex models of electrolytes and ionic liquids.

##  1 Introduction

Electrostatic correlations can significantly affect the structure and
thermodynamic properties of the electrical double layer 1, 2, resulting in
qualitative differences from mean-field Poisson-Boltzmann (PB) theory, such as
like-charge attraction 3, 4 or over-screening of surface charge. Critical
applications in biology, colloids, separations, or electrochemistry rely on or
operate in the regime where correlation effects are critical.

Numerous models have been proposed to capture electrostatic correlations,
typically with a complicated mathematical structure. Outwaithe derived a
modified Poisson Boltzmann models to account for the fluctuation potential of
a single ion interacting with a charged wall 5. The hypernetted chain
approximation and mean spherical approximation closure to the Ornstein-Zernike
equation involve solving integral equations involving the direct correlation
functions of bulk charged spheres 6, 7, 8, 9 for the equilibrium structure.
Further work based on classical Density Functional Theory determines
equilibrium properties based on the minimization of an integro-differential
free energy functional. Kierlik and Rosinberg implemented a model (termed the
bulk fluid model 10) which captures correlations based on an perturbative
expansion of density with direct correlation functions as an input from the
mean spherical approximation 11. Voudkavinova et al. analyzed the accuracy of
two other related density functional theories (reference fluid density 12,
functionalized mean spherical approximation 13, and bulk fluid 11, 14) in
comparison to Monte Carlo simulations, finding the reference fluid density
approach to be most accurate 10. These theories implement the accurate
Fundamental Measure Theory functional developed by Rosenfeld to describe
excluded volume effects 15, 16. The additional electrostatic interactions
beyond mean field are included in the excess free energy separately, without
any modification to the mean-field electrostatic part of the free energy.

While these approaches often produce accurate density profiles, they can be
involved to implement to a broad range of applications, for different
geometries or dynamic problems, especially compared to the classical PB
theory. To our knowledge, they also have not yet been shown to recover the
correlated behavior of the counterion-only limit for counterions of
infinitesimal size. A simpler mathematical structure could help with the
application and interpretation of electrostatic correlations to a wider class
of problems in physics, including electrokinetics, colloidal interactions, and
electrochemical transport/reactions.

Figure 1: The scenarios considered in the application of the BSK theory.

Bazant, Storey and Kornyshev (BSK) proposed a continuum framework to account
for the nonlocal dielectric permittivity of ionic liquids resulting from ion-
ion correlations 17 with a simple mathematical structure, building on
intermediate coupling approximations of Santangelo 18 and Hatlo and Lue 19 for
the one-component plasma. The model captures correlations based on expansions
in terms of electric field, rather than ion density, in the free energy
functional which leads to a higher-order Poisson equation. In so doing, the
electrostatic correlations are included self-consistently in the definition of
the electrostatic potential whose gradient determines the electrostatic force
on an ion in the diffuse layer. The BSK theory provides a simple framework to
predict charge density oscillations and over-screening phenomena in a variety
of electrokinetic , electrochemical, biophysical, and colloidal phenomena in
electrolytes and ionic liquids. The equations require a similar level of
complexity to solve compared to the PB theory, which allows them to be applied
to a broad group of applications. The theory was used to describe
electrosmotic 20 and electrophoretic mobility 21 reversals in multivalent and
concentrated electrolytes, as well as electroconvective instabilities in ionic
liquids 22. It was also applied to the dynamics 23, 24, 25, 26, 27 and
electrosorption 28, 29, 30, 31 at electrochemical interfaces for ionic liquids
and concentrated solvent-in-salt electrolytes, including storage 32 and
transport 33 in nanoporous media. Electrostatic correlations have a profound
effect on colloidal interactions4, 34, 35, where they can induce like-charge
attraction in multivalent electrolytes, also predicted by the BSK theory. The
activity and solvation energy of electrolytes at high concentration was also
studied including the electrostatic correlation effect 36, 37, 38, 39, 40 as
well as the extent of ion pairing in confinement41. Finally, the electrostatic
correlations given by the BSK theory were important in describing the
conduction through biological ion channels 42, 43, 44, 45, 46. Despite the
numerous applications, fundamental questions remain about the proper boundary
conditions and correlation length required to complete the BSK theory.

Here, we show that the appropriate boundary constraint for the higher-order
Poisson equation is based on an interfacial stress balance. With corrected
boundary conditions, the BSK theory becomes exact in the strong and weak
coupling limits for the one-component plasma, and agrees with Monte Carlo (MC)
simulations in intermediate coupling. We also suggest scaling relationships
for the correlation length without steric constraints in one-component plasma
and in multivalent electrolytes, for all the scenarios in Fig. 1. We show how
the correlation length can have a simple physical interpretation based on the
correlation hole size of counterions at a charged surface. Although
generalizations are possible, we restrict the analysis to a restricted
primitive electrolyte with hard, spherical ions of equal size in a constant
permittivity, $\epsilon$, medium and smeared out surface charge density,
$q_{s}$, and neglect all concentrated-solution effects, so as to isolate
electrostatic correlations.

##  2 Theory

The BSK free energy functional is given by:

$\begin{split}G=&\int_{V}d\textbf{r}\,\left\\{g+\rho\phi-\frac{\epsilon}{2}\left[\left(\nabla\phi\right)^{2}+\ell_{c}^{2}\left(\nabla^{2}\phi\right)^{2}+\dots\right]\right\\}\\\
&+\oint_{S}d\textbf{r}_{\textbf{s}}\,q_{s}\phi.\end{split}$ (1)

Here, $g=(H-TS)/V$ is the enthalpy and entropy density, $\rho$ is the charge
density, and $\phi$ is the electrostatic potential. For simplicity, the free
energy is truncated after the first correlation contribution, although higher
order terms can be considered 19. While the original authors performed a
gradient expansion to arrive at Eq. 1 17, 20, the mathematical procedure is
equivalent to modifying the interaction potential between ions from
$U_{\alpha\beta}(r)=z_{\alpha}z_{\beta}\ell_{B}\mid r\mid^{-1}$ to
$U_{\alpha\beta}(r)=z_{\alpha}z_{\beta}\ell_{B}\mid r\mid^{-1}(1-e^{-\mid
r\mid/\ell_{c}})$ 25, 18. The modified interaction potential is solved in the
mean-field limit. Thus the BSK theory is a phenomenological correction to PB
within the mean-field approximation, by subtracting out interactions with
smeared out charges within a correlation length $\ell_{c}$, which should scale
as the size of the correlation hole. The modified Poisson equation results by
finding the extremal of the functional ($\frac{\delta G}{\delta\phi}=0$):

$\epsilon(\ell_{c}^{2}\nabla^{2}-1)\nabla^{2}\phi=\rho.$ (2)

Eq. 2 is a statement of Maxwell’s equation, $\nabla\cdot\mathbf{D}=\rho$ where
the displacement field is $\mathbf{D}=\hat{\epsilon}\mathbf{E}$ with a non-
local permittivity operator
$\hat{\epsilon}=\epsilon(\ell_{c}^{2}\nabla^{2}-1)$ applied on the electric
field, $\mathbf{E}=-\nabla\phi$, in a medium of constant permittivity,
$\epsilon$. PB theory is recovered when $\ell_{c}=0$. Note that the definition
of electrostatic potential itself has changed by adding the higher order
correlation terms, without violating Maxwell’s equation. In other words, the
potential that determines the energy of an ion in the double layer must
satisfy $\nabla\cdot\left(\hat{\epsilon}\mathbf{E}\right)=\rho$ rather than
$\nabla\cdot\left(\epsilon\mathbf{E}\right)=\rho$. In this way, the
electrostatic correlation contribution to the electrostatic energy for an ion
is included self consistently within the electrostatic framework, rather than
being added as additional corrections in the excess chemical potential. An
advantage of this approach is that the electrostatic force per unit charge of
an ion is captured directly with $\mathbf{E}$, meaning that the diffuse
potential $\phi$ here could be measured experimentally at an electrode (if
also accounting for the potential drop in the Stern layer).

The charge density at equilibrium, $\rho=\sum_{i}z_{i}ec_{i}$, will be
determined by the constraint that the electrochemical potential for each ion
is a constant at equilibrium. The electrochemical potential can be defined as
the variation of the Gibbs free energy with respect to concentration 47,
$\mu_{i}=\frac{\delta G}{\delta c_{i}}$, or

$\mu_{i}=\mu_{i}^{\theta}+kT\ln(c_{i})+z_{i}e\phi+\mu_{i}^{\mathrm{ex}}$ (3)

where the first term is a reference value, the second term is the ideal
entropy contribution, the third term is the electrostatic potential
contribution, and the fourth term is the excess electrochemical potential.

The first open question in applying BSK theory is that of additional boundary
conditions, beyond Maxwell’s equation $\hat{n}\cdot\mathbf{D}=q_{s}$.
Presumably, the boundary condition must take care of the unaccounted short-
range part of $U_{\alpha\beta}$. In the original BSK formulation and all
subsequent works, the boundary condition of $\hat{n}\cdot\nabla^{3}\phi=0$ was
applied, with the justification that the correlation effects should disappear
at the interface 20, 25, 24, 33, 41, 48, 21, 34, 42, 26, 28, 27, 29, 49. The
theory provided reasonable agreement to simulation and experimental results
for ionic liquids and multivalent electrolytes. However, the boundary
conditions have not yet been proved or validated systematically.

The second open question is the choice of correlation length, which has been
arbitrarily set to the Bjerrum length for electrolytes 17, 21, and the ion
diameter for ionic liquids 17. The theory is ultimately very sensitive to the
choice of boundary conditions and correlation length. Here, we analyze the
boundary condition in terms of a stress balance at the interface and then
validate $\ell_{c}$ by comparison to MC simulations.

###  2.1 Interfacial balance

Applying the Gibbs-Duhem equation at constant temperature to the electrolyte
and screened surface charges, following 50, and neglecting the external
electrostatic work done on the system, gives $dP=\sum_{i}c_{i}d\mu_{i}$.
Taking the gradient in three-dimensional space and applying the definition of
the electrochemical potential:

$-\mathbf{f}=\nabla P=kT\sum_{i}\nabla
c_{i}+\rho\nabla\phi+\sum_{i}c_{i}\nabla\mu_{i}^{\mathrm{ex}},$ (4)

where $\mathbf{f}$ is the total thermodynamic force. The first and third terms
on the RHS of Eq. 4 correspond to the gradient of osmotic pressure,
$\nabla\Pi$. For an ideal solution, $\mu_{i}^{\mathrm{ex}}$=0. The gradient of
the defined thermodynamic pressure is equivalent to the divergence of the
total stress tensor of the electrolyte system,
$\mathbf{f}=\nabla\cdot{\mathbf{\uptau}}$. The total stress tensor is composed
of an osmotic pressure component, $\Pi$ and a Maxwell stress tensor,
$\mathbf{\uptau}_{e}$, such that
$\mathbf{\uptau}=-\Pi\,\mathrm{I}+\mathbf{\uptau}_{e}$. The component of
interest in this analysis, $\mathbf{\uptau}_{e}$, can be defined by:

$\nabla\cdot\mathbf{\uptau}_{e}=\rho\mathbf{E}=\nabla\cdot(\hat{\epsilon}\mathbf{E})\mathbf{E}$
(5)

in a constant $\epsilon$ medium. Plugging in for the charge density using the
BSK Eq. 2 and performing integration by parts, one arrives at an expression
for the Maxwell stress tensor for a fluid with a non-local permittivity,

$\begin{split}\mathbf{\uptau}_{e}=&\epsilon\mathbf{EE}-\frac{1}{2}\epsilon\mathbf{E}^{2}\,\mathbf{I}+\epsilon{\ell_{c}}^{2}\Big{[}\left(\mathbf{E}\cdot\nabla^{2}\mathbf{E}\right)\,\mathbf{I}-\mathbf{E}\left(\nabla^{2}\mathbf{E}\right)\\\
&-\left(\nabla^{2}\mathbf{E}\right)\mathbf{E}+\frac{1}{2}\left(\nabla\cdot\mathbf{E}\right)^{2}\,\mathbf{I}\Big{]},\end{split}$
(6)

as derived in the Supporing Information. While the above equation was derived
for constant $\epsilon$ and $\ell_{c}$, the expression is identical if these
parameters vary. For varying $\epsilon$ or $\ell_{c}$, the Korteweg-Helmholtz
force density must be included in the electrostatic stress 51,

$\nabla\cdot\mathbf{\uptau}_{e}=\rho\mathbf{E}-\frac{1}{2}\mathbf{E}^{2}\nabla\epsilon+\frac{1}{2}(\nabla\cdot\mathbf{E})^{2}\nabla(\epsilon{\ell_{c}}^{2}).$
(7)

Within the distance of closest approach of the ions to the surface,
correlations cannot affect the value of the Maxwell stress at the surface,
$\mathbf{\uptau}_{e,\mathrm{surf}}$, generated by the surface charge density.
The mechanical equilibrium problem therefore requires continuity in the
electrostatic stress tensor evaluated in the solution and at the surface,

$\mathbf{\uptau}_{e}-\mathbf{\uptau}_{e,\mathrm{surf}}=0.$ (8)

At a uniformly-charged, flat surface without a dielectric jump, the Maxwell
stress tensor is simply
$\mathbf{\uptau}_{e,\mathrm{surf}}={q_{s}}^{2}/(2\epsilon)\hat{n}\hat{n}$,
while the Maxwell stress tensor in the electrolyte is given by Eq. 6. Equating
these two expressions, and substituting in $\hat{n}\cdot\mathbf{D}=q_{s}$ we
arrive at a final boundary for a potential varying in one coordinate
direction:

$\hat{n}\cdot\ell_{c}\nabla^{3}\phi=\nabla^{2}\phi\Big{\rvert}_{S}$ (9)

applied at the distance of closest approach of the ion with the wall.

The method amounts to applying the Contact Theorem to the correlated
electrolyte in the absence of correlations, shown below for
$\mu_{i}^{\mathrm{ex}}=0$ at a flat electrode with constant charge density
without a dielectric jump 52, 53:

$P=-\frac{q_{s}^{2}}{2\epsilon}+kT\sum_{i}c_{i}\Big{\rvert}_{S}=-\hat{n}\cdot\mathbf{\uptau}_{e}\cdot\hat{n}+kT\sum_{i}c_{i}\Big{\rvert}_{S}.$
(10)

The Contact Theorem is a statement of mechanical equilibrium, where the
repulsive osmotic pressure contribution is balanced by the electrostatic
attraction from the Maxwell stress. Without the constraint from Eq. 8, the BSK
theory does not obey this simple relationship which should be valid even for
dilute electrolytes in the primitive model 54, 55, 56. The procedure of
ensuring continuity in the Maxwell stress can be repeated for any higher order
$\hat{\epsilon}$ by equating the $\mathbf{\uptau}_{e}$ at successive orders of
derivatives. The condition in Eq. 8 is also applicable to any extended
electrolyte mean-field theory with arbitrary models of concentrated solution
activity and solvent polarizability, including interactions with a soft wall.
The approach may even be extended to media with non-local dielectric constant
$\hat{\epsilon}$ driven by solvent polarization 57, 58.

###  2.2 Correlation length scaling

At a highly charged surfaces, the charge-charge correlations are dominated by
the mutual repulsion of counterions at the interface, as demonstrated in the
schematic in Fig. 1(a). The size of a correlation hole of counterions forming
a Wigner crystal is characterized by a length scale $R_{\mathrm{hole}}$:

$R_{\mathrm{hole}}=\left(\frac{1}{6}\right)^{1/4}\left(\frac{ze}{q_{s}}\right)^{1/2}$
(11)

For this work, we assume that the correlation length scales as the size of a
correlation hole, determined by the surface charge denisty:

$\ell_{c}=\alpha R_{\mathrm{hole}}$ (12)

Figure 2: BSK theory compared to MC simulations from 54 with $\alpha=0.50$ for
counterions screening a charged isolated surface. The solid lines are the
predictions of the BSK theory with the boundary condition of
$\hat{n}\cdot\ell_{c}\nabla^{3}\phi=\nabla^{2}\phi\Big{\rvert}_{S}$, the
dashed-dotted lines are the predictions of the BSK theory with the boundary
condition of $\hat{n}\cdot\ell_{c}\nabla^{3}\phi=0$ and the markers are from
the MC simulations. Strong coupling limits are plotted as black dashed lines.
(a) The charge density is plotted as a function of distance from an isolated
surface. (b) The charge density difference relative to the solution to the PB
theory as a function of distance from an isolated surface.

with one parameter $\alpha$, which is considered to be a constant. We will
demonstrate that the fitted scaling with $\alpha=0.50$ works well from the
limit of zero reservoir concentration (one-component plasma) to more
concentrated electrolytes. At high concentration or at low surface charge
densities, this scaling argument breaks down, and the other length scales
might dominate in the correlation length. For example, if the surface charge
tends to zero, then the charge-charge correlations will be dominated by the
Bjerrum length and characteristic mean spacing between ions given by the bulk
concentration. At very large charge densities and for large ion sizes, where
$R_{\mathrm{hole}}$ becomes comparable to $a$, the ion diameter, the ion size
can dominate in determining the charge-charge correlations due to over-
crowding effects. Also, if other length scales are introduced, such as surface
curvature, the correlation length between ions can also be affected, as
demonstrated with the one-component plasma around a thin charged cylinder.

In the Supporting Information, the non-arbitrary scaling of the correlation
length is investigated by comparison to Grand Canonical Monte Carlo
simulations from ref. 59 for a $z:1$ electrolyte. Using the Buckingham-$\Pi$
theorem, we know that the dimensionless correlation length is related to a
power law relationship, at least for small or large values of the
dimensionless variables. Here we choose four length scales from which we can
construct three dimensionless groups: $\ell_{c}$, $\ell_{\mathrm{GC}}$,
$z^{2}\ell_{B}$, and $\lambda_{D}$, such that the powerlaw relationship can be
expressed as:

$\delta_{c}=\alpha_{2}\left(\frac{z^{2}\ell_{B}}{\ell_{\mathrm{GC}}}\right)^{\alpha_{3}}\left(\frac{z^{2}\ell_{B}}{\lambda_{D}}\right)^{\alpha_{4}},$
(13)

The correlation length scaling that arises from the fitting procedure to the
MC data is given by:

$\ell_{c}\thicksim{\ell_{B}}^{1/4}(q_{s}/e)^{-1/8}{C_{\mathrm{ref}}}^{-1/6},$
(14)

with a fitted scale of:

$\delta_{c}=0.35\left(\frac{z^{2}\ell_{B}}{\ell_{\mathrm{GC}}}\right)^{-1/8}\left(\frac{z^{2}\ell_{B}}{\lambda_{D}}\right)^{2/3}.$
(15)

Note that the fitted exponents are expressed in terms of fractions to
emphasize their relationship to the intrinsic length scales in the system.

A detailed investigation of how correlations are affected as a function of
surface charge density, ion valency, concentration, ion size, and surface
curvature could motivate a more nuanced scaling of the correlation length,
based on direct analysis of the charge-charge correlation function, including
variations in the correlation length as a function of the distance from a
charged surface. In this work, we isolate the electrostatic correlation
effects for a dilute electrolyte at highly charged surfaces. For the purposes
of the analysis in the main text, we will assume the correlation length to be
$\ell_{c}=0.50R_{\mathrm{hole}}$ for all the scenarios investigated in Fig. 1.
For the results with the fitted correlation length scaling in 15, one can
refer to the Supporting Information.

###  2.3 One-component plasma

Considering a system of point-like counterions neutralizing a uniformly
charged surface, the importance of correlations is governed by a coupling
constant,

$\Xi=2\pi z^{3}{\ell_{B}}^{2}q_{s}/e,$ (16)

which is a measure of the correlation hole size, $R_{\mathrm{hole}}$, compared
to the characteristic ion distance from the surface, the Gouy-Chapman length,

$\ell_{\mathrm{GC}}=e(2\pi z\ell_{B}q_{s})^{-1}$ (17)

such that $\Xi\sim{R_{\mathrm{hole}}}^{2}/{\ell_{\mathrm{GC}}}^{2}$. Here, $z$
is the ion valency, $\ell_{B}$ is the Bjerrum length, and $e$ is an elementary
charge. In the weak coupling limit ($\Xi<<1$), PB theory is valid. In the
strong coupling limit ($\Xi>>1$), counterions interact with the electric
potential created by the surface since ion-surface interactions dominate 54,
60, 55.

Now we consider applying the mechanical constraint, starting with the one-
component plasma of infinitesimally small size with $\mu^{\mathrm{ex}}=0$. The
one-component plasma consists of a single mobile ionic species which
neutralizes the charge of a smeared out surface charge density. We can non-
dimensionalize lengths with the Gouy-Chapman length, the potential by the
thermal voltage for the counterion, $\phi_{0}=\frac{kT}{ze}$, and the charge
density by $\rho_{0}=2\pi\ell_{B}q_{s}^{2}e^{-1}$. Here, we assume that the
correlation length scales with the size of a correlation hole at the surface,
$\delta_{c}=\ell_{c}/\ell_{\mathrm{GC}}=\alpha_{0}\sqrt{\Xi}$ Using
$\thicksim$ to denote non-dimensionalized variables:

${\alpha_{0}}^{2}\Xi\tilde{\nabla}^{4}\tilde{\phi}-\tilde{\nabla}^{2}\tilde{\phi}=2\tilde{\rho}=2e^{-\tilde{\phi}}$
(18)

with boundary conditions of

$\begin{split}&\hat{n}\cdot({\alpha_{0}}^{2}\Xi\tilde{\nabla}^{3}\tilde{\phi}-\tilde{\nabla}\tilde{\phi})=-2\\\
&\hat{n}\cdot\alpha_{0}\sqrt{\Xi}\tilde{\nabla}^{3}\tilde{\phi}=\tilde{\nabla}^{2}\tilde{\phi}\end{split}$
(19)

at $\tilde{x}=0$, where $\alpha_{0}=1.36\alpha$ is an order one constant
proportional to $\alpha$. Therefore, the importance of the higher order
derivative is governed by the coupling parameter, $\Xi$.

The solution to these equations is compared to the results of MC simulations
in Fig. 2 for a one-component plasma screening a plane of charge. The BSK
theory reproduces the behavior of the one-component plasma from weak coupling,
in intermediate coupling, and matches the strong coupling limit with
$\alpha=0.50$. Furthermore, the BSK theory with the boundary condition of
$\hat{n}\cdot\nabla^{3}\phi=0$, represented by the dashed-dotted lines in Fig.
2a, does not accurately represent the data at intermediate or strong coupling.

Figure 3: BSK theory compared to MC simulations from 54 with $\alpha=0.50$
between two like-charged surfaces. The solid lines are the predictions of the
BSK theory, and the markers are from the MC simulations. Strong coupling
limits are plotted as dashed lines. (a) The charge density is plotted between
two surfaces with separation $\tilde{d}=2$. (b) The pressure is calculated as
a function of separation distance between the two plates. As the coupling
increases, the pressures between the like-charged surfaces become attractive
(negative) rather than repulsive (positive). The dimensionless pressure is
$\tilde{P}=Pe^{2}/(2\pi\ell_{B}q_{s}^{2})$.

We can also consider the one-component plasma between two charged surfaces of
equal charge density with the same sign, confining the counterions in a gap of
dimensionless distance $\tilde{d}$, as shown in Fig. 3a. In Fig. 3b, the
pressure is plotted as a function of separation distances between two charged
surfaces with different coupling parameters, using Eq. 8 and using the same
value for $\alpha$. The BSK theory again provides good agreement with the
results of the MC simulations at all the coupling parameters.

Figure 4: BSK theory compared to MC simulations from 61 using $\alpha=0.50$
for the counterion density around a charged cylinder for $\xi=4$. The solid
lines are the results of applying Eq. 18 and the markers are the MC simulation
results from 61. The weak coupling, strong coupling, and re-normalized strong
coupling needle limits are plotted 61.

Another critical question is the validity of Eq. 8 at a curved interface. The
simplest model system to test the hypothesis is the one-component plasma
surrounding a charged cylinder of radius
$\xi=R_{\mathrm{cyl}}/\ell_{\mathrm{GC}}$ at infinite dilution, corresponding
to a cylindrical cell with outer radius $R_{\mathrm{out}}\rightarrow\infty$.
As shown in Fig. 4, the BSK equation reproduces the results of the weak and
strong coupling limits correctly. However, similar to the strong coupling
expansion of 62, the theory does not correctly describe the renormalization of
charge arising from Manning condensation in the needle limit, where a fraction
$f=1-1/\xi$ of the charge is “condensed" onto cylinder 63. The charge density
must be multiplied by this fraction, $f$, in order to match the strong
coupling expansion taking into account the charge renormalization in the
needle limit 61, 64. The smaller the radius of curvature, the more likely that
the configuration of correlated ions is influenced by curvature. In the
“needle limit," where $\sqrt{\Xi}/\xi>>1$, ions are distributed in a nearly
linear fashion along the cylindrical backbone with spacing scaling as
$\tilde{a}\thicksim\Xi/\xi$, which may also be the relevant scaling for the
correlation length in this regime rather than $\sqrt{\Xi}$. Supporting Figs.
S1 and S2 show the results choosing $\delta_{c}=\Xi/\xi$, but the counterion
condensation transition is still not captured.

Figure 5: BSK theory compared to MC simulations 59 of multivalent electrolytes
with $\alpha=0.50$. (a) An example of the charge density profile for a 2:1
electrolyte at 0.1 M concentration compared to the GCMC simulations and PB
theory. (b) The difference between the predictions of the BSK theory and the
simulations from PB theory predictions.

##  3 Electrolytes

A more useful and relevant application of the BSK theory is to describe the
distribution of charges in electrolytes and ionic liquids, as was originally
proposed. Here, we focus on the dilute electrolyte limit, to isolate
electrostatic correlations directly, without complications from overcrowding.

If the BSK equation for a z:1 electrolyte with salt concentration
${C_{\mathrm{ref}}}$ is non-dimensionalized with the thermal voltage
$\tilde{\phi}=(e\phi)/(kT)$ and the Debye length

$\lambda_{D}=\sqrt{\frac{\epsilon kT}{(z^{2}+z)e^{2}{C_{\mathrm{ref}}}}},$
(20)

$\tilde{\nabla}=\lambda_{D}\nabla$ and $\delta_{c}=\ell_{c}/\lambda_{D}$ the
BSK equation becomes:

$\delta_{c}^{2}\tilde{\nabla}^{4}\tilde{\phi}-\tilde{\nabla}^{2}\tilde{\phi}=\tilde{\rho}=\frac{ze^{-z\tilde{\phi}}-ze^{\tilde{\phi}}}{z^{2}+z}.$
(21)

The boundary conditions are similarly modified to:

$\begin{split}&\hat{n}\cdot(\delta_{c}^{2}\tilde{\nabla}^{3}\tilde{\phi}-\tilde{\nabla}\tilde{\phi})=\tilde{q}_{s}\\\
&\hat{n}\cdot\delta_{c}\tilde{\nabla^{3}\phi}=\tilde{\nabla}^{2}\tilde{\phi}.\end{split}$
(22)

The agreement of the predicted charge density profiles from Eq. 21 with the
GCMC data is very good, as exhibited in Fig. 4 for a 0.1 M 2:1 electrolyte. In
the Supporting Information, the results are expanded to a more complete set of
comparisons with simulations. It is seen with $\ell_{c}=0.50R_{\mathrm{hole}}$
or with $\ell_{c}$ determined by Eq. 15, that the BSK theory can correct the
PB charge density profiles, including an overscreening transition. Larger
errors from the BSK theory are incurred at large concentration, where the
current assumption of $\mu^{\mathrm{ex}}_{i}=0$ breaks down.

One implication of the boundary condition is that the differential capacitance
for $\ell_{c}=0$ is equivalent to the case of $\ell_{c}\neq 0$ if
$\mu_{i}^{\mathrm{ex}}=0$. Therefore, the differential capacitance for the
correlated, dilute electrolyte is given by the traditional Gouy-Chapman
equation:

$C_{D}=\frac{\epsilon}{\lambda_{D}}\cosh\left(\frac{\tilde{\phi}_{D}}{2}\right),$
(23)

in stark contrast to the original work in the limit of $\mu_{i}^{ex}=0$ 17,
20. It would be interesting to explore the implications of the boundary
condition on electrokinetic reversals, electrochemical interfaces, biological
channels, or colloidal phenomena 20, 25, 24, 33, 41, 48, 21, 34, 42, 26, 28,
27, 29. For example, the DLVO theory of colloidal interactions can be modified
to include attractive correlation effects 4.

##  4 Further extensions of the theory

###  4.1 Charged Dielectric Interfaces:

Note that the Maxwell stress condition (Eq. 6) has only been stated without a
dielectric jump. The stress condition may need further validation at a
dielectric interface. A more general statement of matching the Maxwell stress
with and without correlations might be given by a jump condition between the
two media:

$\hat{n}\cdot[\mathbf{\uptau}_{e,1}-\mathbf{\uptau}_{e,2}]=\hat{n}\cdot[\mathbf{\uptau}_{e,1}-\mathbf{\uptau}_{e,2}]_{\ell_{c}=0}.$
(24)

For a uniformly-charged, flat interface without a dielectric jump,
$\tau_{e,2}=0$, so the RHS of the above equation reduces to
$\hat{n}\cdot[\mathbf{\uptau}_{e,1}-\mathbf{\uptau}_{e,2}]_{\ell_{c}=0}=\frac{{q_{s}}^{2}}{2\epsilon_{1}}\hat{n}$
.

###  4.2 Concentrated Electrolytes and Ionic Liquids:

The present analysis attempts to isolate the effect of ion correlations in a
dilute electrolyte. Ion size effects, particularly for $a/\lambda_{D}>1$ will
require further validation to properly account for correlations guided by ion
size combined with electrostatics. A non-local free energy functional might be
necessary to capture the size correlations in concentrated solutions 16, 65,
in conjunction with electrostatic correlations. Short-range bulk correlations
66 are not captured in this theory. Furthermore, if surface charges are
discrete rather than smeared out, the contact condition may change 67. For an
arbitrary mixture of ions with different valency, the effective correlation
length will depend upon correlations between each pair of species, although
the correlations at high surface charge density will still be dominated by the
most highly charged counterion.

##  5 Conclusions

The phenomenological BSK theory describes non-local, discrete correlation
effects with a higher-order, local, continuum description of the free energy
quite well. The remarkable agreement of the theory with the one-component
plasma and primitive model electrolyte suggest that higher-order, continuum
equations can properly account for correlation effects, as long as the
appropriate constraints are imposed at boundaries. The formalism used here
could be extended to the ionic liquid limit, although ion pairing, short-range
non-electrostatic correlations, and “spin glass" ordering 68 might preclude a
simple continuum description. Further modifications are needed to capture the
long range screening exhibited in ionic liquids and concentrated
electrolytes69, as well as density oscillations expected in overcrowded
systems 16. Even so, the BSK theory captures important features of
electrostatic correlations, including like-charge attraction and
overscreening, driven by electrostatic interactions of spatially correlated
counterions. Furthermore, unlike many previous approaches, all the
electrostatic forces are contained self-consistently within the electrostatic
potential, $\phi$. Detailed analysis of experimental data is needed to
determine the competing effects of surface adsorption reactions modifying
fixed surface charge 70 or the overscreening/ like-charge attraction effects
modeled by the BSK theory 4.

##  6 Acknowledgements

This work was supported by an Amar G. Bose Research Grant (electrolyte
calculations) and the Center for Enhanced Nanofluidic Transport (one-component
plasma calculations). JPD is also supported by the National Science Foundation
Graduate Research Fellowship under Grant No. 1122374. JPD would like to
acknowledge useful discussions with Michael McEldrew, Amir Levy, Tingtao Zhou,
Dimitrios Fraggedakis, and Mohamed Mirzadeh.

## References

  * Levin and Levin 2002 Levin, Y.; Levin, Y. Electrostatic correlations: from plasma to biology. _Rep. Prog. Phys_ 2002, _65_ , 1577–1632
  * Yu Grosberg et al. 2002 Yu Grosberg, A.; Nguyen, T. T.; Shklovskii, B. I. Colloquium: The physics of charge inversion in chemical and biological systems. 2002,
  * Pellenq et al. 1997 Pellenq, R.-M.; Caillol, J.; Delville, A. Electrostatic attraction between two charged surfaces: A (N, V, T) Monte Carlo simulation. _The Journal of Physical Chemistry B_ 1997, _101_ , 8584–8594
  * Misra et al. 2019 Misra, R. P.; de Souza, J. P.; Blankschtein, D.; Bazant, M. Z. Theory of Surface Forces in Multivalent Electrolytes. _Langmuir_ 2019, _35_ , 11550–11565
  * Outhwaite and Bhuiyan 1983 Outhwaite, C. W.; Bhuiyan, L. B. An improved modified Poisson–Boltzmann equation in electric-double-layer theory. _Journal of the Chemical Society, Faraday Transactions 2: Molecular and Chemical Physics_ 1983, _79_ , 707–718
  * Henderson and Blum 1978 Henderson, D.; Blum, L. Some exact results and the application of the mean spherical approximation to charged hard spheres near a charged hard wall. _The Journal of Chemical Physics_ 1978, _69_ , 5441–5449
  * Henderson et al. 1979 Henderson, D.; Blum, L.; Smith, W. R. Application of the hypernetted chain approximation to the electric double layer at a charged planar interface. _Chemical Physics Letters_ 1979, _63_ , 381–383
  * Lozada-Cassou et al. 1982 Lozada-Cassou, M.; Saavedra-Barrera, R.; Henderson, D. The application of the hypernetted chain approximation to the electrical double layer: Comparison with Monte Carlo results for symmetric salts. _The Journal of Chemical Physics_ 1982, _77_ , 5150–5156
  * Kjellander et al. 1992 Kjellander, R.; Åkesson, T.; Jönsson, B.; Marčelja, S. Double layer interactions in mono-and divalent electrolytes: A comparison of the anisotropic HNC theory and Monte Carlo simulations. _The Journal of Chemical Physics_ 1992, _97_ , 1424–1431
  * Voukadinova et al. 2018 Voukadinova, A.; Valiskó, M.; Gillespie, D. Assessing the accuracy of three classical density functional theories of the electrical double layer. _Physical Review E_ 2018, _98_ , 012116
  * Kierlik and Rosinberg 1991 Kierlik, E.; Rosinberg, M. Density-functional theory for inhomogeneous fluids: adsorption of binary mixtures. _Physical Review A_ 1991, _44_ , 5025
  * Gillespie et al. 2002 Gillespie, D.; Nonner, W.; Eisenberg, R. S. Coupling Poisson–Nernst–Planck and density functional theory to calculate ion flux. _Journal of Physics: Condensed Matter_ 2002, _14_ , 12129
  * Roth and Gillespie 2016 Roth, R.; Gillespie, D. Shells of charge: a density functional theory for charged hard spheres. _Journal of Physics: Condensed Matter_ 2016, _28_ , 244006
  * Rosenfeld 1993 Rosenfeld, Y. Free energy model for inhomogeneous fluid mixtures: Yukawa-charged hard spheres, general interactions, and plasmas. _The Journal of chemical physics_ 1993, _98_ , 8126–8148
  * Rosenfeld 1989 Rosenfeld, Y. Free-energy model for the inhomogeneous hard-sphere fluid mixture and density-functional theory of freezing. _Physical review letters_ 1989, _63_ , 980
  * Roth 2010 Roth, R. Fundamental measure theory for hard-sphere mixtures: a review. _Journal of Physics: Condensed Matter_ 2010, _22_ , 063102
  * Bazant et al. 2011 Bazant, M. Z.; Storey, B. D.; Kornyshev, A. A. Double Layer in Ionic Liquids: Overscreening versus Crowding. _Physical Review Letters_ 2011, _106_ , 046102
  * Santangelo 2006 Santangelo, C. D. Computing counterion densities at intermediate coupling. _Physical Review E_ 2006, _73_ , 041512
  * Hatlo and Lue 2010 Hatlo, M. M.; Lue, L. Electrostatic interactions of charged bodies from the weak- to the strong-coupling regime. _EPL (Europhysics Letters)_ 2010, _89_ , 25002
  * Storey and Bazant 2012 Storey, B. D.; Bazant, M. Z. Effects of electrostatic correlations on electrokinetic phenomena. _Physical Review E_ 2012, _86_ , 056303
  * Stout and Khair 2014 Stout, R. F.; Khair, A. S. A continuum approach to predicting electrophoretic mobility reversals. _Journal of Fluid Mechanics_ 2014, _752_ , R1
  * Wang et al. 2017 Wang, C.; Bao, J.; Pan, W.; Sun, X. Modeling electrokinetics in ionic liquids. _Electrophoresis_ 2017, _38_ , 1693–1705
  * Zhao 2011 Zhao, H. Diffuse-charge dynamics of ionic liquids in electrochemical systems. _Physical Review E_ 2011, _84_ , 051504
  * Jiang et al. 2014 Jiang, X.; Huang, J.; Zhao, H.; Sumpter, B. G.; Qiao, R. Dynamics of electrical double layer formation in room-temperature ionic liquids under constant-current charging conditions. _Journal of Physics: Condensed Matter_ 2014, _26_ , 284109
  * Kondrat et al. 2015 others,, et al. Dynamics of Ion Transport in Ionic Liquids. _Physical review letters_ 2015, _115_ , 106101
  * Alijó et al. 2015 Alijó, P. H.; Tavares, F. W.; Biscaia Jr, E. C.; Secchi, A. R. Effects of electrostatic correlations on ion dynamics in alternating current voltages. _Electrochimica Acta_ 2015, _152_ , 84–92
  * Alidoosti and Zhao 2018 Alidoosti, E.; Zhao, H. On the Impact of Electrostatic Correlations on the Double-Layer Polarization of a Spherical Particle in an Alternating Current Field. _Langmuir_ 2018, _34_ , 5592–5599
  * Lee et al. 2013 Lee, D.; Im, D.; Kang, I. Electric Double Layer at the Interface of Ionic Liquid–Dielectric Liquid under Electric Field. _Langmuir_ 2013, _29_ , 1875–1884
  * McEldrew et al. 2018 McEldrew, M.; Goodwin, Z. A.; Kornyshev, A. A.; Bazant, M. Z. Theory of the double layer in water-in-salt electrolytes. _The journal of physical chemistry letters_ 2018, _9_ , 5840–5846
  * Shalabi et al. 2019 Shalabi, A.; Daniels, L.; Scott, M.; Mišković, Z. Differential capacitance of ionic liquid interface with graphene: The effects of correlation and finite size of ions. _Electrochimica Acta_ 2019, _319_ , 423–434
  * Xie and Jiang 2018 Xie, D.; Jiang, Y. Nonlocal Poisson-Fermi double-layer models: Effects of nonuniform ion sizes on double-layer structure. _Physical Review E_ 2018, _97_ , 052610
  * Yang et al. 2019 Yang, Y. D.; Moon, G. J.; Oh, J. M.; Kang, I. S. Discrete and Continuum Analyses of Confinement Effects of an Ionic Liquid on the EDL Structure and the Pressure Acting on the Wall. _The Journal of Physical Chemistry C_ 2019, _123_ , 2516–2525
  * Jiang et al. 2016 Jiang, X.; Liu, Y.; Qiao, R. Current rectification for transport of room-temperature ionic liquids through conical nanopores. _The Journal of Physical Chemistry C_ 2016, _120_ , 4629–4637
  * Moon et al. 2015 Moon, G. J.; Ahn, M. M.; Kang, I. S. Osmotic pressure of ionic liquids in an electric double layer: Prediction based on a continuum model. _Physical Review E_ 2015, _92_ , 063020
  * Santos et al. 2017 Santos, M. S.; Biscaia Jr, E. C.; Tavares, F. W. Effect of electrostatic correlations on micelle formation. _Colloids and Surfaces A: Physicochemical and Engineering Aspects_ 2017, _533_ , 169–178
  * Liu and Eisenberg 2015 Liu, J.-L.; Eisenberg, B. Poisson–Fermi model of single ion activities in aqueous solutions. _Chemical Physics Letters_ 2015, _637_ , 1–6
  * Schlumpberger and Bazant 2017 Schlumpberger, S.; Bazant, M. Z. Simple Theory of Ionic Activity in Concentrated Electrolytes. _arXiv preprint arXiv:1709.03106_ 2017,
  * Liu and Eisenberg 2018 Liu, J.-L.; Eisenberg, B. Poisson-Fermi modeling of ion activities in aqueous single and mixed electrolyte solutions at variable temperature. _The Journal of chemical physics_ 2018, _148_ , 054501
  * Nakamura 2018 Nakamura, I. Effects of Dielectric Inhomogeneity and Electrostatic Correlation on the Solvation Energy of Ions in Liquids. _The Journal of Physical Chemistry B_ 2018, _122_ , 6064–6071
  * Liu and Li 2019 Liu, J.-L.; Li, C.-L. A generalized Debye-Hückel theory of electrolyte solutions. _AIP Advances_ 2019, _9_ , 015214
  * Huang 2018 Huang, J. Confinement Induced Dilution: Electrostatic Screening Length Anomaly in Concentrated Electrolytes in Confined Space. _The Journal of Physical Chemistry C_ 2018, _122_ , 3428–3433
  * Liu 2013 Liu, J.-L. Numerical methods for the Poisson–Fermi equation in electrolytes. _Journal of Computational Physics_ 2013, _247_ , 88–99
  * Liu and Eisenberg 2013 Liu, J.-L.; Eisenberg, B. Correlated ions in a calcium channel model: a Poisson–Fermi theory. _The Journal of Physical Chemistry B_ 2013, _117_ , 12051–12058
  * Liu and Eisenberg 2014 Liu, J.-L.; Eisenberg, B. Poisson-Nernst-Planck-Fermi theory for modeling biological ion channels. _The Journal of chemical physics_ 2014, _141_ , 12B640_1
  * Liu and Eisenberg 2014 Liu, J.-L.; Eisenberg, B. Analytical models of calcium binding in a calcium channel. _The Journal of Chemical Physics_ 2014, _141_ , 075102
  * Liu and Eisenberg 2015 Liu, J.-L.; Eisenberg, B. Numerical methods for a Poisson-Nernst-Planck-Fermi model of biological ion channels. _Physical Review E_ 2015, _92_ , 012711
  * Bazant 2013 Bazant, M. Z. Theory of chemical kinetics and charge transfer based on nonequilibrium thermodynamics. _Accounts of chemical research_ 2013, _46_ , 1144–1160
  * dos Santos et al. 2009 dos Santos, A.; Diehl, A.; Levin, Y. Electrostatic correlations in colloidal suspensions: Density profiles and effective charges beyond the Poisson–Boltzmann theory. _The Journal of chemical physics_ 2009,
  * Gupta and Stone 2018 Gupta, A.; Stone, H. A. Electrical Double Layers: Effects of Asymmetry in Electrolyte Valence on Steric Effects, Dielectric Decrement, and Ion–Ion Correlations. _Langmuir_ 2018, _34_ , 11971–11985
  * Bazant et al. 2009 Bazant, M. Z.; Kilic, M. S.; Storey, B. D.; Ajdari, A. Towards an understanding of induced-charge electrokinetics at large applied voltages in concentrated solutions. _Advances in colloid and interface science_ 2009, _152_ , 48–88
  * Woodson and Melcher 1968 Woodson, H. H.; Melcher, J. R. Electromechanical dynamics. 1968,
  * Alawneh and Henderson 2007 Alawneh, M.; Henderson, D. Monte Carlo simulation of the double layer at an electrode including the effect of a dielectric boundary. _Molecular Simulation_ 2007, _33_ , 541–547
  * Henderson et al. 1979 Henderson, D.; Blum, L.; Lebowitz, J. L. An exact formula for the contact value of the density profile of a system of charged hard spheres near a charged wall. _Journal of Electroanalytical Chemistry and Interfacial Electrochemistry_ 1979, _102_ , 315–319
  * Moreira and Netz 2002 Moreira, A.; Netz, R. Simulations of counterions at charged plates. _The European Physical Journal E_ 2002, _8_ , 33–58
  * Burak et al. 2004 Burak, Y.; Andelman, D.; Orland, H. Test-charge theory for the electric double layer. _Physical Review E_ 2004, _70_ , 016102
  * Carnie and Chan 1981 Carnie, S. L.; Chan, D. Y. Ionic adsorption from a primitive model electrolyte—nonlinear treatment. _The Journal of Chemical Physics_ 1981, _75_ , 3485–3494
  * Kornyshev et al. 1978 Kornyshev, A.; Rubinshtein, A.; Vorotyntsev, M. Model nonlocal electrostatics. I. _Journal of Physics C: Solid State Physics_ 1978, _11_ , 3307
  * Kornyshev 1981 Kornyshev, A. Nonlocal screening of ions in a structurized polar liquid—new aspects of solvent description in electrolyte theory. _Electrochimica Acta_ 1981, _26_ , 1–20
  * Valiskó et al. 2018 Valiskó, M.; Kristóf, T.; Gillespie, D.; Boda, D. A systematic Monte Carlo simulation study of the primitive model planar electrical double layer over an extended range of concentrations, electrode charges, cation diameters and valences. _AIP Advances_ 2018, _8_ , 025320
  * Grosberg et al. 2002 Grosberg, A. Y.; Nguyen, T.; Shklovskii, B. Colloquium: the physics of charge inversion in chemical and biological systems. _Reviews of modern physics_ 2002, _74_ , 329
  * Mallarino et al. 2013 Mallarino, J. P.; Tellez, G.; Trizac, E. Counterion density profile around charged cylinders: the strong-coupling needle limit. _The Journal of Physical Chemistry B_ 2013, _117_ , 12702–12716
  * Naji and Netz 2005 Naji, A.; Netz, R. R. Counterions at charged cylinders: criticality and universality beyond mean-field theory. _Physical review letters_ 2005, _95_ , 185703
  * Manning 1969 Manning, G. S. Limiting Laws and Counterion Condensation in Polyelectrolyte Solutions I. Colligative Properties. _The Journal of Chemical Physics_ 1969, _51_
  * Cha et al. 2017 Cha, M.; Yi, J.; Kim, Y. W. Hidden Criticality of Counterion Condensation Near a Charged Cylinder. _Scientific Reports_ 2017, _7_ , 10551
  * Gillespie 2015 Gillespie, D. A review of steric interactions of ions: Why some theories succeed and others fail to account for ion size. _Microfluidics and Nanofluidics_ 2015, _18_ , 717–738
  * Goodwin et al. 2017 Goodwin, Z. A.; Feng, G.; Kornyshev, A. A. Mean-Field Theory of Electrical Double Layer In Ionic Liquids with Account of Short-Range Correlations. _Electrochimica Acta_ 2017, _225_ , 190–197
  * Moreira and Netz 2002 Moreira, A. G.; Netz, R. R. Counterions at charge-modulated substrates. _EPL (Europhysics Letters)_ 2002, _57_ , 911
  * Levy et al. 2018 Levy, A.; McEldrew, M.; Bazant, M. Z. Spin-Glass Charge Ordering in Ionic Liquids. _arXiv preprint arXiv:1808.06156_ 2018,
  * Smith et al. 2016 Smith, A. M.; Lee, A. A.; Perkin, S. The electrostatic screening length in concentrated electrolytes increases with concentration. _The journal of physical chemistry letters_ 2016, _7_ , 2157–2163
  * Mugele et al. 2015 Mugele, F.; Bera, B.; Cavalli, A.; Siretanu, I.; Maestro, A.; Duits, M.; Cohen-Stuart, M.; Van Den Ende, D.; Stocker, I.; Collins, I. Ion adsorption-induced wetting transition in oil-water-mineral systems. _Scientific reports_ 2015, _5_ , 10519