# Collaborative Position Reasoning Network for Referring Image Segmentation

Jianjian Cao, Beiya Dai, Yulin Li, Xiameng Qin and Jingdong Wang Jianjian Cao
is with School of Information Science and Technology, Fudan University,
Shanghai, China. (Email: jjcao22@m.fudan.edu.cn).Beiya Dai is with Department
of Computer Science, Notional University of Defense Technology, Changsha,
China. (Email: beiya_dai@nudt.edu.cn).Yulin Li, Xiameng Qin and Jingdong Wang
are with Baidu Inc., Beijing, China. (Email<EMAIL_ADDRESS><EMAIL_ADDRESS>wangjingdong@baidu.com).

###### Abstract

Given an image and a natural language expression as input, the goal of
referring image segmentation is to segment the foreground masks of the
entities referred by the expression. Existing methods mainly focus on
interactive learning between vision and language to enhance the multi-modal
representations for global context reasoning. However, predicting directly in
pixel-level space can lead to collapsed positioning and poor segmentation
results. Its main challenge lies in how to explicitly model entity
localization, especially for non-salient entities. In this paper, we tackle
this problem by executing a Collaborative Position Reasoning Network (CPRN)
via the proposed novel Row-and-Column interactive (RoCo) and Guided Holistic
interactive (Holi) modules. Specifically, RoCo aggregates the visual features
into the row- and column-wise features corresponding two directional axes
respectively. It offers a fine-grained matching behavior that perceives the
associations between the linguistic features and two decoupled visual features
to perform position reasoning over a hierarchical space. Holi integrates
features of the two modalities by a cross-modal attention mechanism, which
suppresses the irrelevant redundancy under the guide of positioning
information from RoCo. Thus, with the incorporation of RoCo and Holi modules,
CPRN captures the visual details of position reasoning so that the model can
achieve more accurate segmentation. To our knowledge, this is the first work
that explicitly focuses on position reasoning modeling. We also validate the
proposed method on three evaluation datasets. It consistently outperforms
existing state-of-the-art methods.

###### Index Terms:

Referring Image Segmentation, Position Reasoning, Transformer.

##  I Introduction

Referring image segmentation (RIS) aims to predict a pixel-level segmentation
mask in the image corresponding to entity referred by the natural language
expression. As shown in Fig. 1, it can identify the entities of interest by
the description of free-form referring expressions, which are not restricted
to pre-defined object categories. RIS requires the algorithms to explore the
relationship between language and vision so that the style of referred entity
can be more flexible than traditional segmentation tasks. Hence, RIS can be
regarded as an open-ended task and has a wide range of potential applications
in interactive image editing and human-robot interaction, etc. It has
attracted the attention of many researchers in the intersection of vision and
language.

Figure 1: Comparison of visualization results between state-of-the-art methods
and our proposed approach. The first row shows that the existing methods are
prone to positioning errors on some Non-Salient targets with the small-scale.
The second row shows that for some complex referring expressions, the
reasoning ability of the previous models is not enough to position the target
accurately.

Since RIS involves visual and linguistic domains, it is challenging especially
in modeling the fine-grained interactions and aligning implicit relationships
among the two modalities. As shown in Fig. 2, existing works can be roughly
divided into four types according to the network structure. (a) A
straightforward way to extract contextual knowledge to produce the final
result via a simple concatenation-convolution scheme such as dynamic filters,
LSTM and attention mechanism. This solution [5, 6] aggregates the visual and
linguistic features without a deep understanding, which could not effectively
explore the relationships between the two modalities. (b) Another line of
works [9, 10] process each word in the referring expression to learn cross-
modal interaction in a sequential manner. However, they consider each word as
an equal contribution. This may have trouble distinguishing the target with
long referring expressions. (c) Alternative works [52, 53] establish several
attributes (object, location and relationship to other objects) of referring
expressions to improve the scores of cross-modal matching. This design help
refine the segmentation results but lacks global context information and
relies on the proposals generated by object detectors. (d) At last, a series
of works have been proposed to progressively integrate contextual information
at multiple levels. LAVT [66] fuses the linguistic and visual features into
each stage of the network and captures segmentation masks with a lightweight
decoder. BRINet [34] considers the interaction through a bi-directional cross-
modal attention module that uses both visual and textual guidances to capture
their dependencies, realizing the compatibility between vision and textural
features. They focuses on utilizing the inference module to enhance the visual
and textual interaction achieves the best results. Nevertheless, all these
methods do not directly focus on the issue of position reasoning and fail to
locates the referred entity from background.

Most previous works tackle the referring problem utilizing efficient cross-
modality feature interaction to explore semantic contextual representations.
Specifically, mainstream frameworks firstly extract visual and linguistic
features respectively, and then introduce diverse operations to solve the
interactive learning. Although these methods have achieved remarkable
performance, the limitation of them is that the global context modelings still
lack sufficient fine-grained visual concepts which is essential for position
reasoning. Fig. 1 shows the visualization examples in which the segmentation
masks remain unsatisfactory because of the distraction of background and non-
salient objects. From the perspective of human cognition, the RIS model
focuses on positioning the entity regions that well match the expressions, and
then refine the precise segmentation. The fine-grained semantic features help
the model distinguish the referred entity from other analogs.

Figure 2:  Conceptual Comparisons of typical architectures. (a) Simply
combining the two modals of information. (b) Dividing the referring expression
and fusing it with visual features progressively. (c) Exploring the
relationship between visual and textual features by matching their attributes.
(d) Adopting inference module in a multi-semantic-level progressively-fusion
network architecture. $I$, $E$, $V$ and $L$ denotes image, natural language
expression, visual feature and texture feature, respectively.

Recently, the Transformer has achieved great success in the area of Natural
Language Processing and Computer Vision. The state-of-the-art RIS methods
introduce Transformer architectures to strengthen the ability of multi-modal
feature fusion and global information modeling. VLT [60] uses a transformer to
build a network with an encoder-decoder attention mechanism to enhance global
contextual information. LAVT [66] utilizes the multi-stage design in the Swin
Transformer to form a hierarchical language-aware visual coding scheme. CRIS
[64] leverages the pre-trained model CLIP [59] and contrastive learning
strategy to achieve text-pixel alignment. Although the transformer can bring a
certain performance improvement to the RIS model, the challenge of position
reasoning still exists and has not been well solved.

In this paper, we address the problem of position reasoning in RIS and propose
a Collaborative Positioning Reasoning Network (CPRN) for leveraging the
hierarchical context of images for position reasoning. As illustrated in Fig.
3, in our model, the features passed through two parallel pathways can capture
the local and global information for accurate localization and fine-grained
segmentation. In detail, the Row-and-Column interactive (RoCo) module
generates the correlation between horizontal and vertical feature maps with
linguistic features. The Guided Holistic interactive (Holi) module keeps the
holistic feature map to ensure the integrity of global information. Meanwhile,
a global guidance path directs the RoCo’s positioning information into the
Holi module to enhance entity perception reasoning and suppress the irrelevant
redundancy from background. The output features of RoCo and Holi are merged
via a Feed Forward Network (FFN). Finally, we devise a Multi-Scale decoder to
aggregate multi-level features for accurate referring segmentation.

Figure 3: Illustration of the difference between (a) previous methods and (b)
our model. Previous works apply the holistic feature map in Cross-modal
Attention. Differently, we use two parallel branches. The above one divides
the holistic feature map into horizontal and vertical maps, and the lower one
keeps the holistic feature map. They are fused with the textual feature,
respectively, getting a better location for the referent.

In summary, this paper makes the following contributions:

  * •

We propose a novel Collaborative Positioning Reasoning Network (CPRN) to
explicitly settle the position reasoning issue in RIS. And the proposed CPRN
can be used as a flexible block adaptable to any inference-based framework.

  * •

We propose a Row-and-Column interactive (RoCo) module to explicitly locate the
referent by dividing the holistic feature map into row- and column-wise maps
and integrating them separately with textual features.

  * •

We propose a Guided Holistic interactive (Holi) module to retain a
comprehensive perception of all pixels in an image, for fine-grained
segmentation. Furthermore, the global guidance path is designed to enhance the
localization of Holi by incorporating the RoCo’s positioning information.

  * •

Extensive experiments on all three challenging datasets show that the proposed
CPRN plays an important role in improving the positioning performance of
referring image segmentation. And our model achieves superior performance
compared to state-of-the-art methods.

##  II Related Work

###  II-A Referring Image Segmentation.

Given an image and a natural language expression, the goal of Referring image
segmentation is to produce a segmentation mask in the image corresponding to
entities referred by the natural language expression. The RIS task is firstly
introduced in [5], which directly concatenates both visual and textual
features to generate the final mask. RRN [6] considers the multi-scale
semantics in the visual encoding step and employs ConvLSTM [16] in the feature
fusion step. Later, word attention [7] extracts keywords in the image regions
to suppress noises in the referring expression and highlight the target
object. RMI [9] directly combines visual features with each word feature from
a language LSTM to recurrently refine segmentation results. DMNet [10]
utilizes a dynamic filter for each word to further enhances this interaction.
Further, relation inference is applied to capture visual and textual
modalities.

With the application of the attention mechanism more and more widely, some
work uses the attention mechanism to extract visual content corresponding to
language expression. STEP [8] emphasizes the attention from image to word by
computing dependencies between each visual region and each word, to guide the
segmentation recurrently. CMSA [15] is exploited in respectively to capture
global interaction information between image regions and words via Cross-modal
self-attention. CMPC [58] firstly employs entity and attribute words to
perceive all the related entities. Then, the relational words are adopted to
highlight the correct entity, as well as suppress other irrelevant ones by
multi-modal graph reasoning. BRINet [34] uses both visual and linguistic
guidances to capture the dependencies between multi-modal features. LSCM [29]
models interaction between visual and textural information under the guidance
of DPT-WG [43]. ReSTR [63] is the first convolution-free architecture for RIS,
unifying two different modal network topologies with Transformer. CRIS [64]
uses the pre-trained model CLIP [59] and contrast learning strategies to
achieve text pixel alignment. MaIL [65] introduces a new modal information
mask mode and designs a simpler encoder-decoder pipeline and a mask-image-
language three-mode encoder. LAVT [66] fuse the linguistic and visual features
into each stage of the network and captures segmentation masks with a
lightweight decoder.

However, these works merely adopt holistic visual information in multi-modal
interaction, leading to inaccurate object location. In this work, we introduce
a collaborative position reasoning method by row-and-column interaction, in
addition to holistic multi-interactive inference, and achieve satisfied
segmentation results.

###  II-B Multi-modal Interaction.

A lot of multi-modal interaction researchers are interested in combining
natural language processing with visual understanding. At first, [44]
demonstrates that if relevant data from different modalities is available at
training time, better features can be learned. TFN [45], LMF [46], and T2FN
[47] are proposed to capture both intra- and inter-modal dynamics
simultaneously. MulT [48] aligns data from different modalities implicitly,
which leverages cross-modal attention modules for each modality on a high
level, and each of them is responsible for aligning the target modality vector
with the complementary modal vector. [42] introduces Auto-Fusion and GAN-
Fusion learning to compress information from different modalities while
preserving the context and GAN-Fusion regularizes the learned latent space
given context from complementing modalities, making the network decide the
fusion manner. MCF [49] puts forward reshaping feature vectors into circulant
matrices and defining two types of interaction operations between vectors and
matrices. [50] realizes bidirectional multi-layer fusion from both channel-
level and pixel-level through two fusion operations, which can strengthen the
multi-modal feature interactions across channels as well as enhance the
spatial feature discrimination. For a given query (image or language), [57]
simply considers the keys and values from all input tokens, it just merges the
input from both ways, this multi-modal attention is called Merge attention.
[32] approach is that given a query from one modality (e.g., image), keys and
values can only be obtained from another modality (e.g., language), this
multi-modal attention is called co-attention. These methods for multi-modal
interaction are based on holistic feature maps.

In this paper, we designed a Row-and-Column interactive (RoCo) module, which
decompose the holistic feature map and used row- and column-wise information
to interact with textual feature, respectively, to establish the local
association between visual and linguistic patterns.

Figure 4: The overall architecture of our method. The CPRN block directly
fuses features from two modal inputs of image $\bf{V}_{i}$ and text $\bf{L}$
to generate cross-modal feature representations. It includes RoCo interactive
module and Holi interactive module. The multi-scale decoder generates
segmentation results based on the interactive representation of multi-modal
features at different stages.

###  II-C Location Mechanism.

In semantic segmentation tasks, in addition to multimodal feature fusion, the
problem of locating reference images cannot be ignored. At present, some work
has been done in locating target reference objects. [67] uses the prior
extractor to extract the prior, and then uses the before to generate a prior
region map of the query image, which is used to locate objects. MCN [12]
jointly learns two tasks, Citation Representation Comprehension (REC) and
Segmentation (RES). To address the problem of conflicting predictions between
the two tasks, he proposed an adaptive soft non-localization suppression
(ASNLS) design, a post-processing method that suppresses responses in
irrelevant regions in the RES based on the predictions of RECs. LTS [62]
proposes a localization module to obtain the corresponding visual content of
the expression and uses the obtained object prior as the visual localization
guidance for the subsequent segmentation module. Unlike the localization
module, which includes two forms of simple filters and Transformers, the
proposed CPRN block locates the visual area that responds to the linguistic
expression by the row-column position information. There is a class of methods
to obtain object localization information by using additional external
sources, such as MAttNet [38] and lang2seg [13]. These two methods use Mask
R-CNN [40] to pre-process and post-process the image when segmenting the
image. Although Mask R-CNN provides localization and segmentation of objects
in images, greatly improving the performance of the model, our collaborative
positioning and reasoning network performs much better on the three benchmark
datasets, demonstrating the superiority of our approach in positioning. In
addition, the idea of CCNet [14] is somewhat similar to ours and it obtains
dense contextual information for semantic segmentation through two recurrent
cross-attention (RCCA) modules, which aggregates associated information via
rows and columns. Different from CCNet, we design two parallel interactive
modules, Roco and Holi, where Roco leverages the row and column information to
explicitly locate the referent, and Holi utilizes the global image information
for fine segmentation.

##  III Methodology

Fig. 4 illustrates the overall architecture of our Collaborative Positioning
Reasoning Network, which integrates the proposed CPRN block for referring
image segmentation. We first elaborate on the motivation of our approach in
Sec. III-A. Given an image and a natural language expression as input, we
extract the visual and linguistic features on different semantic levels,
respectively (Sec. III-B). Then, they are fused and fed into the CPRN
inference block (Sec. III-C), which is composed of two modules, to highlight
the referent entities. One is the Row-and-Column interactive (RoCo) module,
the other is the Guided Holistic interactive (Holi) module. After that, the
two pathways are merged using a Feed Forward Network (FFN) to enhance the
reasoning features. Finally, the Multi-Scale Decoder module (Sec. III-D) is
used to perform the different stage feature fusion and refine the final
segmentation mask.

###  III-A Motivation

It is essential for RIS task to mine relation information between vision and
language via feature interaction. Some works [15, 6, 29] consider the multi-
scale information to find the referent. Since they only consider holistic
visual information, inappropriate segmentation results exist. A main problem
is that they cannot accurately locate the object. We propose the Collaborative
Positioning Reasoning Network (CPRN) utilizing two parallel pathways to
sufficiently aggregate object position (RoCo module) while capturing holistic
information (Holi module) between visual and textual modalities, as
illustrated in Fig. 4. For solving the referent positioning issue, we
decompose the visual feature into row- and column-wise features, which will
interact with the textual features separately, to locate the object in both
horizontal and vertical directions. The multi-modal features of the two
directions will assign the location of the referent object. Meanwhile, the
positioning effect of the RoCo module will also guide the Holi module, helping
it to more accurately locate and segment the referent. By the mutual
enhancement between the RoCo and Holi modules, our method can perform reliable
joint reasoning, which greatly improves the localization and segmentation of
referent entities.

Figure 5: Illustration of our Collaborative Positioning Reasoning Network
(CPRN). First, the visual features $\bf{V}_{i}$ generate vertical features and
horizontal features via the Row-Column Aggregation. These two visual features
are obtained through cross attention layers and it generates the multi-modal
features with semantic information $\bf{v}_{h}^{hw}$ and $\bf{v}_{w}^{hw}$.
Then, the final output of RoCo module are obtained through expand and addition
operations. At the same time, $\bf{mask}_{roco}$ with positioning information
is calculated, which guides the Holi module through the global guidance
pathway for fine-grained segmentation. The Holi module utilizes
$\bf{Mask}_{roco}$ and $\bf{Mask}_{holi}$ to generate the final output via a
designed Guided Attention layer. Expand represents bilinear interpolation
operation, and for convenience of representation, we ignore some residual
connections.

###  III-B Feature Extraction

Taking an image $I$ and a referring expression $E$ with $T$ words as input, we
firstly use the Swin Transformer [23] to extract visual features at different
stages. Let $V_{i}\in\mathbb{R}^{H_{i}\times W_{i}\times C_{i}}$,
$i\in\\{2,3,4,5\\}$, denotes the visual features, corresponding to the 1st,
2nd, 3rd and 4th stages of Swin Transformer network, where $H_{i}$, $W_{i}$
and $C_{i}$ are the dimensions of height, width, and visual feature channels,
respectively. Besides, the spatial coordinate features are used to capture
more spatial information. For each stage, we also define an 8-D spatial
coordinate feature denoted as $P_{i}\in\mathbb{R}^{H_{i}\times W_{i}\times
8}$, $i\in\\{2,3,4,5\\}$ at each spatial position as the implementation in
[9]. Then, a new fused visual feature $\mathbf{V}_{i}$ is obtained by
concatenating the visual feature $V_{i}$ and the spatial coordinate feature
$P_{i}$ followed by a 1$\times$1 convolution layer. We denote a single level
of fused visual features as $\mathbf{V}$ for ease. Next, the linguistic
features $\mathbf{L}=\\{\mathbf{L}_{1},\mathbf{L}_{2},...,\mathbf{L}_{T}\\}$,
$\mathbf{L}_{i}\in\mathbb{R}^{d_{l}},i\in\\{1,2,...,T\\}$ is extracted with a
language encoder BERT [24], where $d_{l}$ and $T$ denote the number of
channels and the number of words. After that, the visual features $\mathbf{V}$
and linguistic features $\mathbf{L}$ are fed into our proposed Collaborative
Positioning Reasoning Network (CPRN) which depicted in Sec. III-C.

###  III-C Collaborative Positioning Reasoning Network

Rather than fusing the visual and linguistic features directly in previous
works, the proposed Collaborative Positioning Reasoning Network (CPRN) pays
more attention to the positioning of referent entities and designed the Row-
and-Column interactive (RoCo) module, which realizes the positioning target by
perceiving the row- and column-wise local features of the image. As illustrate
in Fig. 5, it also designs the Guided Holistic interactive (Holi) module,
which realizes the accurate segmentation of referents by perceiving the global
features of the image. Furthermore, the Feed Forward Network (FFN) is designed
to merge the two parallel pathways, enabling joint reasoning, making the
features used for final segmentation more reliable.

####  III-C1 Row-and-Column interactive module

In the positioning path, the network firstly decomposes the visual feature map
$\mathbf{V_{i}}$ into two parts, _i.e._ a row-wise feature and a column-wise
feature, corresponding to the horizontal and the vertical directions,
respectively. After that, it leverages to interact among the two visual
features and the linguistic features, as shown in Fig. 5. For the convenience
of representation, we remove the $i$ subscript of all variables.

Specifically, the row-wise and column-wise visual features are obtained by the
Row-Colunm Aggregation operations which execute average pooling on
$\mathbf{V}$, with pooling kernel of size $1\times W$ and $H\times 1$. And
each of these two features after pooling is implemented with 1$\times$1
convolution layer and followed by the GeLU function adding nonlinearity:

$\begin{split}\mathbf{v}_{h}&=GeLU\left(\mathbf{w}^{1}_{h}\left(Avg\\_pool_{h}\left(\mathbf{V}\right)\right)+\mathbf{b}^{1}_{h})\right),\\\
\mathbf{v}_{w}&=GeLU\left(\mathbf{w}^{1}_{w}\left(Avg\\_pool_{w}\left(\mathbf{V}\right)\right)+\mathbf{b}^{1}_{w}\right),\end{split}$
(1)

where $\mathbf{v}_{h}\in\mathbb{R}^{H\times C_{h}}$ denotes the row-wise
feature, and $\mathbf{v}_{w}\in\mathbb{R}^{W\times C_{w}}$ denotes the column-
wise feature, $C_{h}=C_{w}=C$. $H$ is the height of vertical feature, $W$
represents the width of horizontal feature, $C_{h}$ and $C_{w}$ define the
number of channels. From the linguistic feature $\mathbf{L}$, it generates
word vectors, $\mathbf{word}_{h_{k}}\in\mathbb{R}^{T\times d_{h}}$,
$\mathbf{word}_{h_{v}}\in\mathbb{R}^{T\times d_{h}}$,
$\mathbf{word}_{w_{k}}\in\mathbb{R}^{T\times d_{w}}$,
$\mathbf{word}_{w_{v}}\in\mathbb{R}^{T\times d_{w}}$, through four 1$\times$1
convolution layers:

$\begin{split}\mathbf{word}_{h_{k}}&=\mathbf{w}^{2}_{h_{k}}\left(\mathbf{L}\right)+\mathbf{b}^{2}_{h_{k}},\\\
\mathbf{word}_{h_{v}}&=\mathbf{w}^{2}_{h_{v}}\left(\mathbf{L}\right)+\mathbf{b}^{2}_{h_{v}},\\\
\mathbf{word}_{w_{k}}&=\mathbf{w}^{2}_{w_{k}}\left(\mathbf{L}\right)+\mathbf{b}^{2}_{w_{k}},\\\
\mathbf{word}_{w_{v}}&=\mathbf{w}^{2}_{w_{v}}\left(\mathbf{L}\right)+\mathbf{b}^{2}_{w_{v}}.\end{split}$
(2)

Then, those word vectors are passed to the row-wise inference and the column-
wise inference branches, respectively, to fully capture the two directional
interactions. In detail, it feeds the row-wise and column-wise visual features
and the corresponding word vectors into two cross-attention mechanisms
separately, to calculate the language perception of the row-and-column level
pixels in the image. Since the implementations of these two cross-attention
mechanisms are the same, for simplicity, we only take the row-wise inference
as an example. Specifically, the common cross-attention mechanisms are
utilized to learn the row-wise influence by feeding the vertical visual
feature $\mathbf{v}_{h}$ to query the linguistic feature
$\mathbf{word}_{h_{k}}$ and generate the vertical linguistic feature.
$Attention$ is the simple Scaled Dot-Product Attention mechanism and can be
expressed by:

$\begin{split}Attention(Q,K,V)&=softmax\left(\frac{QK^{\top}}{\sqrt{d_{k}}}\right)V.\end{split}$
(3)

After obtaining the vertical linguistic feature which have the same shape as
$\mathbf{v}_{h}$, we combine them to produce a set of vertical multi-modal
feature maps $\mathbf{v}_{h}^{Att}$ via element-wise multiplication. Formally,

$\begin{split}\mathbf{v}_{h}^{Att}=Attention\left(\mathbf{v}_{h},\mathbf{word}_{h_{k}},\mathbf{word}_{h_{v}}\right)\odot\mathbf{v}_{h},\end{split}$
(4)

where $\mathbf{v}_{h}^{Att}\in\mathbb{R}^{H\times C_{h}}$, $\odot$ denotes
element-wise multiplication. In the same way, we use another cross-attention
layer to generate the horizontal multi-modal feature maps
$\mathbf{v}_{w}^{Att}\in\mathbb{R}^{W\times C_{w}}$. Finally, we use the
Bilinear Interpolation to resize $\mathbf{v}_{h}$, $\mathbf{v}_{w}$,
$\mathbf{v}_{h}^{Att}$, $\mathbf{v}_{w}^{Att}$ to the scale of the original
image, which are added up as the output of the Row-and-Column interactive
module for further fusion:

$\begin{split}\mathbf{v}_{hw}^{all}=B\left(\mathbf{v}_{h}\right)+B\left(\mathbf{v}_{w}\right)+B\left(\mathbf{v}_{h}^{Att}\right)+B\left(\mathbf{v}_{w}^{Att}\right),\\\
\end{split}$ (5)

where $\mathbf{v}_{hw}^{all}\in\mathbb{R}^{H\times W\times C}$ and $B$ denotes
the Bilinear Interpolation layer. It is worth noting that in order to preserve
the row-wise and column-wise visual features of the image, we also use
Bilinear Interpolation for $\mathbf{v}_{h}$, $\mathbf{v}_{w}$ and add them to
the final output, which is not shown in the above figures.

Furthermore, to enable the positioning effect of the RoCo module to guide the
Holi module, we also design a global guidance path that utilizes the learned
horizontal and vertical attention maps to build the global perception of the
image, giving the Holi module a referent location prior
$\mathbf{Mask}_{roco}\in\mathbb{R}^{H\times W\times T}$, which can be
formulate as

$\begin{split}\mathbf{Mask}_{roco}=\frac{\mathbf{e}_{h}*\mathbf{e}_{w}^{\top}}{{\sum_{HW}\left(\mathbf{e}_{h}*\mathbf{e}_{w}^{\top}\right)}},\\\
\mathbf{e}_{h}=\sigma\left(\frac{\mathbf{word}_{h_{k}}\mathbf{v}_{h}^{\top}}{\sqrt{d_{h}}}\right),\\\
\mathbf{e}_{w}=\sigma\left(\frac{\mathbf{word}_{w_{k}}\mathbf{v}_{w}^{\top}}{\sqrt{d_{w}}}\right),\\\
\end{split}$ (6)

where $\mathbf{e}_{h}\in\mathbb{R}^{H\times T}$,
$\mathbf{e}_{w}\in\mathbb{R}^{W\times T}$ represent the vertical and
horizontal attention maps, respectively. $T$ denote the number of words and
$\sigma$ denote the $softmax$ function. The generated $\mathbf{Mask}_{roco}$
will be used to guide the Holi module which is depicted in Sec. III-C2. In the
above process, some small regions in the holistic feature map can be enhanced.
In other words, some small-scale non-salient objects in the image could be
explicitly located like salient objects, resulting in more accurate
segmentation masks.

####  III-C2 Guided Holistic interactive module

As illustrated in Fig. 5, our Guided Holistic interactive (Holi) module
establishes the attention correlations between the holistic visual features
and the linguistic features. Like previous methods, we maintain the scale of
the visual feature map during the multi-modal feature interaction, which in
turn captures the perception between the language words and the image pixels.
In detail, an 1$\times$1 convolution layer are implemented to obtain the
holistic visual feature $\mathbf{v}_{g}\in\mathbb{R}^{H\times W\times C}$,
which can be formulate as

$\begin{split}\mathbf{v}_{g}&=\mathbf{w}^{3}_{g}(\mathbf{V})+\mathbf{b}^{3}_{g}.\\\
\end{split}$ (7)

In addition, from the linguistic feature $\mathbf{L}$, we also utilize two
1$\times$1 convolution layers to generate
$\mathbf{word}_{g_{k}}\in\mathbb{R}^{T\times d_{g}}$ and
$\mathbf{word}_{g_{v}}\in\mathbb{R}^{T\times d_{g}}$.

$\begin{split}\mathbf{word}_{g_{k}}&=\mathbf{w}^{3}_{g_{k}}\left(\mathbf{L}\right)+\mathbf{b}^{3}_{g_{k}},\\\
\mathbf{word}_{g_{v}}&=\mathbf{w}^{3}_{g_{v}}\left(\mathbf{L}\right)+\mathbf{b}^{3}_{g_{v}}.\\\
\end{split}$ (8)

Then, a novel Guided Attention layer is designed to capture the multi-modal
interactions between the linguistic features and the holistic visual features.
Specifically, under the global guidance of the RoCo module, we first use a
simple attention layer to calculate the holistic attention map
$\mathbf{Mask}_{holi}\in\mathbb{R}^{H\times W\times T}$, and then fuse it with
$\mathbf{Mask}_{roco}$ to generate the guided holistic attention map
$\mathbf{Mask}_{roho}\in\mathbb{R}^{H\times W\times T}$, which could be
defined as

$\begin{split}\mathbf{Mask}_{roho}=\frac{\left(\mathbf{Mask}_{roco}+\mathbf{Mask}_{holi}\right)}{2},\\\
\end{split}$ (9)
$\begin{split}\mathbf{Mask}_{holi}=\sigma\left(\frac{\mathbf{v}_{g}\mathbf{word}_{g_{k}}^{\top}}{\sqrt{d_{g}}}\right).\\\
\end{split}$ (10)

Finally, based on the guided holistic attention map $\mathbf{Mask}_{roho}$, we
can get the holistic linguistic features, which have the same shape as
$\mathbf{v}_{g}$, and then combine them to produce a set of holistic multi-
modal feature maps $\mathbf{v}_{g}^{all}\in\mathbb{R}^{H\times W\times C}$ as
the output of the Guided Holistic interactive module. Formally,

$\begin{split}\mathbf{v}_{g}^{all}=\left(\mathbf{Mask}_{roho}*\mathbf{word}_{g_{v}}\right)\odot\mathbf{v}_{g},\\\
\end{split}$ (11)

where $*$ denotes matrix multiplication and $\odot$ denotes element-wise
multiplication.

####  III-C3 Merging two pathways

In the following steps, a Feed Forward Network is utilized to fuse the outputs
of these two parallel branches. Firstly, we use two convolution layers
followed by ReLU nonlinearity to perform feature projection on
$\mathbf{v}_{hw}^{all}$ and $\mathbf{v}_{g}^{all}$ and mathematically
described as follows

$\begin{split}\mathbf{F}_{hw}&=ReLU\left(\mathbf{w}_{hw}^{4}\left(\mathbf{v}_{hw}^{all}\right)+\mathbf{b}_{hw}^{4}\right),\end{split}$
(12)
$\begin{split}\mathbf{F}_{g}&=ReLU\left(\mathbf{w}_{g}^{4}\left(\mathbf{v}_{g}^{all}\right)+\mathbf{b}_{g}^{4}\right),\end{split}$
(13)

where $\mathbf{F}_{hw},\mathbf{F}_{g}\in\mathbb{R}^{H\times W\times C}$. After
that, the above two multi-model features are added up and follow by a feed
forward network to generate the fused multi-modal features
$\mathbf{F}\in\mathbb{R}^{H\times W\times C}$, involving the joint reasoning
information learned by the RoCo and Holi modules.

$\begin{split}\mathbf{F}&=update(FFN\left(\mathbf{F}_{hw}+\mathbf{F}_{g}\right),\mathbf{V}),\\\
\end{split}$ (14)

where $FFN$ denotes the traditional feed forword network which contains two
linear projection layers and a ReLU nonlinear layer followed by dropout
function. $update$ denotes the residual connection. In fact, $\mathbf{F}$ is
the final output of our proposed CPRN. Under the cooperation and guidance
between the RoCo module and the Holi module, our proposed CPRN block can
better locate the referents and obtain more fine-grained results, making up
for the previous models that only rely on the holistic visual feature map to
locate the referents.

###  III-D Multi-Scale Decoder module

As illustrate in Fig. 4, we combine the multi-modal features of different
stages via a Multi-Scale Decoder module which progressively integrates these
features $\\{\mathbf{F}_{i}\\}_{i=0}^{N}$ from high-level to low-level
semantics as follows:

$\begin{split}\mathbf{Y}_{i}&=Upsampling(Proj\left([\mathbf{Y}_{i+1},\mathbf{F}_{i+1}]\right)).\\\
\end{split}$ (15)

where $Upsampling$ represents upsampling operation on the feature map via
bilinear interpolation and $Proj$ indicates that the linear projection
function is used to transform the channel dimension. Specifically, the output
of final stage $\mathbf{Y}_{N}$ is equal to $\mathbf{F}_{N}$, and $N$
represents the number of different stages. At last, the final feature maps,
$\mathbf{Y}_{1}$, are fed into an $1\times 1$ convolution layer to produce a
2-D probability score map $y^{\prime}\in\left(0,1\right)$ normalized with
sigmoid function. During training, a binary cross entropy loss function are
utilized to calculate the loss between the predicted score map $y^{\prime}$
and ground truth label $y$, which can be formulated as follow:

$\begin{split}L=-\frac{1}{\mathcal{Z}}\sum^{\mathcal{Z}}_{n=0}[y\left(n\right)log(y^{\prime}\left(n\right))+(1-y\left(n\right))log(1-y^{\prime}\left(n\right))]\end{split}$
(16)

where $n$ represents the n-th image pixel, and $\mathcal{Z}$ is the number of
pixels in the input image. The detailed process of our CPRN block is expressed
in algorithm 1.

Input: Images $I$, Language expression $E$;

Output: Segmentation result $y^{\prime}$;

Extracting the feature $\bf{V}$ with the help of Swin Transformer;

Extracting the feature $\bf{L}$ with the help of BERT;

for  _stage $i\to N$_ do

$\mathbf{v}_{h},\mathbf{v}_{w}=RoCo\\_Aggregation\left(\bf{V}\right)$;

$\mathbf{v}_{h}^{Att},\mathbf{v}_{w}^{Att}=Cross\\_Attention\left(\mathbf{v}_{h},\mathbf{v}_{w};\mathbf{L}\right)$;

$\bm{v}_{hw}^{all}=B\left(\bm{v}_{h}\right)+B\left(\bm{v}_{w}\right)+B\left(\bm{v}_{h}^{Att}\right)+B\left(\bm{v}_{w}^{Att}\right)$;

Calculate the referent location prior $\mathbf{Mask}_{roco}$ through the
global guidance path;

Calculate the holistic attention map $\mathbf{Mask}_{holi}$ through a simple
attention layer;

Get the guided holistic attention map $\mathbf{Mask}_{roho}$ by fusing
$\mathbf{Mask}_{roco}$ and $\mathbf{Mask}_{holi}$;

$\mathbf{v}_{g}^{all}=Guided\\_Attention\left(\mathbf{v}_{g};\mathbf{L}|\mathbf{Mask}_{roho}\right)$;

$\mathbf{F}_{hw},\mathbf{F}_{g}=projection(\mathbf{v}_{hw}^{all},\mathbf{v}_{g}^{all})$;

$\mathbf{F}=update\left(FFN\left(\mathbf{F}_{hw}+\mathbf{F}_{g}\right),\mathbf{V}\right)$;

Get the fusion feature $\bf{F}$ through the multi-scale decoder;

Calculate the Segmentation results $y^{\prime}$ through a simple segmentation
head;

Algorithm 1 Framework of our CPRN. TABLE I: Comparison with state-of-the-art methods on three benchmark datasets using _overall IoU_ as metric. U: The UMD split. G: The Google split. | RefCOCO | RefCOCO+ | Gref
---|---|---|---
| val | test A | test B | val | testA | testB | val (U) | test (U) | val(G)
RRN [6] | 55.33 | 57.26 | 53.93 | 39.75 | 42.15 | 36.11 | - | - | 36.45
CSMA [15] | 58.32 | 60.61 | 55.09 | 43.76 | 47.60 | 37.89 | - | - | 39.98
BRINet [34] | 60.98 | 62.99 | 59.21 | 48.17 | 52.32 | 42.11 | - | - | 48.04
CMPC [58] | 61.36 | 64.53 | 59.64 | 49.56 | 53.44 | 43.23 | - | - | 49.05
LSCM [29] | 61.47 | 64.99 | 59.55 | 49.34 | 53.12 | 43.50 | - | - | 48.05
EFN [11] | 62.76 | 65.69 | 59.67 | 51.50 | 55.24 | 43.01 | - | - | 51.93
BUSNet [61] | 63.27 | 66.41 | 61.39 | 51.76 | 56.87 | 44.13 | - | - | 50.56
VLT [60] | 65.65 | 68.29 | 62.73 | 55.50 | 59.20 | 49.36 | 52.99 | 56.65 | 49.76
LTS [62] | 65.43 | 67.76 | 63.08 | 54.21 | 58.32 | 48.02 | 54.40 | 54.25 | -
ReSTR [63] | 67.22 | 69.30 | 64.45 | 55.78 | 60.44 | 48.27 | 54.48 | - | -
CRIS [64] | 70.47 | 73.18 | 66.10 | 62.27 | 68.08 | 53.68 | 59.87 | 60.36 | -
LAVT [66] | 72.73 | 75.82 | 68.79 | 62.14 | 68.38 | 55.10 | 61.24 | 62.09 | 60.50
CPRN (Ours) | 73.42 | 76.65 | 70.84 | 63.58 | 69.44 | 55.84 | 62.81 | 64.25 | 60.92

##  IV Experiments

###  IV-A Datasets and Experiment Setup

TABLE II: Ablation studies on RefCOCO validation set. “&” and “$\|$” represent the series and parallel connection of RoCo module and Holi module, respectively. ”Holi*” represents a simple cross attention mechanism, and ”Holi” represents our proposed guided holi interactive module. Method |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>| Overall IoU | Mean IoU
---|---|---|---|---|---|---|---
baseline (Holi*) | 83.26 | 79.31 | 73.38 | 62.29 | 32.36 | 71.99 | 73.10
RoCo | 79.27 | 74.68 | 67.35 | 50.45 | 15.11 | 66.63 | 67.61
RoCo & Holi* | 80.61 | 75.81 | 67.92 | 50.98 | 17.58 | 68.57 | 69.02
RoCo $\|$ Holi* | 84.56 | 80.65 | 75.10 | 64.21 | 33.90 | 72.79 | 74.29
RoCo $\|$ Holi | 84.58 | 81.21 | 75.91 | 64.28 | 34.04 | 72.96 | 74.48
+FFN | 84.66 | 81.49 | 76.21 | 65.23 | 35.00 | 73.12 | 74.60
+ape (CPRN) | 85.09 | 81.71 | 76.54 | 65.53 | 35.26 | 73.42 | 75.00

Datasets: We use three datasets to evaluate our method: RefCOCO [30], RefCOCO+
[30] and Gref [27].

The RefCOCO [30] set contains 19,994 images and 142,209 citation expressions
for 50,000 objects obtained from the MS COCO dataset [1], and the average
length of Refcoco expressions is 3.61. The set annotation comes from the two-
player game interaction [28], where two or more objects of the same object
class appear in each image.

The RefCOCO+ [30] set contains 141,564 expressions for 49,856 objects in
19,992 images, and the average length of Refcoco+ is 3.53. These images were
also collected from the MS COCO dataset, with a limitation that position words
are not allowed in expressions. Refcoco and Refcoco+ do not limit the number
of objects of the same category to 4, so containing some images with many
objects of the same category, Refcoco, and Refcoco+ both average 3.9 same
category objects per image.

The Gref [27] set is also collected from the MS COCO dataset. There are
104,560 expressions involving 54,822 objects in 26,711 images. The expressions
for this dataset are collected on Mechanical Turk through independent rounds,
rather than using the two-player game. The expressions in Gref are longer and
more complex than RefCOCO and RefCOCO+, with Gref containing an average of
8.43 words. We use the same split as in [27], this dataset has two different
splits, one is UMD and the other is Google, abbreviated as Gref-umd and Gref-
google. Gref has an average of 1.63 same category objects per image.

Metrics: Following previous works [9, 15, 66, 64], we adopt overall
Intersection-over-Union (_Overall IoU_) and _Pre@X_ as our evaluation metrics.
Given the predicted segmentation mask and the ground truth, the _Overall IoU_
metric is the ratio between the intersection and the union of the two, which
is calculated by dividing the total intersection area by the total union area.
Both intersection area and union area are accumulated over all test samples.
The _Pre@X_ measures the percentage of test examples that have _IoU_ score
higher than the threshold $X$. In our experiments,
$X\in\\{0.5,0.6,0.7,0.8,0.9\\}$.

Implementation Details: Given an input image, we resize it to $480\times 480$
and adopt Swin Transformer [23] pre-trained on ImageNet-22K dataset [54] as
our backbone, following previous works [66]. On all three benchmark datasets,
we keep the maximum length of query expression as 20. The language model we
use is a BERT [24] model with 12 layers, a hidden size of 768, and is
initialized with official pre-trained weights. The number of inference stage
$N$ is equal to 4. Our model is optimized with a binary cross-entropy loss,
and we employ the AdamW optimizer [37] with a weight decay of $0.01$. We
employ a learning rate schedule with an initial learning rate set to $5e^{-5}$
and a polynomial learning rate decay. We use the batch size of 32 and train on
8 Tesla V100 with 16 GPU VRAM. During inference, we upsample the prediction
results back to the original image size and use _argmax_ to select the index
on the channel dimension of the score map, no other post-processing operations
are required.

Figure 6: Visualization of the attention map of the Guided Attention layer.
(a) Original image. (b) RoCo Attention map. (c) Holi Attention map. (d) RoHo
Attention map. (e) Segmentation. (f) Ground-truth  Figure 7: Visualization of
comparisons between LAVT and CPRN segmentation results on the small-scale
object and complex language set from the Gref-umd validation set. (a) Original
image. (b) LAVT. (c) CPRN. (d) Ground-truth.  Figure 8:  Visualization of
segmentation and feature map at different stages from the Gref-umd validation
set. The left-most column shows the original image, and the right-most column
illustrates the predicted mask and the ground truth mask. Figure 9:
Visualization of comparisons between LSCM, LAVT and CPRN segmentation results
from the RefCOCO+ testA set. (a) Original image. (b) LSCM. (c) LAVT. (d) CPRN.
(e) Ground-truth.

###  IV-B Quantitative Results

Comparison with State-of-the-arts: To demonstrate the superiority of our CPRN,
we compare it with state-of-the-art methods, including RRN [6], CSMA [15],
BRINet [34], CMPC [58], LSCM [29], EFN [11], BUSNet [61], VLT [60], LTS [62],
ReSTR [63], CRIS [64] and LAVT [66] on three RIS benchmarks. The results of
the comparison with other methods on three datasets using _overall IoU_ as
metrics are displayed in Tab. I. It can be seen that our method consistently
outperforms all previous methods on three benchmarks. In detail, CPRN yields
an average improvement of 7.53% over ReSTR on all three datasets. Similarly,
it achieves a significant improvement in the range of 1.36%-4.71% compared to
CRIS, which had previously achieved the best performance. In particular, our
method also greatly outperforms LAVT using the same backbone, which fully
illustrates the advantages of our proposed CPRN block. Specifically, on the
refcoco+ dataset, our method exceeds LAVT by 1.44% on val split, 1.06% on
testA split, and 0.74% on testB split. Since this dataset does not contain
location information in referring expressions, it turns out that our CPRN
block can enhance the ability to locate referents more than other methods that
lack explicit modeling of positioning. According to Tab. I, the gains are more
obvious on other datasets than on the RefCOCO+ dataset, indicating that the
ability of the CPRN block to locate language-responsive regions has a great
impact on improving model performance. More importantly, compared to the sota
model, our network achieves 1.57% and 2.16% improvement on Gref-umd validation
and test sets. Note that the referring expressions in the Gref dataset are
generally longer, indicating that CPRN can better handle complex long
sentences by positioning interactions between visual and linguistic features.
In the end, the above comparison with existing methods fully demonstrates the
superiority of our method.

TABLE III: Comparison with LAVT on the Gref-umd validation set on a small dataset of small scale objects and complex languages. | Methods |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>| Overall IoU | Mean IoU
---|---|---|---|---|---|---|---|---
small scale | LAVT | 43.96 | 36.24 | 28.32 | 18.81 | 2.57 | 28.54 | 41.63
CPRN | 48.91 | 40.20 | 33.27 | 20.99 | 1.78 | 31.65 | 45.45
complex language | LAVT | 69.08 | 63.77 | 57.00 | 44.69 | 21.74 | 58.26 | 61.65
CPRN | 72.46 | 70.29 | 64.73 | 52.66 | 25.12 | 61.88 | 64.77
TABLE IV: Experiments with five feature combination way of RoCo module on RefCOCO validation set. Methods |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>| Overall IoU | Mean IoU
---|---|---|---|---|---
$f_{1}$ | 83.78 | 74.81 | 34.54 | 72.46 | 73.90
$f_{2}$ | 84.55 | 75.00 | 34.49 | 73.21 | 74.28
$f_{3}$ | 84.83 | 76.38 | 34.08 | 72.93 | 74.75
$f_{4}$ | 85.07 | 76.66 | 34.93 | 73.17 | 74.83
CPRN | 85.09 | 76.54 | 35.24 | 73.42 | 75.00

###  IV-C Ablation Studies

To investigate the relative contribution of each component in the proposed
modules and the localization ability of the CPRN block, we conduct a series of
ablation experiments on the RefCOCO dataset and evaluate it in the validation
set, which is illustrated in Tab. II. In addition, we study the combination of
row- and column-wise features in RoCo module, such as Tab. IV. Furthermore, to
verify that our CPRN block is more effective on some non-salient objects with
small scale and complex language expressions, we also conduct a plenty of
experiments on the Gref-umd dataset and use _overall IoU_ , _mean IoU_ and
_Pre@X_ as evaluation metrics, such as Tab. III.

Effective of RoCo module and Holi module.  To investigate the contribution of
the Row-and-Column interactive (RoCo) module and Guided Holistic interactive
(Holi) module to the overall model performance, we design four sets of
ablation experiments. The baseline network is built with a simple holistic
module (Holi* module) which only contains a cross-attention layer. We analyze:
(1) Holi* module(Holi*), (2) RoCo module(RoCo), (3) RoCo module and Holi*
module are combined in series(RoCo & Holi*), (4) RoCo module and Holi* module
are combined in parallel(RoCo $\|$ Holi*), (5) RoCo module and Holi module
merged in two pathways(RoCo $\|$ Holi). The results in Tab. II show that since
Holi* only uses a simple cross-attention mechanism, the segmentation
performance of the model is not excellent. Due to the RoCo module is only
responsible for learning the modeling of positioning referents and lacks the
perception of the global information of images, the result of the individual
Roco module is poor. In addition, we try the combination of the RoCo module
and Holi* module in simple series and simple parallel, and both of the model
performances do not improve greatly. We also analyzed the reasons for the poor
performance of serial combination. This is because the visual features are
multi-modal interactively fused with language features in the RoCo module, and
then sent into the Holi* module, the global information of the original visual
features will be destroyed. For the parallel method, the model performance can
already exceed using Holi* module alone. Based on Holi module and RoCo module
in parallel, our proposed CPRN has a better ability to localize referring
entities than the scheme only using single Holi* module, scheme only using
single Roco module, simple series scheme, and simple parallel scheme. In order
to further improve the performance, we have made some improvements, adding
absolute postion embedding(ape) to the visual features and adding the FFN
network layer after the combination of the RoCo module and the Holi module.
Through experimental verification, these improvements will bring certain
improvements to the model performance.

Performance on small-scale objects and complex language expressions.  In order
to demonstrate the effectiveness of our CPRN block, especially the positioning
ability of small-scale non-salient objects, and the joint reasoning ability of
complex language expressions, we constructed two small datasets on the Gref-
umd dataset for further ablation studies. In detail, we use thresholds of 0.03
and 18 as criteria for segmenting small-scale objects and complex language
expressions, separately. We consider data with the mask rate less than 0.03 as
small-scale objects, and the proportion of this small dataset is 10.31%.
Besides, We consider data with language expression length longer than 18 after
tokenizer as complex language queries, and this small dataset accounts for
8.46%. The result can be seen from Tab. III that CPRN outperforms the sub-
optimal method LAVT by absolute advantage on those two reconstructed datasets
with small-scale objects and complex languages, and is higher than the Overall
IoU 3.11% and 3.62%. The visualization of these two small datasets also shows
the advantage of our model in Fig. 7. The left column is the visualization of
small-scale objects, and the right column is the visualization of complex
language expressions. The visualization clearly illustrate that our model has
great advantages, especially in solving the segmentation problem of small-
scale objects and complex language expressions in the RIS task.

The combination of row- and column-wise features.  In order to verify the
combination of row- and column-wise muti-modal features, we designed four
different fusion ways to generate $\mathbf{{v}_{hw}^{all}}$, which are the
functions $f_{1}$, $f_{2}$, $f_{3}$ and $f_{4}$ as as follows:

$\displaystyle
f_{1}=\left(\mathbf{v}_{h}^{Att}\otimes\mathbf{v}_{w}^{Att}\right)+\textbf{V}$
(17) $\displaystyle
f_{2}=\left(\mathbf{v}_{h}^{Att}\otimes\mathbf{v}_{w}^{Att}\right)*\textbf{V}$
(18) $\displaystyle
f_{3}={C}\left({B}\left(\mathbf{v}_{h}+\mathbf{v}_{h}^{Att}\right),{B}\left(\mathbf{v}_{w}+\mathbf{v}_{w}^{Att}\right)\right)$
(19) $\displaystyle
f_{4}={C}\left({C}\left({B}(\mathbf{v}_{h}),{B}(\mathbf{v}_{h}^{Att})\right),{C}\left({B}(\mathbf{v}_{w}),{B}(\mathbf{v}_{w}^{Att})\right)\right)$
(20)

where $B$ denotes the Bilinear Interpolation and $C$ is a function implemented
as concatenating in the channel dimension, and then connect a $1\times 1$
convolution. Actually, the combination function we use in CPRN block is Eq.5
and the experimental results in Tab. IV also demonstrate that it is optimal
way.

###  IV-D Visualization

Fig. 6 shows the visualization of attention maps in the Guided attention
layer, which are RoCo Attention map (b), Holi Attention map (c), and RoCo and
Holi Attention map(d). In order to better highlight the positioning effect of
our CPRN block, the visualization of the RoCo attention map is the effect of
superimposing the row and column features together. As can be seen from the
visual attention map, our RoCo attention map can assist Holi attention in
distinguishing confusing instance objects and guide the Holi module to segment
the correct reference object. Fig. 8 is a visualization of the features at
different stages, we can observe that the feature maps of each stage in CPRN
(ie $\rm Stage_{4}$, $\rm Stage_{3}$, $\rm Stage_{2}$, $\rm Stage_{1}$) can
accurately locate the semantic concepts referred by natural language
expression. Fig. 9 shows the visualization results of LSCM (Fig. 9(b)), LAVT
(Fig. 9(c)), CPRN (Fig. 9(d)) and ground-truth (Fig. 9(e)) on RefCOCO+ testA
set. The referring expressions in the RefCOCO+ dataset do not include words
representing spatial or positional information, which places higher demands on
the ability to understand the appearance of objects. A comparison of
visualization results shows that our proposed CPRN block can positioning
referring entities effectively even in the absence of explicit location
information. Furthermore, we are able to reason about complex textual
information to obtain the final segmented referents.

Taking the first case as an example, given the query expression ”back to us
white”, CPRN can obtain the location information of the instance referent.
Moreover, for the language expression ”man on floor”, our method can also
accurately locate the target. Both LSCM and LAVT cannot accurately localize
language-responsive region relying solely on global information of visual
image. Furthermore, it can be seen that our CPRN can accurately segment
referring objects even in language representations of different lengths and
complex scenes, such as ”black coat looking at suitcase black hat”, LSCM model
mis-segments multiple objects, LAVT model is determine difficultly whether the
segmentation goal is ”black coat” or ”suitcase black hat”, but CPRN can
distinguish segmentation objects, and rows 4 and 5 also show similar results.

From the overall visualization results, it can be seen that CPRN can
accurately positioning entities without bringing too much redundant mask
information, which is crucial for improving the performance of referring image
segmentation task.

##  V Conclusion

In this work, we propose a collaborative positioning reasoning network for
referring image segmentation task, which can efficiently locate the referring
entities with detailed edges, even small-scale objects or incomprehensible
natural languages Express. Under the architecture of multi-semantic inference
network, we adopt RoCo module and Holi module in parallel for each semantic
stage. After dividing the overall visual feature map into horizontal direction
map and vertical direction map, RoCo Module fuses them with texture
information respectively, and the position of referring objects can be more
accurate. The Holi module preserves the overall feature map to ensure the
integrity of the global information, while the RoCo module guides the Holi
module through a global guided pathway to generate correct segmentation
results. The proposed method achieves state-of-the-art performance on three
challenge benchmarks.

## References

  * [1] Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Dollár, Piotr and Zitnick, C Lawrence, “Microsoft coco: Common objects in context,” in _European conference on computer vision_ , 2014.
  * [2] Wang, Liwei and Li, Yin and Lazebnik, Svetlana, “Learning Deep Structure-Preserving Image-Text Embeddings,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [3] Xu, Kelvin and Ba, Jimmy and Kiros, Ryan and Cho, Kyunghyun and Courville, Aaron and Salakhutdinov, Ruslan and Zemel, Richard and Bengio, Yoshua, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention,” in _Computer Science_ , pp. 2048-2057, 2015.
  * [4] Antol, Stanislaw and Agrawal, Aishwarya and Lu, Jiasen and Mitchell, Margaret and Batra, Dhruv and Zitnick, C Lawrence and Parikh, Devi, “VQA: Visual Question Answering,” in _International Conference on Computer Vision (ICCV)_ , 2015.
  * [5] Hu, R. and Rohrbach, M. and Darrell, T, “Segmentation from Natural Language Expressions,” in _European Conference on Computer Vision_ , 2016.
  * [6] Li, Ruiyu and Li, Kaican and Kuo, Yi-Chun and Shu, Michelle and Qi, Xiaojuan and Shen, Xiaoyong and Jia, Jiaya, “Referring Image Segmentation via Recurrent Refinement Networks,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018.
  * [7] Shi, Hengcan and Li, Hongliang and Meng, Fanman and Wu, Qingbo, “Key-Word-Aware Network for Referring Expression Image Segmentation,” in _European Conference on Computer Vision_ , 2018.
  * [8] Chen, Ding Jie and Jia, Songhao and Lo, Yi Chen and Chen, Hwann Tzong and Liu, Tyng Luh, “See-Through-Text Grouping for Referring Image Segmentation,” in _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * [9] Liu, Chenxi and Lin, Zhe and Shen, Xiaohui and Yang, Jimei and Lu, Xin and Yuille, Alan, “Recurrent Multimodal Interaction for Referring Image Segmentation,” in _IEEE Computer Society_ , 2017.
  * [10] MargFfOy-Tuay, E. and JC Pérez and Botero, E. and P Arbeláez, “Dynamic Multimodal Instance Segmentation guided by natural language queries,” in _CoRR_ , 2018.
  * [11] Feng, G. and Hu, Z. and Zhang, L. and Lu, H, “Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation,” in _CoRR_ , 2021.
  * [12] Luo, G. and Zhou, Y. and Sun, X. and Cao, L. and Wu, C. and Deng, C. and Ji, R, “Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation,” in _CVPR_ , 2020.
  * [13] Chen, Y. W. and Tsai, Y. H. and Wang, T. and Lin, Y. Y. and Yang, M. H, “Referring Expression Object Segmentation with Caption-Aware Consistency,” in _BMVC_ , 2019.
  * [14] Huang, Zilong and Wang, Xinggang and Huang, Lichao and Huang, Chang and Wei, Yunchao and Liu, Wenyu, “Ccnet: Criss-cross attention for semantic segmentation,” in _ICCV_ , 2019.
  * [15] Ye, Linwei and Rochan, Mrigank and Liu, Zhi and Wang, Yang, “Cross-Modal Self-Attention Network for Referring Image Segmentation,” in _CVPR_ , 2019.
  * [16] Shi, Xingjian and Chen, Zhourong and Wang, Hao and Yeung, Dit Yan and Wong, Wai Kin and Woo, Wang Chun, “Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting,” in _CoRR_ , 2015.
  * [17] Long, Jonathan and Shelhamer, Evan and Darrell, Trevor, “Fully Convolutional Networks for Semantic Segmentation,” in _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, no. 4, pp. 640-651, 2015.
  * [18] Lin, Guosheng and Milan, Anton and Shen, Chunhua and Reid, Ian, “RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [19] Badrinarayanan, Vijay and Kendall, Alex and Cipolla, Roberto, “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation,” in _IEEE Transactions on Pattern Analysis & Machine Intelligence_, pp. 1-1, 2017.
  * [20] Zhao, Hengshuang and Shi, Jianping and Qi, Xiaojuan and Wang, Xiaogang and Jia, Jiaya, “Pyramid Scene Parsing Network,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [21] Hang Zhang and Kristin J. Dana and Jianping Shi and Zhongyue Zhang and Xiaogang Wang and Ambrish Tyagi and Amit Agrawal, “Context Encoding for Semantic Segmentation,” in _CoRR_ , 2018.
  * [22] Chen, L. C. and Papandreou, G and Kokkinos, I and Murphy, K and Yuille, A. L, “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs,” in _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 4, pp. 834-848, 2018.
  * [23] Liu, Ze and Lin, Yutong and Cao, Yue and Hu, Han and Wei, Yixuan and Zhang, Zheng and Lin, Stephen and Guo, Baining, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _ICCV_ , 2021.
  * [24] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in _NAACL_ , 2019.
  * [25] Chen, Liang Chieh and Papandreou, George and Schroff, Florian and Adam, Hartwig, “Rethinking Atrous Convolution for Semantic Image Segmentation,” in _CoRR_ , 2017.
  * [26] Hochreiter and Sepp and Schmidhuber and Jurgen, “Long short-term memory,” in _Neural Computation_ , 1997.
  * [27] Mao, Junhua and Huang, Jonathan and Toshev, Alexander and Camburu, Oana and Murphy, Kevin, “Generation and Comprehension of Unambiguous Object Descriptions,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [28] Kazemzadeh, Sahar and Ordonez, Vicente and Matten, Mark and Berg, Tamara, “ReferItGame: Referring to Objects in Photographs of Natural Scenes,” in _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2014.
  * [29] Tianrui Hui, Si Liu, Shaofei Huang, Guanbin Li, Sansi Yu, Faxi Zhang, and Jizhong Han , “Linguistic structure guided context modeling for referring image segmentation,” in _European Conference on Computer Vision_ , 2020.
  * [30] Yu, Licheng and Poirson, Patric and Yang, Shan and Berg, Alexander C. and Berg, Tamara L, “Modeling Context in Referring Expressions,” in _ECCV_ , 2016.
  * [31] Escalante, Hugo Jair and Hernandez, Carlos A. and Gonzalez, Jesus A. and Lopez-Lopez, A. and Montes, Manuel and Morales, Eduardo F. and Sucar, L Enrique and Villasenor, Luis and Grubinger, Michael, “The segmented and annotated IAPR TC-12 benchmark,” in _Computer Vision and Image Understanding_ , vol. 114, no. 4, pp. 419-42, 2010.
  * [32] Lu, Jiasen and Batra, Dhruv and Parikh, Devi and Lee, Stefan, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” in _Advances in neural information processing systems_ , 2019.
  * [33] Hui, Tianrui and Liu, Si and Huang, Shaofei and Li, Guanbin and Han, Jizhong, “Linguistic Structure Guided Context Modeling for Referring Image Segmentation,” in _ECCV_ , 2020.
  * [34] Hu, Zhiwei and Feng, Guang and Sun, Jiayu and Zhang, Lihe and Lu, Huchuan, “Bi-Directional Relationship Inferring Network for Referring Image Segmentation,” in _CVPR_ , 2020.
  * [35] Everingham, Mark and Gool, Luc Van and Williams, Christopher K. I. and Winn, John and Zisserman, Andrew, “The Pascal Visual Object Classes (VOC) Challenge,” in _International Journal of Computer Vision_ , vol. 88, no. 2, pp. 303-338, 2010.
  * [36] Krhenbühl, Philipp and Koltun, Vladlen, “Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials,” in _Curran Associates Inc._ , 2012.
  * [37] Loshchilov, I. and Hutter, F., “Decoupled Weight Decay Regularization,” in _ICLR_ , 2019.
  * [38] Yu, Licheng and Lin, Zhe and Shen, Xiaohui and Yang, Jimei and Lu, Xin and Bansal, Mohit and Berg, Tamara L, “MAttNet: Modular Attention Network for Referring Expression Comprehension,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018.
  * [39] Ben-Younes, Hedi and Cadene, Rémi and Cord, Matthieu and Thome, Nicolas, “MUTAN: Multimodal Tucker Fusion for Visual Question Answering,” in _ICCV_ , 2017.
  * [40] He, Kaiming and Gkioxari, Georgia and Piotr Dollár and Girshick, Ross, “Mask R-CNN,” in _IEEE Transactions on Pattern Analysis & Machine Intelligence_, 2017.
  * [41] Hou, Qibin and Zhang, Li and Cheng, Ming Ming and Feng, Jiashi, “Strip Pooling: Rethinking Spatial Pooling for Scene Parsing,” in _CVPR_ , 2020.
  * [42] Sahu, Gaurav and Vechtomova, Olga, “Adaptive Fusion Techniques for Multimodal Data,” in _EACL_ , 2021.
  * [43] Chen, Danqi and Manning, Christopher, “A Fast and Accurate Dependency Parser using Neural Networks,” in _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2014.
  * [44] Ngiam, Jiquan and Khosla, Aditya and Kim, Mingyu and Nam, Juhan and Ng, Andrew Y, “Multimodal Deep Learning,” in _International Conference on Machine Learning_ , 2009.
  * [45] Amir Zadeh and Minghai Chen and Soujanya Poria and Erik Cambria and Louis-Philippe Morency, “Tensor Fusion Network for Multimodal Sentiment Analysis,” in _CoRR_ , 2017.
  * [46] Liu, Zhun and Shen, Ying and Lakshminarasimhan, Varun Bharadhwaj and Liang, Paul Pu and Zadeh, Amir and Morency, Louis Philippe, “Efficient Low-rank Multimodal Fusion with Modality-Specific Factors,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018.
  * [47] Liang, Paul Pu and Liu, Zhun and Tsai, Yao Hung Hubert and Zhao, Qibin and Salakhutdinov, Ruslan and Morency, Louis Philippe, “Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization,” in _CoRR_ , 2019.
  * [48] Tsai, Yao Hung Hubert and Bai, Shaojie and Liang, Paul Pu and Kolter, J. Zico and Salakhutdinov, Ruslan, “Multimodal Transformer for Unaligned Multimodal Language Sequences,” in _CoRR_ , 2019.
  * [49] Aming Wu , Yahong Han, “Multi-modal Circulant Fusion for Video-to-Language and Backward,” in _International Joint Conference on Artificial Intelligence_ , 2018.
  * [50] Yikai Wang and Fuchun Sun and Ming Lu and Anbang Yao, “Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion,” in _ACMMM_ , 2020.
  * [51] Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, Lukasz and Polosukhin, Illia, “Attention Is All You Need,” in _CoRR_ , 2017.
  * [52] Licheng Yu and Zhe Lin and Xiaohui Shen and Jimei Yang and Xin Lu and Mohit Bansal and Tamara L. Berg, “MAttNet: Modular Attention Network for Referring Expression Comprehension,” in _CoRR_ , 2018.
  * [53] Chen, Yi Wen and Tsai, Yi Hsuan and Wang, Tiantian and Lin, Yen Yu and Yang, Ming Hsuan, “Referring Expression Object Segmentation with Caption-Aware Consistency,” in _BMVC_ , 2019.
  * [54] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in _CVPR_ , 2009.
  * [55] Ronneberger, Olaf and Fischer, Philipp and Brox, Thomas, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in _IEEE Access_ ,vol. PP, no. 99, pp. 1-1, 2015.
  * [56] Escalante, Hugo Jair and Hernández, Carlos A and Gonzalez, Jesus A and López-López, Aurelio and Montes, Manuel and Morales, Eduardo F and Sucar, L Enrique and Villasenor, Luis and Grubinger, Michael, “The segmented and annotated IAPR TC-12 benchmark,” in _Computer vision and image understanding_ ,vol. 114, no. 4, pp. 419–428, 2010.
  * [57] Chen, Yen-Chun and Li, Linjie and Yu, Licheng and El Kholy, Ahmed and Ahmed, Faisal and Gan, Zhe and Cheng, Yu and Liu, Jingjing, “Uniter: Universal image-text representation learning,” in _ECCV_ , 2020.
  * [58] Shaofei Huang and Tianrui Hui and Si Liu and Guanbin Li and Yunchao Wei and Jizhong Han and Luoqi Liu and Bo Li, “Referring Image Segmentation via Cross-Modal Progressive Comprehension,” in _CVPR_ , 2020.
  * [59] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, “Learning transferable visual models from natural language supervision,” in _ICML_ , 2021.
  * [60] Ding, Henghui and Liu, Chang and Wang, Suchen and Jiang, Xudong, “Vision-language transformer and query generation for referring segmentation,” in _ICCV_ , 2021.
  * [61] Yang, Sibei and Xia, Meng and Li, Guanbin and Zhou, Hong-Yu and Yu, Yizhou, “Bottom-up shift and reasoning for referring image segmentation,” in _ICCV_ , 2021.
  * [62] Jing, Ya and Kong, Tao and Wang, Wei and Wang, Liang and Li, Lei and Tan, Tieniu, “Locate then segment: A strong pipeline for referring image segmentation,” in _CVPR_ , 2021.
  * [63] Kim, Namyup and Kim, Dongwon and Lan, Cuiling and Zeng, Wenjun and Kwak, Suha, “ReSTR: Convolution-free Referring Image Segmentation Using Transformers,” in _CVPR_ , 2022.
  * [64] Wang, Zhaoqing and Lu, Yu and Li, Qiang and Tao, Xunqiang and Guo, Yandong and Gong, Mingming and Liu, Tongliang, Suha, “Cris: Clip-driven referring image segmentation,” in _CVPR_ , 2022.
  * [65] Li, Zizhang and Wang, Mengmeng and Mei, Jianbiao and Liu, Yong, “MaIL: A Unified Mask-Image-Language Trimodal Network for Referring Image Segmentation,” in _arXiv_ , 2021.
  * [66] Yang, Zhao and Wang, Jiaqi and Tang, Yansong and Chen, Kai and Zhao, Hengshuang and Torr, Philip HS, “LAVT: Language-Aware Vision Transformer for Referring Image Segmentation,” in _CVPR_ , 2022.
  * [67] Liu, Weide and Wu, Zhonghua and Ding, Henghui and Liu, Fayao and Lin, Jie and Lin, Guosheng, “Few-shot segmentation with global and local contrastive learning,” in _arXiv_ , 2021.


# Pragmatic constraints and pronoun reference disambiguation: the possible and
the impossible

Ernest Davis
Dept. of Computer Science
New York University

###### Abstract

Pronoun disambiguation in understanding text and discourse often requires the
application of both general pragmatic knowledge and context-specific
information. In AI and linguistics research, this has mostly been studied in
cases where the referent is explicitly stated in the preceding text nearby.
However, pronouns in natural text often refer to entities, collections, or
events that are only implicitly mentioned previously; in those cases the need
to use pragmatic knowledge to disambiguate becomes much more acute and the
characterization of the knowledge becomes much more difficult. Extended
literary texts at times employ both extremely complex patterns of reference
and extremely rich and subtle forms of knowledge. Indeed, it is occasionally
possible to have a pronoun that is far separated from its referent in a text.
In the opposite direction, pronoun use is affected by considerations of focus
of attention and by formal constraints such as a preference for parallel
syntactic structures; these can be so strong that no pragmatic knowledge
suffices to overrule them.

##  1 Semantics, context, and pronoun disambiguation

It is well known that pronoun disambiguation in understanding text and
discourse often requires the application of both general pragmatic knowledge
and context-specific information (Rohde, 2018).

The point is vividly illustrated by Winograd schemas (Levesque, Davis, and
Morgenstern 2012) such as the well-known example,

> The trophy doesn’t fit in the brown suitcase because it is too big.
> The trophy doesn’t fit in the brown suitcase because it is too small.

In the first sentence the pronoun it must refer to the trophy; in the second,
it must refer to the suitcase. The sentences differ only in the final words
big and small, so that distinction, and the relation of the size of two
objects to the possibility of fitting one inside the other, must be
determining the referent of the pronoun.

Similarly, consider the pair of sentences

> Joan made sure to thank Susan for all the help she had given.
> Joan made sure to thank Susan for all the help she had received.

In the first sentence the referent of she is Susan, whereas in the second the
referent is Joan, reflecting the fact that someone who receives help thanks
someone who gives it.

Current natural language processing technology is able to achieve high levels
of success on collections of these kinds of examples (Kocijan et al. 2022) by
using the patterns of word associations extracted by deep learning networks as
proxies for this kind of underlying knowledge. But there is no question that
people, in understanding these texts, are drawing on a deep knowledge of the
world (Lake and Murphy, 2020) and, I would argue, little doubt that, in the
long run, truly intelligent AI systems will need to do likewise.

The above sentences can be expanded to make the issues of pronoun
disambiguation more complex while keeping them well-formed and intelligible.

> I tried packing the trophy in the brown suitcase, but it was too large, and
> I couldn’t close it. So instead I’ll get the red suitcase, because it should
> be large enough that it will fit in it.
>
> Joan made sure to thank Susan for all the help she had given her when she
> had been in trouble. Her generosity and sympathy had made an enormous
> difference to her.

More complex examples are possible:

> Mary White called Susan Brown to let her know that her daughter Jennifer had
> come over after school to the Whites’ house to play with their own daughter
> Edith. She thanked her for calling, because she had really started to wonder
> why she hadn’t come home.

Both AI and linguistic analysis have focused on short texts like the above,
for ease of analysis; but in extended texts, the complexities of reference and
the subtlety of the knowledge needed for disambiguation can be much greater.
For example, the following passage occurs in part 2, chapter 2 of Jane
Austen’s Emma. Four female characters are involved: Emma Woodhouse, Jane
Fairfax, Jane’s aunt Miss Bates, and Miss Bates’ mother. The dinner party also
includes at least Emma’s father and Mr. Knightley. I have put personal
pronouns in bold face, for emphasis.111I am told that readers who encounter
this out of the context of the novel find some of these pronouns hard to
resolve.

> Upon the whole, Emma left her [Jane Fairfax] with such softened, charitable
> feelings, as made her look around in walking home, and lament that Highbury
> afforded no young man worthy of giving her independence; nobody that she
> could wish to scheme about for her.
>
> These were charming feelings — but not lasting. Before she had committed
> herself by any public profession of eternal friendship for Jane Fairfax, or
> done more towards a recantation of past prejudices and errors, than saying
> to Mr. Knightley, “She certainly is handsome; she is better than handsome!”
> Jane had spent an evening at Hartfield with her grandmother and aunt, and
> every thing was relapsing much into its usual state. Former provocations
> reappeared. The aunt was as tiresome as ever; more tiresome, because anxiety
> for her health was now added to admiration of her powers; and they had to
> listen to the description of exactly how little bread and butter she ate for
> breakfast, and how small a slice of mutton for dinner, as well as to see
> exhibitions of new caps and new workbags for her mother and herself and
> Jane’s offences rose again. They had music; Emma was obliged to play; and
> the thanks and praise which necessarily followed appeared to her an
> affectation of candour, an air of greatness, meaning only to shew off in
> higher style her own very superior performance. She was, besides, which was
> the worst of all, so cold, so cautious!

Some of these disambiguations are straightforward, some quite subtle.222Also
subtle is the identification of Jane as the speaker of those thanks and praise
that seemed to Emma an affectation of candor. Presumably everyone at the
dinner party gave Emma thanks and praise for her playing, but it was only
Jane’s that she found affected. Teasing out all the forms of knowledge that
are used in all these disambiguations would require a lengthy analysis.
However, I note that the fact that the narrative is given from Emma’s point of
view — a fact about narrative structure — is important in disambiguating
“appeared to her” in the second to last sentence.

##  2 Antecedents that are entirely missing

There are also a number of cases where a third-person pronoun can be used with
no antecedent noun phrase. Some of these are natural, some are literary. Cases
II and III are common and established enough that they are listed in the
Oxford English Dictionary as distinct meanings of they.

I. A pronoun may refer to an action or event that appears as a verb phrase.
E.g. in “John hiked for three hours but it tired him out” the word it refers
to the hike.

II. They can mean people in general or some implicit collection of people.
Sometimes they is just equivalent to “Some people” as in “They say that a shot
of whiskey is good for a cold.” But often it refers to a specific collection
of people which much be inferred from context. For instance, in “I went to the
hospital but they told me to go home”, they refers to the hospital staff. In
the earlier quotation from Emma, in the phrase “they had to listen” the word
they refers to everyone at the dinner other than Miss Bates herself; in the
phrase “They had music”, it refers to everyone at the dinner. Neither group is
otherwise named here by a noun phrase.

This is possible even when there are other possible antecedents and other
occurrences of they: “Jack and Otto went to the hospital, but they told them
to go home.”

However, there are limits to this; it would be a stretch to say, with no
antecedent
(*) I went to the park and they were all enjoying the pleasant spring weather.
It would be impossible to say
(*) Between January 2020 and March 2021, more than 2,500,000 of them died of
Covid-19.
(I use the linguists’ convention of marking anomalous sentences with an
asterisk.)

In the Ira Gershwin lyric, “The way you wear your hat / The way you sip your
tea / The memory of all that / No, no, they can’t take that away from me,” the
phrase they can’t means “No one can”.

The first-person plural pronoun we and the second-person pronoun you, which
generally have no antecedent, raise similar issues. We refers to some
collection of people that includes the speaker; you refers to some collection
of people being addressed; but which particular collection has to be
determined from context, and indeed is often deliberately vague.

III. The Oxford English Dictionary gives as one definition of the word they:
“People in authority collectively, regarded as impersonal and oppressive”
(OED). A couple of illustrative quotes:
“English political thinking is much governed by ‘they’. ‘They’ are the higher-
ups, the mysterious powers.” — George Orwell.
“They really have to do something about all the potholes in this city.”

IV. The protagonists of a poem or short story (e.g. “Stories” by I.L. Peretz)
may be denoted throughout by he and she for literary effect. An extreme
example is the novel, Wolf Hall, by Hilary Mantel, in which the protagonist,
Thomas Cromwell, is always denoted as he and never named; but this at times
becomes hard for the reader to follow.

V. There is a literary trope in which characters or the narrator use pronouns
without need for antecedent to refer to powerful, malevolent figures whose
name is either unknown or dangerous to speak: for instance, human beings in
Bambi, or Queen Ayesha in Rider Haggard’s novel She. In J.R.R. Tolkien’s The
Two Towers, Pippin says “Then he came” (italics in the original) without
antecedent to mean Sauron in describing their encounter through the Palantir.
In Conan Doyle’s The Valley of Fear, Porlock writes, “Dear Mr. Holmes, I will
go no further in this matter. It is too dangerous — he suspects me,” he being
Professor Moriarty. In the case of they this often combines with (III): “I
suppose you mean we ought to spend all our time sucking up to Them, and
currying favour, and dancing attendance on Them like you do.” (C.S. Lewis, The
Silver Chair). (Here Them is a clique of bullies at a school.) In this usage,
the pronoun tends to be typographically marked: put in quotation marks, or
italicized, or capitalized, presumably to be read aloud to a particular
emphasis — the quote from The Valley of Fear is an exception. It is more
common in direct quotation than in narration — Haggard’s She is an exception.

##  3 Demonstrative pronouns referring to a category

Somewhat similar to the ambiguity of “we” and “they” discussed above, it is
possible to use the pronouns “this” and “that” in such a way that, though the
grammatical antecedent is a specific individual or event, the actual semantics
is some category that includes the individual.333Thanks to Doug Hofstadter for
bringing this to my attention. Such cases are often ambiguous, and if they can
be disambiguated, they require pragmatic constraints.

I have not found any examples in literature, but here is an artificial
example.

> Joe was way behind on the programming project for his operating systems
> course, and it was due the next day, so he turned off his phone and stayed
> up all night finishing it. He had never done that before.

What exactly had Joe never done before?

  * 1.

Stayed up all night?

  * 2.

Stayed up all night doing school work?

  * 3.

Stayed up all night doing a programming project (either for a course or for
fun)?

  * 4.

Stayed up all night doing a course for this particular class? or doing this
specific project?

  * $1^{\prime}$–$4^{\prime}$.

Turned off his phone and [1–4]?

##  4 Excluded forms

There are limits on the ways in which a pronoun is allowed to connect to a
referent, even when pragmatic constraints decisively rule out any other
possible referent, as the examples in table 1 demonstrate:

> 1\. ? The trophy doesn’t fit in the suitcase, because the trophy is an
> awkward shape and it is too small.
> 2\. * When Joan broke her leg, Susan had given her a lot of help, but Mary
> had not bothered to ask after her. She had been very generous.
> 3\. * Ann has two sons. Barbara has no children. Her children are in
> elementary school.
> 4\. * Robert woke up at 9:00 AM, while Samuel woke up at 6:00 AM, so he had
> more time to get ready for school.
> 5\. * Robert woke up at 9:00 AM, but Samuel woke up at 6:00 AM, so he had
> less time to get ready for school.
> 6\. * The child was screaming after the baby bottle and toy fell. Since he
> was full, it stopped his crying.

Table 1: Pronoun references in which pragmatic constraints do not disambiguate

(Examples 4 and 6 are from Sakaguchi et al., 2020.) Semantically, it in (1)
must refer to the suitcase; she in (2) must refer to Susan; her in (3) must
refer to Ann; he in (4) must refer to Samuel; he in (5) must refer to Robert;
and it in (6) must refer to bottle — at least, that is the consensus of the
crowd workers who created (6) and the other crowd workers who endorsed it. To
my ear, 2, 3, 4, and 6 are completely impossible; and 1 and 5 is at best
extremely unnatural.444I am just using my own “linguistic intuition”; I have
not checked this with empirical studies.

Note that, (4) and (5) are fine with the conjunctions reversed:

> $4^{\dagger}$ Robert woke up at 9:00 AM, but Samuel woke up at 6:00 AM, so
> he had more time to get ready for school.
> $5^{\dagger}$ Robert woke up at 9:00 AM, while Samuel woke up at 6:00 AM, so
> he had less time to get ready for school.
>

Adding a number-agreement constraint fixes (1), and adding gender-agreement
constraints fixes (4) and at least helps in (5), but neither gender nor number
constraints help in (2) or (3) (table 2).

$1^{\prime}$. The trophies don’t fit in the suitcase, because they are an
awkward shape and it is too small.
$2^{\prime}$. * When Joan broke her leg, Susan had given her a lot of help,
but Bob had not bothered to ask after her. She had been very generous.
$2^{\prime\prime}$. * When Joan broke her leg, the Millers had given her a lot
of help, but Bob had not bothered to ask after her. They had been very
generous.
$3^{\prime}$. * Ann has two sons. Fred has no children. Her children are in
elementary school.
$3^{\prime\prime}$. * Ann and David have two sons. Barbara has no children.
Their children are in elementary school.
$4^{\prime}$. Robert woke up at 9:00 AM, while Sarah woke up at 6:00 AM, so
she had more time to get ready for school.
$5^{\prime}$. ? Robert woke up at 9:00 AM, but Sarah woke up at 6:00 AM, so he
had less time to get ready for school.

Table 2: Pronoun references with gender and number constraints. These do not
always disambiguate.

The question is, why are sentences (1-6) excluded whereas the examples
discussed earlier — much more complex in structure, much less decisive in
their pragmatic constraints — acceptable?

It seems plausible to conjecture that to some extent the issue here is one of
narrative focus. The conjecture would be that there a rule that the referent
of a pronoun must be sufficiently in focus in the narrative. Because of the
syntactic form of the texts, in sentences 1-6 the referents that are
pragmatically possible (the suitcase, Susan, Ann, Samuel, Robert, and the
bottle, respectively) are out of focus, and therefore the reference is
impossible.

However, this is obviously largely explanation of ignotum per ignotis; without
a mechanism for determining what the objects in focus at any point in a text,
this raises more questions than it answers. Why does the semantically vacuous
difference between (4) and (5) in table 1 so entirely change the focus? How is
it that, in the passage from Emma, Emma, Jane, and Miss Bates all remain
sufficiently in focus that a pronoun can refer to any of them? If there is
such a rule of focus involved in the variants of sentence (1), why do the
constraints of number agreement easily suffice to overrule it, while the
pragmatic constraints do not?

Kehler (2015) similarly gives a number of examples where the form of the
sentence either forces a pragmatically implausible reading or conflicts with
pragmatics in a way that makes the sentence unintelligible. (Kehler carried
out tests with human subjects to confirm his intuitions.)

> (a) The demonstrators were denied a permit by the city council because they
> feared violence.
> (b) The city council denied the demonstrators a permit because they felt
> strongly that the best way to draw attention to current political issues is
> to advocate violence.
> (c) Norm lent his car to his brother’s girlfriend. He doesn’t own one.
> (d) Margaret Thatcher admires Hillary Clinton, and George W. Bush absolutely
> worships her.

In (a) readers generally interpret they as referring to the demonstrators. In
(b) readers “garden path”: they initially interpret they as meaning “the city
council” and then consciously backtrack and correct when they reach the end of
the sentence. In (c) readers end up confused rather than deciding that he
refers to the brother. In (d) readers interpret her as meaning Clinton rather
than Thatcher, despite the fact that, when the example was devised, Thatcher
was much more plausible. In cases (a), (b), and (d), readers are letting
syntactic parallelism trump pragmatic plausibility. In (b), in contrast to
Winograd’s original example, presumably the distance between they and the
disambiguating “advocate violence” is so large that readers cannot suspend
judgement, so they go with the syntactically parallel reading and then
withdraw it. (Why (b) leads to garden-pathing whereas (c) and (d) just lead to
puzzlement is not at all clear.)

Rohde (2018) proposes a Bayesian framework for pronoun reference
disambiguation: probabilities based on the form of the sentence are combined
with probabilities derived from pragmatic considerations as dictated by Bayes’
law. The cases that we have considered are not inconsistent with a Bayesian
framework — very little is, if you adopt the right framework and set the
priors properly (Davis, 2013) — but they are certainly an awkward fit. One
would have supposed that the assumption that a sentence is meaningful has a
strong prior; that the pragmatic constraints in these sentence have very
strong priors (e.g. the probability that Barbara both has no children and that
her children are in elementary school is zero); and that the probabilities
associated with these rather nebulous considerations of focus would be
comparatively indeterminate. Apparently, that is not the case.

##  5 Remote pronoun antecedents in literature

In literature555It would probably be frowned on in other forms of writing. it
is occasionally possible to use pronouns whose nearest antecedent occurs much
earlier, outside the categories of section 2.

There is a remarkable instance in Dickens’ David Copperfield. In chapter 31,
Steerforth seduces Emily and flees with her. Steerforth is mentioned again, in
passing at the beginning of chapter 34 (p. 474 in my copy). Steerforth is
unmentioned in chapters 35-39. In chapter 40, Mr. Peggoty meets with David
Copperfield and describes his search in Europe for Emily; in this entire
chapter there is only one mention of Steerforth, by a pronoun: “He was nowt to
me now.” (p. 566; italics in the original). The next mention after that is in
chapter 46 (p. 645):

> “Pray, has this girl been found?
>
> “No.”
>
> “And yet she has run away!” …
>
> “Run away?” I repeated
>
> “Yes! From him,” she said with a laugh.

Another example is from Dickens’ Bleak House:

> “Jo,” repeats Allan, looking at him with attention, “Jo! Stay. To be sure! I
> recollect this lad some time ago being brought before the coroner.”
>
> “Yes, I see you once afore at the inkwhich,” whimpers Jo. “What of that?
> Can’t you never let such an unfortnet as me alone? An’t I unfortnet enough
> for you yet? How unfortnet do you want me fur to be? I’ve been a-chivied and
> a-chivied, fust by one on you and nixt by another on you, till I’m worritted
> to skins and bones. The inkwhich warn’t MY fault. I done nothink. He wos
> wery good to me, he wos; he wos the only one I knowed to speak to, as ever
> come across my crossing.”

He in the last sentence refers to the subject of the coroner’s inquest,
Captain Hawdon, who has not been mentioned since chapter 40. Hawdon is an
important figure in the back-story, being the unknown father of the first-
person protagonist Esther Summerson, but a rather minor figure in the novel as
a whole; and many male characters are mentioned between chapters 40 and 46.

However, the identification presents no difficulty, either for Allan Woodcourt
or for the reader. The context of the inquest would probably suffice, but
additionally, in chapter 11, which describes the inquest, Jo uses the
identical phrase “He wos wery good to me, he wos,” there unambiguously of
Hawdon, and Dickens marks it by repeating it at the end of the chapter.

A third example is from At Swim-Two-Birds by Flann O’Brien:

> On the morning of the third day thereafter, said Finn, he was flogged until
> he bled water.

The referent of he is a hypothetical “man who mixed his utterance with the
honeywords of Finn”. Finn is continuing his own train of thought, which in the
novel has been interrupted for two pages by an unrelated conversation between
two other people. The distance is much shorter, but the referent is even more
minor, and, unlike the quotes from Dickens, is completely unrelated to the
immediate context.

It can be objected that all three is in direct quotation, and anything is
permitted in direct quotation. However, at least with the quotation from Bleak
House, if we modify the passage to take it out of direct quotation, the
reference remains clear, though certainly the literary quality suffers:

> “Yes, I see you once afore at the inkwhich,” whimpers Jo. “What of that?
> Can’t you never let such an unfortnet as me alone? An’t I unfortnet enough
> for you yet? How unfortnet do you want me fur to be? I’ve been a-chivied and
> a-chivied, fust by one on you and nixt by another on you, till I’m worritted
> to skins and bones. The inkwhich warn’t MY fault. I done nothink.” Jo
> continues to whimper that he had been very good to Jo; he was the only
> person who came across Jo’s crossing that Jo had known to speak to.

(It would be hard to recast the quotes from David Copperfield or At Swim-Two-
Birds in this way.)

More importantly for our purposes, in all three cases, the reader identifies
the reference, and the expression does not seem unnatural. They are not
problematic in the manner of the starred sentences in section 4.

Narrative focus is clearly relevant in the quotes from David Copperfield,
where Emily is the subject, and in Bleak House, where the inquest is the
subject. The immediate context is entirely irrelevant in At Swim-Two-Birds;
the effect is startling and humorous. That is a much smaller gap, so the
reader still has in mind Finn’s previous conversation.

The most striking kind of example of this kind would be the use of he or she
by the narrator a significant distance from the last mention of the person, in
some fairly straightforward666At Swim-Two-Birds is anything but
straightforward narrative, and Bleak House has a somewhat unusual narrative
form, but their complexities are not relevant to this issue. narrative text. I
expect that such examples exist, but there is no easy way to search for them.

##  6 Conclusion

These kinds of phenomena have been studied in linguistics, but, as far as I
have found, only in disambiguating third person pronouns, only with short
texts, and only when the possible antecedents are given explicitly. It has
been unstudied in AI studies of text interpretation, because those are
concerned with texts that do arise, not with explaining why impossible texts
cannot arise; and in AI studies of text generation, because a generator that
is conservative in its use of pronouns can easily avoid the issue. It does
become significant when one is involved in creating a data set of hard
pronoun-resolution problems, because then one has to deliberately skate close
to the rather fine line that separates pronoun usages that are difficult for
an AI to resolve from pronoun usages in these ways; and it is important that
all the texts in a dataset are clearly proper usage (Kocijan et al. 2022). The
phenomenon is in any case a curious one, and an adequate explanation might
well shed some light on larger questions of how pronouns are used and
understood.

### Acknowledgements

Thanks to Yuling Gu, Doug Hofstadter, Andrew Kehler, and Leora Morgenstern for
helpful discussions.

### References

Davis, Ernest (2013). The relevance of proofs of the rationality of
probability theory to automated reasoning and cognitive models. Unpublished.
https://cs.nyu.edu/faculty/davise/papers/ProbProofs.pdf

Kehler, Andrew. (2005). Testing for Common Sense: Thoughts on Pronoun
Interpretation and the Winograd Schema Challenge. Workshop on Language &
Common Sense: Integrating Across Psychology, Linguistics, and Computer Science
CogSci-2015.

Kocijan, Vid, Thomas Lukasiewicz, Ernest Davis, Gary Marcus, and Leora
Morgenstern (2020). The defeat of the Winograd Schema Challenge. arXiv
preprint arXiv:2004.13831; submitted to Artificial Intelligence.
https://arxiv.org/abs/2004.13831

Lake, Brenden M., and Gregory L. Murphy (2020). Word meaning in minds and
machines. arXiv preprint arXiv:2008.01766. https://arxiv.org/abs/2008.01766

Levesque, Hector, Ernest Davis, and Leora Morgenstern (2012). The Winograd
schema challenge. KR-2012.

Rohde, Hannah. 2018. Pronoun interpretation and production. In Chris Cummins
and Napoleon Katsos (eds.) The Oxford Handbook of Experimental Semantics and
Pragmatics. Oxford University Press.

Sakaguchi, Keisuke, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi (2020).
Winogrande: An adversarial Winograd schema challenge at scale. In Proceedings
of the AAAI Conference on Artificial Intelligence, pp. 8732-8740.