# ViewCo: Discovering Text-Supervised Segmentation Masks via Multi-View
Semantic Consistency

Pengzhen Ren1, Changlin Li2, Hang Xu3, Yi Zhu3, Guangrun Wang4, Jianzhuang
Liu3,
Xiaojun Chang2, Xiaodan Liang1,5
1Sun Yat-sen University 2ReLER, AAII, University of Technology Sydney
3Huawei Noah’s Ark Lab 4University of Oxford 5MBZUAI
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author.

###### Abstract

Recently, great success has been made in learning visual representations from
text supervision, facilitating the emergence of text-supervised semantic
segmentation. However, existing works focus on pixel grouping and cross-modal
semantic alignment, while ignoring the correspondence among multiple augmented
views of the same image. To overcome such limitation, we propose multi-View
Consistent learning (ViewCo) for text-supervised semantic segmentation.
Specifically, we first propose text-to-views consistency modeling to learn
correspondence for multiple views of the same input image. Additionally, we
propose cross-view segmentation consistency modeling to address the ambiguity
issue of text supervision by contrasting the segment features of Siamese
visual encoders. The text-to-views consistency benefits the dense assignment
of the visual features by encouraging different crops to align with the same
text, while the cross-view segmentation consistency modeling provides
additional self-supervision, overcoming the limitation of ambiguous text
supervision for segmentation masks. Trained with large-scale image-text data,
our model can directly segment objects of arbitrary categories in a zero-shot
manner. Extensive experiments show that ViewCo outperforms state-of-the-art
methods on average by up to 2.9%, 1.6%, and 2.4% mIoU on PASCAL VOC2012,
PASCAL Context, and COCO, respectively. 111Code release:
https://github.com/pzhren/ViewCo

##  1 Introduction

Recently, vision-language contrastive learning (Radford et al. (2021); Li et
al. (2021a)) has attracted a lot of attention because it can obtain more
generalized feature representation. And at the same time, it can also make use
of abundant image-text pairs to avoid labor-intensive annotation costs.
Vision-language pre-training (VLP) models have exhibited

Figure 1: Illustration of text description ambiguity. Text descriptions are
highly abstract and difficult to be semantically aligned with images. Cross-
view semantic consistency modeling can effectively alleviate the effect of the
text description ambiguity issue.

impressive potential in various visual (Xu et al. (2022); Mu et al. (2021);
Radford et al. (2021)) and multimodal (Wang et al. (2021b); Kim et al. (2021))
tasks, including text-supervised semantic segmentation (Xu et al. (2022);
Ghiasi et al. (2021); Xu et al. (2021); Zabari & Hoshen (2021); Zhou et al.
(2021a), which uses text instead of traditional dense labels for supervision
to achieve zero-shot semantic segmentation. It provides a feasible solution
for learning segmentation masks without mask annotation.

However, existing works with CLIP-based (Radford et al. (2021)) segmentation
(Xu et al. (2022; 2021); Zhou et al. (2021a)) mainly focus on pixel grouping
or cross-modal semantic alignment. They have the following two obvious
limitations: (i) the excessive strictness of image-text correspondence; and
(ii) the ambiguity of text description. First, in vanilla vision-language
contrastive learning, each image-text pair is regarded as a unique positive
pair, while all the other combinations are regarded as negative ones. This
image-text correspondence is actually too rigorous. In fact, one textual
description may correspond to different images. The excessive strictness is
not conducive to the model learning high-level cross-modal semantic
correspondences. Therefore, more relaxed vision-language contrastive learning
needs to be considered. Second, the ambiguity of textual descriptions is also
a key challenge. Compared with the traditional semantic segmentation pipeline
that uses dense annotations as supervision information (Touvron et al. (2021);
Ren et al. (2022)), the CLIP-based segmentation methods (Xu et al. (2022;
2021); Zhou et al. (2021a)) use text as supervision, which is easier to access
but more noisy and ambiguous.

This is mainly because compared with traditional segmentation annotations,
text descriptions are often more abstract and do not contain location
information. Moreover, the background in the image is usually ignored in the
description. In some cases, the objects in the image do not even exist in the
text description (see Figure 1). Such ambiguity is common in the textual
supervision in vision-language pre-training. In the semantic segmentation
task, the ambiguity of textual supervision makes the segmented object-label
correspondence very fragile. Therefore, Fully mining the information carried
by the dataset itself may need to be considered.

Figure 2:  The consistent comparison of semantic segmentation results in
multiple views of a “horse”. (a) GroupViT: the semantic segmentations of
different views are inconsistent. (b) ViewCo: the semantic segmentations of
different views are much more consistent. Here, x, u, and v represent the
segmentation results on the original image and views u, v, respectively.

On the other hand, visual self-supervision (Caron et al. (2021); He et al.
(2022); Chen et al. (2020a); Zhou et al. (2021b)) has been widely used for
visual pre-training. It includes two categories: reconstructing masked images
(He et al. (2022); Zhou et al. (2021b)) and multicrop image contrast (Caron et
al. (2021); Chen et al. (2020a)). For example, SLIP (Mu et al. (2021))
introduces contrastive learning of multicrop visual consistency for VLP.
MaskCLIP (Dong et al. (2022)) introduces a visual self-supervised task of
reconstructing masked images. They utilize visual self-supervision to provide
more useful information for VLP models. However, the semantic consistency of
multiple views of an image in segmentation and cross-modal contrast have not
received enough attention and research.

Based on the above observations, in this paper, we explore the impact of
multi-view semantic consistency on the task of text-supervised semantic
segmentation through visual self-supervision. To this end, we propose multi-
View Consistency learning (ViewCo), which aims at discovering text-supervised
segmentation masks via multi-view semantic consistency. Specifically, we
propose text-to-views consistency modeling to alleviate the excessive
strictness of image-text correspondence in vanilla vision-language contrastive
learning. It enables the model to benefit from the dense assignment of visual
features by encouraging different crops to align with the same text. This
relaxed one-to-many contrast mechanism also facilitates the learning of multi-
view consistent semantics, enabling the model to acquire high-level cross-
modal alignment capabilities. Moreover, as shown in Figure 1, to alleviate the
ambiguity issue of textual supervision, we propose cross-view segmentation
consistency modeling. It overcomes the limitation imposed by textual ambiguity
by providing additional self-supervision to vision-language contrastive
learning via cross-view segmentation consistency. ViewCo uses the proposed
text-to-views consistency modeling for vision-language cross-modal contrastive
learning and additionally enables cross-view segmentation consistency modeling
by contrasting the segment features of Siamese visual encoders. As shown in
Figure 2, with the help of the two consistency modeling schemes, ViewCo
establishes a solid semantic correspondence in different views, and the
semantics in different views maintain a good consistency. The semantic
consistency of GroupViT in different views is difficult to guarantee.

Overall, ViewCo’s design is simple and effective. We train it on large-scale
image-text pair datasets CC12M (Changpinyo et al. (2021)) and YFCC (Thomee et
al. (2016)). In the inference stage, we use the similarity scores between the
segmentation embeddings generated by the teacher network and the label prompts
to assign labels to the image masks for zero-shot semantic segmentation.
Compared with the state-of-the-art methods, ViewCo achieves an average
improvement of 2.9%, 1.6%, and 2.4% mIoU on PASCAL VOC2012, PASCAL Context,
and COCO, respectively. Our contributions can be summarized as follows:

  * •

We propose a novel one-to-many text-to-views consistency modeling that
improves the model’s ability of high-level cross-modal semantic alignment by
encouraging different crops of an image to align with the same text.

  * •

To alleviate the problem of supervision failure that may arise from text
ambiguity, we propose cross-view segmentation consistency modeling to provide
additional self-supervision for the vision branch and encourage the model to
generate consistent segmentation masks for different views.

  * •

ViewCo consistently outperforms the state-of-the-art methods on PASCAL
VOC2012, PASCAL Context, and MS-COCO when pre-trained on CC12M or CC12M+YFCC.

##  2 Related Work

Vision-Language Pretraining. In recent years, vision-language pre-training
models (Chen et al. (2020b); Desai & Johnson (2021); Li et al. (2020a; 2021a;
b)) have developed rapidly with the help of large-scale image-text pair data
available on the Internet. Recently, VLP models such as CLIP (Radford et al.
(2021)), ALIGN (Li et al. (2021a)), and SLIP (Mu et al. (2021)) have made
great progress in visual representation learning by using contrastive
learning. And they have been successfully transferred to various downstream
tasks, such as visual question answering (Antol et al. (2015); Zhou et al.
(2020)) and visual reasoning (Zellers et al. (2019)). In particular, CLIP
(Radford et al. (2021)) uses the image-text matching relationship for
contrastive learning, and the learned model can be directly transferred to
ImageNet classification (Deng et al. (2009)) in a zero-shot manner without any
fine-tuning. This success is also found on zero-shot semantic segmentation (Xu
et al. (2022)). However, the one-to-one contrastive learning mechanism between
image and text in the vanilla VLP pipeline is too strict, which is not
conducive to the model learning high-level cross-modal semantic alignment.
Based on the above observations, this paper proposes one-to-many text-to-views
consistency modeling. It relaxes the original one-to-one correspondence by
encouraging different crops of an image to match the same text, allowing the
model to benefit from the dense assignment of the visual features.

Visual Self-Supervision. This framework relies on the information carried by
the image itself for self-supervision without any additional annotation
information. Visual self-supervision is mainly divided into generative (He et
al. (2022); Bao et al. (2021)) and contrastive (He et al. (2020a); Caron et
al. (2021); Chen et al. (2020a)). A generative model allows the model to learn
the feature representation of the image by reconstructing the masked image.
Contrastive models focus more on learning-centric global representations.
Since semantic segmentation requires dense prediction of images, generative
models may not help much because

Figure 3:  Comparison of single-view text-to-image contrastive learning (left)
and multi-view text-to-views contrastive learning (right).

they destroy the original structure and information of images. On the other
hand, the contrastive visual self-supervised model can provide the required
multi-view features for ViewCo’s text-to-views consistency modeling. Moreover,
this visual contrastive learning can provide additional visual self-
supervision information for the VLP model to alleviate the risk of supervision
failure caused by text ambiguity. Therefore, this paper focuses on the help of
visual contrastive learning for semantic segmentation consistency.

Consistent Semantics. Capturing consistent semantics is one of the main
challenges shared by many tasks such as cross-modality and visual
understanding. Vision-language contrastive learning (Radford et al. (2021)) is
essential to encode different modal data into the same feature space and
enforces the features sharing the same semantics to get closer, and the
features with different semantics to be pushed away. Similarly, multicrop
image-level semantic consistency is also the core idea of visual self-
supervised contrastive learning (Caron et al. (2021); Chen et al. (2020a); He
et al. (2020a)). For example, DenseCL (Wang et al. (2021a)) performs pixel-
level dense contrastive learning on dense output vectors from multiple views,
which is not helpful to the learning of high-level global semantic
information. Further, GroupViT (Xu et al. (2022)) uses text as supervision and
achieves pixel grouping by capturing the contextual consistent semantics of
images. However, in the text-supervised semantic segmentation task, the
ambiguous properties of text relative to dense annotations result in that the
semantic consistency of images sharing the same semantics cannot be
sufficiently guaranteed in the embedding space. Furthermore, the strict one-
to-one correspondence between image and text in the vanilla VLP model is also
not conducive to the true alignment of high-level cross-modal semantics.
Figure 3 (left) illustrates the above observation: although one of the views
of an image (e.g., the solid circle) is already close to the corresponding
text embedding, other views (e.g., the dashed circles) may still be far away.
Previous VLP methods generally only focus on the alignment of a single view
with text. In contrast, as shown in Figure 3 (right), ViewCo focuses on text-
to-views consistency modeling, doing one-to-many matching in cross-modal
contrastive learning.

##  3 Multi-View Consistent Learning

Figure 4: Framework of ViewCo. It is mainly composed of a cross-view
segmentation consistency module and a text-to-views consistency module. The
visual branch adopts a visual self-supervised model, which consists of teacher
$f_{t}$ and student $f_{s}$ networks with the same structure. $f_{t}$ and
$f_{s}$ are the bottom-up segmentation backbone that outputs segment features
of the image.

As shown in Figure 4, our ViewCo is mainly composed of a cross-view
segmentation consistency module and a text-to-views consistency module. We
describe these two modules in Sections 3.1 and 3.2, respectively, and
summarize the final loss function in Section 3.3.

###  3.1 cross-view segmentation consistency module

As shown in Figure 4 (left), given a batch of image-text pairs
$\\{(x_{i}^{I},x_{i}^{T})\\}_{i=1}^{B}$, two random augmentations are
performed on the input image $x_{i}^{I}$, generating two warped views $u$ and
$v$. We use GroupViT (Xu et al. (2022)) as the bottom-up segmentation backbone
of ViewCo, where each view is segmented into $K$ segment tokens. For each of
the views (e.g., $u$), this process is expressed as:
$Z^{u_{s}}_{\text{Seg}}=\\{Z_{\text{seg}_{k}}^{u_{s}},k=1,...,K\\}=f_{s}(u)\in\mathbb{R}^{K\times
d},$ where $Z_{\text{seg}_{k}}^{u_{s}}\in\mathbb{R}^{d}$ is the $k$-th segment
feature from $f_{s}$, and $d$ is the dimensionality of the segment feature.
Similarly, we have $Z_{\text{seg}_{k}}^{v_{s}}$ and the segment features
$Z_{\text{seg}_{k}}^{u_{t}}$ and $Z_{\text{seg}_{k}}^{v_{t}}$ from the teacher
network $f_{t}$. We update the parameters of $f_{t}$ using the exponential
moving average (EMA) He et al. (2020b) of the parameters of $f_{s}$. For
example, let $\theta_{i}$ and $\overline{\theta}_{i}$ be the parameters of
$f_{s}$ and $f_{t}$ at training step $i$, respectively, and then
$\overline{\theta}_{i}$ is updated as:
$\overline{\theta}_{i}=\alpha\overline{\theta}_{i-1}+(1-\alpha)\theta_{i}$,
where $\alpha$ is a hyper-parameter for smoothing the update. In addition, the
standard contrastive loss function, called InfoNCE (Oord et al. (2018)), is
considered in this paper, for an encoded query $q$ and a set of encoded
samples $k=\\{k_{0},k_{1},k_{2},...\\}^{N}$ that are the keys of a dictionary,
we have:

$\mathcal{L}_{\textit{NCE}}(q,k)=-\log\frac{\text{exp}(q\cdot
k_{+}/\tau)}{\sum_{i=0}^{N}\text{exp}(q\cdot k_{i}/\tau)},$ (1)

where $\tau$ is a learnable temperature parameter. And $q$ and $k_{+}$ are
positive pairs, and the other $(N-1)$ pairs are negative.

Intuitively, the segment features obtained from different crops of the same
image should be roughly the same, i.e., cross-view segmentation consistency.
To this end, for the semantic segmentation task, we replace the image-level
contrastive learning in previous methods (Caron et al. (2021); Zhou et al.
(2021b)) with cross-view segmentation consistency learning within images.
Therefore, we define the minimization training objective of the cross-view
segmentation consistency module in ViewCo as:

$\mathcal{L}^{t\leftrightarrow s}_{[\text{Seg}]}=\mathcal{L}^{t\rightarrow
s}_{[\text{Seg}]}+\mathcal{L}^{s\rightarrow t}_{[\text{Seg}]}.$ (2)

It is a bi-directional contrast loss between the segment features from the
teacher $f_{t}$ and the student $f_{s}$. $\mathcal{L}^{t\rightarrow
s}_{[\text{Seg}]}$ considers two pairs of views (i.e., $(u_{t},v_{s})$ and
$(v_{t},u_{s})$) outputted by $f_{t}$ and $f_{s}$. The segment features of
$(u_{t},v_{s})$ from the same image are multiplied
($Z^{u_{t}}_{\text{seg}}\cdot{Z^{v_{s}}_{\text{seg}}}^{T}\in\mathbb{R}^{K\times
K}$) after $l_{2}$ normalization. In the image branch of ViewCo, we use the
EMA policy for parameter updates, so the learnable grouping tokens on the
corresponding position IDs of different views of the same image are highly
correlated, and they have the same semantics. Therefore, the semantic pairs
$\\{(Z^{u_{t}}_{\text{seg}_{i}},Z^{v_{s}}_{\text{seg}_{i}}),i=1,...,K\\}$ on
the diagonal are regarded as positive, and the other $K(K-1)$ pairs
$\\{(Z^{u_{t}}_{\text{seg}_{i}},Z^{v_{s}}_{\text{seg}_{j}}),i,j=1,...,K,i\neq
j\\}$ are regarded as negative. Therefore, the contrastive loss
$\mathcal{L}^{t\rightarrow s}_{[\text{Seg}]}$ of the teacher-to-student
segment features is defined as $\mathcal{L}^{t\rightarrow
s}_{[\text{Seg}]}=\mathcal{L}_{u_{t}\rightarrow
v_{s}}+\mathcal{L}_{v_{t}\rightarrow u_{s}}$, more specifically:

$\mathcal{L}^{t\rightarrow
s}_{[\text{Seg}]}=-\frac{1}{KB}\sum_{i=1}^{B}\sum_{k=1}^{K}(\mathcal{L}_{\textit{NCE}}(Z^{u_{t}}_{\text{seg}_{k}},\\{Z^{v_{s}}_{\text{seg}_{k}}\\}_{k=1}^{K})+\mathcal{L}_{\textit{NCE}}(Z^{v_{t}}_{\text{seg}_{k}},\\{Z^{u_{s}}_{\text{seg}_{k}}\\}_{k=1}^{K})).$
(3)

Similarly, the contrastive loss $\mathcal{L}^{s\rightarrow t}_{[\text{Seg}]}$
of the student-to-teacher segment features is defined as
$\mathcal{L}^{s\rightarrow t}_{[\text{Seg}]}=\mathcal{L}_{u_{s}\rightarrow
v_{t}}+\mathcal{L}_{v_{s}\rightarrow u_{t}}$, more specifically:

$\mathcal{L}^{s\rightarrow
t}_{[\text{Seg}]}=-\frac{1}{KB}\sum_{i=1}^{B}\sum_{k=1}^{K}(\mathcal{L}_{\textit{NCE}}(Z^{u_{s}}_{\text{seg}_{k}},\\{Z^{v_{t}}_{\text{seg}_{k}}\\}_{k=1}^{K})+\mathcal{L}_{\textit{NCE}}(Z^{v_{s}}_{\text{seg}_{k}},\\{Z^{u_{t}}_{\text{seg}_{k}}\\}_{k=1}^{K})).$
(4)

Figure 5a shows the positive and negative pairs for cross-view segmentation
consistency learning in the vision branch.

(a) $\mathcal{L}_{[\text{Seg}]}$

(b) $\mathcal{L}_{[\text{Text-Views}]}$

Figure 5: Illustration of the contrastive loss of (a) cross-view segmentation
consistency modeling and (b) text-to-views consistency modeling.
$Z^{I_{i}}_{\text{Seg}_{k}}$ is the $k$-th semantic feature of the $i$-th
image (i.e., view $u$ or $v$). $Z_{i}^{I_{v}}$ and $Z_{i}^{I_{u}}$ are the
embeddings of the views $v$ and $u$ of the $i$-th image, respectively.

###  3.2 text-to-views consistency module

Previous methods (Radford et al. (2021); Xu et al. (2022)) build visual-
linguistic semantic correspondences by performing a contrastive loss on image-
text pairs. In this paper, we consider the contrastive learning between
multiple views and text, using one-to-many text-to-views consistency modeling
instead of one-to-one text-to-image contrastive learning. The model learns to
capture intra-modal and inter-modal semantic consistency through the alignment
of multi-view images and text.

Specifically, for a given image-text pair $(x_{i}^{I},x_{i}^{T})$, by
performing two different augmentations to the input image, we have a triplet
$(u_{i},v_{i},x_{i}^{T})$. As shown in Figure 4 (right), in the training
phase, we take the output $(Z_{i}^{u},Z_{i}^{v})$ of the view pair
$(u_{i},v_{i})$ through the student network $f_{s}$ and the output $Z_{i}^{T}$
of the text encoder $E_{T}$ to calculate the contrastive loss respectively.
The visual embeddings $(Z_{i}^{u},Z_{i}^{v})$ and text embedding $Z_{i}^{T}$
are mapped to the same feature space through two MLPs, respectively, before
performing the final $l_{2}$ regularization. This procedure is represented as:
$Z_{i}^{I_{u}}=\text{MLP}({\text{AvgPool}({Z^{u_{i}}_{[\text{Seg}]}})}),Z^{u_{i}}_{[\text{Seg}]}=f_{s}(u_{i});Z_{i}^{I_{v}}=\text{MLP}({\text{AvgPool}({Z^{v_{i}}_{[\text{Seg}]}})}),Z^{v_{i}}_{[\text{Seg}]}=f_{s}(v_{i}).$
The multi-view feature $Z_{i}^{I}=\\{Z_{i}^{I_{u}},Z_{i}^{I_{v}}\\}$ and text
embedding $Z^{T}_{i}$ constitute positive pairs, and the other $2B(B-1)$ pairs
are negative pairs. The contrastive loss of text-to-views consistency modeling
is defined as follows:

$\mathcal{L}_{I_{\\{u,v\\}}\leftrightarrow
T}=\mathcal{L}_{I_{\\{u,v\\}}\rightarrow T}+\mathcal{L}_{T\rightarrow
I_{\\{u,v\\}}},$ (5)

where the contrastive loss of views $I_{\\{u,v\\}}$-to-text is defined as:

$\mathcal{L}_{I_{\\{u,v\\}}\rightarrow
T}=-\frac{1}{KB}\sum_{i=1}^{B}\sum_{k=1}^{K}(\mathcal{L}_{\textit{NCE}}(Z_{i}^{I_{u}},\\{Z_{i}^{T}\\}_{i=1}^{B})+\mathcal{L}_{\textit{NCE}}(Z_{i}^{I_{v}},\\{Z_{i}^{T}\\}_{i=1}^{B})).$
(6)

and the contrastive loss of text-to-views $I_{\\{u,v\\}}$ is defined as:

$\mathcal{L}_{T\rightarrow
I_{\\{u,v\\}}}=-\frac{1}{KB}\sum_{i=1}^{B}\sum_{k=1}^{K}(\mathcal{L}_{\textit{NCE}}(Z_{i}^{T},\\{Z_{i}^{I_{u}}\\}_{i=1}^{B})+\mathcal{L}_{\textit{NCE}}(Z_{i}^{T},\\{Z_{i}^{I_{v}}\\}_{i=1}^{B})).$
(7)

Additionally, in order to further enhance the association between multi-view
semantics and text semantics, we also compute the multi-label image-text
contrastive loss (Xu et al. (2022)) of multi-view and “prompted text” pairs
$\\{(Z_{i}^{I_{u}},\\{Z_{i}^{t_{m}}\\}_{m=1}^{M})_{i=1}^{B},(Z_{i}^{I_{v}},\\{Z_{i}^{t_{m}}\\}_{m=1}^{M})_{i=1}^{B}\\}$,
where $\\{Z_{i}^{t_{m}}\\}_{m=1}^{M}$ are the embeddings of the additional $M$
text prompts $\\{T_{i}^{m}\\}^{M}_{m=1}$ generated by the $i$-th text
$x_{i}^{T}$ according to the “prompt engineering” mechanism (Radford et al.
(2021)). $(Z_{i}^{I_{u}},\\{Z_{i}^{t_{m}}\\}_{m=1}^{M})$, i.e., the embedding
of the $i$-th image view $u$ and the generated $M$ text embeddings
$\\{Z_{i}^{t_{m}}\\}_{m=1}^{M}$ are positive pairs, and the other combinations
are negative pairs. Therefore, similar to Eq.(5), the multi-label contrastive
loss of multi-view $I_{\\{u,v\\}}$ and multi-prompt $\\{T^{m}\\}_{m=1}^{M}$ is
defined as:

$\mathcal{L}_{I_{\\{u,v\\}}\leftrightarrow\\{T^{m}\\}_{m=1}^{M}}=\mathcal{L}_{I_{\\{u,v\\}}\rightarrow\\{T^{m}\\}_{m=1}^{M}}+\mathcal{L}_{\\{T^{m}\\}_{m=1}^{M}\rightarrow
I_{\\{u,v\\}}}.$ (8)

First, the views-to-prompts loss is the average of the losses of the two
views. Considering a single view, e.g. $u$, the contrastive loss of $u$ to all
the prompts is defined as:

$\mathcal{L}_{I_{u}\rightarrow\\{T^{m}\\}_{m=1}^{M}}=-\frac{1}{B}\sum_{i=1}^{B}\left(\log\frac{\sum_{m=1}^{M}\text{exp}(Z_{i}^{I_{u}}\cdot
Z_{i}^{t_{m}}/\tau)}{\sum_{m=1}^{M}\sum_{j=1}^{B}\text{exp}(Z_{i}^{I_{u}}\\!\cdot\\!Z_{j}^{t_{m}}/\tau)}\right)\\!.$
(9)

Second, the contrastive loss of multi-prompt-to-views is defined as:

$\mathcal{L}_{\\{T^{m}\\}_{m=1}^{M}\rightarrow
I_{\\{u,v\\}}}\\!=\\!-\frac{1}{2MB}\sum_{m=1}^{M}\sum_{i=1}^{B}(\mathcal{L}_{\textit{NCE}}(Z_{i}^{t_{m}},\\{Z_{i}^{I_{u}}\\}_{i=1}^{B})+\mathcal{L}_{\textit{NCE}}(Z_{i}^{t_{m}},\\{Z_{i}^{I_{v}}\\}_{i=1}^{B})).$
(10)

In particular, a similar work to our text-to-views consistency module is
DeCLIP (Li et al. (2021b)). It believes that the text description may only be
a small part of the image, so in addition to the global view in CLIP (Radford
et al. (2021)), DeCLIP also adds a local view for image self-supervision,
which may cause information leakage. In addition, DeCLIP uses EDA (Wei & Zou
(2019)) as a text augmentation strategy. The augmented text still contains
multiple semantics, which is not helpful to the alignment of local semantics
in segmentation tasks. In contrast, ViewCo uses self-supervision of two local
views to ensure the difficulty of the task, while using a “prompt engineering”
mechanism to obtain an augmented text with a single semantic. Combining one-
to-many alignment can help ViewCo to better mine consistent segmentation
semantics in images.

###  3.3 Overall Loss Function

Finally, the total loss of ViewCo is the sum of the cross-view segmentation
consistency contrastive loss and the two cross-modal contrastive losses:

$\mathcal{L}=\mathcal{L}^{t\leftrightarrow
s}_{[\text{Seg}]}+\mathcal{L}_{I_{\\{u,v\\}}\leftrightarrow
T}+\mathcal{L}_{I_{\\{u,v\\}}\leftrightarrow\\{T^{m}\\}_{m=1}^{M}}.$ (11)

##  4 Experiments

###  4.1 Implementation Details

Architecture. In the cross-view segmentation consistency module, $f_{t}$ and
$f_{s}$ have the same network structure. The parameters of $f_{t}$ are updated
using the exponential moving average of the parameters of $f_{s}$. We use
GroupViT (Xu et al. (2022)) with two stages as the backbone for semantic
feature extraction of ViewCo’s visual branch. It is built on ViT-S
(Dosovitskiy et al. (2020); Touvron et al. (2021)) with 12 Transformer layers.
The input image size is $224\times 224$, the patch size is $16\times 16$, and
the hidden dimensionality is 384. The 2-stage GroupViT finally outputs 8
segment tokens, i.e., $K=8$). Following Radford et al. (2021), ViewCo’s text
encoder $E_{T}$ consists of 12 Transformer layers with a hidden feature
dimensionality of 256.

Training and Inference. In the training phase, we use CC12M (Changpinyo et al.
(2021)) and the filtered YFCC (Thomee et al. (2016)) as training datasets,
which contain 12M and 14M image-text pairs, respectively. See A.1 of the
supplementary material for more training details. In the inference phase,
following (Xu et al. (2022); Radford et al. (2021)), the image is segmented by
associating the image patches with the $K$ segment tokens outputted by the
teacher network $f_{t}$. The semantics in the images are further classified by
computing the similarity of the $K$ visual-semantic embeddings to the text
embeddings generated from the class labels of the test dataset.

Zero-Shot Transfer to Semantic Segmentation. We evaluate ViewCo on the task of
zero-shot transfer to semantic segmentation on the validation sets of PASCAL
VOC 2012 (Everingham et al. (2010)), PASCAL Context (Mottaghi et al. (2014))
and COCO Stuff (Lin et al. (2014)) datasets. The three datasets contain 20,
59, and 80 foreground classes and an additional background class,
respectively. During inference, following GroupViT (Xu et al. (2022)), ViewCo
predicts only the foreground classes by thresholding the softmax-normalized-
similarity between the embedding of the outputted image segments and the text
segmentation labels. The thresholds on PASCAL VOC 2012, PASCAL Context, and
COCO are set to 0.95, 0.35, and 0.95, respectively. We resize each input image
to have a shorter side of 448.

###  4.2 Comparisons with Recent Methods

We first compare the performance of ViewCo with some ViT-S-based zero-shot
baselines. Then, to further evaluate the performance of ViewCo on the zero-
shot semantic segmentation task, we compare ViewCo with some fully supervised
transfer and CLIP-based models.

Arch. | Method |  Mask mIoU (%)
---|---|---
ViT | pixel-wise | 20.1
ViT | K-means | 25.0
ViT | Mean-shift | 20.7
ViT | Spectral clustering | 19.7
GroupViT | - | 51.2
ViewCo (ours) | - | 52.4
Table 1: Comparison with zero-shot baselines on PASCAL VOC 2012.

Comparison with Zero-Shot Baselines. Table 1 shows the performance comparison
of ViewCo and zero-shot baselines on PASCAL VOC 2012. Among them, the four
ViT-based baselines train vision and text encoders through the image-text
contrastive loss defined in CLIP (Radford et al. (2021)). They adopt four
different pixel grouping methods: pixel-wise, K-means, Mean-shift (Comaniciu &
Meer (2002)), and Spectral clustering (Shi & Malik (1997)) respectively. And
GroupViT (Xu et al. (2022)) uses the bottom-up patch grouping mechanism. As
shown in Table 1, ViewCo significantly outperforms the CLIP-trained ViT and
GroupViT (52.4% vs. 51.2%) baselines. It is worth noting that ViewCo and
GroupViT adopt the same segmentation backbone, indicating that ViewCo can
effectively improve the model’s ability of segmentation and cross-modal
semantic alignment with the help of the two consistent semantic modelings.

| Pre-training |  | Transfer (mIoU (%))
---|---|---|---
Arch | Model | Dataset | Supervision | Zero-Shot |  PASCAL VOC |  PASCAL Context | COCO
ViT | DeiT | ImageNet | class | ✗ | $53.0$ | $35.9$ | -
DINO | ImageNet | self | ✗ | $39.1$ | $20.4$ | -
DINO | CC12M+YFCC | self | ✗ | $37.6$ | $22.8$ | -
MoCo | ImageNet | self | ✗ | $34.3$ | $21.3$ | -
MoCo | CC12M+YFCC | self | ✗ | $36.1$ | $23.0$ | -
CLIP | CLIP | LAION-20M | text | ✓ | - | $13.5$ | 8.2
GroupViT | CC12M | text | ✓ | 41.1 | $18.2$ | 18.4
GroupViT111The latest version of GroupViT reports the results of training on the CC3M+CC12M+YFCC dataset. | CC12M+YFCC | text | ✓ | 51.2 | 22.3 | 20.9
SLIP | LAION-20M | text & self | ✓ | - | $12.3$ | 8.8
CLIP-MAE | LAION-20M | text & self | ✓ | - | $16.8$ | 11.8
MaskCLIP | LAION-20M | text & self | ✓ | - | $17.7$ | 11.8
ViewCo (ours) | CC12M | text & self | ✓ | $\mathbf{45.7}$ | 20.8 | 20.6
ViewCo (ours) | CC12M+YFCC | text & self | ✓ | $\mathbf{52.4}$ | 23.0 | 23.5
Table 2:  Comparisons with recent methods. Zero-shot means that the model is
directly transferred to the semantic segmentation task without any fine-tuning
on the target dataset.

Comparison with Other SoTA Methods. These methods include one fully supervised
baseline DeiT (Touvron et al. (2021)), two visual self-supervised baselines
DINO (Caron et al. (2021)) and MoCo (He et al. (2020a)), two vision-language
contrastive learning baselines CLIP (Radford et al. (2021)) and GroupViT (Xu
et al. (2022)), two vision-language contrast and visual self-supervised
learning combined baselines SLIP (Mu et al. (2021)) and CLIP-MAE (Dong et al.
(2022)), and one vision-language contrast and self-distillation combined
baseline MaskCLIP (Dong et al. (2022)).

Table 2 shows the mIoU performance comparison between ViewCo and the SoTA
methods on PASCAL VOC 2012, PASCAL Context, and COCO validation sets. ViewCo
significantly outperforms them on all three datasets. Compared to GroupViT,
when pre-trained on CC12M, ViewCo achieves a 4.6% mIoU improvement on PASCAL
VOC. Similarly, when pre-trained on CC12M+YFCC, ViewCo achieves a 2.6% mIoU
improvement on COCO compared to GroupViT. Similar to ViewCo, SLIP, MaskCLIP,
and CLIP-MAE all use additional supervision information in the vision branch
of the VLP models. Compared with them, ViewCo still has clear advantages in
PASCAL Context and COCO. In addition, ViewCo obtains segmentation performance
close to the fully supervised DeiT on PASCAL VOC, which again demonstrates the
effectiveness of ViewCo for zero-shot semantic segmentation.

###  4.3 Analysis

In this section, for the convenience of comparison, we use CC12M as the pre-
training dataset by default for the ablation of ViewCo components, qualitative
analysis, and image classification performance comparison. GroupViT (Xu et al.
(2022)) is used as the baseline for ViewCo.

| Visual branch |  COCO mIoU (%)
---|---|---
GroupViT | - | 18.4
GroupViT+ | image-level | 18.6
ViewCo | semantic-level |  19.1(0.7$\uparrow$)
Table 3: Image-level contrast vs. semantic-level contrast. “-” indicates that
no visual self-supervision module is used. GroupViT+ represents modifying the
corresponding component in GroupViT.

Image-Level Contrast vs. Semantic-Level Contrast. To ablate the role of the
cross-view segmentation consistency module in the vision branch, we add an
image-level contrastive module to GroupViT in the visual branch, where we
first calculate the average of the $K$ segment tokens outputted by the teacher
and student networks, and then perform contrastive learning. For ViewCo, we
remove the text-to-views consistency module and directly average pool the
multi-view features outputted by the student network. To be consistent with
GroupViT, we use the pooled visual features for contrastive learning with text
embeddings. As shown in Table 3, adding a visual self-supervised module for
vision-language contrastive learning can improve the performance of the model
on semantic segmentation by improving the quality of visual feature learning.
Furthermore, the improved performance (i.e., 19.1 vs. 18.6) of semantic-level
learning relative to image-level contrastive learning suggests that the cross-
view segmentation consistency module can further improve the performance by
capturing the consistency of cross-view semantic segmentation.

| Visual branch |  Vision -language contrast |  COCO mIoU (%)
---|---|---|---
GroupViT+ | image-level | single | 18.6
GroupViT+ | image-level | multiple |  19.7 (1.1$\uparrow$)
ViewCo | semantic-level | single | 19.1
ViewCo | semantic-level | multiple |  20.6 (1.5$\uparrow$)
Table 4: Vision-language contrast: single-view vs. multi-view. “single” and
“multiple” denote the number of image views used in vision-language
contrastive learning.

Vision-Language Contrast: Text-to-Image vs. Text-to-Views. We further ablate
the text-to-views consistency module in ViewCo. In single-view vision-language
contrastive learning, we use the average embedding of multi-view features
outputted by the student network and the text embedding for contrastive
learning during training. As shown in Table 4, text-to-views consistency
modeling significantly improves the performances of the models compared to
single-view text-to-image (i.e., 1.1% and 1.5%). This indicates that text-to-
views consistency modeling has better high-level semantic alignment
capabilities than text-to-image single-view modeling. This is exactly what
previous methods of single-view vision-language contrastive learning do not
have.

Figure 6:  Comparison of semantic segmentation of images in rare scenes. (a)
Image segmentation and semantic prediction. (b) Image segmentation. ViewCo can
better learn high-level cross-modal semantic alignment with the help of two
consistency modeling schemes.

Qualitative Analysis. Figure 2 shows some visualization results of multi-view
semantic segmentation consistency for ViewCo and GroupViT. As shown in Figure
2(a), in GroupViT, the semantic segmentations of different views from the same
image are inconsistent. For example, in image $x$, “umbrella” is misclassified
as “cow”, and in view $u$, “umbrella” is misclassified as “horse”. There is
also the problem of inconsistent semantic segmentations between views $u$ and
$v$. As shown in Figure 2(b), the semantic segmentation in different views in
ViewCo is completely consistent. This shows that our cross-view segmentation
consistency modeling and text-to-views consistency modeling in ViewCo are
effective.

|  Pre-training dataset | Zero-shot
---|---|---
| Acc@1 (%) | Acc@5 (%)
GroupViT | CC12M | 37.5 | 65.5
ViewCo | CC12M |  39.5 (2.0$\uparrow$) |  68.4 (2.9$\uparrow$)
ViT | CC12M+YFCC | 42.4 | -
GroupViT | CC12M+YFCC | 42.9 | 71.7
ViewCo | CC12M+YFCC |  46.3 (3.4$\uparrow$) |  74.0 ($2.3\uparrow$)
Table 5: Zero-shot performance on ImageNet.

To evaluate ViewCo’s ability to perform semantic segmentation through semantic
understanding in rare scenes, we show the more visual comparison in Figure 6.
The images of rare scenes are selected from the Internet. In Figure 6(a), we
use the class labels of the PASCAL VOC 2012 dataset as the label set for the
images. ViewCo’s segmentation and prediction results in rare scenes are
significantly better than GroupViT’s. This indicates that ViewCo can better
understand high-level semantics in images through consistent semantic
learning. In Figure 6(b), we only focus on the model’s ability to segment
images in rare scenes. Compared to GroupViT, ViewCo handles the details of
image segmentation much better.

More visual comparison results are shown in Figure 7 of A.2 of the
supplementary material. In addition, we also visually compare the segmentation
consistency of ViewCo and GroupViT on different views in A.3. Finally, we
present an analysis of ViewCo’s cross-view segmentation consistency in A.4.

Image Classification. We also evaluate the classification performance of
ViewCo. As shown in Table 5, ViewCo significantly outperforms ViT (i.e., 46.3%
vs. 42.4%) and GroupViT (i.e., 46.3% vs. 42.9%), showing that ViewCo achieves
better cross-modal semantic alignment through text-to-views consistency
modeling.

##  5 Conclusion

We propose a novel and simple multi-view consistency learning (ViewCo) for
text-supervised semantic segmentation. To deal with the problems of
excessively strict image-text correspondence and ambiguous text supervision in
the VLP model, ViewCo models the text-to-views consistency and cross-view
segmentation consistency. ViewCo can generate consistent segmentations and
better capture high-level cross-modal semantic alignment. We expect that this
exploration of multi-view consistent learning is also applicable to other VLP
tasks.

##  6 Acknowledgment

This work was supported in part by National Key R&D Program of China under
Grant No.2020AAA0109700, National Natural Science Foundation of China (NSFC)
under Grant No.61976233, Guangdong Outstanding Youth Fund (Grant
No.2021B1515020061), Shenzhen Fundamental Research Program (Project
No.RCYX20200714114642083, No.JCYJ20190807154211365). We thank MindSpore and
CAAI-Huawei MindSpore Open Fund for the partial support of this work, which is
a new deep learning computing framwork222https://www.mindspore.cn.

## References

  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 2425–2433, 2015.
  * Bao et al. (2021) Hangbo Bao, Li Dong, and Furu Wei.  Beit: Bert pre-training of image transformers.  _arXiv preprint arXiv:2106.08254_ , 2021.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 9650–9660, 2021.
  * Changpinyo et al. (2021) Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 3558–3568, 2021.
  * Chen et al. (2020a) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International Conference on Machine Learning_ , pp. 1597–1607. PMLR, 2020a.
  * Chen et al. (2020b) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In _European Conference on Computer Vision_ , pp. 104–120. Springer, 2020b.
  * Comaniciu & Meer (2002) Dorin Comaniciu and Peter Meer.  Mean shift: A robust approach toward feature space analysis.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 24:603–619, 2002.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 248–255. Ieee, 2009.
  * Desai & Johnson (2021) Karan Desai and Justin Johnson.  Virtex: Learning visual representations from textual annotations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 11162–11173, 2021.
  * Dong et al. (2022) Xiaoyi Dong, Yinglin Zheng, Jianmin Bao, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al.  Maskclip: Masked self-distillation advances contrastive language-image pretraining.  _arXiv preprint arXiv:2208.12262_ , 2022.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2020.
  * Everingham et al. (2010) Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  _International Journal of Computer Vision_ , 88(2):303–338, 2010.
  * Ghiasi et al. (2021) Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin.  Open-vocabulary image segmentation.  _arXiv preprint arXiv:2112.12143_ , 2021.
  * He et al. (2020a) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9729–9738, 2020a.
  * He et al. (2020b) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 9729–9738, 2020b.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 16000–16009, 2022.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _International Conference on Machine Learning_ , pp. 5583–5594. PMLR, 2021.
  * Kingma & Ba (2015) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _ICLR (Poster)_ , 2015.
  * Li et al. (2020a) Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 11336–11344, 2020a.
  * Li et al. (2021a) Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in Neural Information Processing Systems_ , 34:9694–9705, 2021a.
  * Li et al. (2020b) Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu.  Hero: Hierarchical encoder for video+ language omni-representation pre-training.  _arXiv preprint arXiv:2005.00200_ , 2020b.
  * Li et al. (2021b) Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  _arXiv preprint arXiv:2110.05208_ , 2021b.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European Conference on Computer Vision_ , pp. 740–755. Springer, 2014.
  * Loshchilov & Hutter (2016) Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with restarts.  _Learning_ , 10:3, 2016.
  * Mottaghi et al. (2014) Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille.  The role of context for object detection and semantic segmentation in the wild.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 891–898, 2014.
  * Mu et al. (2021) Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  Slip: Self-supervision meets language-image pre-training.  _arXiv preprint arXiv:2112.12750_ , 2021.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pp. 8748–8763. PMLR, 2021.
  * Ren et al. (2022) Pengzhen Ren, Changlin Li, Guangrun Wang, Yun Xiao, Qing Du, Xiaodan Liang, and Xiaojun Chang.  Beyond fixation: Dynamic window visual transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 11987–11997, 2022.
  * Shi & Malik (1997) Jianbo Shi and Jitendra Malik.  Normalized cuts and image segmentation.  _Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pp. 731–737, 1997.
  * Thomee et al. (2016) Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  _Communications of the ACM_ , 59(2):64–73, 2016\.
  * Touvron et al. (2021) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _International Conference on Machine Learning_ , pp. 10347–10357. PMLR, 2021.
  * Wang et al. (2021a) Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li.  Dense contrastive learning for self-supervised visual pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 3024–3033, 2021a.
  * Wang et al. (2021b) Zekun Wang, Wenhui Wang, Haichao Zhu, Ming Liu, Bing Qin, and Furu Wei.  Distilled dual-encoder model for vision-language understanding.  _arXiv preprint arXiv:2112.08723_ , 2021b.
  * Wei & Zou (2019) Jason Wei and Kai Zou.  Eda: Easy data augmentation techniques for boosting performance on text classification tasks.  _arXiv preprint arXiv:1901.11196_ , 2019.
  * Xu et al. (2022) Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang.  Groupvit: Semantic segmentation emerges from text supervision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18134–18144, 2022.
  * Xu et al. (2021) Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai.  A simple baseline for zero-shot semantic segmentation with pre-trained vision-language model.  _arXiv preprint arXiv:2112.14757_ , 2021.
  * Zabari & Hoshen (2021) Nir Zabari and Yedid Hoshen.  Semantic segmentation in-the-wild without seeing any segmentation examples.  _arXiv preprint arXiv:2112.03185_ , 2021.
  * Zellers et al. (2019) Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  From recognition to cognition: Visual commonsense reasoning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6720–6731, 2019.
  * Zhou et al. (2021a) Chong Zhou, Chen Change Loy, and Bo Dai.  Extract free dense labels from clip.  _arXiv preprint arXiv:2112.01071_ , 2021a.
  * Zhou et al. (2021b) Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  Image bert pre-training with online tokenizer.  In _International Conference on Learning Representations_ , 2021b.
  * Zhou et al. (2020) Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao.  Unified vision-language pre-training for image captioning and vqa.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 13041–13049, 2020.

##  Appendix A Appendix

###  A.1 Training details

We conduct all the experiments on 24 $\times$ Nvidia V100 GPUs. Following Xu
et al. (2022), we set the batch size to 4096, and the learning rate is
initialized to 0.0016 and decayed via the cosine schedule (Loshchilov & Hutter
(2016)). We use the Adam optimizer (Kingma & Ba (2015)) with a weight decay of
0.05, and train ViewCo for 30 epochs with 5 initial epochs containing linear
warm-up. The number $M$ of generated text prompts is set to 3, and the text
templates are the same as Radford et al. (2021). Additionally, we perform
contrastive learning using the semantic embeddings outputted by the student
network $f_{s}$ and the text branch. They are mapped to the same latent space
through two-layer MLPs.

###  A.2 Additional Qualitative Results

Figure 7:  Qualitative comparison on PASCAL VOC 2012. The red boxes indicate
obvious segmentation errors. ViewCo has better segmentation details than
GroupViT (Xu et al. (2022)).

Visual comparison on PASCAL VOC 2012. Figure 7 shows the semantic segmentation
visualization results of five groups of GroupViT and ViewCo. ViewCo can obtain
better segmentation results than GroupViT.

###  A.3 Multi-View Segmentation Consistency Comparison

Figure 8:  Visual comparison between GroupViT and ViewCo on multi-view (normal
and rotated view) segmentation consistency. The red boxes indicate obvious
segmentation errors. Compared with GroupViT, ViewCo has better segmentation
consistency across multiple views and stable segmentation performance for
rotated views.

The segmentation consistency comparison between GroupViT and ViewCo on
multiple views is shown in Figure 8. We obtain two augmented views by cropping
or rotating augmentation and then adopt the weights trained on CC12M by
GroupViT and ViewCo to evaluate the segmentation consistency of the two models
on multiple views. It can be seen that the segmentation of GroupViT on normal
and rotated views is inconsistent, and GroupViT performs poorly on rotated
views compared to normal views. In contrast, ViewCo’s segmentation on normal
and rotated views maintains good consistency and is not disturbed by image
rotation. This shows that ViewCo’s cross-view segmentation consistency module
learns different view consistency semantics well.

###  A.4 Cross-view segmentation consistency visualization analysis

Figure 9:  A visual example of ViewCo’s cross-view segmentation consistency.

In addition, we further analyze the effect of image flip augmentation on
ViewCo’s cross-view segmentation consistency module by visualization in Figure
9. Specifically, we visualized the segmentation results of the 8 segment
semantics output by the second grouping block in the teacher and student
networks of different views. It should be noted that we did not perform any
post-processing on the visualization results, but only numbered the main
semantics in the image for the convenience of visualization. Image
segmentation shows less than 8 semantics, mainly because in the grouping block
the model determines which segment semantics each image patch belongs to by
computing an attention map of shape 196 $\times$ 8 (196 is the number of image
patches). This means that some segments may not have corresponding image
patches. And, in the segmentation results from different views of the same
image, image regions with the same color represent the same location IDs and
the same semantics. From the visualization results in Figure 9, it can be
found that the semantics in the segmentation maps of different views output by
the teacher and student networks maintain a good consistency in the
corresponding position IDs, which again justifies the design of ViewCo’s
cross-view segmentation consistency module.


# Formation of Multiple Populations of M5 (NGC 5904)

Jae-Woo Lee Department of Physics and Astronomy, Sejong University
209 Neungdong-ro, Gwangjin-Gu, Seoul, 05006, Republic of Korea
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

With our new Ca-CN-CH-NH photometry, we revisit the globular cluster (GC) M5.
We find that M5 is a mono-metallic GC with a small metallicity dispersion. Our
carbon abundances show that the $\sigma$[C/Fe] of the M5 CN-s population, with
depleted carbon and enhanced nitrogen abundances, is significantly large for a
single stellar population. Our new analysis reveals that the M5 CN-s
population is well described by the two stellar populations: the CN-sI, being
the major CN-s component, with the intermediate carbon and nitrogen abundance
and the CN-sE with the most carbon-poor and nitrogen-rich abundance. We find
that the CN-sE is significantly more centrally concentrated than the others,
while CN-w and CN-sI have similar cumulative radial distributions. The red
giant branch bump $V$ magnitude, the helium abundance barometer in mono-
metallic populations, of individual populations appears to be correlated with
their mean carbon abundance, indicating that carbon abundances are
anticorrelated with helium abundances. We propose that the CN-sE formed out of
gas that experienced proton-capture processes at high temperatures in the
innermost region of the proto-GC of M5 that resided in a dense ambient density
environment. Shortly after, the CN-sI formed out of gas diluted from the
pristine gas in the more spatially extended region, consistent with the
current development of numerical simulations by others.

Stellar populations (1622); Population II stars (1284); Hertzsprung Russell
diagram (725); Globular star clusters (656); Chemical abundances (224);
Stellar evolution (1599); Red giant branch (1368)

##  1 Introduction

Recent photometric and spectroscopic studies of globular clusters (GCs) in our
Galaxy and nearby galaxies revealed the ubiqutous nature of multiple
populations (MPs) in GCs and opened a new golden era in the field of stellar
populations (e.g, Carretta et al., 2009; Lee et al., 2009; Milone et al.,
2017; Bastian & Lardo, 2018; Gratton et al., 2019). Previously considered as
simple and dull, understanding the formation and evolution of GCs requires
complex and somewhat fine-tuned processes. One of the remarkable aspects is
that the second generation (SG) of stars, which is believed to form in more
spatially concentrated inner regions of GCs out of gas ejected from the
clusters’ first generation (FG) of stars (e.g., D’ercole et al., 2008), is the
major component of GCs with MPs. Recently, sophisticated numerical simulations
have become available and they help us to delineate how the second generation
of stars formed and dynamically evolved under certain assumptions (e.g.,
Bekki, 2019; Calura et al., 2019).

Major observational breakthroughs have been made via the Hubble Space
Telescope (e.g., Milone et al., 2017) and the ground-based multi-object
spectroscopy with large aperture telescopes (e.g., Carretta et al., 2009).
However, these approaches have some drawbacks: broadband photometry has a
potential degeneracy problem for individual elemental abundances, while
conventional spectroscopy has a potential contamination problem by nearby
stars in crowded regions.

Surface carbon and nitrogen abundances, the key elements in the study of GC
MPs, can illuminate the internal nucleosynthesis of evolved GC stars and even
an entire cluster’s evolutionary history. However, due to observational
limitations, both in the HST photometry and the ground-based spectroscopy that
we mentioned above, reliable carbon and nitrogen abundances are often poorly
known.

In this Letter, we revisit the GC M5 (NGC 5904) with our new photometric
system, optimized to study carbon and nitrogen abundances of red giant branch
(RGB) stars in crowded fields. In our previous Ca-CN photometric study of the
cluster, we found that M5 contains two MPs with identical radial distributions
but different structural and kinematical properties (Lee, 2017, 2019a, 2019b).
Our new Ca-CN-CH-NH observations of the cluster reveal new population with
interesting physical properties that we were not able to detect in our
previous study. Here we present the [Fe/H], [C/Fe], and [N/Fe] of individual
RGB stars and new insight on the formation and evolution of MPs in M5.

##  2 New Observations

During the last decade, we developed a new photometric system that can measure
Ca II H and K lines, NH, CN, and CH molecular band absorption strengths at
$\lambda$3360, $\lambda$3883, and $\lambda$4250, respectively, to provide
reliable [Fe/H], [C/Fe], and [N/Fe] of individual RGB stars using theoretical
fine model grids with various stellar input parameters and elemental
abundances (see Lee, 2015, 2017, 2019a, 2019b; Lee & Sneden, 2021, for our new
filter system). Our method can avoid the crowding effects with potential
contamination by neighboring stars, which is one of the limitations in the
classical spectroscopic study of the GC stars, in particular, in the central
part of GCs. Therefore, our approach can guarantee a more complete sample for
the GC study.

Observations for our Strömgren and Ca-CN photometry were conducted using the
CTIO 1.0 m telescope in 21 nights in seven runs from 2007 May to 2014 May (see
Lee, 2017, for details). In addition, we obtained the JWL34 and JWL43
photometry using the KPNO 0.9 m telescope in 11 nights in three separate runs
from 2017 February and 2019 July.111The transmission functions for our JWL33
and JWL43 filters can be found in Figure 3 of Lee & Sneden (2021) and Figure 1
of Lee (2019b), respectively.

The raw data handling were described in detail in our previous works (Lee,
2015, 2017). The photometry of M5 and standard stars were analyzed using
DAOPHOTII, DAOGROW, ALLSTAR and ALLFRAME, and COLLECT-CCDAVE-NEWTRIAL packages
(Stetson, 1994). The total number of stars in our M5 field from our ALLFRAME
run was more than 60,000.

Finally, we derived the astrometric solutions for individual stars using
coordinates of more than 3500 stars extracted from the Gaia Early Data Release
3 (EDR3, Gaia Collaboration, 2020) and the IRAF IMCOORS package. The rms
errors of our fit are very small, 0$\farcs$008 and 0$\farcs$006 along the R.A.
and the decl., respectively.

Figure 1:  CMDs of M5 membership stars based on the proper motion study of the
Gaia EDR3 (Gaia Collaboration, 2020). We only show the stars with radial
distance from the center larger than 1′. Discrete bimodal RGB sequences can be
seen in the M5 $cn_{\rm JWL}$ CMD, while a broad RGB sequence can be seen in
the $ch_{\rm JWL}$,$nh_{\rm JWL}$, and $m1$ CMDs (see also Lee, 2017).

##  3 Results

###  3.1 Color-Magnitude Diagrams

Throughout this work, we will use our own photometric indices already defined
in our previous works (Lee, 2017, 2019a, 2019b; Lee & Sneden, 2021),

$\displaystyle hk_{\rm JWL}$ $\displaystyle=$ $\displaystyle({\rm Ca}_{\rm
JWL}-b)-(b-y).$ (1) $\displaystyle nh_{\rm JWL}$ $\displaystyle=$
$\displaystyle({\rm JWL34}-b)-(b-y).$ (2) $\displaystyle cn_{\rm JWL}$
$\displaystyle=$ $\displaystyle{\rm JWL39}-{\rm Ca}_{\rm JWL},$ (3)
$\displaystyle ch_{\rm JWL}$ $\displaystyle=$ $\displaystyle({\rm
JWL43}-b)-(b-y).$ (4)

Note that the $hk_{\rm JWL}$ index measures the absorption strengths of Ca II
H and K lines and it is a good measure of metallicity (Anthony-Twarog et al.,
1991; Lee et al., 2009; Lee, 2015). The $nh_{\rm JWL}$, $cn_{\rm JWL}$, and
$ch_{\rm JWL}$ are excellent measures of the NH band at $\lambda$3360, the CN
band at $\lambda$3883, and the CH G band at $\lambda$4250, respectively, for
cool stars (Lee, 2017, 2019b, 2020; Lee & Sneden, 2021).

We made use of the proper motions from the Gaia EDR3 to select the cluster’s
membership stars, following the method similar to those used in our previous
studies (see, e.g., Lee, 2020; Lee & Sneden, 2021, and references therein). We
derived the mean values of proper motions of M5 with iterative sigma-clipping
calculations, finding that, in units of mas yr-1, ($\mu_{\rm
R.A.}\times\cos\delta$, $\mu_{\rm decl.}$) = (4.058, $-$9.873) with standard
deviations along the major axis of the ellipse of 0.845 mas yr-1 and along the
minor axis of 0.686 mas yr-1. We considered that stars within 4$\sigma$ from
the mean values to be M5 member stars. Then we selected our target RGB stars
with $-$2 mag $\leq$ $V-V_{\rm HB}$ $\leq$ 2 mag from our multicolor
photometry. In Figure 1, we show our new color-magnitude diagrams (CMDs) for
M5 membership stars. The $cn_{\rm JWL}$ CMD shows discrete double RGB
sequences (see also Lee, 2017, 2019a, 2019b). On the other hand, our $ch_{\rm
JWL}$, $nh_{\rm JWL}$, and $m1$ CMDs show very broad RGB sequences, due to
spreads in the carbon and nitrogen abundances as we will discuss later (also
see Lee & Sneden, 2021).

We also note that M5 contains a very broad range of the horizontal branch (HB)
morphology (e.g., see Gratton et al., 2013). Compared to M3, which is slightly
more metal-poor than M5 (Lee & Sneden, 2021), M5 has a well developed blue HB,
most likely due to the presence of the dispersion in its helium abundance as
we will discuss later (see also Lee, 2017).

Table 1: Mean [Fe/H] values. | [Fe/H] | No.
---|---|---
All |  $-$1.295 $\pm$ 0.039 | 842
CN-w |  $-$1.290 $\pm$ 0.047 | 242
CN-s |  $-$1.296 $\pm$ 0.043 | 600
CN-sI |  $-$1.293 $\pm$ 0.032 | 477
CN-sE |  $-$1.323 $\pm$ 0.036 | 124
Ivans et al. (2001) |  $-$1.29 $\pm$ 0.05 | 19
Carretta et al. (2009) |  $-$1.35 $\pm$ 0.02 | 136
Gratton et al. (2013) |  $-$1.33 $\pm$ 0.02 | 30
Husser et al. (2020) |  $-$1.16 $\pm$ 0.20 | 863
Table 2: Dispersions in Carbon Abundances of the Faint RGBs | M5 |  | M3
---|---|---|---
| CN-w | CN-s |  | CN-w | CN-s
$\sigma$[C/Fe] |  | 0.074 | 0.136 |  | 0.068 | 0.083
Figure 2:  (a) A plot of [C/Fe] vs. [N/Fe] for well measured [$\sigma$(hk)
$\leq$ 0.01 mag] M5 RGB stars with $-$2 mag $\leq$ $V-V_{\rm HB}$ $\leq$ 2
mag. The blue color denotes the CN-w population, while the red and dark-green
colors denote the CN-s-I and CN-s-E populations. (b) A plot of [C/Fe] vs. $V$
magnitude, showing that the boundary between the CN-sI and CN-sE is not set at
a fixed [C/Fe] but reflects the variation of carbon abundance against the $V$
magnitude due to the internal mixing during the evolution of low-mass stars.

###  3.2 Elemental Abundances and Populational Tagging

In our previous study of M5, we performed populational tagging based on the
$cn_{\rm JWL}$ distribution of RGB stars, showing a discrete bimodal
distribution (Lee, 2017, 2019a, 2019b). Here, we take a similar path that will
reveal three different populations in M5.

Using the populational tagging for the CN-w and CN-s populations222The CN-w
and CN-s populations of our study are defined to be groups of RGB stars with
weak CN band strengths at $\lambda$3883 (i.e., carbon-normal and nitrogen-
normal) and strong CN band strengths (i.e., carbon-poor and nitrogen-rich),
respectively, following the conventional nomenclature (e.g., Norris et al.,
1981). from our previous study (Lee, 2017, 2019a, 2019b), we derived
photometric elemental abundances from our color indices using the similar
method that we developed in our previous work (Lee & Sneden, 2021). We
obtained the Dartmouth model isochrones for [Fe/H] = $-$1.5, $-$1.4, $-$1.3,
$-$1.2, and $-$1.1 dex with [$\alpha$/Fe] = +0.4 dex, and the age of 12.5 Gyr
(Dotter et al., 2008). We interpolated the effective temperatures and surface
gravities from $M_{V}$ = 3.5 to $-$2.5 mag with a magnitude step size of
$\Delta M_{V}$ = 0.2 mag. Using these stellar parameters, we constructed a
series of synthetic spectra with varying elemental abundances with abundance
step sizes of [X/Fe] = 0.2 dex. For our synthetic spectrum calculations, we
used the 2011 version of the local thermodynamic equilibrium (LTE) line
analysis code MOOG that includes Rayleigh scattering from neutral hydrogen
(Sneden, 1973; Sobeck et al., 2011) and the atomic/molecular line lists
generated from the $linemake$ facility.333Available at
https://github.com/vmplacco/linemake. In total, we calculated more than
210,000 synthetic spectra for our current work. Finally, individual synthetic
spectra were convolved with our filter transmission functions to be converted
to our photometric system.

The photometric metallicity of individual RGB stars can be calculated using
the following relation (also see Appendices of Lee & Sneden, 2021),

$\displaystyle{\rm[Fe/H]}$ $\displaystyle\approx$ $\displaystyle f_{1}(hk_{\rm
JWL},~{}M_{V}),$ (5)

and our results are given in Table 1. We obtained [Fe/H] = $-$1.295 $\pm$
0.039 $\pm$ 0.001 dex (the errors are for the standard deviation and the
standard error) and our result appears to be consistent with those from other
researchers listed in Table 1 to within measurement uncertainties. Note that
we used only those RGB stars with low measurement errors, $\sigma$(hk) $\leq$
0.01 mag, which will ensure that our photometric metallicity of individual
stars is not affected by measurement error. Also importantly, the proper
motion study of Gaia EDR3 is not complete in the central part of the cluster.
Consequently, our RGB sample with photometric elemental abundances is not a
complete sample and tends to be biased toward the outer part of cluster, where
the degree of crowding effect from the nearby stars is less severe. In total,
we measured metallicity for 242 CN-w and 600 CN-s RGB stars. As shown in the
table, our photometric [Fe/H] values of each population are in excellent
agreement with each other, suggesting that M5 is a mono-metallic GC to within
$\sigma$[Fe/H] $\lesssim$ 0.05 dex.

The photometric carbon and nitrogen abundances can be estimated using the
above 842 RGB stars as follows,

$\displaystyle{\rm[C/Fe]}$ $\displaystyle\approx$ $\displaystyle f_{2}(ch_{\rm
JWL},~{}{\rm[Fe/H]},~{}M_{V}),$ (6) $\displaystyle{\rm[N/Fe]}$
$\displaystyle\approx$ $\displaystyle f_{3}(nh_{\rm
JWL},~{}{\rm[Fe/H]},~{}M_{V}).$ (7)

In Figure 2, we show a plot of [C/Fe] versus [N/Fe] of the M5 RGB stars with
$-$2 mag $\leq$ $V-V_{\rm HB}$ $\leq$ 2 mag, showing a strong carbon–nitrogen
anticorrelation, which is a natural consequence of the CN-cycle hydrogen
burning.

It should be noted that the carbon abundance spread in the M5 CN-s RGB stars
is very large. In Table 2, we compare the $\sigma$[C/Fe] of M5 and M3 RGB
stars fainter than their RGB bumps (RGBBs). It is believed that RGB stars
fainter than the RGBB maintain their initial [C/Fe] and [N/Fe] abundances and
are not affected by the CN cycle accompanied by a noncanonical thermohaline
deep mixing that can significantly alter the surface [C/Fe] and [N/Fe] during
the evolution of low-mass stars (Charbonnel & Zahn, 2007). We note that the
$\sigma$[C/Fe] of the CN-w and CN-s populations in M3 are comparable to each
other (Lee & Sneden, 2021). In sharp contrast, the $\sigma$[C/Fe] the M5 CN-s
stars is about twice as large as that of the M5 CN-w population, indicating
that the M5 CN-s population may contain multiple subpopulations.444See Figure
5(a) of Lee & Sneden (2021) and Figure 9 of Lee (2019b) for the $\parallel
ch_{\rm JWL}$ extents of the CN-s populations in M3 and M5, where  $\parallel
ch_{\rm JWL}\equiv\frac{ch_{\rm JWL}-ch_{\rm JWL,red}}{ch_{\rm
JWL,red}-ch_{\rm JWL,blue}},$ (8) and $ch_{\rm JWL,blue}$ and $ch_{\rm
JWL,red}$ are the fiducials of the red and the blue sequences of the $ch_{\rm
JWL}$ index, respectively. One can find that the extent of the $\parallel
ch_{\rm JWL}$ of the M5 CN-s population is significantly large compared to
that of the M3 CN-s. Note that the previous spectroscopic study of M5 by
Carretta et al. (2009) showed an extended Na-O anticorrelation. They proposed
that M5 may contain three different populations, the primordial, the
intermediate, and the extreme components, based on the location on the plot of
[O/Fe] versus [Na/Fe].555Note that Carretta et al. (2009) separated three
populations in M5 from fixed [O/Fe] and [Na/Fe] values, without considering
elemental abundance variations against luminosity, due to the existence of
evolutionary effects, such as an internal deep mixing, as Lee (2010) showed.
Therefore, their primordial, intermediate, and extreme components are not
exactly the same as our CN-w, CN-sI, and CN-sE populations (see also Lee,
2017).

In our previous study of the cluster (Lee, 2017, 2019a, 2019b), we studied the
MPs of M5 RGB stars based on the $cn_{\rm JWL}$ distribution, finding two
distinctive populations, the CN-w and CN-s. Careful examination of the
$\parallel ch_{\rm JWL}$ distribution of the M5 CN-s population may suggest
that it can be well described by a bimodal distribution. In order to
distinguish the CN-s subpopulations, we applied the expectation maximization
(EM) algorithm for the two-component Gaussian mixture model, finding the
number ratio of $n$(CN-sI):$n$(CN-sE) = 79:21 ($\pm$3). As shown in Figure 2,
our CN-sI and CN-sE are those with the intermediate [C/Fe] and the most carbon
depleted populations, respectively, and they are roughly corresponding to the
intermediate and extreme components by Carretta et al. (2009), respectively.
Including the CN-w population, which is corresponding to the primordial
component devised by Carretta et al. (2009), the M5 populational number ratio
becomes $n$(CN-w):$n$(CN-sI):$n$(CN-sE) = 29:56:15 ($\pm$3).

Figure 3:  CRDs of the individual populations in M5. Vertical gray dashed
lines are for the core and half-light radii. Note that the CN-sE stars are
significantly centrally concentrated than other populations.

###  3.3 Cumulative Radial Distributions

The cumulative radial distributions (CRDs) of MPs in GCs can provide pivotal
information on the formation and the dynamical evolution of MPs. Recent
numerical simulations may suggest that the SG with extreme helium abundance
formed first in the innermost part of GCs out of gas that experienced proton-
capture processes at high temperatures, while the SG with modest helium
enhancement formed out of interstellar media diluted from the pristine gas in
the more spatially extended region. The initial formation location of the SG
of stars may depend on the physical environment that proto-GCs resided (Calura
et al., 2019). Also, the dynamical effects on smaller stellar masses of the
helium-enhanced populations can alter initial CRDs during the course of GC
evolution, although the tidal field of the host galaxy could be a major
component in determining the relative CRDs of MPs (e.g., see Fare et al.,
2018)

In our previous study, we showed that CRDs of the M5 CN-w and CN-s populations
are statistically identical up to more than five half-light radius. Here we
repeated similar tasks for the three populations with low measurement
uncertainties and we show our results in Figure 3 and Table 3. The figure
clearly shows that the CN-w CRD is in good agreement with that of the CN-s [=
CN-sI \+ CN-sE]. We performed Kolmogorov–Smirnov (K-S) and Anderson–Darling
(A-D) tests to see if they are statistically similar distributions. It is a
well-known fact that the K-S test can be sensitively dependent on the near
center of the distribution and less dependent on the edges of the
distribution, while the A-D test is known to be less vulnerable to such
problems. Our results show that the CN-w and CN-s are most likely drawn from
the identical parent distributions. Since the CN-sI is the major component of
the CN-s, the CRD of the CN-sI would be similar to that of the CN-s. On the
other hand, the CRD of CN-sE may tell a different story. The figure clearly
shows that the CN-sE is the most centrally concentrated. Our K-S tests suggest
that the CRD of the CN-sE is significantly different from the others.
Therefore, our results strongly suggest that the CN-sE population must have
formed in the innermost region of M5.

Table 3: $p$ Values Returned from the K-S and A-D Tests for Individual CRDs Populations | K-S | A-D
---|---|---
CN-w | vs. | CN-s | 0.505 | 0.211
CN-s | vs. | CN-sI | 0.361 | 0.098
CN-s | vs. | CN-sE | 1.05$\times 10^{-4}$ | 2.18$\times 10^{-6}$
CN-w | vs. | CN-sI | 0.099 | 0.147
CN-w | vs. | CN-sE | 1.35$\times 10^{-5}$ | 1.11$\times 10^{-6}$
CN-sI | vs. | CN-sE | 6.51$\times 10^{-7}$ | 1.49$\times 10^{-9}$
Figure 4:  Plots of $(b-y)$ vs. $V$ CMDs and generalized differential LFs of
the CN-w, CN-sI, and CN-sE RGB stars. We show the RGBB $V$ magnitudes of
individual populations with gray dotted lines. The numbers in the bottom of
each panel denote the mean [C/Fe] values of the RGB stars fainter than the
RGBB.

###  3.4 Red Giant Branch Bump Magnitudes

Helium is the second most abundant element but it cannot be detected directly
in GC RGB stars due to the lack of any measurable spectral lines.666For
example, see §6.3.1 of Lee (2019b, and references therein) for the
spectroscopic helium abundance measurements using the photospheric He I
$\lambda$5876Å line and chromospheric He I $\lambda$10830Å line. Instead,
indirect methods, such as the RGBB magnitudes of a single population, can play
an important role to probe the helium contents in GCs. During the evolution of
low-mass stars, RGB stars experience slower evolution and temporary drop in
luminosity when the very thin H-burning shell crosses the discontinuity in the
chemical composition and lowered mean molecular weight left by the deepest
penetration of the convective envelope during the ascent of the RGB, the so-
called RGBB (e.g., see Renzini & Fusi Pecci, 1988). It is well understood
that, at a given age, the RGBB $V$ magnitude decreases with metallicity and
increases with helium abundance. In the study of MPs in a GC, very accurate
differential photometry can be attained and, therefore, one can accurately
estimate relative helium contents among MPs in a mono-metallic GC (Lee, 2015,
2017, 2018; Milone et al., 2018; Lee & Sneden, 2021).

We derived the generalized differential luminosity functions (LFs) and the
RGBB $V$ magnitudes for individual populations in M5 and we show our results
in Figure 4. We obtained the RGBB $V$ magnitudes of 15.034, 14.962, 14.968,
and 14.944 ($\pm$0.030) mag for the CN-w, CN-s, CN-sI, and CN-sE,
respectively. Note that our new measurements for the limited number of CN-w
and CN-s stars are consistent with our previous results, 15.038 and 14.970
($\pm$0.030), respectively (Lee, 2017). Our photometric metallicities of the
three populations are in excellent agreement and, therefore, the difference in
helium abundance is most likely responsible for the RGBB magnitude difference.
As we argued previously (Lee, 2017), our results suggest that the CN-sI is
likely enhanced in helium by $\Delta$Y $\approx$ 0.026 $\pm$ 0.017 with
respect to the CN-w. The RGBB $V$ magnitude of the CN-sE population is the
brightest and it could be the most helium-enhanced population by $\Delta$Y
$\approx$ 0.036 $\pm$ 0.017 with respect to the CN-w population. Our inferred
helium abundances are in good agreement with that of D’Antona & Caloi (2008),
who estimated the helium enhancement of the M5 SG by $\Delta$Y $\approx$ 0.02
– 0.07 with respect to the FG from their synthetic HB models.

##  4 Summary

With our new photometric indices and theoretical fine model grids for various
stellar parameters and abundances using synthetic spectra, we derived [Fe/H],
[C/Fe], and [N/Fe], the key elements in the GC MP study, of individual RGB
stars in M5. Our [Fe/H] measurements suggest that M5 is a mono-metallic GCs
with a small metallicity dispersion, [Fe/H] = $-$1.295 $\pm$ 0.039 $\pm$ 0.001
dex .

We showed that the dispersion in [C/Fe] of the M5 CN-s is very large compared
to those of the M5 CN-w or the M3 CN-w and CN-s, suggesting that the M5 CN-s
is composed of MPs. Our new analysis revealed that the M5 CN-s can be well
described by at least two populations, the CN-sI and CN-sE, and, as a
consequence, M5 contains at least three distinctive MPs, including the CN-w.
We obtained the number ratio of $n$(CN-w):$n$(CN-sI):$n$(CN-sE) = 29:56:15
($\pm$3). The CN-sE is significantly more centrally concentrated than the
others, while the CN-s and CN-sI have the similar CRDs as the CN-w does,
consistent with our previous result (Lee, 2017).

We found a correlation between the mean [C/Fe] and the RGBB $V$ magnitude in
individual populations in M5, in the sense that the RGBB $V$ magnitude
increases with the carbon abundance. Since the three populations in M5 have
almost identical metallicity, the difference in the RGBB $V$ magnitude can be
interpreted as the difference in helium abundance. We estimated that the CN-sI
and CN-sE populations are enhanced in helium by $\Delta$Y $\approx$ 0.026
$\pm$ 0.017 and 0.036 $\pm$ 0.017, respectively, with respect to the CN-w
population, which is thought to have a normal helium abundance for its
metallicity.

Our results for the CRDs and inferred helium abundances can be nicely
explained by recent numerical simulations by others (e.g., Fare et al., 2018;
Calura et al., 2019). After the termination of the CN-w supernovae explosions,
the CN-sE (i.e., the most helium-enhanced SG by Calura et al. 2019) formed out
of gas that experienced proton-capture processes at high temperatures, rich in
helium and nitrogen and poor in carbon, in the innermost region. Shortly
after, the CN-sI (i.e., the modest helium-enhanced SG by Calura et al. 2019)
formed out of gas diluted from the pristine gas in the more spatially extended
region. It is likely that the proto-GC of M5 resided in a dense density
external gas environment where the modest helium-enhanced population will be
the major component (Calura et al., 2019), consistent with our new result for
M5.

The different degree of diffusion processes due to slightly smaller stellar
masses with helium abundance cannot be completely ruled out to explain the
similar CRDs between the CN-w and CN-sI. However, as Fare et al. (2018)
discussed, the Milky Way tidal field would be a more important factor to
determine the CRDs of M5, since it has a very elongated orbit in our Galaxy,
with the perigalacticon distance of 2.90 $\pm$ 0.05 kpc and the apgalacticon
distance of 24.20 $\pm$ 1.00 (Baumgardt et al., 2019). At the same time, M5
must have experienced a significant degree of shock-induced mass-loss when it
passed through the Galactic disk and bulge (e.g., Gnedin & Ostriker, 1997),
that may cause the truncation in the radial distributions of the CN-w and CN-
sI, resulting in similar CRDs between the two (e.g., see Lee, 2018).

J.-W.L. thanks an anonymous referee for a careful review of the paper and many
helpful suggestions. He acknowledges financial support from the Basic Science
Research Program (grant No. 2019R1A2C2086290) through the National Research
Foundation of Korea (NRF) and from the faculty research fund of Sejong
University in 2019.

## References

  * Anthony-Twarog et al. (1991) Anthony-Twarog, B. J., Laird, J. N., Payne, D., & Twarog, B. A. 1991, AJ, 101, 1902
  * Bastian & Lardo (2018) Bastian, N., & Lardo, C. 2018, ARA&A, 56, 3
  * Baumgardt et al. (2019) Baumgardt, H., Hilker, M., Sollima, A., & Bellini, A. 2019, MNRAS, 482, 5138
  * Bekki (2019) Bekki, K. 2019, A&A, 622, 53
  * Carretta et al. (2009) Carretta E., Bragaglia, A., Gratton, R.G., et al. 2009, A&A, 505, 117
  * Charbonnel & Zahn (2007) Charbonnel, C., & Zahn, J.-P. 2007, A&A, 467, L15
  * Calura et al. (2019) Calura, F., D’Ercole, A., Vesperini, E., Vanzella, E., & Sollima, A. 2019, MNRAS, 489, 3269
  * D’Antona & Caloi (2008) D’Antona, F., & Caloi, V. 2008, MNRAS, 390, 693
  * D’ercole et al. (2008) D’Ercole, A., Vesperini, E., D’Antona, F., McMillan, S. L. W., & Recchi, S. 2008, MNRAS, 391, 825
  * Dotter et al. (2008) Dotter, A., Chaboyer, B., Jevremović, D., Kostov, V., Baron, E., Ferguson, J. W. 2008, ApJS, 178, 89
  * Fare et al. (2018) Fare, A., Webb, J. J., & Sills, A. 2018, MNRAS, 481, 3027
  * Gaia Collaboration (2020) Gaia Collaboration, Brown, A. G. A., Vallenari, A., et al. 2020a, arXiv:2012.01533
  * Gnedin & Ostriker (1997) Gendin, O. Y., & Ostriker, J. P. 1997, ApJ, 474, 233
  * Gratton et al. (2019) Gratton, R., Bragaglia, A., Carretta, E., et al. 2019, A&A Rev., 27, 8
  * Gratton et al. (2013) Gratton, R., Lucatello, S., Sollima, A., et al. 2013, A&A, 549, 41
  * Husser et al. (2020) Husser, T.-O., Latour, M., Brinchmann, J., et al. 2020, A&A, 635, 114
  * Ivans et al. (2001) Ivans, I. I., Kraft, R. P., Sneden, C., et al. 2001, AJ, 122, 1438
  * Lee (2010) Lee, J.-W. 2010, MNRAS, 405, L36
  * Lee (2015) Lee, J.-W. 2015, ApJS, 219, 7
  * Lee (2017) Lee, J.-W. 2017, ApJ, 844, 77
  * Lee (2018) Lee, J.-W. 2018, ApJS, 238, 24
  * Lee (2019a) Lee, J.-W. 2019a, ApJ, 872, 41
  * Lee (2019b) Lee, J.-W. 2019b, ApJ, 883, 166
  * Lee (2020) Lee, J.-W. 2020, ApJ, 888, L6
  * Lee et al. (2009) Lee, J.-W., Kang, Y.-W., Lee, J., & Lee, Y.-W. 2009, Nature, 462, 480
  * Lee & Sneden (2021) Lee, J.-W., & Sneden, C. 2021, ApJ, 909, 167
  * Milone et al. (2018) Milone A. P., Marino, A.F., Renzini, A. et al. 2018, MNRAS, 481, 5098
  * Milone et al. (2017) Milone, A. P., Piotto, G., Renzini, A., et al. 2017, MNRAS, 464, 3636
  * Norris et al. (1981) Norris, J., Cottrell, P. L., Freeman, K. C., & Da Costa, G. S. 1981, ApJ, 244, 205
  * Renzini & Fusi Pecci (1988) Renzini, A., & Fusi Pecci, F. 1988, ARA&A, 26, 199
  * Sneden (1973) Sneden, C. 1973, PhD thesis, The University of Texas
  * Sobeck et al. (2011) Sobeck, J. E., Kraft, R. P., Sneden, C., et al. 2011, AJ, 141, 175
  * Stetson (1994) Stetson P. B. 1994, PASP, 106, 250