11institutetext: Tsinghua University 22institutetext: Shanghai Qi Zhi
Institute
22email: {renwang435<EMAIL_ADDRESS><EMAIL_ADDRESS>
# CYBORGS: Contrastively Bootstrapping Object Representations by Grounding in
Segmentation

Renhao Wang 11 Hang Zhao 1122⋆⋆ Yang Gao equal advising1122

###### Abstract

Many recent approaches in contrastive learning have worked to close the gap
between pretraining on iconic images like ImageNet and pretraining on complex
scenes like COCO. This gap exists largely because commonly used random crop
augmentations obtain semantically inconsistent content in crowded scene images
of diverse objects. In this work, we propose a framework which tackles this
problem via joint learning of representations and segmentation. We leverage
segmentation masks to train a model with a mask-dependent contrastive loss,
and use the partially trained model to bootstrap better masks. By iterating
between these two components, we ground the contrastive updates in
segmentation information, and simultaneously improve segmentation throughout
pretraining. Experiments show our representations transfer robustly to
downstream tasks in classification, detection and segmentation.111Code and
pretrained models available at https://github.com/renwang435/CYBORGS

##  1 Introduction

Many self-supervised contrastive methods have come to rival and even surpass
the performance of fully supervised methods on a number of tasks, including
object detection [3, 44], semantic segmentation [17, 41], video understanding
[23, 30], and image classification [13, 8]. A large portion of these methods
rely on random cropping to select positive pairs of image subregions for a
self-supervised instance-level discrimination task. Recently, many works have
found this random cropping strategy succeeds for iconic image pretraining, but
struggles when applied to pretraining on complex scene images. Treating two
random crops from the same image as containing semantically similar
information works well for images with singular, dominant subjects, like those
in ImageNet. But such an assumption inevitably fails due to inconsistent
learning signals in scene images full of diverse objects [40, 9, 33]. To
address this issue, prior works have generated random crops in an object-aware
manner [44, 3, 34]. By localizing objects with unsupervised algorithms (e.g.
selective search), these works are able to ground random crops around singular
objects, validating the assumption that such crops contain similar information
about objects.

We argue that utilizing _pixel-level_ object information can be even more
effective than detection-level boxes. By parsing random crop contents with
segmentation masks, we can turn a pair of crops into a diverse wealth of
similar and dissimilar object regions, facilitating contrastive self-
supervised learning. To fully realize this idea in SSL frameworks, we also
need to meet two important criteria. Firstly, these masks should be obtained
in an _unsupervised_ manner. Secondly, we want to _avoid preprocessing
pipelines_ to obtain pseudo-segmentations (_e.g._ , graph cut algorithms),
which often lack awareness of object-level semantics and require human domain
knowledge for good performance [20, 50].

Figure 1: Mutually improving representation learning and semantic
segmentation. In the first stage, we use available segmentation masks to
ground contrastive learning. In the second stage, we use representations from
the backbone $f_{\theta}$ to bootstrap improved segmentation masks.

To this end, we propose in this work to perform segmentation and concept
learning _jointly_ (Fig. 1). In the first stage of our framework, we ground
self-supervised learning with segmentation information to train a
representation backbone. In a periodic second stage, we leverage these
representations to bootstrap segmentation masks, which can subsequently be fed
back to the first stage to further improve representations. By iterating
between these two core stages, we develop representations which strongly
generalize to many downstream tasks, and are especially well-aligned with
object detection and segmentation. Furthermore, to ameliorate issues of
representation collapse, we also optimize a clustering consistency objective
during the first stage. We show that the formulation of this loss fits
naturally within any contrastive framework, and helps improve masks more
reliably between bootstrap cycles. Thus, in ContrastivelY Bootstrapping Object
Representations by Grounding in Segmentation (cyborgs), our contributions are
fourfold:

  1. 1.

We develop the first framework which performs end-to-end, joint self-
supervised learning of object-level representations and semantic segmentation,
while removing entirely the need for heuristic preprocessing of pseudo-
segmentations.

  2. 2.

We show how to bootstrap segmentation masks robustly by directly clustering on
feature maps obtained from a partially pretrained backbone.

  3. 3.

We demonstrate how to regularize contrastive updates in our framework with an
intra-/inter-view cluster consistency loss that is well-aligned with the
hyperspherically-distributed contrastive embeddings.

  4. 4.

With pretraining on complex scene images such as COCO, we demonstrate that
grounding in segmentation leads to representations which transfer
competitively to a diversity of downstream tasks and real-world, long-tail
objects and scene semantics.

##  2 Related Work

#### Self-supervised representation learning.

SSL methods utilize internal structure as a source of supervision to learn
general representations, including auxiliary tasks such as context prediction
[13], solving jigsaws [35], inpainting [37], colorization [51], or orientation
prediction [27]. Most relevant to our work is contrastive learning, where the
goal is to perform instance discrimination, concentrating positive pairs and
separating negative pairs of feature embeddings in a latent space [19, 8, 36,
15]. Despite their convincing performance on downstream tasks, the majority of
current contrastive-based methods are pretrained on ImageNet, and subject to
strong object-centric bias and poor visual grounding [40, 34, 21, 38].

To this end, a number of emerging methods examine self-supervised
representaton learning on in-the-wild, scene image datasets such as COCO [40,
33, 45]. CAST improves visual grounding by ensuring crops overlap readily with
object regions identified by saliency masks, and guides representation
learning using a Grad-CAM loss [39, 40]. ORL uses a pretrained self-supervised
model to approximate object-level semantic correspondence, thus improving
positive-negative identification for contrastive refinement of the pretrained
model [45].

Going a step further, cyborgs and other works obtain object-level semantics
through pixel level pseudo-labeling [20, 41, 52, 1, 17]. For example, DetCon
[20] involves unsupervised preprocessing of images to obtain masks, and uses
these masks to aggregate features over object regions for contrastive
learning. Crucially, all other previous methods suffer from the disadvantage
that mask proposals are generated i) via graph-based algorithms requiring
heuristic hyperparameter decisions, and ii) only once before training, with no
further learning. In contrast, by integrating object mask proposals and
contrastive pretraining into the same loop, cyborgs iteratively refines and
improves both segmentation quality and learned representation quality,
jointly.

#### Unsupervised segmentation and clustering.

The use of clustering-based approaches in SSL has a long history [5, 6, 31].
DeepCluster is a seminal work which proposed to train a CNN by alternating
between feature clustering to obtain class pseudo-labels, and learning to
predict those very labels [5]. PCL and SwAV combine a clustering objective
with a contrastive objective, directly encoding semantic structure learned by
clustering into a latent representation space [6, 31]. Instead of directly
improving features by learning to cluster feature prototypes, cyborgs
primarily uses clustering as a mechanism to improve segmentation.

Indeed, clustering-based algorithms have recently found application in a
number of unsupervised and self-supervised image segmentation works [22, 50,
52, 11]. Both pixel and region-level contrastive learning methods have been
employed to i) improve semantic segmentation for better representation
learning [52, 46], and ii) vice versa [22, 26, 11]. To the best of our
knowledge, cyborgs is the first work to consider these two well-studied tasks
as complementary, iteratively synergizing them together via a bootstrapping
paradigm. Additionally, cyborgs does not directly optimize for segmentation
quality via pixel-level losses, and aims to improve segmentation strictly
insofar as it aids in representation learning.

##  3 CYBORGS

We now describe the details in our proposed framework. In Section 3.1, we
provide an overview of the abstractions in our work. At its core, we require
iteration between two components: a contrastive objective capable of
leveraging masks to train an encoder, and an unsupervised method to generate
masks from a (partially) trained encoder. In Section 3.2 and Section 3.3, we
describe the particular instantiations of these two components in our
demonstration of the framework. Finally, in Section 3.4, we show how to
construct a self-supervised consistency loss to guide mask generation.

###  3.1 CYBORGS Framework Abstraction

Following typical contrastive learning frameworks in vision, we begin with a
given RGB image $\mathbf{I}\in\mathbb{R}^{3\times H\times W}$ of height $H$
and width $W$, and two transformations $t,t^{\prime}$ independently sampled
from data augmentation pipelines $\mathcal{T},\mathcal{T^{\prime}}$. For the
time being, we assume we also have ground truth semantic segmentation masks
$\\{\mathbf{M}\\}\in[0,1]^{C\times H\times W}$. Each $H\times W$ binary mask
$\mathbf{M}$ describes pixel-wise class membership for a particular class, for
$C$ total classes. Applying the transformations to
$\mathbf{I},\\{\mathbf{M}\\}$ yields two augmented views
$\mathbf{v}=t(\mathbf{I}),\mathbf{v^{\prime}}=t^{\prime}(\mathbf{I})$, and two
semantic maps
$\\{\mathbf{m}\\}=t(\\{\mathbf{M}\\}),\\{\mathbf{m^{\prime}}\\}=t^{\prime}(\\{\mathbf{M}\\})$.
Note that every $\mathbf{m}$ contains object-level assignments spatially
aligned with view $\mathbf{v}$, and likewise every $\mathbf{m^{\prime}}$
aligns with $\mathbf{v^{\prime}}$. After passing view $\mathbf{v}$ to a
(fully) convolutional encoder $f_{\theta}$ for featurization, we can extract a
(sub)set of intermediate feature maps
$\\{\mathbf{F}\\}=\\{\mathbf{y}^{[1]},\dots,\mathbf{y}^{[l]}\\}$, where
$\mathbf{y}^{[l]}=f_{\theta}^{[l]}(\mathbf{v})$ for layer $l$. Doing the same
for view $\mathbf{v^{\prime}}$ yields feature maps
$\\{\mathbf{F^{\prime}}\\}$.

These feature maps inherently contain spatial and latent information about the
image, which we can leverage using the segmentation masks. The core idea is
conceptually simple and lightweight: we can sample arbitrary regions in the
feature maps and apply the binary masks
$\\{\mathbf{m}\\},\\{\mathbf{m^{\prime}}\\}$ to filter out groups of features
which correspond to the same underlying object regions. Applying mean pooling,
concatenation, or some other general aggregation operator to these groups
yields feature vectors containing similar and dissimilar object-level
semantics. These positive-negative pairs allow us to use a flexible class of
contrastive objectives to train our encoder $f_{\theta}$. Note that this
naturally requires upsampling or downsampling either the masks or the feature
maps to the same spatiality, and our framework is entirely agnostic to these
details. But a more immediate problem is obtaining reasonable masks
$\\{\mathbf{M}\\}$ to begin with.

Figure 2: CYBORGS Training Framework. We sample over the feature maps for
different views, using segmentation masks to identify similar and dissimilar
object regions. These are aggregated into positive and negative feature
vectors, respectively, for the contrastive objective $\mathcal{L}_{mask}$
(Section 3.2). Periodically, we also backprop through a clustering consistency
loss $\mathcal{L}_{clus}$ (Section 3.4).

A crucial assumption we have thus maintained is that ground truth segmentation
masks are available. Indeed, without specification of how object regions
correspond to each other across views, the very notion of positives and
negatives for a contrastive formulation becomes ill-defined. Previous works
which have relied on such masks in a similar fashion have used simple spatial
heuristics such as grid-based masks, or more complex unsupervised algorithms
such as graph cut segmentations [1, 52, 20]. Ultimately, we find that these
approaches yield unsatisfactory masks which are semantics-unaware, or require
significant hand-tuning, especially when employed on scene images. But
composing a learning-based procedure is non-trivial; the contrastive objective
cannot backpropagate through the non-differentiable augmentations
$t,t^{\prime}$ and modify a mask $\mathbf{m}$ directly.

To this end, our framework bootstraps segmentation masks using representations
from the partially trained model $f_{\theta}$. This idea is motivated by two
insights. Firstly, the contrastive objective directly improves the encoder
$f_{\theta}$, and thus leveraging the features from $f_{\theta}$ can help us
obtain semantic-aware masks which correspondingly improve over the course of
training. Secondly, recall that the ultimate goal of our framework is to
improve representation learning. Since downstream transfer of representations
takes places on $f_{\theta}$, using the representations from $f_{\theta}$ to
construct our segmentation masks ensures that representation quality and
bootstrapped mask quality are tied together. Implementation-wise, our
framework is agnostic to the actual algorithm employed for mask generation,
with the only constraint being that the method cannot rely on ground truth
supervision. For concreteness, we illustrate in Section 3.3 how to generate
robust masks using a simple KMeans clustering-based algorithm on the feature
maps $\\{\mathbf{F}\\}$. By iterating between contrastive updating of
$f_{\theta}$ and unsupervised generation of masks, we mutally improve our
representations and segmentations.

###  3.2 Mask-Dependent Contrastive Learning

To demonstrate the utility of our framework, we first choose the loss function
from [20] as the particular instantiation of a mask-based contrastive
objective for training our encoder in the first stage. We provide a high level
review here.

In [20], $\\{\mathbf{F}\\}$ is a single $2048\times 7\times 7$ feature map
extracted from the final layer of a standard ResNet-50 encoder processing view
$\mathbf{v}$ (before average pooling). The entire feature map is sampled, and
the segmentation masks in $\\{\mathbf{m}\\}$ are spatially downsampled
accordingly. Aggregation of $\\{\mathbf{F}\\}$ is obtained via mask-based
pooling for each $\mathbf{m}\in\\{\mathbf{m}\\}$:

$\mathbf{h_{m}}=\frac{1}{\sum_{i,j}\mathbf{m}[i,j]}\sum\limits_{i,j}\mathbf{m}[i,j]\mathbf{F}[i,j]$
(1)

Feature map $\\{\mathbf{F^{\prime}}\\}$ is similarly aggregated after
processing view $\mathbf{v^{\prime}}$ with a target encoder $f_{\xi}$,
yielding $\mathbf{h^{\prime}_{m^{\prime}}}$. For additional asymmetry,
$\mathbf{h_{m}}$ is further transformed by an online projector $g_{\theta}$
and predictor $q_{\theta}$ to obtain
$\mathbf{v_{m}}=q_{\theta}(g_{\theta}(\mathbf{h_{m}}))$, and
$\mathbf{h_{m^{\prime}}}$ is transformed by a target projector $g_{\xi}$ to
obtain $\mathbf{v^{\prime}_{m^{\prime}}}=g_{\xi}(h^{\prime}_{m^{\prime}})$.
The target parameters $\xi$ are updated as an exponential moving average (EMA)
of their online counterparts $\theta$. The final mask-based contrastive
objective is given by:

$\mathcal{L}_{contrastive}=\mathbb{E}_{\mathbf{m},\mathbf{m^{\prime}}\sim\\{\mathbf{m}\\},\\{\mathbf{m^{\prime}}\\}}\left[-\log\frac{\exp(\mathbf{v_{m}}\cdot\mathbf{v^{\prime}_{m^{\prime}}})}{\exp(\mathbf{v_{m}}\cdot\mathbf{v^{\prime}_{m^{\prime}}})+\sum_{n}\exp(\mathbf{v_{m}}\cdot\mathbf{v_{n}})}\right]$
(2)

for negative pooled features $\\{\mathbf{v}_{n}\\}$ sampled from different
masks and images.

In addition, inspired by prior art demonstrating that different layers within
a CNN encode information at different semantic resolutions [28, 18, 47], we
also extract and utilize features from throughout ResNet-50, instead of
relying solely on features from the final convolutional map as in [20]. By
fusing these features together spatially (after upsampling or downsampling),
downstream learning is able to leverage information across the semantic
spectrum, from low-level local structure, to high-level global style. Further
details are available in the appendix.

###  3.3 Bootstrapping Segmentation Masks

Recall that our framework is agnostic to the particular algorithm used in the
second stage bootstrapping of better segmentation masks. For simplicity, we
illustrate the details of this stage using a classic KMeans clustering
algorithm.

Figure 3: Bootstrapping Masks. To generate the segmentation masks, we perform
simple KMeans clustering on a feature map from the trained backbone, with a
dynamic number of clusters.

More formally, we begin by considering a batch of $B$ input RGB images
$\\{\mathbf{I}\\}\in\mathbb{R}^{B\times 3\times H\times W}$, and a (fully)
convolutional backbone $f_{\theta}$ which has been trained in a self-
supervised fashion via the objective in Eq. 2. We choose a particular layer
$\boldsymbol{\ell}$, and extract the feature map
$\mathbf{y}^{[\boldsymbol{\ell}]}_{\theta}=f_{\theta}^{[\boldsymbol{\ell}]}(\\{\mathbf{I}\\})\in\mathbb{R}^{B\times
D_{F}\times H_{F}\times W_{F}}$. We omit the layer index $\boldsymbol{\ell}$
and online encoder parameters $\theta$ for brevity, so that
$\mathbf{y}\triangleq\mathbf{y}^{[\boldsymbol{\ell}]}_{\theta}$. We then
flatten the feature maps and $\ell 2$-normalize feature-wise, generating a
matrix of features $\mathbf{F}\in\mathbb{R}^{(B\cdot H_{F}\cdot W_{F})\times
D_{F}}$. Given a hyperparameter $K$, representing the number of clusters (or
unique object classes) within the segmentation mask, we perform spherical
$K$-means clustering on $\mathbf{F}$, ending up with a matrix of feature
prototypes
$\mathbf{P}=\\{\mathbf{\mu}_{1},\mathbf{\mu}_{2},\dots,\mathbf{\mu}_{K}\\}\in\mathbb{R}^{D_{F}\times
K}$. We assign to each cell in the original feature map
$\mathbf{y}^{[\boldsymbol{\ell}]}$ a cluster label based on their Euclidean
distances to the prototypes in $\mathbf{P}$. Finally, we broadcast the class
assignments back to the original dimensions of the image $\mathbf{I}$ via
nearest neighbor interpolation, akin to [9].

#### Periodic bootstrapping.

Performing such a clustering operation on every epoch to regenerate the
segmentations can be expensive. Even if computation was not an issue, we
empirically find that representations do not improve monotonically with
epochs, so bootstrapping masks too frequently can actually lead to worse
masks. Moreover, as a result of an undertrained encoder $f_{\theta}$ at the
beginning of training, we obtain poorer early clusterings; noisy masks lead to
noisy gradients for updating the encoder, and vice versa. Thus, to avoid
representation collapse, we periodically bootstrap the segmentation masks
every $N$ epochs, where $N$ is a hyperparameter much greater than 1.

#### Scale-dynamic sampling.

The choice of $K$ also merits discussion. Given access to some oracle, a
natural choice might be to set $K$ equal to the number of unique object
classes within the image. However, as a number of prior works have identified,
the semantic context provided by extra “distractor” classes outside of the
main object classes can serve as a useful signal for clustering [24, 5, 9].
But increasing $K$ also requires more images within the bootstrapping batch to
perform KMeans reliably on the features, reducing the scalability of our
method.

To balance these motivations, for every batch of images where we wish to
bootstrap segmentations, we dynamically sample integer $K$ uniformly between
$K_{min}=2$ and $K_{max}=256$, inclusive. Intuitively, $K_{min}=2$ represents
a mask which imparts the model with simple foreground-background semantics,
while the upper bound of $K_{max}=256$ yields an oversegmentation (COCO offers
only 81 labeled object segmentation classes.) By varying $K$ in such a
fashion, not only do we maintain efficiency in bootstrapping, but we also
reintroduce our model to information of varying semantic scale on every
bootstrap cycle. As we show in Section 4.4, this technique improves the
robustness of our representations.

###  3.4 Consistency as a Curriculum for Segmentation

Despite the use of periodic bootstrapping and scale-dynamic sampling, we find
that the long training schedules employed in contrastive learning can still
lead to divergence between our representation learning and semantic
segmentation objectives. This is because our framework up to now improves the
segmentation only _implicitly_. While we are optimizing on every iteration our
contrastive objective in Eq. 2, regularly improving our encoded
representations, the bootstrapping of masks is optimization-free with respect
to the encoder. Without an update signal to explicitly encode the semantics of
desirable vs. non-desirable segmentations, the encoder over-prioritizes the
goal of representation learning, and can diverge from a feature distribution
which yields good segmentations.

#### Clustering consistency.

To this end, we reuse a universal paradigm in contrastive learning: similar
objects across different scenes and different views should have similar
labels. We introduce a clustering consistency loss, similar to that employed
in [11], which can be applied more regularly every $M$ epochs, where $M$ is
more frequent than the every $N$ epochs used per bootstrapping cycle.

Concretely, recall the feature map $\mathbf{y}\in\mathbb{R}^{D_{F}\times
H_{F}\times W_{F}}$ and feature prototypes
$\mathbf{P}=\\{\mathbf{\mu}_{1},\mathbf{\mu}_{2},\dots,\mathbf{\mu}_{K}\\}\in\mathbb{R}^{D_{F}\times
K}$ we obtained in Section 3.3 after processing view $\mathbf{v}$ using the
online encoder $f_{\theta}$. We obtain a similar map $\mathbf{y}^{\prime}$ and
set of prototypes
$\mathbf{P^{\prime}}=\\{\mathbf{\mu^{\prime}}_{1},\mathbf{\mu^{\prime}}_{2},\dots,\mathbf{\mu^{\prime}}_{K}\\}$
after featurizing $\mathbf{v}^{\prime}$ with the target encoder $f_{\xi}$.
Consider the feature at pixel $[i,j]$ within $\mathbf{y}$, for an arbitrary
$1\leq i\leq H_{F}$ and $1\leq j\leq W_{F}$. With a slight abuse of notation,
we let $\mathbf{\mu}_{[i,j]}$ represent the prototype this feature is assigned
to under $\mathbf{P}$ (and similarly, $\mathbf{\mu^{\prime}}_{[i,j]}$ the
assignment of $\mathbf{y^{\prime}}[i,j]$ under $\mathbf{P^{\prime}}$). Then we
define a clustering consistency loss via:

$\mathcal{L}_{clus}=\frac{1}{H_{F}W_{F}}\sum\limits_{i=1}^{H_{F}}\sum\limits_{j=1}^{W_{F}}\overbrace{d\left(\mathbf{\mu}_{[i,j]},\mathbf{y}_{[i,j]}\right)+d\left(\mathbf{\mu^{\prime}}_{[i,j]},\mathbf{y^{\prime}}_{[i,j]}\right)}^{intra-
loss}+\overbrace{d\left(\mathbf{\mu^{\prime}}_{[i,j]},\mathbf{y}_{[i,j]}\right)+d\left(\mathbf{\mu}_{[i,j]},\mathbf{y^{\prime}}_{[i,j]}\right)}^{inter-
loss}$

(3)

where $d(\cdot,\cdot)$ is some distance function. Intuitively, intra-cluster
consistency enforces that under one scene, object regions with similar
features should be clustered into similar prototypes. Similarly, inter-cluster
consistency enforces that under different scenes, we still wish for features
from different regions corresponding to similar objects to be assigned to the
same prototype. This forces our learned prototypes to be invariant to
differences between views and generalize to object-centric semantics, which
translates readily to higher fidelity segmentation masks during bootstrapping
updates.

To formulate $d(\cdot,\cdot)$, we draw inspiration from recent work which
demonstrates that the infoNCE objective in contrastive learning promotes a
feature space which is uniformly distributed on the unit hypersphere [42]. The
von Mises-Fisher (vMF) distribution defines a probability density over a unit
hypersphere, making it a natural candidate to characterize the feature space
learned by our mask-based contrastive objective in Eq. 2. We refer readers to
a comprehensive treatment in [14] for details. In our setting, we can assume a
vMF mixture model where each feature $\mathbf{y}$ is drawn uniformly from one
of $K$ vMF distributions, each parameterized by a feature clustering prototype
$\mathbf{\mu}_{1},\mathbf{\mu}_{2},\dots,\mathbf{\mu}_{K}$, and sharing a
common concentration hyperparameter $\kappa$. Then our clustering consistency
loss objective is formulated as maximizing the posterior likelihood of a
particular encoded feature $\mathbf{y}$ being assigned to its corresponding
cluster $c$ under this mixture, with $1\leq c\leq K$. That is, we seek to
minimize the negative log-likelihood given by:

$d(\mathbf{\mu}_{[i,j]},\mathbf{y})=-\log
p(\mathbf{\mu}_{[i,j]=c}\mid\mathbf{y},\mathbf{\mu}_{1},\mathbf{\mu}_{2},\dots,\mathbf{\mu}_{K})=-\log\frac{\exp\left(\kappa\mathbf{\mu}_{[i,j]}^{T}\mathbf{y}\right)}{\sum\limits_{c^{\prime}=1}^{K}\exp\left(\kappa\mathbf{\mu}_{c^{\prime}}^{T}\mathbf{y}\right)}$
(4)

The vMF clustering loss objective described in Eq. 3 also serves an additional
purpose towards the beginning of our pretraining pipeline. In the total
absence of reliable masks before the first bootstrapping cycle, we train our
encoder $f_{\theta}$ strictly with the loss in Eq. 3, setting $K$ to a fixed
parameter depending on the median number of objects per scene in our dataset
(_e.g._ , for COCO, we use $K=8$). This _vMF warmup period_ of $W$ epochs
($W=5$ in our work) serves to burn in our encoder. After more reasonable
representations have been learned, we immediately bootstrap the masks, and
subsequent epochs using a combination of the mask-based contrastive loss in
Eq. 2 and the vMF clustering loss in Eq. 3, as their respective periods $N$
and $M$ dictate. For a comprehensive outlining of our algorithm flow, we refer
readers to the pseudocode presented in the appendix.

##  4 Experiments

In our experiments, we aim to demonstrate that joint learning of general
representations and semantic segmentation can be successfully accomplished via
our bootstrapping method. We show strong performance on multiple downstream
tasks (Section 4.2), surprisingly robust segmentation performance over a long-
tailed distribution of objects (Section 4.3), and a convincing array of
ablations which validate our design choices and methodological contributions
(Section 4.4).

###  4.1 Experimental Settings

#### Datasets.

Given our primary goal of learning on images in the wild, we follow previous
works [40, 43, 33] and pretrain on the train2017 split of the MS COCO dataset
[32]. With $\sim$118k images of natural settings, MS COCO is widely adopted as
a benchmark more reflective of real-world scenarios across a breadth of
downstream tasks of interest, such as object detection or instance
segmentation. For a relevant quantitative comparison, note that the heavily
object-dominant ImageNet dataset contains on average 1.1 objects per image,
whereas the average scene image in COCO contains 7.3 objects [43]. Crucially,
we use no scene-level, object-level, or pixel-level label information in our
pretraining pipeline.

#### Implementation details.

To enable easy comparison to other SSL works in similar settings [40, 45, 33,
43], we use a ResNet-50 backbone in all of our models. Other architectural
details such as the dimensionality of projection and prediction MLPs described
in Section 3.2 follow directly from BYOL [15].

For our mask-based contrastive objective in Eq. 2, we aggregate features from
$\texttt{res2},\texttt{res3},\texttt{res4}$, downsampling all layers to a
spatial resolution of $7\times 7$. This allows us to leverage a lightweight
but comprehensive semantic hierarchy. We bootstrap the segmentation masks
every $N=100$ epochs, performing clustering on the feature map from res2.b2 in
batches of 16 images (where b2 refers to block 2). We use a vMF warmup period
of 5 epochs; outside the warmup period, the vMF clustering loss is employed
every 5 epochs with weight $\lambda=0.1$ and $\kappa=10$. In pretraining, we
use the LARS optimizer [49] with a batch size of 64 across 8 NVIDIA RTX 3090s
for 800 epochs. The initial learning rate is set to 0.1, and the weight decay
is $1.5e^{-6}$. Clustering is implemented via GPU-accelerated mini-batch
approximation using the FAISS library [25].

###  4.2 Main Results: Representation Learning

| Method |  | VOC07 clf. |  |  IN-1k, 1% Labels |  |  IN-1k, 10% Labels |  |  VOC Detection |  |  COCO Instance Segmentation
---|---|---|---|---|---|---|---|---|---|---|---
|  | mAP |  |  Top-1 acc. |  Top-5 acc. |  |  Top-1 acc. |  Top-5 acc. |  |  AP${}^{\text{bb}}_{\text{}}$ |  AP${}^{\text{bb}}_{\text{50}}$ |  AP${}^{\text{bb}}_{\text{75}}$ |  |  AP${}^{\text{bb}}_{\text{}}$ |  AP${}^{\text{bb}}_{\text{50}}$ |  AP${}^{\text{bb}}_{\text{75}}$ |  AP${}^{\text{mk}}_{\text{}}$ |  AP${}^{\text{mk}}_{\text{50}}$ |  AP${}^{\text{mk}}_{\text{75}}$
1) |  simclr [8] |  | 78.1 |  | 23.4 | 46.4 |  | 52.2 | 77.4 |  | – | – | – |  | 37.0 | 56.8 | 40.3 | 33.7 | 53.8 | 36.1
2) |  moco-v2 [10] |  | 82.2 |  | 28.2 | 54.7 |  | 57.1 | 81.7 |  | 54.7 | 81.0 | 60.6 |  | 38.5 | 58.1 | 42.1 | 34.8 | 55.3 | 37.3
3) |  byol [15] |  | 84.5 |  | 28.4 | 55.9 |  | 58.4 | 82.7 |  | 55.5 | 81.7 | 61.7 |  | 39.5 | 59.3 | 43.2 | 35.6 | 56.5 | 38.2
4) |  bai et al. [1] |  | – |  | – | – |  | – | – |  | 57.1 | 82.1 | 63.8 |  | 39.8 | 59.6 | 43.7 | 35.9 | 56.9 | 38.6
5) |  densecl [43] |  | 83.8 |  | – | – |  | – | – |  | 56.7 | 81.7 | 63.0 |  | 39.6 | 59.3 | 43.3 | 35.7 | 56.5 | 38.4
6) |  cast [40] |  | 73.1 |  | – | – |  | – | – |  | 54.2 | 80.1 | 59.9 |  | 36.7 | 56.7 | 39.9 | 33.6 | 53.6 | 35.8
7) |  orl [45] |  | 86.7 |  | 31.0 | 58.9 |  | 60.5 | 84.2 |  | 55.8 | 82.1 | 62.3 |  | 40.3 | 60.2 | 44.4 | 36.3 | 57.3 | 38.9
8) |  cyborgs(ours) |  | 86.9 |  | 31.3 | 59.4 |  | 61.7 | 84.2 |  | 58.0 | 83.0 | 64.3 |  | 42.0 | 62.6 | 46.2 | 38.0 | 59.7 | 40.8

Table 1: Transfer Learning on Downstream Tasks. We report strong, state-of-
the-art performance across linear classification on VOC07, semi-supervised
finetuning on ImageNet-1k, and transfer on VOC object detection and COCO
instance segmentation. All methods are pretrained on COCO with a ResNet-50
backbone, and finetuned on the reported datasets.

We follow standard downstream transfer-based protocols to evaluate the
strength of representations learned by cyborgs. In particular, we begin with
_frozen_ linear evaluation on VOC07 and semi-supervised transfer on
ImageNet-1k. In comparison to similar state-of-the-art self-supervised methods
pretrained on COCO, we achieve improvements of $+0.2$ mAP for VOC07 and
$+0.3\%,+1.2\%$ in top-1 accuracy for semi-supervised 1% and 10% on IN-1k,
respectively. While these gains are only incremental, image classification
requires semantic-level knowledge [38, 48, 2], whereas we design our method
around leveraging pixel-level information, and so even marginal gains are a
surprising windfall.

While linear probing has been treated as the gold standard for assessing
feature quality, that strong performance in tasks such as detection and
segmentation are even more reflective of potent learned representations. We
demonstrate convincing state-of-the-art on PASCAL VOC detection and COCO
instance segmentation. In comparison to a strong and well-established BYOL
baseline, we provide a +2.5 AP improvement on the former, and a +2.5 and +2.4
AP improvement on the latter. Our results on segmentation in particular are
noteworthy; while we do not make use of pixel annotations, our bootstrapping
scheme clearly aligns with latent information critical to the segmentation
task. To further verify this robust segmentation performance, we also perform
transfer-based evaluation on CityScapes semantic segmentation [12], as well as
LVIS long-tailed instance segmentation [16], which we detail in the appendix.
We achieve state-of-the-art on these datasets amongst all other previous SSL
methods pretrained on COCO, with a substantial +3.4 AP and +3.3 AP improvement
on LVIS, demonstrating that our framework can also generalize to unseen object
structures and semantics.

###  4.3 Segmentation Quality

We first confirm qualitatively (Fig. 4) that bootstrapped masks generated by
cyborgs are indeed semantically meaningful. Note that our clustering-based
segmentations easily extend beyond the original labeled classes of COCO,
despite receiving no ground truth information about pixel labels throughout
pretraining.

|  |  |
---|---|---|---
|  |  |
|  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(a) Raw RGB

(b) KMeans Mask

(c) CRF-Refined Mask

(d) Ground Truth

Figure 4: Bootstrapped segmentation masks from a CYBORGS-pretrained encoder on
COCO. We show KMeans segmentations on the bilinearly upsampled feature maps
for visual quality. During actual bootstrapping, we first segment the feature
map, before performing nearest neighbors upsampling, and do not perform CRF
refinement on the mask. Colors do not necessarily correspond across images
(rows) or between mask types (columns), but are consistent within a single
image itself.

#### How does cyborgs work with such noisy masks?

In addition to the masks generated by clustering on the feature maps from the
backbone encoder, we also show the mask resulting from refinement using a
fully connected conditional random field (CRF), using the distances to feature
prototypes in latent space as priors, following the protocol described in
previous works [29, 7]. We argue that although the raw masks at a pixel-level
appear to be noisy, their easy refinement into masks closely aligned with
ground truth masks indicates that the encoded features are quite well aligned
with object-level concepts at the semantic level.

Type of Mask mIoU Random Crop 15.9 5x5 Grid 18.7 FH Masks 27.7 Obj. Bounding
Boxes 29.0 cyborgs Masks (ours) 33.6 Ground Truth Masks 35.2

(a) Importance of bootstrapping.

(b) Masks (blue) and representations (red) improve jointly.

Figure 5: Using semantic segmentation performance on COCO-Stuff-10k to
evaluate bootstrapping value. (a) Replacing our bootstrapping segmentation
core with static boxes from other unsupervised heuristics leads to decreased
performance. (b) Note a single epoch of using improved masks can lead to
significant gains (epoch 100).

#### Why bootstrap masks?

To further demonstrate the robustness of our bootstrapping process for mask
generation, we retrain cyborgs using alternative masks. Instead of
bootstrapping masks, we employ random cropping masks (i.e. all pixels in the
scene belong to the same class), a $5\times 5$ spatial grid mask and
Felzenszwalb-Huttenlocher (FH) masks used in [20, 1], detection-level object
masks acquired via selective search pre-processing, and ground truth masks
available in COCO. These masks are generated before pretraining and remain
fixed, supplanting our bootstrapping algorithm. Given the unsupervised
generalization of masks generated under our framework to a long-tailed
distribution of objects (c.f. Fig. 4), we evaluate the representations by
transferring the trained backbones to a ResNet-50 FCN and finetuning end-to-
end on COCO-Stuff-10k semantic segmentation. COCO-Stuff-10k is a _densely_
labeled subset of COCO, comprising of 9k images for training and 1k images for
testing, across 171 semantic categories [4]. We verify in Fig. 5a that
bootstrapping mask-level information through cyborgs outperforms detection-
level boxes obtained from selective search, and nears performance of
pretraining with fixed, stable ground truth masks.

#### Joint improvement of masks and representations.

The harmonious interplay between the representation learning and semantic
segmentation components of our framework is one of our major contributions. To
ascertain that representations and segmentation quality mutually improve over
pretraining, we continue to assess semantic segmentation performance on the
COCO-Stuff-10k dataset, for saved checkpoints throughout various stages of
pretraining. For a batch of input images, we extract frozen feature maps from
the same layer we use to bootstrap segmentation masks (res2.b2), and
bilinearly interpolate to the original image dimensions. We then add a single
layer of 1x1 convolutions to predict the pixel labels, yielding a final setup
akin to linear probing in transfer-based evaluation.

Because only this last layer is trainable in the resulting model, segmentation
performance is heavily dependent on the quality of the extracted feature maps.
Since these are exactly the inputs to our KMeans segmentation algorithm, we
obtain transfer results which correlate readily with the quality of our
bootstrapped masks. To evaluate representation quality in the same pretrained
models, we transfer the ResNet-50 backbones, unfreeze all layers, and add an
FCN head, finetuning on COCO-Stuff-10k end-to-end. We perform these
evaluations for cyborgs models pretrained for 5, 25, 50, 100, 200, 500 and 800
epochs. As seen in Fig. 5b, this evaluation scheme demonstrates that mask
quality and representation quality improve jointly over the course of
pretraining. Note that we bootstrap masks for the first time at the
_beginning_ of epoch 100 using our partially pretrained backbone; a subsequent
iteration over the entire dataset is sufficient to improve both mask and
representation semantics significantly.

###  4.4 Ablations and Discussion

All ablation models are pretrained using a ResNet-50 backbone, and evaluations
are performed on PASCAL VOC detection for faster turnaround time.

N |  AP${}_{\text{\tiny all}}$
---|---
0 |  9.75
10 |  52.6
50 |  58.3
100 |  58.0
200 |  55.0
400 |  52.7
(a)  Bootstrapping frequency. Mask bootstrapping too often or not enough leads
to poor performance.

Case |  AP${}_{\text{\tiny all}}$
---|---
CRF |  58.6
No CRF |  58.0
|
|
|
|
(b)  CRF in bootstrapping. Refining bootstrapped masks with CRFs during
pretraining is not necessary.

Layers |  AP${}_{\text{\tiny all}}$
---|---
2 |  55.1
3 |  55.8
4 |  52.7
2+4 |  56.4
2+3+4 |  58.0
|
(c)  Features in contrastive objective. Leveraging a semantic hierarchy of
features is important.

Layers |  AP${}_{\text{\tiny all}}$
---|---
2.b2 |  58.0
2+3 |  58.2
2+4 |  54.4
2+3+4 |  55.0
|
|
(d)  Features in KMeans. Earlier maps are more amenable to KMeans
segmentation.

K |  AP${}_{\text{\tiny all}}$
---|---
$K=2$ |  22.6
$K=81$ |  42.5
$K=256$ |  33.2
$K\sim\mathcal{U}[2,256]$ |  58.0
|
(e)  Scale-dynamic sampling. Dynamically sampling cluster resolution for
KMeans segmentation works best.

Loss |  AP${}_{\text{\tiny all}}$
---|---
Euclidean |  53.8
vMF |  58.0
|
|
|
(f)  Clustering loss. Euclidean distance for Eq. 4 leads to significant
performance degradation.

M |  AP${}_{\text{\tiny all}}$
---|---
0 |  41.8
1 |  58.6
5 |  58.0
10 |  57.8
50 |  55.6
(g)  vMF Loss Frequency. Applying the vMF curriculum more regularly leads to
stronger performance.

$\lambda$ |  AP${}_{\text{\tiny all}}$
---|---
0 |  41.5
0.001 |  42.3
0.01 |  57.8
0.1 |  58.0
1 |  57.2
(h)  vMF Loss Weight. Performance is sensitive to the presence but not weight
of the vMF loss.

Table 2: Ablations for design choices in cyborgs. We report average precision
(AP) for object detection on PASCAL VOC test2007. Default settings corresp. to
Table 1 are highlighted in gray.

#### Bootstrapping frequency.

We perform a sensitivity analysis on the bootstrapping frequency parameter
$N$, where we regenerate the masks on epoch $N,\,\,2N,\dots$, using feature
maps from the improving encoder (Table 2a). Using only the initial masks
obtained under vMF warmup for pretraining (_i.e._ , $N=0$) leads to collapsed
performance. Moreover, bootstrapping the masks too frequently ($N$ = 10) also
leads to a performance drop, consistent with our hypothesis in Section 3.3
that unstable masks which are changing too rapidly can lead to
representational collapse. Finally, we also note that bootstrapping too
_infrequently_ (_i.e._ , $N=400$) is similarly suboptimal, validating our
default chosen schedule.

#### CRF-refinement of masks.

Given the qualitative improvements of the CRF-refined masks when performing
the final evaluation (c.f. Fig. 4), a natural consideration is to apply CRF
post-processing to the masks during every bootstrapping cycle. As we show in
Table 2b, this brings only incremental improvements to the resulting
representations, which we believe do not justify the increase in computational
complexity. This result also further validates our claim in Section 4.3 that
the representations under cyborgs are already well-aligned in latent space
with respect to the semantic segmentation task.

#### Usage of multiple layers of features.

Throughout our method, there are two points where we potentially use feature
maps from multiple layers of our encoder backbone. The first is in the
aggregation of features for our contrastive objective in Eq. 2. We show in
Table 2c that using $\texttt{res2},\texttt{res3},\texttt{res4}$ from our
backbone in combination is crucial to performance. This further verifies that
leveraging information from across the semantic spectrum learned by the
encoder is vital.

The second point is in the bootstrapping of masks, where we use only the
feature map from res2.b2 of our backbone. We show in Table 2d that feature
aggregation across multiple layers does not help here. One explanation for
this phenomenon is curse of dimensionality; a simple KMeans clustering
procedure on extremely high dimensional features aggregated across multiple
layers may result in clusters with few or no points.

#### Scale-dynamic sampling.

We also perform an ablation on dynamically sampling the semantic resolution of
masks during bootstrapping. We compare with a foreground-background masks
($K=2$), object-level masks ($K=81$ categories from COCO), and clustering with
the same number of unique labels as the default graph-based segmentation
algorithm used in [20]. As shown in Table 2e, fixing the KMeans cluster
dimension at any level reduces the performance of cyborgs. This validates our
choice to provide the encoder with diverse levels of detail through
bootstrapped masks of varying semantic resolution.

#### vMF clustering loss.

We verify several properties of the clustering loss in Eq. 4. As seen in Table
2f, basing the loss on the vMF distribution, which aligns better with our
hyperspherically-distributed embeddings, results in better transfer
performance. We also examine the frequency at which the vMF clustering loss is
applied, and how sensitive our method is to the weight of this loss. Table 2g
validates that applying the loss more frequently increases downstream transfer
performance. In combination with Table 2h we note the weight of the loss does
not dramatically influence performance, but its presence is important; at
$\lambda=0,0.001$ or if $M=0$ (_i.e._ , no application of vMF loss), the
performance collapses.

##  5 Conclusion

We have proposed cyborgs, a novel self-supervised framework which learns
object-level representations and semantic segmentation jointly, in an end-to-
end fashion. In pretraining on complex scene images, our representations
transfer competitively to a diverse array of downstream tasks, with
particularly strong alignment with a long-tailed distribution of object-level
segmentation semantics.

Acknowledgements: YG is supported by the Ministry of Science and Technology of
the People’s Republic of China, the 2030 Innovation Megaprojects “Program on
New Generation Artificial Intelligence” (Grant No. 2021AAA0150000). YG is also
supported by a grant from the Guoqiang Institute, Tsinghua University. RW
would like to thank Yu Sun and Yingdong Hu for valuable edits to the paper,
without which this work would not be possible.

## References

  * [1] Bai, Y., Chen, X., Kirillov, A., Yuille, A., Berg, A.C.: Point-level region contrast for object detection pre-training. arXiv preprint arXiv:2202.04639 (2022)
  * [2] Ballard, D.H., Zhang, R.: The hierarchical evolution in human vision modeling. Topics in Cognitive Science 13(2), 309–328 (2021)
  * [3] Bar, A., Wang, X., Kantorov, V., Reed, C.J., Herzig, R., Chechik, G., Rohrbach, A., Darrell, T., Globerson, A.: Detreg: Unsupervised pretraining with region priors for object detection. arXiv preprint arXiv:2106.04550 (2021)
  * [4] Caesar, H., Uijlings, J., Ferrari, V.: Coco-stuff: Thing and stuff classes in context. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1209–1218 (2018)
  * [5] Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 132–149 (2018)
  * [6] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 9912–9924. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper/2020/file/70feb62b69f16e0238f741fab228fec2-Paper.pdf
  * [7] Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)
  * [8] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
  * [9] Chen, T., Luo, C., Li, L.: Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems 34 (2021)
  * [10] Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 (2020)
  * [11] Cho, J.H., Mall, U., Bala, K., Hariharan, B.: Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16794–16804 (2021)
  * [12] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3213–3223 (2016)
  * [13] Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: Proceedings of the IEEE international conference on computer vision. pp. 1422–1430 (2015)
  * [14] Gopal, S., Yang, Y.: Von mises-fisher clustering models. In: International Conference on Machine Learning. pp. 154–162. PMLR (2014)
  * [15] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., Valko, M.: Bootstrap your own latent - a new approach to self-supervised learning. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 21271–21284. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper/2020/file/f3ada80d5c4ee70142b17b8192b2958e-Paper.pdf
  * [16] Gupta, A., Dollar, P., Girshick, R.: Lvis: A dataset for large vocabulary instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5356–5364 (2019)
  * [17] Hamilton, M., Zhang, Z., Hariharan, B., Snavely, N., Freeman, W.T.: Unsupervised semantic segmentation by distilling feature correspondences. In: International Conference on Learning Representations (2021)
  * [18] Hariharan, B., Arbeláez, P., Girshick, R., Malik, J.: Hypercolumns for object segmentation and fine-grained localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 447–456 (2015)
  * [19] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9729–9738 (2020)
  * [20] Hénaff, O.J., Koppula, S., Alayrac, J.B., Oord, A.v.d., Vinyals, O., Carreira, J.: Efficient Visual Pretraining with Contrastive Detection. International Conference on Computer Vision (2021)
  * [21] Herranz, L., Jiang, S., Li, X.: Scene recognition with cnns: objects, scales and dataset bias. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 571–579 (2016)
  * [22] Hwang, J.J., Yu, S.X., Shi, J., Collins, M.D., Yang, T.J., Zhang, X., Chen, L.C.: Segsort: Segmentation by discriminative sorting of segments. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7334–7344 (2019)
  * [23] Jabri, A., Owens, A., Efros, A.: Space-time correspondence as a contrastive random walk. Advances in neural information processing systems 33, 19545–19560 (2020)
  * [24] Ji, X., Henriques, J.F., Vedaldi, A.: Invariant information clustering for unsupervised image classification and segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9865–9874 (2019)
  * [25] Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with gpus. IEEE Transactions on Big Data (2019)
  * [26] Ke, T.W., Hwang, J.J., Yu, S.X.: Universal weakly supervised segmentation by pixel-to-segment contrastive learning. In: International Conference on Learning Representations (2021)
  * [27] Komodakis, N., Gidaris, S.: Unsupervised representation learning by predicting image rotations. In: International Conference on Learning Representations (ICLR) (2018)
  * [28] Kornblith, S., Shlens, J., Le, Q.V.: Do better imagenet models transfer better? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2661–2671 (2019)
  * [29] Krähenbühl, P., Koltun, V.: Efficient inference in fully connected crfs with gaussian edge potentials. Advances in neural information processing systems 24 (2011)
  * [30] Kuang, H., Zhu, Y., Zhang, Z., Li, X., Tighe, J., Schwertfeger, S., Stachniss, C., Li, M.: Video contrastive learning with global context. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3195–3204 (2021)
  * [31] Li, J., Zhou, P., Xiong, C., Hoi, S.C.: Prototypical contrastive learning of unsupervised representations. In: ICLR (2021)
  * [32] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
  * [33] Liu, S., Li, Z., Sun, J.: Self-emd: Self-supervised object detection without imagenet. arXiv preprint arXiv:2011.13677 (2020)
  * [34] Mo, S., Kang, H., Sohn, K., Li, C.L., Shin, J.: Object-aware contrastive learning for debiased scene representation. Advances in Neural Information Processing Systems 34 (2021)
  * [35] Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: European conference on computer vision. pp. 69–84. Springer (2016)
  * [36] Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
  * [37] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2536–2544 (2016)
  * [38] Purushwalkam, S., Gupta, A.: Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. Advances in Neural Information Processing Systems 33, 3407–3418 (2020)
  * [39] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
  * [40] Selvaraju, R.R., Desai, K., Johnson, J., Naik, N.: Casting your model: Learning to localize improves self-supervised representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11058–11067 (2021)
  * [41] Van Gansbeke, W., Vandenhende, S., Georgoulis, S., Van Gool, L.: Unsupervised semantic segmentation by contrasting object mask proposals. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 10052–10062 (October 2021)
  * [42] Wang, T., Isola, P.: Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In: International Conference on Machine Learning. pp. 9929–9939. PMLR (2020)
  * [43] Wang, X., Zhang, R., Shen, C., Kong, T., Li, L.: Dense contrastive learning for self-supervised visual pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3024–3033 (2021)
  * [44] Xiao, T., Reed, C.J., Wang, X., Keutzer, K., Darrell, T.: Region similarity representation learning. arXiv preprint arXiv:2103.12902 (2021)
  * [45] Xie, J., Zhan, X., Liu, Z., Ong, Y.S., Loy, C.C.: Unsupervised object-level representation learning from scene images. arXiv preprint arXiv:2106.11952 (2021)
  * [46] Xiong, Y., Ren, M., Zeng, W., Urtasun, R.: Self-supervised representation learning from flow equivariance. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 10191–10200 (October 2021)
  * [47] Xu, J., Wang, X.: Rethinking self-supervised correspondence learning: A video frame-level similarity perspective. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 10075–10085 (October 2021)
  * [48] Yang, C., Wu, Z., Zhou, B., Lin, S.: Instance localization for self-supervised detection pretraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3987–3996 (2021)
  * [49] You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., Hsieh, C.J.: Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962 (2019)
  * [50] Zhang, F., Torr, P., Ranftl, R., Richter, S.: Looking beyond single images for contrastive semantic segmentation learning. Advances in Neural Information Processing Systems 34 (2021)
  * [51] Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: European conference on computer vision. pp. 649–666. Springer (2016)
  * [52] Zhang, X., Maire, M.: Self-supervised visual representation learning from hierarchical grouping. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 16579–16590. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper/2020/file/c1502ae5a4d514baec129f72948c266e-Paper.pdf


# UPDATED MEASUREMENT OF $R_{K}$ AT LHCb

D. LANCIERINI  Physik Institut, Universität Zürich,
Winterthurerstrasse 190, 8057, Zürich

###### Abstract

In the Standard Model of particle physics, charged leptons of different
flavour couple to the electroweak force carriers with the same interaction
strength. This property, known as lepton flavour universality, was found to be
consistent with experimental evidence in a wide range of particle decays.
Lepton flavour universality can be tested by comparing branching fractions in
ratios such as ${R_{K}}=\mathcal{B}(\mbox{$B^{+}\rightarrow
K^{+}\mu^{+}\mu^{-}$})/\mathcal{B}(\mbox{$B^{+}\rightarrow
K^{+}e^{+}e^{-}$})$. This observable is measured using proton-proton collision
data recorded with the LHCb detector at CERN’s Large Hadron Collider
corresponding to an integrated luminosity of 9$\rm{\leavevmode\nobreak\
fb}^{-1}$. For a dilepton invariant mass range of
$q^{2}\in[1.1,6.0]\text{\,Ge\kern-1.00006ptV}^{2}$, the measured value of
${R_{K}}=0.846\,^{+\,0.042}_{-\,0.039}\,{}^{+\,0.013}_{-\,0.012}$, where the
first uncertainty is statistic and the second systematic, is in tension with
the Standard Model predicted value at the ${3.1}\sigma$ level raising evidence
for lepton flavour universality violation in $B^{+}\rightarrow
K^{+}\ell^{+}\ell^{-}$ decays.

##  1 Introduction

The Standard Model (SM) of particle physics represents our best understanding
of the properties and interactions of fundamental particles. It received
thorough experimental confirmation since it’s formulation in the 1960’s.
However, it is not a complete theory of fundamental interactions as it is
unable to include a description of gravity and leaves some observed phenomena
unexplained, such as the matter-antimatter asymmetry or the apparent dark-
matter content of the Universe. It also does not incorporate neutrino
oscillations and their non-zero masses. Extensions of the SM, referred to as
”new physics” (NP), are formulated to explain these phenomena and they predict
the existence of new particles and interactions whose signatures can be
observed in direct or indirect searches at particle accelerators. Natural
units where $\hbar=c=1$ as well as charge conjugation are implied throughout
this proceedings.

###  1.1 Rare beauty-quark decays

Signatures of NP can be searched for by comparing decay rates of hadrons,
where hadrons are bound states of quarks, with the corresponding SM
predictions. Among the wide range of decay processes the SM is able to provide
a precise prediction of, transitions where a $B^{+}$ hadron decays into a kaon
$K^{+}$, and two charged leptons $\ell^{+}\ell^{-}$ are of great interest.
Since at the quark level they involve a $b\to s\ell^{+}\ell^{-}$ transition,
these processes are mediated by flavour-changing neutral currents, which in
the SM only occur via electroweak loops (as can be seen in the left-hand side
of Fig. 1 and are thus greatly suppressed [1]. However, virtual contributions
from new particles, an example represented on the right hand side of Fig. 1,
could compete with amplitudes of the same strength as the SM, resulting in a
modification of the observed branching fraction with respect to the SM
prediction. Rare $B^{+}$ meson decays are thus sensitive to virtual
contributions from new particles which could have masses that are inaccessible
to direct searches for resonances, even at Large Hadron Collider beam
energies.

Figure 1: Feynman diagrams mediated by left (right) SM (NP) processes. While
in the SM contribution to $b\to s\ell^{+}\ell^{-}$ transitions only occurs at
loop level, since it’s a flavour-changing neutral current, possible NP models
involving a leptoquark [2] exchange would allow this process to occur at tree
level.

###  1.2 Neutral current anomalies

Since 2012, several measurements of observables involving $b\to
s\ell^{+}\ell^{-}$ transitions have shown deviations from SM predictions.
While there is no single result exhibiting a $5\sigma$ deviation from the SM,
the pattern of deviations, collectively denoted as ”neutral current
anomalies”, is striking. In fact, the latest measurements published by LHCb
[4]${}^{,}\,$[5] all point to a smaller branching fractions for muons in the
intermediate $q^{2}$ region, where $q^{2}$ is the di-lepton invariant mass
squared. Precise SM predictions of $b\to s\ell^{+}\ell^{-}$ processes require
good control over non-local QCD effects, such as charmloop contributions,
which are known to suffer from potentially large theoretical uncertainties.
However, as these non-perturbative effects are lepton flavour universal, they
cancel in the ratios of branching fractions $R_{H_{s}}$:

$R_{H_{s}}=\frac{\int_{q^{2}_{\textrm{min}}}^{q^{2}_{\textrm{max}}}\frac{{\rm
d}\mathcal{B}(\mbox{$H_{b}\rightarrow H_{s}\mu^{+}\mu^{-}$})}{{\rm
d}q^{2}}{\rm
d}q^{2}}{\int_{q^{2}_{\textrm{min}}}^{q^{2}_{\textrm{max}}}\frac{{\rm
d}\mathcal{B}(\mbox{$H_{b}\rightarrow H_{s}e^{+}e^{-}$})}{{\rm d}q^{2}}{\rm
d}q^{2}}.$ (1)

Where the differential branching fractions are integrated over the di-lepton
invariant mass squared range
$q^{2}\in[q^{2}_{\textrm{min}},q^{2}_{\textrm{max}}]$ and the final (initial)
state $H_{s(b)}$ particles contain a valence strange (beauty) quark. Due to
the universality of the coupling of SM gauge bosons to different lepton
flavours, the SM prediction for $R_{H_{s}}$ is unity up to $\mathcal{O}(1\%)$
QED corrections [8] and negligible phase space effects arising from the
different final state lepton masses, hence any sizable deviation from unity
would unambiguously indicate breaking of lepton flavour universality (LFU).

The LHCb collaboration [6]${}^{,}\,$[7] performed the most precise
measurements of LFU sensitive observables, such as $R_{K^{*}}$ [9] and $R_{K}$
[10], and both the results exhibit a tension above $2\sigma$ with respect to
the SM expectation. In 2021 LHCb presented an update to the previous
measurement of $R_{K}$ [11] with the addition of the data collected during
2017 and 2018, effectively doubling size of the analysed data set.

##  2 Experimental strategy

The measurement of $R_{K}$ is performed using two main ingredients: yields of
the decays of interest and the efficiency to select them. However, this
involves direct comparison of efficiencies to select muons and electrons decay
modes, which interact differently in the detector. While muons traverse the
LHCb detector interacting feebly with the material up to the muon stations
where they are absorbed, electrons lose a significant amount of energy to
bremsstrahlung radiation. This has a degrading effect not only on the mass
resolution of the electron with respect to the muon mode, but also on the
electrons reconstruction and tracking efficiencies. The main challenge of the
measurement consists of controlling for the efficiency of the selection
requirements placed to isolate signal candidates and reduce background
sources, as these could have systematically different effects on the two
lepton flavours.

In order to suppress such systematic effects, $R_{K}$ is measured as a double
ratio with respect to the resonant $B^{+}\rightarrow
K^{+}J/\psi(\ell^{+}\ell^{-})$ mode, which is known to obey LFU to the sub-
percent level [12] and has a branching fraction of orders of magnitude larger
than the rare $B^{+}\rightarrow K^{+}\ell^{+}\ell^{-}$ mode:

$\displaystyle\begin{split}{R_{K}}&=\frac{N(K^{+}\mu\mu)}{N(K^{+}ee)}\frac{\varepsilon(K^{+}ee)}{\varepsilon(K^{+}\mu\mu)}\Big{/}\underbrace{\frac{N(K^{+}J/\psi(\mu\mu))}{N(K^{+}J/\psi(ee))}\frac{\varepsilon(K^{+}J/\psi(ee))}{\varepsilon(K^{+}J/\psi(\mu\mu))}}_{r_{J/\psi}}\\\
&=\frac{N(K^{+}\mu\mu)}{N(K^{+}J/\psi(\mu\mu))}\frac{\varepsilon(K^{+}J/\psi(\mu\mu))}{\varepsilon(K^{+}\mu\mu)}\Big{/}\frac{N(K^{+}ee)}{N(K^{+}J/\psi(ee))}\frac{\varepsilon(K^{+}J/\psi(ee))}{\varepsilon(K^{+}ee)}\\\
\end{split}$ (2)

In Eq. (2), $N(X)$ and $\varepsilon(X)$ represent respectively the yields and
efficiencies of selecting the decay of a $B^{+}$ meson into X. After
rearranging the terms in the $R_{K}$ definition of Eq. (2) one can see that
the double ratio procedure allows to compare efficiencies to select same
lepton flavours in the final states, canceling out systematic uncertainties
that are common to both rare and resonant modes thanks to the kinematic
overlap of the two decays. The selection requirements are common to both
resonant and rare modes up to the cut in $q^{2}$ and of the reconstructed mass
of the $K^{+}\ell^{+}\ell^{-}$ system. Further selection is applied to reduce
background sources and retain only tracks and $B^{+}$ decay vertices with good
fit quality.

###  2.1 Efficiencies calibration

Selection efficiencies are estimated using simulated samples, and successively
corrected using clean and high statistics calibration data samples. The
efficiencies calibration procedure consists of correcting simulation
mismodeling using weights that are assigned to simulated events in order to
align the selection performance to the one observed in control mode data. The
calibration procedure involves correction of the $B^{+}$ kinematics, the
efficiency of the particle identification as well as trigger selections and
mismatches in the resolution of the $q^{2}$ and reconstructed mass of the
$B^{+}$ candidate.

Figure 2: The ratio $r_{J/\psi}$ relative to its average value $\langle
r_{J/\psi}\rangle$ as a function of $p_{T}$ (left) and distributions of the
$B^{+}$ transverse momentum (left). The distribution from the
$B^{+}\rightarrow K^{+}J/\psi$ decays is similar to that of the corresponding
$B^{+}\rightarrow K^{+}\ell^{+}\ell^{-}$ decays such that the differential
measurement of $r_{J/\psi}$ tests the kinematic region relevant for the
$R_{K}$ measurement.

###  2.2 Procedure validation

The experimental method is validated through several cross checks that aim at
both verifying the control of the selection efficiencies and the cancellation
of the systematic effects in the double ratio procedure. Percent-level control
of the efficiencies is verified with a direct comparison of the
$B^{+}\rightarrow K^{+}J/\psi(e^{+}e^{-})$ and $B^{+}\rightarrow
K^{+}J/\psi(\mu^{+}\mu^{-})$ branching fractions, $r_{J/\psi}$ as defined in
Eq. (2). This is a stringent cross check of the understanding of the
efficiencies as $r_{J/\psi}$ doesn’t exploit the systematics uncertainties
suppression of the double ratio. The value obtained using the full dataset is
found to be $r_{J/\psi}=0.981\pm 0.020$ where the error containing both
systematic and statistic components is dominated by the systematic
uncertainty, as expected. This value is within $1\sigma$ of the LFU
expectation.

The control of the efficiencies is further validated by performing
$r_{J/\psi}$ single and double-differentially in variables which are relevant
to the detector response. An example is shown in Fig. 2 on the left, where
$r_{J/\psi}$ is found to be uniform in bins of transverse momentum of the
$B^{+}$ meson. This cross check allows to quantify the effect that the
remaining deviations from flatness would have on $R_{K}$ if they were due to a
genuine mismodelling of the efficiencies rather than statistical fluctuations.
Thanks to the large kinematic overlap between the rare and resonant modes, as
can be seen in Fig. 2 (right), the impact on $R_{K}$ is within the estimated
systematic uncertainty. Similarly, double differential computations of the
$r_{J/\psi}$ ratio do not show any trend and are within the systematic
uncertainties assigned to $R_{K}$.

The validity of the efficiencies estimation procedure and the suppression of
the systematic effects due to electron muon detection differences is further
tested by performing the double ratio:

$R_{\psi(2S)}=\frac{N(K^{+}\psi(2S)(\mu\mu))}{N(K^{+}J/\psi(\mu\mu))}\frac{\varepsilon(K^{+}J/\psi(\mu\mu))}{\varepsilon(K^{+}\psi(2S)(\mu\mu))}\Big{/}\frac{N(K^{+}\psi(2S)(ee))}{N(K^{+}J/\psi(ee))}\frac{\varepsilon(K^{+}J/\psi(ee))}{\varepsilon(K^{+}\psi(2S)(ee))}$
(3)

The result obtained is $R_{\psi(2S)}=0.997\pm 0.011$ where the uncertainty
includes both statistical and systematic effects. The result is well
compatible with the LFU hypothesis for $\psi(2S)\rightarrow\ell^{+}\ell^{-}$
decays and confirms the cancellation of systematic effects to sub-percent
level in the double ratio.

##  3 Results

The value of $R_{K}$ is extracted as a parameter of interest from an unbinned
extended maximum likelihood fit which is performed simultaneously to the
$B^{+}\rightarrow K^{+}\mu^{+}\mu^{-}$ and $B^{+}\rightarrow K^{+}e^{+}e^{-}$
data, the projections are shown in Fig. 3

Figure 3: Distribution of the invariant mass for candidates with (left)
electron and (right) muon pairs in the final state for the rare
$B^{+}\rightarrow K^{+}\ell^{+}\ell^{-}$ signal channels.

The mass-shape parameters are derived from the calibrated simulation. It can
be noted that the statistics of the muon data sample is higher than the
electron case due to lower electron reconstruction efficiency, as discussed in
Sec. 2. In both muon and electron fits the signal shape forms a peaking
structure which in the electron case has a long elongated tail at lower
reconstructed $B^{+}$ masses owing to the fact that the bremsstrahlung
recovery algorithm is not perfect and not always the energy lost is correctly
recovered. Moreover, due to the poorer mass resolution of the electrons, the
selected data sample is polluted by backgrounds coming from the resonant
$B^{+}\rightarrow K^{+}J/\psi(e^{+}e^{-})$ mode or from partially
reconstructed backgrounds such as, for example, $B^{0}\rightarrow
K^{*0}(K^{+}\pi^{-})e^{+}e^{-}$ decays in which the $\pi^{-}$ is not
reconstructed. In both electrons and muon fits, candidates formed from random
track combinations, extend thoughout the fit mass window and correspond to the
combinatorial background.

The obtained value of $R_{K}$ is corrected by taking into account the bias
introduced by the choice of fit model, which is found to be small, and the
result reads:

${R_{K}}=0.846\,^{+\,0.042}_{-\,0.039}\,{}^{+\,0.013}_{-\,0.012}$ (4)

Where the first uncertainty is statistical, and the second is systematic. As
expected the error on the result is dominated by the statistical contribution
rather than the systematic one, the dominant contributions to the systematic
uncertainties being the choice of the fit model for signal and partially
reconstructed decays while the effects due to trigger and kinematics
corrections amount to the per mille level showing robustness of the double
ratio method.

The obtained result, being the most precise measurement of $R_{K}$ to date
(Fig. 4), has a p-value under the SM hypothesis of $0.1\%$. This corresponts
to a deviation from the SM prediction of ${3.1}\sigma$, and therefore
constitutes the first evidence of LFU violation in $B^{+}\rightarrow
K^{+}\ell^{+}\ell^{-}$ decays obtained with a single experiment.

Another result can be obtained by combining the extracted value of $R_{K}$
with the result for the branching fraction measurement ${\rm
d}\mathcal{B}(\mbox{$B^{+}\rightarrow K^{+}\mu^{+}\mu^{-}$})/{\rm d}q^{2}$
from [4] to obtain ${\rm d}\mathcal{B}(\mbox{$B^{+}\rightarrow
K^{+}e^{+}e^{-}$})/{\rm d}q^{2}$ in the same dilepton invariant mass region

$\frac{d\mathcal{B}(B^{+}\rightarrow K^{+}e^{+}e^{-})}{{\rm
d}q^{2}}=(28.6^{+1.5}_{-1.4}\textrm{ (stat.) }\pm 1.3\textrm{ (syst.) })\times
10^{-9}\,\text{\,Ge\kern-1.00006ptV}^{-2}$ (5)

Figure 4: Left: Comparison between $R_{K}$ measurements. In addition to the
LHCb result (black), the Babar [13] (green) and Belle [14] (blue)
collaborations are shown. Right: likelihood function from the fit to
$B^{+}\rightarrow K^{+}\ell^{+}\ell^{-}$ candidates profiled as a function of
$R_{K}$. The extent of the dark, medium and light blue regions shows the
values allowed for $R_{K}$ at $1,3,5\sigma$ levels. The red line indicates the
SM prediction.

##  4 Summary

The observable $R_{K}$ is measured using 9 $\rm{\leavevmode\nobreak\ fb}^{-1}$
of proton-proton collision data recorded by the LHCb experiment and the result
is found to be in tension with the SM prediction at the level of $3.1\sigma$.
This result constitutes evidence for the violation of LFU in $B^{+}\rightarrow
K^{+}\ell^{+}\ell^{-}$ decays.

## References

## References

  * [1] S. L. Glashow, J. Iliopoulos, and L. Maiani. Weak interactions with lepton-hadron symmetry. PRD, 2:1285–1292, Oct 1970.
  * [2] C. Cornella, D. A. Faroughy, J. Fuentes-Martin, G. Isidori, M. Neubert, Reading the footprints of the B-meson flavor anomalies, arxiv.org:2103.16558.
  * [3] New Physics in Rare B Decays after Moriond 2021 arxiv:2103.13370.
  * [4] R. Aaij et al. Differential branching fractions and isospin asymmetries of $B\rightarrow K^{(*)}\mu^{+}\mu^{-}$ decays. JHEP 2014, 133 (2014).
  * [5] Measurements of the S-wave fraction in $B^{0}\rightarrow K^{+}\pi^{-}\mu^{+}\mu^{-}$ decays and the $B^{0}\rightarrow K^{*}(892)\mu^{+}\mu^{-}$ differential branching fraction. JHEP 2016, 47 (2016).
  * [6] A. A. Alves Jr. et al. The LHCb detector at the LHC. JINST, 3:S08005, 2008.
  * [7] R. Aaij et al. LHCb detector performance. Int. J. Mod. Phys., A30:1530022, 2015.
  * [8] On the standard model predictions for $R_{K}$ and $R_{K^{*}}$. EPJC 76, 440 (2016).
  * [9] R. Aaij et al. Test of lepton universality with $B^{0}\rightarrow K^{*0}\ell^{+}\ell^{-}$ decays JHEP 08 (2017) 055.
  * [10] R. Aaij et al. Search for Lepton-Universality Violation in $B^{+}\rightarrow K^{+}\ell^{+}\ell^{-}$ Decays. PRL 122 (2019) 19, 191801.
  * [11] R. Aaij et al. Test of lepton universality in beauty-quark decays. arXiv:2103.11769.
  * [12] P. A. Zyla et al. Review of particle physics. PTEP, 2020(8):083C01, 2020.
  * [13] J. P. Lees et al. Measurement of branching fractions and rate asymmetries in the rare decays $B\rightarrow K^{(*)}\ell^{+}\ell^{-}$. PRD, 86:032012, 2012.
  * [14] S. Choudhury et al. Test of lepton flavor universality and search for lepton flavor violation in $B\rightarrow K\ell\ell$ decays. JHEP, 03:105, 2021.