[table]capposition=top

11institutetext: Johns Hopkins University, Baltimore MD 21218, USA
22institutetext: Indian Institute of Science, Bangalore 560012, India
22email<EMAIL_ADDRESS>22email:
<EMAIL_ADDRESS>
# Learning to Count in the Crowd from Limited Labeled Data

Vishwanath A. Sindagi 11 Rajeev Yasarla 11 Deepak Sam Babu 22 R. Venkatesh
Babu 22 Vishal M. Patel 11

###### Abstract

Recent crowd counting approaches have achieved excellent performance. However,
they are essentially based on fully supervised paradigm and require large
number of annotated samples. Obtaining annotations is an expensive and labour-
intensive process. In this work, we focus on reducing the annotation efforts
by learning to count in the crowd from limited number of labeled samples while
leveraging a large pool of unlabeled data. Specifically, we propose a Gaussian
Process-based iterative learning mechanism that involves estimation of pseudo-
ground truth for the unlabeled data, which is then used as supervision for
training the network. The proposed method is shown to be effective under the
reduced data (semi-supervised) settings for several datasets like
ShanghaiTech, UCF-QNRF, WorldExpo, UCSD, _etc_. Furthermore, we demonstrate
that the proposed method can be leveraged to enable the network in learning to
count from synthetic dataset while being able to generalize better to real-
world datasets (synthetic-to-real transfer).

###### Keywords:

Crowd counting, semi-supervised learning, pseudo-labeling, domain adaptation,
synthetic to real transfer

##  1 Introduction

Due to its significance in several applications (like video surveillance [12,
50, 44], public safety monitoring [57], microscopic cell counting [15],
environmental studies [23], _etc_.), crowd counting has attracted a lot of
interest from the deep learning research community. Several convolutional
neural network (CNN) based approaches have been developed that address various
issues in counting like scale variations, occlusion, background clutter [17,
58, 18, 36, 42, 22, 39, 37, 3, 28, 19, 43, 33, 34, 2], _etc_. While these
methods have achieved excellent improvements in terms of the overall error
rate, they follow a fully-supervised paradigm and require several labeled data
samples. There is a wide variety of scenes and crowded scenarios that these
networks need to handle to in the real world. Due to a distribution gap
between the training and testing environments, these networks have limited
generalization abilities and hence, procuring annotations becomes especially
important. However, annotating data for crowd counting typically involves
obtaining point-wise annotations at head locations, and this is a labour
intensive and expensive process. Hence, it is infeasible to procure
annotations for all possible scenarios. Considering this, it is crucial to
reduce the annotation efforts, especially for crowd counting methods which get
deployed in a wide variety of scenarios.

(a) (b)

Figure 1: Results of semi-supervised learning experiments. (a) ShanghaiTech A
(b) UCF-QNRF. For both datasets, the error increases with reduction in the
%-age of labeled data. By leveraging the unlabeled dataset using the proposed
GP-based framework, we are able to reduce the error considerably. Note that
$\mathcal{D_{L}}$ and $\mathcal{D_{U}}$ indicate labeled and unlabeled
dataset, respectively.

With the exception of a few works [6, 22, 55], reducing annotation efforts
while maintaining good performance is relatively less explored for the task of
crowd counting. Hence, in this work, we focus on learning to count using
limited labeled data while leveraging unlabeled data to improve the
performance. Specifically, we propose a Gaussian Process (GP) based iterative
learning framework where we augment the existing networks with capabilities to
leverage unlabeled data, thereby resulting in overall improvement in the
performance. The proposed framework follows a pseudo-labeling approach, where
we estimate the pseudo-ground truth (pseudo-GT) for the unlabeled data, which
is then used to supervise the network. The network is trained iteratively on
labeled and unlabeled data. In the labeled stage, the network weights are
updated by minimizing the $L_{2}$ error between predictions and the ground-
truth (GT) for the labeled data. In addition, we save the latent space vectors
of the labeled data along with the ground-truths. In the unlabeled stage, we
first model the relationship between the latent space vectors of the labeled
images along with the corresponding ground-truth and unlabeled latent space
vectors jointly using GP. Next, we estimate the pseudo-GT for the unlabeled
inputs using the GP modeled earlier. This pseudo-GT is then used to supervise
the network for the unlabeled data. Minimizing the error between the unlabeled
data predictions and the pseudo-GT results in improved performance. Fig. 1
illustrates the effectiveness of the proposed GP-based framework in exploiting
unlabeled data on two datasets (ShanghaiTech-A [60] and UCF-QNRF[10]) in the
reduced data setting. It can be observed that the proposed method is able to
leverage unlabeled data effectively resulting in lower error across various
settings.

The proposed method is evaluated on different datasets like ShanghaiTech [60],
UCF-QNRF [10], WorldExpo [58], UCSD [4], _etc_. in the reduced data settings.
In addition to obtaining lower error as compared to the existing methods [22],
the performance drop due to less data is improved by a considerable margin.
Furthermore, the proposed method is effective for learning to count from
synthetic data as well. More specifically, we use labeled synthetic crowd
counting dataset (GCC [55]) and unlabeled real-world datasets (ShanghaiTech
[60], UCF-QNRF [10], WorldExpo [58], UCSD [5]) in our framework, and show that
it is able to generalize better to real-world datasets as compared to recent
domain adaptive crowd counting approaches [55]. To summarize, the following
are our contributions:

  * •

We propose a GP-based framework to effectively exploit unlabeled data during
the training process, resulting in improved overall performance. The proposed
method consists of iteratively training over labeled and unlabeled data. For
the unlabeled data, we estimate the pseudo-GT using the GP modeled during
labeled phase.

  * •

We demonstrate that the proposed framework is effective in semi-supervised and
synthetic-to-real transfer settings. Through various ablation studies, we show
that the proposed method is generalizable to different network architectures
and various reduced data settings.

##  2 Related Work

Crowd Counting. Traditional approaches in crowd counting ([16, 31, 7, 9, 15,
27, 56]) typically involved feature extraction techniques and training
regression algorithms. Recently, CNN-based approaches like [54, 58, 36, 1, 51,
26, 60, 36, 42] have surpassed the traditional approaches by a large margin in
terms of the overall error rate. Most of these methods focus on addressing the
issue of large variations in scales. Approaches like [60, 36, 42] focus on
improving the receptive field. Different from these, approaches like [28, 41,
47, 32] focus on effective ways of fusing multi-scale information from deep
networks. In addition to scale variation, recent approaches have addressed
other issues in crowd counting like improving the quality of predicted density
maps using adversarial regularization [42, 37], use of deep negative
correlation-based learning for obtaining more generalizable features, and
scale-based feature aggregation [3]. Most recently, several methods have
employed additional information like segmentation and semantic priors [61,
53], attention [20, 45, 46], perspective [38], context information [21],
multiple-views [59] and multi-scale features [11], adaptive density maps [52]
into the network. In other efforts, researchers have made important
contributions by creating large-scale datasets for counting like UCF-QNRF
[10], GCC [55] and JHU-CROWD [48, 49]. For a more detailed discussion on these
methods, the reader is referred to recent comprehensive surveys [43, 8]

Learning from limited data.  Recent research in crowd counting has been
largely focused on improving the counting performance in the fully-supervised
paradigm. Very few works like [6, 22, 55] have made efforts on minimizing
annotation efforts. Loy _et al._[6] proposed a semi-supervised regression
framework that exploit underlying geometric structures of crowd patterns to
assimilate the count estimation of two nearby crowd pattern points in the
manifold. However, this approach is specifically designed for video-based
crowd counting.

Recently, Liu _et al._[22] proposed to leverage additional unlabeled data for
counting by introducing a learning to rank framework. They assume that any
sub-image of a crowded scene image is guaranteed to contain the same number or
fewer persons than the super-image. They employ pairwise ranking hinge loss to
enforce this ranking constraint for unlabeled data in addition to the $L_{2}$
error to train the network. In our experiments we observed that this
constraint is almost always satisfied, and it provides relatively less
supervision over unlabeled data.

Babu _et al._[35] focus on a different approach, where they train 99.9% of
their parameters from unlabeled data using a novel unsupervised learning
framework based on winner-takes-all (WTA) strategy. However, they still train
the remaining set of parameters using labeled data.

Wang _et al._[55] take a totally different approach to minimize annotation
efforts by creating a new synthetic crowd counting dataset (GCC).
Additionally, they propose a Cycle-GAN based domain adaptive approach for
generalizing the network trained on synthetic dataset to real-world dataset.
However,there is a large gap in terms of the style and also the crowd count
between the synthetic and real-world scenarios. Domain adaptive approaches
have limited abilities in handling such scenarios. In order to obtain
successful adaptation, the authors in [55] manually select the samples from
the synthetic dataset that are closer to the real-world scenario in terms of
crowd count for training the network. This selection is possible when one has
information about the count from the real-world datasets, which violates the
assumption of lack of unlabeled data in the target domain for unsupervised
domain adaptation.

Considering the drawbacks of existing approaches, we propose a new GP-based
iterative training framework to exploit unlabeled data.

##  3 Preliminaries

In this section, we briefly review the concepts (crowd counting, semi-
supervised learning and Gaussian Process) that are used in this work.

Crowd counting. Following recent works [58, 60], we employ the approach of
density estimation technique. That is, an input crowd image is forwarded
through the network, and the network outputs a density map. This density map
indicates the per-pixel count of people in the image. The count in the image
is obtained by integrating over the density map. For training the network
using labeled data, the ground-truth density maps are obtained by imposing 2D
Gaussians at head location $x_{g}$ using $D(x)=\sum_{{x_{g}\in
S}}\mathcal{N}(x-x_{g},\sigma)$. Here, $\sigma$ is the Gaussian kernel’s scale
and $S$ is the list of all locations of people.

Problem formulation. We are given a set of labeled dataset of input-GT pairs
($\\{x,y\\}\in\mathcal{D_{L}}$) and a set of unlabeled input data samples
$x\in\mathcal{D_{U}}$. The objective is to fit a mapping-function $f(x|\phi)$
(with parameters defined by $\phi$) that accurately estimates target label $y$
for unobserved samples. Note that this definition applies to both semi-
supervised setting and synthetic-to-real transfer setting. In the case of
synthetic-to-real transfer, the synthetic dataset is labeled and hence, can be
used as the labeled dataset ($\mathcal{D_{L}}$). Similarly, the real-world
dataset is unlabeled and can be used as the unlabeled dataset
($\mathcal{D_{U}}$).

In order to learn the parameters, both labeled and unlabeled datasets are
exploited. Typically, loss functions such as $L_{1}$, $L_{2}$ or cross entropy
error are used for labeled data. For exploiting unlabeled data
$\mathcal{D_{U}}$, existing approaches augment $f(x|\phi)$ with information
like shape of the data manifold [25] via different techniques such as
enforcing consistent regularization [13], virtual adversarial training [24] or
pseudo-labeling [14]. In this work, we employ pseudo-labeling based approach
where we estimate pseudo-GT for unlabeled data, and then use them for
supervising the network using traditional supervised loss functions.

Gaussian process. A Gaussian process (GP) $f(v)$ is an infinite collection of
random variables, any finite subset of which have a joint Gaussian
distribution. A GP is fully specified by its mean function ($m(v)$) and
covariance function $K(v,v^{\prime})$. These are defined below:

$\displaystyle m(v)$ $\displaystyle=\mathbb{E}[f(v)],$ (1)
$\displaystyle{K}\left(v,v^{\prime}\right)$
$\displaystyle=\mathbb{E}\left[(f(v)-m(v))\left(f\left(v^{\prime}\right)-m\left(v^{\prime}\right)\right)\right],$
(2)

where $v,v^{\prime}\in\mathcal{V}$ denote the possible inputs that index the
GP. The covariance matrix is computed from the covariance function ${K}$ which
expresses the notion of smoothness of the underlying function. GP can then be
formulated as follows:

$f(v)\sim\mathcal{GP}(m(v),K(v,v^{\prime})+\sigma_{\epsilon}^{2}I),$ (3)

where $I$ is identity matrix and $\sigma_{\epsilon}^{2}$ is the variance of
the additive noise. Any collection of function values is then jointly Gaussian
as follows

$f(V)=\left[f\left(v_{1}\right),\ldots,f\left(v_{n}\right)\right]^{T}\sim\mathcal{N}\left(\mu,K(V,V^{\prime})+\sigma_{\epsilon}^{2}I\right),$
(4)

with mean vector and covariance matrix defined by the GP as mentioned earlier.
To make predictions at unlabeled points, one can compute a Gaussian posterior
distribution in closed form by conditioning on the observed data. For more
details, we refer the reader to [29].

##  4 GP-based iterative learning

Fig. 2 gives an overview of the proposed method. The network is constructed
using an encoder $f_{e}(x,\phi_{e})$ and a decoder $f_{d}(z,\phi_{d})$, that
are parameterized by $\phi_{e}$ and $\phi_{d}$, respectively. The proposed
framework is agnostic to the encoder network, and we show in the experiments
section that it generalizes well to architectures such as VGG16 [40],
ResNet-50 and ResNet-101 [30]. The decoder consists of a set of 2 conv-relu
layers (see supplementary material for more details). Typically, an input
crowd image $x$ is forwarded through the encoder network to obtain the
corresponding latent space vector $z$. This vector is then forwarded through
the decoder network to obtain the crowd density output $y$, _i.e_ ,
$y=f_{d}(f_{e}(x,\phi_{e}),\phi_{d})$.

We are given a training dataset,
$\mathcal{D}=\mathcal{D_{L}}\cup\mathcal{D_{U}}$, where
$\mathcal{D_{L}}=\\{x_{l}^{i},y_{l}^{i}\\}_{i=1}^{N_{l}}$ is a labeled dataset
containing $N_{l}$ training samples and
$\mathcal{D_{U}}=\\{x_{u}^{i}\\}_{i=1}^{N_{u}}$ is an unlabeled dataset
containing $N_{u}$ training samples. The proposed framework effectively
leverages both the datasets by iterating the training process over labeled
$\mathcal{D_{L}}$ and unlabeled datasets $\mathcal{D_{U}}$. More specifically,
the training process consists of two stages: (i) Labeled training stage: In
this stage, we employ supervised loss function $\mathcal{L}_{s}$ to learn the
network parameters using labeled dataset, and (ii) Unlabeled training stage:
We generate pseudo GTs for the unlabeled data points using the GP formulation,
which is then used for supervising the network on the unlabeled dataset. In
what follows, we describe these stages in detail.

Figure 2: Illustration of the proposed framework. Training is performed
iteratively over labeled and unlabeled data. For labeled data, we minimize the
$L_{2}$ error between the predictions and GT. For unlabeled data, we minimize
the $L_{2}$ error between the predictions and pseudo-GT.

###  4.1 Labeled stage

Since the labeled dataset $\mathcal{D_{L}}$ comes with annotations, we employ
$L_{2}$ error between the predictions and the GTs as supervision loss for
training the network. This loss objective is defined as follows:

$\mathcal{L}_{s}=\mathcal{L}_{2}=\|y^{pred}_{l}-y_{l}\|_{2},$ (5)

where $y^{pred}_{l}=g(z_{l},\phi_{d})$ is the predicted output, $y_{l}$ is the
ground-truth, $z=h(x,\phi_{e})$ is the intermediate latent space vector. Note
that, the subscript $l$ in the above quantities indicate that these are
defined for labeled data.

Along with performing supervision on the labeled data, we additionally save
feature vectors $z_{l}^{i}$’s from the intermediate latent space in a matrix
$F_{z_{l}}$. Specifically, $F_{z_{l}}=\\{z_{l}^{i}\\}_{i=1}^{N_{l}}$. This
matrix is used for computing the pseudo-GTs for unlabeled data at a later
stage. The dimension of $F_{z_{l}}$ matrix is $N_{l}\times M$. Here, $M$ is
the dimension of the latent space vector $z_{l}$. In our case, the latent
space vector dimension is $64\times 32\times 32$ (see supplementary material
for more details), which is reshaped to $1\times 65,536$. Hence, $M=65,536$.

###  4.2 Unlabeled stage

Since the unlabeled data $\mathcal{D_{U}}$ does not come with any GT
annotations, we estimate pseudo-GTs which are then used as supervision for
training the network on unlabeled data. For this purpose, we model the
relationship between the latent space vectors of the labeled images
$F_{z_{l}}$ along with the corresponding GT $T_{y_{l}}$ and unlabeled latent
space vectors $z^{pred}_{u}$ jointly using GP.

Estimation of pseudo-GT: As discussed earlier, the training process iterates
over labeled $\mathcal{D_{L}}$ and unlabeled data $\mathcal{D_{U}}$. After the
labeled stage, the labeled latent space vectors $F_{z_{l}}$ and their
corresponding GT density maps $T_{y_{l}}$ are used to model the function $t$
which maps the relationship between the latent vectors and the output density
maps as, $y=t(z)$. Using GP, we model this function $t(.)$ as an infinite
collection of functions of which any finite subset is jointly Gaussian. More
specifically, we jointly model the distribution of the function values $t(.)$
of the latent space vectors of the labeled and the unlabeled samples using GP
as follows:

$P(t(z)|\mathcal{D_{L}},F_{z_{l}},T_{y_{l}})\sim\mathcal{GP}(\mu,K(F_{z_{l}},F_{z_{l}})+\sigma_{\epsilon}^{2}I),$
(6)

where $\mu$ is the function value computed using GP, $\sigma_{\epsilon}^{2}$
is set equal to 1, and $K$ is the kernel function. Based on this, the
conditional joint distribution for the latent space vector $z^{k}_{u}$ of the
$k^{th}$ unlabeled sample $x^{k}_{u}$ can be expressed as the following
Gaussian distribution:

$P(t(z_{u}^{k})|\mathcal{D_{L}},F_{z_{l}},T_{z_{l}})=\mathcal{N}(\mu_{u}^{k},\Sigma_{u}^{k}),$
(7)

where

$\mu_{u}^{k}=K(z_{u}^{k},F_{z_{l}})[K(F_{z_{l}},F_{z_{l}})+\sigma_{\epsilon}^{2}I]^{-1}T_{y_{l}},$
(8) $\displaystyle\Sigma_{u}^{k}={}$ $\displaystyle
K(z_{u}^{k},z_{u}^{k})-K(z_{u}^{k},F_{z_{l}})[K(F_{z_{l}},F_{z_{l}})+\sigma_{\epsilon}^{2}I]^{-1}K(F_{z_{l}},z_{u}^{k})+\sigma_{\epsilon}^{2}$
(9)

where $\sigma_{\epsilon}^{2}$ is set equal to 1 and $K$ is a kernel function
with the following definition:

$K(Z,Z)_{k,i}=\kappa(z_{u}^{k},z_{l}^{i})=\frac{\langle
z_{u}^{k},z_{l}^{i}\rangle}{|z_{u}^{k}|\cdot|z_{l}^{i}|}.$ (10)

Considering the large dimensionality of the latent space vector,
$K(F_{z_{l}},F_{z_{l}})$ can grow quickly in size especially if the number of
labeled data samples $N_{l}$ is high. In such cases, the computational and
memory requirements become prohibitively high. Additionally, all the latent
vectors may not be necessarily effective since these vectors correspond to
different regions of images in terms of content and size/density of the crowd.
In order to overcome these issues, we use only those labeled vectors that are
similar to the unlabeled latent vector. Specifically, we consider only $N_{n}$
nearest labeled vectors corresponding to an unlabeled vector. That is, we
replace $F_{z_{l}}$ by $F_{z_{l},n}$ in Eq. (7)-(9). Here
$F_{z_{l},n}=\\{z_{l}^{j}:z_{l}^{j}\in nearest(z_{u}^{k},F_{z_{l}},N_{n})\\}$,
and $T_{y_{l},n}=\\{y^{j}_{l}:z_{l}^{j}\in
nearest(z_{u}^{k},F_{z_{l}},N_{n})\\}$ with $nearest(p,Q,N_{n})$ being a
function that finds top $N_{n}$ nearest neighbors of $p$ in $Q$.

The pseudo-GT for unlabeled data sample is given by the mean predicted in Eq.
(8), _i.e_ , $y_{u,pseudo}^{k}=\mu_{u}^{k}$. The $L_{2}$ distance between the
predictions $y^{k}_{u,pred}=g(z_{u}^{k},\phi_{e})$ and the pseudo-GT
$y_{u,pseudo}^{k}$ is used as supervision for updating the parameters of the
encoder $f_{e}(\cdot,\phi_{e})$ and the decoder $f_{d}(.,\phi_{d})$.

Furthermore, the pseudo-GT estimated using Eq. (8) may not be necessarily
perfect. Errors in pseudo-GT will limit the performance of the network. To
overcome this, we explicitly exploit the variance modeled by the GP.
Specifically, we minimize the predictive variance by considering Eq. (9) in
the loss function. As discussed earlier, using all the latent space vectors of
labeled data may not be necessarily effective. Hence, we minimize the variance
$\Sigma_{u,n}^{k}$ computed between $z^{k}_{u}$ and the $N_{n}$ nearest
neighbors in the latent space vectors using GP. Thus, the loss function during
the unlabeled stage is defined as:

$\mathcal{L}_{un}=\frac{1}{|\Sigma_{u,n}^{k}|}\|{y}^{k}_{u,pred}-{y}_{u,pseudo}^{k}\|_{2}+\log\Sigma_{u,n}^{k},$
(11)

where $y^{k}_{u,pred}$ is the crowd density map prediction obtained by
forwarding an unlabeled input image $x_{u}^{k}$ through the network,
$y_{u,pseudo}^{k}=\mu_{u}^{k}$ is the pseudo-GT (see Eq. (8)), and
$\Sigma_{u,n}^{k}$ is the predictive variance obtained by replacing
$F_{z_{l}}$ in Eq. (9) with $F_{z_{l},n}$.

###  4.3 Final objective function

We combine the supervised loss Eq. (5) and unsupervised loss Eq. (11) to
obtain the final objective function as follows:

$\mathcal{L}_{f}=\mathcal{L}_{s}+\lambda_{un}\mathcal{L}_{un},$ (12)

where $\lambda_{un}$ is a hyper-parameter that weighs the unsupervised loss.

##  5 Experiments and results

In this section, we discuss the details of the various experiments conducted
to demonstrate the effectiveness of the proposed method. Since the proposed
method is able to leverage unlabeled data to improve the overall performance,
we performed evaluation in two settings: (i) Semi-supervised settings: In this
setting, we varied the percentage of labeled samples from 5% to 75%. We first
show that with the base network, there is performance drop due to the reduced
data. Later, we show that the proposed method is able to recover a major
percentage of the performance drop. (ii) Synthetic-to-real transfer settings:
In this setting, the goal is to train on synthetic dataset (labeled), while
adapting to real-world dataset. Unlabeled images from the real-world are
available during training. In both settings, the proposed method is able to
achieve better results as compared to recent methods. Details of the datasets
are provided in the supplementary material.

###  5.1 Semi-supervised settings

In this section, we conduct experiments in the semi-supervised settings by
reducing the amount of labeled data available during training. The rest of the
samples in the dataset are considered as unlabeled samples wherever
applicable. In the following sub-sections, we present comparison of the
proposed method in the 5% setting with other recent methods. For comparison,
we used 4 datasets: ShanghaiTech (SH-A/B)[60], UCF-QNRF [10], WorldExpo [58]
and UCSD [4]. This is followed by a detailed ablation study involving
different architectures and various percentages of labeled data used during
training. For ablation, we chose ShanghaiTech-A and UCF-QNRF datasets since
they contain a wide diversity of scenes and large variation in count and
scales.

Implementation details. We train the network using Adam optimizer with a
learning rate of $10e-{5}$ and a momentum of $0.9$ on an NVIDIA Titan Xp GPU.
We use batch size of 24. During training, random crops of size $256\times 256$
are used. During inference, the entire image is forwarded through the network.
For evaluation, we use mean absolute error ($MAE$) and mean squared error
($MSE$) metrics, which are defined as:
$MAE=\frac{1}{N}\sum_{i=1}^{N}|y_{i}-y^{\prime}_{i}|$ and
$MSE=\sqrt{\frac{1}{N}\sum_{i=1}^{N}|y_{i}-y^{\prime}_{i}|^{2}}$,
respectively. Here, $N$ is the total number of test images, $y_{i}$ is the
ground-truth/target count of people in the image and $y^{\prime}_{i}$ is the
predicted count of people in to the $i^{th}$ image. We set aside 10% of the
training set for the purpose of validation. The hyper-parameter $\lambda_{un}$
was chosen based on the validation performance. More details are provided in
the supplementary.

Table 1: Comparison of results in SSL settings. Reducing labeled data to 5%
results in performance drop by a big margin as compared to 100% data.
ResNet-50 was used as the encoder network for all the methods. RL: Ranking-
Loss. GP: Gaussian-Process. AG: Average Gain %${}^{\ref{ftn:ag}}$.

Method | $\mathcal{D_{L}}$ | $\mathcal{D_{U}}$ | SH-A | SH-B | UCF-QNRF | WExpo | UCSD
---|---|---|---|---|---|---|---
MAE | MSE | AG | MAE | MSE | AG | MAE | MSE | AG | MAE | AG | MAE | MSE | AG
ResNet-50 (Oracle) | 100% | - | 76 | 126 | - | 8.4 | 14.5 | - | 114 | 195 | - | 10.1 | - | 1.7 | 2.1 | -
ResNet-50 ($\mathcal{D_{L}}$-only) | 5% | - | 118 | 211 | - | 21.2 | 34.2 | - | 186 | 295 | - | 14.2 | - | 2.2 | 2.8 | -
ResNet-50+RL | 5% | 95% | 115 | 208 | 2.0 | 20.1 | 32.9 | 4.0 | 182 | 291 | 1.7 | 14.0 | 0.01 | 2.2 | 2.8 | 0
ResNet-50+GP(Ours) | 5% | 95% | 102 | 172 | 16 | 15.7 | 27.9 | 22 | 160 | 275 | 10 | 12.8 | 10 | 2.0 | 2.4 | 12

Comparison with recent approaches. Here, we compare the effectiveness of the
proposed method with a recent method by Liu _et al._[22] on 4 different
datasets. In order to get a better understanding of the overall improvements,
we also provide the results of the base network with (i) 100% labeled data
supervision that is the oracle performance, and (ii) 5% labeled data
supervision.

For all the methods (except oracle), we limited the labeled data used during
training to 5% of the training dataset. Rest of the samples were used as
unlabeled samples. We used ResNet-50 as the encoder network. The results of
the experiments are shown in Table 1. For all the experiments that we
conducted, we report the average of the results for 5 trials. The standard
deviations are reported in the supplementary. We make the following
observations for all the datasets: (i) Compared to using the entire dataset,
reducing the labeled data during training (to 5%) leads to significant
increase in error. (ii) The proposed GP-based framework is able to reduce the
performance drop by a large margin. Further, the proposed method achieves an
average gain (AG)111$AG=\frac{G_{mae}+G_{mse}}{2}$,
$G_{mae}=\frac{mae_{(\mathcal{D_{U}+D_{L}})}-mae_{(\mathcal{D_{L}})}}{mae_{(\mathcal{D_{L}})}}$,
$G_{mse}=\frac{mse_{(\mathcal{D_{U}+D_{L}})}-mse_{(\mathcal{D_{L}})}}{mse_{(\mathcal{D_{L}})}}$
of anywhere between 10%-22% over the $\mathcal{D_{L}}$-only baseline across
all datasets. (iii) The proposed method is able to leverage the unlabeled data
more effectively as compared to Liu _et al._[22]. This is because the authors
in [22] using a ranking loss on the unlabeled data which is based on the
assumption that sub-image of a crowded scene is guaranteed to contain the same
or fewer number of people compared to the entire image. We observed that this
constraint is satisfied naturally for most of the unlabeled images, and hence
it provides less supervision (see supplementary material for a detailed
analysis).

Table 2: Results of ablation study with different %-ages of labeled data. The
proposed method achieves significant gains across different percentages of
labeled data. We used ResNet-50 as the encoder network for all the
experiments. AG: Average Gain %${}^{\ref{ftn:ag}}$.

$\mathcal{D_{L}}$ % | SH-A | UCF-QNRF
---|---|---
No-GP ($\mathcal{D_{L}}$-only) | GP ($\mathcal{D_{L}+D_{U}}$) |  AG % | No-GP ($\mathcal{D_{L}}$-only) | GP ($\mathcal{D_{L}+D_{U}}$) |  AG %
MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE
5 | 118 | 211 | 102 | 172 | 16 | 186 | 295 | 160 | 275 | 10
25 | 110 | 160 | 91 | 149 | 12 | 178 | 252 | 147 | 226 | 14
50 | 102 | 149 | 89 | 148 | 6.1 | 158 | 250 | 136 | 218 | 13
75 | 93 | 146 | 88 | 139 | 4.7 | 139 | 240 | 129 | 210 | 9.8
100 | 76 | 126 | - | - | - | 114 | 195 | - | - | -

Ablation of labeled data percentage. We conducted an ablation study where we
varied the percentage of labeled data used during the training process. More
specifically, we used $4$ different settings: 5%, 25%, 50% and 75%. The
remaining data were used as unlabeled samples. We used ResNet-50 as the
network encoder for all the settings. This ablation study was conducted on 2
datasets: ShanghaiTech-A (SH-A) and UCF-QNRF. The results of this ablation
study are shown in Table 2. It can be observed for both datasets that as the
percentage of labeled data is reduced, the performance of the baseline network
drops significantly. However, the proposed GP-based framework is able to
leverage unlabeled data in all the cases to reduce this performance drop by a
considerable margin. Fig. 3 and 4 show sample qualitative results on
ShanghaiTech-A and UCF-QNRF datasets for the semi-supervised protocol with 5%
labeled data setting. It can be observed that the proposed method is able to
predict the density maps more accurately as compared to the baseline method
that does not consider unlabeled data.

(a) (b) (c) (d)

Figure 3: Results of SSL experiments on the ShanghaiTech-A [60] dataset using
the 5% labeled data setting. (a): Input. (b) No-GP (c) Proposed Method (d)
Ground-truth.

(a) (b) (c) (d)

Figure 4: Results of SSL experiments on the UCF-QNRF [10] dataset using the 5%
labeled data setting. (a): Input. (b) No-GP (c) Proposed Method (d) Ground-
truth. Table 3: Results of ablation study with different networks. The
proposed method is able to exploit unlabeled data irrespective of different
architectures. We used 5% of the training data as labeled set, and the rest as
unlabeled samples. AG: Average Gain %${}^{\ref{ftn:ag}}$.

Net | $\mathcal{D_{L}}$% | SH-A | UCF-QNRF
---|---|---|---
| No-GP($\mathcal{D_{L}}$-only)
---
| GP($\mathcal{D_{L}+D_{U}}$)
---
AG % |  | No-GP ($\mathcal{D_{L}}$-only)
---
| GP ($\mathcal{D_{L}+D_{U}}$)
---
AG %
MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE
ResNet-50 | 100 | 76 | 126 | - | - | - | 114 | 195 | - |  | -
5 | 118 | 211 | 102 | 172 | 16 | 186 | 295 | 160 | 275 | 10
ResNet-101 | 100 | 76 | 117 | - | - | - | 116 | 197 | - | - | -
5 | 131 | 200 | 110 | 162 | 18 | 196 | 324 | 174 | 288 | 11
VGG16 | 100 | 74 | 118 | - | - | - | 120 | 197 | - |  | -
5 | 121 | 205 | 112 | 163 | 14 | 188 | 316 | 175 | 291 | 7.4

Architecture ablation. We conducted an ablation study where we evaluated the
proposed method using different architectures. More specifically, we used
different networks like ResNet-50, ResNet-101 and VGG16 as encoder network.
The ablation was performed on 2 datasets: ShanghaiTech-A (SH-A) and UCF-QNRF.
For all the experiments, we used 5% of the training dataset as labeled
dataset, and the rest were used as unlabeled samples. The results of this
experiment are shown in Table 3. Based on these results, we make the following
observations: (i) Since networks like VGG16 and ResNet-101 have higher number
of parameters, they tend to overfit more in the reduced-data setting as
compared to ResNet-50. (ii) The proposed GP-based method obtains consistent
gains by leveraging unlabeled dataset across different architectures.

Figure 5: Histogram for pseudo-GT errors ($err_{pseudo}^{u}$) and prediction
errors ($err_{pred}^{u}$) on unlabeled data during training. Note that pseudo-
GT errors are concentrated on the lower end, implying that they are more
closer to the ground truth as compared to the predictions. Hence, pseudo-GTs
provide meaningful supervision.

Pseudo-GT Analysis. In order to gain a deeper understanding about the
effectiveness of the proposed approach, we plot the histogram of normalized
errors with respect to the predictions $y_{pred}^{u}$ of the network and the
pseudo-GT $y_{pseudo}^{u}$ for the unlabeled data during the training process.
Specifically, we plot histograms of $err_{pred}^{u}$ and $err_{pseudo}^{u}$,
where $err_{pred}^{u}=\frac{|y_{pred}^{u}-y_{gt}^{u}|}{y_{gt}^{u}}$ and
$err_{pseudo}^{u}=\frac{|y_{pseudo}^{u}-y_{gt}^{u}|}{y_{gt}^{u}}$. Here,
$y_{gt}^{u}$ is the actual GT corresponding to the unlabeled data sample. The
plot is shown in Fig. 5. It can be observed that the pseudo-GT errors are
concentrated in the lower end of the error region as compared to the
prediction errors. This implies that the pseudo-GTs are more closer to the GTs
than the predictions. Hence, the pseudo-GTs obtained using the proposed method
are able to provide good quality supervision on the unlabeled data.

###  5.2 Synthetic-to-Real transfer setting

Recently, Wang _et al._[55] proposed a synthetic crowd counting dataset (GCC)
that consists of 15,212 images with a total of 7,625,843 annotations. The
primary purpose of this dataset is to reduce the annotation efforts by
training the networks on the synthetic dataset, thereby eliminating the need
for labeling. However, due to a gap between the synthetic and real-world data
distributions, the networks trained on synthetic dataset perform poorly on
real-world images. In order to overcome this issue, the authors in [55]
proposed a Cycle-GAN based domain adaptive approach that additionally enforces
SSIM consistency. More specifically, they first learn to translate from
synthetic crowd images to real-world images using SSIM-based Cycle-GAN. This
transfers the style in the synthetic image to more real-world style. The
translated synthetic images are then used to train a counting network (SFCN)
that is based on ResNet-101 architecture.

While this approach improves the error over the baseline methods, its
performance is essentially limited in the case of large distribution gap
between real and synthetic images. Moreover, the authors in [55] perform a
manual selection of synthetic samples for training the network. This
selections ensures that only samples that are closer to the real-world images
in terms of the count are used for training. Such a selection is not feasible
in the case of unsupervised domain adaptation where we have no access to
labels in the target dataset.

Table 4: Comparison of results in synthetic-to-real transfer settings. We
train the network on synthetic crowd counting dataset (GCC), and leverage the
training set of real-world datasets without any labels. We used the same
network as described in [55].

Method | SH-A | SH-B | UCF-QNRF | UCF-CC-50 | WExpo
---|---|---|---|---|---
MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
No Adapt | 160 | 217 | 22.8 | 30.6 | 276 | 459 | 487 | 689 | 42.8
Cycle GAN [62] | 143 | 204 | 24.4 | 39.7 | 257 | 401 | 405 | 548 | 32.4
SE Cycle GAN [55] | 123 | 193 | 19.9 | 28.3 | 230 | 384 | 373 | 529 | 26.3
Proposed Method | 121 | 181 | 12.8 | 19.2 | 210 | 351 | 355 | 505 | 20.4

The proposed GP-based framework overcomes these drawbacks easily and can be
extended to the synthetic-to-real transfer setting as well. We consider the
synthetic data as labeled training set and real-world training set as
unlabeled dataset, and train the network to leverage the unlabeled dataset.
The results of this experiment are reported in Table 4. We used the same
network (SFCN) and training process as described in [55]. As it can be
observed, the proposed method achieves considerable improvements compared to
the recent approach. Since we estimate the pseudo-GT for unlabeled real-world
images and use it as supervision directly, the distribution gap that the
network needs to handle is much lesser. This results in better performance
compared to the domain adaptive approach [55]. Unlike [55], we train the
network on the unlabeled data and hence, we do not need to perform any
synthetic sample selection. Fig. 6 and 7 show sample qualitative results on
the ShanghaiTech-A and UCF-QNRF datasets for the synthetic-to-real transfer
protocol. The proposed method is able to predict the density maps more
accurately as compared to the baseline.

(a) (b) (c) (d)

Figure 6: Results of Synthetic-to-Real transfer experiments on ShanghaiTech-A
dataset. (a): Input. (b) No Adapt (c) Proposed Method (d) Ground-truth.

(a) (b) (c) (d)

Figure 7: Results of Synthetic-to-Real transfer experiments on the UCF-QNRF
[10] dataset. (a): Input. (b) No Adapt. (c) Proposed Method. (d) Ground-truth.

##  6 Conclusions

In this work, we focused on learning to count in the crowd from limited
labeled data. Specifically, we proposed a GP-based iterative learning
framework that involves estimation of pseudo-GT for unlabeled data using
Gaussian Processes, which is then used as supervision for training the
network. Through various experiments, we show that the proposed method can be
effectively used in a variety of scenarios that involve unlabeled data like
learning with less data or synthetic to real-world transfer. In addition, we
conducted detailed ablation studies to demonstrate that the proposed method
generalizes well to different network architectures and is able to achieve
consistent gains for different amounts of labeled data.

## Acknowledgement

This work was supported by the NSF grant 1910141.

## Supplementary Material

Due to limited space in the main paper, we present additional details about
the proposed method and experiments in the supplementary.

### Encoder and Decoder Architecture

Here, we provide details of the encoder and decoder architecture for all the
experiments.

Encoder: In the main paper, we conducted experiments with 4 different networks
for the encoder: For semi-supervised experiments, we used Res50, Res101 and
VGG16. For learning from synthetic data we used Res101-SFCN [55] .Following
are the details:

(i) Res50: First 3 layers of Res50 are used as the encoder.

(ii) Res101: First 3 layers of Res101 are used as the encoder.

(iii) VGG16: First 10 layers of VGG16 are used as the encoder.

(iv) Res101-SFCN: We use the network exactly as described in [55]. In this
network, the layers until final dilated conv layer are considered as a part of
the encoder.

For all the above networks, the features of the final encoder layer are
forwarded through a $1\times 1$ conv layer to reduce the dimensionality to 64
channels. The output of this $1\times 1$ conv is the feature embedding in the
latent space which is used in GP modeling. Since the train crop size is
$256\times 256$, the intermediate feature maps in the latent space is of
dimension $64\times 32\times 32$.

Decoder: We use the same decoder in all the semi-supervised learning
experiments. The decoder consists of 2 conv-relu layers. The first one is a
$3\times 3$ conv layer, that takes in 64 channels and outputs 64 channels. The
final layer is a a $1\times 1$ layer that takes in 64 channels and outputs 1
channel which is the density map. The final conv layer is followed by an
bilinear-upsampling layer that upsamples the output density to the resolution
of the input image.

In case of learning from the synthetic data, since we use the same network as
in [55], all the layers after the dilated conv layers are used as decoder.

### Dataset Details

In this section, we provide details of the different datasets used for
evaluating the proposed method in the main paper.

ShanghaiTech [60]:This dataset contains 1198 annotated images with a total of
330,165 people. This dataset consists of two parts: Part A with 482 images and
Part B with 716 images. Both parts are further divided into training and test
datasets with training set of Part A containing 300 images and that of Part B
containing 400 images. Rest of the images are used as test set.

UCF-QNRF [10]: UCF-QNRF is a large crowd counting dataset with 1535 high-
resolution images and 1.25 million head annotations. There are 1201 training
images and 334 test images. It contains extremely congested scenes where the
maximum count of an image can reach 12865.

WorldExpo [58]: The WorldExpo’10 dataset was introduced by Zhang _et al._.
[58] and it contains 3,980 annotated frames from 1,132 video sequences
captured by 108 surveillance cameras. The frames are divided into training and
test sets. The training set contains 3,380 frames and the test set contains
600 frames from five different scenes with 120 frames per scene. They also
provided Region of Interest (ROI) map for each of the five scenes.

UCSD [4]: The UCSD dataset crowd counting dataset consists of 2000 frames from
a single scene. These scenes contain relatively sparse crowds with the number
of people ranging from 11 to 46 per frame. A region of interest (ROI) is pro-
vided for the scene in the dataset. Of the 2000 frames, frames 601 through
1400 are used for training while the remaining frames are held out for
testing.

GCC [55]:GTA V Crowd Counting Dataset (GCC) is a large-scale synthetic dataset
based on an electronic game, which consists of 15,212 crowd images. GCC
provides three evaluation strategies (random splitting, cross-camera,and
cross-location evaluation).

Table 5: Effect of $\lambda_{un}$ on ShanghaiTech Part-A val set. $\lambda_{un}$ | MAE | MSE
---|---|---
0.0 | 102 | 175
0.2 | 100 | 162
0.4 | 89 | 149
0.6 | 85 | 140
0.8 | 88 | 147
1.0 | 92 | 156

### Hyper-parameter $\lambda_{un}$

In this section, we study the effect of $\lambda_{un}$ on the overall
performance. $\lambda_{un}$ weighs the unsupervised loss function in the Eq.
12 of main paper. For this study, we use the ShanghaiTech A dataset, due to
its wide variety of scenes and diversity in the count. We conducted this
experiment for the 5% data setting where 5% of the data was used as labeled
data and rest was used as unlabeled data. We used Res50 encoder. Note that we
perform the evaluation on the held-out validation set (and not on the test
set). The results for different values of $\lambda_{un}$ are shown in Table 5.

We observed that the performance peaks when the value of $\lambda_{un}$ is
$0.6$. $\lambda_{un}=0$ corresponds to only labeled data. This is the baseline
performance. As we increase $\lambda_{un}$, we observe that the error
improves. However, for $\lambda_{un}>0.6$, we see a small drop. This is
because the network would not have learned to optimal level at the initial
stages of training. Due to this the pseud-GT will be erroneous, and hence,
using high weight for unsupervised at initial stages prohibits the network
from reaching optimal performance.

Based on this experiment, we use $\lambda_{un}=0.6$ for all the experiments.

Table 6: Semi-supervised experiments with recent crowd counting methods. We
used 5% of the training data as labeled set, and the rest as unlabeled
samples. AG: Average Gain %${}^{\ref{ftn:ag}}$.

Net | $\mathcal{D_{L}}$% | SH-A | UCF-QNRF
---|---|---|---
| No-GP($\mathcal{D_{L}}$-only)
---
| GP($\mathcal{D_{L}+D_{U}}$)
---
AG % |  | No-GP ($\mathcal{D_{L}}$-only)
---
| GP ($\mathcal{D_{L}+D_{U}}$)
---
AG %
MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE
Res101-SFCN | 100 | 74 | 114 | - | - | - | 113 | 196 | - | - | -
5 | 128 | 199 | 109 | 160 | 17 | 193 | 323 | 172 | 282 | 12
CSRNet | 100 | 71 | 112 | - | - | - | 123 | 195 | - |  | -
5 | 120 | 200 | 111 | 159 | 14 | 187 | 310 | 171 | 293 | 7.0

Table 7: Results in SSL settings. Reducing labeled data to 5% results in
performance drop by a big margin as compared to 100% data. Res50 was used as
the encoder network for all the methods. RL: Ranking-Loss. GP: Gaussian-
Process. AG: Average Gain %${}^{\ref{ftn:ag}}$.

Method | $\mathcal{D_{L}}$ | $\mathcal{D_{U}}$ | SH-A | SH-B | UCF-QNRF | WExpo | UCSD
---|---|---|---|---|---|---|---
MAE | MSE | AG | MAE | MSE | AG | MAE | MSE | AG | MAE | AG | MAE | MSE | AG
Ours | 5% | 95% | 102 $\pm$ 0.8 | 172 $\pm$ 2.1 | 16 | 15.7 $\pm$ 0.9 | 27.9 ($\pm$ 1.1) | 22 | 160 $\pm$ 2.4 | 275 $\pm$ 3.1 | 10 | 12.8 $\pm$ 0.5 | 10 | 2.0 $\pm$ 0.05 | 2.4 $\pm$ 0.09 | 12

Table 8: Results for synthetic-to-real transfer settings. We train the network
on synthetic crowd counting dataset (GCC), and leverage the training set of
real-world datasets without any labels. We used the same network and
training/evaluation protocol as in [55].

Method | SH-A | SH-B | UCF-QNRF | UCF-CC-50 | WExpo
---|---|---|---|---|---
MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
Ours | 121 $\pm$ 0.6 | 181 $\pm$ 1.6 | 12.8$\pm$ 0.3 | 19.2$\pm$ 0.9 | 210$\pm$ 2.7 | 351$\pm$ 4.1 | 355 $\pm$ 4.4 | 505$\pm$ 5.9 | 20.4 $\pm$ 0.9

### Additional Architecture Ablation

In this section, we conducted additional architecture ablation experiments
using two recent crowd counting techniques: CSRNet [19] and Res101-SFCN [55].
WE use the 5% data-setting, where we use 5% of the data as labeled and rest as
unlabeled. We evaluated both these methods on ShanghaiTech-A (SH-A) and UCF-
QNRF datasets.For CSRNet, we use the layers upto the last dilated conv as the
encoder. For the decoder, we use 2 conv layers as described earlier.

The results of this experiment are shown in Table 6. In addition to MAE/MSE,
we rerport Average Gain (AG)222$AG=\frac{G_{mae}+G_{mse}}{2}$,
$G_{mae}=\frac{mae_{(\mathcal{D_{U}+D_{L}})}-mae_{(\mathcal{D_{L}})}}{mae_{(\mathcal{D_{L}})}}$,
$G_{mse}=\frac{mse_{(\mathcal{D_{U}+D_{L}})}-mse_{(\mathcal{D_{L}})}}{mse_{(\mathcal{D_{L}})}}$.
We observed consistent gains in both the cases when we used the proposed GP-
based method to leverage unlabeled data.

### Multiple Trials

In this section, we report the standard-deviations for the experiments with
our proposed method corresponding to Table 1 and Table 4 in the main paper.
See Table 7 and Table 8. Note that the standard deviations are computed using
5 trials.

## References

  * [1] Arteta, C., Lempitsky, V., Zisserman, A.: Counting in the wild. In: European Conference on Computer Vision. pp. 483–498. Springer (2016)
  * [2] Babu Sam, D., Sajjan, N.N., Venkatesh Babu, R., Srinivasan, M.: Divide and grow: Capturing huge diversity in crowd images with incrementally growing cnn. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3618–3626 (2018)
  * [3] Cao, X., Wang, Z., Zhao, Y., Su, F.: Scale aggregation network for accurate and efficient crowd counting. In: European Conference on Computer Vision. pp. 757–773. Springer (2018)
  * [4] Chan, A.B., Liang, Z.S.J., Vasconcelos, N.: Privacy preserving crowd monitoring: Counting people without people models or tracking. In: Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on. pp. 1–7. IEEE (2008)
  * [5] Chan, A.B., Vasconcelos, N.: Bayesian poisson regression for crowd counting. In: 2009 IEEE 12th International Conference on Computer Vision. pp. 545–551. IEEE (2009)
  * [6] Change Loy, C., Gong, S., Xiang, T.: From semi-supervised to transfer counting of crowds. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2256–2263 (2013)
  * [7] Chen, K., Loy, C.C., Gong, S., Xiang, T.: Feature mining for localised crowd counting. In: European Conference on Computer Vision (2012)
  * [8] Gao, G., Gao, J., Liu, Q., Wang, Q., Wang, Y.: Cnn-based density estimation and crowd counting: A survey. arXiv preprint arXiv:2003.12783 (2020)
  * [9] Idrees, H., Saleemi, I., Seibert, C., Shah, M.: Multi-source multi-scale counting in extremely dense crowd images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2547–2554 (2013)
  * [10] Idrees, H., Tayyab, M., Athrey, K., Zhang, D., Al-Maadeed, S., Rajpoot, N., Shah, M.: Composition loss for counting, density map estimation and localization in dense crowds. In: European Conference on Computer Vision. pp. 544–559. Springer (2018)
  * [11] Jiang, X., Xiao, Z., Zhang, B., Zhen, X., Cao, X., Doermann, D., Shao, L.: Crowd counting and density estimation by trellis encoder-decoder network. arXiv preprint arXiv:1903.00853 (2019)
  * [12] Kang, D., Ma, Z., Chan, A.B.: Beyond counting: Comparisons of density maps for crowd analysis tasks-counting, detection, and tracking. arXiv preprint arXiv:1705.10118 (2017)
  * [13] Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 (2016)
  * [14] Lee, D.H.: Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks
  * [15] Lempitsky, V., Zisserman, A.: Learning to count objects in images. In: Advances in Neural Information Processing Systems. pp. 1324–1332 (2010)
  * [16] Li, M., Zhang, Z., Huang, K., Tan, T.: Estimating the number of people in crowded scenes by mid based foreground segmentation and head-shoulder detection. In: Pattern Recognition, 2008. ICPR 2008. 19th International Conference on. pp. 1–4. IEEE (2008)
  * [17] Li, T., Chang, H., Wang, M., Ni, B., Hong, R., Yan, S.: Crowded scene analysis: A survey. IEEE Transactions on Circuits and Systems for Video Technology 25(3), 367–386 (2015)
  * [18] Li, W., Mahadevan, V., Vasconcelos, N.: Anomaly detection and localization in crowded scenes. IEEE transactions on pattern analysis and machine intelligence 36(1), 18–32 (2014)
  * [19] Li, Y., Zhang, X., Chen, D.: Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1091–1100 (2018)
  * [20] Liu, N., Long, Y., Zou, C., Niu, Q., Pan, L., Wu, H.: Adcrowdnet: An attention-injective deformable convolutional network for crowd understanding. arXiv preprint arXiv:1811.11968 (2018)
  * [21] Liu, W., Salzmann, M., Fua, P.: Context-aware crowd counting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5099–5108 (2019)
  * [22] Liu, X., van de Weijer, J., Bagdanov, A.D.: Leveraging unlabeled data for crowd counting by learning to rank. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
  * [23] Lu, H., Cao, Z., Xiao, Y., Zhuang, B., Shen, C.: Tasselnet: Counting maize tassels in the wild via local counts regression network. Plant Methods 13(1), 79 (2017)
  * [24] Miyato, T., Maeda, S.i., Koyama, M., Ishii, S.: Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence 41(8), 1979–1993 (2018)
  * [25] Oliver, A., Odena, A., Raffel, C.A., Cubuk, E.D., Goodfellow, I.: Realistic evaluation of deep semi-supervised learning algorithms. In: Advances in Neural Information Processing Systems. pp. 3235–3246 (2018)
  * [26] Onoro-Rubio, D., López-Sastre, R.J.: Towards perspective-free object counting with deep learning. In: European Conference on Computer Vision. pp. 615–629. Springer (2016)
  * [27] Pham, V.Q., Kozakaya, T., Yamaguchi, O., Okada, R.: Count forest: Co-voting uncertain number of targets using random forest for crowd density estimation. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3253–3261 (2015)
  * [28] Ranjan, V., Le, H., Hoai, M.: Iterative crowd counting. In: European Conference on Computer Vision. pp. 278–293. Springer (2018)
  * [29] Rasmussen, C.E.: Gaussian processes in machine learning. In: Summer School on Machine Learning. pp. 63–71. Springer (2003)
  * [30] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems. pp. 91–99 (2015)
  * [31] Ryan, D., Denman, S., Fookes, C., Sridharan, S.: Crowd counting using multiple local features. In: Digital Image Computing: Techniques and Applications, 2009\. DICTA’09. pp. 81–88. IEEE (2009)
  * [32] Sam, D.B., Babu, R.V.: Top-down feedback for crowd counting convolutional neural network. In: Thirty-second AAAI conference on artificial intelligence (2018)
  * [33] Sam, D.B., Peri, S.V., Kamath, A., Babu, R.V., et al.: Locate, size and count: Accurately resolving people in dense crowds via detection. arXiv preprint arXiv:1906.07538 (2019)
  * [34] Sam, D.B., Peri, S.V., Mukuntha, N., Babu, R.V.: Going beyond the regression paradigm with accurate dot prediction for dense crowds. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 2853–2861. IEEE (2020)
  * [35] Sam, D.B., Sajjan, N.N., Maurya, H., Babu, R.V.: Almost unsupervised learning for dense crowd counting. In: Thirty-Third AAAI Conference on Artificial Intelligence (2019)
  * [36] Sam, D.B., Surya, S., Babu, R.V.: Switching convolutional neural network for crowd counting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)
  * [37] Shen, Z., Xu, Y., Ni, B., Wang, M., Hu, J., Yang, X.: Crowd counting via adversarial cross-scale consistency pursuit. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
  * [38] Shi, M., Yang, Z., Xu, C., Chen, Q.: Revisiting perspective information for efficient crowd counting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7279–7288 (2019)
  * [39] Shi, Z., Zhang, L., Liu, Y., Cao, X., Ye, Y., Cheng, M.M., Zheng, G.: Crowd counting with deep negative correlation learning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
  * [40] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)
  * [41] Sindagi, V.A., Patel, V.M.: Cnn-based cascaded multi-task learning of high-level prior and density estimation for crowd counting. In: Advanced Video and Signal Based Surveillance (AVSS), 2017 IEEE International Conference on. IEEE (2017)
  * [42] Sindagi, V.A., Patel, V.M.: Generating high-quality crowd density maps using contextual pyramid cnns. In: The IEEE International Conference on Computer Vision (ICCV) (Oct 2017)
  * [43] Sindagi, V.A., Patel, V.M.: A survey of recent advances in cnn-based single image crowd counting and density estimation. Pattern Recognition Letters (2017)
  * [44] Sindagi, V.A., Patel, V.M.: Dafe-fd: Density aware feature enrichment for face detection. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 2185–2195. IEEE (2019)
  * [45] Sindagi, V.A., Patel, V.M.: Ha-ccn: Hierarchical attention-based crowd counting network. arXiv preprint arXiv:1907.10255 (2019)
  * [46] Sindagi, V.A., Patel, V.M.: Inverse attention guided deep crowd counting network. arXiv preprint (2019)
  * [47] Sindagi, V.A., Patel, V.M.: Multi-level bottom-top and top-bottom feature fusion for crowd counting. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1002–1012 (2019)
  * [48] Sindagi, V.A., Yasarla, R., Patel, V.M.: Pushing the frontiers of unconstrained crowd counting: New dataset and benchmark method. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1221–1231 (2019)
  * [49] Sindagi, V.A., Yasarla, R., Patel, V.M.: Jhu-crowd++: Large-scale crowd counting dataset and a benchmark method. arXiv preprint arXiv:2004.03597 (2020)
  * [50] Toropov, E., Gui, L., Zhang, S., Kottur, S., Moura, J.M.: Traffic flow from a low frame rate city camera. In: Image Processing (ICIP), 2015 IEEE International Conference on. pp. 3802–3806. IEEE (2015)
  * [51] Walach, E., Wolf, L.: Learning to count with cnn boosting. In: European Conference on Computer Vision. pp. 660–676. Springer (2016)
  * [52] Wan, J., Chan, A.: Adaptive density map generation for crowd counting. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1130–1139 (2019)
  * [53] Wan, J., Luo, W., Wu, B., Chan, A.B., Liu, W.: Residual regression with semantic prior for crowd counting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4036–4045 (2019)
  * [54] Wang, C., Zhang, H., Yang, L., Liu, S., Cao, X.: Deep people counting in extremely dense crowds. In: Proceedings of the 23rd ACM international conference on Multimedia. pp. 1299–1302. ACM (2015)
  * [55] Wang, Q., Gao, J., Lin, W., Yuan, Y.: Learning from synthetic data for crowd counting in the wild. arXiv preprint arXiv:1903.03303 (2019)
  * [56] Xu, B., Qiu, G.: Crowd density estimation based on rich features and random projection forest. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 1–8. IEEE (2016)
  * [57] Zhan, B., Monekosso, D.N., Remagnino, P., Velastin, S.A., Xu, L.Q.: Crowd analysis: a survey. Machine Vision and Applications 19(5-6), 345–357 (2008)
  * [58] Zhang, C., Li, H., Wang, X., Yang, X.: Cross-scene crowd counting via deep convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 833–841 (2015)
  * [59] Zhang, Q., Chan, A.B.: Wide-area crowd counting via ground-plane density maps and multi-view fusion cnns. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8297–8306 (2019)
  * [60] Zhang, Y., Zhou, D., Chen, S., Gao, S., Ma, Y.: Single-image crowd counting via multi-column convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 589–597 (2016)
  * [61] Zhao, M., Zhang, J., Zhang, C., Zhang, W.: Leveraging heterogeneous auxiliary tasks to assist crowd counting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 12736–12745 (2019)
  * [62] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2223–2232 (2017)


11institutetext: Zentrum für Astronomie der Universität Heidelberg,
Astronomisches Rechen-Institut, Mönchhofstr. 12, 69120 Heidelberg, Germany;
11email<EMAIL_ADDRESS>22institutetext: University of
California Los Angeles, Department of Physics & Astronomy, Los Angeles, CA,
USA 33institutetext: Max-Planck-Institut für Extraterrestrische Physik,
Gießenbachstraße 1, 85748 Garching, Germany 44institutetext: Institute of
Physics, Laboratory of Astrophysics, Ecole Polytechnique Fédérale de Lausanne
(EPFL), 1290 Sauverny, Switzerland

# The Halos and Environments of Nearby Galaxies (HERON) Survey IV: Complexity
in the boxy galaxies NGC 720 and NGC 2768

Andreas J. Koch-Hansen 11 Anna Pasquali 11 R. Michael Rich 22 Ortwin Gerhard
33 Oliver Müller 44

The shapes of galaxies, in particular their outer regions, are important
guideposts to their formation and evolution. Here we report on the discovery
of strongly box-shaped morphologies of the, otherwise well-studied, elliptical
and lenticular galaxies NGC 720 and NGC 2768 from deep imaging. The boxiness
is strongly manifested in the shape parameter $A_{4}/a$ of $-0.04$ in both
objects, and also significant center shifts of the isophotes of $\sim$ 2–4 kpc
are seen. One reason for such asymmetries commonly stated in the literature is
a merger origin, although the number of such cases is still sparse and the
exact properties of the individual boxy objects is highly diverse. Indeed, for
NGC 2768, we identify a progenitor candidate (dubbed Pelops) in the residual
images, which appears to be a dwarf satellite that is currently merging with
NGC 2768. At its absolute magnitude of Mr of $-$12.2 mag, the corresponding
Sersic radius of 2.4 kpc is more extended than those of typical dwarf galaxies
from the literature. However, systematically larger radii are known to occur
in systems that are in tidal disruption. This finding is bolstered by the
presence of a tentative tidal stream feature on archival GALEX data. Finally,
further structures in the fascinating host galaxy comprise rich dust lanes and
a vestigial X-shaped bulge component.

###### Key Words.:

Galaxies: formation — Galaxies: halos — Galaxies: individual: NGC 720, NGC
2768 — Galaxies: interactions — Galaxies: elliptical and lenticular, cD —
Galaxies: structure

##  1 Introduction

The morphology of galaxies holds important clues about their formation and
evolutionary processes, be it in the spiral-vs-elliptical dichtomy,
prominently illustrated in the Hubble tuning fork and its extensions (Hubble,
1926; de Vaucouleurs, 1959), or via including irregular objects that often are
the result of past or ongoing tidal interactions or mergers (Arp, 1966; Tal et
al., 2009).

One particular case are galaxies with isophotes that significantly deviate
from smooth ellipses111 Commonly quantified in terms of the fourth-order
Fourier parameter in an isophote analysis The nomenclature of these parameters
differs amongst the literature. Here, we follow the internal naming of our
used IRAF ellipse task, which denotes the isophote-intensity weighted fourth
moment ($B_{4}/a$) as $A_{4}/a$ (e.g., Bender & Moellenhoff, 1987;
Jedrzejewski, 1987; Bender et al., 1988, 1989) and similar for the third
moments ($B_{3}/a\equiv A_{3}/a$). . The most extreme deviations can tend
towards a “disky” ($A_{4}/a>0$) or a “boxy” ($A_{4}/a<0$) shape. Graham et al.
(2012) report on a rectangular dwarf galaxy (LEDA 074886; $M_{R}=-17.3$ mag)
with a very high negative boxiness parameter ranging from $A_{4}/a$ = $-0.05$
to $-0.08$ between 3 and 5 kpc, which they dubbed the “Emerald Cut Galaxy” (in
the following “ECG”). One possible reason for the ECG’s boxiness discussed in
their work is the edge-on merger of two spiral galaxies. However, Graham et
al. (2012) emphasize that there are only a few highly boxy examples known in
the literature, yet the details of their shapes are diverse, and accordingly
pinning down one single formation channel is unrealistic. This picture has
hardly changed in the literature over the past decade.

Here, we report on the discovery of boxy morphologies in the halos of two,
otherwise well-studied, galaxies in the Local Volume: NGC 720 (E5) and 2768
(E6/S0), each with stellar masses of a few $\times$1011 M⊙ (Rembold et al.,
2005; Forbes et al., 2012; Pastorello et al., 2014). Rich et al. (2019) lists
their halo shape as “boxy” and “round”, respectively, purely based on visual
inspection. Prompted by previous, shallower works that did not detect any
peculiarities in these objects, this begs the question of whether their
boxiness is an intrinsic property of the individual galaxy or if it might
represent the general presence of disks or other substructures (e.g., Pasquali
et al., 2007). To this end, we employ new deep imaging from the “Halos and
Environments of Nearby Galaxies” (HERON) survey (Rich et al., 2019), bolstered
by archival data from the Sloan Digital Sky Survey (SDSS), and the Galaxy
Evolution Explorer (GALEX; Bianchi & GALEX Team 2000; Morrissey et al. 2007)
to investigate the shapes of those two particularly boxy galaxies. This paper
is organized as follows: in Sect. 2, we describe the images that are the basis
of our study. Sect. 3 is dedicated to the structural analysis of the two
galaxies. Finally, we discuss our findings in terms of the formation histories
of either object in Sect. 4.

##  2 Observations: Centurion 28 imaging

Out of the sample of 119 HERON galaxies the two objects of this study were
chosen by eye based on their optical appearance and indications of boxiness.
In the following, we briefly introduce the two data sets employed in our
structural analysis.

The images used in this work were taken as part of the HERON survey and
details of their reduction are given in Rich et al. (2019), alongside a
general characterization of the galaxies, for instance in terms of their halo
sizes. Imaging for the two targets was acquired in Oct. and Nov. 2011 with the
28-inch Centurion (C28) telescope at the Polaris Observatory Association in
Lockwood Valley, California (Rich et al., 2012; Brosch et al., 2015; Koch et
al., 2017; Rich et al., 2019). The pixel scale of the detector is
0.82$\arcsec$ pixel-1, which for NGC 720 has been resampled by a factor of two
in either dimension. This corresponds to 218 and 88 pc per pixel at the
adopted distances of NGC 720 and 2768, respectively222The distances were taken
as 27.38 Mpc and 22.15 Mpc to NGC 720 and NGC2768, respectively (see Rich et
al. 2019 and references therein)..

The fields around the galaxies were exposed for 13$\times$300 s (NGC 720) and
3$\times$300 s (NGC 2768) using a broad-band Astrodon Luminance filter, which
has a bandwidth from 4000 to 7000 Å and thus effectively acts as a wide Sloan
$r$-filter. As a result, the images reach surface brightnesses of 29.9 and
28.9 mag sq.arcsec-1 for NGC 720 and NGC 2768, respectively. The seeing
conditions of the observations were rather low, at 6.4$\arcsec$ for NGC 720
and 3.5$\arcsec$ for NGC 2768. Fig. 1 shows the full C28 images for either
galaxy.

The fundamental coordinate system was attached to the images using the public
service astrometry.net (Lang et al., 2010), which builds on blind pattern
matching. Finally, we obtained the photometric calibration by performing
aperture photometry of stellar sources within the images using SExtractor
(Bertin & Arnouts, 1996), and cross-matching the results to the $r$-band
photometry of the 14th data release of the Sloan Digital Sky Survey (SDSS
DR14; Abolfathi et al. 2018).

Figure 1: Full C28 images of NGC 720 (top panel) and NGC 2768 (bottom panel)
on a linear stretch. North is up, East is left. We also indicate scale bars of
10$\arcmin$ (blue line) and 20 kpc (red line) at the adopted distance to the
galaxies.

##  3 Analysis

We performed an isophotal analysis of both galaxies using IRAF’s ellipse task
(as was also done by Graham et al. 2012), which fits basic ellipse parameters
to the flux, with the option to include higher order parameters, where our
chief focus lies on $A_{4}/a$. To prepare the images, stellar sources were
masked within IRAF and we used a 2$\sigma$-clipping within ellipse to
interpolate over any possible residual flux. The resulting radial profiles of
the key shape parameters are shown in Figs. 2 and 4 for either galaxy down to
3$\sigma$ above the sky, and in the following we discuss the implications for
their boxiness and potential formation scenarios. For the case of the surface
brightness profile, the magnitudes have been corrected for extinction by
$A_{r}=0.036$ mag for NGC 720 and $A_{r}=0.103$ mag for NGC 2768 (Schlafly &
Finkbeiner, 2011).

###  3.1 NGC 720

In the first series of HERON papers, Rich et al. (2019) traced NGC 720 down to
a surface brightness of 29.9 mag arcsec-2 and stated a halo diameter at a
flat-rate 28.0 mag arcsec-2 by ocular inspection. The respective halo “size”
thus extends to 23 times its half-light radius (at 6.7 kpc), and the profile
sampled in this work reaches to about 10 half-light radii. We note, however,
that our profiles are truncated at a surface brightness of 3$\sigma$ above the
background. NGC 720 has been classified as an E5 galaxy with a total mass of
3.29$\times$1011 M⊙ (Rembold et al., 2005) and only little rotation. Also our
ellipticity profile (middle left of Fig. 2) meanders around the corresponding
value of $\epsilon$$\sim$0.4–0.6.

Figure 2: Photometric and morphological parameters of NGC 720 from ellipse,
shown as a function of semi-major axis distance. The half-light radius is
indicated.

Of main interest for our present work is the intensity-weighted mean
boxi-/diskiness parameter $A_{4}/a$. This is typically measured from 2 seeing
radii to 1.5 half-light radii (e.g., Carter, 1978, 1987; Bender et al., 1989;
Hao et al., 2006; Graham et al., 2012). To guide the eye, galaxies that are
labelled “boxy” (excluding dwarfs) have parameters ranging from $-0.02$ to
around zero, with values reported as low as $-$0.04 (e.g., Hao et al., 2006).
The boxiness parameters around $-$0.04 are effectively rare in the literature.
Here, the ECG stands out in having very low values of $-$0.05 down to $-0.08$
between 3 and 5 kpc (Graham et al., 2012). The boxy nature of NGC 720 has
already been noted by Rich et al. (2019) “by eye” and is now quantitatively
confirmed (bottom left panel of Fig. 2), reaching an $A_{4}/a$ of $-0.04$ in
its outskirts, which uniquely classifies this galaxy as a boxy one, albeit to
a lesser extent than the ECG.

Moreover, we find a strongly varying center position for NGC 720 throughout
the annuli, varying by 10–20 px ($\sim$2–4 kpc at the adopted distance). This
is likely due to the same event that caused the boxiness of the isophotes,
which we conjecture to be a merger (see also Sect. 3.2.2). The residual image
(Fig. 3) displays further butterfly-shaped features, which are a common
feature if the disk component is not properly modelled and removed, thus
revealing complexities in the disk such as dust lanes.

Figure 3: Model subtracted image of NGC 720, covering $25\arcmin\times
20\arcmin$. North is up, East is left.

###  3.2 NGC 2768

This galaxy has been traced down to 28.9 mag arcsec-2 by Rich et al. (2019)
and its diameter at the 28 mag arcsec-2 level is reported as 96 kpc,
corresponding to $\sim$13 effective radii (the latter being 7.6 kpc). In turn,
the data under scrutiny here cover approximately six half-light radii (above
3$\sigma$ of the sky) before the background hampered a further meaningful
analysis. NGC 2768 already appears boxy to the eye (Fig. 1), which is
bolstered by the shape profiles in Fig. 4 (bottom right panel). The low values
of $A_{4}/a$ of $-0.04$ in its outer regions render it a clear contender for a
boxy galaxy. Its nature as a purported lenticular to elliptical (E5) galaxy is
also sustained by our derived ellipticity profile (middle left panel of Fig.
4).

Figure 4: Same as Fig. 2, but for NGC 2768.

As for NGC 720, we find a significant center shift across the full isophotes,
amounting to as much as 20 px in $x$ and $y$-direction, which corresponds to
$\sim$1.8 kpc at the used pixel scale and adopted distance to the target
galaxy. In fact, the most pronounced shifts for both galaxies, appears after
200–300 arcsec.

A hint of the peculiarity in this galaxy was already found by Pulsoni et al.
(2018), who measured the kinematics of planetary nebulae out to 5 Re,
resulting in a non-point-symmetric distribution. They found this object to be
a fast rotator out to large radii, and they quantified its asymmetry with
similar parameters to ours (viz., $c_{4}$ and $s_{4}$, accounting for sine and
cosine projections). Pulsoni et al. (2018) judged these asymmetries as “likely
real” as they have also already been seen in the deep optical images of Duc et
al. (2015).

####  3.2.1 A disk in NGC 2768

A prominent dust lane, as visible as black stripes on the model-subtracted
image of NGC 2768 (Fig. 5), had already been noted by Kim (1989) and a hint of
it can also be seen on our original C28 image (Fig. 1) when using a proper
stretch. Moreover, we note the possible presence of a vestigial x-shaped bulge
structure as is also known to exist in the Milky Way (McWilliam & Zoccali,
2010).

Figure 5: Model subtracted image of NGC 2768, covering $20\arcmin\times
13\arcmin$.

An identical isophotal analysis on the HST data as on our C28 imaging reveals
the exact same features. Finally, we note that we have also investigated
archival SDSS images in the $r$-band, which confirm the presence of the dust.

####  3.2.2 The progenitor that built NGC 2768

Fig. 6 is the result of masking not only the stars before running ellipse, but
also masking the dust features mentioned in Sect. 3.2.1. This model subtracted
image clearly shows the presence of a large plume towards the West of NGC
2768’s center, which we consider to be the ongoing merger that caused the
strong distortions of NGC 2768’s isophotes, which we henceforth dub Pelops333
Son of Tantalos. According to Greek mythology, Pelops was “tidally disrupted”
(rather, chopped to pieces) to feast the Gods.. Using GALFIT (Peng et al.,
2002) we fitted a Sersic-profile to the model-subtracted image. The result of
this fit is shown in Fig. 6 and its basic parameters are summarized in Table
1. GALFIT explicitly accounts for the point spread function (PSF) of the
images in its fitting so that the stated radii are the ones obtained after
decovolution with the PSF profile. Similarly, this is considered (internally
within GALFIT) for the error analysis.

Figure 6: Left panel: a $1.5\arcmin\times 1.3\arcmin$ image (thus encompassing
$\sim$4 half-light radii) centered on Pelops. The middle panel shows the best-
fit ($\chi^{2}/\nu$=0.96) GALFIT-model, and the right panel depicts the
residual image. Table 1: Parameters of the NGC 2768 merger candidate, Pelops.

Parameter | Value | Methodaa$a$G, C28: GALFIT values from our C28 images; A, SDSS: Aperture photometry within $r_{e}$ on the SDSS residual images; D: Derived
---|---|---
$\alpha$ | 09:11:24.6 | G, C28
$\delta$ |  $+$60:02:18.2 | G, C28
MR |  $-$12.2 $\pm$ 0.2 | G, C28
$\mu_{e}$ | 22.96 $\pm$ 0.01 | G, C28
$r_{e}$ | 2.4 $\pm$ 0.3 kpc | G, C28
$n$ | 0.37 $\pm$ 0.02 | G, C28
$e$ | 0.55 $\pm$ 0.01 | G, C28
P.A. | 88.4∘ $\pm$ 1.1∘ | G, C28
$u_{0}$ | 19.19 $\pm$ 0.26 | A, SDSS
$g_{0}$ | 17.93 $\pm$ 0.14 | A, SDSS
$r_{0}$ | 17.00 $\pm$ 0.09 | A, SDSS
$i_{0}$ | 16.65 $\pm$ 0.08 | A, SDSS
$z_{0}$ | 16.50 $\pm$ 0.07 | A, SDSS
($g-r$)0 | 0.93$\pm$0.17 | D
($r-i$)0 | 0.35$\pm$0.12 | D
444

Assuming this blob feature (Casey et al., 2023) to be at the same distance as
NGC 2768, we determine its absolute magnitude as M${}_{r}=-12.2\pm$0.12 mag.
The best-fit Sersic index was determined as 0.37$\pm$0.02, which is rather
small and typical of disrupting galaxies and indicates that a simple Sersic-
modeling is not adequate anymore. Furthermore, we determine an axis ratio of
0.55$\pm$0.01 and the position angle of Pelops of 88.4${}^{\circ}\pm
1.09^{\circ}$.

Figure 7: GALEX NUV image of NGC 2768. The location of the Pelops overdensity
identified in Fig. 6 is indicated with a green circle. The purported stream is
highlighted by the red lines in the bottom panel. The image covers
$14\arcmin\times 7\arcmin$.

To find further evidence of the reality of the merging galaxy, we consulted
archival data from the Galaxy Evolution Explorer (GALEX; Bianchi & GALEX Team
2000; Morrissey et al. 2007), which targeted the surroundings of NGC 2768 in
the context of the NGA campaign (Gil de Paz et al., 2007) in Feb. 2005. While
no detection can be made in the Far-ultraviolet (UV) image, the 182-s exposure
in the Near-UV clearly depicts the host galaxy (Fig. 7). Furthermore, we note
the presence of a vestigial stream and possibly an extension into a further
arc to the South of the host galaxy. These are indicated by eye in the bottom
panel of Fig. 7. A natural suspicion is that the structure we see could be a
reflection. However, according to the GALEX
documentation555http://www.galex.caltech.edu/wiki/Public:Documentation/Chapter_8#Ghosts,
ghosts chiefly appear at 30–60$\arcsec$ above and below the bright source
along the $y$-direction on the detector. In our case, the arcs appear at a
much larger separation of 140$\arcsec$ with an even larger extent and a shape
that does not resemble the ghostly donut-shapes. Therefore, we deem it
unlikely, that the purported stream is an artefact.

####  3.2.3 Properties of Pelops from SDSS images

The same feature also stands out in identically model-subtracted SDSS images
in the $g$, $r$, and $i$-bands, confirming that we are most likely seen a real
feature, while being fainter in the $u$\- and $z$-bands. The (lack of) depth
of the SDSS prevents us from obtaining any meaningful structural or
photometric parameters from GALFIT. However, we performed aperture photometry
by simply adding the calibrated flux on each image within one effective
radius, both of the merger candidate and of the host galaxy. The magnitudes
were dereddened using the dust maps of Schlafly & Finkbeiner (2011) and the
extinction law of Cardelli et al. (1989). Here, it worth noticing that both
the $g-r$ and $r-i$ colours of both objects are in very good agreement to
within the (Poisson) errors.

Based on our photometry, we consulted the E-MILES simple stellar population
(SSP) models (Vazdekis et al., 2016), which we computed for a LMC-like
metallicity and with the universal initial mass function of Kroupa (2001) for
53 ages between 0.03 Gyr and 14 Gyr. For each age, we varied the intrinsic
reddening, AV, between 0 and 2 in steps of 0.05. Next, we computed the reduced
$\chi^{2}_{\nu}$ between the predicted SSP colours and the ones measured from
the SDSS images. This results in a best-fit age of $6.5^{+5.5}_{-3.8}$ Gyr and
an intrinsic reddening of $0.15\pm 0.15$ mag. The according mass-to-light
(M/L) ratio in the $r$-band is found to be 1.9${}^{+1.0}_{-0.9}$ in Solar
units, which is rather on the low side for a typical dwarf galaxy (e.g., Koch,
2009). This would imply, adopting the satellite’s absolute magnitude, a total
mass of $\sim$107 M⊙.

####  3.2.4  Pelops in context

The surface brightness at the effective radius is fully in agreement with
those of dwarf galaxies in various environments (Fig. 8, middle panel).
However, at its absolute magnitude this merging candidate appears too large by
a factor of a few, when compared to typical galaxies of similar magnitude. Its
corresponding half-light radius is 2.4$\pm$0.3 kpc, and an investigation of
systematically more extended objects is often used to confirm the presence and
absence of tidal disruptions (Koch et al., 2017). Indeed, the contender within
NGC 2768 lies within $\pm$1.5 mag of the strongly disrupted NGC 4449B (Rich et
al., 2012) and HCC-087 (Koch et al., 2012) at similarly large radii. We note,
however, that the shown literature sample is given in the Johnson-Cousins
R-band (and in parts transformed from Sloan $g$\- and $i$-band magnitudes;
Byun et al. 2020), whereas our value is in Sloan-$r$ and converted from our
luminance measurement so that a slight offset in magnitude can be inherent. We
further note that we adopted a single Sersic profile, which does not account
for tidal features, while other galaxies in the literature may employ other
types of profiles, adding to the discrepancy. Two data points in the bottom
panel Fig. 8 are worth mentioning: firstly, the Local Group dwarf spheroidal
And XIX, which, at M${}_{\rm R}=-9.74$ and R${}_{\rm h}=1.7$ kpc (McConnachie,
2012), stands out in the radius-magnitude diagram. Also this ultra-diffuse
object is a result of tidal interactions with its host galaxy, M31 (Collins et
al., 2020). Secondly, Antlia 2 has been named “an enormous Galactic dwarf
satellite” (Torrealba et al., 2019), the properties of which are also
indicative of a strong tidal evolution. We therefore conclude that the
overdensity is, by effective and absolute magnitude, a common dwarf galaxy in
the process of tidal interactions with the host, NGC 2768.

Figure 8: Location of Pelops (red point) on the magnitude-radius plot (bottom
panel) and in relation with the surface brightness at effective radius (middle
panel) and the Sersic-index (top panel). Data from dwarf galaxies in various
groups and clusters are shown as black points (Chiboucas et al., 2009; Müller
et al., 2015; McConnachie, 2012; Muñoz et al., 2015; Park et al., 2017; Byun
et al., 2020). Indicated in blue and green are the dwarf spheroidals And XIX
and Antlia 2, which are the most extended objects in the Local Group.

To place the merging galaxy in context, we compared its absolute magnitude
with the Local Group dwarf sample (McConnachie, 2012). Albeit given in the
V-band, the satellite to NGC 2768 appears similar to Andromeda II, which,
intriguingly, might be the remnant of a merger in itself (Amorisco et al.,
2014). Our candidate has a luminosity of $\sim$5$\times 10^{6}$ L⊙. Adopting
the M/L-ratio determined above, this results in a mass of the dwarf candidate
on the order of 107 M⊙. If the NGC 2768 merger and And II were of similar type
and the present object had the same mass-to-light-ration as And II ($\sim$20
Côté et al. 1999) this would yield a mass ten times larger, $\sim$108 M⊙. This
compares to the host galaxy’s (disk plus bulge) mass of $\sim 10^{11}$ M⊙
(Forbes et al., 2012). No matter which M/L is used, we are thus facing a minor
merger, which is still seemingly capable of deforming the structure of the
galaxy and inducing many of its morphological and dynamical properties.

##  4 Discussion

Upon visual inspection of the rich HERON dataset, we detected unusually boxy
isophotes of two, otherwise well-studied, galaxies. Such objects are hitherto
rare, and in one case we could even identify a merger candidate, which we
believe has caused the isophotal distortions.

Graham et al. (2012) suggested an edge-on merger of two disk galaxies as the
origin of the ECG, one of the most rectangular galaxies known. In their
scenario, initial gas was driven inwards and formed an inner disk, while
larger radii experienced a dissipationless merger. The “boxiness” parameters
of the two well-known objects measured in our study are not as extreme as for
the ECG ($A_{4}/a$ of $-0.08$ vs. our $-0.04$). While no remnant could be
identified for NGC 720, a merger origin of its morphology is bolstered by
Rembold et al. (2005), who identified this galaxy as an unequal-mass merger
remnant based on its kinematics from longslit spectroscopy.

NGC 2768 has been classified as an E5 galaxy (Pastorello et al., 2014), but
has also been named E6/S0 (Zanatta et al., 2018; Rich et al., 2019). Already
early on, it was found that its gas has different kinematics from the stars in
the inner regions (Fried & Illingworth, 1994), hinting at a dynamically
special history. Similarly, Forbes et al. (2012) performed a bulge/disk
decomposition based on various photometric and kinematic tracers (planetary
nebulae, stars, and globular clusters) and found that the disk of NGC 2768
rotates rapidly, with its velocity dispersion decreasing with radius. In
contrast, the bulge turned out to be pressure supported with only a slow
rotation. As the resulting ratio of the disk’s rotational velocity to its
velocity dispersion resembles that of a spiral galaxy, Forbes et al. (2012)
conclude that NGC 2768 is a transformed late-type galaxy. Similarly, Zanatta
et al. (2018), also using globular clusters and planetary nebulae as tracers,
note that NGC 2768’s red (i.e., old) globular cluster system displays
rotation, most pronounced at inner radi (R$<$1 kpc), indicating that mergers
seem to have played an important role. Overall, lenticular galaxies can evolve
from spiral galaxies via various processes that remove most of their gas and
erase spiral structures (e.g., Byrd & Valtonen, 1990; Bournaud et al., 2005;
Zanatta et al., 2018). Interestingly, also the ECG shows a disk-bar like
structure in its very center, where a solid body rotation indicates the
presence of a central disk (Forbes et al., 2011; Graham et al., 2012).

An obvious question is how frequent galaxies with such strongly boxy
morphologies are in the (Local) Universe. Several similar contenders are
reported in the literature (see Graham et al. 2012 and references therein),
although none of them display such a boxiness as the ECG. For instance,
Bidaran et al. (2020) find boxy isophotes in the Virgo cluster dE galaxy VCC
0608, which also has a severe misalignment between the photometric and
kinematic position angles, indicative of a past merger. As Graham et al.
(2012) discuss, all boxy galaxies have very individually different properties.
Here, different galaxy types are covered, and head-tail structures and warps
have also been identified. As a result, it remains difficult to uniquely
identify one tailor-made mechanism to produce boxy isophotes.

Hao et al. (2006) reported that only 19 out of 847 (i.e., 2.2%) in their
sample of nearby early-type galaxies (elliptical and lenticular) from the SDSS
are boxy, with $-0.02<A_{4}/a<-0.01$, while the remaining 97.8% show disky
isophotes, i.e. $A_{4}/a>-0.01$). This fraction is in agreement with the
number of boxy early-type galaxies in the Virgo Cluster Catalog (Binggeli et
al., 1985, VCC) as identified in Graham et al. (2012) and Bidaran et al.
(2020), which add up to $\sim 3$% the VCC’s population. This highlights that
boxy galaxies are still a rare species that await further detections and
require more in-depth investigations.

###### Acknowledgements.

We thank the anonymous referee for a constructive report. The authors warmly
thank W. Byun for sharing his data on dwarf galaxies. AJKH and AP gratefully
acknowledge funding by the Deutsche Forschungsgemeinschaft (DFG, German
Research Foundation) – Project-ID 138713538 – SFB 881 (“The Milky Way
System”), subprojects A03, A05, A11, B05. OM is grateful to the Swiss National
Science Foundation for financial support under the grant number PZ00P2_202104.

## References

  * Abolfathi et al. (2018) Abolfathi, B., Aguado, D. S., Aguilar, G., et al. 2018, ApJS, 235, 42
  * Amorisco et al. (2014) Amorisco, N. C., Evans, N. W., & van de Ven, G. 2014, Nature, 507, 335
  * Arp (1966) Arp, H. 1966, ApJS, 14, 1
  * Bender et al. (1988) Bender, R., Doebereiner, S., & Moellenhoff, C. 1988, A&AS, 74, 385
  * Bender & Moellenhoff (1987) Bender, R. & Moellenhoff, C. 1987, A&A, 177, 71
  * Bender et al. (1989) Bender, R., Surma, P., Doebereiner, S., Moellenhoff, C., & Madejsky, R. 1989, A&A, 217, 35
  * Bertin & Arnouts (1996) Bertin, E. & Arnouts, S. 1996, A&AS, 117, 393
  * Bianchi & GALEX Team (2000) Bianchi, L. & GALEX Team. 2000, Mem. Soc. Astron. Italiana, 71, 1117
  * Bidaran et al. (2020) Bidaran, B., Pasquali, A., Lisker, T., et al. 2020, MNRAS, 497, 1904
  * Binggeli et al. (1985) Binggeli, B., Sandage, A., & Tammann, G. A. 1985, AJ, 90, 1681
  * Bournaud et al. (2005) Bournaud, F., Jog, C. J., & Combes, F. 2005, A&A, 437, 69
  * Brosch et al. (2015) Brosch, N., Kaspi, S., Niv, S., & Manulis, I. 2015, Ap&SS, 359, 9
  * Byrd & Valtonen (1990) Byrd, G. & Valtonen, M. 1990, ApJ, 350, 89
  * Byun et al. (2020) Byun, W., Sheen, Y.-K., Park, H. S., et al. 2020, ApJ, 891, 18
  * Cardelli et al. (1989) Cardelli, J. A., Clayton, G. C., & Mathis, J. S. 1989, ApJ, 345, 245
  * Carter (1978) Carter, D. 1978, MNRAS, 182, 797
  * Carter (1987) Carter, D. 1987, ApJ, 312, 514
  * Casey et al. (2023) Casey, K. J., Greco, J. P., Peter, A. H. G., & Davis, A. B. 2023, MNRAS
  * Chiboucas et al. (2009) Chiboucas, K., Karachentsev, I. D., & Tully, R. B. 2009, AJ, 137, 3009
  * Collins et al. (2020) Collins, M. L. M., Tollerud, E. J., Rich, R. M., et al. 2020, MNRAS, 491, 3496
  * Côté et al. (1999) Côté, P., Mateo, M., Olszewski, E. W., & Cook, K. H. 1999, ApJ, 526, 147
  * de Vaucouleurs (1959) de Vaucouleurs, G. 1959, Handbuch der Physik, 53, 275
  * Duc et al. (2015) Duc, P.-A., Cuillandre, J.-C., Karabal, E., et al. 2015, MNRAS, 446, 120
  * Forbes et al. (2012) Forbes, D. A., Cortesi, A., Pota, V., et al. 2012, MNRAS, 426, 975
  * Forbes et al. (2011) Forbes, D. A., Spitler, L. R., Graham, A. W., et al. 2011, MNRAS, 413, 2665
  * Fried & Illingworth (1994) Fried, J. W. & Illingworth, G. D. 1994, AJ, 107, 992
  * Gil de Paz et al. (2007) Gil de Paz, A., Boissier, S., Madore, B. F., et al. 2007, ApJS, 173, 185
  * Graham et al. (2012) Graham, A. W., Spitler, L. R., Forbes, D. A., et al. 2012, ApJ, 750, 121
  * Hao et al. (2006) Hao, C. N., Mao, S., Deng, Z. G., Xia, X. Y., & Wu, H. 2006, MNRAS, 370, 1339
  * Hubble (1926) Hubble, E. P. 1926, ApJ, 64, 321
  * Jedrzejewski (1987) Jedrzejewski, R. I. 1987, MNRAS, 226, 747
  * Kim (1989) Kim, D.-W. 1989, ApJ, 346, 653
  * Koch (2009) Koch, A. 2009, Astronomische Nachrichten, 330, 675
  * Koch et al. (2017) Koch, A., Black, C. S., Rich, R. M., et al. 2017, Astronomische Nachrichten, 338, 503
  * Koch et al. (2012) Koch, A., Burkert, A., Rich, R. M., et al. 2012, ApJ, 755, L13
  * Kroupa (2001) Kroupa, P. 2001, MNRAS, 322, 231
  * Lang et al. (2010) Lang, D., Hogg, D. W., Mierle, K., Blanton, M., & Roweis, S. 2010, AJ, 139, 1782
  * McConnachie (2012) McConnachie, A. W. 2012, AJ, 144, 4
  * McWilliam & Zoccali (2010) McWilliam, A. & Zoccali, M. 2010, ApJ, 724, 1491
  * Morrissey et al. (2007) Morrissey, P., Conrow, T., Barlow, T. A., et al. 2007, ApJS, 173, 682
  * Muñoz et al. (2015) Muñoz, R. P., Eigenthaler, P., Puzia, T. H., et al. 2015, ApJ, 813, L15
  * Müller et al. (2015) Müller, O., Jerjen, H., & Binggeli, B. 2015, A&A, 583, A79
  * Park et al. (2017) Park, H. S., Moon, D.-S., Zaritsky, D., et al. 2017, ApJ, 848, 19
  * Pasquali et al. (2007) Pasquali, A., van den Bosch, F. C., & Rix, H. W. 2007, ApJ, 664, 738
  * Pastorello et al. (2014) Pastorello, N., Forbes, D. A., Foster, C., et al. 2014, MNRAS, 442, 1003
  * Peng et al. (2002) Peng, C. Y., Ho, L. C., Impey, C. D., & Rix, H.-W. 2002, AJ, 124, 266
  * Pulsoni et al. (2018) Pulsoni, C., Gerhard, O., Arnaboldi, M., et al. 2018, A&A, 618, A94
  * Rembold et al. (2005) Rembold, S. B., Pastoriza, M. G., & Bruzual, G. 2005, A&A, 436, 57
  * Rich et al. (2012) Rich, R. M., Collins, M. L. M., Black, C. M., et al. 2012, Nature, 482, 192
  * Rich et al. (2019) Rich, R. M., Mosenkov, A., Lee-Saunders, H., et al. 2019, MNRAS, 490, 1539
  * Schlafly & Finkbeiner (2011) Schlafly, E. F. & Finkbeiner, D. P. 2011, ApJ, 737, 103
  * Tal et al. (2009) Tal, T., van Dokkum, P. G., Nelan, J., & Bezanson, R. 2009, AJ, 138, 1417
  * Torrealba et al. (2019) Torrealba, G., Belokurov, V., Koposov, S. E., et al. 2019, MNRAS, 488, 2743
  * Vazdekis et al. (2016) Vazdekis, A., Koleva, M., Ricciardelli, E., Röck, B., & Falcón-Barroso, J. 2016, MNRAS, 463, 3409
  * Zanatta et al. (2018) Zanatta, E. J. B., Cortesi, A., Chies-Santos, A. L., et al. 2018, MNRAS, 479, 5124