# Class-Incremental Learning via Knowledge Amalgamation††thanks: Paper
published at ECML PKDD 2022

Marcus de Carvalho
School of Computer Science and Engineering
Nanyang Technological University
50 Nanyang Ave, Singapore 639798
<EMAIL_ADDRESS>
&Mahardhika Pratama
School of Computer Science and Engineering
University of South Australia
Adelaide, Australia
<EMAIL_ADDRESS>
&Jie Zhang
School of Computer Science and Engineering
Nanyang Technological University
50 Nanyang Ave, Singapore 639798
<EMAIL_ADDRESS>
&Yajuan Sun
Singapore Institute of Manufacturing Technology
2 Fusionopolis Way, Singapore 138634
<EMAIL_ADDRESS>

###### Abstract

Catastrophic forgetting has been a significant problem hindering the
deployment of deep learning algorithms in the continual learning setting.
Numerous methods have been proposed to address the catastrophic forgetting
problem where an agent loses its generalization power of old tasks while
learning new tasks. We put forward an alternative strategy to handle the
catastrophic forgetting with knowledge amalgamation (CFA), which learns a
student network from multiple heterogeneous teacher models specializing in
previous tasks and can be applied to current offline methods. The knowledge
amalgamation process is carried out in a single-head manner with only a
selected number of memorized samples and no annotations. The teachers and
students do not need to share the same network structure, allowing
heterogeneous tasks to be adapted to a compact or sparse data representation.
We compare our method with competitive baselines from different strategies,
demonstrating our approach’s advantages. Source-code: github.com/Ivsucram/CFA

_Keywords_ Continual Learning $\cdot$ Transfer Learning $\cdot$ Knowledge
Distillation.

##  1 Introduction

Computational learning systems driven by the success of deep learning have
obtained great success in several computational data mining and learning
system as computer vision, natural language processing, clustering, and many
more [1]. However, although deep models have demonstrated promising results on
unvarying data, they are susceptible to catastrophic forgetting when applied
to dynamic settings, i.e., new information overwrites past experiences,
leading to a significant drop in performance of previous tasks.

In other words, current learning systems depend on batch setting training,
where the tasks are known in advance, and the training data of all classes are
accessible. When new knowledge is introduced, an entire retraining process of
the network parameters is required to adapt to changes. This becomes
impractical in terms of time and computational power requirements with the
continual introduction of new tasks.

To overcome catastrophic forgetting, learning agents must integrate continuous
new information to enrich the existing knowledge. The model must then prevent
the new information from significantly obstructing the acquired knowledge by
preserving all or most of it. A learning system that continuously learns about
incoming new knowledge consisting of new classes is called a class-incremental
learning agent.

A class-incremental solution showcases three properties:

  1. 1.

It should learn from a data stream that introduces different classes at
different times,

  2. 2.

It should provide a multi-class inference for the learned classes at any
requested time,

  3. 3.

Its computational requirements should be bounded, or grow slowly, to the
number of classes learned.

Many strategies and approaches in the continual learning field attempt to
solve the catastrophic forgetting problem in the class-incremental scenario.
Regularization techniques [5] identify essential parameters for inference of
previous tasks and avoid perturbing them when learning new tasks. Knowledge
distillation methods have also been used [6], where knowledge from previous
tasks and incoming tasks are jointly optimized. Inspired by work in
reinforcement learning, memory replay has also been an important direction
explored by researchers [19], where essential knowledge acquired from previous
experiences is re-used for faster training, or retraining, of a learning
agent.

In this paper, we propose a catastrophic forgetting solution based on
knowledge amalgamation (CFA). Given multiple trained teacher models - each on
a previous task - knowledge amalgamation aims to suppress catastrophic
forgetting by learning a student model that handles all previous tasks in a
single-head manner with only a selected number of memorized samples and no
annotations. Furthermore, the teachers and the students do not need to share
the same structure so that the student can be a compact or sparse
representation of the teachers’ models.

A catastrophic forgetting solution based on the knowledge amalgamation
approach is helpful because it allows heterogeneous tasks to be adapted to a
single-head final model. At the same time, knowledge amalgamation explores the
relationship between the tasks without needing any identifier during the
amalgamation process, being smoothly integrated into already existing learning
pipelines. This approach can be perceived as a post-processing continual
learning solution, where a teacher model is developed for each task and
flexibly combined into a single compact model when inference is required. As a
result, it does not need to maintain specific network architectures for each
task.

Contributions:

  * •

A novel class-incremental learning approach via knowledge amalgamation which:

    * –

Allows teachers and students to present different structures and tasks;

    * –

Integrable into existing learning pipelines (including non-continual ones);

##  2 Related Work

A neural network model needs to learn a series of tasks in sequences in the
continual learning setting. Thus, only data from the current task is available
during training. Furthermore, classes are assumed to be clearly separated. As
a result, catastrophic forgetting occurs when a new task is introduced and the
model loses its generalization power of old tasks through learning.

Currently, there are three scenarios in which a continual learning experiment
can be configured. Task-incremental learning is the easiest of the scenarios,
as a model receives knowledge about which task needs to be processed. In this
scenario, models with task-specific components are the standard, where the
multi-headed output layer network represents the most common solution.

The second scenario, referred to as domain-incremental learning, does not have
task identity available during inference, and models only need to solve the
given task without inferring its task.

Finally, class-incremental learning, the third scenario, requires that the
models must solve each task seen so far while at the same time inferring its
task. The currently proposed method falls into this scenario. Furthermore,
most real-world problems of incrementally learning new classes of objects also
belong to this scenario.

Existing works to handle the continual learning problem are mainly divided
into three categories:

  * •

Structure-based approach: One reason for catastrophic forgetting is that a
neural network’s parameters are optimized for new tasks and no longer for
previous ones. This suggests that not optimizing the entire network or
expanding the internal model structure to deal with the new tasks while
isolating old network parameters could attenuate catastrophic forgetting. PNN
[8] pioneered this approach by adding new components to the network and
freezing old task parameters during training. Context-dependent gating (XdG)
[17] is a simple but popular approach to randomly assigning nodes to tasks.
However, these approaches are limited to the task-incremental learning
scenario by design, as task identity is required to select the correct task-
specific components during training.

  * •

Regularization-based approach: When task knowledge is only available during
training time, training a different part of the network for each task can
still happen, but then the whole network is used through inference. Standard
methods in this approach estimate the importance of the network parameters for
the previously learned tasks and penalize future changes accordingly. Elastic
Weight Consolidation (EWC) [16] and its online counterpart (EWCo) [32] adopt
the Fisher information matrix to estimate the importance of the network
synapses. Synaptic intelligence (SI) [18] utilizes an accumulated gradient to
quantify the significance of the network parameters.

  * •

Memory-based approach: This strategy replays old, or augmented, samples stored
in memory when learning a new task. Learning without forgetting (LWF) [6] uses
a pseudo-data strategy where it labels the samples of the current tasks using
the model trained on the previous tasks, resulting in training that mixes
hard-target (likely category according to previous tasks) with soft-target
(predicted probabilities within all classes). Gradient episodic memory (GEM)
[21] and Averaged GEM (A-GEM) [22] successfully boost continual learning
performance by the usage of exact samples stored in memory to estimate the
forgetting case and to constraint the parameter updates accordingly. Gradient-
based Sample Selection (GSS) [27] focuses on optimizing the selection of
samples to be replayed. Dark Experience Replay (DER/DER++) [28] and Function
Distance Regularization (FDR) [29] use past samples and soft outputs to align
past and current outputs. Hindsight Anchor Learning (HAL) [26] adds additional
objectives into replaying, aiming to reduce forgetting of key learned data
points.

Alternatively, methods can also take advantage of generative models for
pseudo-rehearsal. For example, Deep Generative Replay (DGR) [19] utilizes a
separated generative model sequentially trained on all tasks to generate
samples from their data distribution. Additionally, knowledge distillation can
be combined with DGR (DGR+distill) [20] to pair generated samples with soft
target knowledge.

The proposed CFA is a memory-based approach that presents a novel way to
perform continual learning using knowledge amalgamation, a derivation of
knowledge distillation, and domain adaptation to merge several teacher models
into a single student model.

###  2.1 Domain Adaptation

Transfer learning (TL) [2] is defined by the reuse of a model developed for a
task to improve the learning of another task. Neural networks have been
applied to TL because of their power in representing high-level features.

While there are many sub-topics of TL, we are deeply interested in domain
adaptation (DA) [10]. While there are many approaches to measure and reduce
the disparity between the distributions of these two domains, Maximum Mean
Discrepancy (MMD) [11] and Kullback-Leibler divergence (KL) [12] are widely
used in the literature. Our approach uses KL to approximate the representation
of learning distributions between the teachers and the student, differing from
the original knowledge amalgamation method [3], where the MMD approach is
applied.

###  2.2 Knowledge Distillation

Knowledge distillation (KD) [9] is a method of transferring learning from one
model to the other, usually by compression, where a larger teacher model
supervises the training of a smaller student model. One of the benefits of KD
is that it can handle heterogeneous structures, i.e., the teacher and the
student do not need to share the same network structure. Instead, the teacher
supervises the student training via its logits, also called the soft target.
In other words, KD minimizes the distance between the student network output
$\hat{z}$ and the logits $z$ from a teacher network, generated from an
arbitrary input sample:

$\mathcal{L}_{KD}=||\hat{z}-z||_{2}^{2}$ (1)

Although KD has become a field itself in the machine learning community, many
approaches are still performed under a single teacher-student relationship,
with a sharing task [9]. Contrary to these constraints, our method can process
multiple and heterogeneous teachers, condensing their knowledge into a single
student model covering all tasks.

###  2.3 Knowledge Amalgamation

Knowledge Amalgamation (KA) [3, 4] aims to acquire a compact student model
capable of handling the comprehensive joint objective of multiple teacher
models, each specialized in their task. Our approach extends the concept of
knowledge amalgamation in [24, 3] to the continual learning environment.

##  3 Problem Formulation

In continual learning, within the class-incremental learning scenario, we
experience a stream of data tuples $(x_{i},y_{i})$ that satisfies
$(x_{i},y_{i})\stackrel{{\scriptstyle iid}}{{\sim}}P_{t_{i}}(X,Y)$, containing
an input $x_{i}$ and a target $y_{i}$ organized into sequential tasks
$t_{i}\in\mathcal{T}={1,...T}$, where the total number of tasks $T$ is unknown
a priori. The goal is to learn a predictor
$f:\mathcal{X}\times\mathcal{T}\rightarrow\mathcal{Y}$, which can be queried
at any time to predict the target vector y associated to a test sample $x$,
where $(x,y)\sim P_{t}$. Such test pair can belong to a task that we have
observed in the past or the current task.

We define the knowledge amalgamation task as follows. Assume that we are given
N teacher models ${t}_{i=1}^{N}$ trained a priori, each of which implements a
specific task $T$. Let $\mathcal{D}_{i}$ denote the set of classes handled by
model $t_{i}$. Without loss of generality, we assume
$\mathcal{D}_{i}\neq\mathcal{D}_{j},\forall i\neq j$. In other words, for any
pair of models $t_{i}$ and $t_{j}$, we assume they classify different tasks.
The goal of knowledge amalgamation is to derive a compact single-head student
model that can infer all tasks, in other words, to be able to simultaneously
classify all the classes in $\mathcal{D}=\cup_{i=1}^{N}\mathcal{D}_{i}$. In
other words, the knowledge amalgamation mechanism is done in the post-
processing manner where all teacher models trained to a specific task are
combined into a single model to perform comprehensive classification as per
its teacher models. This approach provides flexibility over existing continual
learning approaches because a teacher model can be independently built for a
specific task. Their knowledge can later be amalgamated into a student model
without loss of generalization power.

##  4 Proposed method

Input: Teacher models $\mathcal{T}_{N}$, Task Memory $\mathcal{M}$, Student
model $\mathcal{S}$, number of epochs

Output: Amalgamated Model $\mathcal{S}$

1

2for  _epoch in epochs_ do

3 M $\leftarrow$ shuffle(M); # Optional

4 for  _sample $m$ in $\mathcal{M}$_ do

5 begin Joint Representation Learning:

6 $\mathcal{L}_{M}=\mathcal{L}_{R}=0$; # Loss initialization

7 $\hat{f}_{S}\leftarrow f_{S}=\leftarrow F_{S}$; # Student encoder

8 for  _$T_{i}$ in $\mathcal{T}_{N}$_ do

9 $\hat{f}_{T_{i}}\leftarrow f_{T_{i}}\leftarrow F_{T_{i}}$; # Teacher encoder

10 $F^{\prime}_{T_{i}}\leftarrow f_{T_{i}}\leftarrow\hat{f}_{T_{i}}$; #
Teacher decoder

11
$\mathcal{L}_{M}$=$\mathcal{L}_{M}+H(\hat{f}_{S},\hat{f}_{T_{i}})-H(\hat{f}_{S})$;
# Eq. 3

12 $\mathcal{L}_{R}$ = $\mathcal{L}_{R}$ \+ $||F^{\prime}_{T_{i}}$ \-
$F_{T_{i}}||_{2}^{2}$; # Eq. 4

13

14

15

16 begin Soft Domain Adaptation:

17 $y_{T}\leftarrow\mathcal{T}_{N}(m)$; # Stacked teachers’ soft output

18 $D_{{\text{KL}_{\text{soft}}}}=H(\hat{y}_{S},y_{T})-H(\hat{y}_{S})$; # Eq.
5

19

20

21 $\mathcal{L}=\alpha
D_{{\text{KL}_{\text{soft}}}}+(1-\alpha)(\mathcal{L}_{M}+\mathcal{L}_{R})$; #
Eq. 6

22 $S_{\theta}=S_{\theta}-\lambda\nabla\mathcal{L}$; # Parameter learning

23

24

Algorithm 1 CFA

In this section, we introduce the proposed CFA and its details. The knowledge
amalgamation element is an extension of [3, 4] and consists of two parts: a
joint representational learning and a soft domain adaptation.

Figure 1: A summarized workflow of the proposed CFA. It consists of two parts:
Joint representation learning and Soft domain adaptation. In joint
representation learning, the features of the teachers (showing two here) and
those to be learned by the students are first transformed into a joint space.
Later on, soft domain adaptation enforces a domain invariant feature space
between the student and teachers via KL. Figure 2: An illustration of the
shared extractor sub-network. A sub-network is shared between the teachers and
the student during the joint representation learning procedure. This shared
extractor aims to create a domain-invariant space via KL, which is then
decompressed back into the student model.

###  4.1 Joint Representation Learning

Figure 1 depicts the joint representation learning scheme. The features of the
teachers and those to be learned from the students are first transformed into
a common feature space, and then two loss terms are minimized. First, a
feature ensemble loss $\mathcal{L}_{M}$ encourages the features of the student
to approximate those of the teachers in the joint space. Then a reconstruction
loss $\mathcal{L}_{R}$ ensures the transformed features can be mapped back to
the original space with minimum possible errors.

####  4.1.1 Adaptation Layer

The adaptation layer aligns the teachers’ and students’ output feature
dimensions via a 1 x 1 convolution kernel [13] that generates a predefined
output length with different input sizes. Let $F_{S}$ and $F_{T_{i}}$ be
respectively the original features of the student and teacher $T_{i}$, and
$f_{S}$ and $f_{T_{i}}$ their respective aligned features. In our
implementation, $f_{S}$ and $f_{T_{i}}$ have the same size of $F_{S}$ and
$F_{T_{i}}$.

####  4.1.2 Shared Extractor

Once the aligned features are derived, a naive approach would be to directly
average the features of the teachers $f_{T_{i}}$ as that of the student
$f_{S}$. However, due to domain discrepancy of the training data and
architectural differences of the teacher networks, the roughly aligned
features may remain heterogeneous. To this end, the teachers and students
share the parameters of a small learnable sub-network, illustrated in Figure
2. This shared extractor consists of three consecutive residual blocks of $1$
stride. It converts $f_{T_{i}}$ and $f_{S}$ into the common representation
spaces $\hat{f}_{T_{i}}$ and $\hat{f}_{S}$. In our implementation,
$\hat{f}_{T_{i}}$ and $\hat{f}_{S}$ is half the size of $f_{T_{i}}$ and
$f_{S}$.

####  4.1.3 Knowledge Amalgamation

To amalgamate knowledge from heterogeneous teachers, we enforce a domain
invariant feature space between the student and teachers via the KL
divergence, computed as follows:

$D_{KL_{i}}(\hat{f}_{S}||\hat{f}_{T_{i}})=H(\hat{f}_{S},\hat{f}_{T_{i}})-H(\hat{f}_{S}),$
(2)

where $H(\hat{f}_{S},\hat{f}_{T_{i}})$ is the cross entropy of
$\hat{f}_{T_{i}}$ and $\hat{f}_{S}$ and $H(\hat{f}_{S})$ is the entropy of
$\hat{f}_{S}$.

We then aggregate all such pairwise KL losses between each teacher and the
student, as shown in Figure 2, and write the overall discrepancy
$\mathcal{L}_{M}$ in the shared space as:

$\mathcal{L}_{M}=\sum_{i=1}^{N}D_{KL_{i}},$ (3)

To further enhance the joint representation learning, we add an autoencoder
[23] reconstruction loss between the original teachers’ feature space. Let
${F^{\prime}_{T_{i}}}$ denote the reconstructed feature of teacher $T_{i}$,
the reconstruction loss $\mathcal{L}_{R}$ is defined as

$\mathcal{L}_{R}=\sum_{i=1}^{N}||F^{\prime}_{T_{i}}-F_{T_{i}}||_{2},$ (4)

###  4.2 Soft Domain Adaptation

Apart from learning the teacher’s features, the student is also expected to
produce identical or similar inferences as the teachers do. We thus also take
the teachers’ predictions by feeding unlabelled input samples to them and then
supervise the student’s training.

We assume that all teacher models handle non-overlapping classes, then
directly stack their score vectors and use them as the student’s target. A
similar strategy can be used for teachers with overlapping classes, where the
logits of repeating classes can be summed or averaged, but we do not explore
it here. Instead of directly applying a cross-entropy loss between the student
output and the teachers’ soft output, as most knowledge distillation solutions
do, we also enforce a domain invariant space into the discriminative (fully-
connected) layers of the student by applying the KL between the student output
and the stacked teachers’ soft output.

Let $y_{T}$ denote the stacked teachers’ soft output and $\hat{y}_{S}$ denote
the corresponding student soft output, then KL is applied as:

$D_{KL_{soft}}(\hat{y}_{S}||y_{T})=H(\hat{y}_{S},y_{T})-H(\hat{y}_{S})$ (5)

###  4.3 Final Loss

We incorporate the loss terms in Eqs 3, 4 and 5 into our final loss function.
The whole framework is trained end-to-end by optimizing the following
objective:

$\mathcal{L}=\alpha
D_{KL_{soft}}(\hat{y}_{S}||y_{T})+(1-\alpha)(\mathcal{L}_{M}+\mathcal{L}_{R})$
(6)

where $\alpha\in[0,1]$ is a hyper-parameter to balance the three terms of the
loss function. By optimizing this loss function, the student network is
trained from the amalgamation of its teachers without annotations.

##  5 Experiments

We evaluate CFA and its baselines under four benchmarks. Then, an ablation
study gives further insight regarding CFA memory usage and internal
procedures. Finally, we executed all CFA experiments using the same structure
for the teachers and students; a ResNet18 backbone [25] as a feature extractor
and two fully-connected layers ahead of it.

###  5.1 Replay Memory

To retrieve proper111Meaning, related to the original data distribution logits
from the teachers, CFA uses the replay memory strategy, where it records some
previous samples to be replayed during the amalgamation process. The nearest-
mean-of-exemplars strategy was used to build the replay memory, but any other
sample selection strategy can be used.

####  5.1.1 Nearest-Mean-of-Exemplars strategy

Consider $t_{i}(x)$ the logits of a teacher $t_{i}$ on a specific task $i$. We
compute the mean exemplar for each class in class $y$ as
$\mu_{y}=\frac{1}{||\mathcal{D}_{i}||}\sum_{x\in\mathcal{D}_{i}}t_{i}(x)$. A
sample $x$ is then added to the memory if there is free space or by descending
sorting out the memory and $x$ by their $L_{2}$ distance.

###  5.2 Baselines setup

We set up two different configurations of CFA. CFA${}_{\text{fixed}}$ uses the
nearest-mean-of-exemplars replay memory strategy with a fixed memory footprint
of $1000$ samples. Meanwhile, CFA${}_{\text{grow}}$ uses the teachers’
confidence replay memory strategy with a growing memory allowing $1000$
samples per task. Hence, CFA${}_{\text{fixed}}$ memory footprint maintains the
same, independent of the number of classes learned so far222Storage of the
original teacher models parameters is still required, usually in secondary
memory, as HDD or SSD., while CFA${}_{\text{grow}}$ memory footprint slowly
grows are more classes are introduced.

All teachers and students have the same architecture, a pre-trained ResNet18
feature-extractor followed by two fully-connected layers, a
$\mathbb{R}^{1000\times 500}$ followed by a
$\mathbb{R}^{500\times\text{output}}$. CFA is optimized under Adam with
learning rate $\lambda=10^{-4}$, hyper-parameter $\alpha=0.5$, and $100$
training epochs.

The other baselines are based on the source-code release by [28]. Their
configuration is also detailed in the supplemental document. The ones which
are memory-based contains a memory budget of $1000$ samples per task, making
them similar to CFA${}_{\text{grow}}$.

All methods have been evaluated using the same computation environment, a
Windows machine with an Intel Core i9-9900K 5.0 GHz with 32GB of main memory
and an Nvidia GeForce 2080 Ti.

###  5.3 Metrics

The continual learning protocol is followed, where we observe three metrics:

$\textbf{\text{Average Accuracy:
}}\text{ACC}=\frac{1}{T}\sum_{i=1}^{T}R_{T,i}$ (7)

$\textbf{\text{Backward Transfer:
}}\text{BWT}=\frac{1}{T-1}\sum_{i=1}^{T-1}R_{T,i}-R_{i,i}$ (8)

$\textbf{\text{Forward Transfer:
}}\text{FWT}=\frac{1}{T-1}\sum_{i=2}^{T}R_{i-1,i}-\bar{b}_{i}$ (9)

where $R\in\mathbf{R}^{TxT}$ is a test classification matrix, where $R_{i,j}$
represents the test accuracy in task $t_{j}$ after completely learn $t_{i}$.
The details are given by [21].

Baseline | Metric (%) | SplitMNIST | SplitCIFAR10 | SplitCIFAR100 | SplitTinyImageNet
---|---|---|---|---|---
EWCo [16, 32] | ACC | 19.13 $\pm$ 0.02 | 18.57 $\pm$ 2.04 | 7.91 $\pm$ 0.79 | 7.39 $\pm$ 0.03
BWT | -97.37 $\pm$ 0.26 | -88.51 $\pm$ 5.18 | -83.80 $\pm$ 1.56 | -71.79 $\pm$ 0.62
FWT | -13.36 $\pm$ 1.38 | -10.09 $\pm$ 5.64 | -1.02 $\pm$ 0.15 | -0.44 $\pm$ 0.16
LWF [6] | ACC | 19.20 $\pm$ 0.06 | 16.27 $\pm$ 4.55 | 9.13 $\pm$ 0.43 | 0.41 $\pm$ 0.20
BWT | -96.52 $\pm$ 0.94 | -89.24 $\pm$ 9.60 | -83.34 $\pm$ 5.57 | -50.33 $\pm$ 3.25
FWT | -11.92 $\pm$ 2.01 | -11.05 $\pm$ 1.88 | 0.16 $\pm$ 0.71 | -0.25 $\pm$ 0.03
ER [30] | ACC | 23.41 $\pm$ 0.60 | 69.07 $\pm$ 3.31 | 27.41 $\pm$ 2.94 | 12.33 $\pm$ 1.23
BWT | -93.83 $\pm$ 0.92 | -24.15 $\pm$ 14.17 | -66.35 $\pm$ 1.22 | -71.07 $\pm$ 2.35
FWT | -8.83 $\pm$ 3.14 | -11.84 $\pm$ 0.40 | -0.98 $\pm$ 0.08 | -0.5 $\pm$ 0.05
AGEM [22] | ACC | 9.19 $\pm$ 0.65 | 13.49 $\pm$ 4.12 | 0.94 $\pm$ 0.32 | 1.55 $\pm$ 0.32
BWT | -40.52 $\pm$ 46.02 | -47.26 $\pm$ -47.26 | 2.99 $\pm$ 5.74 | -14.93 $\pm$ 2.12
FWT | -8.97 $\pm$ 3.54 | -6.06 $\pm$ -6.06 | 0.74 $\pm$ 1.74 | -0.55 $\pm$ 0.20
DER [28] | ACC | 60.85 $\pm$ 2.87 | 72.99 $\pm$ 6.43 | 32.60 $\pm$ 9.77 | 23.62 $\pm$ 3.30
BWT | -42.80 $\pm$ 3.81 | -22.71 $\pm$ 5.00 | -44.64 $\pm$ 8.54 | -52.19 $\pm$ 3.66
FWT | -12.25 $\pm$ 2.71 | -9.36 $\pm$ 8.93 | -0.93 $\pm$ 0.09 | -0.46 $\pm$ 2.12
DER++ [28] | ACC | 72.86 $\pm$ 0.95 | 77.86 $\pm$ 7.59 | 38.82 $\pm$ 8.28 | 23.94 $\pm$ 2.52
BWT | -24.64 $\pm$ 1.21 | -16.27 $\pm$ 5.71 | -49.03 $\pm$ 7.60 | -43.82 $\pm$ 5.95
FWT | -12.59 $\pm$ 0.48 | -6.26 $\pm$ 8.81 | -0.91 $\pm$ 0.07 | -0.26 $\pm$ 2.16
FDR [29] | ACC | 78.08 $\pm$ 3.41 | 48.00 $\pm$ 5.36 | 32.26 $\pm$ 5.51 | 13.30 $\pm$ 1.64
BWT | -21.73 $\pm$ 4.36 | -86.58 $\pm$ 4.37 | -62.87 $\pm$ 5.83 | -67.08 $\pm$ 1.69
FWT | -10.10 $\pm$ 1.13 | -11.41 $\pm$ 2.95 | -0.87 $\pm$ 7.29 | -0.67 $\pm$ 0.22
GSS [27] | ACC | 24.69 $\pm$ 0.80 | 43.96 $\pm$ 2.86 | 13.94 $\pm$ 0.30 | 9.60 $\pm$ 0.84
BWT | -91.69 $\pm$ 1.04 | -55.71 $\pm$ 2.57 | -78.21 $\pm$ 0.32 | -69.36 $\pm$ 0.28
FWT | -10.31 $\pm$ 1.96 | -10.56 $\pm$ 3.30 | -0.39 $\pm$ 0.55 | -0.53 $\pm$ 0.05
HAL [26] | ACC | 88.25 $\pm$ 0.46 | 50.11 $\pm$ 1.18 | 11.00 $\pm$ 2.87 | 3.23 $\pm$ 0.11
BWT | -13.61 $\pm$ 0.62 | -47.01 $\pm$ 2.14 | -44.74 $\pm$ 1.84 | -32.68 $\pm$ 4.10
FWT | -8.81 $\pm$ 3.28 | -11.70 $\pm$ 1.69 | -0.97 $\pm$ 0.26 | -0.24 $\pm$ 0.31
CFA${}_{\text{fixed}}$ (Ours) | ACC | 83.51 $\pm$ 1.35 | 74.96 $\pm$ 0.46 | 27.76 $\pm$ 2.28 | 23.44 $\pm$ 2.55
BWT | -7.95 $\pm$ 1.53 | -14.25 $\pm$ 1.76 | -16.41 $\pm$ 1.49 | -17.58 $\pm$ 1.89
FWT | 69.46 $\pm$ 9.41 | 54.28 $\pm$ 6.57 | 26.91 $\pm$ 3.17 | 20.49 $\pm$ 5.50
CFA${}_{\text{grow}}$ (Ours) | ACC | 89.25 $\pm$ 3.66 | 79.40 $\pm$ 1.15 | 38.74 $\pm$ 3.26 | 32.50 $\pm$ 3.35
BWT | 69.77 $\pm$ 1.31 | 49.00 $\pm$ 2.78 | 11.49 $\pm$ 2.65 | 23.33 $\pm$ 3.45
FWT | 91.77 $\pm$ 5.18 | 65.67 $\pm$ 8.22 | 21.84 $\pm$ 4.26 | 32.58 $\pm$ 5.12
Table 1: Numerical results over five execution runs.

###  5.4 Benchmarks

SplitMNIST is a standard continual learning benchmark that adapts the entire
MNIST problem [15] into five sequential tasks, with a total of 10 classes.

SplitCIFAR10 features the incremental class problem where the full CIFAR10
problem [14] is divided into five sequential tasks, with a total of 10
classes.

SplitCIFAR100 features the incremental class problem where the complete
CIFAR100 problem [14] is divided into 10 sequential subsets, totalling 100
classes.

SplitTinyImageNet features the incremental class problem where 200 classes
from the full ImageNet [31] are resized to 64x64 colored pixels and divided
into 10 sequential tasks.

###  5.5 Numerical results

We compare CFA${}_{\text{fixed}}$ and CFA${}_{\text{grow}}$ against one
regularization-based approaches (EWCo), one knowledge distillation approaches
(LWF333A multi-class implementation was put forward to deal with class-
incremental learning, as in [28]), and six memory-based approaches (AGEM, DER,
DER++, FDR, GSS, HAL).

Table 1 presents a metric summary between the chosen baselines and benchmarks.
It demonstrates that CFA${}_{\text{fixed}}$ and CFA${}_{\text{grow}}$ are
comparable, or even stronger, in comparison with the current state-of-the-art
methods, specially when we take in consideration that CFA${}_{\text{fixed}}$
presents a fixed memory footprint. Furthermore, both CFA${}_{\text{fixed}}$
and CFA${}_{\text{grow}}$ have great BWT and FWT metrics, with
CFA${}_{\text{grow}}$ being the only model providing positive values to all
metrics. This means that CFA signalizes some zero-shot learning [21], although
not explicitly focused here.

Furthermore, the most outstanding achievement of CFA is achieving good
continual learning performance when applied to an offline environment while
maintaining competitive results. In other words, all other methods are fully
continual learning procedures, requiring an organization to shift its entire
learning pipeline from scratch. In contrast, CFA leverages the power of
individual teachers trained on the tasks in an online or offline environment.
This scenario is expected in current organization pipelines, saving costs in
an inevitable paradigm shift from offline to online learning agent
technologies.

###  5.6 Ablation study

####  5.6.1 Memory Analysis

Benchmark | Memory budget | CFA${}_{\text{fixed}}$ | CFA${}_{\text{grow}}$ | DER | DER++ | FDR
---|---|---|---|---|---|---
Split CIFAR10 | 100 | 48.87 $\pm$ 5.26 | 61.36 $\pm$ 2.02 | 46.77 $\pm$ 3.12 | 51.91 $\pm$ 4.21 | 39.60 $\pm$ 4.54
200 | 61.20 $\pm$ 4.35 | 69.33 $\pm$ 1.54 | 58.41 $\pm$ 3.23 | 64.92 $\pm$ 6.15 | 44.49 $\pm$ 4.31
500 | 69.53 $\pm$ 3.30 | 74.63 $\pm$ 1.20 | 65.63 $\pm$ 5.95 | 72.45 $\pm$ 6.85 | 48.20 $\pm$ 5.30
1000 | 74.96 $\pm$ 0.46 | 79.40 $\pm$ 1.15 | 72.99 $\pm$ 6.43 | 77.86 $\pm$ 7.59 | 41.91 $\pm$ 6.42
2000 | 76.45 $\pm$ 1.90 | 82.21 $\pm$ 2.52 | 73.81 $\pm$ 5.12 | 77.44 $\pm$ 8.90 | 47.39 $\pm$ 7.01
Split CIFAR100 | 100 | 7.75 $\pm$ 1.20 | 21.34 $\pm$ 2.30 | 13.23 $\pm$ 0.00 | 22.88 $\pm$ 4.90 | 12.23 $\pm$ 4.50
200 | 12.87 $\pm$ 2.12 | 26.01 $\pm$ 2.53 | 19.98 $\pm$ 0.00 | 23.78 $\pm$ 5.20 | 14.74 $\pm$ 2.24
500 | 21.23 $\pm$ 2.23 | 30.59 $\pm$ 3.54 | 26.53 $\pm$ 5.23 | 31.45 $\pm$ 6.43 | 22.26 $\pm$ 4.21
1000 | 27.76 $\pm$ 2.28 | 38.74 $\pm$ 3.26 | 32.60 $\pm$ 9.77 | 38.82 $\pm$ 8.28 | 32.26 $\pm$ 5.51
2000 | 41.50 $\pm$ 3.45 | 47.54 $\pm$ 3.18 | 36.78 $\pm$ 9.88 | 43.45 $\pm$ 8.54 | 33.12 $\pm$ 5.40
Table 2: ACC(%) metrics over different budget memories.

Table 2 put the strongest baselines face to face to compare how their
accuracies change over different memory budgets. CFA${}_{\text{fixed}}$
maintains a competitive performance, even though it presents a fixed memory
footprint. So, even though it performs similarly to DER, DER++, and FDR, it
benefits from using less memory and being applied to current offline learning
pipelines.

####  5.6.2 Joint representation learning analysis

Description | Split CIFAR10
---|---
$\alpha=1.0$ | $\text{JRL}(\times)\text{SDA}(\checkmark)$ | 35.78 $\pm$ 10.78
$\alpha=0.5$ | $\text{JRL}(\checkmark)\text{SDA}(\checkmark)$ | 74.96 $\pm$ 0.46
$\alpha=0.0$ | $\text{JRL}(\checkmark)\text{SDA}(\times)$ | 69.68 $\pm$ 2.23
Table 3: ACC(%) results with varying hyper-parameter $\alpha$ of the
CFA${}_{\text{fixed}}$ with 1000 of memory budget, controlling the influence
of the Joint Representation Learning (JRL) and Soft Domain Adaptation (SDA)
into its main loss.

As shown in Table 3, Joint Representation Learning (JRL) is the main
adaptation driver, responsible for driving the student’s latent space to
represent different tasks. Furthermore, as we are using the same architecture
for the teachers and students, the difference between $\alpha=1.0$ and
$\alpha=0.5$ is not that significant here but immensely important when dealing
with entire heterogeneous structures, as noted by [3]. When JRL is disabled,
the model has difficulties learning high-feature representations only with the
soft domain adaptation (SDA), resulting in tremendous catastrophic forgetting.

##  6 Conclusion

This paper proposes CFA, an approach to handle catastrophic forgetting for the
class-incremental environment with knowledge amalgamation. CFA can amalgamate
the knowledge of multiple heterogeneous trained teacher models, each for a
previous task, into a single-headed student model capable of handling all
tasks altogether.

We compared CFA with a set of competitive baselines under the class-
incremental learning scenario, yielding positive generalization with excellent
average accuracy and knowledge transfer capabilities, backed by backward and
forward knowledge transfer metrics. At the same time, CFA demonstrated some
zero-shot learning aptitude and handled an enormous number of classes
simultaneously.

CFA presents a novel approach towards continual learning using knowledge
amalgamation, enabling easy integration to current learning pipelines and
shifting from offline to online learning with a performance similar to or
superior to the best of the only-online existing methods. Our approach is
perceived as a post-processing approach of continual learning, distinguishing
itself from existing approaches. Our future work is directed to explore the
continual learning problem in multi-stream environments.

##  7 Acknowledgement

This work is financially supported by National Research Foundation, Republic
of Singapore under IAFPP in the AME domain (contract no.: A19C1A0018).

## References

  * [1] Gama, J.: Knowledge discovery from data streams. CRC Press, Boca Raton, Fl. (2010)
  * [2] Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. on Knowl. and Data Eng. 22(10), 1345–1359 (Oct 2010)
  * [3] Luo, S., Wang, X., Fang, G., Hu, Y., Tao, D., Song, M.: Knowledge amalgamation from heterogeneous networks by common feature learning. In: Kraus, S. (ed.) Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019. pp. 3087–3093. ijcai.org (2019). https://doi.org/10.24963/ijcai.2019/428, https://doi.org/10.24963/ijcai.2019/428
  * [4] Shen, C., Wang, X., Song, J., Sun, L., Song, M.: Amalgamating knowledge towards comprehensive classification. Proceedings of the AAAI Conference on Artificial Intelligence 33(01), 3068–3075 (Jul 2019). https://doi.org/10.1609/aaai.v33i01.33013068, https://ojs.aaai.org/index.php/AAAI/article/view/4165
  * [5] Lee, S.W., Kim, J.H., Jun, J., Ha, J.W., Zhang, B.T.: Overcoming catastrophic forgetting by incremental moment matching. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. p. 4655–4665. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (2017)
  * [6] Li, Z., Hoiem, D.: Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(12), 2935–2947 (2018). https://doi.org/10.1109/TPAMI.2017.2773081
  * [7] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T.: Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114(13), 3521–3526 (2017). https://doi.org/10.1073/pnas.1611835114, https://www.pnas.org/content/114/13/3521
  * [8] Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural networks. ArXiv abs/1606.04671 (2016)
  * [9] Hinton, G., Dean, J., Vinyals, O.: Distilling the knowledge in a neural network. pp. 1–9 (03 2014)
  * [10] , Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., Vaughan, J.W.: A theory of learning from different domains. Mach. Learn. 79(1–2), 151–175 (May 2010)
  * [11] Gretton, A., Borgwardt, K.M., Rasch, M.J., Schölkopf, B., Smola, A.: A kernel two-sample test. Journal of Machine Learning Research 13(25), 723–773 (2012), http://jmlr.org/papers/v13/gretton12a.html
  * [12] Joyce, J.M.: Kullback-Leibler Divergence, pp. 720–722. Springer Berlin Heidelberg, Berlin, Heidelberg (2011). https://doi.org/10.1007/978-3-642-04898-2327, https://doi.org/10.1007/978-3-642-04898-2327
  * [13] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1–9 (2015). https://doi.org/10.1109/CVPR.2015.7298594
  * [14] Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of Toronto (2009)
  * [15] LeCun, Y., Cortes, C.: MNIST handwritten digit database. ATT Labs (2010)
  * [16] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G.,Rusu, A.A., Milan, K., Quan, J., Ramalho, T.: Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114(13), 3521–3526 (2017). https://doi.org/10.1073/pnas.1611835114, https://www.pnas.org/content/114/13/3521
  * [17] Masse, N.Y., Grant, G.D., Freedman, D.J.: Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proceedings of the National Academy of Sciences 115(44), E10467–E10475 (2018). https://doi.org/10.1073/pnas.1803839115, https://www.pnas.org/content/115/44/E10467
  * [18] Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 3987–3995. PMLR (06–11 Aug 2017), https://proceedings.mlr.press/v70/zenke17a.html
  * [19] Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017), https://proceedings.neurips.cc/paper/2017/file/0efbe98067c6c73dba1250d2beaa81f9-Paper.pdf
  * [20] van de Ven, G.M., Tolias, A.S.: Three scenarios for continual learning. ArXiv abs/1904.07734 (2019)
  * [21] Lopez-Paz, D., Ranzato, M.A.: Gradient episodic memory for continual learning. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017), https://proceedings.neurips.cc/paper/2017/file/f87522788a2be2d171666752f97ddebb-Paper.pdf
  * [22] Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M.: Efficient lifelong learning with a-gem. ArXiv abs/1812.00420 (2019)
  * [23] Rumelhart, D.E., McClelland, J.L.: Learning Internal Representations by Error Propagation, pp. 318–362 (1987)
  * [24] Shen, C., Wang, X., Song, J., Sun, L., Song, M.: Amalgamating knowledge towards comprehensive classification. ArXiv abs/1811.02796 (2019)
  * [25] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
  * [26] Chaudhry, A., Gordo, A., Dokania, P.K., Torr, P., Lopez-Paz, D.: Using hindsight to anchor past knowledge in continual learning (2020). https://doi.org/10.48550/ARXIV.2002.08165, https://arxiv.org/abs/2002.08165
  * [27] Aljundi, R., Lin, M., Goujaud, B., Bengio, Y.: Gradient Based Sample Selection for Online Continual Learning. Curran Associates Inc., Red Hook, NY, USA (2019)
  * [28] Buzzega, P., Boschini, M., Porrello, A., Abati, D., Calderara, S.: Dark experience for general continual learning: a strong, simple baseline. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 15920–15930. Curran Associates, Inc. (2020)
  * [29] , Benjamin, A.S., Rolnick, D., Kording, K.: Measuring and regularizing networks in function space (2018). https://doi.org/10.48550/ARXIV.1805.08289, https://arxiv.org/abs/1805.08289
  * [30] Ratcliff, R.: Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review 97 2, 285–308 (1990)
  * [31] Tavanaei, A.: Embedded encoder-decoder in convolutional networks towards explainable ai. ArXiv abs/2007.06712 (2020)
  * [32] Schwarz, J., Czarnecki, W., Luketina, J., Grabska-Barwinska, A., Teh, Y.W., Pascanu, R., Hadsell, R.: Progress amp; compress: A scalable framework for continual learning. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80, pp. 4528–4537. PMLR (10–15 Jul 2018), https://proceedings.mlr.press/v80/schwarz18a.html


11institutetext: University of Guelph, ON, Canada
Memorial University of Newfoundland, NL, Canada
Dalian Maritime University, Dalian, China

# CrowdMLP: Weakly-Supervised Crowd Counting via Multi-Granularity MLP

Mingjie Wang  Jun Zhou  Hao Cai  Minglun Gong

###### Abstract

Existing state-of-the-art crowd counting algorithms rely excessively on
location-level annotations, which are burdensome to acquire. When only count-
level (weak) supervisory signals are available, it is arduous and error-prone
to regress total counts due to the lack of explicit spatial constraints. To
address this issue, a novel and efficient counter (referred to as CrowdMLP) is
presented, which probes into modelling global dependencies of embeddings and
regressing total counts by devising a multi-granularity MLP regressor. In
specific, a locally-focused pre-trained frontend is cascaded to extract crude
feature maps with intrinsic spatial cues, which prevent the model from
collapsing into trivial outcomes. The crude embeddings, along with raw crowd
scenes, are tokenized at different granularity levels. The multi-granularity
MLP then proceeds to mix tokens at the dimensions of cardinality, channel, and
spatial for mining global information. An effective proxy task, namely Split-
Counting, is also proposed to evade the barrier of limited samples and the
shortage of spatial hints in a self-supervised manner. Extensive experiments
demonstrate that CrowdMLP significantly outperforms existing weakly-supervised
counting algorithms and performs on par with state-of-the-art location-level
supervised approaches.

###### Keywords:

Weakly-Supervised Crowd Counting, multi-granularity MLP, Self-supervised Proxy
Task

##  1 Introduction

Object counting strives to estimate the total number of objects dispersing
across still images [35] or dynamic video sequences [25]. It has increasingly
drawn attention from computer vision community, thanks to its wide spread
societal applications, e.g., social distance monitoring [28], traffic
surveillance [48], counting in agriculture [24] and metropolis crowd
management [26].

Figure 1: The categories of crowd counter under distinct supervisory signals.
(a) Learn from an entire set of location-level density maps. (b) Learn from
density maps of partially selected images. (c) Learn from density maps of
partial regions in crowd scenes. (d) Our approach learns directly from less
time-consuming count-level (weak) annotations.

Early counting approaches either adopt body detection-based techniques [12] or
recur to count mapping [5], which perform unsatisfactorily when encountering
highly congested scenes. To evade the issue, Lempitsky et al. [13] introduces
density maps, thereby casting the counting problem into that of predicting
pixel-wise density values. Benefiting from the boom of CNNs, density map
prediction has become the mainstream choice in the realm of crowd counting
[49, 15, 18, 10, 44]; see Figure 1(a).

Albeit the ever-improving counting accuracy, it is notorious that training
such pixel-to-pixel probability predictor demands plenty of location-level
(strong) annotations. The prevailing datasets with such annotations are
generated by manually placing dots at the centroids of people heads across the
whole samples (e.g., 1.51 million dots are labelled for the JHU-CROWD++
dataset [32]), which is tedious and time-consuming. Therefore, a variety of
recent approaches have been proposed to relieve the labelling burden by
employing as few annotated samples as possible. For example, AL-AC [50] uses
limited annotated images via active labelling strategy, whereas other works
[23, 34] design semi-supervised schemes to synthesize pseudo ground truths for
supervising a larger host of unlabelled images; see Figure 1(b). More
recently, Xu et al. [44] make further efforts to reduce the areas of pixel-
wise labels by leveraging fixed partial region in each density map; see Figure
1(c). But even so, the protocol of using scarce location-level supervision has
yet been dispensed with and the requirement of labour-intensive and memory-
consuming dot maps persists.

If we rethink the necessity of location-wise spatial constraints, then
training crowd counters with count-level annotations, i.e. the total number of
objects of interest (see Figure 1(d)), has many merits. These include: _i_)
Training models to predict accurate 2D density maps or dot maps introduces a
domain gap between training and inference phases since the models are
evaluated solely on total counts, impeding the leaning of object-agnostic
repeat patterns; _ii_) 2D density maps are derived from ground-truth dot maps
through a heuristically-defined procedure with inductive bias, which could
confuse the models on what to learn; _iii_) Density map requires up to 1,000
times more memory space compared to the global counts, e.g., 362 MB vs. 364 KB
for the testing set in ShanghaiTech Part A [49]; and _iv_) More essentially,
collecting data with single ground-truth counts can be much easier in many
real-world scenarios. For example, when the same scene is captured by multiple
cameras, the manual count obtained on one view can serve as ground truths for
all captured multi-view images. Furthermore, once an initial count is known
for a scene with controlled access, the ground truths for future frames can be
obtained easily by adding/subtracting numbers of people entering/leaving.
Hence, count-level annotations hold great promises in building large-scale
benchmark datasets for multifarious counting problems.

Recently, a resurge of multi-layer perceptrons (MLP) has lead to cutting-edge
purely MLP-based paradigms for image classification [6, 37], which benefit
from intrinsic advantages of underlying fully-connected layers: more global
receptive fields with less inductive bias than _de facto_ CNNs component,
simpler than self-attention layers in Transformer [38]. Inspired by these
achievements, this paper presents a novel weakly-supervised CrowdMLP
architecture, which exploits global-range receptive fields without considering
the highly-costly self-attention mechanism and attains better performance than
the transformer-based counterpart [16]. As shown in Figure 2, CrowdMLP is
featured by the collaboration between a locally-focused CNN frontend and a
devised global-oriented _multi-granularity MLP regressor_. Low-level
representations with limited receptive fields are extracted by the pre-trained
CNN frontend. These feature maps, along with raw input scenes, are then
reshaped and linearly projected to construct a pyramid of token sequences at
multiple levels of granularity. The holistic MLP regressor proceed to
characterize these tokens and produce embeddings with global-range
dependencies, which are fed into a high-level MLP head to directly predict a
single and continuous count value.

While training with count-level supervisory signals avoids inductive bias, the
lack of location information often causes the performance to degenerate. An
auxiliary counting-specific proxy task, namely _Split-Counting_ , is designed
to stem the degradation by randomly splitting the input into two complementary
parts and enforcing the sum counts for the two parts to be close to the count
for the whole (see Figure 2). This strategy implicitly enhance spatial
awareness in a self-supervised manner, leading to accuracy gains.

In a nutshell, our contributions are summarized as follows:

  1. 1.

A novel CrowdMLP is proposed to mitigate the burdensome location-level
annotations by directly regressing total counts. By proposing multi-
granularity MLP regressor with less inductive bias, global-range receptive
fields at distinct semantics levels are exploited, while circumventing
cumbrous self-attention module in Transformer.

  2. 2.

To avoid performance loss caused by the lack of spatial hints in count-level
supervision, a self-supervised proxy task is introduced to implicitly impose
spatial cues and remedy the accuracy degradation.

  3. 3.

Extensive experiments demonstrate that CrowdMLP offers massive performance
gains over existing weakly-supervised approaches (i.e. CNN- and Transformer-
based counterparts) and competitive results as fully-supervised ones. They
also shed light on the density- and spatial-awareness properties of CrowdMLP.

Figure 2: The overall architecture of CrowdMLP, which is featured by coarse-
to-fine token streams, multi-granularity MLP, and self-supervised proxy task.

##  2 Related Work

###  2.1 Counting with Fully-supervised Settings

Early approaches tackle the problem of crowd counting by detecting body parts
[12, 42, 39, 14] or regressing crowd numbers [5]. However, when handling
highly congested scenes with dense crowd, severe occlusions, and clutter
backgrounds, these traditional methods perform poorly. With the impressive
capability of CNNs [31] and the introduction of density map [13], numerous
CNN-based algorithms are proposed to predict density maps. Among these modern
methods, MCNN [49] employs multiple columns with different kernel sizes to
extract multi-scale features. Switch-CNN [1] uses an auxiliary classification
sub-network to determine which branch should be selected for the regressor at
each image patch. SANet [3] adopts inception module into the feature encoder
and presents a local pattern consistency loss. CSRNet [15] stimulates a line
of studies on enlarging the receptive fields via dilated convolutions. CAN
[20] and SASNet [36] investigate the impacts of attention mechanisms on
accuracy. Furthermore, BL [27], BM-Count [17], and P2PNet [35] propose to
maximize confidence of head centres with only dot annotation.

###  2.2 Counting With Weakly-supervised Settings

For the sake of reducing the burden of annotations, a suite of models attach
importance to weak supervisory signals. L2R [21] is presented to leverage
abundantly-available crowd scenes from the Internet via an auxiliary sorting
task with self-supervised settings. Liu et al. [23] design a self-training
strategy to train a robust feature extractor from unlabelled samples.
Simultaneously, Sindagi et al. [34] incorporate a learning mechanism of
Gaussian process to generate pseudo labels for unlabelled data. Considering
how the model parameters evolve, GWTA-CCNN [29] updates 99.9% of the
parameters in an auto-encoder manner using unlabelled crowd images, whereas
remaining weights of the last two layers are fine-tuned with density map
supervision. Xu et al. [44] use stochastically-selected partial sub-region in
each density map to train the model.

Although the above approaches make use of extra unlabelled images or regions,
location-level dot or density maps supervisions are still decisive and vital
for the final performance. To get rid of such dependency, Yang et al. [45]
move towards implicitly modelling the hidden relationships in count numbers
and predicting the crowd number, which is credited to the side sorting sub-
networks and hand-craft soft labels. Nevertheless, the performance achieved is
unsatisfactory since CNN-based regressor is incompetent to hunt for global
receptive fields. The model is also too sophisticated, making it prone to
overfitting. Different from their work, we probe into a model fed by count-
level supervisory signals from the perspective of multi-granularity token
sequences.

###  2.3 Cutting-edge Learning Paradigms

The past two years witness the focus of preeminent learning paradigms what aim
to compete with _de facto_ CNNs. Transformer-based ViT [7] opens a new avenue
for treating image patches the similar way as word tokens and attains
competitive results on image classification. Furthermore, the fabric-like
models, i.e. DETR [4], shows the effectiveness of combining hybrid convolution
and transformer. More recently, Tolstikhin et al. [37] trigger a line of
research to exploit the intrinsic merits of MLPs. Albeit the splendid
performance in image classification, the potentials for regression still have
not been dived into. Moreover, multi-head attention mechanism [38] has become
the standard component in Transformer and intuitively allows for attending to
varying parts of the token sequence by passing the input token through the
same self-attention unit several times in parallel. Inspired by the merits of
multi-head design, we propose the multi-granularity MLP to enrich token
embeddings. Considering the drastic scale shifts in the crowd counting, we
further refine the plain multi-head method by injecting tokenizations with
distinct patch sizes before individual MLP head, which remarkably differs from
the multiple attention heads with strictly same inputs and multi-level
operations in CNN-based models [33, 9].

##  3 Methodology

As shown in Figure 2, the proposed count-level framework (CrowdMLP) is
comprised of pre-trained CNN frontend, multi-granularity MLP, and self-
supervised proxy task.

###  3.1 Coarse-to-fine Token Streams

As aforementioned, global-range receptive fields are pivotal for count-level
regression due to the lack of local constraints. To this end, we propose an
effective global-oriented MLP regressor which tokenizes the crowd images by
breaking it up into a sequence of patches with lower density levels. Apart
from the global dependencies, limited training samples and drastic scale
variation are prone to impel the model to collapse into trivial outcomes.
Hence, a tokenization structure is designed for marrying the advantages of
locally-focused features (pre-trained CNNs) and global-range relationships
(MLPs) in a hybrid fashion. In specific, given a square input
$I\in\mathcal{R}^{H\times H\times 3}$, where $H\times H$ is image resolution,
we first reshape $I$ into a sequence of raw patches of size $16\times 16$,
denoted as $R_{r}$. Meanwhile, following the wide-spread algorithms [15] for
crowd counting, we incorporate first ten layers of a VGG-16 pre-trained on
ImageNet as the frontend to gather crude and low-level features that embrace
the implicit spatical hints and the property of translation invariance. The
extracted low-level features, with the resolution of
$\frac{H}{8}\times\frac{H}{8}$, go through a 1$\times$1 convolution layer to
shorten channel width and suppress model complexity. The resulted feature maps
are denoted as $C_{I}$.

To remedy the lack of multi-scale cues in MLPs-based models and increase the
cardinality of the model, $C_{I}$ are split with patch sizes of $16\times 16$,
$8\times 8$ and $4\times 4$, respectively, leading to three sets of feature
sequences ($F_{r}$). Then, $F_{r}$ and $R_{r}$ are linearly projected to
construct four token streams $\\{T_{1}\in\mathcal{R}^{(\frac{H}{8\times
16})^{2}\times D},T_{2}\in\mathcal{R}^{(\frac{H}{8\times 8})^{2}\times
D},T_{3}\in\mathcal{R}^{(\frac{H}{8\times 4})^{2}\times
D},T_{4}\in\mathcal{R}^{(\frac{H}{16})^{2}\times D}\\}$ at identical token
dimension $D$, yielding rich coarse-to-fine tokens for further capturing
embeddings with global receptive fields at diverse granularity. The overall
process can be defined as follows:

$\begin{array}[]{l}T_{1}=LP(S_{16\times 16}(F_{r})),T_{2}=LP(S_{8\times
8}(F_{r}))\\\ T_{3}=LP(S_{4\times 4}(F_{r}))\\\ T_{4}=D_{r}(LP(S_{8\times
8}(R_{r}))),\end{array}$ (1)

where $S$ denotes split and reshape operations, whereas $LP$ is linear
projection. 2D Dropout ($D_{r}$) with drop rate of 0.2 is employed for raw
token sequence to make embeddings robust against noise by stochastically
dropping a small set of patches before each epoch.

###  3.2 Multi-granularity MLP Regressor

The MLP-based counter is built upon multiple fundamental transformations,
denoted as $T_{mlp}$, and each of them consists of two plain MLPs (token
mixing and channel mixing). Each kind of mixing operator involves two fully-
connected layers, non-linear ReLU activation function, and batch normalization
$BN$. $T_{mlp}$ can be written as follows:

$\begin{array}[]{l}F_{m}=ReLU(f_{1}(w_{0},X)),~{}~{}F_{m}=Dropout(F_{m})\\\
F_{m}=ReLU(f_{2}(w_{1},F_{m})),F_{m}=Dropout(F_{m})\\\
Y=BN(F_{m}+X),\end{array}$ (2)

where $f_{1}$ and $f_{2}$ indicate two plain fully-connected layers, $w_{0}$
and $w_{1}$ the corresponding learnable parameters, $X$ and $Y$ the input and
output embeddings, and $F_{m}$ the intermediate embedding vector.

While token- and channel-mixing communicate information along spatial and
channel directions, the interaction across different granularities is not yet
considered. This would damage the establishment of complementary embeddings at
diverse semantic levels and lead to one-sided long-range dependencies. To
attenuate this limitation and mine token embeddings with coarse-to-fine global
receptive fields in depth, we present a simple yet effective multi-granularity
MLP structure. To the best of our knowledge, this is the first attempt to
address the issues of scale/density changes through global-oriented multi-
granularity MLP. By imposing this structure on the four input token streams
$T_{p}=\\{T_{1},T_{2},T_{3},T_{4}\\}$, our model allows for assembling
distinct granularities of token sequences covering varying head scales. Four
MLP-based heads $H_{1},H_{2},H_{3},H_{4}$ with depth of 1 are imposed on
$T_{p}$ according to the Eq.(2), and generate a pool of embeddings with multi-
level global receptive fields, denoted as $E_{1},E_{2},E_{3},E_{4}$.
Sequentially, these higher-level embeddings are concatenated as a holistic
embedding vector $\hat{E}$ involving adequate semantics. $\hat{E}$ is then fed
into a higher-capable MLP encoder $M_{top}$ with depth of 3 to refine
representations and to further mix feature hints at dimensions of cardinality,
spatial, and channel. Finally, a light-weigh MLP-based count head $C_{top}$ is
plugged at the top of counter to predict the total count $P_{I}\in R^{1\times
1}$ as a continuous and uncontrolled value. We denote the concatenation as
$\odot$ and the overall procedure is depicted as:

$\small\begin{array}[]{l}E_{1}=H_{1}(T_{1}),~{}E_{2}=H_{2}(T_{2}),~{}E_{3}=H_{3}(T_{3}),~{}E_{4}=H_{4}(T_{4})\\\
\hat{E}=(E_{1}\odot E_{2}\odot E_{3}\odot
E_{4}),~{}P_{I}=C_{top}(M_{top}(\hat{E}))\end{array}$ (3)

###  3.3 Self-supervised Proxy Split-Counting

Recently, self-supervised learning is in vogue in computer vision, which has
lead to scale on a mass of annotation-free training data and capture robust
semantic information. This rule-based learning paradigm provides supervisory
signals by pre-designing pivotal surrogate tasks. Towards the goals of
decoupling the feature spatial hints and regularizing the learning procedure,
we propose a novel proxy task for counting-specific regression, namely _Split-
Counting_. It is designed with the consideration of enforcing the model to
capture global-range spatial cues and regress complementary counts accurately.
Specifically, we randomly select a rectangular region of stochastic resolution
from input $I$ to form a positive decoupling ($I_{P}$) and the complement
region is referred as a negative decoupling ($I_{N}$). The generation of
pairwise decouplings can be formulated as:

$I_{P}=I\times M,I_{N}=I\times(1-M),$ (4)

where $I=I_{N}\cup I_{P}$ and $I_{N}\cap I_{P}=\emptyset$. $M$ is randomly-
generated mask maps with values of 1 or 0, whereas $\times$ represents pixel-
wise multiplication.

The pairwise decouplings ($I_{N},I_{P}$) are taken as the input of our Split-
Counting. For simplicity, we denote three lines of regressions as
$P_{I}=M_{1}(\theta,I)$, $P_{P}=M_{2}(\theta,I_{P})$, and
$P_{N}=M_{3}(\theta,I_{N})$, where $\theta$ indicates the shared parameters of
CrowdMLP and $P$ are the predicted counting results. By doing so, the intra
relationship hidden in a single count value is explicitly decoupled without
resorting to extra modules, and the missing spatial constraint is revived.
Another key benefit is to reduce regression difficulty by implicitly learning
an ensemble ($\hat{M}$) of sub-regressors.

###  3.4 Analysis on Split-Counting

To theoretically demonstrate the impacts of Split-Counting on lowering
empirical error of the entire model, we denote the ensemble regressor as
$\hat{M}$ and three implicit sub-regressors as $M_{1}$, $M_{2}$ and $M_{3}$.
Hence, we have $\hat{M}=\frac{1}{2}(M_{1}+(M_{2}+M_{3}))$. Following the
analysis in DNCL [47] and bias-variance decomposition theory [2], given the
target crowd count $Y$, the prediction error expectation is decomposed into
the combination of bias “$Bias$”, variance “$Var$” and covariance “$Cov$” as
follows:

$\begin{array}[]{l}E[(\hat{M}-Y)^{2}]=Bias(\hat{M},Y)^{2}+Var(\hat{M})\\\
=(E[\hat{M}]-Y)^{2}+E[(\hat{M}-E[\hat{M}])^{2}]\\\
=(\frac{1}{2}((E[M_{1}]-Y)+(E[M_{1}]+E[M_{2}]-Y)))^{2}\\\
+\frac{1}{4}(E[(M_{1}-E(M_{1}))^{2}]+E[(M_{2}+M_{3}-E(M_{2}+M_{3}))^{2}])\\\
+Cov(M_{1},M_{2},M_{3})\\\
=Bias(\hat{M},Y)^{2}+Var(\hat{M})+Cov(M_{1},M_{2},M_{3})\end{array}$ (5)

As for the average regression error of each individual regressor, it is
straightforward to show:

$\begin{array}[]{l}\frac{1}{2}(E[(M_{1}-Y)^{2}]+E[(M_{2}+M_{3}-Y)^{2}])\\\
=Bias(\hat{M},Y)^{2}+2Var(\hat{M})+\frac{1}{2}(Cov(M_{1},\hat{M})+Cov(M_{2}+M_{3},\hat{M}))\end{array}$
(6)

Then, the variance between sub-individuals and the ensemble model can be
transformed as the following equation:

$\begin{array}[]{l}\frac{1}{2}(E[(M_{1}-\hat{M})^{2}]+E[(M_{2}+M_{3}-\hat{M})^{2}])\\\
=2Var(\hat{M})-Var(\hat{M})-Cov(M_{1},M_{2},M_{3})\\\
+\frac{1}{2}(Cov(M_{1},\hat{M})+Cov(M_{2}+M_{3},\hat{M}))\end{array}$ (7)

Finally, the regression error of the ensemble model $\hat{M}$ (see equation
(5)) can be reformulated as the summation of all individual errors and a
negative term:

$\begin{array}[]{l}E[(\hat{M}-Y)^{2}]=\frac{1}{2}(E[(M_{1}-Y)^{2}]+E[(M_{2}+M_{3}-Y)^{2}])\\\
-\frac{1}{2}(E[(M_{1}-\hat{M})^{2}]+E[(M_{2}+M_{3}-\hat{M})^{2}])\\\
=\frac{1}{2}(E[(M_{1}-Y)^{2}]+E[(M_{2}+M_{3}-Y)^{2}])-E[(M_{1}-\hat{M})^{2}]\end{array}$
(8)

This indicates that the quadratic error of the ensemble regressor is
theoretically guaranteed to be less than or equal to the average of the main
baseline and the proxy regressors. Hence, the proposed _Split-Counting_ is
analytically proven to play a positive role in lowering empirical error of the
entire model.

###  3.5 Objective Function

In our framework, we utilize $L_{1}$ loss between the predicted count $P_{I}$
and ground-truth count $C_{gt}$ as the pivotal supervisory signal:
$L_{C}=|P_{I}-C_{gt}|$. In addition, we derive a self-supervised loss
($L_{SS}$) and an integral loss ($L_{I}$) from the proxy task as
regularization constraints; see Figure 2. Given the outputs of the self-
supervised Split-Counting, $P_{P}$ and $P_{N}$, they are defined as:

$L_{SS}=|(P_{P}+P_{N})-P_{I}|,L_{I}=|(P_{P}+P_{N})-C_{gt}|$ (9)

The objective function of our CrowdMLP is formulated as
$L=L_{C}+\frac{1}{2}(L_{SS}+L_{I})$.

##  4 Experiments

###  4.1 Datasets

Four widely-used benchmark datasets are utilized to verify the proposed
method. ShanghaiTech dataset [49] consists of two parts: Part A and Part B.
This dataset includes 1198 crowd images with total 330,165 annotated people.
Classical Part A contains 482 congested internet images while Part B consists
of 716 sparse images with a fixed size of 768$\times$1024 captured from
outside streets. UCF_QNRF [8] benchmark is collected from the website and
includes 1,553 images with a total number of 1,252,642 people, in which 1201
images are used for training and 334 crowd scenes in testing set. JHU-CROWD++
[32] is a large-scale crowd dataset including 4,372 images, in which 2,722
images are taken for training, 1,600 scenes for testing, and 500 samples are
used for validation. It has a total of 1.51 million annotations on centroids
of people and the number of crowd dispersing across crowd scenes ranges from 0
to 25,791. Although NWPU [41] is also a popular large-scale dataset, it does
not provide the ground truths for testing samples. Hence, the performance on
large-scale dataset is evaluated using JUH-CROWD++ here.

###  4.2 Implementation Details

The CrowdMLP is optimized using Adam algorithm with the initial learning rate
of 1e-5. The mini-batch size is set to 12 and the learning rate is degraded by
a MultiStepLR scheduler. The internal token dimension $D$ is fixed as 256 in
our experiments. For all datasets, we first reshape the raw images to keep the
max side of 1024 and the other of 768. During the training phase, we randomly
crop a batch of patches with the fixed size of 256$\times$256 from the
reshaped crowd scenes in an online manner. Stochastic flipping and random
lighting with the probability of 0.5 are chosen as data augmentation
operations. At the inference time, we design a sliding window strategy with
the size of 256$\times$256 to infer a set of sub-counts and their summation is
computed as the final count prediction. Two commonly-used metrics, Mean
Absolute Error (MAE) and Mean Square Error (MSE), are employed for performance
evaluation: $MAE=\frac{1}{N}\sum_{i=1}^{N}|C_{i}-C_{i}^{gt}|$ and
$MSE=\frac{1}{N}\sum_{i=1}^{N}(C_{i}-C_{i}^{gt})^{2}$, where $N$ represents
the number of testing images, $C_{i}^{gt}$ is the ground-truth total number of
people across the $i_{th}$ image and the $C_{i}$ indicates the corresponding
predicted count value.

###  4.3 Comparison with State-of-the-art

As shown in Table 1, on ShanghaiTech Part A, Part B, and JHU-CROWD++ datasets,
CrowdMLP outperforms the previous state-of-the-art weakly-trained approach
(TransCrowd-GAP) significantly, with 19.7%, 24.7%, and 13.3% MSE reductions,
respectively. On UCF-QNRF, the improvement is subtle with 3.2% lower MAE but
1.1% higher MSE.

As a powerful count-level regressor, CrowdMLP also substantially minimizes the
performance gap between count-level and location-level supervised approaches.
On ShanghaiTech Part A, it is slightly behind the best fully-supervised P2PNet
(57.828 _vs._ 52.74 on MAE and 84.412 _vs._ 85.06 on MSE), but outperforms
other approaches. For example, when compared to the up-to-date BL algorithm,
CrowdMLP brings 17.1% and 4.51% MAE reduction on ShanghaiTech Part A and Part
B, respectively. Furthermore, on the large-scale JHU-CROWD++ dataset, which
demands more arduous distribution learning, CrowdMLP performs the best among
all algorithms that have performances reported to-date. This observation
supports our hypothesis that count-level labels provide ample cues for
counting, even though they do not explicitly provide spatial information.

Methods | Label | Part A | Part B | UCF-QNRF | JHU++
---|---|---|---|---|---
Level | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE
ADCrowdNet [19] | Density Map | 63.2 | 98.9 | 7.6 | 13.9 | - | - | - | -
PACNN [30] | Density Map | 62.4 | 102.0 | 7.6 | 11.8 | - | - | - | -
CAN [20] | Density Map | 62.3 | 100.0 | 7.8 | 12.2 | 107 | 183 | 100.1 | 314.0
MBTTBF [33] | Density Map | 60.2 | 94.1 | 8.0 | 15.5 | 97.5 | 165.2 | 81.8 | 299.1
DSSINet [18] | Density Map | 60.63 | 96.04 | 6.85 | 10.34 | 99.1 | 159.2 | 133.5 | 416.5
S-DCNet [43] | Density Map | 58.3 | 95.0 | 6.7 | 10.7 | 104.4 | 176.1 | - | -
ASNet [10] | Density Map | 57.78 | 90.13 | - | - | 91.59 | 159.71 | - | -
AMRNet [22] | Density Map | 61.59 | 98.36 | 7.02 | 11.00 | 86.6 | 152.2 | - | -
DM-Count [40] | Point Map | 59.7 | 95.7 | 7.4 | 11.8 | 85.6 | 148.3 | - | -
BL [27] | Point Map | 62.8 | 101.8 | 7.7 | 12.7 | 88.7 | 154.8 | 75.0 | 299.9
P2PNet [35] | Point Map | 52.74 | 85.06 | 6.25 | 9.9 | 85.32 | 154.5 | - | -
MATT [11] | Count+Density | 80.1 | 129.4 | 11.7 | 17.5 | - | - | - | -
Sorting [45] | Count | 104.6 | 145.2 | 12.3 | 21.2 | - | - | - | -
TransCrowd-T [16] | Count | 69.0 | 116.5 | 10.6 | 19.7 | 98.9 | 176.1 | 76.4 | 319.8
TransCrowd-G [16] | Count | 66.1 | 105.1 | 9.3 | 16.1 | 97.2 | 168.5 | 74.9 | 295.6
CrowdMLP (ours) | Count | 57.828 | 84.412 | 7.667 | 12.127 | 94.130 | 170.324 | 67.568 | 256.154
Table 1:  Quantitative comparisons with state-of-the-art approaches under
different annotation levels on four datasets. Best results under each
annotation level are shown in boldface. Our approach significantly outperforms
existing count-level approaches. It also performs better than most location-
level approaches on the ShanghaiTech Part A dataset, while attaining the best
reported accuracy on JHU-CROWD++.

###  4.4 Ablation Study

Ablation studies are conducted on ShanghaiTech Part A dataset to thoroughly
investigate the effects of each component. For simplicity, the presented
CrowdMLP is denoted as the Baseline.

The Effects of Coarse-to-fine Token Streams. We first ablate the effects of
structuring token sequences at multiple levels of granularity by removing each
sub-encoder. Table 2 shows that removing individual encoders from the pyramid
incurs performance loss, which confirms the importance of the devised multi-
granularity MLP.

Methods | Parameters | Part A
---|---|---
MAE | MSE
w.o. Raw Token | 23.70M | 60.411 | 88.613
w.o. 16$\times$16 Token | 17.65M | 61.896 | 93.894
w.o. 8$\times$8 Token | 23.74M | 61.697 | 87.717
w.o. 4$\times$4 Token | 24.56M | 59.294 | 87.483
Baseline | 26.63M | 57.828 | 84.412
Table 2: Ablation study on multi-granularity structure. The first row removes
the token sequence for raw image patches from the coarse-to-fine token
streams, whereas each of the following 3 rows removes those for feature
patches of corresponding sizes. Performance degradation is observed when any
token sequence is removed.

Using Different Learning Paradigms. Next, we compare the regression capability
of MLPs-based structure against CNNs and Transformer paradigms. A CSRNet-like
CNNs-based model (Crowd-CNN) is constructed for directly counting under the
count-level supervision. For fair comparison, the width and depth of Crowd-CNN
is carefully adjusted to match the number of parameters in the baseline. A
second variant (Crowd-Transformer) is obtained by replacing all MLPs
transformations in the baseline with transformer in ViT that have the same
depth and dimension.

Table 3 shows that Crowd-CNN delivers a seriously degraded performance, which
may be attributed to the fact that CNNs unit is incapable of modelling long-
range receptive fields while focusing on locality [45]. The lower errors
achieved by Crowd-Transformer and CrowdMLP verify the importance of global-
range receptive fields for count-level regressors. Albeit the acceptable
accuracy, Crowd-Transformer suffers from increased model scale (36.43M _vs._
26.63M) and high computational complexity. Our observation is consistent with
the recent $S^{2}$-MLPv2 [46] for image classification.

Methods | Parameters | Part A
---|---|---
MAE | MSE
Crowd-CNN | 26.63M | 159.925 | 251.649
Crowd-Transformer | 36.43M | 61.121 | 92.922
Baseline | 26.63M | 57.828 | 84.412
Table 3: The effects of different regression paradigms involving CNNs,
Transformers, and MLPs.  Figure 3: Cluster visualization of testing scenes in
ShanghaiTech Part A by imposing T-SNE on learned intermediate features. Scenes
with varying attributes fall into different clusters. From left to right, the
zoomed-in views show examples from selected clusters. Figure 4: Visualization
on sliding window counting results on a crowd scene example with drastic
scale, density and semantics shifts across the whole image. The total
predicted count is 718.4, whereas the ground truth is 717. Figure 5:
Embeddings cluster visualization of testing images in ShanghaiTech Part A
using T-SNE technique. Each image is shown as a colour dot, where the colour
represents the crowd number/densities. While the images are clustered
primarily by scene features (see Figure 3), high-density images tend to group
together (green cluster). The zoomed-in view depicts corresponding real scenes
in green cluster.

The Effect of Proxy Split-Counting. To evaluate the impacts of the proxy task,
we also train a variant of CrowdMLP without the self-supervised _Split-
Counting_ on ShanghaiTech Part A. The comparison shows that Split-Counting
helps to bring 0.55% improvement on MAE and 0.32% improvement on MSE. This
confirms our theoretical analysis that the proxy task is beneficial for
implicitly decoupling spatial constraints and lowering the empirical error.

###  4.5 Visualization and Analysis

To discern learned features in depth under the count-level supervisions and
investigate whether or not CrowdMLP, trained without any location signals, is
aware of scale variation, density shift, and spatial semantics, we visualize
the embeddings using T-SNE technique and present a typical inference example
with drastic scale/density/semantics changes across the whole image. The
clustering results in Figures 3 and 5 show that CrowdMLP learns discriminative
features with underlying repetitive patterns for diverse scenes. Photos with
similar properties appear to be grouped into same clusters. For example, on
the leftmost (blue) cluster, we have street photos with large scale variation
where crowd only show up at the bottom of the images, whereas on the rightmost
(red) cluster, we have overhead views with uniformly distributed crowd.

Simultaneously, Figure 4 exhibits the behaviour of sliding-window strategy
with the size of 256$\times$256 at the testing phase. It can be observed that,
although lacking location-level prior, our weakly-supervised CrowdMLP can
distinguish spatial semantics (the background and foreground people). For the
common issues of scale variations and density shifts, our model seems to work
pretty well and predict people number precisely under the variation trend of
scale and density; see red and blue dotted regions in the Figure 4.

###  4.6 Conclusion

In this paper, a novel weakly-supervised CrowdMLP is presented to mitigate the
burden of the labour-intensive location-level annotations and learn to
directly regress crowd counts using only count-level labels. In specific, the
multi-granularity MLP regressor are creatively designed to capture global
receptive fields at varying levels of granularity. Besides, a promising proxy
task, namely split-counting, is introduced for regularizing the learning
procedure of the crowd regressor, implicitly decoupling spatial constraints,
and avoiding performance degradation. The split-counting also brings
potentials for exploiting massive unlabelled or cross-domain data. Extensive
experiments have demonstrated the effectiveness of the presented method. For
future work, we plan to explore the potential of CrowdMLP for cross-domain
counting and/or domain adaptation.

## References

  * [1] Babu Sam, D., Surya, S., Venkatesh Babu, R.: Switching convolutional neural network for crowd counting. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5744–5752 (2017)
  * [2] Brown, G., Wyatt, J.L., Tino, P., Bengio, Y.: Managing diversity in regression ensembles. Journal of machine learning research 6(9) (2005)
  * [3] Cao, X., Wang, Z., Zhao, Y., Su, F.: Scale aggregation network for accurate and efficient crowd counting. In: Proceedings of the European conference on computer vision (ECCV). pp. 734–750 (2018)
  * [4] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European conference on computer vision. pp. 213–229. Springer (2020)
  * [5] Chan, A.B., Vasconcelos, N.: Bayesian poisson regression for crowd counting. In: 2009 IEEE 12th international conference on computer vision. pp. 545–551. IEEE (2009)
  * [6] Chen, S., Xie, E., et al.: Cyclemlp: A mlp-like architecture for dense prediction. arXiv preprint arXiv:2107.10224 (2021)
  * [7] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [8] Idrees, H., Tayyab, M., Athrey, K., Zhang, D., Al-Maadeed, S., Rajpoot, N., Shah, M.: Composition loss for counting, density map estimation and localization in dense crowds. In: Proceedings of the european conference on computer vision (ECCV). pp. 532–546 (2018)
  * [9] Jiang, X., Zhang, L., Lv, P., Guo, Y., Zhu, R., Li, Y., Pang, Y., Li, X., Zhou, B., Xu, M.: Learning multi-level density maps for crowd counting. IEEE transactions on neural networks and learning systems 31(8), 2705–2715 (2019)
  * [10] Jiang, X., Zhang, L., Xu, M., Zhang, T., Lv, P., Zhou, B., Yang, X., Pang, Y.: Attention scaling for crowd counting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4706–4715 (2020)
  * [11] Lei, Y., Liu, Y., et al.: Towards using count-level weak supervision for crowd counting. PR (2021)
  * [12] Leibe, B., Seemann, E., Schiele, B.: Pedestrian detection in crowded scenes. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05). vol. 1, pp. 878–885. IEEE (2005)
  * [13] Lempitsky, V., Zisserman, A.: Learning to count objects in images. Advances in neural information processing systems 23 (2010)
  * [14] Li, M., Zhang, Z., Huang, K., Tan, T.: Estimating the number of people in crowded scenes by mid based foreground segmentation and head-shoulder detection. In: 2008 19th international conference on pattern recognition. pp. 1–4. IEEE (2008)
  * [15] Li, Y., Zhang, X., Chen, D.: Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1091–1100 (2018)
  * [16] Liang, D., Chen, X., et al.: Transcrowd: Weakly-supervised crowd counting with transformer. arXiv preprint arXiv:2104.09116 (2021)
  * [17] Liu, H., Zhao, Q., Ma, Y., Dai, F.: Bipartite matching for crowd counting with point supervision. In: International Joint Conference on Artificial Intelligence (2021)
  * [18] Liu, L., Qiu, Z., Li, G., Liu, S., Ouyang, W., Lin, L.: Crowd counting with deep structured scale integration network. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1774–1783 (2019)
  * [19] Liu, N., Long, Y., Zou, C., Niu, Q., Pan, L., Wu, H.: Adcrowdnet: An attention-injective deformable convolutional network for crowd understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3225–3234 (2019)
  * [20] Liu, W., Salzmann, M., Fua, P.: Context-aware crowd counting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5099–5108 (2019)
  * [21] Liu, X., Van De Weijer, J., Bagdanov, A.D.: Leveraging unlabeled data for crowd counting by learning to rank. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7661–7669 (2018)
  * [22] Liu, X., Yang, J., Ding, W., Wang, T., Wang, Z., Xiong, J.: Adaptive mixture regression network with local counting map for crowd counting. In: European Conference on Computer Vision. pp. 241–257. Springer (2020)
  * [23] Liu, Y., Liu, L., Wang, P., Zhang, P., Lei, Y.: Semi-supervised crowd counting via self-training on surrogate tasks. In: European Conference on Computer Vision. pp. 242–259. Springer (2020)
  * [24] Lu, H., Cao, Z., Xiao, Y., Zhuang, B., Shen, C.: Tasselnet: counting maize tassels in the wild via local counts regression network. Plant methods 13(1), 1–17 (2017)
  * [25] Ma, Y.J., Shuai, H.H., Cheng, W.H.: Spatiotemporal dilated convolution with uncertain matching for video-based crowd estimation. IEEE Transactions on Multimedia (2021)
  * [26] Ma, Z., Hong, X., Wei, X., Qiu, Y., Gong, Y.: Towards a universal model for cross-dataset crowd counting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3205–3214 (2021)
  * [27] Ma, Z., Wei, X., Hong, X., Gong, Y.: Bayesian loss for crowd count estimation with point supervision. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6142–6151 (2019)
  * [28] Punn, N.S., Sonbhadra, S.K., Agarwal, S., Rai, G.: Monitoring covid-19 social distancing with person detection and tracking via fine-tuned yolo v3 and deepsort techniques. arXiv preprint arXiv:2005.01385 (2020)
  * [29] Sam, D.B., Sajjan, N.N., Maurya, H., Babu, R.V.: Almost unsupervised learning for dense crowd counting. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 8868–8875 (2019)
  * [30] Shi, M., Yang, Z., Xu, C., Chen, Q.: Revisiting perspective information for efficient crowd counting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7279–7288 (2019)
  * [31] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [32] Sindagi, V., Yasarla, R., Patel, V.M.: Jhu-crowd++: Large-scale crowd counting dataset and a benchmark method. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020)
  * [33] Sindagi, V.A., Patel, V.M.: Multi-level bottom-top and top-bottom feature fusion for crowd counting. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1002–1012 (2019)
  * [34] Sindagi, V.A., Yasarla, R., Babu, D.S., Babu, R.V., Patel, V.M.: Learning to count in the crowd from limited labeled data. In: European Conference on Computer Vision. pp. 212–229. Springer (2020)
  * [35] Song, Q., Wang, C., Jiang, Z., Wang, Y., Tai, Y., Wang, C., Li, J., Huang, F., Wu, Y.: Rethinking counting and localization in crowds: A purely point-based framework. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3365–3374 (2021)
  * [36] Song, Q., Wang, C., Wang, Y., Tai, Y., Wang, C., Li, J., Wu, J., Ma, J.: To choose or to fuse? scale selection for crowd counting. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 2576–2583 (2021)
  * [37] Tolstikhin, I.O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al.: Mlp-mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems 34 (2021)
  * [38] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [39] Viola, P., Jones, M.J., Snow, D.: Detecting pedestrians using patterns of motion and appearance. International Journal of Computer Vision 63(2), 153–161 (2005)
  * [40] Wang, B., Liu, H., Samaras, D., Nguyen, M.H.: Distribution matching for crowd counting. Advances in Neural Information Processing Systems 33, 1595–1607 (2020)
  * [41] Wang, Q., Gao, J., Lin, W., Li, X.: Nwpu-crowd: A large-scale benchmark for crowd counting and localization. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020). https://doi.org/10.1109/TPAMI.2020.3013269
  * [42] Wu, B., Nevatia, R.: Detection of multiple, partially occluded humans in a single image by bayesian combination of edgelet part detectors. In: Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1. vol. 1, pp. 90–97. IEEE (2005)
  * [43] Xiong, H., Lu, H., Liu, C., Liu, L., Cao, Z., Shen, C.: From open set to closed set: Counting objects by spatial divide-and-conquer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8362–8371 (2019)
  * [44] Xu, Y., Zhong, Z., Lian, D., Li, J., Li, Z., Xu, X., Gao, S.: Crowd counting with partial annotations in an image. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15570–15579 (2021)
  * [45] Yang, Y., Li, G., Wu, Z., Su, L., Huang, Q., Sebe, N.: Weakly-supervised crowd counting learns from sorting rather than locations. In: European Conference on Computer Vision. pp. 1–17. Springer (2020)
  * [46] Yu, T., Li, X., et al.: S2mlpv2: Improved spatial-shift mlp architecture for vision. arXiv preprint arXiv:2108.01072 (2021)
  * [47] Zhang, L., Shi, Z., Cheng, M.M., Liu, Y., Bian, J.W., Zhou, J.T., Zheng, G., Zeng, Z.: Nonlinear regression via deep negative correlation learning. IEEE transactions on pattern analysis and machine intelligence 43(3), 982–998 (2019)
  * [48] Zhang, S., Wu, G., Costeira, J.P., Moura, J.M.: Fcn-rlstm: Deep spatio-temporal neural networks for vehicle counting in city cameras. In: Proceedings of the IEEE international conference on computer vision. pp. 3667–3676 (2017)
  * [49] Zhang, Y., Zhou, D., Chen, S., Gao, S., Ma, Y.: Single-image crowd counting via multi-column convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 589–597 (2016)
  * [50] Zhao, Z., Shi, M., Zhao, X., Li, L.: Active crowd counting with limited supervision. In: European Conference on Computer Vision. pp. 565–581. Springer (2020)