# Fast-ParC: Position Aware Global Kernel for ConvNets and ViTs

Tao Yang, Haokui Zhang, Wenze Hu, Changwen Chen, , Xiaoyu Wang  Haokui Zhang,
Wenze Hu and Xiaoyu Wang are with Intellifusion, Shenzhen, China, 518000.
Tao Yang and Changwen Chen are with CHEN Lab, Department of Computing, Hong
Kong Polytechnic University, Hong Kong, China, 999077.
Work done during an internship at Intellifusion. A preliminary version[1] of
this work has been presented in the ECCV 2022Manuscript received xxxx, xxxx;

###### Abstract

Transformer models have made tremendous progress in various fields in recent
years. In the field of computer vision, vision transformers (ViTs) also become
strong alternatives to convolutional neural networks (ConvNets), yet they have
not been able to replace ConvNets since both have their own merits. For
instance, ViTs are good at extracting global features with attention
mechanisms while ConvNets are more efficient in modeling local relationships
due to their strong inductive bias. A natural idea that arises is to combine
the strengths of both ConvNets and ViTs to design new structures. In this
paper, we propose a new basic neural network operator named position-aware
circular convolution (ParC) and its accelerated version Fast-ParC. The ParC
operator can capture global features by using a global kernel and circular
convolution while keeping location sensitiveness by employing position
embeddings. Our Fast-ParC further reduces the $\mathcal{O}(n^{2})$ time
complexity of ParC to $\mathcal{O}(n\log{}n)$ using Fast Fourier Transform.
This acceleration makes it possible to use global convolution in the early
stages of models with large feature maps, yet still maintains the overall
computational cost comparable with using 3x3 or 7x7 kernels. The proposed
operation can be used in a plug-and-play manner to 1) convert ViTs to pure-
ConvNet architecture to enjoy wider hardware support and achieve higher
inference speed; 2) replacing traditional convolutions in the deep stage of
ConvNets to improve accuracy by enlarging the effective receptive field.
Experiment results show that our ParC op can effectively enlarge the receptive
field of traditional ConvNets, and adopting the proposed op benefits both ViTs
and ConvNet models on all three popular vision tasks, image classification,
object detection, and semantic segmentation.
Source code will be available at
https://github.com/yangtao2019yt/Fast_ParC.git.

###### Index Terms:

global receptive field, position aware circular convolution, positional
embedding, pure convolution operation, Fast Fourier Transform

##  1 Introduction

Vision transformer has been a rising in recent years. Transformer was firstly
proposed in 2017 to solve the NLP task[2]. In 2020, Dosovitskiy et al.[3]
directly applied the original transformer to image classification task and
found it achieved better results than convolutional networks (ConvNets), when
pretrained on large datasets (e.g. ImageNet-21K or JFM-300M[4]). ViT and its
variants are then widely used on other downstream vision tasks such as object
detection[5][6], semantic segmentation[7], and multi-modal tasks like human
object interactions (HOI)[8], text-to-image (T2I) synthesis[9], etc.. Despite
the great success of transformers, they still cannot replace ConvNets
completely. As is summarized in previous works[1][10][11][12], when compared
with ViTS, ConvNets have better hardware support and are easier to train. In
addition, ConvNets still dominate in the domain of lightweight models[13][14]
for mobile and other edge computing scenarios.

Figure 1: Visualization of the Effective Receptiev Fielf(ERF)[15] of different
convolutional neural networks(CNNs). ParC significantly increase the CNNs’ ERF
to global. Code are based on the repository pulished by RepLKNet[16].

Both transformers and ConvNets have their distinct features. For transformers,
the widely recognized multi-head attention mechanism is designed to capture
long range pair-wise relationships between tokens, which provides transformers
with powerful global modeling ability. While empowered by this representation
capability, it however requires higher computation budget as well. The time
complexity for self-attention is quadratic over the number of tokens, and
hence slow to process high resolution feature maps. Different from it,
convolution operation excel at extracting local information. It captures the
local information within a small sliding-window, usually 3x3, and reuses the
same convolution kernel for different inputs and different spatial positions.
It can be interpreted as an effective implicit weight sharing scheme, making
the required parameters of the convolution grows only linearly with the input.
Furthermore, ConvNets have been well studied and used for much longer time, so
they enjoy some other unique advantages as well. For example, compression
algorithms like pruning[17] and quantization[18] for ConvNets are mature. As
for hardware implementations, there are also many existing acceleration
schemes (e.g. Winograd[19], FFT[20], im2col[21]) whether on general platforms
like CPU, GPU or dedicated accelerators like FPGA, ASIC. In a word,
convolution operation is cheaper in implementation, but it cannot capture
global relationship like self-attention does. Clearly, these exist a
complementary relationship between transformers’ representation ability and
ConvNets’ efficiency, both of which are indispensable for practical
applications.

There are some recent works to combine the merits of transformers and
ConvNets. PVT[22], Swin[23] and CoAtNet[24] attempt to reintroduce inductive
bias of convolution (such as its sliding-window strategy), to help transformer
models to learn better. Works like LeViT[25], MobileViT[11],
EfficientFormer[26] focus on designing efficient hybrid architectures. Most of
these works bring the two kind of networks together, but they fail to tackle
the key problem: the extra computational and engineering complexity of the
newly introduced attention operator. It’s natural to ask: if it is possible to
design a new operator different from self-attention and traditional
convolution, that has the advantages of both sides simultaneously?

In this paper, we construct such a new plug-and-play operator named ParC,
combining the merits of both transformers and ConvNets. ParC has a global
receptive field, since we use global kernels ($K_{h}=H$ or $K_{w}=W$) and
adopt circular convolution scheme. We then adopt a explicit learnable
positional embedding before convolution to keep our model positional
sensitive. As shown in Fig. 1, different ConvNet models improve their
effective receptive fields to global by simply applying the proposed ParC
operator. As ParC uses pure convolution operation, it is efficient to be
deployed on different platforms. Lastly, we take apart 2D convolution into two
1D convolution to overcome the increase in FLOPs/parameters. Based on the
above design, we achieve the goal of extracting global feature, while still
keeping a low cost in terms of space and time complexity. Through experiments,
we verify the effectiveness of the new operator in a wide range of tasks and
models. In short, the contribution of this paper can be summarized as the
following three points:

  1. 1.

An effective new operator ParC is proposed, combining the merits of both ViTs
and ConvNets. Experiments demonstrated the advantages of ParC by applying it
to a wide ranges of models, including MobileViT[27], ResNet50[28],
MobileNetV2[14] and ConvNext[27]. We also evaluate these models on multiple
tasks, including classification, detection and segmentation.

  2. 2.

Fast-ParC is proposed to overcome the problem that the complexity of ParC
become overwhelming when the resolution of input feature is large. Fast-Parc
is theoretically equivalent to ParC, e.g. their outputs are identical when
given the same input. However it is far more efficient than ParC when given a
large resolution (e.g. 112$\\!\times\\!$122). Fast-ParC extends the usage
scenarios of ParC, making it a more widely applicable operator.

  3. 3.

The internal mechanism of the new operator is analyzed. By visualization, we
show several distinct differences between ParC and vanilla convolution. We
show that the effective receptive field (ERF)[15] of vanilla ConvNet is very
limited, while the ParC-based Network has a global ERF indeed. We also show by
Grad-CAM[29] that ParC-based networks are more comprehensive than vanilla
ConvNets in focusing on the important regions of images. We also provide
detailed analysis over the difference between ParC and vanilla convolution.

##  2 Relater work

###  2.1 Theoretical/Effective Receptive Field

Hebel et al.[30] found in neuroscience that the neurons in the shallow layer
extracts local features only, and the scope covered are accumulated layer by
layer, called the ”receptive field (RF)”. Since the success of VGGNet[31], the
design of CNN architecture follow a similar pattern[28][32][14][13] \- using a
stacking of small kernels like 3$\\!\times\\!$3 instead of larger kernels.
Some previous work give the theoretical computation of CNN’s receptive
field[33][34], namely theoretical receptive field (TRF) - under which concept,
the receptive field of two layers of 3$\\!\times\\!$3 equals one layer of
5$\\!\times\\!$5\. Nevertheless, some works[15][34] cast doubt on this view,
since in fact the importance of pixel degraded quickly from the center to the
edge in a feature map. Later, the effective receptive field (ERF) was proposed
to measure the region of input image which could actually impact the neurons’
activation pattern. Luo et al.[15] back-propagate the center pixel and compute
the partial derivative of input image to examine this region. And by studying
a sequence of convolution networks, they found the effective receptive field
is usually much smaller than their theoretical counterparts. SKNet[35] adopts
attention mechanisms in selecting appropriate receptive field. RF-Next[36]
proposes a NAS-based workflow to search the receptive fields for models
automatically. These works show that a proper decision of receptive field
could be quite beneficial for networks’ performance. Recent work also found
that enlarging the receptive field of convolution network can lead to better
model performance. We call them ”Large Kernel Convolution Network”, which will
be discussed later in Section 2.3.

###  2.2 Vision Transformer and Hybrid Structures

ViTs achieves impressive performance on various vision tasks. However, the
original ViT[3] has some restrictions. For instance, it is heavy weight, has
low computational efficiency and is hard to train. Subsequent variants of ViTs
are proposed to overcome these problems. From the point of improving training
strategy, Touvron et al. [37] proposed to use knowledge distillation to train
ViT models, and achieved competitive accuracy with less pre-training data. To
further improve the model architecture, some researchers attempted to optimize
ViTs by learning from ConvNets. Among them, PVT [22] and CVT [38] insert
convolutional operations into each stage of ViT to reduce the number of
tokens, and build hierarchical multi-stage structures. Swin transformer [23]
computes self attention within shifted local windows. PiT[39] jointly use
pooling layer and depth wise convolution layer to achieve channel
multiplication and spatial reduction. CCNet[40] propose a simplified version
of self attention mechanism called criss-cross attention and inserts it into
ConvNets to build ConvNets with global receptive field. These papers clearly
show that some techniques of ConvNets can be applied on vision transformers to
design better vision transformer models.

Another popular line of research is combining elements of ViTs and ConvNets to
design new backbones. Graham et al. mixed ConvNet and transformer in their
LeViT[25] model, which significantly outperforms previous ConvNet and ViT
models with respect to the speed/accuracy trade-off. BoTNet[41] replaces the
standard convolution with multi-head attention in the last few blocks of
ResNet. ViT-C[42] adds early convolutional stems to vanilla ViT. ConViT[43]
incorporates soft convolutional inductive biases via a gated positional self-
attention. The CMT[10] block consists of depth wise convolution based local
perception unit and a light-weight transformer module. CoatNet[24] merges
convolution and self-attention to design a new transformer module, which
focuses on both local and global information.

###  2.3 Large Kernel Convolution Network

Early ConvNets such as AlexNet[44] and GoogleNet[45] uses big kernel like
5$\\!\times\\!$5 or 7$\\!\times\\!$7\. But since the success of VGGNet[31],
stacking small kernels like 33 and 1x1 becomes believed to be an efficient
choice for computation and storage. Recently, inspired by the success of
vision transformers, big kernels are reused as a powerful tool for improving
model’s performance again. ConvNext[27] modernizes a standard ResNet towards
the design of a vision transformer by introducing a series of incremental but
effective designs, where 7$\\!\times\\!$7 depth-wise convolution is used
following the spirit of windowed-SA in Swin[23]. RepLKNet[16] scales up
convolution kernel to 31$\\!\times\\!$31 and obtain a performance gain, but
the re-parameterization trick used would burden the training process, and an
extra conversion step is needed for model deployment. Later, Rao et al. use an
even larger kernel of 51$\\!\times\\!$51 with dynamic sparsity[46]. GFNet [47]
replaces the SA (self-attention) in transformer blocks with a global Fourier
convolution, implemented with FFT.

Our work is most closely related to RepLKNet[16] and GFNet [47]. Both these
methods and our proposed ParC focus on enlarging the effective receptive
field, but our proposed op is different from the following perspectives: 1)
Our ParC uses learnable position embeddings to keep the result feature map
position sensitive. This is important for location sensitive tasks such as
semantic segmentation and object detection. Experiments in ablation study also
verify this point. 2) Our ParC adopts light weight designs. RepLKNet use heavy
2D convolution kernels, and GFNet use a learnable complex weight matrix with
shape $2CHW$, while ours uses two one dimensional convolutions, reducing the
kernels to $CH$ or $CW$. 3) Different from RepLKNet and GFNet which emphasis
network designs holistically, our proposed ParC is a new basic operator, which
can be inserted into ViTs and ConvNets in a plug-and-play manner. Our
experimental results in Section 4.1 and 4.2 verifies this point. In addition,
we also propose Fast-ParC, which further broaden the usage scenario of ParC.

Figure 2: Illustration of the position aware circular convolution,
corresponding with Eq. 2 The circular convolution part is amplified with dash
line connection. ParC-H and ParC-V follows similar strategy, but are conducted
in different axis.

##  3 The Proposed Fast-ParC Operator

In this section, we first introduce the proposed ParC operator by comparing it
with the vanilla convolution operator. Then, we present the FFT accelerated
version of ParC, named Fast-ParC. Finally, we explain how to use the proposed
ParC in ViT and ConvNet models.

###  3.1 ParC Operation

####  3.1.1 Vanilla Depth-wise Convolution

To describe an 1D depth-wise convolution conducted in horizontal dimension
(noted as Conv1d-H) on a 4D input tensor shaped as
$B\\!\times\\!C\\!\times\\!H\\!\times\\!W$, we could firstly focus on one
specific channel. We denote the output as
$\mathbf{y}=\\{y_{1},...,y_{H-1}\\}$, input as
$\mathbf{x}=\\{x_{0},x_{1},...,x_{H-1}\\}$, the convolution weight as
$\mathbf{w}=\\{w_{0},w_{1},...,w_{K_{h}-1}\\}$. The Pytorch-style convolution
(i.e. F.conv1d) with zero-padding can then be expressed as:

$\displaystyle y_{i}=\sum_{k=0}^{K_{h}-1}w_{k}\cdot x_{k+i-K_{h}/2},\quad
i=0,1,...,H_{y}-1$ (1)

where $K_{h}/2$ is used to offset the extra paddings of $K_{h}/2$ scalars on
both sides of the input. Eq.1 shows that $y_{i}$ is a function of its local
neighboring input (i.e. $x_{i-K_{h}/2}$, …, $x_{K_{h}/2-1+i}$), and the size
of the neighborhood is controlled by the kernel size $K_{h}$. As a
consequence, it is impossible for a single layer of small kernel convolution
to collect long distance information. To solve this shortcoming of vanilla
convolution, we propose our ParC with global receptive field.

####  3.1.2 ParC: Positional-Aware Circular Convolution

Define $\mathbf{w}=\\{w_{0},w_{1},...,w_{K_{h}-1}\\}$ as the kernel weight,
and $\mathbf{pe}=\\{pe_{0},pe_{1},...,pe_{K_{h}-1}\\}$ as the positional
embedding. Corresponding with Fig. 2, ParC could be described as:

$\displaystyle y_{i}=\sum_{k=0}^{H-1}w_{k}^{H}\cdot(x_{(k+i){\rm mod}H}^{p})$
(2) $\displaystyle\mathbf{w}^{H}=f(\mathbf{w},H)$
$\displaystyle\mathbf{x}^{p}=\mathbf{x}+f(\mathbf{pe},H)$

where $i=0,1,...,H-1$. $w$ is the learnable kernel of a fixed size (specified
by hyper-paramters) and $w^{H}$ is the adjusted learnable kernel whose size
match the corresponding input feature map size. $pe$ denotes position
embedding. Here, we adopt interpolation function $f(\cdot,N)$ (e.g. bi-linear,
bi-cubic) to adapt the sizes of kernels and position embedding (from $K_{h}$
to $H$). mod denotes the modulo operation.

Compared to vanilla convolution, the ParC operator has four major differences:
1) global kernel; 2) circular convolution; 3) positional embedding; 4) 1D
decomposition. To extract global feature effectively, these designs are all
essential, which is demonstrated later by ablation experiments in Section 4.4.
In the following, we will elaborate more on the reasons of these design
differences:

Global kernel and Circular Convolution. To extract global relationships across
the whole input map, ParC adopts global kernels, the size of which are the
same as the size of the corresponding feature maps, denoting as $K_{h}=H$ or
$K_{w}=W$. In some architectures this is shrunk by half in each stage. For
example in ResNet50 or ConvNeXt, the feature resolution are $[56,28,14,7]$
respectively for four stage. But simply enlarging the size of the ordinary
convolution kernel itself cannot extract the global relationship effectively.
Because of the use of zero padding, even if the kernel size increases to the
resolution, the kernel weight will be aligned to the zero paddings, which
cannot offer useful information except for absolute location. This effect
happens most extremely when kernel is aligned to the edge of a picture - for a
2D convolution, $3/4$ of the inputs are actually zeros instead. Therefore, we
additionally propose to use the circular convolution. When doing circular
convolution, kernel weight are always aligned to valid pixels during window
sliding, as is shown in Fig. 2.

Positional Embedding. As concluded in previous work[48], vanilla convolution
could encode positional information whenever it uses zero-padding. Circular
convolution, however, reuses the input picture in a periodic way, this will
lose part of the location information. To conquer this, we introduce learnable
position encoding, which is inserted before the circular convolution. In the
following experiment, we demonstrate that this is very important for models’
performance, especially for downstream tasks which are sensitive to spatial
information.

1D Decomposition.  Finally, in order to ensure acceptable cost in model size
and computation. We split the 2D convolution and position encoding into
$H$(horizontal) and $V$(vertical) directions, which reduce the number of
parameters and FLOPs from $\mathcal{O}(H\\!\times\\!W)$ to
$\mathcal{O}(H\\!+\\!W)$, this is a considerable compression when the
resolution is large.

Implementation of Circular Convolution. Conceptually, the circular convolution
needs to be implemented separately from ordinary convolutions because of the
extra modulus op when computing the index of convoluted pixels. In practice,
it could be easily implemented by padding the input feature map with its copy
using ’concat’ function before calling the ordinary 1D convolution routines
(See Algorithm 1)

When considering vertical dimension $W$ and the channel dimension $C$, the Eq.
2 could be extended as

$\displaystyle\mathbf{Y}_{i,j,c}=\sum_{k=0}^{H-1}\mathbf{W}_{k,c}^{H}\cdot(\mathbf{X}_{(k+i){\rm
mod}H,j,c}^{p})$ (3)

$\forall i\in[0,H\\!-\\!1]$, $\forall j\in[0,W\\!-\\!1]$ and $\forall
c\in[0,C\\!-\\!1]$, which is the full representation of a one layer depth-wise
ParC-H with channels $C$, and input resolution $H\\!\times\\!W$ . In
ResNet50-ParC, we also extend the per channel ParC to its dense counterpart
and reintroduce the channel interaction, which can be expressed as:

$\displaystyle\mathbf{Y}_{i,j,c_{o}}=\sum_{c_{i}=0}^{C_{i}-1}\sum_{k=0}^{H-1}\mathbf{W}_{k,c_{i}}^{H}\cdot(\mathbf{X}_{(k+i){\rm
mod}H,j,c_{i}}^{p})$ (4)

considering $\forall i\in[0,H\\!-\\!1]$, $\forall j\in[0,W\\!-\\!1]$, $\forall
c_{i}\in[0,C_{i}\\!-\\!1]$ and $\forall c_{o}\in[0,C_{o}\\!-\\!1]$.

Figure 3: The theoretical FLOPs of different convolutions operations,
estimated with channel number and batch size as 1. Dash line indicates
operation is with local receptive field, solid line indicates operation is
with global receptive field.

# B: batch size, C: channel number

# H: height, W: weight

# fft1d: FFT conduct in one dimension indicates by ’dim’.

# ifft1d: inverse FFT, normalized with 1/N by in default

def ParC_H(x, weight, bias):

B, C, H, W = x.shape

# periodic extension

x_cat = torch.cat([x, x[:, :, :-1, :]], dim=-2)

# spatial-ParC

x = F.conv2d(x_cat, weight, bias, padding=0, groups=C)

return x

def Fast_ParC_H(x, weight, bias):

B, C, H, W = x.shape

# input FFT

x = fft1d(x, dim=-2)

# weight FFT

weight = fft1d(weight, dim=-2)

# Fourier-ParC

x = x * torch.conj(weight).view(1, C, H, 1)

# output iFFT

x = ifft1d(x, dim=-2).real

x = x + bias.view(1, C, 1, 1)

return x

Algorithm 1 PyTorch style Pseudo code of ParC-H and Fast-ParC-H. Codes are
based on depth-separable version. (H means horizontal. It could be easily
extended to ParC-V by conducting these operations in vertical dimension.)

###  3.2 Fast-ParC: Speed up ParC with its FFT Equivalent Form

As shown in Fig 3, when feature resolution is small (e.g. 7$\\!\times\\!$7),
applying ParC can extract global feature and reduce the computation complexity
effectively. But as input resolution grows, the complexity of ParC surpass
7$\\!\times\\!$7 convolution rapidly. To overcome this issue, we propose an
accelerated version of ParC named Fast-ParC. Fast-ParC is much more efficient
than ParC when feature resolution is large (e.g. 56$\\!\times\\!$56). In fact,
Fast-ParC, while enjoying global extracting ability, is even more efficient
than 7$\\!\times\\!$7 convolution within a wide interval of resolution.

Figure 4: ParC-MetaFormer block used to replace transformer blocks in ViTs or
hybrid structures (e.g. MobileViT[11]). ParC-MetaFormer block adopt a
MetaFormer[49] like block structure, and it keeps the 3 merits of transformer
block: 1) global receptive field; 2) positional-aware; 3) data-driven.

We derive Fast-ParC with the help of Fast Fourier Transform(FFT). It is well
known that linear convolution operation could be facilitate by FFT[50]. But
according to convolution theorem[50], for discrete signal, dot product in the
Fourier domain is closer to circular convolution in spatial domain. This is
exactly one of the distinctive differences between ParC and ordinary
convolution. In addition, the other two features of ParC, global kernel and
right padding, also fit well with the default mode of convolution in Fourier
domain. This interesting fact allows us to develop a very neat and beautiful
frequency domain implementation for ParC. Define $x(n)$, $w(n)$, $y(n)$ as
input, weight and output sequence in time domain, $X(k)$, $W(k)$ and $Y(k)$ as
the sequence in Fourier domain, we could get the following equivalence
relationship:

Theorem 1: ParC equivalence theorem

Spatial Form: $\displaystyle y(n)$
$\displaystyle=\textstyle\sum_{i=0}^{N-1}w(i)x((n+i))_{N}$ (5)
$\displaystyle\qquad\Updownarrow$ Fourier Form: $\displaystyle Y(k)$
$\displaystyle=W^{\ast}(n)X(n)$

Eq. 5 shows two mathematical expressions which are strictly equivalent. ParC
in spatial domain requires a convolution style operation, however in Fourier
domain it turns into a simple per-element multiplication. Based on this, we
propose a Fourier domain version of ParC operation, named Fast-ParC. It can be
proved theoretically that Fast-ParC is strictly equivalent to ParC in spatial
domain (See Appendix A.1). The error between these two implementations is
negligible. we can choose the appropriate implementation of ParC respectively
for training and inference, depending on the platform that are actually used.
This provides ParC with powerful flexibility. Algorithm 1 shows the pseudo-
code. The advantages of Fast-ParC are clear:

Firstly, the multiplication complexity of 1D-FFT with length N is only
$\mathcal{O}(NlogN)$, while 1D convolution in spatial domain requires
$\mathcal{O}(N^{2})$.

Corresponding with Table I, it’s clear that the complexity of spatial
convolution surpasses the FFT-based convolution by a large margin, when N is
large. ParC use a global kernel and circular convolution, which matches the
default spatial format of Fourier convolution. Considering downstream task
like detection or segmentation with multiple instances, usually much higher
resolution is needed. For example for COCO[51], the commonly used resolution
for testing is 1280$\\!\times\\!$800, and for ADE20k[52] is
2048$\\!\times\\!$512\. When N is large, Fast-ParC can save model’s FLOPs and
achieve a better acceleration. Fast-ParC also allows us to use ParC for the
shallower stages with a acceptable budget in computation. This is necessary
for implementation of ParC in novel architectures[53].

Another advantage actually comes from the software/hardware support for FFT.
Since FFT is a classical signal processing algorithm, many platforms have off-
the-peg support for its acceleration. When ParC is applied to customized
platform like FPGA, many resources (e.g. on-chip DSP, pre-designed IP core)
could be utilized effectively. Besides, universal computing platforms have
out-of-the-box toolkit for it as well (e.g. CPU: torch.fft[54], numpy.fft[55];
GPU: cuFFT[56]). The flexibility of Fast-ParC allows us to choose the better
implementation with respect to different standard (e.g. most throughput, least
memory occupation), and with respect to the actual preferences for algorithms
in different computing platforms.

Operation | Theoretical complexity
---|---
Self-Attention | $\mathcal{O}(CH^{2}W^{2}+C^{2}HW)$
Conv2d | $\mathcal{O}(CHWK_{x}K_{y})$
ParC | $\mathcal{O}(CHW(H+W))$
FFT-ParC | $\mathcal{O}(CHW(\lceil\log_{2}{H}\rceil+\lceil\log_{2}{W}\rceil)$
TABLE I: The theoretical complexity of multiplication for different
convolution operation. The Complexity of FFT-ParC are with base-2-FFT. One
complex multiplications are treated as 4 real multiplications.

Luckily, Fast-ParC is completely equivalent to ParC, and the replacement does
not require any additional transformations. We can choose the specific
implementation form of ParC according to the requirements of the platform. As
ParC and Fast-ParC owns the highest level of flexibility, the implementation
used for training and inference could also be decoupled. Further exploration
of other efficient convolution operations based on FFT is also a very
promising direction, and we leave this issue for future.

Figure 5: The hierarchical structure of ParC-based ConvNets. Fig. (a) shows
the inner structure of an ParC or Dw-ParC(i.e. depth-wise version ParC)
operator. Fig. (b)(c)(d) illustrate 3 different ParC-ConvNet blocks. Fig (e)
shows the ParC-ConvNet structure, which indicates we replace the last several
blocks in the deep stages and remain the structure in the shallow stages.

###  3.3 Applying ParC on ViTs and ConvNets

To validate the effectiveness of ParC as a plug-and-play meta-operator, we
built a series of ParC-based Models based on the operations proposed in
Section 3.1. Here, baseline models includes both ViTs and ConvNets.
Specifically, for ViTs, MobileViT [11] is selected as baseline, as it achieved
the best parameter/accuracy trade-off among the lightweight hybrid-structures
proposed recently. ResNet50 [28], MobileNetv2 [14] and ConvNext [27] are
adopted as ConvNet baselines. ResNet50 is the most widely used model in
practical applications. MobileNetV2 is the most popular model in mobile
devices. ConvNext is the first ConvNet which remains pure ConvNet architecture
while integrates some characteristics of ViTs. All of the four models we
adopted here are representative.

####  3.3.1 ParC-ViTs

ParC-MetaFormer Block. As shown in Fig 4 and Fig 5, ConvNets and ViTs have big
difference in outer layer structure. ViTs generally adopt meta-former block as
basic architecture. To apply ParC operator on ViTs, we design ParC-MetaFormer
block and use it to replace transformer blocks in ViTs.

Adopting MetaFormer like structure. A MetaFormer[49] block is the block
structure that ViTs use most frequently, and it generally consists of a
sequence of two components: a token mixer and a channel mixer. Both two
components use residual structure. We adopt ParC as a token mixer to build a
ParC-MetaFormer block. We do this because ParC can extract global features and
interacts information among pixels from global space, which meets the
requirement of token mixer module. And unlike the self-attention whose
complexity is quadratic, ParC are much more efficient in computation.
Replacing this part with ParC can reduce computational cost significantly. In
ParC-MetaFormer Block, we adopt a serial structure of ParC-H and ParC-V.
Considering symmetry, half of the channels go though ParC-H firstly, and
others go though ParC-V firstly (as shown in Fig 4).

Adding channel wise attention in channel mixer part. Though ParC keeps the
global receptive field and positional embedding, another benefit ViTs have
against ConvNets is data-driven. In ViTs, self attention module can adapt
weights according to input. This makes ViTs data-driven models, which can
focus on important features and suppress unnecessary ones, bringing better
performance. Previous literature[57][58][59] already explained the importance
of keep model data driven. By replacing the self-attention with the proposed
global circular convolution, we get a pure ConvNet which can extract global
features. But the replaced model is no longer data-driven. To compensate, we
insert channel wise attention module into channel mixer part, as shown in Fig.
4. Following SENet[57], we first aggregate spatial information of input
features $x\in\mathbb{R}^{c\times h\times w}$ via global average pooling and
get aggregated feature $x_{a}\in\mathbb{R}^{c\times 1\times 1}$ Then we feed
$x_{a}$ into a multi-layer perception to generate channel wise weight
$a\in\mathbb{R}^{c\times 1\times 1}$ The $a$ is multiplied with $x$ channel
wise to generate the final output.

MobieViT-ParC Network. Currently, existing hybrid structures can be basically
divided into three main structures, including serial structure[25][42],
parallel structure[12] and bifurcate structure[11][24]. Among all three
structures, the third one achieves best performance for now. MobileViT[11]
also adopts the bifurcate structure. Inspired by this, based on MobileViT, we
build our model with bifurcate structure as well. MobileViT consists of two
major types of modules. Shallow stages consist of MobileNetV2 blocks, which
have local receptive field. Deep stages are made up of ViT blocks, which enjoy
global receptive field. We keep all MobileNetV2 blocks and replace all ViT
blocks with corresponding ParC blocks. This replacement converts the model
from hybrid structure to pure ConvNet, while reserving its global feature
extracting ability.

Frameworks | Models | Source |  Parmas (M) |  Top1 Accu (%)
---|---|---|---|---
Pre-ConvNet | SuffleNetV2(2.0x) | ECCV 2018 | 5.5 | 74.5
MobileNetV3(1.0x) | ICCV 2019 | 5.4 | 75.2
EfficientNet-B0 | ICML 2019 | 5.3 | 76.3
ResNet-101-SE | CVPR 2018 | 49.3 | 77.6
ViTs | DeiT-2G | ICML 2021 | 9.5 | 77.6
Swin-1G | ICCV 2021 | 7.3 | 77.3
Hybrid Structures (ConvNet+ViT) | CoaT-Lite-T | ICCV 2021 | 5.7 | 76.6
LeViT-128S | ICCV 2021 | 7.8 | 76.6
Mobile-Former | CVPR 2022 | 9.4 | 76.7
$\text{MobileViT-S}^{\dagger}$ | ICLR 2022 | 5.6 | 78.4
$\text{EfficientFormer-L1}^{\dagger}$ | arXiV 2022 | 12.3 | 76.0
Post-ConvNet | ConvNext-XT | CVPR 2022 | 7.4 | 77.5
$\text{MobileViT-ParC-S}^{\dagger}$ | ECCV 2022 | 5.0 | 78.6
TABLE II: Classification results on ImageNet-1K. $\dagger$ means running in
MobileViT[11] training framework. Pre-ConvNet indicates classical ConvNets
before ViTs appeared, Post-ConvNet integrates merits of both ViTs and
ConvNets.

####  3.3.2 ParC-ConvNets

For ParC-ConvNets, we focus on providing ConvNets with global receptive field.
Replacing vanilla convolution with ParC operation (as shown in Fig. 5 (a)), we
build different ParC-based blocks (as shown in Fig. 5 (b)(c)(d)). Previous
hybrid structure works[25][42][11] draw a similar conclusion: models with
local-extracting blocks in early stages and global-extracting blocks in deep
stages achieve the best performance. Since ParC owns a global receptive field,
we insert ParC-based block into ConvNets following this rule (as shown in Fig.
5 (e)).

ParC BottleNeck and ResNet50-ParC Network. ResNet[28] is one of the most
classical ConvNet. Simply replacing the 3$\\!\times\\!$3 convolution of the
original ResNet50 BottleNeck with a ParC operator, we can obtain the ParC
BottleNeck (see Fig 5 (b)). Since the characteristics of ParC-H and ParC-V
might be significantly different, no channel interaction is introduced between
them. This is similar to adopt group convolution[60] with group=2. The main
part of ResNet can be divided into 4 stages, and each stage consists of a
couple of repeated BottleNeck blocks. Specifically, ResNet50 has [3, 4, 6, 3]
blocks respectively in four stages. By replacing the last $1/2$ of the
penultimate stage of ResNet50 and the last $1/3$ of the last stage with ParC
BottleNeck, we obtain ResNet50-ParC.

ParC-MobileNetV2 Block and MobileNetV2-ParC Network. MobileNetV2[14] is a
typical representative of the lightweight model. By replacing the
3$\\!\times\\!$3 depthwise convolution in Inverted Bottleneck with depthwise
ParC, we get the ParC-MobileNetV2 block (see Fig 5 (c)). MobileNetV2 is much
slimmer and deesper than the Resnet50, with blocks number of [1, 2, 3, 4, 3,
3, 1] respectively for 7 stages. MobilenetV2-ParC could be obtained by
replacing the last $1/2$ blocks of stage 4 and the last $1/3$ blocks of stage
[5, 6] with ParC-MobileNetV2 blocks.

ParC-ConvNeXt Block and ConvNeXt-ParC Network. ConvNeXt[27] take a sequence of
modifications on original ResNet50 structure to learn from transformers.
During this, 3$\\!\times\\!$3 convolution is replaced with 7$\\!\times\\!$7
depthwise convolution. This enlarges the local receptive field, however it
still cannot grad global information. We further replace 7$\\!\times\\!$7
depthwise convolution in ConvNeXt Block by depthwise ParC. By this we obtain
the ParC-ConvNeXt block (see Fig 5 (d)). Replacing the last $1/3$ blocks of
the last two stages of ConvNeXt with ParC-ConvNeXt blocks, an example of
ConvNeXt-ParC is obtained. We reduce the basic channel number in ConvNeXt-T to
48 (i.e. [48, 96, 192, 384] for each stage) to get a light weight ConvNeXt-XT,
which is more welcome in deploying on edge computing devices and it also has
shorter experimental period.

Note that, in ParC-MetaFormer, a sequence of ParC-H and ParC-W is adopted keep
the receptive field consistent with self-attention, as this design is used to
replace self-attention. In ParC-ConvNets, we adopt a parallel of ParC-H and
ParC-V (each one single layer) as shown in Fig. 5. According to experimental
results, this setting could already provide enough performance gain against
the vanilla ConvNets. In fact, as we do not use only one ParC-ConvNet block,
the ParC-ConvNets still has global receptive field.

Models |  Params (M) | mAP
---|---|---
MobileNetV2 | 4.3 | 22.1
MixNet | 4.5 | 22.3
MNASNet | 4.9 | 23.0
MobileViT | 2.7 | 24.8
VGG | 35.6 | 25.1
ResNet50 | 22.9 | 25.2
MobileViT-S | 5.7 | 27.7
MobileViT-ParC-S | 5.2(↓0.5) | 28.8(↑1.1)

Models |  Params (M) | mIoU
---|---|---
MobileNetV1 | 11.2 | 75.3
MobileNetV2 | 4.5 | 75.7
MobileViT-XXS | 1.9 | 73.6
MobileViT-XS | 2.9 | 77.1
ResNet101 | 22.9 | 80.5
MobileViT-S | 6.4 | 79.1
MobileViT-ParC-S | 5.8(↓0.6) | 79.7(↑0.6)

TABLE III: Object detection results on MS-COCO (Left) and Semantic
segmentation experiments on PASCAL VOC (Right) about ParC-based transformer
models. We compare the mAP/mIoU with more other models.

##  4 Experiment

To test the capability of ParC as a plug-and-play operator. We applied it to a
wide range of models including Transformers and ConvNets. We take different
views when plugging ParC into these two kind of models. For transformer
models, the main purpose of applying ParC is to make model more hardware
friendly while keep its ability of global extracting, thus we use ParC to
replace self-attention. For ConvNets, we use ParC mainly to enable networks
with global extracting ability, while keeping it parameter/FLOPs friendly,
thus we use ParC to replace vanilla convolution.

###  4.1 Experiment on Vision Transformer Models

In the transformer experiments, we focus on inserting the ParC operator into
the most recently proposed MobileViT framework. We show the overall advantages
of the proposed MobileViT-ParC on three typical vision tasks, including image
classification, object detection and semantic segmentation.

####  4.1.1 Image Classification on ImageNet-1K

We conduct image classification experiments on ImageNet-1k, the most widely
used benchmark dataset for this task. We train the proposed MobileViT-ParC
models on the training set of ImageNet-1K, and report top-1 accuracy on the
validation set.

Training setting. As we adopt MobileViT like structure as our outer framework,
we train our model using a very similar training strategy as well. To be
specific, we train each model for 300 epochs on 8 V100 or A100 GPUs with AdamW
optimizer[61], where the maximum learning rate, minimum learning rate, weight
decay and batchsize are set to 0.004, 0.0004, 0.025 and 1024 respectively.
Optimizer momentum $\beta_{1}$ and $\beta_{2}$ of the AdamW optimizer are set
to 0.9 and 0.999 respectively. We use the first 3000 iterations as warm up
stage. We adjust learning rate following the cosine schedule. For data
augmentation, we use random cropping, horizontal flipping and multi-scale
sampler. We use label smoothing[62] to regularize the networks and set
smoothing factor to 0.1. We use Exponential Moving Average (EMA)[63]. More
details of the training settings and link to source code will be provided in
supplementary materials.

Results comparison. The experiment results of image classification and
comparison with other models are listed in Table II. Table II shows that
MobileViT-ParC-S and MobileViT-S beat other model by a clear margin. The
proposed MobileViT-ParC-S achieves highest classification accuracy, and have
fewer parameters than most models. Compared with the second best model
MobileViT-S, our MobileViT-ParC-S decreases the number of parameters by 11%
and increases the top-1 accuracy by 0.2 percentage. Light-weight models.
Firstly, comparing results of light-weight ConvNets with that of ViTs, light-
weight ConvNets show much better performance. Secondly, comparing the popular
ConvNets before ViT appears (pre-ConvNets), ViTs and hybrid structures, hybrid
structures achieve the bes t performance. Therefore improving ConvNets by
learning from the merits of ViT is feasible. Finally, the proposed MobileViT-
ParC achieves the best performance among all comparison models. So indeed by
learning from ViT design, performance of pure light-weight ConvNets can be
improved significantly.

####  4.1.2 Object detection on MS-COCO

We use MS-COCO[64] datasets and its evaluation protocol for object detection
experiments. Following[14][11], we take single shot object detection (SSD)[65]
as the detection framework and use separable convolution to replace the
standard convolutions in the detection head.

Experiment setting. Taking models pretrained on ImageNet-1K as backbone, we
finetune detection models on training set of MS-COCO with AdamW optimizer for
200 epochs. Batchsize and weight decay are set to 128 and 0.01. We use the
first 500 iterations as warm up stage, where the learning rate is increased
from 0.000001 to 0.0009. Both label smoothing and EMA are used during
training.

Results comparison. Table III lists the corresponding results. Similar to
results in image classification, MobileViT-S and MobileViT-ParC-S achieve the
the second best and the best in terms of mAP. Compared with the baseline
model, MobileViT-ParC-S shows advantages in both model size and detection
accuracy.

####  4.1.3 Semantic segmentation on VOC/COCO

Experiment setting. DeepLabV3 is adopted as the semantic segmentation
framework. We fine tune segmentation models on training set of PASCAL VOC[66]
and COCO[51] dataset, then evaluate trained models on validation set of PASCAL
VOC using mean intersection over union (mIoU) and report the final results for
comparison. We fine tune each model for 50 epochs with AdamW. Readers may
refer to more details about training settings in supplementary materials.

Row | Model |  Parms (M) | FLOPs (G) | Top1 Accu (%)
---|---|---|---|---
1 | ResNet50 | 25.6 | 4.1 |  79.1 79.6(↑0.5)
2 | ResNet50-ParC | 23.7 | 4.0
3 | MobileNetV2 | 3.51 | 0.6 |  70.2 71.1(↑0.9)
4 | MobileNetV2-ParC | 3.54 | 0.6
5 | ConvNeXt-XT | 7.44 | 1.1 |  77.5 78.3(↑0.8)
6 | ConvNeXt-ParC-XT | 7.41 | 1.1
TABLE IV: Classification results on ImageNet-1K about ParC based ConvNets. All models are running in ConvNext’s [27] training recipe.  Backbone | Instance Detection | Instance Segmentation
---|---|---
Model |  Params (M) | $\text{AP}^{\text{box}}$ | $\text{AP}_{\text{50}}^{\text{box}}$ | $\text{AP}_{\text{75}}^{\text{box}}$ | $\text{AP}_{\text{S}}^{\text{box}}$ | $\text{AP}_{\text{M}}^{\text{box}}$ | $\text{AP}_{\text{L}}^{\text{box}}$ | $\text{AP}^{\text{mask}}$ | $\text{AP}_{\text{50}}^{\text{mask}}$ | $\text{AP}_{\text{75}}^{\text{mask}}$ | $\text{AP}_{\text{S}}^{\text{mask}}$ | $\text{AP}_{\text{M}}^{\text{mask}}$ | $\text{AP}_{\text{L}}^{\text{mask}}$
ResNet50 | 25.6 | 47.5 | 65.6 | 51.6 | 29.9 | 51.1 | 61.5 | 41.1 | 63.1 | 44.6 | 23.6 | 44.7 | 55.3
Resnet50-ParC | 23.7(↓1.9) | 48.1(↑0.6) | 66.4 | 52.3 | 30.3 | 51.2 | 62.3 | 41.8(↑0.7) | 64.0 | 45.1 | 24.6 | 45.2 | 56.0
MobileNetV2 | 3.51 | 43.7 | 61.9 | 47.6 | 27.0 | 46.1 | 58.3 | 37.9 | 59.1 | 40.8 | 21.4 | 40.5 | 52.5
$\text{MobileNetV2-ParC}^{\dagger}$ | 3.54 | 44.3(↑0.6) | 62.7 | 47.8 | 28.0 | 47.0 | 58.6 | 39.0(↑1.1) | 60.3 | 42.1 | 23.0 | 41.8 | 53.1
ConvNeXt-XT | 7.44 | 47.2 | 65.6 | 51.4 | 30.1 | 50.2 | 62.3 | 41.0 | 62.9 | 44.2 | 23.8 | 44.2 | 55.9
ConvNext-ParC-XT | 7.41 | 47.7(↑0.5) | 66.2 | 52.0 | 30.4 | 50.9 | 62.7 | 41.5(↑0.5) | 63.6 | 44.6 | 24.4 | 44.7 | 56.5
TABLE V: Instance detection and segmentation results on COCO dataset of convolution models and ParC-based convolution models. Code is based on MMdetection. $\dagger$ means a 3x expansion of learnable parameters to ParC weight is applied when transferred from ImageNet (i.e. size 224x224) to COCO (i.e. size 800x1333). Backbone | Evaluation w/o TTA | Evaluation w/ TTA
---|---|---
Model |  Params (M) | mIoU | mAcc | aAcc | $\text{mIoU}_{\text{aug}}$ | $\text{mAcc}_{\text{aug}}$ | $\text{aAcc}_{\text{aug}}$
ResNet50 | 25.6 | 42.27 | 52.91 | 79.88 | 43.75 | 53.50 | 80.71
ResNet50-ParC | 23.7(↓1.9) | 44.32(↑2.05) | 54.66 | 80.80 | 44.69(↑0.94) | 54.38 | 81.33
MobileNetV2 | 3.51 | 38.66 | 48.75 | 77.98 | 39.26 | 48.27 | 78.87
$\text{MobileNetV2-ParC}^{\dagger}$ | 3.54 | 39.25(↑0.59) | 49.64 | 78.35 | 39.62(↑0.36) | 48.57 | 79.03
ConvNeXt-XT | 7.44 | 42.17 | 54.18 | 79.72 | 43.07 | 54.27 | 80.44
ConvNeXt-ParC-XT | 7.41 | 42.32(↑1.15) | 54.48 | 80.30 | 43.09(↑0.02) | 54.41 | 80.76
TABLE VI: Sementic segmentaion result on ADE20k dataset of convolution models
and ParC-based convolution models. Test Time Argumentation(TTA) schemes used
by us include multi-scaling of [0.5, 1.75] and flipping. $\dagger$ means a 4x
expansion of learnable parameters to ParC weight is applied when transferred
from ImageNet (i.e. size 224x224) to ADE20k (i.e. size 512x2048).

Results comparison. Results are summarized in Table III. We can see that
MobileViT-S and MobileViT-ParC-S have the best trade-off between model scale
and mIOU. Compared with ResNet-101, MobileViT-S and MobileViT-ParC-S achieve
competitive mIOU, while having much fewer parameters.

###  4.2 Experiment on Convolutional Models

In the convolutional networks experiment, we insert the ParC module into
ResNet50, MobileNetV2 and ConvNeXt. These part of experiments also include the
three typical vision tasks, classification, detection and segmentation. All of
the following models are trained and evaluated under ConvNeXt’s[27] settings.

####  4.2.1 Image Classification on ImageNet-1K

Training setting. Most of our training settings for convolutional models are
from ConvNeXt’s[27] guide. To be specific, we use 8 2080Ti or 3090 to train
each model for 300 epochs, with AdamW optimizer. The learning rate increase
linearly in the beginning 20 epochs as warm-up, and then decay with cosine
schedule. The batch size, base learning rate, weight decay, momentum
$\beta_{1}$ and $\beta_{2}$ are setting as 4096, 0.004, 0.05, 0.9 and 0.999,
respectively. Data augmentations used include Mixup, Cutmix, RandAugment, and
Random Erasing. Regularization method used include Stochastic Depth[67] and
Label Smoothing[62]. Notice in this section NO Exponential Moving Average
(EMA) skill is used, since in most experiment we observed that the original
model owns a higher accuracy than its EMA counterpart.

Results comparison. We show the results of different types ConvNets on
ImageNet-1K at Table IV. It’s clear that all of the three ParC enhanced models
beat their prototypes. Especially for ResNet50, using ParC design improves
classification accuracy by 0.5%, while reducing 2M parameters and saving
computation cost slightly. Comparing lightweight model MobileNetV2 and
MobileNetV2-ParC, our operator shows a extremely obvious advantage with an
improvement of 0.9%, with a slightly cost on latency. ConvNeXt-ParC-XT exceed
original ConvNeXt-XT by 0.8%, too. Generally speaking, ParC-based
convolutional models have almost the same FLOPs and parameters with the
original models, but have higher accuracy.

####  4.2.2 Instance detection and segmentation on COCO

Training setting. We use ConvNets pretrained on ImageNet (in Section 4.2.1) as
backbones and then fine-tune with Cascade Mask-RCNN[68] as detection framework
on COCO dataset[51]. We set epoch as 36, learning rate as 2e-4 (decays by 10
at epoch 27 and 33). We set momentum $\beta_{1}$ and $\beta_{2}$ as 0.9 and
0.999, weight decay as 0.05, stochastic depth as 0.4. The code is built base
on the official implementation of MMDection[69] toolboxes. FP16 training is
NOT used for precision concern. While training ResNet50 and ResNet50-ParC, we
frozen the first stage of models. To train ConvNeXt-XT and ConvNeXt-ParC-XT,
we set the basic layer-wise learning rate decay to 0.7. For training networks
with Batch Normalization (BN) layers, we set BN layers to evaluation mode
throughout the whole process. We use muti-scale training on training set and
report the box AP and mask AP on validation set. More detailed training and
testing configurations could be seen in our repository.

For MobileNetV2-ParC, since the resolution amplification in our training
recipes is about 3 times from ImageNet (224$\\!\times\\!$224) to COCO
(800$\\!\times\\!$1333). Specifically, taking ParC kernels in stage 3 as an
example, we extend 14$\\!\times\\!$1-sized ParC-H kernels and
1$\\!\times\\!$14-sized ParC-W kernels to 21$\\!\times\\!$1 and
1$\\!\times\\!$42 respectively. We extend kernels pretrained on Imagenet-1K to
as initialization kernels for detection and segmentation models.

Figure 6: The parameters-accuracy trade-off (left) and latency-accuracy trade-
off (right) of ParC-based models and their vanilla counterparts. Arrows in the
left figure indicate the accuracy changes, in the right figure indicate the
latency changes. ${\dagger}$ and $\dagger$ indicate results measured on
different platforms (see Table VII for detail.) Figures show that: 1)
Replacing self-attention with ParC decreases ViTs’ parameters/latency
drastically, with slight improvement in performance; 2) Replacing convolution
with ParC improves the performance over a wide range of ConvNets, with slight
growth in models’ parameters/latency.

Results comparison. The result of instance detection and segmentation is
listed in Table V, which is almost consistent with the classification result
on ImageNet. The ParC-based models outperform the vanilla models by a clear
margin in both box AP and mask AP. Specifically, ResNet50-ParC improves by 0.6
in box AP and 0.7 in mask AP, MobileNetV2-ParC improves by 0.6 in box AP and
1.1 in mask AP, ConvNeXt-ParC-XT improves by 0.5 in both box AP and mask AP.

####  4.2.3 Semantic segmentation on ADE20K

Training setting. We use convolutional model pretrained on ImageNet as
backbone and fine-tune with UperNet[70] as the framework on ADE20K[52]
dataset. We set max iteration to 16000. We set basic learning rate as 1, using
a linear warmup and the a poly deacy schedule. We set momentum, $\beta_{1}$
and $\beta_{2}$ to 0.9, 0.9 and 0.999, respectively. Weight decay and
stochastic depth are set to 0.05 and 0.4. The code is built base on the
official implementation of MMSegmentation[71] toolboxes. F16 training is NOT
used for precision concern. For networks with BN, setting all BN layer to
evaluation mode during fine-tuning. We frozen the first stage of ResNet50 and
ResNet50-ParC. For ConvNeXt-XT and ConvNeXt-ParC-XT, we use a basic layer-wise
learning rate decay of 0.9. For MobileNetV2-ParC, we extend the learnable
parameters of ParC to its 4$\\!\times\\!$times and use the interpolation
result as initialization. We do this adaptation step because the resolution of
ADE20k (i.e. 512$\\!\times\\!$2048) is larger than ImageNet (i.e.
224$\\!\times\\!$224). We use muti-scale training and report mIoU, mAcc and
aAcc results on validation set. Follow ConvNeXt[27] and Swin[23], we enable
’slide’ mode during evaluation. By doing this, the original picture would be
cut into several patches and passed to network individually. Specifically, we
set crop size as 512$\\!\times\\!$512, stride as 341 (take the average for
overlapping area). We test the original single-scale mIoU, and also the mIoU
with test time argumentation (TTA). The TTA used include: 1) flipping; 2)
multi-scaling of range [0.5, 1.75]. Result are shown in Table VI.

Results comparison. The ParC models improves a lot to be compared with the
orginal models. Inserting ParC operation improves about 1.15 in ConvNeXt-XT.
In traditional network, it has a even better performance. Especially,
ResNet50-ParC outperforms the original ResNet50 by 2.05, MobileNetV2-ParC
outperforms MobileNetV2 by 0.59 in mIoU. TTA improves performance for all
ConvNet models test here. But, it is clear that vanilla ConvNets could get
more benefit from it. Comparing the row 3-5 with row 6-8 in Table VI, we could
conclude that ParC’s advantage degrades significantly after TTA. Even so, ParC
version models still show better performance than the vanilla version.

Another interesting fact is that: this significant performance gain is closely
related to the ’512$\\!\times\\!$512-sized crop’ strategy during evaluation.
If we set the test mode to ’whole’, which means to feed the whole
512$\\!\times\\!$2048-sized picture to network, we could also observe a
degradation of ParC-based models’ advantage. This is in accord with what we
see when enabling TTA. Based on these results, we conclude one possible
explanation - ParC is not robust enough to resolution change. While using
ImageNet-1K for pre-training, we empirically set the meta-kernel size to 14 or
7 and use bi-linear interpolation to get a global kernel during practical
forward process. Though interpolated kernel works, but it might be a sub-
optimum technique, causing instability to resolution changes. A more promising
technique is to generate the kernel dynamically. And since zero-padding
convolution does not change the resolution of feature map, it might be a
candidate for kernel generating module. But we unfortunately failed to
accomplish such a version with better performance than Static-ParC, and this
might related with normalization or optimization problem. But after all,
Dynamic-ParC is still a promising direction for future work.

Model |  Parms (M) |  FLOPs (G) | Devices |  Speed (ms) |  Top1 Accu (%)
---|---|---|---|---|---
MobileViT-S | 5.6 | 4.0 | RK3288 | 457 | 78.4
MobileViT-ParC-S | 5.0 | 3.5 | RK3288 | 353 | 78.6
MobileViT-S | 5.6 | 4.0 | DP2000 | 368 | 78.4
MobileViT-ParC-S | 5.0 | 3.5 | DP2000 | 98 | 78.6
$\text{ResNet50}^{\ast}$ | 26 | 4.1 | CPU | 98 | 79.1
$\text{ResNet50-ParC}^{\ast}$ | 24 | 4.0 | CPU | 98 | 79.6
$\text{MobileNetV2}^{\ast}$ | 3.5 | 0.6 | CPU | 24 | 70.2
$\text{MobileNetV2-ParC}^{\ast}$ | 3.5 | 0.6 | CPU | 27 | 71.1
$\text{ConvNeXt-XT}^{\ast}$ | 7.4 | 1.1 | CPU | 47 | 77.5
$\text{ConvNeXt-ParC-XT}^{\ast}$ | 7.4 | 1.1 | CPU | 48 | 78.3
TABLE VII: Applying ParC-Net designs on different backbones and comparing inference speeds of different models. CPU used here is Xeon E5-2680 v4, DP2000 is the code name of a in house unpublished low power neural network processor that highly optimizes the convolutions. $\ast$ denotes the models are trained under ConvNeXt[27] hyperparameters settings, which may not be the optimal. Latency is measured with batch size 1. Row | Task | Kernel | CA | PE | Params (M) | Top1/mAP/mIoU
---|---|---|---|---|---|---
1 | Classification | Baseline | - | - | 5.6 | 78.35
2 | Classification | BK $L/2$ | T | F | 5.0 | 78.46
3 | Classification | BK $L/4$ | T | F | 5.0 | 78.45
4 | Classification | ParC | F | F | 5.3 | 78.50
5 | Classification | ParC | T | F | 5.0 | 78.63
6 | Classification | ParC | T | T | 5.0 | 78.63
7 | Detection | Baseline | - | - | 5.7 | 27.70
8 | Detection | ParC | T | F | 5.7 | 27.50
9 | Detection | ParC | T | T | 5.7 | 28.50
10 | Segmentation | Baseline | - | - | 6.4 | 79.10
11 | Segmentation | ParC | T | F | 5.8 | 79.20
12 | Segmentation | ParC | T | T | 5.8 | 79.70
TABLE VIII: Ablation study conducting conducted under the framework of MobileViT[11]. BK $L/n$ denotes Big kernel with length of $1/n$ times $L$ (resolution), CA denotes channel-wise attention, and PE denotes position embedding. Model |  Params (M) |  Global Kernel |  Positional Embedding |  Circular Convolution |  ImageNet Top1 Accu (%)
---|---|---|---|---|---
ConvNeXt-XT | 7.44 | - | - | - | 77.5
ConvNeXt-BK-XT | 7.40 | T | F | F | 77.8
ConvNeXt-ParC-XT | 7.41 | T | T | T | 78.3
TABLE IX: Ablation study conducting conducted under the framework of
ConvNeXt[27]. Table shows that applying a big kernel but without circular
padding cannot boost the performance maximally.

###  4.3 Inference Speed Test

In this section, we offer the latency experiments of ParC-based networks
measured on multiple platforms. These experiments are conducted to demonstrate
two facts:

  1. 1.

ParC could significantly reduce inference latency, when applied to transformer
models. To demonstrate this, we deploy MobileViT-ParC on a widely used low
power chip Rockchip RK3288, and an in house low power neural network processor
DP2000. We use ONNX[72] and MNN[73] to port these models to chips and time
each model for 100 iterations to measure the average inference speed. The
latency result is then compared with the vanilla MobieViT.

  2. 2.

ParC could significantly boost models’ performance with a negligible increase
in complexity, when applied to convolutional models. To demonstrate this, we
measure the parameters, FLOPs and latency of ParC-based ConvNets on Xeon
E5-2680 v4 CPU. The result is then compared with their vanilla counterparts.

As shown in row 1-4 of Table VII, compared with baseline, ParC-Net is 23%
faster on Rockchip RK3288 and 3.77× faster On DP2000. Besides less FLOPs
operations, we believe this speed improvement is also brought by two factors:
a) Convolutions are highly optimized by existing tool chains that are widely
used to deploy models into these resource constrained devices; b) Compared
with convolutions, transformers are more data bandwith demanding as computing
the attention map involves two large matrices K and Q, whereas in convolutions
the kernel is a rather small matrix compared with the input feature map. In
case the bandwith requirement exceeds that of the chip design, the CPU will be
left idle waiting for data, resulting in lower CPU utilization and overall
slower inference speed.

Results in rows 5-10 show that our proposed ParC-Net universally improves
performances of common ConvNets. For ResNet50 (rows 5-6), it improves accuracy
by 0.5 with less parameters and FLOPs, and almost bring no increase to
latency. For MobileNetV2 (rows 7-8) and ConvNeXt (rows 9-10), it improves by
0.9 with slightly increase on budget.

As is illustrated in Fig. 6, by simply replacing the self-attention or
convolution operation with our new operator: 1) our Parc op improves accuracy
of transformer models by 0.2% on ImageNet classification task and saves as
much as 73% of its inference latency; 2) It improves performance of ConvNets
by 0.9% at most in accuracy on the same task without introducing obvious
negative impacts on inference speed and model size. These results demonstrate
that ParC, as a plug-and-play operator, can be plug into different models for
different purpose. But whether be apply to ViTs or ConvNets, ParC-based models
always have a satisfactory trade-off between performance and complexity.

###  4.4 Ablation Study

Using the MobileViT as a baseline model, we further conduct ablation analysis
to verify the effectiveness of the 2 components we proposed in our ParC-
MetaFormer block, Table VIII shows the results.

  1. 1.

Positional Aware Circular Convolution. The proposed ParC has two major
characteristics: a) Circular convolution brings global receptive field; b) PE
keeps spatial structure information. Experiment results confirm that both
characteristics are important. Results in rows 1-3 show that, using big kernel
can also improve accuracy, but the benefit of it reaches a saturation point
when kernel size reaches a certain level. This results are consistent with the
statement claimed in [27]. Using ParC can further improve accuracy, as shown
in rows 2-3 and 5-6. Introducing PE to ParC is also necessary. As we explained
in Section 3.1.2, using circular convolution alone can indeed capture global
features but it disturbs the original spatial structures. For classification
task, PE has no impact (rows 5-6). However, for detection and segmentation
tasks which are sensitive to spatial location, abandoning PE hurts
performances (rows 9-10 and 12-13).

  2. 2.

Channel Wise Attention. Results in rows 4 and 5 show that using channel wise
attention can improves performance. But compared with ParC, adopting channel
wise attention bring less benefit to model, which indicates that the ParC is
the main beneficial component in ParC-MetaFormer block.

Table IX further shows the ablation analysis we conducted on ConvNeXt’s
framework. In ConvNeXt-BK-XT, we adopt a vanilla large kernel the same size as
stage resolutions (i.e. 13 and 7), but no positional embedding and circular
convolution are used. Result shows that the ParC still owns a significant
advantage. This is consistent with the shortcomings of vanilla convolution
that we mentioned in Section 3.1.2.

In summary, ParC has two key designs: 1)adopting circular convolution with
global kernel to ensure the global receptive field; 2) using position
embedding to keep position sensitiveness. Both are important.

Framework | Training | Testing | Top1 Accu (%) | Row
---|---|---|---|---
ConvNeXt-XT | Vanilla | Vanilla | 77.52% | 1
ParC | ParC | 78.31% | 2
Fast-ParC | 78.31% | 3
TABLE X: ConvNeXt-XT with different implementations of ParC. Result shows that
switching between Fast-ParC and ParC could be done without any extra-
transition step. Figure 7: Spatial weight distributions of 1D Big Kernel (BK)
and ParC of the 7th block of stage 3, respectively sampled from ConvNeXt-BK-XT
and ConvNeXt-ParC-XT. Weight at each spatial location is calculated with the
mean absolute value of all channels with in one layer. Clearly, weights learnt
by vanilla big kernel convolution and ParC follow different spatial
distributions.

###  4.5 Fast-ParC Experiments

In Section 3.2 we proposed Fast-ParC, which could offer as an more efficient
alternative for ParC when input resolution is large. It can be theoretically
proved that Fast-ParC is equivalent to ParC. In this Section, we further
demonstrate these two attributes, equivalence and acceleration, by experiment.

Equivalence Test. To prove the equivalence of ParC and Fast-ParC in real
neural networks, we choose ConvNeXt-XT as framework and apply it with ParC or
Fast-ParC.

Result listed in Table X shows that the switching between Fast-ParC and ParC
need no extra transition step. Based on our code implementation, it is easy to
verify that the magnitude of the error between the outputs of Fast-ParC and
ParC is about 1e-7 averagely. Comparing row 2 with 3, we could draw the
conclusion that this error is acceptable for deep neural network, the
switching between Fast-ParC and ParC does not affect inference. This result
proves the equivalence of ParC and Fast-ParC, and demonstrates that one could
choose the better form of ParC according to the actual running experiment.

Latency Test. To test the actual acceleration improvement that Fast-ParC
provided, we measure the latency of different operators on Intel(R) Core(TM)
i7-11800H for comparison. We test the latency of one single convolution layer.
We set the input batch size as 1, channel and dimension as 96, and choose
several commonly used resolutions (i.e. 28, 56, 112, 224). Under each
resolution setting, we report the latency of Conv2d with kernel size
3$\\!\times\\!$3, Conv2d with kernel size 7$\\!\times\\!$7, ParC and Fast-
ParC. Each operation need to repeat for 200 times as warm-up, and then repeat
100 times for actual latency measuring. We use rfft/irfft instead of fft/ifft
to further speed up the calculation and save storage.

Resolution | Operation | FLOPs (M) | Latency (ms) | Row
---|---|---|---|---
28x28 | DWConv2d 3x3 | 0.68 | 0.07 | 1
DWConv2d 7x7 | 3.69 | 0.09 | 2
ParC | 2.11 | 0.34 | 3
Fast-ParC | 1.83 | 0.38 | 4
56x56 | DW-Conv2d 3x3 | 2.70 | 0.97 | 5
DW-Conv2d 7x7 | 14.8 | 1.00 | 6
ParC | 16.9 | 1.94 | 7
Fast-ParC | 8.49 | 0.79 | 8
112x112 | DW-Conv2d 3x3 | 10.8 | 3.16 | 9
DW-Conv2d 7x7 | 59.0 | 3.10 | 10
ParC | 134 | 9.09 | 11
Fast-ParC | 38.7 | 5.08 | 12
224x224 | DW-Conv2d 3x3 | 43.4 | 12.8 | 13
DW-Conv2d 7x7 | 236 | 13.2 | 14
ParC | 1079 | 44.2 | 15
Fast-ParC | 174 | 18.5 | 16
TABLE XI: The actual inference latency of different operations under different
resolution. Tests are conducted with CPU as Intel(R) Core(TM) i7-11800H,
torch=1.12.0. We set input batch size as 1, channels number as 96, resolution
as 28, 56, 112, 224 respectively.

As shown in Table XI, compared with ParC, Fast-ParC is more efficient in FlOPs
and has some advantages in run-time latency. Overall speaking, row 7-8, 11-12,
15-16 still show that Fast-ParC are indeed faster than ParC when input
resolution is large. For example with resolution as 224$\\!\times\\!$224,
Fast-ParC are 2.39 times faster than ParC. This is corresponding with the
theoretical analysis we given in Section 3.2. Row 3-4 show that when
resolution is relatively small, like 28$\\!\times\\!$28, the advantage brought
by Fast-ParC might be covered by its shortcoming (e.g. paralleling, memory
requirement), thus the acceleration effect is not obvious. Therefore, we
recommend the use of Fast-ParC when the resolution is large, especially for
downstream tasks such as classification and detection. Besides, choosing the
proper form according to actual resolution during run-time might also be a
feasible strategy. For example, according to Table XI, we could set a
threshold of 56, to distinguish whether to use ParC or Fast-ParC during actual
run-time.

##  5 Discussion

In this section, we firstly analyze the spatial distribution of weight learnt
by of vanilla convolution and ParC. Base on it, we further elaborate that they
still learn a similar feature extraction pattern.

Besides, to analyze the mechanism about ParC, we provide the result of two
commonly used visualization scheme, Effective Receptive Field (ERF)[15] and
Grad-CAM[29]. Result shows that ParC-based models are able to grab global
feature even under high resolution, and this makes model more comprehensive
when capture instance related semantic information.

###  5.1 Different Spatial Weights Distribution

We visualize kernels learned in big kernel traditional convolution and our
proposed ParC convolution and show them in Fig.7. An interesting phenomenon is
found here. In kernels of traditional convolution, from middle to both sides,
values range from big to small. Differently, kernels of ParC exactly present
an opposite distribution. However, when we take the $K_{h}/2$-sized shift into
consideration, two kind of distribution will be the same.

In summary, although the proposed ParC summing up information in a different
way, it inherently follows the feature extraction rule in traditional
convolution.

###  5.2 ParC Provides Model with Global Effective Receptive Field

Figure 8: Grad-CAM visualization of ConvNeXt-ParC-XT and vanilla ConvNeXt-XT.

Based on Fig 1, we can draw several conclusions:

  1. 1.

ParC-ConvNets obtain global effective receptive field. Notice models with ParC
have a clearly larger ERF than the vanilla ConvNets. Vanilla ConvNets mostly
focus on the center round area (red) of pictures, and its sensitivity drops
quickly as radius grows, usually vanishes in the edge/corner region (dark
blue). ParC models however owns a global ERF. Though it focus especially on
the cruciform regions (red), it still has non-zero value on other parts
(yellow or shallow blue). A more obvious merit is that ParC-ConvNeXt’s ERF do
not decay radially.

  2. 2.

ERF from different architectures show significantly different shape. ResNet50
seems to have a square outline, the MobileNetV2 and ConvNext-XT however own a
much more round outline. This might be closely related with the activation
function the networks used (ReLU, ReLU6 and GELU). Similar phenomenon[15] are
reported to exist between Sigmoid and ReLU, too. Apart from difference in
outline, ConvNeXt’s ERF also shows a square shadow inside of its round
outline. This might be resulted from its non-overlap downsampling layer. In
contrast, ResNet and MobileNetV2 have a round shadow in its round outline.

  3. 3.

The ERF of ConvNeXt is relatively larger than ResNet50 and MobileNetV2. This
is a reasonable conclusion - since ConvNeXt use 7$\\!\times\\!$7 kernel to
replace 3$\\!\times\\!$3 kernel in ResNet50. However, this only expands local
receptive fields, thus figure shows that in vanilla ConvNeXt-XT’s ERF, the
magnitude still decays radially. ParC integrate global information, offering
ConvNeXt-ParC-XT a effective receptive field over the whole image. As a
consequence, ConvNeXt-ParC-XT obtain a better performance, indicating that
vanilla large kernels are not as good as global kernels. A parallel work,
RepLKNet[16], also mentioned that increasing kernel size of ConvNeXt further
to 31$\\!\times\\!$31 could be beneficial.

###  5.3 ParC Helps Model to form More Comprehensive Focuses

The proposed ParC-based models generally follow this rule: extract local
information in its shallow layer, then integrate global information in this
deeper layer. This helps model to focus on semantic information that might be
hard for vanilla models to catch. We utilize Grad-CAM[29] to visualise the
semantic important regions of vanilla ConvNets and ParC-ConvNets. From Fig. 8,
the benefit brought by ParC in focusing on semantic important regions could be
concluded into two major points:

  1. 1.

For picture with single object, ParC helps model to capture the whole
instance, instead of focusing on some specific parts of it. The images in
columns 1,2 and 4 could support this state. While capturing features, Vanilla
ConvNet focuses on the distinguishing feature like shark tail, cockscomb or
the periphery of instances. But ParC-ConvNet seems to have a focus covering
the entirely instance, while those distinguishing features are only sub-
regions within it.

  2. 2.

For picture with multiple object, ParC helps model to capture more
individuals. As shown in columns 3 and 5 in Fig 8, when facing a picture with
multiple instances, vanilla ConvNet tends to miss some of them but ParC-
ConvNet could capture all. This characteristic of ParC might be beneficial for
dense prediction task.

##  6 Conclusion

We design a novel plug-and-play operator named ParC (Positional Aware Circular
Convolution). ParC owns a global receptive field like self-attention used in
ViT, but could be supported more conveniently by different hardware platforms
since it use pure convolution operation. We demonstrate it boosts networks’
performance on classification whether to be inserted on transformer based
networks or convolution based networks. Besides, these ParC-based models shows
superiority on downstream tasks as well. We also analyzed the inner mechanism
and its difference comparing with vanilla big kernel convolution, and then
give some convincing explanations to its superiority. Fast-ParC, an FFT-based
version of ParC is also proposed for applying ParC in condition of large
resolution. Fast-ParC operation is capable of maintaining low computation
budget even with high input resolution, making ParC a competitive general
choice for most of computer vision tasks.

## Appendix A

###  A.1 Proof of Equivalence of ParC and Fast-ParC

To reuse the classical expression form in signal analysis, a vector of length
N could be considered as a discrete sequence with N non-zero points, namely:

$\displaystyle\mathbf{x}=\\{x_{1},x_{2},...,x_{N-1}\\}\iff x(n),\quad 0\leq
n\leq N-1$

Besides, we use $((x(n)))_{N}$ to represent the periodic extension with $N$
points for sequence $x(n)$.

We first introduce an property of the sum of periodic sequence before get into
Discrete Fourier Transformer (DFT), because it will be frequently used in the
proof of following lemmas. For sequence $x((n))_{N}$, which is with period
$N$. The following lemma always holds:

Lemma 1: Sum of Periodic Sequence

$\displaystyle\sum_{i=0}^{N-1}((x(n)))_{N}$ $\displaystyle=$
$\displaystyle\sum_{i=m}^{N+m-1}((x(n)))_{N},$ $\displaystyle\forall
m\in\mathcal{N}$ (6)

This lemma shows the sum of any $N$ successive points of a periodic sequence
with period $N$ is the same, which is trivial and thus we give it without a
proof.

Usually, the DFT of a N point sequence is still a N point sequence, so we use
the following expression to denote a pair of sequences that are counterpart in
Fourier domain and spatial domain:

$\displaystyle\mathcal{F}^{-1}\\{X(k)\\}$ $\displaystyle=x(n),$ $\displaystyle
0\leq n\leq N-1$ $\displaystyle\mathcal{F}\\{x(n)\\}$ $\displaystyle=X(k),$
$\displaystyle 0\leq k\leq N-1$

while $\mathcal{F}$ and $\mathcal{F}^{-1}$ indicate the DFT and IDFT
respectively. The specific mathematical expression of DFT and IDFT could be
denoted as:

$\displaystyle X(k)$
$\displaystyle=\sum_{n=0}^{N-1}x(n)e^{-j\frac{2\pi}{N}nk}=\sum_{n=0}^{N-1}x(n)W^{nk}_{N}$
$\displaystyle x(n)$
$\displaystyle=\sum_{n=0}^{N-1}\frac{1}{N}X(k)e^{j\frac{2\pi}{N}nk}=\sum_{n=0}^{N-1}\frac{1}{N}X(k)W^{-nk}_{N}$

in which $W_{N}^{k}=e^{-j\frac{2\pi}{N}k}$ is used as a shorthand.

There exist many exquisite transformation relation for $x(n)$ and $X(k)$,
among which the time-domain shifting property and time-domain flipping
property are two commonly used theorems. They which could be written as
follows:

Lemma 2: Time-domain Flipping Property

$\displaystyle\mathcal{F}\\{x((-n))_{N}\\}$
$\displaystyle=\sum_{n=0}^{N-1}x((-n))_{N}W^{nk}_{N}$ (7)
$\displaystyle=X^{\ast}(k)$

in which $(\cdot)^{\ast}$ indicates the conjugate of $\cdot$.

Lemma 3: Time-Domain Shifting Property

$\displaystyle\mathcal{F}\\{x((n-m))_{N}\\}$
$\displaystyle=\sum_{n=0}^{N-1}x((n-m))_{N}W^{nk}_{N}$ (8)
$\displaystyle=W^{mk}_{N}\cdot X(k)$

The proof can be easily down with method of substitution, after which Eq. 6
could offer great use of resetting the index to $0\sim N\\!-\\!1$:

Proof 1: Proof of Lemma 2

$\displaystyle\mathcal{F}\\{x((-n))_{N}\\}$
$\displaystyle=\sum_{n=0}^{N-1}x((-n))_{N}W^{nk}_{N}$
$\displaystyle\xlongequal{i=-n}\sum_{i=0}^{-N+1}x((i))_{N}W^{-ik}_{N}$
$\displaystyle=[\sum_{i=-N+1}^{0}x((i))_{N}W^{ik}_{N}]^{\ast}$
$\displaystyle=[\sum_{i=0}^{N-1}x((i))_{N}W^{ik}_{N}]^{\ast}\text{~{}~{}(Eq.~{}\ref{lemma:sum_periodic})}$
$\displaystyle=X^{\ast}(k)$

The proof of lemma 3 follows a similar strategy:

Proof 2: Proof of Lemma 3

$\displaystyle\mathcal{F}\\{x((n-m))_{N}\\}$
$\displaystyle=\sum_{n=0}^{N-1}x((n-m))_{N}W^{nk}_{N}$
$\displaystyle\xlongequal{i=n-m}\sum_{i=-m}^{N-1-m}x((i))_{N}W^{(i+m)k}_{N}$
$\displaystyle=W^{mk}_{N}[\sum_{i=-m}^{N-1-m}x((i))_{N}W^{ik}_{N}]$
$\displaystyle=W^{mk}_{N}[\sum_{i=0}^{N-1}x((i))_{N}W^{ik}_{N}]\text{~{}~{}(Eq.~{}\ref{lemma:sum_periodic})}$
$\displaystyle=W^{mk}_{N}\cdot X(k)$

It’s easy to tell that the spatial-ParC we gave in Eq. 5 is equivalent to the
classical form of cross-correlation operation. The following lemma holds:

Lemma 4: The equivalence between ParC and cross-correlation

$\displaystyle w(n)\otimes x((n))_{N}$
$\displaystyle=\sum_{n=0}^{N-1}w(n)x((n+m))_{N}$ (9)
$\displaystyle=\sum_{n=0}^{N-1}x(n)w((n-m))_{N}$

in which $\otimes$ is used to represent ParC operation. (Notice when we talk
about ParC, sometime we omit the positional embedding part cause it can be
separated from circular convolution easily, while the latter one takes the
majority of computation bottleneck.) This could be proved using a similar
strategy as shown in Proof 1 and Proof 2:

Proof 3: Proof of Lemma 4

$\displaystyle w(n)\otimes x((n))_{N}$
$\displaystyle=\sum_{n=0}^{N-1}w(n)x((n+m))_{N}$
$\displaystyle=\sum_{n=0}^{N-1}w((n))_{N}x((n+m))_{N}$
$\displaystyle\xlongequal{i=n+m}\sum_{i=m}^{N-1+m}w((i-m))_{N}x((i))_{N}$
$\displaystyle=\sum_{i=m}^{N-1+m}x((i))_{N}w((i-m))_{N}$
$\displaystyle=\sum_{i=0}^{N-1}x((i))_{N}w((i-m))_{N}\text{~{}~{}(Eq.~{}\ref{lemma:sum_periodic})}$
$\displaystyle=\sum_{i=0}^{N-1}x(i)w((i-m))_{N}$
$\displaystyle=\sum_{n=0}^{N-1}x(n)w((n-m))_{N}$

Using the 4 aforementioned lemmas, we could prove the two different
implementation shown in Eq. 5 is strictly equivalent:

Proof 4: Proof of ParC Equivalence Theorem 5

$\displaystyle Y(k)$ $\displaystyle=\mathcal{F}\\{y(n)\\}$ (10)
$\displaystyle=\mathcal{F}\\{\sum_{n=0}^{N-1}w(n)x((n+m))_{N}\\}$
$\displaystyle=\mathcal{F}\\{\sum_{n=0}^{N-1}x(m)w((n-m))_{N}\\}\text{~{}~{}(Eq.~{}\ref{lemma:parc_equal})}$
$\displaystyle=\sum_{m=0}^{N-1}[\sum_{n=0}^{N-1}w((n-m))_{N}x(m)]W^{mk}_{N}$
$\displaystyle=\sum_{m=0}^{N-1}x(m)\sum_{m=0}^{N-1}w((n-m))_{N}W^{mk}_{N}$
$\displaystyle=\sum_{m=0}^{N-1}x(m)[\sum_{m=0}^{N-1}w((m-n))_{N}W^{mk}_{N}]^{\ast}\text{~{}~{}(Eq.~{}\ref{lemma:time_flipping})}$
$\displaystyle=\sum_{m=0}^{N-1}x(m)W^{\ast}(k)W^{mk}_{N}\text{~{}~{}(Eq.~{}\ref{lemma:time_shifting})}$
$\displaystyle=W^{\ast}(k)\sum_{m=0}^{N-1}x(m)W^{mk}_{N}$
$\displaystyle=W^{\ast}(k)\cdot X(k)$

###  A.2 Theoretical Time Complexity of ParC and Fast-ParC

For simplicity, we compute the multiplication time as FLOPs for the following
examples. Considering vanilla 2D convolution with kernel shaped $K_{h}\times
K_{w}$ and input feature map $H\times W$, every slide include $K_{h}\times
K_{w}$ times of MUL operation. If using zero-padding to make the output owns
the same size as input, the entire map cost $HWK_{h}K_{w}$ MUL operations in
spatial. Take channel dimension into consideration, we have the following
equation:

$\displaystyle FLOPs(\text{DW-Conv2d})$ $\displaystyle=CHWK_{h}K_{w}$ (11)
$\displaystyle=\mathcal{O}(CHWK_{h}K_{w})$ $\displaystyle
FLOPs(\text{Conv2d})$ $\displaystyle=C_{i}C_{o}HWK_{h}K_{w}$
$\displaystyle=\mathcal{O}(C_{i}C_{o}HWK_{h}K_{w})$

ParC uses positional embedding, global kernel and circular convolution. We use
positional embedding by add it to feature map, thus this requires $HW$ ADDs
and $0$ MULs. Circular convolution could be considered as a different padding
scheme, it make no difference in computational complexity comparing with
vanilla convolution. ParC use global kernel and 1D decomposition, which means
for $C/2$ we use $K_{h}=H,\ K_{w}=1$ and for the other $C/2$ we use $K_{h}=1,\
K_{w}=W$. Reuse Eq. 11, we could know ParC requires MULs of:

$\displaystyle FLOPs(\text{ParC})$ $\displaystyle=\frac{1}{2}CHW(H\cdot
1)+\frac{1}{2}CHW(1\cdot W)$ $\displaystyle=\frac{1}{2}CHW(H+W)$
$\displaystyle=\mathcal{O}(CHW(H+W))$

It’s wide acknowledged that FFT facilitate convolution when length $N$ is big.
For base-2 FFT with $N$ points, every butterfly transform costs $N/2\cdot
logN$ MUL operation. And 1D-FFT is completely parallel in the other spatial
axis and channel axis. Thus for input map and weight, horizontal 1D-FFT
requires $WH/2\cdot log_{2}\lceil H\rceil$ and $H/2\cdot log_{2}\lceil
H\rceil$ MULs respectively; and the output requires $WH/2\cdot log_{2}\lceil
H\rceil$ for 1D-iFFT. The circular convolution becomes element-wise
multiplication of expended weights and input feature map, this requires $HW$
MULs. Since FFT turns real numbers into complex numbers and $1$ complex number
multiplication equal $4$ real number multiplications, the result should be
times by $4$. Finally, all these operations are parallel in channel axis,
hence the result should be times by $C$:

$\displaystyle FLOPs(\text{Fast-ParC})$ $\displaystyle=\frac{1}{2}C\cdot
4(WHlog_{2}\lceil H\rceil+\frac{1}{2}Hlog_{2}\lceil H\rceil+HW)$
$\displaystyle\quad+\frac{1}{2}C\cdot 4(HWlog_{2}\lceil
W\rceil+\frac{1}{2}Wlog_{2}\lceil W\rceil+HW)$
$\displaystyle=2CHW(log_{2}\lceil H\rceil+log_{2}\lceil W\rceil)$
$\displaystyle\quad+CHlog_{2}\lceil H\rceil+CWlog_{2}\lceil W\rceil+4CHW$
$\displaystyle=\mathcal{O}(CHW(log_{2}\lceil H\rceil+log_{2}\lceil W\rceil))$

## References

  * [1] H. Zhang, W. Hu, and X. Wang, “Parc-net: Position aware circular convolution with merits from convnets and transformer,” in _European Conference on Computer Vision_ , 2022.
  * [2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [3] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [4] C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting unreasonable effectiveness of data in deep learning era,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 843–852.
  * [5] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European conference on computer vision_. Springer, 2020, pp. 213–229.
  * [6] Y. Fang, B. Liao, X. Wang, J. Fang, J. Qi, R. Wu, J. Niu, and W. Liu, “You only look at one sequence: Rethinking transformer in vision through object detection,” _arXiv preprint arXiv:2106.00666_ , 2021.
  * [7] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr, and L. Zhang, “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 6877–6886.
  * [8] F. Z. Zhang, D. Campbell, and S. Gould, “Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 20 104–20 112.
  * [9] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” _arXiv preprint arXiv:2204.06125_ , 2022.
  * [10] J. Guo, K. Han, H. Wu, Y. Tang, X. Chen, Y. Wang, and C. Xu, “Cmt: Convolutional neural networks meet vision transformers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 12 175–12 185.
  * [11] S. Mehta and M. Rastegari, “Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer,” _arXiv preprint arXiv:2110.02178_ , 2021\.
  * [12] Y. Chen, X. Dai, D. Chen, M. Liu, X. Dong, L. Yuan, and Z. Liu, “Mobile-former: Bridging mobilenet and transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5270–5279.
  * [13] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _International conference on machine learning_. PMLR, 2019, pp. 6105–6114.
  * [14] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4510–4520.
  * [15] W. Luo, Y. Li, R. Urtasun, and R. Zemel, “Understanding the effective receptive field in deep convolutional neural networks,” 2017. [Online]. Available: https://arxiv.org/abs/1701.04128
  * [16] X. Ding, X. Zhang, Y. Zhou, J. Han, G. Ding, and J. Sun, “Scaling up your kernels to 31x31: Revisiting large kernel design in cnns,” 2022. [Online]. Available: https://arxiv.org/abs/2203.06717
  * [17] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” _arXiv preprint arXiv:1510.00149_ , 2015.
  * [18] A. Ren, T. Zhang, S. Ye, J. Li, W. Xu, X. Qian, X. Lin, and Y. Wang, “Admm-nn: An algorithm-hardware co-design framework of dnns using alternating direction methods of multipliers,” in _Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems_ , 2019, pp. 925–938.
  * [19] X. Liu, J. Pool, S. Han, and W. J. Dally, “Efficient sparse-winograd convolutional neural networks,” _arXiv preprint arXiv:1802.06367_ , 2018\.
  * [20] V. Podlozhnyuk, “Fft-based 2d convolution,” _NVIDIA white paper_ , vol. 32, p. 1, 2007.
  * [21] A. Vasudevan, A. Anderson, and D. Gregg, “Parallel multi channel convolution using general matrix multiplication,” in _2017 IEEE 28th international conference on application-specific systems, architectures and processors (ASAP)_. IEEE, 2017, pp. 19–24.
  * [22] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 568–578.
  * [23] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 012–10 022.
  * [24] Z. Dai, H. Liu, Q. V. Le, and M. Tan, “Coatnet: Marrying convolution and attention for all data sizes,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 3965–3977, 2021.
  * [25] B. Graham, A. El-Nouby, H. Touvron, P. Stock, A. Joulin, H. Jégou, and M. Douze, “Levit: a vision transformer in convnet’s clothing for faster inference,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 12 259–12 269.
  * [26] Y. Li, G. Yuan, Y. Wen, E. Hu, G. Evangelidis, S. Tulyakov, Y. Wang, and J. Ren, “Efficientformer: Vision transformers at mobilenet speed,” _arXiv preprint arXiv:2206.01191_ , 2022.
  * [27] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 11 976–11 986.
  * [28] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [29] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.
  * [30] D. H. Hubel and T. N. Wiesel, “Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex,” _The Journal of physiology_ , vol. 160, no. 1, p. 106, 1962.
  * [31] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [32] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [33] A. Araujo, W. Norris, and J. Sim, “Computing receptive fields of convolutional neural networks,” _Distill_ , 2019. [Online]. Available: https://distill.pub/2019/computing-receptive-fields
  * [34] H. Le and A. Borji, “What are the receptive, effective receptive, and projective fields of neurons in convolutional neural networks?” _arXiv preprint arXiv:1705.07049_ , 2017.
  * [35] X. Li, W. Wang, X. Hu, and J. Yang, “Selective kernel networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 510–519.
  * [36] S. Gao, Z.-Y. Li, Q. Han, M.-M. Cheng, and L. Wang, “Rf-next: Efficient receptive field search for convolutional neural networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * [37] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 10 347–10 357.
  * [38] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 3–19.
  * [39] B. Heo, S. Yun, D. Han, S. Chun, J. Choe, and S. J. Oh, “Rethinking spatial dimensions of vision transformers,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 11 936–11 945.
  * [40] Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu, “Ccnet: Criss-cross attention for semantic segmentation,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 603–612.
  * [41] A. Srinivas, T.-Y. Lin, N. Parmar, J. Shlens, P. Abbeel, and A. Vaswani, “Bottleneck transformers for visual recognition,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 16 519–16 529.
  * [42] T. Xiao, M. Singh, E. Mintun, T. Darrell, P. Dollár, and R. Girshick, “Early convolutions help transformers see better,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 30 392–30 400, 2021.
  * [43] S. d’Ascoli, H. Touvron, M. L. Leavitt, A. S. Morcos, G. Biroli, and L. Sagun, “Convit: Improving vision transformers with soft convolutional inductive biases,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 2286–2296.
  * [44] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, 2012.
  * [45] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [46] S. Liu, T. Chen, X. Chen, X. Chen, Q. Xiao, B. Wu, M. Pechenizkiy, D. Mocanu, and Z. Wang, “More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity,” _arXiv preprint arXiv:2207.03620_ , 2022.
  * [47] Y. Rao, W. Zhao, Z. Zhu, J. Lu, and J. Zhou, “Global filter networks for image classification,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [48] X. Chu, Z. Tian, B. Zhang, X. Wang, X. Wei, H. Xia, and C. Shen, “Conditional positional encodings for vision transformers,” _arXiv preprint arXiv:2102.10882_ , 2021.
  * [49] W. Yu, M. Luo, P. Zhou, C. Si, Y. Zhou, X. Wang, J. Feng, and S. Yan, “Metaformer is actually what you need for vision,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 819–10 829.
  * [50] C. D. McGillem and G. R. Cooper, _Continuous and discrete signal and system analysis_. Harcourt School, 1991\.
  * [51] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [52] B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba, “Semantic understanding of scenes through the ade20k dataset,” _International Journal of Computer Vision_ , vol. 127, no. 3, pp. 302–321, 2019.
  * [53] J. Li, X. Xia, W. Li, H. Li, X. Wang, X. Xiao, R. Wang, M. Zheng, and X. Pan, “Next-vit: Next generation vision transformer for efficient deployment in realistic industrial scenarios,” _arXiv preprint arXiv:2207.05501_ , 2022\.
  * [54] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [55] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. Fernández del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant, “Array programming with NumPy,” _Nature_ , vol. 585, p. 357–362, 2020.
  * [56] M. Fatica, “Cuda toolkit and libraries,” in _2008 IEEE hot chips 20 symposium (HCS)_. IEEE, 2008, pp. 1–22.
  * [57] J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, “Squeeze-and-excitation networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, no. 8, pp. 2011–2023, 2020.
  * [58] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 3–19.
  * [59] Y. Li, Y. Chen, X. Dai, D. Chen, M. Liu, L. Yuan, Z. Liu, L. Zhang, and N. Vasconcelos, “Micronet: Improving image recognition with extremely low flops,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 468–477.
  * [60] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” _arXiv preprint arXiv:1611.05431_ , 2016.
  * [61] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , 2017.
  * [62] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [63] B. T. Polyak and A. B. Juditsky, “Acceleration of stochastic approximation by averaging,” _SIAM journal on control and optimization_ , vol. 30, no. 4, pp. 838–855, 1992.
  * [64] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [65] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _European conference on computer vision_. Springer, 2016, pp. 21–37.
  * [66] M. Everingham, S. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” _International journal of computer vision_ , vol. 111, no. 1, pp. 98–136, 2015.
  * [67] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger, “Deep networks with stochastic depth,” in _European conference on computer vision_. Springer, 2016, pp. 646–661.
  * [68] Z. Cai and N. Vasconcelos, “Cascade r-cnn: Delving into high quality object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 6154–6162.
  * [69] K. Chen, J. Wang, J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu, Z. Zhang, D. Cheng, C. Zhu, T. Cheng, Q. Zhao, B. Li, X. Lu, R. Zhu, Y. Wu, J. Dai, J. Wang, J. Shi, W. Ouyang, C. C. Loy, and D. Lin, “MMDetection: Open mmlab detection toolbox and benchmark,” _arXiv preprint arXiv:1906.07155_ , 2019.
  * [70] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun, “Unified perceptual parsing for scene understanding,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 418–434.
  * [71] M. Contributors, “MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark,” https://github.com/open-mmlab/mmsegmentation, 2020.
  * [72] K. Z. e. a. Junjie Bai, Fang Lu, “Onnx: Open neural network exchange,” _GitHub repository_ , 2019.
  * [73] X. Jiang, H. Wang, Y. Chen, Z. Wu, L. Wang, B. Zou, Y. Yang, Z. Cui, Y. Cai, T. Yu _et al._ , “Mnn: A universal and efficient inference engine,” _Proceedings of Machine Learning and Systems_ , vol. 2, pp. 1–13, 2020.

|  Tao Yang is an undergraduate student at Honshen Honers School, Chongqing
University. He will receive his B.S. degree in Electronic Engineering in 2023.
In 2022, He spent 3 months as an intern at AI Lab, Intellifusion. He is now
working as a research assistant at CHEN Lab, The Hong Kong Polytechnic
University (PolyU). His research interests include deep learning, computer
vision, and model compression.
---|---
|  Haokui Zhang received the Ph.D degree and the MS degree in computer
application technology from Shannxi Provincial Key Laboratory of Speech and
Image Information Processing in 2021 and 2016. From 2018 to 2020, he spent two
years as a Joint-Ph.D with the fund of CSC at AIML Lab, the University of
Adelaide. He is currently working as senior algorithm expert at AI Lab of
Intellifusion. His research interests cover approximate nearest neighbor
search, semantic segmentation, depth estimation, neural architecture search
and hyperspectral image classification.
---|---
|  Wenze Hu received a PhD degree from Department of Statistics, University of
California, Los Angeles in 2012, and was a postdoctoral scholar at the Center
for Vision Cognition Learning and Art in UCLA. He joined Google in August,
2013. He is currently working as technical director at AI Lab of
Intellifusion. His research interest include computer vision and statistical
modeling.
---|---
|  Changwen Chen received his BS from the University of Science and Technology
of China in 1983, MSEE from the University of Southern California in 1986, and
Ph.D. from the University of Illinois at Urbana-Champaign in 1992. He is
currently Chair Professor of Visual Computing at The Hong Kong Polytechnic
University. Before his current position, he served as Dean of the School of
Science and Engineering at The Chinese University of Hong Kong, Shenzhen from
2017 to 2020. He also served as an Empire Innovation Professor at the
University at Buffalo, the State University of New York from 2008 to 2021. He
was Allen Henry Endow Chair Professor at the Florida Institute of Technology
from 2003 to 2007. He was on the faculty of Electrical and Computer
Engineering at the University of Rochester from 1992 to 1996 and on the
faculty of Electrical and Computer Engineering at the University of Missouri-
Columbia from 1996 to 2003. His research interests include multimedia
communication, multimedia systems, mobile video streaming, Internet of Video
Things (IoVT), image/video processing, computer vision, deep learning,
multimedia signal processing, and immersive mobile video. He is an IEEE Fellow
(2005), a SPIE Fellow (2007), and an elected member of the Academia Europaea
(2021).
---|---
|  Xiaoyu Wang received the BE degree from the University of Science and
Technology of China, in 2006, the MS degree in statistics from the University
of Missouri, Columbia, in 2012, and the PhD degree in electrical and computer
engineering from the University of Missouri, Columbia. He is currently chief
scientist at Intellifusion. Before joining Intellifusion, he was a founding
member of Snap Research and served as chair of computer vision from 2015 to
2017. He was a research scientist at NEC Labs America from 2012-2015. He
served as area chair for WACV 2017, CVPR 2022 and CVPR 2023. His team was the
runner-up winner of ImageNet Object Recognition Challenge in 2013. His
research interests include on Computer Vision and Machine Learning.
---|---