# SCSC: Spatial Cross-scale Convolution Module to Strengthen both CNNs and
Transformers

Xijun Wang1∗, Xiaojie Chu2∗, Chunrui Han2, Xiangyu Zhang2
1The University of Maryland, College Park, Maryland, USA. 2MEGVII Technology,
Beijing, China
<EMAIL_ADDRESS>{chuxiaojie, hanchunrui<EMAIL_ADDRESS>
###### Abstract

This paper presents a module, Spatial Cross-scale Convolution (SCSC), which is
verified to be effective in improving both CNNs and Transformers. Nowadays,
CNNs and Transformers have been successful in a variety of tasks. Especially
for Transformers, increasing works achieve state-of-the-art performance in the
computer vision community. Therefore, researchers start to explore the
mechanism of those architectures. Large receptive fields, sparse connections,
weight sharing, and dynamic weight have been considered keys to designing
effective base models [39, 24, 63, 44]. However, there are still some issues
to be addressed: large dense kernels and self-attention are inefficient, and
large receptive fields make it hard to capture local features. Inspired by the
above analyses and to solve the mentioned problems, in this paper, we design a
general module taking in these design keys to enhance both CNNs and
Transformers. SCSC introduces an efficient spatial cross-scale encoder and
spatial embed module to capture assorted features in one layer. On the face
recognition task, FaceResNet with SCSC can improve 2.7% with 68% fewer
FLOPs111The number of multiply-adds operations. ∗Equal contribution. This work
was completed when Xijun Wang was an intern at MEGVII. and 79% fewer
parameters. On the ImageNet classification task, Swin Transformer with SCSC
can achieve even better performance with 22% fewer FLOPs, and ResNet with CSCS
can improve 5.3% with similar complexity. Furthermore, a traditional network
(e.g., ResNet) embedded with SCSC can match Swin Transformer’s performance.

##  1 Introduction

Figure 1: Illustration of Spatial Cross-scale Convolution Module (SCSC) with
different receptive fields in one layer. First, we decrease the input feature
maps’ channel to get the channel reduced features maps, which can save the
channel-wise convolution computational cost. Second, we apply $m$ different
Spatial encoder to get $m$ different receptive fields feature maps. Third, we
use Spatial Embed Module to merge the $m$ feature maps to get the embedded
feature maps. Finally, we recover the channel number as same as the input
feature maps to get the output feature maps.

Vision transformers have achieved impressive breakthroughs in a variety of
tasks, including classification, object detection, segmentation, video action
recognition, and _etc_. [19, 49, 39], making transformers promising backbones
for vision applications. Transformers have a strong representation ability for
their special designs, supporting a variety of data forms (tensor, set,
sequence, graph, and _etc_.), and being robust to blocks and noise [12, 18,
57, 42]. As a result, more and more researchers are trying to figure out what
makes transformers so powerful. Han et al. [24] and [63] analyze vision
transformers from the respective of receptive fields, sparse connectivity,
weight sharing, and dynamic weight, which has been considered as desired
properties in model architecture design.

For receptive field, [19, 39, 17, 46] have shown that large kernels can
efficiently introduce large receptive fields and can partially avoid the
optimization problem caused by the increase of model depth. [46, 13, 8, 43, 6]
has proved that large kernel (kernel size $\geq 5\times 5$) applied to CNN can
obtain competitive performance, especially for downstream tasks. However, the
intensive computational cost of large dense kernels makes it hard to be widely
used in practice. For Transformer, multi-head self-attention layer can
simulate a convolutional layer by linear projection operations [12], to some
extent, transformer acts like a global receptive field CNN network. But the
high computational cost of self-attention still hinder transformers to be
applied in practice.

To alleviate the intensive computational complexity, [39] and [24] have
explored the sparse connectivity property, [39] proposed shifted windowing
scheme by limiting self-attention computation to local windows. And [24]
utilizes $7\times 7$ depth-wise convolution[9, 28] to simulate the local
window scheme. However, the efficiency and spatial modeling ability still can
be improved. Moreover, the receptive field in one layer could be more diverse.

For weight sharing, both CNN and Swin-Transformer [39] have applied weight
sharing mechanism to obtain computational efficiency. The difference is CNNs
share weights across spatial dimension while Swin-Transformer shares weight
across channel dimension.

For dynamic weight, there are many works [30, 32, 59, 3] illustrate its
effectiveness on CNN. And transformers’ self-attention structure uses dynamic
weight for each input instance so that model capacity is increased.

To conclude, all the four properties listed above in various architectures
have potential to improve performance or efficiency. Convolution is efficient
because of its spatial sharing pattern, and transformers have a large capacity
because of their self-attention’s large receptive field and dynamic scheme.
Therefore, convolution and self-attention have complementary qualities, and a
well-designed module that combines all desirable properties is possible.

To deal with all the above limitations and exploit complementary qualities of
Transformer and CNN once for all, in this work, we propose a Spatial Cross-
scale Convolution Module (SCSC), which can capture microscopic and macroscopic
feature representation synchronously without intensive computational cost. As
shown in Figure 1, for SCSC, different from the mainstream CNN’s small kernel
size or the Transformer’s global receptive field, we design an intermediate
expression for the receptive field and spatial modeling ability by using a
wide range of kernel sizes from $3\times 3$ to $13\times 13$, depth-wise
convolution is applied for its efficiency. Small kernels (e.g., $3\times 3$)
have advantage in modeling low-level and detailed information as shown in the
upper red path in Figure 1, large kernels (e.g., $13\times 13$) can handle the
semantic dependence in a large receptive field as shown in the lower green
path in Figure 1. Therefore, keeping both of them can obtain diverse spatial
representations. Furthermore, we design an efficient spatial embed module to
aggregate the different spatial representation features, which can capably
integrate different levels of information. As a result, in our proposed SCSC,
we can acquire different receptive fields in one layer, share weight in both
spatial-wise and channel-wise, exploit depth-wise convolution to hold the
sparse connectivity property, and apply the spatial embed module to bring
dynamic connection across the channel dimension.

Moreover, we have evaluated the proposed SCSC in different tasks. On ImageNet
classification, Swin-T embedded with SCSC obtains 81.6% (VS Swin-T: 81.3%)
Top-1 accuracy with 22% (1G) fewer FLOPs. ConvNet embedded with SCSC obtains
82.3% Top-1 accuracy. On MS1M face recognition, FaceResNet embedded with SCSC
achieves 95.6% (VS FaceResNet: 92.6%) rank-1 face identification accuracy with
68% fewer FLOPs and 79% fewer parameters . On COCO detection, Swin-SCSC with
Mask R-CNN gains 43.2% (VS Swin-T: 42.7) box AP with 23G fewer FLOPs. On
ADE20K segmentation, ResNet-SCSC achieve 45.7% mIoU (VS Swin-T: 44.4%). These
experiments demonstrate the effectiveness of our proposed SCSC.

To summarize, we make the following contributions:
1) Present a high capacity and effective convolution module SCSC, which can
dynamically combine a large range of receptive fields (pixel-wise) in one
layer to enhance the presentation ability.
2)  Architectures applied with SCSC can obtain better performance with fewer
computational cost and parameters. Furthermore, SCSC is a general module and
can be applied to strengthen both CNNs and Transformers.
3) SCSC module can power the classical neural networks (e.g., ResNet50) to
achieve comparable performance with strong Transformers (e.g., Swin).

##  2 Related Work

###  2.1 Vision Transformers

Recently, increasing Vision Transformers obtain state-of-the-art performance
on visual tasks [34, 23]. To improve the original vision Transformer (ViT)
[19], [10] offers a conditional positional encoding (CPE) technique. Unlike
prior fixed or learnable positional encodings, which are pre-defined and
independent of input tokens, CPE is dynamically produced and conditioned by
the immediate neighborhood of the input tokens. For self-attention, [14]
provides gated positional self-attention (GPSA), a kind of positional self-
attention that includes a ”soft” convolutional inductive bias. And [21]
presents a new attention mechanism called external attention, which
substitutes self-attention in existing popular architectures. External
attention is linear in complexity and implicitly considers all data samples’
correlations. By preserving encoder branches at various scales while engaging
attention across scales, [55] introduces a co-scale mechanism to image
Transformers. [62] creates a progressive tokenization system in order to
overcome the restriction of ViT when training from scratch on a medium-sized
dataset such as ImageNet.

To make Transformer more efficient, [55] introduces Convolutional Vision
Transformer (CvT), which enhances the performance and efficiency of Vision
Transformer (ViT) by incorporating convolutions into ViT to provide the best
of both designs. [39] proposes a hierarchical Transformer named Swin
Transformer that computes its representation using Shifted windows. The
shifted windowing scheme brings greater efficiency by limiting self-attention
computation to non-overlapping local windows while also allowing for cross-
window connection, making Swin an efficient and effective Vision Transformer
architecture. To go further, [17] creates CSWin Self-Attention, which divides
multi-heads into parallel groups and conducts self-attention in horizontal and
vertical stripes.

For different transformers design and learning strategies, [49] constructs
competitive convolution free transformers DeiTs, which can compete with the
state-of-the-art on ImageNet without using any external data at that time.
They also present a transformer-specific teacher-student method. [51] offers
Pyramid Vision Transformer (PVT), which addresses the challenges of applying
Transformer to a variety of dense prediction applications. Moreover,
CrossFormer [52] uses cross-scale convolution as a downsample (embedding)
layer which blends each embedding with multiple patches of different scales
for self-attention module.

Instead of proposing algorithms only to complement the shortcomings of ViTs,
our method absorbs the advantages of vision transformer (e.g. large receptive
field) and go a further step to benefit both transformer and CNN.

###  2.2 Large Kernel and Spatial Modeling

In the exploration of large kernels, DetNAS [8] chooses large-kernel blocks in
low-level layers and deep blocks in high-level layers. [43] discovers that the
large kernel (and effective receptive field) plays a crucial role when we have
to do classification and localization tasks at the same time (e.g., semantic
segmentation). To solve both classification and localization challenges in
semantic segmentation, [43] presents a Global Convolutional Network. Deeplab
[6] applies ”atrous convolution” with upsampled filters for dense feature
extraction for semantic segmentation and expands it even further to atrous
spatial pyramid pooling, which stores objects and visual information at
several scales. ConvNet [40] revisits the use of large kernel-sized ($7\times
7$) convolutions and RepLKNet [16] further scale up receptive fields using
$31\times 31$re-parameterized large depth-wise convolutions. However, RepLKNet
[16] focus on large models with a large number of parameters ($\geq$ 79M).

To better model spatial information, [13] presents deformable ConvNets for
modeling dense spatial transformation to learn receptive fields adaptively.
Inception family (e.g., GoogLeNet [47] and Inception-V4 [46]) extract multi-
scale features by different convolutional kernels and fuse them statically
using concatenation. DRConv [3] learns a guided mask to assign different
customized weights to different spatial regions for better spatial
representation. SKNet [36] proposes a dynamic selection technique in CNNs that
allows each neuron to modify its receptive field size adaptively based on
different scales of input.

Compared with these methods, we additionally introduce large range of
receptive field (especially large receptive field), and consider weight
sharing and dynamic mechanism at the same time.

###  2.3 Dynamic Mechanism

With the prevalence of data dependency mechanism [1, 31, 50] , which
emphasizes to extract more customized feature for diverse representation [41].
Benefited from the data dependency mechanism, networks can flexibly adjust
themselves, including the structure and parameters, to automatically fit the
fickle information to improve the representation ability of neural networks.
[5, 54] indicate that different regions in the spatial dimension are not
equally important in representation learning and should be processed
differently. For instance, activation in important regions needs to be
amplified to play a dominant role in the forward propagation. SKNet [36]
designs a dynamic module to channel-wisely select suitable receptive fields
based on channel attention and achieves better performance. It dynamically
restructures the networks for the sake of different receptive fields with
dilated convolutions [60, 61].

From the aspect of dynamic weights, CondConv [59] obtains dynamic weights by
the dynamical linear combination of several weights. And the specialized
convolution kernels for each sample are learned in a way similar to the
mixture of experts. In the spatial domain, to handle object deformations,
Deformable Kernels [20] directly resamples the original kernel space to adapt
the effective receptive field (ERF) while leaving the receptive field
untouched. Local Relation Networks [29] adaptively determine aggregation
weights for spatial dimension based on the compositional relationship of local
pixel pairs. Non-local [53] operation computes the response at each position
as the weighted sum of the features at all positions, which can make it
capture long-range dependencies. Different from above dynamic methods, apart
from dynamic filters, DRConv achieves a dynamic guided mask to automatically
determine the distribution of multiple filters so that it can process variable
distribution of spatial semantics. However, these methods have high
computational and memory complexity which significantly limits the efficiency
of the model.

###  2.4 CNN VS Transformers

Apart from designing new Vision Transformers, some works focus on exploring
the relationship between Transformer and CNN. [63] performs empirical research
on various DNN frameworks (e.g., CNN, Transformer, and MLP) in order to grasp
their benefits and drawbacks better. [12] shows that self-attention layers can
learn to behave similar to convolutional layers. [48] discovers that CNN-based
pre-trained models are competitive and outperform their Transformer
counterparts in some NLP settings. [24] recasts local attention as a channel-
wise locally-connected layer and empirically find that the models based on
depth-wise convolution with lower processing complexity perform on par with or
somewhat better than Swin Transformer. In this paper, we propose an effective
general convolution module, which narrows the gap between CNNs and
Transformers.

##  3 Method

Figure 2: Design details of Spatial Cross-scale Convolution Module (SCSC).

For CNNs and Transformers, CNNs can exploit large kernels to obtain large
receptive fields, and Transformers can get global receptive fields through
self-attention. Nevertheless, both large dense kernels and self-attention
mechanisms cannot avoid the high computational cost. Moreover, large kernels
may hinder the model to capture local features. Therefore, find an efficient
way to obtain large effective receptive fields and keep the feature diversity
meanwhile attracts increasing attention.

Since depth-wise convolution is widely used in modern CNNs for its efficiency
and effectiveness, it becomes feasible solution to introduce large receptive
fields without intensive computational cost. As analyzed in [24], depth-wise
convolution can resemble local attention (Local Vision Transformer, e.g.,
Swin-Transformer [39]) in sparse connectivity. But the fixed large windows or
kernels may not be a good manner to capture local features and still have
space to be improved.

Therefore, based on depth-wise convolution, we applied a wide range of kernel
size from $3\times 3$ to $13\times 13$ to obtain effective receptive fields,
so as to capture microscopic and macroscopic feature representation. This way
can capture different receptive fields in one layer and strengthen the spatial
modeling ability. Furthermore, we put forward an efficient spatial embed
module to combine different level spatial features, which can further enhance
the overall presentation ability.

|  | downsp. rate
---
(output size)
Swin | Swin-SCSC | ResNet | ResNet-SCSC
stage1 |  | 4×
---
(56×56)
| concat 4×4, 96-d, LN
---
$\left[\begin{array}[]{c}\text{win. sz. 7\texttimes 7}\\\ \text{dim 96, head
3}\end{array}\right]\times 2$
| concat 4×4, 96-d, LN
---
SCSC block
$\left[\begin{array}[]{c}\text{kernel sz.}\\\
\text{[3,11]}\end{array}\right]\times 2$
| bottleneck block
---
$\left[\begin{array}[]{c}\text{kernel sz.}\\\
\text{[1,3]}\end{array}\right]\times 3$
| SCSC block
---
$\left[\begin{array}[]{c}\text{kernel sz.}\\\
\text{[3,9,13]}\end{array}\right]\times 3$
stage2 |  | 8×
---
(28×28)
| concat 2×2, 192-d, LN
---
$\left[\begin{array}[]{c}\text{win. sz. 7\texttimes 7}\\\ \text{dim 96, head
3}\end{array}\right]\times 2$
| concat 2×2, 192-d, LN
---
SCSC block
$\left[\begin{array}[]{c}\text{kernel sz.}\\\
\text{[3,9]}\end{array}\right]\times 2$
| bottleneck block
---
$\left[\begin{array}[]{c}\text{kernel sz.}\\\
\text{[1,3]}\end{array}\right]\times 4$
| SCSC block
---
$\left[\begin{array}[]{c}\text{kernel sz.}\\\
\text{[3,7,11]}\end{array}\right]\times 5$
stage3 |  | 16×
---
(14×14)
| concat 2×2, 384-d, LN
---
$\left[\begin{array}[]{c}\text{win. sz. 7\texttimes 7}\\\ \text{dim 96, head
3}\end{array}\right]\times 6$
| concat 2×2, 384-d, LN
---
SCSC block
$\left[\begin{array}[]{c}\text{kernel sz.}\\\
\text{[3,7]}\end{array}\right]\times 6$
| bottleneck block
---
$\left[\begin{array}[]{c}\text{kernel sz.}\\\
\text{[1,3]}\end{array}\right]\times 6$
| SCSC block
---
$\left[\begin{array}[]{c}\text{kernel sz.}\\\
\text{[3,5,7]}\end{array}\right]\times 12$
stage4 |  | 32×
---
(7×7)
| concat 2×2, 768-d, LN
---
$\left[\begin{array}[]{c}\text{win. sz. 7\texttimes 7}\\\ \text{dim 96, head
3}\end{array}\right]\times 2$
| concat 2×2, 768-d, LN
---
SCSC block
$\left[\begin{array}[]{c}\text{kernel sz.}\\\
\text{[3,5]}\end{array}\right]\times 2$
| bottleneck block
---
$\left[\begin{array}[]{c}\text{kernel sz.}\\\
\text{[1,3]}\end{array}\right]\times 3$
| SCSC block
---
$\left[\begin{array}[]{c}\text{kernel sz.}\\\
\text{[3,5]}\end{array}\right]\times 3$
Table 1:  Detailed architecture specifications. The kernel size (sz.) of our
SCSC can be any combination for the specific tasks, here we just give some
examples.

###  3.1 SCSC: Spatial Cross-scale Convolution Module

For clear illustration, we split SCSC into four steps: Channel Reduction,
Spatial Cross-scale Encoder, Spatial Combination, and Channel Recover. Taking
a $K$ layers CNN for example, the input of the $k_{th}$ layer can be denoted
as $X^{k}\in\mathbb{R}^{N\times C\times H\times W}$, where N, C, H, W are
batchsize, channel, height, and width respectively. As shown in Figure 2, for
Channel Reduction, we use conv $1\times 1$ as the mapping function,

$\displaystyle X^{k}_{d}=W^{k}_{d}\otimes X^{k},$ (1)

where $\otimes$ denotes convolutional operation,
$X^{k}_{d}\in\mathbb{R}^{N\times C//m\times H\times W}$, m is a constant and
means how many different kernels in Spatial Cross-scale Encoder. Then we
calculate the Spatial Cross-scale Encoder by

$\displaystyle
X^{k}_{s}=\left\\{X^{k}_{s_{1}},...,X^{k}_{s_{i}},...,X^{k}_{s_{m}}\right\\},$
(2) $\displaystyle X^{k}_{s_{i}}=W^{k}_{s_{i}}\otimes X_{d}^{k}\ ,\
W^{k}_{s_{i}}\in\mathbb{R}^{C//m\times C//m\times v\times v},$ (3)

in which $i\in[1,m],v\in[3,13],X^{k}_{s_{i}}\in\mathbb{R}^{N\times C//m\times
H^{\prime}\times W^{\prime}}$. After getting different level spatial features,
we design a Spatial Embed Module to dynamically fuse multi-scale features at
each spatial position (i.e., pixel level).

$\displaystyle\mathcal{M}^{k}=SEM(X^{k}_{d}),$ (4)

in $SEM()$, we use $1\times 1$ conv to reduce the channel number to $m*g$,
then apply sigmoid function, the result noted as
$\mathcal{M}^{k}\in\mathbb{R}^{N\times m*g\times H^{\prime}\times
W^{\prime}}$, $g$ is a constant and means the combination times, which can
bring more dynamic combinations. We reshape the $X^{k}_{s}$ to $N\times
g*H^{\prime}*W^{\prime}\times C//m//g\times m$, and $\mathcal{M}^{k}$ to
$N\times g*H^{\prime}*W^{\prime}\times m\times 1$. and then we can get the
embedded feature,

$\displaystyle X^{k}_{e}=Matmul(X^{k}_{s},\mathcal{M}^{k}),$ (5)

where $X^{k}_{e}\in\mathbb{R}^{N\times C//m\times H^{\prime}\times
W^{\prime}}$.

Finally, we product Channel Recover operation by

$\displaystyle X^{k}_{o}=W^{k}_{r}\otimes X^{k}_{e},$ (6)

and $X^{k}_{o}\in\mathbb{R}^{N\times C\times H^{\prime}\times W^{\prime}}$.
All the operations are differentiable, so our objective function is

$\displaystyle\min\limits_{W^{k}_{d},W^{k}_{s},W_{\mathcal{M}^{k}},W^{k}_{r}|_{k=1}^{K}}\frac{1}{n}\sum_{i=1}^{n}\mathcal{L}(Y_{i};\hat{Y}|X_{i}),$
(7)

in which $W$ is the learnable weights, $(X_{i},Y_{i}),i\in[1,n]$ is the input
image and label, $\hat{Y}$ is the predicted label. $\mathcal{L()}$ is the loss
function.

###  3.2 Architecture

The proposed SCSC module can be easily embedded in any existing architecture.
As shown in Table 1, taking the most used CNN ResNet for example, we directly
replace the original bottleneck block of ResNet-50 with our SCSC block. For
Transformer, we replace the self-attention with the proposed SCSC module, in
which the pre-linear-projections and post-linear-projections over the values
can be regarded as the 1×1 convolutions in our design.

Model | Image Resolution | Kernel Size Range | Params | FLOPs | Top-1 Acc. (%)
---|---|---|---|---|---
ResNet-50 [27] | $224\times 224$ |  $1\times 1$ to $3\times 3$ | 26M | 4.1G | 76.2
ResNet-101 [27] | $224\times 224$ |  $1\times 1$ to $3\times 3$ | 45M | 7.9G | 77.4
ResNet-152 [27] | $224\times 224$ |  $1\times 1$ to $3\times 3$ | 60M | 11.6G | 78.3
SKNet-50 [36] | $224\times 224$ |  $1\times 1$ to $5\times 5$ | 27M | 4.5G | 79.3
SKNet-100 [36] | $224\times 224$ |  $1\times 1$ to $5\times 5$ | 49M | 8.5G | 79.8
SENet [30] | $224\times 224$ |  $1\times 1$ to $3\times 3$ | 22M | 3.9G | 79.9
ResNeXt [58] | $224\times 224$ |  $1\times 1$ to $3\times 3$ | 25M | 4.2G | 80.1
ResNet-SCSC-V1 (Ours) | $224\times 224$ |  $1\times 1$ to $13\times 13$ | 10M | 1.7G | 79.4
ResNet-SCSC-V2 (Ours) | $224\times 224$ |  $1\times 1$ to $13\times 13$ | 12M | 2.2G | 80.3
ResNet-SCSC-V3 (Ours) | $224\times 224$ |  $1\times 1$ to $13\times 13$ | 25M | 4.5G | 81.5
ViT-B/16 [19] | $384\times 384$ | Global Receptive Field | 86M | 55.4G | 77.9
ViT-L/16 [19] | $384\times 384$ | Global Receptive Field | 307M | 190.7G | 76.5
DeiT-S [49] | $224\times 224$ | Global Receptive Field | 22M | 4.6G | 79.8
DW-Conv.-T [24] | $224\times 224$ | $7\times 7$ | 24M | 3.8G | 81.3
Swin-T [39] | $224\times 224$ | Local Window $7\times 7$ | 28M | 4.5G | 81.3
ConvNet-T [40] | $224\times 224$ | $7\times 7$ | 28M | 4.5G | 82.1
Swin-T-SCSC (Ours) | $224\times 224$ |  $1\times 1$ to $11\times 11$ | 22M | 3.5G | 81.6
ConvNet-T-SCSC (Ours) | $224\times 224$ |  $1\times 1$ to $11\times 11$ | 28M | 4.5G | 82.2
Table 2:  Comparison of Top-1 classification accuracy with different
architectures (CNNs and Transformers) and some state-of-the-art backbones on
ImageNet.

###  3.3 Discussion and Limitation

Some existing methods have contributed to exploring the effectiveness of
multi-scale features, but they are suboptimal in terms of efficiency. For
example, PSP [26] and ASPP [6] use sampling operators (e.g., pooling or atrous
convolution) to efficiently extract multi-scale features, but this comes with
the expense of losing spatial information. CrossFormer and GoogleNet use
resource-intensive operations, such as regular convolution with large kernels,
to extract multi-scale features, and they fuse them through static
concatenation. However, this approach significantly increases model
parameters.

In contrast, our SCSC approach not only introduces a wide range of receptive
fields for effective representation learning but also incorporates weight
sharing and dynamic mechanisms for greater efficiency. This allows our model
to achieve high performance while keeping the model size and computation cost
low.

In details, first, our SCSC is compact and effective, the Channel Reduction
step decreases the size of the input feature maps, and then inputs feature
maps into the Spatial Cross-scale Encoder, which consists of depth-wise
convolution with a wide range of kernel size to model the spatial information.
This manner can effectively avoid the intensive computational cost. In the
meanwhile, the wide range of kernel size provides different receptive fields
in one layer, small kernels for the detailed local information, large kernels
for the semantic dependence. Furthermore, we design a dynamic Spatial Embed
Module to merge the different spatial information. The proposed SCSC naturally
aggregates the advantage of CNN and Transformer, small kernels and weight
sharing from the CNN, large receptive fields and dynamic from the Transformer.
However, it still has its limitation: depth-wise convolution may need
exceptional acceleration in practice.

##  4 Experiments

In this section, we evaluate the effectiveness of our proposed SCSC module by
embedding it into the classical CNN backbones (ResNet-50 [27], MobileFaceNet
[7] and FaceResNet [15]), and state-of-the-art Swin-Transformer [39]
respectively. We conduct experiments for SCSC-based architectures on ImageNet
[45], MS1M-V2 [22], COCO 2017 [38] and ADE20K[64] in terms of image
classification, face recognition, object detection and segmentation.

###  4.1 Classification

We take classical ResNet-50 [27] and state-of-the-art Swin-Transformer [39] as
the backbone to evaluate SCSC by replacing their original components
respectively.

Settings: The ImageNet 2012 dataset [45] is a well-known image classification
dataset includes $1.28$ million training images and 50k validation images from
1000 classes. All our models are trained on the whole training dataset and
validated using the single-crop top-1 validation accuracy. The training
settings follow [39], all models in our experiments are trained for 300 epochs
with AdamW [35] optimizer and a cosine decay learning rate scheduler starts
from 0.001/0.0005. We set batch size as 1024 and weight decay as 0.05.

Resnet50 with SCSC. We directly replace the original bottleneck block with our
SCSC block. Following the mainstream works, we refer to the four residual
stages of ResNet-50 as c2, c3, c4, c5, respectively. And the block number for
the four stages are 3, 4, 6, 3, respectively. For ResNet-SCSC-V1, the four
stages’ input/output channel number are 96, 192, 384, 512, expansion is set as
2. The kernel sets for the four stages are [3,9,13], [3,7,11], [3,5,7] and
[3,5] respectively. We increase the numbers of blocks in c2,c3,c4,c5 from
3,4,6,3 to 3,4,8,3 so that the FLOPs can match the original network. For
ResNet-SCSC-V2, most setting keep the same as ResNet-SCSC-V1, except
increasing the numbers of blocks in c2,c3,c4,c5 from 3,4,8,3 to 3,5,12,3. For
ResNet-SCSC-V3, most settings keep the same as ResNet-SCSC-V2, except the
expansion is set as 3. We list the architecture of ResNet-SCSC-V3 in Table 1.

Swin-Transformer with SCSC. We replace local self-attention with our SCSC
module. The pre-linear-projections and post-linear-projections over the values
can be regarded as the 1×1 convolutions in our design.

The results are shown in Table 2. As can be seen, for ResNet-SCSC-V1 and
ResNet-SCSC-V2, we use much less computational cost and parameters to obtain
competitive results. For ResNet-SCSC-V3, with comparable model complexity, we
successfully make the performance of a classical CNN architecture (ResNet)
match the Transformers’ performance. For Swin-Transformer, with our SCSC
module, it achieves a higher accuracy with 1G (22%) FLOPs fewer computational
cost. These results show that SCSC-based architectures not only have a
considerable improvement, but also saving computational cost and parameters,
demonstrating the effectiveness of our method.

###  4.2 Face Recognition

We further evaluate the effectiveness of our SCSC on Face Recognition task.
MobileFaceNet [7] and FaceResNet [15] are applied as our backbone with input
size $96\times 96$.

Settings: MS1M-V2 dataset is a large-scale face dataset with 5.8M images from
85k celebrities. We use a refined semi-automatic version of the MS-Celeb-1M
dataset [22] which consists of 1M photos from 100k identities for training.
The dataset we use for validation is MegaFace [33], which includes 1M images
of 60k identities as the gallery set and 100k images of 530 different
individuals. We use SGD with a momentum of 0.9 to optimize the model, and the
batch size is 512. We train all the models for 420k iterations. The learning
rate begins with 0.1 and is divided by 10 at 252k, 364k, and 406k iterations.
For evaluation, we use face identification metric which refers to the rank-1
accuracy on MegaFace as the evaluation indicator.

MobileFaceNet with SCSC. We directly replace the original bottleneck block
with our SCSC block. In [7], there are 5 stages, we denote them as s1, s2, s3,
s4, s5, respectively. And the block number for the five stages are 5, 1, 6,
1,2, respectively. For MobileFaceNet-SCSC, the kernel sets for the five stages
are [3,9], [3,7], [3,7], [3,5] and [3,5] respectively. The kernel size is
relatively small because of the input image size is small. We set expansion as
3 so that not reducing too many FLOPs since MobileFaceNet is already an
efficient network.

FaceResNet with SCSC. As in FaceResNet [15], we denote the four residual
stages of FaceResNet as c2, c3, c4, c5, respectively. And the block number for
the four stages are 3, 2, 2, 2. For FaceResNet-SCSC, We set the numbers of
blocks in c2,c3,c4,c5 as 6,6,6,4. The kernel sets for the four stages are
[5,11], [3,9], [3,5] and [3,3] respectively.

Model | Params | FLOPs | Acc.(%)
---|---|---|---
MobileFaceNet [7] | 0.98M | 162M | 90.9
MobileFaceNet-SCSC | 0.89M | 146M | 92.0
FaceResNet [15] | 40.3M | 1050M | 92.9
FaceResNet-SCSC | 8.3M | 330M | 95.6
Table 3: Results of SCSC on Megaface. “Acc.” refers to the rank-1 face
identification accuracy with 1M distractors.

As Table 3 shows, given that MobileFaceNet is already an efficient network,
MobileFaceNet-SCSC still outperforms the baseline by 1.1% gain even with fewer
FLOPs and parameters. Moreover, FaceResNet-SCSC surpasses FaceResNet by a
large margin of 2.7% accuracy with 68% fewer computational cost and 79% fewer
parameters, further indicating the superiority of our proposed SCSC.

Backbone | box AP | mask AP | FLOPs | Params
---|---|---|---|---
R50 [27] | 38.2 | 34.7 | 260.1G | 44.2M
R101 [27] | 40.0 | 36.1 | 336.2G | 63.2M
X101-32x4d [58] | 41.9 | 37.5 | 340.0G | 62.8M
X101-64x4d [58] | 42.8 | 38.4 | 493.4G | 101.9M
Swin-T [39] | 42.7 | 39.3 | 263.8G | 47.8M
ResNet-SCSC | 44.0 | 40.4 | 270.8G | 44.9M
Swin-SCSC | 43.2 | 39.6 | 240.5G | 41.8M
Table 4: Results of object detection and instance segmentation on the COCO mini-val with Mask R-CNN (1x schedule). FLOPs are measured on an 800 $\times$ 1280 image. Backbone | box AP | mask AP | FLOPs | Params
---|---|---|---|---
R50 [27] | 46.3 | 43.4 | 739G | 82M
DeiT-S [49] | 48.0 | 41.4 | 889G | 80M
Swin-T [39] | 50.4 | 43.7 | 742G | 86M
DW Conv.-T [24] | 49.9 | 43.4 | 730G | 82M
ResNet-SCSC | 50.3 | 43.7 | 749G | 83M
Swin-SCSC | 49.9 | 43.2 | 719G | 80M
Table 5: Results of object detection and instance segmentation performance on
the COCO mini-val with Cascade Mask R-CNN (3x schedule). FLOPs are measured on
an 800 $\times$ 1280 image.

###  4.3 Detection

To evaluate the effectiveness of our SCSC on object detection, we utilize the
COCO 2017 dataset [38] which consists of 80k train images and 40k val images.

Settings: we exploit Mask R-CNN [25] and Cascade Mask R-CNN [2] framework with
FPN [37]. For Mask-RCNN, the implementation is based on MMDetection [4]. We
train the detection networks on 8 GPU with mini-batch 2 per GPU for 1x
schedule (12 epochs). We use AdamW optimizer, and the initial learning rate is
0.0001, started after 500 iteration warmup and decayed by 0.1 times at the 8th
and 11th epoch. We use stochastic drop path regularization of 0.2 and weight
decay of 0.05. All other hyper-parameters follow the default settings in
MMDetection [4]. For Cascade Mask-RCNN, we follow the implementation,
training, and test settings from Swin Transformer [39]. Backbone weights are
initialized by the parameters of Swin-SCSC and ResNet-SCSC respectively.

As shown in Table 4, we compare Swin-SCSC and ResNet-SCSC with standard
ConvNets, i.e., ResNe(X)t, and previous Transformer networks, e.g., Swin-T
with Mask R-CNN. The comparisons are conducted by changing only the backbones
with other settings unchanged. Our ResNet-SCSC architecture achieves 5.0%
improvement over baseline ResNet-50. Compared with strong baseline Swin-T,
ResNet-SCSC improves 1.3% box AP and 1.1% mask AP over Swin-T. Swin-SCSC
achieves better performance even with $23G$ FLOPs fewer computational cost and
$6M$ fewer parameters, illustrating that SCSC also works on downstream tasks.

###  4.4 Semantic Segmentation

Backbone | mIoU | FLOPs | Params
---|---|---|---
R50 [27] | 42.1 | 952G | 67M
R101 [27] | 43.8 | 1029G | 86M
DeiT-S [49] | 42.9 | 1099G | 52M
DW Conv.-T [24] | 45.5 | 928G | 56M
Swin-T [39] | 44.4 | 941G | 60M
ResNet-SCSC | 45.7 | 956G | 64M
Swin-SCSC | 44.0 | 916G | 54M
Table 6: Results of semantic segmentation on the ADE20K val set with UperNet.
FLOPs are measured on an 512 $\times$ 2048 image.

Settings: ADE20K[64] is a widely used semantic segmentation dataset, covering
a broad range of 150 semantic categories. It has 25K images, with 20K for
training, 2K for validation, and another 3K for testing. We utilize the
implementation of UperNet [56] in MMSegmentation [11] as our base framework
for its high efficiency. We use the same setting as the Swin Transformer [39].
Models are trained on 8 GPUs with mini-batch 2 per GPU for 160k iterations. We
employ the AdamW optimizer with an initial learning rate of 0.00006, a weight
decay of 0.01, a scheduler that uses linear learning rate decay, and a linear
warmup of 1,500 iterations. SyncBN and stochastic depth with the ratio of 0.3
is applied for Swin-SCSC. The experimental results are reported as single-
scale testing. All other hyper-parameters follow the default settings in
MMSegmentation [11].

Table 6 lists the mIoU, FLOPs and model size (number of parameters) for
different backbones. Compared with Swin-T, our ResNet-SCSC achieves +1.3 mIoU
higher (45.7 vs. 44.4) than Swin-T with slightly larger model size and
computation cost. Our Swin-SCSC achieves comparable performance with 25G less
computation cost and 6M fewer parameters.

###  4.5 Ablation Study

We conduct a series of experiments to evaluate the impact of specific design
choices in our spatial Cross-scale Convolution Module. We use the ImageNet
classification dataset [45] for our experiments, and follow the same
experimental settings as described in section 4.1 by default.

The effectiveness of combination number $g$ in Spatial Embed Module: We use
ResNet-SCSC to explore which combination number $g$ in Spatial Embed Module is
better. We set $g$ as 2, 4, 8, respectively, and the Top-1 accuracy is 81.1%,
81.5%, and 81.2% respectively. Therefore, we set $g=4$ by default.

The Effect of Spatial Embed Module: We design the Spatial Embed Module (SEM)
for fusing multi-scale features dynamically. For ResNet-SCSC, if we only use
7x7 kernel size, we will get 80.6% top-1 accuracy. Adding Spatial Cross-scale
Encoder, the accuracy can reach 81.0%. Further adding Spatial Embed Module, we
can obtain 81.6% accuracy. This shows the effectiveness of dynamic fusing by
using our SEM.

Figure 3: Influence of DW kernel size in Spatial Embed Module. We set DW
kernel size as $3\times 3$, $5\times 5$, $7\times 7$, $9\times 9$, $11\times
11$, $13\times 13$, SCSC setting, respectively. Larger kernel will result
better performance, but when the kernel size increase too large, the accuracy
begin to decline. Our SCSC achieves a large improvement by using different
receptive fields in one layer

Kernel size: We train Swin-SCSC for 100 epochs to explore the choices of
kernel sizes. We set the size of depth-wise (DW) kernels with $3\times 3$,
$5\times 5$, $7\times 7$, $9\times 9$, $11\times 11$, $13\times 13$, as well
as our SCSC setting. As shown in Figure 3, larger kernels results in better
accuracy but when the kernel size exceeds $11\times 11$, accuracy begins to
decline. Our results demonstrate that both excessively small and excessively
large kernel sizes are suboptimal. Furthermore, using different receptive
fields in one layer achieves the highest accuracy and significantly
outperforms other methods. This suggests that utilizing multiple receptive
fields can effectively enhance the performance of model.

##  5 Conclusion and Future Work

Different from the mainstream CNN’s small kernel size and the Transformer’s
global receptive field, we increase the spatial modeling ability by using a
wide range of kernel sizes. Furthermore, we design an efficient spatial
embedding module to merge the different spatial representation features. As a
result, the proposed SCSC naturally combines the advantage of CNNs and
Transformers, small kernel and weight sharing from the CNN, large receptive
field and dynamic from the Transformer. Intensive experiments illustrate the
effectiveness and generalization of our SCSC. In the future, we will explore
more about the relationship between CNNs and Transformers.

## References

  * [1] Alan Allport.  Visual attention.  The MIT Press, 1989.
  * [2] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6154–6162, 2018.
  * [3] Jin Chen, Xijun Wang, Zichao Guo, Xiangyu Zhang, and Jian Sun.  Dynamic region-aware convolution.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8064–8073, 2021.
  * [4] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al.  Mmdetection: Open mmlab detection toolbox and benchmark.  arXiv preprint arXiv:1906.07155, 2019.
  * [5] Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua.  Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5659–5667, 2017.
  * [6] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 40(4):834–848, 2017.
  * [7] Sheng Chen, Yang Liu, Xiang Gao, and Zhen Han.  Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices.  In Chinese Conference on Biometric Recognition (CCBR), pages 428–438, 2018.
  * [8] Yukang Chen, Tong Yang, Xiangyu Zhang, Gaofeng Meng, Xinyu Xiao, and Jian Sun.  Detnas: Backbone search for object detection.  pages 6642–6652, 2019.
  * [9] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1251–1258, 2017.
  * [10] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen.  Conditional positional encodings for vision transformers.  arXiv preprint arXiv:2102.10882, 2021.
  * [11] MMSegmentation Contributors.  MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation, 2020.
  * [12] Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi.  On the relationship between self-attention and convolutional layers.  arXiv preprint arXiv:1911.03584, 2019.
  * [13] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In IEEE International Conference on Computer Vision (ICCV), pages 764–773, 2017.
  * [14] Stéphane d’Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun.  Convit: Improving vision transformers with soft convolutional inductive biases.  arXiv preprint arXiv:2103.10697, 2021.
  * [15] Xiaohan Ding, Chunlong Xia, Xiangyu Zhang, Xiaojie Chu, Jungong Han, and Guiguang Ding.  Repmlp: re-parameterizing convolutions into fully-connected layers for image recognition.  arXiv preprint arXiv:2105.01883, 2021.
  * [16] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding.  Scaling up your kernels to 31x31: Revisiting large kernel design in cnns.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11963–11975, 2022.
  * [17] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo.  Cswin transformer: A general vision transformer backbone with cross-shaped windows.  arXiv preprint arXiv:2107.00652, 2021.
  * [18] Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas.  Attention is not all you need: Pure attention loses rank doubly exponentially with depth.  arXiv preprint arXiv:2103.03404, 2021.
  * [19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations (ICLR), 2021\.
  * [20] Hang Gao, Xizhou Zhu, Steve Lin, and Jifeng Dai.  Deformable kernels: Adapting effective receptive fields for object deformation.  arXiv preprint arXiv:1910.02940, 2019.
  * [21] Meng-Hao Guo, Zheng-Ning Liu, Tai-Jiang Mu, and Shi-Min Hu.  Beyond self-attention: External attention using two linear layers for visual tasks.  arXiv preprint arXiv:2105.02358, 2021.
  * [22] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao.  Ms-celeb-1m: A dataset and benchmark for large-scale face recognition.  In European Conference on Computer Vision (ECCV), pages 87–102, 2016.
  * [23] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al.  A survey on visual transformer.  arXiv preprint arXiv:2012.12556, 2020.
  * [24] Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang.  On the connection between local attention and dynamic depth-wise convolution.  In International Conference on Learning Representations, 2021.
  * [25] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In IEEE International Conference on Computer Vision (ICCV), pages 2961–2969, 2017.
  * [26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Spatial pyramid pooling in deep convolutional networks for visual recognition.  IEEE transactions on pattern analysis and machine intelligence, 37(9):1904–1916, 2015.
  * [27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [28] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [29] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin.  Local relation networks for image recognition.  In IEEE International Conference on Computer Vision (ICCV), pages 3464–3473, 2019.
  * [30] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7132–7141, 2018.
  * [31] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  In Advances in Neural Information Processing Systems (NeurIPS), pages 2017–2025, 2015.
  * [32] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool.  Dynamic filter networks.  In Advances in Neural Information Processing Systems (NeurIPS), pages 667–675, 2016.
  * [33] Ira Kemelmacher-Shlizerman, Steven M Seitz, Daniel Miller, and Evan Brossard.  The megaface benchmark: 1 million faces for recognition at scale.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4873–4882, 2016.
  * [34] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah.  Transformers in vision: A survey.  arXiv preprint arXiv:2101.01169, 2021.
  * [35] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [36] Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang.  Selective kernel networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 510–519, 2019.
  * [37] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2117–2125, 2017.
  * [38] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European Conference on Computer Vision (ECCV), pages 740–755. Springer, 2014.
  * [39] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [40] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976–11986, 2022.
  * [41] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten.  Exploring the limits of weakly supervised pretraining.  In European Conference on Computer Vision (ECCV), pages 181–196, 2018.
  * [42] Sayak Paul and Pin-Yu Chen.  Vision transformers are robust learners.  arXiv preprint arXiv:2105.07581, 2021.
  * [43] Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun.  Large kernel matters–improve semantic segmentation by global convolutional network.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4353–4361, 2017.
  * [44] Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou.  Global filter networks for image classification.  In Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [45] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision (IJCV), pages 211–252, 2015.
  * [46] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In Association for the Advancement of Artificial Intelligence (AAAI), 2017.
  * [47] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [48] Yi Tay, Mostafa Dehghani, Jai Gupta, Dara Bahri, Vamsi Aribandi, Zhen Qin, and Donald Metzler.  Are pre-trained convolutions better than pre-trained transformers?  arXiv preprint arXiv:2105.03322, 2021.
  * [49] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International Conference on Machine Learning (ICML), pages 10347–10357. PMLR, 2021.
  * [50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems (NeurIPS), pages 5998–6008, 2017.
  * [51] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  arXiv preprint arXiv:2102.12122, 2021.
  * [52] Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu.  Crossformer: A versatile vision transformer hinging on cross-scale attention.  In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
  * [53] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7794–7803, 2018.
  * [54] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In European Conference on Computer Vision (ECCV), pages 3–19, 2018\.
  * [55] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang.  Cvt: Introducing convolutions to vision transformers.  arXiv preprint arXiv:2103.15808, 2021.
  * [56] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In European Conference on Computer Vision (ECCV), pages 418–434, 2018.
  * [57] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He.  Feature denoising for improving adversarial robustness.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 501–509, 2019.
  * [58] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1492–1500, 2017.
  * [59] Brandon Yang, Gabriel Bender, Quoc V Le, and Jiquan Ngiam.  Condconv: Conditionally parameterized convolutions for efficient inference.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [60] Fisher Yu and Vladlen Koltun.  Multi-scale context aggregation by dilated convolutions.  arXiv preprint arXiv:1511.07122, 2015.
  * [61] Fisher Yu, Vladlen Koltun, and Thomas Funkhouser.  Dilated residual networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 472–480, 2017.
  * [62] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan.  Tokens-to-token vit: Training vision transformers from scratch on imagenet.  arXiv preprint arXiv:2101.11986, 2021.
  * [63] Yucheng Zhao, Guangting Wang, Chuanxin Tang, Chong Luo, Wenjun Zeng, and Zheng-Jun Zha.  A battle of network structures: An empirical study of cnn, transformer, and mlp.  arXiv preprint arXiv:2108.13002, 2021.
  * [64] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Semantic understanding of scenes through the ade20k dataset.  International Journal of Computer Vision (IJCV), 127(3):302–321, 2019.


[a]Juhi Dutta

# Searching for top squarks from the landscape at HL-LHC

###### Abstract

Supersymmetric models with low electroweak fine-tuning are more prevalent on
the string landscape than fine-tuned models. We assume a fertile patch of
landscape vacua containing the minimal supersymmetric standard model (MSSM) as
a low-energy EFT. Such models are characterized by light higgsinos in the mass
range of a few hundred GeV whilst top squarks are in the 1-2.5 TeV range.
Other sparticles are generally beyond current LHC reach. We evaluate prospects
for top squark searches of the expected natural SUSY at HL-LHC.

##  1 Introduction

Supersymmetry is one of the best-motivated candidates for Beyond Standard
Model (BSM) to address several issues of Nature. While early estimates of
naturalness placed stringent upper limits on masses of supersymmetric
particles, such as $m_{\widetilde{t}_{1}}\leq$ 300-400 GeV for $\Delta_{BG}<$
10-30 where $\Delta_{BG}$ is the Barbieri-Guidice measure of naturalness [1],
defined as

$\Delta_{BG}=max_{i}|\frac{p_{i}\partial m^{2}_{Z}}{m^{2}_{Z}\partial
dp_{i}}|.$

where $p_{i}$ are the fundamental free parameters of the theory and subsequent
measures such as $\Delta_{HS}$ required third generation squarks to be less
than 500 GeV[12, 11], current experimental searches from LHC place rather
stringent limits $\sim\mathcal{O}$(TeV) on the lightest stops. However, the
naturalness measures considered earlier turned out to be large overestimates
of the actual-finetuning[13, 15, 16, 14]. Recently, more conservative
estimates of electroweak fine-tuning $\Delta_{EW}$ [6] have come up to resolve
the naturalness issue where $\Delta_{EW}$ is

$\frac{m^{2}_{Z}}{2}=\frac{m^{2}_{H_{d}}+\Sigma^{d}_{d}-(m^{2}_{H_{u}}+\Sigma^{u}_{u})\tan^{2}\beta}{\tan^{2}\beta-1}-\mu^{2}$
(1)

the ratio of the largest term on the right-hand side of the eq. 1 to
$\frac{m^{2}_{Z}}{2}$, where $m_{Z}$ is the mass of the $Z$ boson,
$m^{2}_{H_{u}}$ and $m^{2}_{H_{d}}$ refer to the Higgs soft breaking masses
coupling to the up-type and down-type quarks respectively,
$\tan\beta=\frac{v_{u}}{v_{d}}$ ($v_{q}$ being the vaccuum expectation value
of $H_{q}$, where $q=(u,d)$), and $\Sigma^{q}_{q}$, refer to the loop
contributions from the particles and sparticles to the Higgs sector (with
dominant one-loop contributions from the lightest top squarks) and $\mu$ is
the higgsino mass parameter. Such a conservative measure of naturalness
imposes relatively relaxed constraints on sparticle masses which are allowed
up to several TeV at little cost to finetuning since their contributions to
the weak scale are suppressed by loop factors.

Another possible resolution arises from the string landscape picture. In the
string landscape, where order of 10500 vacua solutions arise from
compactification from 10 to 4 spacetime dimensions, each vacuum solution
corresponds to a different set of 4-d low energy effective field theory law of
physics. The string landscape provides a natural setting for Weinberg’s
anthropic solution to the cosmological constant problem [7] in an eternally
inflating multiverse. In the same spirit, one tries to address the origin of
the SUSY breaking scale in the string landscape. Supersymmetric models with
low electroweak fine-tuning are expected to be more prevalent on the string
landscape than fine-tuned models [10]. In this work, we determine the
properties of the stops from the landscape and prospects of observing them at
the upcoming HL-LHC.

##  2 MSSM from the string landscape

Assuming a fertile patch of landscape vacua containing the minimal
supersymmetric standard model (MSSM) as low energy effective field theory
[14], the landscape statistically favours large soft terms via a power law
[3],

$f_{SUSY}=m^{2n_{F}+n_{D}-1}_{soft}$

where $n_{F}$ and $n_{D}$ refer to the number of F-term and D-term SUSY
breaking terms and where $f_{\text{SUSY}}$ is the expected statistical
distribution of landscape soft terms. A statistical pull by the landscape to
large soft terms is balanced by the requirement of a derived value of the weak
scale in the pocket universe ($PU$), which is not too far from its measured
value in our universe ($OU$) given by the ABDS window [4]

$\frac{m^{PU2}_{Z}}{2}=\frac{m^{2}_{H_{d}}+\Sigma^{d}_{d}-(m^{2}_{H_{u}}+\Sigma^{u}_{u})\tan^{2}\beta}{\tan^{2}\beta-1}-\mu^{2}_{PU}$

such that $m^{PU}_{weak}\sim(0.5-5)m^{OU}_{weak}$ in order to allow for
complex nuclei (and hence atoms) in our universe. The string landscape
approach to soft SUSY breaking within the MSSM statistically predicts a Higgs
boson with mass 125 GeV and is characterized by light higgsinos in the 100-400
GeV range, lightest top squarks are in the 1-2.5 TeV range with large
trilinear soft terms which helps to push $m_{h}$ $\sim$ 125 GeV and other
squarks beyond the HL-LHC reach.

Figure 1: The probability distributions for the lightest CP-even Higgs mass
$m_{h}$.

To obtain a measure of stringy naturalness, we implement a linear scan[5] over
the NUHM2 parameter space. Fig. 1 shows the probability distribution for the
lightest CP-even Higgs mass, $m_{h}$ from the string landscape with a n=1 draw
to large soft terms. We observe that the distribution is peaked towards
$\sim$125 GeV while $m_{\widetilde{t}_{1}}$ has a large number of events above
the TeV scale within 1-2.5 TeV with a peak $\sim 1.5$ TeV. The reach of LHC
Run 3 and HL-LHC would probe the peak probability region in the coming years,
making the search for light top-squarks of supersymmetry a highly motivated
priority.

Figure 2: The probability distribution of the lightest top squark
$m_{\widetilde{t}_{1}}$ (left) and mixing angle $\cos\theta$ in the stop
sector (right).

Fig. 2 (right) shows the variation of the cosine of mixing angle in the stop
sector vs. the lightest stop mass. It is clear from the plot, $\cos\theta\sim
0.1$ over most of the stop mass range suggesting that the lightest stop is
largely dominated by the right-handed top squark. Thus, the light top-squark
decays comparably via $b\widetilde{\chi}^{\pm}_{1}$ and
$t\widetilde{\chi}^{0}_{i}$ yielding mixed final states of
$b\bar{b}+\not{E}_{T}$ , $t\bar{b}/t\bar{b}+\not{E}_{T}$ and
$t\bar{t}+\not{E}_{T}$.

Figure 3: The variation of the branching ratios of $\widetilde{t}_{1}$ vs. the
stop mass, $m_{\widetilde{t_{1}}}$ into the different decay modes. Figure 4:
Stop pair production cross-section at NLO at $\sqrt{s}=14$ TeV at the HL-LHC.

##  3 Collider study

In this section, we discuss the collider prospects of the lightest stops at
the high luminosity LHC (HL-LHC). We choose a benchmark point shown in Table 1
consistent with current experimental constraints from flavour physics and LHC
data and scan over the parameter $A_{t}$ to vary the mass of the stop in the
range 800-2200 GeV. The production cross-section of the lightest stops is
shown in Fig. 4 for $\sqrt{s}=14$ TeV. For the collider analyses, we consider
the signal final states: $b\bar{b}+\not{E}_{T}$, 1 t + 1 b $+\not{E}_{T}$ and
2 t $+\not{E}_{T}$. The dominant SM backgrounds are
$t\bar{t},b\bar{b}Z,t\bar{t}Z,t\bar{t}W,b\bar{b}W$ and single top production
channel. Using boosted jet techniques to reconstruct top jets with $p_{T}>400$
GeV and $R=1.5$ and relying on hard kinematic observables such as
$\not{E}_{T}>400$ GeV, $H_{T}>1400$ GeV, $L_{T}(=H_{T}+\not{E}_{T})>1400$ GeV,
min($m_{T}(t,\not{E}_{T}),m_{T}(b,\not{E}_{T}))\geq 175$ GeV,
$\Delta\Phi(b,\not{E}_{T})\geq 40^{\circ}$, $\Delta\Phi(J,\not{E}_{T})\geq
30^{\circ}$ to suppress the SM background for the $tb+\not{E}_{T}$, we observe
that the key kinematic variable to discriminate between signal and backgrounds
is $M_{T_{2}}$ especially at the tails of the distribution as shown in Fig. 5
for $\sqrt{s}=14$ TeV and integrated luminosity of $3000$ fb-1. Similar cuts
(see [5]) are imposed on the $bb+\not{E}_{T}$ and $tt+\not{E}_{T}$ channels.
In all cases, the top-squark pair production is revealed as an enhancement in
the $m_{T_{2}}$ distribution at high values of $m_{T_{2}}$. A combined reach
of all channels at HL-LHC lead to a reach of $\sim$1.7 TeV at 5$\sigma$ as
seen in Fig. 5(right) and $\sim$2 TeV at 2$\sigma$ [5].

Parameters | Benchmark point
---|---
$m_{0}$ | 5 TeV
$m_{1/2}$ | 1.2 TeV
$A_{0}$ | -8 TeV
$\tan\beta$ | 10
$\mu$ | 250 GeV
$m_{A}$ | 2 TeV
$m_{\tilde{g}}$ | 2830 GeV
$m_{\tilde{t}_{1}}$ | 1714 GeV
$m_{\tilde{t}_{2}}$ | 3915 GeV
$m_{\tilde{\chi}_{1}^{\pm}}$ | 261.7 GeV
$m_{\tilde{\chi}_{2}^{\pm}}$ | 1020.6 GeV
$m_{\tilde{\chi}_{1}^{0}}$ | 248.1 GeV
$m_{\tilde{\chi}_{2}^{0}}$ | 259.2 GeV
$m_{\tilde{\chi}_{3}^{0}}$ | 541.0 GeV
$m_{\tilde{\chi}_{4}^{0}}$ | 1033.9 GeV
$m_{h}$ | 124.7 GeV
$\Omega_{\tilde{\chi}_{1}}^{std}h^{2}$ | 0.016
$\sigma^{SI}(\tilde{\chi}_{1}^{0},p)$ (pb) | $2.2\times 10^{-9}$
$\sigma^{SD}(\tilde{\chi}_{1}^{0},p)$ (pb) | $2.9\times 10^{-5}$
$\Delta_{\rm EW}$ | 22
Table 1: Input parameters (TeV) and masses (GeV) for the stringy natural SUSY
benchmark point from the NUHM2 model with $m_{t}=173.2$ GeV using Isajet 7.88.

Figure 5: Distribution of $m_{T_{2}}$ for the final state 1$b$+1$t$ \+
$\not{E}_{T}$ and expected 5$\sigma$ discovery plane for the lightest stop at
the HL-LHC.

##  4 Summary and Conclusions

In this work, we have investigated the properties of top squarks from the
string landscape where a power-law draw to large soft terms is expected. The
derived value of the weak scale must lie within the ABDS window in order to
allow for complex nuclei (and hence atoms) in each anthropically-allowed
pocket universe. Under this stringy naturalness requirement, we find
$m_{\widetilde{t}_{1}}\sim$ 1-2.5 TeV with large mixing which also facilitates
to lift $m_{h}$ to 125 GeV while minimizing the top squark contributions to
the weak scale. Despite of the large mixing, the lighter top-squark is mainly
a right-squark, and lead to mixed final states of $b\bar{b}+\not{E}_{T}$,
$t\bar{t}+\not{E}_{T}$ and $tb+\not{E}_{T}$. Using boosted jet techniques to
investigate the reach of stops at HL-LHC it is possible to reach
$m_{\widetilde{t}_{1}}\simeq 1.7$ at 5$\sigma$ and $\simeq 2$ TeV at 2$\sigma$
therefore covering most of the stringy natural parameter space at HL-LHC.

## Acknowledgments

The author acknowledges support from the HEP Dodge Family Endowment Fellowship
at the Homer L.Dodge Department of Physics $\&$ Astronomy at the University of
Oklahoma. The author thanks her collaborators H.Baer, V.Barger, D.Sengupta and
K.Zhang for the successful completion of the work Ref. [5], on which this talk
is based on.

## References

  * [1] R. Barbieri and G. F. Giudice, Nucl. Phys. B 306 (1988), 63-76 doi:10.1016/0550-3213(88)90171-X
  * [2] H. Baer, V. Barger and D. Mickelson, Phys. Rev. D 88 (2013) no.9, 095013 doi:10.1103/PhysRevD.88.095013 [arXiv:1309.2984 [hep-ph]].
  * [3] M. R. Douglas, doi:10.1142/9789814412551_0012 [arXiv:1204.6626 [hep-th]].
  * [4] V. Agrawal, S. M. Barr, J. F. Donoghue and D. Seckel, Phys. Rev. D 57 (1998), 5480-5492 doi:10.1103/PhysRevD.57.5480 [arXiv:hep-ph/9707380 [hep-ph]].
  * [5] H. Baer, V. Barger, J. Dutta, D. Sengupta and K. Zhang, doi:10.1103/PhysRevD.108.075027 [arXiv:2307.08067 [hep-ph]].
  * [6] H. Baer, V. Barger, P. Huang, A. Mustafayev and X. Tata, Phys. Rev. Lett. 109 (2012), 161802 doi:10.1103/PhysRevLett.109.161802 [arXiv:1207.3343 [hep-ph]].
  * [7] S. Weinberg, Phys. Rev. Lett. 59 (1987), 2607 doi:10.1103/PhysRevLett.59.2607
  * [8] H. Baer, V. Barger, D. Martinez and S. Salam, Phys. Rev. D 108 (2023) no.3, 035050 doi:10.1103/PhysRevD.108.035050 [arXiv:2305.16125 [hep-ph]].
  * [9] C. Brust, A. Katz, S. Lawrence and R. Sundrum, JHEP 03 (2012), 103 doi:10.1007/JHEP03(2012)103 [arXiv:1110.6670 [hep-ph]].
  * [10] H. Baer, V. Barger, S. Salam, D. Sengupta and K. Sinha, Eur. Phys. J. ST 229 (2020) no.21, 3085-3141 doi:10.1140/epjst/e2020-000020-x [arXiv:2002.03013 [hep-ph]].
  * [11] C. Brust, A. Katz, S. Lawrence and R. Sundrum, JHEP 03 (2012), 103 doi:10.1007/JHEP03(2012)103 [arXiv:1110.6670 [hep-ph]].
  * [12] R. Kitano and Y. Nomura, Phys. Rev. D 73 (2006), 095004 doi:10.1103/PhysRevD.73.095004 [arXiv:hep-ph/0602096 [hep-ph]].
  * [13] H. Baer, V. Barger and D. Mickelson, Phys. Rev. D 88 (2013) no.9, 095013 doi:10.1103/PhysRevD.88.095013 [arXiv:1309.2984 [hep-ph]].
  * [14] H. Baer, V. Barger, D. Martinez and S. Salam, Phys. Rev. D 108 (2023) no.3, 035050 doi:10.1103/PhysRevD.108.035050 [arXiv:2305.16125 [hep-ph]].
  * [15] A. Mustafayev and X. Tata, Indian J. Phys. 88 (2014), 991-1004 doi:10.1007/s12648-014-0504-8 [arXiv:1404.1386 [hep-ph]].
  * [16] H. Baer, V. Barger, D. Mickelson and M. Padeffke-Kirkland, Phys. Rev. D 89 (2014) no.11, 115019 doi:10.1103/PhysRevD.89.115019 [arXiv:1404.2277 [hep-ph]].