# Kaleido-BERT: Vision-Language Pre-training on Fashion Domain

Mingchen Zhuge1,† Dehong Gao1,† Deng-Ping
Fan${}^{2,\,{\color[rgb]{1,0,1}\textrm{{\char 0\relax}}}}$
Linbo Jin1 Ben Chen1 Haoming Zhou1 Minghui Qiu1 Ling Shao${}^{2~{}}$
1 Alibaba Group 2 Inception Institute of AI (IIAI)
http://dpfan.net/Kaleido-BERT

###### Abstract

We present a new vision-language (VL) pre-training model dubbed Kaleido-BERT
††$\dagger$ Equal; * Corresponding author: Deng-Ping Fan
(dengpfan@gmail.com)., which introduces a novel kaleido strategy for fashion
cross-modality representations from transformers. In contrast to random
masking strategy of recent VL models, we design alignment guided masking to
jointly focus more on image-text semantic relations. To this end, we carry out
five novel tasks, , rotation, jigsaw, camouflage, grey-to-color, and blank-to-
color for self-supervised VL pre-training at patches of different scale.
Kaleido-BERT is conceptually simple and easy to extend to the existing BERT
framework, it attains state-of-the-art results by large margins on four
downstream tasks, including text retrieval (R@1: 4.03% absolute improvement),
image retrieval (R@1: 7.13% abs imv.), category recognition (ACC: 3.28% abs
imv.), and fashion captioning (Bleu4: 1.2 abs imv.). We validate the
efficiency of Kaleido-BERT on a wide range of e-commerical websites,
demonstrating its broader potential in real-world applications.

\begin{overpic}[width=433.62pt]{Framework-v9.png} \end{overpic}

figureVision-Language (VL) pre-training architecture on fashion. We propose a
novel VL pre-training architecture (Kaleido-BERT), which consists of a Kaleido
Patch Generator (KPG), Attention-based Alignment Generator (AAG), and
Alignment Guided Masking (AGM) strategy to learn better VL feature embeddings.
Kaleido-BERT achieves the state-of-the-art on the standard public Fashion-Gen
dataset and deploys to the online system (a).

##  1 Introduction

Transformers [14, 68], first designed for Natural Language Processing (NLP),
have achieved great success in a number of other areas as well [5, 11],
including the vision (, Selfie [66], DETR [6], ViT [34], and PVT [69]) and
vision-language (ViLBERT [45], VL-BERT [60], OSCAR [42]) communities. However,
for VL Pre-Training Model (PTM), current approaches, such as VL-BERT [60] and
UNITER [9] focus on learning text and image representation of a general domain
(, coarse matching). As such, these techniques will benefit for general cross-
modality representation learning.

No. | Pre-Training Model | Year | Pub. | Architecture | Training Set | Core Idea | Domain | Pre-train  | Finetune | Vision Feature | Code
---|---|---|---|---|---|---|---|---|---|---|---
1 |  VisualBERT [40] | 2019 | arXiv | 1-Stream | Coco |  First Image-Text PTM _AFAK_ | Image | ✓ | U | RoI | Torch
2 | CBT [63] | 2019 | arXiv | 2-Stream | Kinetics [30] | Noise contrastive estimation loss | Video | ✓ | U/G/O | Frame | N/A
3 |  VideoBERT [64] | 2019 | ICCV | 1-Stream | SC |  First Video-Text PTM _AFAK_ | Video | ✓ | G/O | Frame | N/A
4 | B2T2 [2] | 2019 | EMNLP | 1-Stream | CC | Explicitly reference RoI to text before inputting PTM | Image | ✓ | U | RoI | Tensorflow
5 |  LXMERT [38] | 2019 | EMNLP | 2-Stream | VG+Coco | Three encoders for ROIs, language, and cross-modality features | Image | ✓ | U | RoI | Torch
6 | ViLBERT [45] | 2019 | NeurlIPS | 2-Stream | CC | Cross-modality co-attention layers | Image | ✓ | U | RoI | Torch
7 |  ImageBERT [55] | 2020 | arXiv | 1-Stream | CC+VG+SC | Pre-training with a large-scale Image-Text dataset | Image | ✓ | U | RoI | N/A
8 | Unicoder-VL [38] | 2020 | AAAI | 1-Stream | CC+SBU | Masked object classification | Image | ✓ | U | RoI | N/A
9 |  VLP [82] | 2020 | AAAI | 1-Stream | CC | Unified en/decoder PTM for VL generation and understanding | Image | ✓ | U/G | RoI | Torch
10 | VL-BERT [61] | 2020 | ICLR | 1-Stream | CC | Visual feature embedding | Image | ✓ | U | RoI | Torch
11 |  VD-BERT [70] | 2020 | EMNLP | 1-Stream |  VisDial [12] | Video-Dialog pre-training | Video | ✓ | O | RoI | Torch
12 | VLN-BERT [48] | 2020 | ECCV | 2-Stream | Matterport3D [7] | Path selection in VL navigation | Image |  | O | RoI | N/A
13 |  HERO [39] | 2020 | EMNLP | 3-Stream | TV+HT100M | Video-subtitle matching & Frame order modeling | Video | ✓ | U | Frame | N/A
14 | XGPT [73] | 2020 | arXiv | 1-Stream | CC+SC | Improve VL generative ability | Image | ✓ | U/G | RoI | N/A
15 |  InterBERT [43] | 2020 | arXiv | 1-Stream | Coco+CC+SBU | Masked group modeling | Image | ✓ | U | RoI | Torch
16 | VILLA [20] | 2020 | NeurlIPS | 2-Stream | Coco+CC+SBU | Adversarial pre-training and finetune | Image | ✓ | U/O | RoI | Torch
17 |  ActBERT [83] | 2020 | CVPR | 1-Stream | HT100M | Global frame and local object regions & Tangled transformer | Video | ✓ | U/O | Frame & RoI | N/A
18 | PREVALENT [24] | 2020 | CVPR | 2-Stream | Matterport3D [7] | Pre-train with image-text-action triplets | Image | ✓ | O | Image | Caffe & C++
19 |  12-IN-1 [46] | 2020 | CVPR | 2-Stream | ES | Multi-task training | Image | ✓ | U | RoI | Torch
20 | Pixel-BERT [27] | 2020 | arXiv | 1-Stream | Coco+VG | Pixel-level VL semantics alignment | Image | ✓ | U | Pixel | N/A
21 |  FashionBERT [21] | 2020 | SIGIR | 1-Stream |  Fashion-Gen [58] | Patches & Adaptive loss | Image | ✓ | U | Patch | Tensorflow
22 | UNITER [9] | 2020 | ECCV | 1-Stream | Coco+VG+CC+SBU | Conditional masking & Word region alignment | Image | ✓ | U | RoI | Torch
23 |  VisDial-BERT [50] | 2020 | ECCV | 2-Stream |  CC+VQA [4] | Adapt ViLBERT for visual dialog | Image |  | O | RoI | Torch
24 | OSCAR [42] | 2020 | ECCV | 1-Stream | ES | Object tags as anchor points | Image | ✓ | U/G | RoI | Torch
25 |  ERNIEL-VIL [78] | 2020 | arXiv | 2-Stream | CC+SBU |  VL PTM with knowledge-enhanced ERNIE [80] | Image | ✓ | U | RoI | Paddle
26 | RVL-BERT [10] | 2020 | arXiv | 1-Stream | VDR [44] | Visual relationship detection with VL-BERT | Image |  | U | RoI | Torch
27 |  UniVL [47] | 2020 | arXiv | 2-Stream | HT100M | Five pre-training objectives & Two pre-training strategies | Video | ✓ | U/G | Frame | N/A
28 | MMFT-BERT [32] | 2020 | EMNLP | 3-Stream | TV | Multi-modal fusion PTM | Image | ✓ | U | RoI | Torch
29 | Kaleido-BERT (OUR) | 2021 | CVPR | 1-Stream |  Fashion-Gen [58] | Kaleido patches & Pre-alignment with masking | Image | ✓ | U/G | Patch & Coordinate | Tensorflow

Table 1: Summary of 28 previous representative cross-modality methods and our
Kaleido-BERT model. Training Set: Coco = MSCOCO Caption [8]. VG = Visual
Genome [35]. CC = Conceptual Caption [59]. SBU = SBU Captions [53]. TV = TVQA
[37]. HT100M = HowTo100M [49]. SC: Self Collection. ES: 12-in-1 and OSCAR
ensemble 12, 5+ datasets, respectively. Finetune: U = Understanding tasks
(classification). G = Generation tasks (image caption). O = Others (action
task).

However, in the various e-commercial situations (, accessories, clothing,
toys), the main goal is to learn the fine-grained representation (short
sleeve, cotton and jersey) rather than only the coarse representation (what,
where) in the general domain. In this case, the current general VL models [9,
60] are sub-optimal for fashion-based tasks [26, 1, 67], and could be
unfavorable when deploying global features based models to attribute-aware
tasks, such as searching for a specific fashion captioning [75] and fashion
catalog/object [15], where it is essential to extract fine-grained features or
similarities [65] from image and text.

In this paper, we propose a novel framework (see fig:Kaleido-BERT) for the
fashion-based tasks. The core idea is to focus on fine-grained representation
learning and to bridge the semantic gaps between text and image. To achieve
this goal, we first introduce an efficient “kaleido” strategy, which extracts
a series of multi-grained image patches for the image-modality. As a result,
our model is named as Kaleido-BERT. This strategy is scalable and largely
alleviates the aforementioned coarse presentation issue by introducing the
patch-variant pre-training scheme. Furthermore, to bridge the semantic gap
between different modalities, attention mechanism is employed to build pre-
alignments between kaleido patches and text tokens. This pre-alignment
information further guides the masking strategy for pre-training. Kaleido-
BERT111https://github.com/mczhuge/Kaleido-BERT/. is forced to explicitly learn
semantic information across modalities. In summary, our contributions are as
follows:

Figure 1: Different utilization of alignment information in VL pre-training
architectures. T = Task. W = Word. V = Visual Token. L = Object Detection
Label. E = Embedding. Trm = Transformer Block.

  * •

Kaleido Patch Generator: We propose the kaleido strategy to generate a kaleido
of multi-grained features. With the related pre-training tasks, rotation,
jigsaw, camouflage, grey-to-color, and blank-to-color, Kaleido-BERT learns
fine-grained cross-modality information and outperforms the fixed-patch or
RoI-based VL models in the fashion domain.

  * •

Attention-based Alignment Generator: Kaleido-BERT introduces the pre-alignment
strategy to infer a cross-modality mapping between kaleido patches and text
tokens. These pre-alignment pairs largely fill the semantic gaps between
modalities.

  * •

Alignment Guided Masking: We present an alignment-guided masking strategy to
explicitly force Kaleido-BERT to learn the semantic connections between vision
and language. Experiments show the importance of the attention-based pre-
alignment and the alignment masking strategy.

##  2 Related Work

There is a large body of VL modeling literature [31, 4, 54, 77, 23, 3, 28, 62,
79], and we briefly introduce the transformer-based methods in this section.
More detailed summary can be found in tab:ModelSummary.

###  2.1 Vision-Language Pre-training

Recent transformer-based pre-training frameworks, such as BERT [14], GPT2
[57], XLNet [76], and GPT3 [5], have revolutionized NLP tasks. Motivated by
these studies, many cross-modal pre-training models for vision-language (,
video/image and text pairs) have been designed. For video-text pair models,
CBT [63] and VideoBERT [64] are pioneering work that study the capability of
pre-training learning. ActBERT [83] and HERO [39] focus more on downstream
applications, while UniVL [47] focuses on both video-language understanding
and generation tasks.

For image-text pair models, they can be categorized into single-stream [40, 2,
38, 82, 61, 70, 73, 43, 27, 41, 9, 21, 42, 10, 55] and two-stream [38, 45, 50,
48, 13, 24, 15, 78] or even three-stream [32] according to the network
architecture of the single-modal input. In single-stream models, the features
of different modalities are directly fed into a Transformer. In contrast, in
two-stream models, they are first processed by two single-modal networks
before fed into a Transformer, and so forth in three-stream models. ViLBERT
[45] claims that the two-stream structure is superior to the single-stream,
while VL-BERT [61] finds that the single-stream models achieve more promising
results, as these models have more cross-modality information interactions.
VisualBERT [40] and B2T2 [2] are single-stream models and derive a unified VL
understanding network. With the same spirit of focusing on generic VL tasks,
many concurrent models, , Unicoder-VL [38], VLP [82], ViLBERT [45], VL-BERT
[61], have been proposed under the BERT framework. In contrast to the boom in
generic tasks (, VCR [9, 78, 43], VQA [32, 38]), other tasks such as visual
relationship detection (RVL-BERT [10]), visual navigation (, PERVALENT [24]
and VLN-BERT [48]), and visual dialog (, VisualD [50], VD-BERT [70]) are still
in their infancy. More recently, Lu et al. [46] shows that multi-task VL
learning can lead to a significant improvement over isolated task learning.
Similarly, OSCAR [42] achieves new state-of-the-art performance on many
representative VL tasks (, image captioning like XGPT [73], image-text
retrieval like Image-BERT [55]).

Advances have also been made by the VirTex [13] model in image classification,
object detection, and instance segmentation filed, by using semantically dense
captions to learn visual representations. Another notable study is the recent
ACL work [41] in which the authors creatively demonstrate that the attention
head of the VL model can perform entity grounding and syntactic grounding.
Unlike all the above-mentioned works, Pixel-BERT [27] considers aligning
vision-language features at a pixel level instead of using region-based image
features.

As shown in fig:Alignment, our Kaleido-BERT focuses on a masking strategy at
the embedding level rather than at the task level (, LXMERT [38] and UNITER
[9]) or input level such as OSCAR [42]. Kaleido-BERT explicitly align the
embedding features between image and text so that it can learn fine-grain
representations for fashion tasks.

###  2.2 Fashion-Based Task

As described in sec:VL pretraining, existing VL models mainly focus on
relatively coarse representations, while less attention has been paid to fine-
grained representation learning for the fashion-based task. There are two
concurrent studies [15, 21] resembling our work. FashionBERT [21] was the
first published work in the fashion domain. The concurrent work, MAAF [15],
aims to derive a modality-agnostic attention fusion strategy to address the
undifferentiated text and image query task. Unlike FashionBERT, which utilizes
a patch-fixed masking strategy, the MAAF adopts an image-level attention
mechanism. We argue that these two schemes restrict the power of the pre-
trained representation learning, especially for the fine-grained fashion task.
As a consequence, a more flexible solution with patch-variant is urgently
required.

To the best of our knowledge, the proposed Kaleido-BERT is the first to
present the effectiveness of alignment guided masking by jointly focusing more
on image-text coherence for the fashion domain.

##  3 Proposed Kaleido-BERT

In this section, we introduce our Kaleido-BERT, which learns the fine-grained
VL features for the fashion domain rather than the coarse representation
features for VL tasks. We adopt the standard transformer designed for NLP to
make our Kaleido-BERT scalable over a varying number of transformer-based VL
learning task.

###  3.1 Model Overview

The architecture of our Kaleido-BERT is illustrated in fig:Kaleido-BERT. There
are five stages: (1) Kaleido-BERT takes two inputs: a text (, image caption or
description) and corresponding image patches generated by our Kaleido Patches
Generator (KPG). Similar to LXMERT [38], each text is represented as a
sequence of tokens and each image is represented as a sequence of kaleido
patches. (2) At the embedding stage, we propose the Attention-based Alignment
Generator (AAG) to generate pre-alignments between text tokens and kaleido
patches so that the image and text are explicitly aligned semantically. (3)
Different from existing random masking strategy, we proposed to adopt an
Alignment Guided Masking (AGM) strategy to relieve the difficulty of cross-
modality modeling. (4) Text tokens and kaleido patches fully interact in
Kaleido-BERT, which gradually learns VL semantic information and produces the
cross-modality fine-grained representations. (5) Finally, we adopt five new
kaleido tasks (, rotation, jigsaw, camouflaged, grey-to-color and blank-to-
color tasks) besides the masked language modeling and image-text matching
tasks to supervise the network. Our implementation is based on the
EasyTransfer 222Tensorflow:
https://github.com/alibaba/EasyTransfer/Huggingface 333 Pytorch:
https://github.com/huggingface/transformers library. We refer the readers to
this de facto standard library for details.

###  3.2 Kaleido Patch Generator

Given an image as input, we obtain the multi-grained patches by the proposed
Kaleido Patch Generator (KPG). As shown in fig:KPG, we can introduce a
saliency detection network444For simplicity, we just utilize a very simple
UNet-like architecture as our foreground segmentation net. (, BAS [56], EGNet
[81], ICON [84] or other SOTAs in the recent paper [17]) to obtain the
foreground mask and then lock (, bounding box proposal) the domain object.
Motivated by the spatial envelope [52] and the block-level strategy [19, 18],
we then split the image into different scales (, 1$\times$1, 2$\times$2,
$\dots$, 5$\times$5). These image patches are just like “kaleido” patches, and
more detailed divisions (, 6$\times$6 or N$\times$N like Pixel-BERT [27]) can
be considered according to the difficulty of the specific task. Finally, we
obtain 55 kaleido patches from each input image. To generate the embeddings of
these patches, we utilize the standard ResNet-50 [25] as our backbone.

Figure 2: Illustration of KPG. See sec:KPG for details. Figure 3: Procedures
of AAG. See sec:AAG for details.

###  3.3 Attention-based Alignment Generator

Attention-based Alignment Generator (AAG) aims to find the coarse alignments
between text tokens and kaleido patches. As shown in fig:AAG, we directly
adopt the famous SAT network [74] as our text generator, which automatically
learns to describe the content of images. At the same time, the SAT network
generates the attention heat-map for each token, from which we infer the
relation between generated tokens and image regions. With the co-occurrence of
the generated tokens and the raw text tokens, as well as the overlap area of
image regions and kaleido patches, we further build the alignments between raw
text tokens and kaleido patches.

Figure 4: Aligned Kaleido Patch Modeling (AKPM). (I) Rotation recognition.
(II) Jigsaw puzzles solving. (III) Camouflaged prediction. (IV) Grey-to-color
modeling. (V) Blank-to-color modeling. Zoomed-in for a better view. See
sec:Pre-trainKBERT for details.

###  3.4 Alignment Guided Masking

The main idea that inspires us to modify the vanilla random masking strategy
is that the pre-aligned $\langle$token, patch$\rangle$ pair provides explicit
semantic relations between two modalities. This alignment can be used in the
pre-training stage, which further forces Kaleido-BERT to explicitly explore
cross-modality semantic information. As shown in fig:Alignment (Left), unlike
the random masking strategy, Alignment Guided Masking (AGM) gives high
priority to masking the pre-alignment pairs. Meanwhile, for each selected pre-
aligned $\langle$token, patch$\rangle$ pair, we randomly mask either the token
part or the patch part, which stimulates Kaleido-BERT to learn the missing
information in one modality by providing the information of the other. When
all pre-alignment pairs are traversed and not-enough tokens or patches are
selected, a random masking strategy is adopted to mask the unaligned tokens
and patches independently. In this way, we obtain the token and patch masking
candidates. AGM strategy works on level-3, level-4, level-5 of kaleido
patches. We do not apply this strategy on level-1 & -2 since masking larger
patches will increase the difficulty of modeling. Empirically, we mask one
patch in level-3, two patches in level-4, and three patches in level-5.

###  3.5 Cross-Modality Transformer

We adopt the original BERT [14] as our cross-modality transformer so that our
Kaleido-BERT can be easily extended. Specifically, for the text side, we
follow FashionBERT [21] to encode the order of the token (, generated via
WordPieces [72]) position as $0,1,2,3,\dots,N$. Our final training corpus for
each sub-word token is obtained by summing up its embedding with the segment
and position embeddings, followed by another layer normalization (LN) layer.
For the image side, we encode the position information by re-organizing it as
5D features ($[x_{1},x_{2},y_{1},y_{2},w*h]$) for each patch. After that, both
patches and location features are fed into a fully-connected (FC) layer in
order to project them into the same embedding space. We obtain visual
embeddings for each patch by summing up three FC outputs (, FC (seg_id), FC
(img_feature), FC (pos_emb))555Similar to ‘segment embeddings’ in BERT, we
conduct a special modality embedding (‘T’ for text, ‘I’ for image) to help the
model distinguish the different modalities. and then passing them through an
LN layer.

###  3.6 Pre-training Kaleido-BERT

To alleviate the VL semantic gap and boost feature representation, we design
three pre-training tasks, , Aligned Masked Language Modeling (AMLM), Image and
Text Matching (ITM) and the proposed Aligned Kaleido Patch Modeling (AKPM)
(which includes five kaleido sub-tasks) to supervise our Kaleido-BERT.

Task #1: AMLM.  Derived from our alignment guided masking strategy, we can
obtain the mask candidates including both token and patch candidates. When
masking indices are determined, we decompose masking words into 10% random
words, 10% unchanged, and 80% [MSK]. The masked-out token sequence is denoted
by $T_{i}=\left\\{t_{1},...[MSK],...,t_{T}\right\\}$, where token $t_{i}$ is
masked out. We feed the hidden output of the last layer of the masked-out
token into a classifier over the standard BERT vocabularies. The AMLM goal is
to predict the masked words based on the observation of their surrounding
tokens and image patches. The objective of the AMLM task is mathematically
written as:

$\small\mathcal{L}_{AMLM}=\sum
CE(t_{i},\mathcal{F}(T,K,\theta)_{MSK\\_hidden}),$ (1)

where $CE$ denotes the cross-entropy loss. $\mathcal{F}$ is the Kaleido-BERT
function. $\mathcal{F}(\cdot)_{MSK\\_hidden}$ denotes the hidden output of
masked-out tokens. $K$ denotes the masked-out kaleido patch sequence.

Task #2: ITM.  The ITM task is transferred by Next Sentence Prediction (NSP)
on the vanilla BERT. In this task, [CLS] is used to indicate the beginning of
the fused representation. The hidden output of [CLS] is fed into an FC layer
and we use the sigmoid function to predict a score between 0 and 1. The text
and image of one positive example are extracted from the same fashion product,
while those of one negative sample are randomly extracted from different
fashion products. The objective of the ITM task is written as:

$\small\mathcal{L}_{ITM}=CE(y_{m},\mathcal{F}(T,K,\theta)_{CLS\\_hidden}),$
(2)

where $y_{m}$ denotes the text and image match label.

Task #3: AKPM.  wei

Sub-Task #i@: Rotation Recognition (RR).  Recent works [22, 29] have compared
various self-supervised learning strategies concluding that predicting image
rotations is among the most effective. Motivated by this, we introduce RR in
our pre-training. Specifically, we force the 1$\times$1 patch of the level-1
kaleido to randomly rotate by an angle
$\theta\in\left\\{0^{\circ},90^{\circ},180^{\circ},270^{\circ}\right\\}$.
During the training process, we use the angle of the rotated patch as the
target label. The hidden output of the $K_{1}$ patch is fed into an FC layer
followed by softmax function. The final softmax output is used to predict the
angle. The objective of the RR task is written as:

$\small\mathcal{L}_{RR}=CE(y_{r},\mathcal{F}(T,K,\theta)_{K_{1}\\_hidden}),$
(3)

where $y_{r}$ denotes the rotation angle.

Sub-Task #ii@: Jigsaw Puzzle Solving (JPS).  JPS [51, 29] has been
demonstrated to be suitable for self-supervised representation learning. Such
a pretext task (also called surrogate task) can mine the spatial relations
among image patches. Based on this insight, we borrow the notion of jigsaw
puzzle to stimulate Kaleido-BERT to learn the potential association from
unordered 2$\times$2 patch lists. For simplicity, we treat the JPS problem as
a classification of the jigsaw permutations ($4!=24$ classes). The network
architecture is similar to RR. The objective of the JPS task is written as:

$\small\mathcal{L}_{JPS}=CE(y_{j},\mathcal{F}(T,K,\theta)_{K_{2}\\_hidden}),$
(4)

where $y_{j}$ denotes the jigsaw permutation.

Sub-Task #iii@: Camouflage Prediction (CP).  To increase the discernment
ability of the model, we introduce another camouflage prediction task to judge
which patch has been replaced666The camouflaged patch is the same scale as the
patch randomly selected selecting from the other product randomly.. With the
help of image and text clues, this task encourages the training process to
observe the diversity among 3$\times$3 patches. We name this task Camouflage
Prediction (CP) because its essence is to camouflage one patch then let the
model detect it. By pre-training our Kaleido-BERT with CP, the framework
achieves a strong capacity to screen out the imparity with varied products.
The CP prediction is also treated as a classification problem and its
objective is denoted by:

$\mathcal{L}_{CP}=CE(y_{c},\mathcal{F}(T,K,\theta)_{K_{3}\\_hidden}),$ (5)

where $y_{c}$ denotes the index of a camouflaged patch.

Sub-Task #iv@: Grey-to-Color Modeling (G2CM).  Different from the masking
strategy in existing models, which simply exchanges image embeddings with zero
paddings, we introduce a smoother G2CM strategy that greys the image patches.
Then we reconstruct the grey patch to a color patch by regression, supervised
by KL-divergence, which better caters to self-supervised learning. The
objective of G2CM is to minimize the G2CM loss:

$\mathcal{L}_{G2CM}=\sum
KLD(k_{4i},\mathcal{F}(T,K,\theta)_{K_{4}\\_hidden}),$ (6)

where $KLD$ denotes the KL-divergence, which aims to minimize the distance of
the reconstructed distribution to the target distribution and $k_{4i}$ is the
masked-out patch(es) of $K_{4}$ kaleido patches.

Sub-Task #v@: Blank-to-Color Modeling (B2CM).  The last sub-task is B2CM.
Similar to other pre-training methods that replace image feature embeddings
with the same-dimension zeros sequence, we also adopt this kind of patch
masking scheme. This strongly tests the learning ability of a model that
captures the contextual information. The objective of B2CM is to minimize the
B2CM loss:

$\mathcal{L}_{B2CM}=\sum
KLD(k_{5i},\mathcal{F}(T,K,\theta)_{K_{5}\\_hidden}),$ (7)

where $k_{5i}$ is the masked-out patch of $K_{5}$.

All in all, we introduce the aligned kaleido patch modeling to enhance the
ability of the model for spatial context structure (, RR and JPS),
classification (, CP), and image generation (, G2CM and B2CM). Finally,
Kaleido-BERT should minimize the overall loss function as:

$\begin{split}&\mathcal{L}_{total}=\mathcal{L}_{AMLM}+\mathcal{L}_{ITM}+\mathcal{L}_{RR}+\mathcal{L}_{JSP}\\\
&+\mathcal{L}_{CP}+\mathcal{L}_{G2CM}+\mathcal{L}_{B2CM}.\end{split}$ (8)

The evaluations of different Kaleido tasks on the validation set are shown in
Fig. 5. As can be seen, the losses decay smoothly, which proves that the pre-
training process carries on as normal, and the designed tasks can be learned
well with Kaleido-BERT.

\begin{overpic}[width=203.80193pt]{./IMG/loss-min.pdf}
\put(23.0,33.0){\normalsize{①}} \put(73.0,33.0){\normalsize{②}}
\put(23.0,20.0){\normalsize{③}} \put(73.0,20.0){\normalsize{④}}
\put(23.0,7.0){\normalsize{⑤}} \put(73.0,7.0){\normalsize{⑥}} \end{overpic}
Figure 5: Evolution of each training loss. The scores are computed from the
validation sets. ①: Rotation loss. ②: Jigsaw loss. ③: Camouflage loss. ④:
Grey-to-Color loss. ⑤: Blank-to-Color loss. ⑥: Total Loss = AKPM + ITM + AMLM.
This shows that Kaleido-BERT can learn from the kaleido strategy.

##  4 Experiments

We evaluate our Kaleido-BERT on four VL tasks by transferring the pre-trained
model to each target task and fine-tuning through end-to-end training.

Table 2: Retrieval performances on Fashion-Gen dataset. Here,
$\text{Sum}\mathcal{R}$=$($Rank@1+Rank@5+Rank@10$)$*100. See
sec:CompetingModel for details.

|  VSE |  VSE++ |  SCAN |  PFAN |  ViLBERT |  VLBERT | FashionBERT | ImageBERT |  OSCAR | Kaleido-BERT
---|---|---|---|---|---|---|---|---|---|---
Tasks |  [33] |  [16] |  [36] |  [71] |  [60] |  [45] |  [21] |  [55] |  [42] | Ours
| Rank@1 | $\uparrow$ | 4.010% | 4.590% | 4.590% | 4.290% | 20.97% | 19.26% | 23.96% | 22.76% | 23.39% |  27.99%(+4.030%)
| Rank@5 | $\uparrow$ | 11.03% | 14.99% | 16.50% | 14.90% | 40.49% | 39.90% | 46.31% | 41.89% | 44.67% |  60.09%(+13.78%)
1.ITR | Rank@10 | $\uparrow$ | 22.14% | 24.10% | 26.60% | 24.20% | 48.21% | 46.05% | 52.12% | 50.77% | 52.55% |  68.37%(+15.82%)
| Rank@1 | $\uparrow$ | 4.350% | 4.600% | 4.300% | 6.200% | 21.12% | 22.63% | 26.75% | 24.78% | 25.10% |  33.88%(+7.130%)
| Rank@5 | $\uparrow$ | 12.76% | 16.89% | 13.00% | 20.79% | 37.23% | 36.48% | 46.48% | 45.20% | 49.14% |  60.60%(+11.46%)
2.TIR | Rank@10 | $\uparrow$ | 20.91% | 28.99% | 22.30% | 31.52% | 50.11% | 48.52% | 55.74% | 55.90% | 56.68% |  68.59%(+11.91%)
| $\text{Sum}\mathcal{R}$ | $\uparrow$ | 75.20 | 94.16 | 87.29 | 101.90 | 218.13 | 212.84 | 251.36 | 241.30 | 251.53 |  319.52

###  4.1 Pre-training Settings

Dataset. For a fair comparison, we follow the same settings as the Top-1
FashionBERT [21] and pre-train the proposed Kaleido-BERT on the Fashion-
Gen777https://fashion-gen.com/ dataset. It contains 67,666 fashion products
accompanied with text descriptions. Each product includes one to six images
from different angles. Among all the image-text pairs, like [21], we use
260,480 for training, and 35,528 for testing.

Implementation Details. Our Kaleido-BERT has: L=12, H=768, A=12. L is number
of stacked Transformer blocks. H denotes the hidden activation, and A means
the number of attention heads. We implement our model with Tensorflow and use
8*Tesla V100 for pre-training. The Adam optimizer is applied with a learning
rate of $2e-5$ and weight decay $1e-4$. We adopt a warming-up strategy for the
first 5K steps.

###  4.2 Downstream Tasks

We evaluate our model for four downstream VL tasks, including Image-Text
Retrieval, Text-Image Retrieval, Category Recognition, and Fashion Captioning.
The four tasks strongly cater to industrial applications in the fashion field.

1\. Image-Text Retrieval (ITR).  Text retrieval is a downstream task that
requires the model to distinguish whether a sentence can effectively describe
an image. We sample the product images and titles as image-sentences pairs
provided by the Fashion-Gen [58] and consider the original product information
as positive samples. At the same time, we shuffle the dataset and consider the
unmatched image-sentence pairs as negative samples. To increase the
difficulty, the positive and negative pairs are selected from the same sub-
category, which is hard for PTM to differentiate. We use Rank@1, Rank@5,
Rank@10 to evaluate the retrieval performance.

2\. Text-Image Retrieval (TIR).  The image retrieval task aims to rank product
images according to their title. Similar to text retrieval, we use the ground-
truth image in the pair as the positive sample and randomly sample 100
unrelated captions from other products in the same sub-category. By predicting
the matching score, Rank@1, @5, @10 are used as metrics.

3\. Category/SubCategory Recognition (CR & SUB). The category is a vital
attribute for describing a product, and is especially useful in many real-life
applications. We consider a classification task that judges the category and
subcategory of a product, such as {HOODIES, SWEATERS}, {TROUSERS, PANTS}. We
directly use a FC layer after [CLS] for these tasks.

4\. Fashion Captioning (FC).  Image captioning has emerged as an important
research topic with a rich literature in computer vision, and the accuracy on
FC can evaluate the generation ability of cross-modality models.

###  4.3 Competing Models

Detailed comparisons for each downstream task are shown in tab:exp_i2t_t2i &
tab:exp_catpred_generation. ($i$) Our Kaleido-BERT achieves significant
improvement on nearly all evaluations, which demonstrates its excellent
understanding and generation ability in fashion domain. ($ii$) We observe that
the FashionBERT approach outperforms ViLBERT and VLBERT. The main difference
between them is that FashionBERT adopts patches as image features, while
ViLBERT and VLBERT extract RoIs as features. This indicates that in the
fashion domain, the patch method is better for extracting image features.
($iii$) ImageBERT and Oscar perform better than VLBERT and ViLBERT by adding
RoI object classification and RoI tags. These two methods provide more
information about the image modality. This, to a certain degree, hints that
more image semantic information (image features, image supervision tasks) is
required to guide model learning. In our Kaleido-BERT, the kaleido strategy
extends from the patch method of FashionBERT [21]. The attention-based
alignment masking and the kaleido pre-training task provide more semantic
information from the image modality. These factors together explain the
superiority of Kaleido-BERT in VL understanding and generation in the fashion
domain.

Table 3:  Category Recognition and Fashion Captioning performances on Fashion-
Gen dataset. Here, $\text{Sum~{}}\mathcal{C}\mathcal{L}\mathcal{S}$=
$($ACC+macro-$\mathcal{F}$$)$*100 and
$\text{Sum~{}}\mathcal{C}\mathcal{A}\mathcal{P}$=Bleu-4+METEOR+ROUGE-L+CIDEr.
See sec:CompetingModel for details.

| FashionBERT | ImageBERT |  OSCAR | Kaleido-BERT
---|---|---|---|---
Tasks |  [21] |  [55] |  [42] |  Ours
| ACC | $\uparrow$ | 91.25% | 90.77% | 91.79% |  95.07% (+3.28%)
3.CR | macro-$\mathcal{F}$ | $\uparrow$ | 0.705 | 0.699 | 0.727 |  0.714 (-0.013)
| ACC | $\uparrow$ | 85.27% | 80.11% | 84.23% |  88.07% (+2.80%)
3.SUB | macro-$\mathcal{F}$ | $\uparrow$ | 0.620 | 0.575 | 0.591 |  0.636 (+0.016)
| $\text{Sum}~{}\mathcal{C}\mathcal{L}\mathcal{S}$ | $\uparrow$ | 309.02 | 298.28 | 307.82 |  318.14
| Bleu-4 | $\uparrow$ | 3.30 | - | 4.50 |  5.70 (+1.2)
| METEOR | $\uparrow$ | 9.80 | - | 10.9 |  12.8 (+1.9)
4.FC | ROUGE-L | $\uparrow$ | 29.7 | - | 30.1 |  32.9 (+2.8)
| CIDEr | $\uparrow$ | 30.1 | - | 30.7 |  32.6 (+1.9)
| $\text{Sum}~{}\mathcal{C}\mathcal{A}\mathcal{P}$ | $\uparrow$ | 72.9 | - | 76.2 |  84.0

Table 4: Ablation studies of 3 vital pre-training factors. See sec:abaltion
for details.

Metrics | KPG | AGM | AKPM
---|---|---|---
Scale-fixed | Kaleido. | Kaleido.+SOD | Random | AGM | B | B+I |  B+I$\sim$II |  B+I$\sim$III |  B+I$\sim$IV |  B+I$\sim$V | B+V
1\. Rank@1 | $\uparrow$ | 24.71 | 26.73(+8.2%) | 27.99(+13.3%) | 26.55 | 27.99(+5.4%) | 25.37 | 25.07(-1.2%) | 26.03(+2.6%) | 26.88(+6.0%) | 26.20(+3.3%) | 27.99(+10.3%) | 24.62(-2.9%)
1\. Rank@5 | $\uparrow$ | 50.05 | 54.55(+9.0%) | 60.09(+20.1%) | 55.13 | 60.09(+8.9%) | 54.97 | 55.14(+0.3%) | 56.31(+2.4%) | 58.34(+6.1%) | 59.13(+7.6%) | 60.09(+9.3%) | 53.78(-2.2%)
1\. Rank@10 | $\uparrow$ | 58.93 | 65.44(+11.0%) | 68.37(+16.0%) | 64.92 | 68.37(+5.3%) | 62.13 | 62.90(+1.2%) | 63.37(+2.0%) | 67.79(+9.1%) | 67.99(+9.4%) | 68.37(+10.0%) | 60.88(-2.0)%
2\. Rank@1 | $\uparrow$ | 30.17 | 32.19(+6.7%) | 33.88(+12.0%) | 32.14 | 33.88(+5.4%) | 31.09 | 30.98(-0.4%) | 32.22(+3.6%) | 33.17(+6.7%) | 33.80(+8.7%) | 33.88(+9.0%) | 30.77(-1.0%)
2\. Rank@5 | $\uparrow$ | 52.29 | 58.40(+11.7%) | 60.60(+15.9%) | 56.99 | 60.60(+6.3%) | 57.35 | 57.44(+0.2%) | 58.73(+2.4%) | 58.55(+2.1%) | 60.57(+5.6%) | 60.60(+5.7%) | 55.95(-2.4%)
2\. Rank@10 | $\uparrow$ | 60.82 | 66.49(+9.3%) | 68.59(+12.8%) | 63.77 | 68.59(+7.6%) | 64.79 | 65.65(+1.3%) | 64.16(-1.0%) | 67.92(+4.8%) | 68.41(+5.6%) | 68.09(+5.1%) | 61.70(-4.8%)
$\text{Sum}~{}\mathcal{R}$ | $\uparrow$ | 276.97 | 303.80(+9.7%) | 319.52(+16.2%) | 299.50 | 319.52(+6.7%) | 295.70 | 297.18(+0.5%) | 300.82(+1.7%) | 312.65(+5.7%) | 316.10(+6.9%) | 319.02(+7.9%) | 287.70(-2.7%)
3\. ACC | $\uparrow$ | 93.44% | 93.45%(+0.0%) | 95.07%(+1.7%) | 92.71% | 95.07%(+2.5%) | 90.94% | 90.82%(-0.1%) | 91.40%(+0.5%) | 93.91%(+3.3%) | 94.05%(+3.4%) | 95.07%(+4.5%) | 88.87(-2.3%)
3\. macro-$\mathcal{F}$ | $\uparrow$ | 0.701 | 0.705(+0.6%) | 0.714(+1.9%) | 0.711 | 0.714(+0.4%) | 0.690 | 0.692(+0.3%) | 0.721(+4.5%) | 0.713(+3.3%) | 0.710(+2.9%) | 0.714(+3.5%) | 0.701(+1.4%)
4\. ACC | $\uparrow$ | 86.89% | 87.61%(+0.8%) | 88.07%(+1.4%) | 87.20% | 88.07(+1.0%) | 81.66% | 81.25%(-0.5%) | 84.44%(+3.4%) | 86.49%(+5.9%) | 88.53%(+8.4%) | 88.07%(+7.9%) | 81.64(+0.0%)
4\. macro-$\mathcal{F}$ | $\uparrow$ | 0.630 | 0.634(+0.6%) | 0.636(+1.0%) | 0.633 | 0.636(+0.5%) | 0.558 | 0.575(+3.0%) | 0.596(+6.8%) | 0.636(+14.0%) | 0.633(+13.4%) | 0.636(+14.0%) | 0.596(+8.4%)
$\text{Sum}~{}\mathcal{C}\mathcal{L}\mathcal{S}$ | $\uparrow$ | 313.43 | 314.96(+0.5%) | 318.14(+1.5%) | 314.31 | 318.14(+1.2%) | 297.40 | 298.77(+0.4%) | 307.54(+3.4%) | 315.30(+6.0%) | 316.88(+6.5%) | 318.14(+7.0%) | 300.21(+0.9%)
5\. Bleu-4 | $\uparrow$ | 4.9 | 5.2(+6.1%) | 5.7(+16.3%) | 5.3 | 5.7(+7.5%) | 4.9 | 5.2(+6.1%) | 5.2(+6.1%) | 5.1(+4.1%) | 5.6(+14.3%) | 5.7(+16.3%) | 5.3(+8.2%)
5\. METEOR | $\uparrow$ | 11.0 | 11.7(+6.4%) | 12.8(+16.4%) | 11.3 | 12.8(+13.3%) | 11.6 | 11.6(+0.0%) | 11.8(+1.7%) | 12.6(+8.6%) | 12.8(+10.3%) | 12.8(+10.3%) | 11.4(-1.7%)
5\. ROUGE-L | $\uparrow$ | 29.8 | 31.5(+5.7%) | 32.9(+10.4%) | 30.3 | 32.9(+8.6%) | 30.4 | 30.7(+1.0%) | 30.8(+1.3%) | 31.9(+4.9%) | 32.7(+7.6%) | 32.9(+8.2%) | 30.6(+0.7%)
5\. CIDEr | $\uparrow$ | 30.9 | 31.3(+1.3%) | 32.6(+5.5%) | 31.7 | 32.6(+2.8%) | 31.0 | 31.5(+1.6%) | 31.4(+1.3%) | 32.0(+3.2%) | 32.3(+4.2%) | 32.6(+5.2%) | 31.3(+1.0%)
$\text{Sum}~{}\mathcal{C}\mathcal{A}\mathcal{P}$ | $\uparrow$ | 76.6 | 79.7(+4.0%) | 84.0(+9.7%) | 78.6 | 84.0(+6.9%) | 77.9 | 79.0(+1.4%) | 79.2(+1.7%) | 81.6(+4.7%) | 83.4(+7.1%) | 84.0(+7.8%) | 78.6(+0.9%)

Figure 6: Single task analysis on our Kaleido-BERT. ACC = Accuracy, KLD =
Kullback-Leibler Divergence. Refer to sec:abaltion for details.

###  4.4 Ablation Study

Three main factors may influence the performance of Kaleido-BERT, including
Input-level: Kaleido Patch Generator (KPG); Embedding-level: Alignment Guided
Masking (AGM); and Task-level: Aligned Kaleido Patch Modeling (AKPM). We thus
perform three main ablation studies to further analyze these
components/strategies of our model. The results are shown in Tab. 4 and Fig.
6.

KPG.  Three attempts we have tried to generate our kaleido patches. Scheme-1:
Similar to [21, 34], the first attempt is to split the fashion images with a
fixed-scale setting. Training with such patches, we obtain 276.97 Sum
$\mathcal{R}$, 313.43 Sum $\mathcal{C}\mathcal{L}\mathcal{S}$ and 76.6 Sum
$\mathcal{C}\mathcal{A}\mathcal{P}$ scores. Scheme-2: We carry out a kaleido
(patch-invariant) scheme to generate patches and achieves +9.7%, +0.5% and
+4.0% relative improvement on each metric. Compared with scheme-1, scheme-2 is
capable of capturing fine-grained representation better. Scheme-3: We further
introduce the salient object detection (SOD) algorithm [81] to avoid a huge
number of patches with blank regions (tabula rasa). We observed 16.2%, +1.5%
and +9.7% relative improvement compared with Scheme-1.

AGM.  The majority of existing masking methods independently mask visual and
text features with a pre-set probability. Such kind masking methods are
usually named as random masking. In this experiment, we compare AGM to random
masking (Random). Not surprisingly, AGM obtains +6.7%, +1.2%, 6.9%
improvements. Compared to random masking, AGM generates more semantic related
masking, which benefits our Kaleido-BERT to better understand multi-modality
information.

AKPM.  To verify the efficiency of the proposed AKPM, we conduct 7 ablation
studies (see fig:curve). The baseline (B) merely consists of the basic ITM and
AMLM. Then we add five sub-task to pre-train the model step by step. For
example, “B + I $\sim$ IV” equals to “B + I + II + III + IV”. Note that
existing models [21] usually use the combination of “ITM + AMLM + B2CM” (B +
V) as the pre-training supervision. As shown in tab:abalation, the improvement
is limited (+0.9%) in terms of
$\text{Sum}~{}\mathcal{C}\mathcal{L}\mathcal{S}$ score, even cause negative
effect (-2.7%) in $\text{Sum}\mathcal{R}$ metric. Interestingly, we naively
replace V (B2CM) with I (RR) will obtains improvement on all downstream tasks
(+0.5%, +0.4% and +1.4%, respectively). Gradually, we observed that the
performance continue improve when we adding the corresponding sub-tasks
sequentially. Meanwhile the negative affect of V has been alleviated, we argue
that V plays the true value when Kaleido-BERT has learned the comprehensive
representations of image embeddings from I$\sim$IV.

##  5 Conclusion

We presented a novel pre-trained vision-language understanding architecture
Kaleido-BERT for fashion-based task. It consists of a kaleido patches
generator, attention-based alignment generator, and alignment guided masking
strategy. These components are easy to implement and cooperate closely to
learn both intra-modal and inter-modal image-text feature embeddings. The
designed Kaleido-BERT is much more efficient than existing models, attains the
new SOTA performance, and largely boosts the accuracy of many downstream tasks
such as Image-Text Retrieval, Category Recognition, and Fashion Captioning.

## References

  * [1] Ziad Al-Halah and Kristen Grauman.  From paris to berlin: Discovering fashion style influences around the world.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10136–10145, 2020.
  * [2] Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter.  Fusion of detected objects in text for visual question answering.  arXiv preprint arXiv:1908.05054, 2019.
  * [3] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In CVPR, pages 6077–6086, 2018.
  * [4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In ICCV, pages 2425–2433, 2015.
  * [5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  arXiv preprint arXiv:2005.14165, 2020.
  * [6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV. Springer, 2020.
  * [7] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  arXiv preprint arXiv:1709.06158, 2017.
  * [8] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325, 2015.
  * [9] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In ECCV, pages 104–120. Springer, 2020.
  * [10] Meng-Jiun Chiou, Roger Zimmermann, and Jiashi Feng.  Rvl-bert: Visual relationship detection with visual-linguistic knowledge from pre-trained representations.  arXiv preprint arXiv:2009.04965, 2020.
  * [11] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Weller Adrian.  Rethinking attention with performers.  arXiv preprint arXiv:2009.14794, 2020.
  * [12] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra.  Visual dialog.  In CVPR, pages 326–335, 2017.
  * [13] Karan Desai and Justin Johnson.  Virtex: Learning visual representations from textual annotations.  arXiv preprint arXiv:2006.06666, 2020.
  * [14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL, 2019.
  * [15] Eric Dodds, Jack Culpepper, Simao Herdade, Yang Zhang, and Kofi Boakye.  Modality-agnostic attention fusion for visual search with text feedback.  arXiv preprint arXiv:2007.00145, 2020.
  * [16] Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler.  Vse++: Improving visual-semantic embeddings with hard negatives.  arXiv preprint arXiv:1707.05612, 2017.
  * [17] Deng-Ping Fan, Ming-Ming Cheng, Jiang-Jiang Liu, Shang-Hua Gao, Qibin Hou, and Ali Borji.  Salient objects in clutter: Bringing salient object detection to the foreground.  In ECCV, pages 186–202, 2018.
  * [18] Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji.  Structure-measure: A new way to evaluate foreground maps.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 4548–4557, 2017\.
  * [19] Deng-Ping Fan, ShengChuan Zhang, Yu-Huan Wu, Yun Liu, Ming-Ming Cheng, Bo Ren, Paul L Rosin, and Rongrong Ji.  Scoot: A perceptual metric for facial sketches.  In ICCV, pages 5612–5622, 2019.
  * [20] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu.  Large-scale adversarial training for vision-and-language representation learning.  In NIPS, 2020.
  * [21] Dehong Gao, Linbo Jin, Ben Chen, Minghui Qiu, Peng Li, Yi Wei, Yi Hu, and Hao Wang.  Fashionbert: Text and image matching with adaptive loss for cross-modal retrieval.  In SIGIR, pages 2251–2260, 2020.
  * [22] Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra.  Scaling and benchmarking self-supervised visual representation learning.  In ICCV, pages 6391–6400, 2019.
  * [23] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In CVPR, pages 6904–6913, 2017.
  * [24] Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao.  Towards learning a generic agent for vision-and-language navigation via pre-training.  In CVPR, pages 13137–13146, 2020.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [26] Wei-Lin Hsiao, Isay Katsman, Chao-Yuan Wu, Devi Parikh, and Kristen Grauman.  Fashion++: Minimal edits for outfit improvement.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5047–5056, 2019.
  * [27] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu.  Pixel-bert: Aligning image pixels with text by deep multi-modal transformers.  arXiv preprint arXiv:2004.00849, 2020.
  * [28] Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh.  Pythia v0. 1: the winning entry to the vqa challenge 2018.  arXiv preprint arXiv:1807.09956, 2018.
  * [29] Longlong Jing and Yingli Tian.  Self-supervised visual feature learning with deep neural networks: A survey.  PAMI, 2020.
  * [30] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950, 2017.
  * [31] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg.  Referitgame: Referring to objects in photographs of natural scenes.  In EMNLP, pages 787–798, 2014.
  * [32] Aisha Urooj Khan, Amir Mazaheri, Niels da Vitoria Lobo, and Mubarak Shah.  Mmft-bert: Multimodal fusion transformer with bert encodings for visual question answering.  arXiv preprint arXiv:2010.14095, 2020.
  * [33] Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel.  Unifying visual-semantic embeddings with multimodal neural language models.  arXiv preprint arXiv:1411.2539, 2014.
  * [34] Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16-16 words: Transformers for image recognition at scale.  In arXiv preprint arXiv:2010.11929, 2020.
  * [35] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  IJCV, 123(1):32–73, 2017.
  * [36] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He.  Stacked cross attention for image-text matching.  In ECCV, pages 201–216, 2018.
  * [37] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg.  Tvqa: Localized, compositional video question answering.  arXiv preprint arXiv:1809.01696, 2018.
  * [38] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In AAAI, pages 11336–11344, 2020.
  * [39] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu.  Hero: Hierarchical encoder for video+ language omni-representation pre-training.  arXiv preprint arXiv:2005.00200, 2020.
  * [40] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [41] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  What does bert with vision look at?  In ACL, pages 5265–5275, 2020.
  * [42] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In ECCV, pages 121–137. Springer, 2020.
  * [43] Junyang Lin, An Yang, Yichang Zhang, Jie Liu, Jingren Zhou, and Hongxia Yang.  Interbert: Vision-and-language interaction for multi-modal pretraining.  arXiv preprint arXiv:2003.13198, 2020.
  * [44] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei.  Visual relationship detection with language priors.  In ECCV, pages 852–869. Springer, 2016.
  * [45] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In NIPS, pages 13–23, 2019.
  * [46] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee.  12-in-1: Multi-task vision and language representation learning.  In CVPR, pages 10437–10446, 2020.
  * [47] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Xilin Chen, and Ming Zhou.  Univilm: A unified video and language pre-training model for multimodal understanding and generation.  arXiv preprint arXiv:2002.06353, 2020.
  * [48] Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra.  Improving vision-and-language navigation with image-text pairs from the web.  arXiv preprint arXiv:2004.14973, 2020.
  * [49] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In ICCV, pages 2630–2640, 2019.
  * [50] Vishvak Murahari, Dhruv Batra, Devi Parikh, and Abhishek Das.  Large-scale pretraining for visual dialog: A simple state-of-the-art baseline.  In ECCV, 2019.
  * [51] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In ECCV, pages 69–84. Springer, 2016.
  * [52] Aude Oliva and Antonio Torralba.  Modeling the shape of the scene: A holistic representation of the spatial envelope.  IJCV, 42(3):145–175, 2001.
  * [53] Vicente Ordonez, Girish Kulkarni, and Tamara L Berg.  Im2text: Describing images using 1 million captioned photographs.  In NIPS, pages 1143–1151, 2011.
  * [54] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In ICCV, pages 2641–2649, 2015.
  * [55] Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti.  Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data.  arXiv preprint arXiv:2001.07966, 2020.
  * [56] Xuebin Qin, Deng-Ping Fan, Chenyang Huang, Cyril Diagne, Zichen Zhang, Adrià Cabeza Sant’Anna, Albert Suàrez, Martin Jagersand, and Ling Shao.  Boundary-aware segmentation network for mobile and web applications.  arXiv preprint arXiv:2101.04704, 2021.
  * [57] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [58] Negar Rostamzadeh, Seyedarian Hosseini, Thomas Boquet, Wojciech Stokowiec, Ying Zhang, Christian Jauvin, and Chris Pal.  Fashion-gen: The generative fashion dataset and challenge.  arXiv preprint arXiv:1806.08317, 2018.
  * [59] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In ACL, pages 2556–2565, 2018.
  * [60] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  In ICLR, 2020.
  * [61] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  In ICLR, 2020.
  * [62] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi.  A corpus for reasoning about natural language grounded in photographs.  In ACL, 2019.
  * [63] Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid.  Learning video representations using contrastive bidirectional transformer.  arXiv preprint arXiv:1906.05743, 2019.
  * [64] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In ICCV, pages 7464–7473, 2019.
  * [65] Reuben Tan, Mariya I Vasileva, Kate Saenko, and Bryan A Plummer.  Learning similarity conditions without explicit supervision.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10373–10382, 2019.
  * [66] Trieu H Trinh, Minh-Thang Luong, and Quoc V Le.  Selfie: Self-supervised pretraining for image embedding.  arXiv preprint arXiv:1906.02940, 2019.
  * [67] Mariya I Vasileva, Bryan A Plummer, Krishna Dusad, Shreya Rajpal, Ranjitha Kumar, and David Forsyth.  Learning type-aware embeddings for fashion compatibility.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 390–405, 2018.
  * [68] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NIPS, pages 5998–6008, 2017.
  * [69] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  arXiv preprint arXiv:2102.12122, 2021.
  * [70] Yue Wang, Shafiq Joty, Michael R Lyu, Irwin King, Caiming Xiong, and Steven CH Hoi.  Vd-bert: A unified vision and dialog transformer with bert.  In EMNLP, 2020.
  * [71] Yaxiong Wang, Hao Yang, Xueming Qian, Lin Ma, Jing Lu, Biao Li, and Xin Fan.  Position focused attention network for image-text matching.  arXiv preprint arXiv:1907.09748, 2019.
  * [72] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  arXiv preprint arXiv:1609.08144, 2016.
  * [73] Qiaolin Xia, Haoyang Huang, Nan Duan, Dongdong Zhang, Lei Ji, Zhifang Sui, Edward Cui, Taroon Bharti, and Ming Zhou.  Xgpt: Cross-modal generative pre-training for image captioning.  arXiv preprint arXiv:2003.01473, 2020.
  * [74] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In ICML, pages 2048–2057, 2015.
  * [75] Xuewen Yang, Heming Zhang, Di Jin, Yingru Liu, Chi-Hao Wu, Jianchao Tan, Dongliang Xie, Jue Wang, and Xin Wang.  Fashion captioning: Towards generating accurate descriptions with semantic rewards.  arXiv preprint arXiv:2008.02693, 2020.
  * [76] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In NIPS, pages 5753–5763, 2019.
  * [77] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola.  Stacked attention networks for image question answering.  In CVPR, pages 21–29, 2016.
  * [78] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang.  Ernie-vil: Knowledge enhanced vision-language representations through scene graph.  arXiv preprint arXiv:2006.16934, 2020.
  * [79] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  From recognition to cognition: Visual commonsense reasoning.  In CVPR, pages 6720–6731, 2019.
  * [80] Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu.  Ernie: Enhanced language representation with informative entities.  arXiv preprint arXiv:1905.07129, 2019.
  * [81] Jia-Xing Zhao, Jiang-Jiang Liu, Deng-Ping Fan, Yang Cao, Jufeng Yang, and Ming-Ming Cheng.  Egnet: Edge guidance network for salient object detection.  In ICCV, pages 8779–8788, 2019.
  * [82] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J Corso, and Jianfeng Gao.  Unified vision-language pre-training for image captioning and vqa.  In AAAI, pages 13041–13049, 2020.
  * [83] Linchao Zhu and Yi Yang.  Actbert: Learning global-local video-text representations.  In CVPR, pages 8746–8755, 2020.
  * [84] Mingchen Zhuge, Deng-Ping Fan, Nian Liu, Dingwen Zhang, Dong Xu, and Ling Shao.  Salient object detection via integrity learning.  arXiv preprint arXiv:2101.07663, 2021.


# Potential Perturbation of the Ionosphere by Megaconstellations and
Corresponding Artificial Re-entry Plasma Dust

S. Solter-Hunt

###### Abstract

500,000 to 1 million satellites are expected in the next decades, primarily to
build internet constellations called megaconstellations. These
megaconstellations are disposable and will constantly re-enter and be
replaced, hence creating a layer of conductive particulate. Here it will be
shown that the mass of the conductive particles left behind from worldwide
distribution of re-entry satellites is already billions of times greater than
the mass of the Van Allen Belts. From a preliminary analysis, the Debye length
in spaceflight regions is significantly higher than non-spaceflight regions
according to CCMC ionosphere data. As the megaconstellations grow, the Debye
length of the satellite particulate may exceed that of the cislunar
environment and create a conductive layer around the earth worldwide. Thus,
satellite reentries may create a global band of plasma dust with a charge
higher than the rest of the magnetosphere. Therefore, perturbation of the
magnetosphere from conductive satellites and their plasma dust layer should be
expected and should be a field of intensive research. Human activity is not
only impacting the atmosphere, it is clearly impacting the ionosphere.

##  I Introduction

The Van Allen Belts are two torus-like regions of trapped solar energizing
particles that protect the atmosphere. The mass of the Van Allen Belts is
0.00018 kilograms. The masses of other parts of the magnetosphere (ring
current, plasmasphere, etc) are not widely estimated but are less dense than
the Van Allen Belts. The mass of one second generation Starlink satellite is
1250 kilograms[8], in which all of the mass will become conductive particulate
in 5 years during re-entry demolishment and will be added to the lower
ionosphere for an indefinite time. Thus, the space industry is adding enormous
amounts of material to the magnetosphere in comparison to the magnetosphere’s
natural level of particulate and this is forming a layer of artificial plasma
dust in the meteor ablation zone. Due to the conductive nature of the
satellite material, this may perturb or change the magnetosphere. A depletion
or change of the magnetosphere could subsequently have an impact on the
atmosphere.

The magnetosphere is known to be weakening by 10-15% [11] and satellites are
typically involved in aiding that analysis. But in the 2020s and 2030s,
satellites will become so numerous that they will form their own dynamic shell
of conductive material. This shell of conductive material is now much greater
than the radius of the earth. As satellites fill LEO (300 km) to GEO (36,000
km), this layer of conductive material extracted from the Earth’s crust is 6
times the radius of earth. A comparison of space industry altitudes and
regions with the Van Allen Belts is shown in Figure 1. Human spaceflight
activity is thus creating vast regions of charged particles that may impact
the Van Allen Belts and other parts of the magnetosphere in unknown ways.

After consulting multiple magnetosphere models, the consensus was that it
would take decades to simulate 500,000 satellites within a magnetosphere model
and evaluate the potential impact. If the megaconstellations and their debris
are perturbing the magnetosphere, their rate of growth is too fast for decades
of simulations considering the potential urgent ramifications for the
atmosphere. This may be a case where neither simulation or experiment can
assess the question prior to the full deployment of 500,000 satellites. Thus,
this must be considered based on overall planetary-level calculations between
the magnetosphere and the megaconstellations.

The space industry must fund more accurate chemical modeling of the
atmosphere, ionosphere, and magnetosphere and the chemicals, debris, and
materials they are adding to it. It is already suspected that the resulting
re-entry alumina may increase ozone depletion [3]. Additionally, since space
is not considered an earth environment, there is no regard for the sensitivity
of the ionosphere, Van Allen Belts, plasmasphere, or magnetosphere. These
plasma systems may indeed be more sensitive than suspected due to their low
density.

In addition to general mass calculations, ionosphere data and specifically the
charge effectiveness or Debye length, is examined. It should be noted that
this data is only possible because of in-situ satellite and rocket
measurements and thus there is no pre-space-industry data available.

##  II Method

A simulation of the magnetosphere and the megaconstellations is not currently
feasible, and the planetary-scale experiment is underway without a direct
ability to diagnose the satellite-magnetosphere relationship because the
satellites themselves detect the changes in the magnetic field. Thus,
calculations on the mass, Debye length, and a small scale model are compared
to gauge the issue. Debye length is studied by extracting the electron
temperature and density from the Community Coordinated Modeling Center of the
NASA Goddard Space Flight Center. Masses and other reentry statistics are
studied from Jonathan’s Space Report[6]. As space-based development and
pollution grow, there is a strong need for more multidisciplinary
investigations and synthesis across materials, aerospace science, chemistry,
plasma physics, and climate.

##  III Discussion

###  III.1 Mass

The Starlink V2 satellite constellation (just one of many planned
megaconstellations) intends to have 42,000 satellites [6], each the mass of a
SUV, truck, or large car (1250+ kg). Each satellite has a planned lifetime of
only 5 years (if successful and many satellites are failing sooner), thus in
order to maintain the megaconstellation, 23 satellites per day will complete a
re-entry burn in the upper atmosphere. This is approximately 26,308 kilograms
(29 tons) of satellite re-entry material every day, just for the Starlink
megaconstellation. Thus, every day, just for this one constellation, the
conductive mass of material added to the upper atmosphere will be 150 million
times greater than the mass of the protective Van Allen Belts. Thus, every
second, just for this one megaconstellation, the mass of approximately 2,000
Van Allen Belts will be deposited into the ionosphere. Thus, the
megaconstellations are creating their own plasma dust region vastly greater in
mass than that of the Van Allen Belts or any other region of the
magnetosphere. The South Atlantic Anomaly (SSA) is a region of the Van Allen
Belts that comes closest to the Earth at 200 km and is avoided by satellites
due to an increase in ionizing radiation. But the satellites themselves may be
creating a more dangerous ionizing radiation zone with their reentry plasma
dust.

Figure 1: Approximate schematic showing relation of human space activities to
the Van Allen Belts. The relatively much-higher density debris and artificial
plasma dust from human space activities are occurring under, through, and
above the Van Allen Belts, suggesting that the much-higher density charged
particles from these activities may disrupt the Van Allen Belts.

As of March 2023, 299 Starlink V1 (first generation) satellites have already
re-entered [6] at a mass of approximately 300 kg each. This amount of material
is 500 million times the mass of the Van Allen Belts. In 2022 alone, the space
industry polluted approximately 2 billion times the mass of Van Allen Belts
(over 500 tons) in reentry particulate and material from all launches. Since
the beginning of the space industry, approximately 20,000 tons of material [7]
have been demolished during reentry, meaning a similar amount may still remain
as plasma dust. This amount is over 100 billion times greater than the Van
Allen Belts.

This re-entry material will be globally distributed since re-entries are
globally distributed. The locations of re-entries are essentially randomly
distributed. Space launch regions likely have greater densities of plasma dust
development. This increase in ionization in these regions could have adverse
impacts for launch success as ions can interfere with electronics.

The layer of growing re-entry particulate does not dissipate when the
satellite demolishes. It stays stagnant in the upper atmosphere/lower
ionosphere for several years before decay into the lower atmosphere in a best
case scenario. In a worst case scenario, the particulate stays there
indefinitely. Since this material is replenished every day, any potential
natural decay to the ground may be negligible. It should be noted that rocket
and satellite exhaust are also creating plasma dust [9]. If the metal dust is
settling into the atmosphere after several years, the repercussions of vast
amounts of metal dust in the atmosphere includes dangers to the ozone [3] but
any other potential impacts are unknown.

###  III.2 Debye Length

Even with 500,000 satellites, the average distance between those satellites
will be in the hundreds of kilometers. However, the satellites will still have
frequent near passes in the hundreds to thousands of feet, possibly overcoming
the Debye length of the ionosphere and magnetosphere in some regions.

More concerning is the density of aluminium particulate and other conductive
metal particulate that is rapidly accumulating in the lower ionosphere after
reentry demolishment of entire vehicles. These metals from satellites are
already clearly measured in the stratosphere [10]. The satellites break up at
60 to 70 km, and here the ionosphere has a Debye length of approximately 1.5
meters on average between day and night conditions [5]. As shown in Figure 2,
the Debye length goes up in the meteor ablation zone, however, this is likely
from space debris and not meteors, as conductive material from satellites has
far exceeded meteor material. If it is known that the Debye length has
increased noticeably from meteor aluminium, then high concern should be placed
upon the vast amounts of satellite aluminium. It is estimated that the meteor
ablation zone gets approximately 50 tons of meteorite material per day - but
the charged dust from that amount is approximately 1 percent (450 kg)[3].
Thus, one next generation Starlink (1250 kg) creates almost 3 times as much
charged dust as all meteorite material in one day. If the amount of meteor
aluminium and the corresponding charged dust has created an increase of Debye
length by approximately 0.3 meters and satellite dust is approximately 60
times that (using the Starlink constellation alone), then assuming
proportionality of this trend, the satellite aluminium from Starlink alone
would potentially increase the Debye length by over 17 meters. This would be
nearly double the Debye length of the Magnetosheath, which is 9.5 meters [4].
The Debye length in the ionosphere and magnetosphere ranges from a few
centimeters to a few tens of meters. Therefore, all satellite reentries are on
a trajectory to create a band of charge higher than the rest of the
magnetosphere. If this estimation is correct, Starlink reentries alone could
be impacting the ionosphere.

Figure 2: In this figure using CCMC data, it is shown that the electron Debye
length increases at the meteor ablation zone in the lower ionosphere. This
could potentially have been from space debris. This data is from 2010 and
space debris has increased dramatically since. If it is believed that there is
an increase in Debye length from meteorite aluminium, then the effect is
stronger for reentry material, which has vastly exceeded the meteorite
aluminium. As shown here, the increase in the Debye length is at the top of
the ablation zone, further suggesting this is from space industry reentries.

The ion Debye length, which includes mobile ions, is given by

$\displaystyle\lambda_{D}^{2}$ $\displaystyle=\frac{\kappa k_{B}T}{4\pi
q^{2}c_{0}^{2}k^{2}}$

where $\kappa$ is the dielectric constant, q is the ion charge, $k_{b}T$ is
the thermal energy, and $c_{0}$ is the ambient concentration of ions.

The electron Debye length, which is extracted here from CCMC data, is given by

$\displaystyle\lambda_{D}^{2}$ $\displaystyle=\frac{\epsilon
k_{B}T_{e}}{N_{e}e^{2}}$

where $\epsilon$ is the permittivity of free space, $T_{e}$ is the electron
temperature, and $N_{e}$ is the electron density.

According ionospheric models, the lower regions of the ionosphere are
particularly difficult for acquiring data and measurements. These regions are
too low for satellite in situ measurements and difficult to sound from the
ground typically because of the low ion/electron densities and high neutral
densities. So data in this region on electron density relies on rocket
measurements[2]. However, data in a region slightly higher than the meteor
ablation zone, already shows there may be concern because spaceflight regions
are showing much higher Debye lengths than non-spaceflight regions, as shown
in Figure 3.

Figure 3: In this figure using CCMC data, it is shown that the electron Debye
length at 100 km in spaceflight regions is much higher than non-spaceflight
regions. The Debye length in spaceflight regions is trending up over time.
This illustrates that human activity is having an impact on the ionosphere.
Here Kennedy Space Flight Center is the spaceflight region and Reykjavik,
Iceland is the non-spaceflight region. It is preferred to compare Debye length
in the meteor ablation zone, but the exact values are more difficult to
acquire because ionosondes and satellite measurements here are less reliable.

###  III.3 Plasma drag

Plasma drag occurs when charged particles are trapped in a planet’s magnetic
field. These charged particles can overtake the magnetosphere depending on
their location and density. Drag is directly related to density and since it
has been shown here that the masses of artificial plasma dust are going to
create regions of charged particles with a much higher density than elsewhere
in the magnetosphere, it should be expected that plasma drag within the meteor
ablation zone is going to overtake the other regions of the magnetosphere.

###  III.4 Small-scale Scenario

In a basic physics consideration, a spherical magnet (analog Earth) surrounded
by a conductive spherical mesh (analog megaconstellations) would induce a drop
in magnetic field outside of the conductive spherical mesh. This would allow
solar energizing particles to more easily reach the satellite regions (and
atmosphere) and cause satellite failures and thus further stratify the
ionosphere with conductive particulate. The complication is that the spherical
mesh in this scenario is very wide (6 times the radius of Earth) and whether
this circumstance is more adverse is unknown.

However, the layer of the conductive particulate from satellite and spacecraft
reentry demolishment may rapidly condense at current rates of activity. This
layer of metal particles in the lower ionosphere, where the Debye length may
become stronger than the natural ionosphere, will become a conductive shell of
material. Inside the shell, the electric field tends toward 0. Thus outside
the shell, the magnetic field tends toward 0.

###  III.5 Magnetosphere Loss on Mars and Dust on Earth

It is thought that the loss of the magnetosphere on Mars may have involved
separation of planetary materials in the core of Mars [13]. The space industry
on Earth is taking vast amounts of conductive materials naturally found on the
surface and in the crust and injecting them into the ionosphere and beyond,
causing a new stratification of planetary material.

Mars, nearer to the asteroid belt, may have endured more meteor ablation
involving aluminium and other metals. This may have played a part in the
magnetosphere erosion, which the space industry is now accelerating with
incessant satellite reentry demolishment.

Additionally, further research regarding the Chicxulub impact [12] indicates
that the dust from the asteroid was a key mechanism in the extinction of
dinosaurs and life in this period. The space industry is replicating this type
of asteroid dust with more extreme chemicals, dynamics, mass, and combinations
that have not had sufficient study at this time. At 1250 kg per satellite,
only approximately 100,000 to 150,000 satellite re-entry demolishments are
needed to meet the approximate mass of the Chicxulub impact. A new estimate
[3] indicates that 1 million satellites are expected to be maintained and
regularly demolished in total.

##  IV Conclusion

With the estimated projected values for the Starlink megaconstellation alone
(approximately one 1250 kg satellite per hour re-entering), every second the
space industry is adding approximately 2,000 times more conductive material
than mass of the Van Allen Belts into the ionosphere. It appears assumed by
the space industry that the magnetosphere and Van Allen Belts are
indestructible, when in fact the nature of their composition is delicate and
billions of times less massive than the conductive material being added to the
magnetosphere on a regular basis. Continued monitoring and analysis of this
artificial charged dust in comparison to the natural charged dust is needed.

It is known that man-made chemicals can endanger the atmosphere, and the
megaconstellations are not just depositing dangerous chemicals, they are
depositing huge masses of conductive material. It is known that injecting
fossil fuels into the atmosphere, originally from below the Earth’s crust, is
risking the habitability of Earth. It appears likewise for injecting metals
from the ground and crust into the ionosphere. Using just estimates from
Starlink reentry masses, the Debye length of the ablation zone may increase to
over 17 meters if the planned reentry demolishment of 23 satellites per day
occurs. This amount, if globally spread and amplified by hundreds of other
planned and developing megaconstellations, would cause perturbation to the
magnetosphere because it would out-compete the Debye length elsewhere in the
nearby magnetosphere and magnetosheath. It may be the case that satellites,
reentry particulate, and conductive space debris may cause further weakening
and perturbations of the magnetosphere.

##  V Acknowledgement

This work was made possible partly by the satellite monitoring and tracking
work of Dr. Jonathan McDowell of Jonathan’s Space Report, as well as by the
open source data of the Community Coordinated Modeling Center (CCMC). The
editing and support of Dr. Samantha Lawler of University of Regina also made
this work possible. I’d like to also acknowledge Dr. Andy Lawrence for leading
the Starlink FCC appeal, which spurred some of this research.

## References

  * [1]

##  VI References

* [2] Bilitza, D., Pezzopane, M., Truhlik, V., et al, 2022, The International Reference Ionosphere Model: A Review and Description of an Ionospheric Benchmark, Reviews of Geophysics, doi: 10.1029/2022RG000792.
* [3] Boley, A. and Byers, M., 2021, Satellite mega-constellations create risks in Low Earth Orbit, the atmosphere and on Earth, 11, 10642, Scientific Reports, doi: 10.1038/s41598-021-89909-7.
* [4] Champion, K. and Schaub, H., 2022, Effective Debye Lengths in Representative Cislunar Regions, 16th Spacecraft Charging Technology Conference.
* [5] Mann, I., Gunnarsdottir, T., Häggström, I. et al, 2019, Radar studies of ionospheric dusty plasma phenomena, doi: 10.1002/ctpp.201900005.
* [6] McDowell, J. Jonathan’s Space Report, 2023, 819, url: http://planet4589.org/space/con/star/stats.html.
* [7] McDowell, J., Jonathan’s Space Report - Reentry masses, 2023, 819, url: http://www.planet4589.com/space/data/reentry/data/remass.txt.
* [8] McDowell, J., Jonathan’s Space Report - Starlink Simulations, 2023, 819, url: https://planet4589.org/astro/starsim/index.html.
* [9] Merlino, R., 2021, Dusty Plasmas: from Saturn’s rings to semiconductor processing devices, 6, Advances in Physics, doi: 10.1080/23746149.2021.1873859.
* [10] Murphy, D., Abou-Ghanem, M., Cziczo, J., et al, 2023, Metals from spacecraft reentry in stratospheric aerosol particles, The Proceedings of the National Academy of Sciences, doi: 10.1073/pnas.2313374120.
* [11] Schiermeier, Q., 2013, Mission to map Earth’s magnetic field readies for take-off, Nature, doi: 10.1038/nature.2013.14212.
* [12] Senel, C., Kaskes, P., Temel, O. et al, 2023, Chicxulub impact winter sustained by fine silicate dust, Nature Geoscience, doi: 10.1038/s41561-023-01290-4.
* [13] Yokoo, S., Hirose, K., Tagawa, S. and et al, 2022, Stratification in planetary cores by liquid immiscibility in Fe-S-H, Nature Communications, doi: 10.1038/s41467-022-28274-z.


# NAM: Normalization-based Attention Module

Yichao Liu
Helmholtz-Zentrum Dresden-Rossendorf
Dresden, Germany
<EMAIL_ADDRESS>
Zongru Shao
Helmholtz-Zentrum Dresden-Rossendorf
Dresden, Germany
Center for Advanced Systems Understanding
Görlitz, Germany
<EMAIL_ADDRESS>
Yueyang Teng
College of Medicine and Biological Information Engineering
Northeastern University
110004 Shenyang, China
<EMAIL_ADDRESS>
Nico Hoffmann
Helmholtz-Zentrum Dresden-Rossendorf
Dresden, Germany
<EMAIL_ADDRESS>

###### Abstract

Recognizing less salient features is the key for model compression. However,
it has not been investigated in the revolutionary attention mechanisms. In
this work, we propose a novel normalization-based attention module (NAM),
which suppresses less salient weights. It applies a weight sparsity penalty to
the attention modules, thus, making them more computational efficient while
retaining similar performance. A comparison with three other attention
mechanisms on both Resnet and Mobilenet indicates that our method results in
higher accuracy. Code for this paper can be publicly accessed at
https://github.com/Christian-lyc/NAM.

##  1 Introduction

Attention mechanisms have been one of the heated research interests in recent
years (Wang et al. (2017); Hu et al. (2018); Park et al. (2018); Woo et al.
(2018); Gao et al. (2019)). It assists deep neural networks to suppress less
salient pixels or channels. Many of the prior studies focus on capturing
salient features with attention operations (Zhang et al. (2020); Misra et al.
(2021)). Those methods successfully exploit the mutual information from
different dimensions of features. However, they lack consideration on the
contributing factors of weights, which is capable of further suppressing the
insignificant channels or pixels. Inspired by Liu et al. (2017), we aim to
utilize the contributing factors of weights for the improvement of attention
mechanisms. We use a scaling factor of batch normalization which uses the
standard deviation to represent the importance of weights. This can avoid
adding fully-connected and convolutional layers, which is used in the SE, BAM
and CBAM. Thus, we propose an efficient attention mechanism – Normalization-
based Attention Module (NAM).

##  2 Related work

Many prior works attempt to improve the performance of neural networks by
suppressing insignificant weights. Squeeze-and-Excitation Networks (SENet) (Hu
et al. (2018)) integrate the spatial information into channel-wise feature
responses and compute the corresponding attention with two multi-layer-
perceptron (MLP) layers. Later, Bottleneck Attention Module (BAM) (Park et al.
(2018)) builds separated spatial and channel submodules in parallel and they
can be embedded into each bottleneck block. Convolutional Block Attention
Module (CBAM) (Woo et al. (2018)) provides a solution that embeds the channel
and spatial attention submodules sequentially. To avoid the ignorance of
cross-dimension interactions, Triplet Attention Module (TAM) (Misra et al.
(2021)) takes account of dimension correlations by rotating the feature maps.
However, these works neglect information from the tuned weights from training.
Therefore, we aim to highlight salient features by utilizing the variance
measurement of the trained model weights.

##  3 Methodology

We propose NAM as an efficient and lightweight attention mechanism. We adopt
the module integration from CBAM (Woo et al. (2018)) and redesign the channel
and spatial attention submodules. Then, a NAM module is embedded at the end of
each network block. For residual networks, it is embedded at the end of the
residual structures. For the channel attention submodule, we use a scaling
factor from batch normalization (BN) (Ioffe and Szegedy (2015)), as shown in
Equation (1). The scaling factor measures the variance of channels and
indicates their importance.

$B_{out}=BN(B_{in})=\gamma\frac{B_{in}-\mu_{\mathcal{B}}}{\sqrt{\sigma^{2}_{\mathcal{B}}+\epsilon}}+\beta$
(1)

where $\mu_{\mathcal{B}}$ and $\sigma_{\mathcal{B}}$ are the mean and standard
deviation of mini batch $\mathcal{B}$, respectively; $\gamma$ and $\beta$ are
trainable affine transformation parameters (scale and shift) (Ioffe and
Szegedy (2015)). The channel attention submodule is shown in Figure 1 and
Equation (2), where $\mathbf{M}_{c}$ represents the output features. $\gamma$
is the scaling factor for each channel, and the weights are obtained as
$W_{\gamma}=\gamma_{i}/\sum_{j=0}\gamma_{j}$. We also apply a scaling factor
of BN to the spatial dimension to measure the importance of pixels. We name it
pixel normalization. The corresponding spatial attention submodule is shown in
Figure 2 and Equation (3), where the output is denoted as $\mathbf{M}_{s}$.
$\lambda$ is the scaling factor, and the weights are
$W_{\lambda}=\lambda_{i}/\sum_{j=0}\lambda_{j}$.

To suppress the less salient weights, we add a regularization term into the
loss function, as shown in Equation (4) (Liu et al. (2017)), where $x$ denotes
the input; $y$ is the output; $W$ represents network weights; $l(\cdot)$ is
the loss function; $g(\cdot)$ is the $l_{1}$ norm penalty function; $p$ is the
penalty that balances $g(\gamma)$ and $g(\lambda)$.

$\mathbf{M}_{c}=sigmoid(W_{\gamma}(BN(\mathbf{F}_{1})))$ (2)
$\mathbf{M}_{s}=sigmoid(W_{\lambda}(BN_{s}(\mathbf{F}_{2})))$ (3)
$Loss=\sum_{(x,y)}l(f(x,W),y)+p\sum g(\gamma)+p\sum g(\lambda)$ (4)

Figure 1: Channel attention mechanism Figure 2: Spatial attention mechanism

##  4 Experiment

In this section, we compare the performance of NAM with SE, BAM, CBAM, and TAM
for ResNet and MobileNet. We evaluate every method with four Nvidia Tesla V100
GPUs on a cluster. We first run ResNet50 on CIFAR-100 (Krizhevsky et al.
(2009)) and use the same preprocess and training configurations as CBAM (Woo
et al. (2018)), with $p$ as 0.0001. The comparison in Table 1 indicates that
NAM with channel or spatial attention alone outperforms the other four
attention mechanisms. We then run MobileNet on ImageNet (Deng et al. (2009))
as it is one of the standard datasets for image classification benchmarks. We
set $p$ as 0.001 and the rest of the configurations the same as CBAM. The
comparison in Table 2 shows that NAM with channel and spatial attention
combined outperforms the other three with similar computation complexity.

Table 1: Classification results on CIFAR-100

Architecture | Parameters | FLOPs | Top-1 Error ($\%$) | Top-5 Error ($\%$)
---|---|---|---|---
ResNet 50 | 23.71M | 1.30G | 22.74 | 6.37
ResNet 50 + SE | 26.22M | 1.31G | 20.29 | 5.18
ResNet 50 + BAM | 24.06M | 1.33G | 19.97 | 5.03
ResNet 50 + CBAM | 26.24M | 1.31G | 19.44 | 4.66
ResNet 50 + TAM | 23.71M | 1.33G | 20.15 | 5.13
ResNet 50 + NAM(ch*) | 23.74M | 1.31G | 19.09 | 4.5
ResNet 50 + NAM(sp*) | 23.71M | 1.31G | 19.38 | 4.72

  * •

$\star$ ch stands for channel attention only; sp indicates spatial attention
only.

Table 2: Classification results on ImageNet Architecture | Parameters | FLOPs | Top-1 Error ($\%$) | Top-5 Error ($\%$)
---|---|---|---|---
MobileNet V2 | 3.51M | 0.31G | 30.52 | 11.20
MobileNet V2 + SE | 3.53M | 0.32G | 29.77 | 10.65
MobileNet V2 + BAM | 3.54M | 0.32G | 29.91 | 10.80
MobileNet V2 + CBAM | 3.54M | 0.32G | 29.74 | 10.66
MobileNet V2 + NAM | 3.51M | 0.32G | 29.34 | 10.18

##  5 Conclusion

We proposed a NAM module that is more efficient by suppressing the less
salient features. Our experiments indicate that NAM provides efficiency gain
on both ResNet and MobileNet. We are conducting a detailed analysis of the
performance of NAM regarding its integration variations and hyper-parameter
tuning. We also plan to optimize NAM with different model compression
techniques to promote its efficiency. In the future, we will investigate its
effects on other deep learning architectures and applications.

## References

  * Wang et al. [2017] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang.  Residual attention network for image classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3156–3164, 2017.
  * Hu et al. [2018] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7132–7141, 2018.
  * Park et al. [2018] Jongchan Park, Sanghyun Woo, Joon-Young Lee, and In So Kweon.  Bam: Bottleneck attention module.  _arXiv preprint arXiv:1807.06514_ , 2018.
  * Woo et al. [2018] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 3–19, 2018.
  * Gao et al. [2019] Zilin Gao, Jiangtao Xie, Qilong Wang, and Peihua Li.  Global second-order pooling convolutional networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3024–3033, 2019.
  * Zhang et al. [2020] Zhizheng Zhang, Cuiling Lan, Wenjun Zeng, Xin Jin, and Zhibo Chen.  Relation-aware global attention for person re-identification.  In _Proceedings of the ieee/cvf conference on computer vision and pattern recognition_ , pages 3186–3195, 2020.
  * Misra et al. [2021] Diganta Misra, Trikay Nalamada, Ajay Uppili Arasanipalai, and Qibin Hou.  Rotate to attend: Convolutional triplet attention module.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 3139–3148, 2021.
  * Liu et al. [2017] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang.  Learning efficient convolutional networks through network slimming.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2736–2744, 2017.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. PMLR, 2015.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.

##  Appendix A Appendix

###  A.1 Comparison of CBAM and NAM regarding the number of parameters

Table 3: The comparison of channel attention for CBAM and NAM on ResNet50 Parameters | CBAM | NAM
---|---|---
Block1 | 512*4*512*4/16*2 ($C*R*C*R/r*2$) | 512*4 ($C*R$)
Block2 | 256*4*256*4/16*2 | 256*4
Block3 | 128*4*128*4/16*2 | 128*4
Block4 | 64*4*64*4/16*2 | 64*4
Overhead | 696320 | 3840

We show a comparison of the number of parameters in CBAM and NAM in Table 4
and 3. They empirically verify the parameter reduction of NAM. In the channel
attention module, $C$ represents the number of the input channels of each
block. $R$ represents the expanding ratio of each block. $r$ represents the
reduction ratio utilized in the MLP to compute the channel attention, which is
set to 16 in CBAM. The kernel size is denoted as $k$, which is 7. In NAM, $H$
and $W$ represent the height and width of the input images respectively. From
Table 4 and 3, we observe a significant parameter reduction in the channel
attention module and an insignificant increase of parameters in the spatial
attention module of NAM against CBAM. As a result, NAM has fewer parameters
than CBAM.

Table 4: The comparison of spatial attention for CBAM and NAM on ResNet50 Parameters | CBAM | NAM
---|---|---
Block1 | 2*1*7*7 ($2*1*k^{2}$) | 32*32 ($H*W$)
Block2 | 2*1*7*7 | 16*16
Block3 | 2*1*7*7 | 8*8
Block4 | 2*1*7*7 | 4*4
Overhead | 392 | 1360