# RS-DFM: A Remote Sensing Distributed Foundation Model for Diverse Downstream
Tasks

Zhechao Wang, Peirui Cheng, Pengju Tian, Yuchao Wang, Mingxin Chen, Shujing
Duan, Zhirui Wang, , Xinming Li, Xian Sun This work was supported by the
National Natural Science Foundation of China under Grant 62331027 and Grant
62076241. (Corresponding author: Zhirui Wang.)Zhechao Wang, Pengju Tian,
Yuchao Wang, Mingxin Chen, and Xian Sun are with the Aerospace Information
Research Institute, Chinese Academy of Sciences, Beijing 100190, China, also
with the Key Laboratory of Network Information System Technology (NIST),
Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing
100190, China, also with the University of Chinese Academy of Sciences,
Beijing 100190, China, and also with the School of Electronic, Electrical and
Communication Engineering, University of Chinese Academy of Sciences, Beijing
100190, China (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>sunxian@aircas.ac.cn).Peirui Cheng, Zhirui Wang and Xinming Li are with the
Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing
100094, China, and also with the Key Laboratory of Network Information System
Technology (NIST), Aerospace Information Research Institute, Chinese Academy
of Sciences, Beijing 100190, China (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>13911729321@139.com).

###### Abstract

Remote sensing lightweight foundation models have achieved notable success in
online perception within remote sensing. However, their capabilities are
restricted to performing online inference solely based on their own
observations and models, thus lacking a comprehensive understanding of large-
scale remote sensing scenarios. To overcome this limitation, we propose a
Remote Sensing Distributed Foundation Model (RS-DFM) based on generalized
information mapping and interaction. This model can realize online
collaborative perception across multiple platforms and various downstream
tasks by mapping observations into a unified space and implementing a task-
agnostic information interaction strategy. Specifically, we leverage the
ground-based geometric prior of remote sensing oblique observations to
transform the feature mapping from absolute depth estimation to relative depth
estimation, thereby enhancing the model’s ability to extract generalized
features across diverse heights and perspectives. Additionally, we present a
dual-branch information compression module to decouple high-frequency and low-
frequency feature information, achieving feature-level compression while
preserving essential task-agnostic details. In support of our research, we
create a multi-task simulation dataset named AirCo-MultiTasks for multi-UAV
collaborative observation. We also conduct extensive experiments, including 3D
object detection, instance segmentation, and trajectory prediction. The
numerous results demonstrate that our RS-DFM achieves state-of-the-art
performance across various downstream tasks.

###### Index Terms:

distributed foundation model, multi-platform collaboration, generalized
feature mapping, information decoupling.

##  I Introduction

Remote sensing foundation models have been widely applied in remote sensing
intelligent perception[1, 2], relying on powerful generalization performance.
Moreover, the constant iteration of remote observation platform computational
resources and the rapid development of lightweight techniques, such as
knowledge distillation and quantization, have provided the necessary
conditions for online inference of remote sensing foundation models. For
example, Ringmo-lite is able to achieve online inference for object detection,
semantic segmentation and other tasks on a single-edge terminal.

However, remote sensing scenarios typically cover extensive areas and present
complex environment structures. The observational data from a single platform
are limited by perspectives and resolutions, making it difficult to encompass
all information on the observed objects and even facing issues such as
environment occlusion and imaging disturbance. Considering the significant
correlation and complementarity among the multi-perspective, multi-resolution
observation data from distinct observation platforms, which can provide more
comprehensive object information, multi-platform collaborative perception is
an effective solution to address the limitations of single-platform
observations. However, current remote sensing foundation models only perform
online perception based on their own observations, failing to fully utilize
the advantages of multi-platform collaborative observation and computational
resources, making it difficult to achieve online perception based on multi-
platform collaborative observation.

Therefore, to fully leverage multi-platform observation data for multi-task
online perception towards extensive remote sensing scenarios, it is essential
to design a foundation model that supports online collaborative perception
across multiple platforms. The model can not only utilize each observation
platform’s computational resources to achieve online independent perception
and analysis but also leverage the observation data and computational
resources from different platforms to fully explore the association
relationships among multi-platform observation data through multi-platform
information extraction and interactive fusion, thereby achieving multi-
platform collaborative perception and analysis.

Figure 1: Illustration of collaborative perception among multiple remote
sensing platforms. In this scene, several platforms observe the same area from
different angles and heights. Through inter-platform collaboration, they
achieve a more comprehensive understanding of the scene, enhancing the
performance of various downstream tasks.

However, constructing an online collaborative perception foundation model
across multiple remote sensing platforms presents two significant challenges.
The first challenge concerns the consistent recognition issue across distinct
platforms. The significant differences in the altitudes and perspectives
across various observation platforms lead to inconsistent recognition of
information sensitivity to their observations. Therefore, there are severe
gaps among multi-platform information, making it challenging to achieve
information fusion across platforms and precise collaborative analysis towards
objects. Additionally, the information compression loss caused by
communication limitations between platforms is the second challenge. Remote
sensing observation platforms need real-time observation information
interaction to perform online collaborative inference. Remote sensing images
have high resolution, and remote sensing platforms are typically distributed
over extensive and complex environments, where the wireless communication
distance and bandwidth have limitations. Therefore, it is inevitable to
compress the information during the real-time information interaction between
different observation platforms. However, during information compression, the
loss of some critical information results in incomplete object information and
even interferes with the observation data itself, thereby reducing the multi-
platform collaborative reasoning capability.

To address these challenges, this paper propose a multi-platform collaborative
inference framework based on the concept of “physical separation, logical
integration”. This framework is enhanced in two key areas: collaborative
information extraction and selective information compression interaction. By
mapping observational information from different altitudes and perspectives
into a unified space, and by decoupling high-frequency and low-frequency
information in remote sensing for selective information compression, we
achieve online multi-task collaborative perception across multiple remote
sensing observation platforms. Specifically, we propose a Remote Sensing
Distributed Foundation Model based on information generalized interaction and
fusion, which enables individual independent perception on a single terminal
and group collaborative perception across multiple platforms. Moreover, we
propose a Generalized BEV Generalization (GBG) module, which utilizes the
geometric prior of remote sensing observation perspectives to calculate the
upper depth limit of pixels. This approach transforms depth estimation from an
absolute classification task into a relative classification task based on the
upper depth limit, thereby enabling the generalized mapping of different
altitudes and perspectives. Finally, we propose a High-Low Frequency Decoupled
Collaboration (HLFDC) module. This module leverages distinct attention
mechanisms to extract high-frequency and low-frequency features. After
employing compression methods for these features, the module transmits the
most critical information with minimal bandwidth. To validate the
effectiveness of our algorithm, we developed a simulated dataset for multi-UAV
collaborative observation, named AirCo-MultiTasks. The dataset contains 5 UAVs
at different flight heights and sets rich scenes and objects, supporting
multi-tasks such as detection, tracking and prediction. Through experimental
validation on this dataset, our algorithm implemented SOTA on various
downstream tasks.

Figure 2: The overall architecture of our proposed RS-DFM framework. For
clarity, we just present the collaboration between two platforms.

The main contributions of this paper can be summarized as follows:

We propose a remote sensing distributed foundation model based on information
generalized interaction and fusion, which enables individual independent
perception on a single terminal and group collaborative perception across
multiple platforms.

We present a generalized BEV generation module, which utilizes remote sensing
observation geometric prior to achieve the generalized mapping of features
from different altitudes and perspectives.

We design a high-low frequency decoupled collaboration module, which reduces
transmission bandwidth and achieves efficient selective information
transmission.

A large-scale multi-task multi-UAV collaborative perception dataset, AirCo-
MultiTasks, is introduced to validate the effectiveness of the proposed
framework.

##  II Related Work

###  II-A Remote Sensing Foundation Model

The value of using foundation models has become standard practice in computer
vision. Many models provide strong baselines and improve performance across
various tasks. ViT[3] breaks through the limitations of traditional CNNs by
utilizing the Transformer[4] architecture. MAE[5] utilizes transformer-based
masked image reconstruction approaches to enhance the model’s robustness and
representation capability and SimMIM[6] further achieves efficient image
modeling through a simplified design.

Foundation models have also found widespread applications in intelligent
remote sensing perception, with numerous excellent models emerging
continuously. RingMo[7] is the first generative self-supervised foundation
model framework in the field of RS. This framework exploits masked image
modeling to obtain general feature representation and increase the accuracy of
various RS perception tasks. Billionmodel[8] introduces the first billion-
scale foundation model in the field of remote sensing, marking the largest
model size explored in this domain. RingMo-SAM[9] can process both optical and
SAR remote sensing data, capable of zero-shot semantic segmentation and object
categories identification, with strong generalization ability.

However, foundation models are not flexible and efficient due to the high
demand for computing and storage resources, making it difficult to adapt to
edge servers or terminals, and cannot support on-orbit RS image perception in
practical applications. RingMo-Lite[10] aims to address the high computational
resource requirements in remote sensing image perception, enabling it to run
on edge devices while maintaining high accuracy and versatility across
multiple tasks.

###  II-B Collaborative perception

Collaborative perception is a pivotal approach for intelligent data
perception. It is efficiently applied in various scenarios by collaboratively
processing information from multiple observation platforms. In autonomous
driving, Who2com[11] implements a handshake mechanism to choose collaborative
partners, while When2com[12] decides the appropriate time to start
collaboration. Where2com[13] utilizes the detection head to guide regions for
sparse interactions. CoCa3D[14] achieves collaborative depth estimation
through considering single-view depth probability and multi-view consistency.
HEAL[15] proposes an extensible heterogeneous collaborative perception
framework, which ensures extensibility by establishing a unified feature space
and aligning new agents to it. CodeFilling[16] introduces codebook-based
message representation and information-filling-driven message selection for
perception-communication trade-off.

Collaborative perception also plays an important role in remote sensing
scenarios. MDCNet[17] proposes a multi-platform distributed collaborative
inference network for object detection. Through the selective communication
strategy for multi-platform feature interaction, the method improves the
accuracy of object detection and optimizes the inference efficiency by
breaking through the limitations of single-platform observation range and
precision. DCNNet[18] propose a terminal-cloud collaborative, progressive
inference mechanism for image classification, with a distributed self-
distillation paradigm designed to integrate and refine in-depth features,
performing efficient knowledge transfer between the terminals and the cloud
network.

Figure 3: Operation flow of the GBG module. This module introduces a
generalized feature mapping through relative depth estimation, which enhances
the accuracy of BEV representation.

##  III Methods

This section presents RS-DFM, a well-designed collaborative perception
framework for remote sensing platforms, which enables multiple platforms to
share visual information, promoting more holistic perception. As depicted in
Fig.2, RS-DFM comprises four components (i.e., feature extraction, GBG module,
HLFDC module, and supports for multiple downstream tasks) to implement the
entire collaboration process. We briefly introduce these components in the
overview, while delving into the innovative designs of the GBG and HLFDC
modules in Section 3 and Section 4, respectively.

###  III-A Overview

Typically, the RS-DFM framework encompasses the following components.

Feature Extraction. Initially, each platform captures and processes its
observations using a shared encoder to extract semantic information. We
utilize the Swin-Tiny[29] backbone as each drone’s encoder due to its low
inference latency and optimized memory usage within the family of transformer-
based backbones.

GBG Module. For collaboration among platforms, image features need to be
transformed from each pixel coordinates to BEV representations in the world
coordinates. Based on the encoded image features, the GBG module provides a
generalized depth estimation for various platforms at different angles and
heights by leveraging the geometric prior. Then, image features and their
corresponding estimated depths are used to generate frustum features, which
are then converted into voxel features with the help of the camera’s intrinsic
and extrinsic parameters. Ultimately, sum-pooling and high-dimensional
reduction are applied to the voxel features to derive BEV representations,
convenient for later collaboration.

HLFDC Module. The HLFDC Module aims to balance enhanced perception with
transmission cost in inter-platform collaboration. Given the characteristics
of remote sensing images, this module achieves information compression by
decoupling high-frequency and low-frequency information. High-frequency
information, which captures fine details, is preserved, whereas low-frequency
information, representing the global structure, is compressed. Following
collaborative interactions, local features and transmitted compressed
collaborative features are fused through pixel-wise weighting, facilitating
subsequent downstream tasks.

Supports for Multiple Downstream Tasks. The collaboration among multiple
platforms aims to enrich their respective perceptual information and is not
limited to a specific task. Consequently, RS-DFM supports various downstream
tasks, including 3D object detection, BEV instance segmentation, and multi-
object tracking. Specifically, the fused features generated by the HLFDC
module are fed into downstream task heads to obtain enhanced perception
results.

###  III-B Generalized BEV Generation Module

Various remote sensing platforms lead to the significant discrepancy in
observations of identical objects due to different viewing angles and
altitudes. Consequently, it’s necessary to convert multi-view observations
into a unified spatial coordinate before information fusion. To this end, we
transform the 2D image features to the BEV coordinate system. The BEV system
offers distinct advantages, particularly in terms of scale and spatial
consistency, which can effectively mitigate gaps among multi-view
observations. Depth estimation serves as a critical component in generating
BEV representations. However, the vast range of aerial observations, coupled
with various estimation intervals, presents challenges for the prevalent lss
method in approximating the depth of each pixel. To address this issue, we
leverage the geometric priors provided by inclined observations and propose a
novel Generalized BEV Generation Module. The procedural methodology of the
proposed module is outlined below.

####  III-B1 Derivation of geometric constraint

In this part, we utilize the camera’s intrinsics and extrinsics parameters to
calculate the corresponding depth upper-bound for each position in the
observation.

According to the pinhole camera model, the transformation between the point
$(x,y,z)$ in the world coordinate system and its projection $(u,v)$ in the
pixel coordinate system can be described by the equation:

$\small\begin{bmatrix}x\\\ y\\\
z\end{bmatrix}=\mathbf{R}^{-1}\left(\mathbf{K}^{-1}\begin{bmatrix}u\\\ v\\\
1\end{bmatrix}{d}-\mathbf{T}\right).$ (1)

Here, the intrinsic matrix of the camera is denoted as
$\mathbf{K}\in\mathbb{R}^{3\times 3}$, and the extrinsics are represented by a
rotation matrix $\mathbf{R}\in\mathbb{R}^{3\times 3}$ and a translation vector
$\mathbf{T}\in\mathbb{R}^{3\times 1}$. The symbol $d$ represents the exact
pixel depth in the camera coordinate system.

For notational brevity, we denote $\mathbf{R}^{-1}\mathbf{K}^{-1}$ as the
matrix $\mathbf{M}=(m_{ij})\in\mathbb{R}^{3\times 3}$ and
$\mathbf{R}^{-1}(-\mathbf{T})$ as the vector
$\mathbf{N}=(n_{i})\in\mathbb{R}^{3\times 1}$. Hence, the equation (1) is
modified as:

$\small\begin{bmatrix}x\\\ y\\\ z\end{bmatrix}=\mathbf{M}\begin{bmatrix}u\\\
v\\\ 1\end{bmatrix}d+\mathbf{N}=\begin{bmatrix}m_{11}&m_{12}&m_{13}\\\
m_{21}&m_{22}&m_{23}\\\ m_{31}&m_{32}&m_{33}\end{bmatrix}\begin{bmatrix}u\\\
v\\\ 1\end{bmatrix}d+\begin{bmatrix}n_{1}\\\ n_{2}\\\ n_{3}\end{bmatrix}.$ (2)

In reference to the world coordinate system, the ground plane is located in
the negative direction of the drone’s y-axis. By setting $y=-H$ in the
equation (2), where $H$ represents the altitude of the drone, the depth upper-
bound of each position in the camera coordinate can be derived as:

$\small{D_{(u,v)}}=\frac{-H-n_{2}}{m_{21}u+m_{22}v+m_{23}}.$ (3)

Following the acquisition of the depth upper bound for each pixel, we utilize
this geometric constraint as a prior for optimizing the depth estimation.

####  III-B2 A relative depth estimation

In contrast to the previous absolute depth estimation, we introduce a relative
depth estimation method to calculate the ratio of the each pixel’s depth to
the depth upper-bound. The global unified depth estimation interval is
replaced by a flexible relative depth estimation interval $(0,1)$ scaled by
the depth upper-bound for each region.

Specifically, we utilize a parametric network $\Phi_{\text{depth}}(\cdot)$,
which generates a pixel-wise categorical distribution of depth, represented as
$d_{\text{pred}}=\Phi_{\text{depth}}(F_{(u,v)})\cdot
D_{(u,v)}\in\mathbb{R}^{D}$. Here, $D$ signifies the number of discretized
probability of depth bins. We treat the estimation as a classification problem
to capture the inherent depth estimation uncertainty and reduce the impact of
erroneous depth estimates, as stated in LSS.

####  III-B3 BEV Generation

The following process transforms 2D observations into BEV representations
based on the aforementioned pinhole camera model, in equation (1). First,
operate the outer product on the encoded features,
$F_{\text{2D}}\in\mathbb{R}^{W\times H\times C}$, and the corresponding
estimated depth, $D_{\text{pred}}=\left[d_{\text{pred}_{(h,w)}}\right]_{(w\in
W,h\in H)}\in\mathbb{R}^{W\times H\times D}$, to obtain the frustum features,
which aim to recover the spatial information of the image. Then, project the
frustum features into BEV pillars using the camera’s intrinsics and
extrinsics. This operation converts multi-view observations into a unified
world coordinate system. Finally, perform z-axis sum pooling at each spatial
position $(x,y,:)$, collapsing the height dimension to generate the BEV
features, $F_{\text{BEV}}\in\mathbb{R}^{X\times Y\times C}$. The obtained BEV
representations facilitate subsequent information exchange among different
platforms.

Figure 4: Operation flow of the HLFDC module. This module compresses the
original local features through two branches: low frequency and high
frequency. In the high-frequency branch, local attention is conducted to
obtain high-frequency features reflecting objects’ edges, shapes, and other
details. These high-frequency features are then condensed into low-dimensional
channels. Conversely, the low-frequency branch preserves the overall
structural information through low-filtered average pooling and global
attention. The high-frequency branch condenses spatial information into down-
sampled representations.

###  III-C High-Low Frequency Decoupled Collaboration Module

Communication among remote sensing platforms is often restricted,
necessitating the partial transmission of valuable information to achieve
efficient collaborative perception. As indicated in the literature[19, 20],
the high-frequency components of feature maps depict object details, while
low-frequency components typically capture the global structure. Given that
remote sensing images contain small and dense foreground objects with rich
background information, we tend to preserve intricate details and compress
global spatial information for lightweight transmission. Therefore, we propose
a collaborative strategy based on high-low frequency decoupling, implemented
as follows:

####  III-C1 High Frequency Information Extraction

The high-frequency branch is utilized to capture local dependencies that
reflect fine details. To this end, we implement a local window self-attention
with the simple non-overlapping window partition.

The original BEV feature map is segmented into multiple local feature maps
using non-overlapping windows of size $M\times M$. Each local feature map is
then reshaped into a feature sequence,
$F_{\text{seq}}\in\mathbb{R}^{M^{2}\times C}$, which serves as the input to
the multi-head self-attention layer. Each self-attention head calculates the
query $\mathbf{Q}$, key $\mathbf{K}$ and value $\mathbf{V}$ matrices with a
linear transformation from $F_{\text{seq}}$,

$\mathbf{Q}=F_{\text{seq}}\mathbf{W}_{q},\mathbf{K}=F_{\text{seq}}\mathbf{W}_{k},\mathbf{V}=F_{\text{seq}}\mathbf{W}_{v},\in\mathbb{R}^{M^{2}\times
D_{h}}$ (4)

where $\mathbf{W}_{q}$, $\mathbf{W}_{k}$,
$\mathbf{W}_{v}\in\mathbb{R}^{C\times D_{h}}$ are learnable parameters and
$D_{h}$ is the number of hidden dimensions for a head. Next, the output of a
self-attention head is a weighted sum over $M^{2}$ value vectors,

$\mathrm{SA}_{h}(F_{\text{seq}})=\mathrm{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{D_{h}}}\right)\mathbf{V}\in\mathbb{R}^{M^{2}\times
D_{h}}.$ (5)

For the multi-head self-attention layer with $N_{h}$ heads, the final output
is computed by a linear projection of the concatenated outputs from each self-
attention head, which can be formulated by:

$\mathrm{MSA}(F_{\text{Seq}})=\mathrm{concat}_{h\in[N_{h}]}[\mathrm{SA}_{h}(F_{\text{Seq}})]\mathbf{W}_{o}\in\mathbb{R}^{M^{2}\times
C^{\prime}},$ (6)

where $\mathbf{W}_{o}\in\mathbb{R}^{(N_{h}\times D_{h})\times C^{\prime}}$ is
a learnable parameter. Herein, we set $C^{\prime}=C/2$ and restructure the
high-frequency feature sequences into their spatial format, which are then
combined to form the complete high-frequency spatial feature
$F_{\text{high}}\in\mathbb{R}^{X\times Y\times C^{\prime}}$.

####  III-C2 Low Frequency Information Extraction

Several studies[21] have indicated the effectiveness of the global attention
mechanism in capturing low-frequency information. However, due to the limited
computational budget, it is expensive to directly perform global attention on
original high-resolution feature maps. Inspired by [22], we first initially
utilize an average pooling operation to achieve dual functions of low-pass
filtering and down-sampling the feature maps. Considering the complementary
high-low frequency, the stride of the average pooling is set to match the size
of the local window in the high-frequency branch.

Specifically, the original BEV feature map is average-pooled with a kernel and
stride of size $M$ to obtain the initial low-frequency signals, which are
projected into $N_{h}$ queries, keys, and values,
$\mathbf{Q},\mathbf{K},\mathbf{V}\in\mathbb{R}^{XY/M^{2}\times D_{h}}$,
respectively. Then, we apply the global multi-head self-attention to capture
the rich low-frequency feature sequences that are finally reshaped to the low-
frequency spatial features $F_{\text{low}}\in\mathbb{R}^{X/M\times Y/M\times
C^{\prime}}$. The process of multi-head self-attention is same with the
equations (4–6).

####  III-C3 Inter-platform Collaborative Features Fusion

After obtaining the high-low frequency decoupled features at each remote
sensing platform, these features are transmitted to other platforms to
facilitate collaborative perception. Upon receiving the collaborative
features, a geometric transformation $G$ is applied to align them with the
local coordinate system, ensuring spatial congruence. Subsequently, the
platform integrates high-frequency and low-frequency information to restore
the original information. As the low-frequency features are average pooled, a
series of transposed convolutions is utilized to up-sample them to the
original size, ensuring the consistence with the shape of the high-frequency
features. Consequently, the processed high-low frequency features are
concatenated to generate the collaborative features $F_{\text{col}}$ for
subsequent fusion, which can be expressed as:

$F_{\text{col}}=\text{concat}[\text{Up}(G(F_{\text{low}}));G(F_{\text{high}})]\in\mathbb{R}^{X\times
Y\times C}.$ (7)

Guided by local features $F_{\text{BEV}}$, the contribution weight $W_{j}$ of
$F_{\text{col,j}}$ towards constructing fused features $F_{\text{fused}}$ is
quantified by:

$W_{j}=\frac{\phi_{\text{conv}}\left([F_{\text{BEV}};F_{\text{col,j}}]\right)}{\sum_{j=1}^{N}\phi_{\text{conv}}\left([F_{\text{BEV}};F_{\text{col,j}}]\right)}\in\mathbb{R}^{X\times
Y\times 1}$ (8)

, where $\phi_{\text{conv}}$ represents the multi-layer convolution
operations. Moreover, a pixel-level weighted fusion is executed to generate
the fused features for subsequent downstream tasks:

$F_{\text{fused}}=\sum_{j=1}^{N}W_{j}F_{\text{col,j}}\in\mathbb{R}^{X\times
Y\times C}.$ (9)

##  IV Experiments

###  IV-A Datasets

Figure 5: Statistical charts for the AirCo-MultiTasks dataset, depicting the
occlusion within a single view, the number of various object types, the
distribution of distances between objects and drones, and the proportion of
objects within observations, respectively.

Through collaboration, UAV swarms can achieve extended perception range,
enhanced robustness, and improved flexibility, leading to better performance
in various tasks under complex environments. However, available datasets for
multi-drone collaboration primarily focus on detection and segmentation,
lacking support for prediction tasks. To provide a more diverse and
challenging benchmark for the multi-drone perception community, we introduce
AirCo-MultiTasks, a simulation dataset based on CARLA[23] that supports multi-
task collaborative perception for multi-UAVs.

Specifically, five collaborative drones are positioned at intersections to
monitor traffic flow from different directions. Three of these drones fly at a
low altitude of 30 meters, one flies at 50 meters, and one flies at a high
altitude of 70 meters, covering an area of approximately 150m $\times$ 150m.
To illustrate the challenges of aerial observations intuitively, we present
several charts to reflect the characteristics of AirCo-MultiTasks, such as
occlusion, long-distant observations, small objects, etc., as shown in Fig.5.

Each UAV is equipped with three types of sensors: a device for capturing RGB
images, a device for environmental perception, and a device for transmitting
perceptual information between UAVs. This setup enables the UAVs to achieve
collaborative perception for various tasks, including 3D object detection, BEV
semantic segmentation, and target trajectory prediction.

The dataset comprises 38,000 synchronous images with a resolution of 1600 x
900, captured by five collaborative drones. It is divided into 170 training
scenes and 30 validation scenes. To facilitate multi-task training and
evaluation, each frame is meticulously annotated with both 2D and 3D labels,
encompassing three primary object categories: vehicles, bicycles, and
pedestrians. This dataset serves as a valuable tool for assessing the
performance of collaborative perception methods across a spectrum of
perception tasks, including 2D and 3D object detection, BEV segmentation, and
trajectory prediction. A detailed breakdown of the dataset is provided below.

Data collection. The proposed dataset was collected using CARLA simulation. We
generated complex traffic scenarios in CARLA and deployed five UAVs at three
different flight heights at intersections or roadsides to jointly monitor
traffic flow in the same area from different perspectives.

Sensor setup. Each UAV is equipped with an RGB optical camera, and UAVs at
different heights observe the same area with different pitch angles. The field
of view (FoV) of each camera is 70°, and the resolution is 1600 $\times$ 900
pixels. The translation (x, y, z) and rotation (w, x, y, z in quaternion) of
each camera are recorded during data collection. With this sensor setup and
flight height configuration, the collaborative region observed by the five
UAVs can cover approximately 150m $\times$ 150m.

Scene creation. Roads, static buildings, and dynamic targets in the simulation
scene are all created by CARLA. We selected 33 locations for data collection
from three high-quality simulation maps provided by CARLA. To enhance the
richness and complexity of the scenes, the weather and traffic flow in each
scene were randomly initialized during data collection.

Traffic flow generation. Vehicles and pedestrians in the simulation scene are
controlled and managed by CARLA. The hundreds of dynamic targets in the scene
inherit the Actor class in CARLA and are uniformly spawned, managed, and
destroyed by the simulator. The number and location of target spawn points in
each map are predetermined, and the spawn points are all located near roads.

Data annotation. We collected synchronous images from all cameras on five
UAVs, brings about a sample size of 5 images. Camera intrinsics and extrinsics
in global coordinates were provided to facilitate coordinate transformation
across the UAVs. During data collection, 3D bounding boxes of vehicles and
pedestrians were synchronously recorded, including location (x, y, z),
rotation (w, x, y, z in quaternion) in the global coordinate system, and
length, width, height, and visibility.

###  IV-B Evaluation Metrics

####  IV-B1 Object Detection

The 3D object detection evaluation metrics are mean Average Precision (mAP),
mean Average Translation Error (mATE), mean Absolute Scale Error (mASE), and
mean Absolute Orientation Error (mAOE). These metrics are defined as follows:

  * •

mAP: This metric measures the average precision of the detected objects across
different classes and various match thresholds. It uses the Average Precision
(AP) metric, which defines a match by thresholding the 2D center distance on
the ground plane instead of intersection over union. AP is calculated as the
normalized area under the precision-recall curve, considering only points
where recall and precision are both above 10%. This approach helps to minimize
the impact of noise often found in low precision and recall regions. Then, an
average is computed over matching thresholds $D=\\{0.5,1,2,4\\}$ meters and
the set of classes $C$:

$\small\operatorname{mAP}=\frac{1}{|C||D|}\sum_{c\in C}\sum_{d\in D}AP_{c,d},$
(10)

where $AP_{c,d}$ is the Average Precision for class $c$ at threshold $d$.

  * •

mATE: This metric evaluates the average translation error of the detection
bounding box centers. It measures the accuracy of the model in detecting the
position of the objects. The specific calculation is as follows:

$\small\operatorname{mATE}=\frac{1}{N}\sum_{i=1}^{N}\|c_{i}-\hat{c}_{i}\|,$
(11)

where $c_{i}$ and $\hat{c}_{i}$ are the ground truth and predicted centers of
the $i$-th bounding box, respectively, and $N$ is the total number of detected
objects.

  * •

mASE: This metric measures the average scale error of the detection bounding
boxes. It evaluates the accuracy of the model in estimating the size of the
objects. The specific calculation is as follows:

$\small\operatorname{mASE}=\frac{1}{N}\sum_{i=1}^{N}\left|s_{i}-\hat{s}_{i}\right|,$
(12)

where $s_{i}$ and $\hat{s}_{i}$ are the ground truth and predicted scales
(e.g., width, height, length) of the $i$-th bounding box, respectively, and
$N$ is the total number of detected objects.

  * •

mAOE: This metric assesses the average orientation error of the detection
bounding boxes. It evaluates the accuracy of the model in estimating the
orientation of the objects. The specific calculation is as follows:

$\small\operatorname{mAOE}=\frac{1}{N}\sum_{i=1}^{N}\left|\theta_{i}-\hat{\theta}_{i}\right|,$
(13)

where $\theta_{i}$ and $\hat{\theta}_{i}$ are the ground truth and predicted
orientations of the $i$-th bounding box, respectively, and $N$ is the total
number of detected objects.

####  IV-B2 Instance Segmentation

We adopt four metrics widely used in previous works [24, 25], including
Intersection-over-Union (IoU), Segmentation Quality (SQ), Recognition Quality
(RQ), and Panoptic Quality (PQ). These metrics are defined as follows:

  * •

IoU: This metric evaluates the segmentation quality of objects. It is
calculated as follows:

$\small\operatorname{IoU}\left(\hat{y}^{\mathrm{seg}},y^{\mathrm{seg}}\right)=\frac{\sum_{h,w}\hat{y}^{\mathrm{seg}}\cdot
y^{\mathrm{seg}}}{\sum_{h,w}\hat{y}^{\mathrm{seg}}+y^{\mathrm{seg}}-\hat{y}^{\mathrm{seg}}\cdot
y^{\mathrm{seg}}},$ (14)

where $\hat{y}^{\mathrm{seg}}$ and $y^{\mathrm{seg}}$ denote the predicted and
ground truth semantic segmentation, respectively.

  * •

SQ: This metric evaluates the segmentation quality of the correctly detected
segments. It is the average IoU of the true positives (TP):

$\small\operatorname{SQ}\left(\hat{y}^{\text{inst}},y^{\text{inst}}\right)=\frac{\sum_{\left(p,q\right)\in
TP}\operatorname{IoU}\left(p,q\right)}{|TP|}.$ (15)

  * •

RQ: This metric evaluates the recognition quality of the predicted segments,
considering both true positives (TP) and the penalties for false positives
(FP) and false negatives (FN):

$\small\operatorname{RQ}\left(\hat{y}^{\text{inst}},y^{\text{inst}}\right)=\frac{|TP|}{|TP|+\frac{1}{2}|FP|+\frac{1}{2}|FN|}.$
(16)

  * •

PQ: This metric considers both the quality of segmentation and the accuracy of
instance recognition, capturing both the overlap between predicted and ground
truth segments and the correct classification of instances. The metric is
expressed as:

$\small\operatorname{PQ}\left(\hat{y}^{\text{inst}},y^{\text{inst}}\right)=\frac{\sum_{\left(p,q\right)\in
TP}\operatorname{IoU}\left(p,q\right)}{\left|TP\right|+\frac{1}{2}\left|FP\right|+\frac{1}{2}\left|FN\right|}.$
(17)

$TP$, $FP$, and $FN$ correspond to true positives, false positives, and false
negatives, respectively. $TP$ are the correctly predicted segments with an IoU
over 0.5 and matching ground truth segments. $FP$ are the predicted segments
that do not match any ground truth segments. $FN$ are the ground truth
segments that are not predicted by the model.

####  IV-B3 Trajectory Prediction

We adopt IoU and Video Panoptic Quality (VPQ) [26, 25] as the temporal
evaluation metrics.

  * •

IoU: For frame-level evaluation, IoU evaluates the segmentation quality of
objects at the present and future frames. The specific calculation is as
follows:

$\small\operatorname{IoU}\left(\hat{y}_{t}^{\mathrm{seg}},y_{t}^{\mathrm{seg}}\right)=\frac{1}{N}\sum_{t=0}^{N-1}\frac{\sum_{h,w}\hat{y}_{t}^{\mathrm{seg}}\cdot
y_{t}^{\mathrm{seg}}}{\sum_{h,w}\hat{y}_{t}^{\mathrm{seg}}+y_{t}^{\mathrm{seg}}-\hat{y}_{t}^{\mathrm{seg}}\cdot
y_{t}^{\mathrm{seg}}},$ (18)

where $N$ denotes the number of output frames, $\hat{y}_{t}^{\mathrm{seg}}$
and $y_{t}^{\mathrm{seg}}$ denote the predicted and ground truth semantic
segmentation at timestamp $t$, respectively.

  * •

VPQ: For video-level evaluation, VPQ reflects the quality of the segmentation
and ID consistency of the instances through the video, expressed as:

$\small\operatorname{VPQ}\left(\hat{y}_{t}^{\text{inst}},y_{t}^{\text{inst}}\right)=\sum_{t=0}^{N-1}\frac{\sum_{\left(p_{t},q_{t}\right)\in
TP_{t}}\operatorname{IoU}\left(p_{t},q_{t}\right)}{\left|TP_{t}\right|+\frac{1}{2}\left|FP_{t}\right|+\frac{1}{2}\left|FN_{t}\right|},$
(19)

where $TP_{t}$, $FP_{t}$, and $FN_{t}$ correspond to true positives, false
positives, and false negatives at timestamp $t$, respectively.

###  IV-C Implementation Details

We adhere to the setups introduced in previous studies [27, 28, 26] for
various BEV-specific downstream tasks. Initially, the raw images, with a
resolution of $900\times 1600$ pixels, are scaled and randomly cropped to
$256\times 704$ pixels. For view transformation, the relative depth estimation
range is set from 0 to 1 and discretized into 100 intervals. For BEV
representations, the spatial ranges for the $x$, $y$, and $z$ axes are
configured as $[-75,75]$ meters, $[-75,75]$ meters, and $[0,10]$ meters,
respectively. Model performance is evaluated in two perceptual scopes: a
$150\,\text{m}\times 150\,\text{m}$ area (long-range) and a
$75\,\text{m}\times 75\,\text{m}$ area (short-range), both with a 0.75-meter
resolution. Specifically, we adopt the backbone of Swin-Tiny [29] as the
encoder for each drone because of its light-weight design and low inference
latency. For the HLFDC module, the size of local window is set to 4. Regard
downstream tasks, BEVDET[27] is employed as the detector for 3D object
detection, Fiery[28] as the segmentator for BEV instance segmentation, and
PowerBEV[26] as the trajectory predictor for BEV trajectory prediction. Our
proposed framework is trained using the AdamW optimizer with an initial
learning rate of $2\times 10^{-4}$. The training process is conducted on four
NVIDIA A40 GPUs, with a batch size of 8 for 20 epochs.

###  IV-D Comparative Baselines for Collaborative perception

Early collaboration involves the integration of raw observations, where multi-
view images are combined to generate BEV representations.

Intermediate collaboration focuses on feature-level interactions to produce
comprehensive BEV representations. These approaches can be either fully
connected or partially connected:

  * •

The fully connected paradigm shares complete features among all members.

    * –

V2VNet [30] performs multi-round message exchanges using graph neural
networks.

    * –

V2X-ViT [31] explores correlations among heterogeneous collaborators through
transformer blocks.

  * •

The partially connected paradigm restricts interactions to certain members or
regions.

    * –

Who2com [11] introduces a handshake mechanism to select collaborative
partners.

    * –

When2com [12] determines the optimal timing for initiating collaboration.

    * –

Where2comm [13] uses the detection head to direct interactions towards
specific regions.

Late collaboration combines individual prediction results from multiple drones
to form a final decision.

###  IV-E Quantitative Evaluation

####  IV-E1 Benchmark Comparison in 3D Object Detection

Tab. I shows that our proposed approach, RS-DFM, achieves the highest
performance in terms of mAP and mATE among all collaborative perception
methods, even much better than the early collaboration without incorporating
generalized feature mapping. This superior performance can be attributed to
our designed GBG module, which facilitates more accurate object localization
and yields outstanding results in position-related metrics. While our RS-DFM
may exhibit slightly lower performance in mASE and mAOE, it still sharply
reduces these errors compared to the majority of collaboration approaches.
Additionally, RS-DFM significantly outperforms No-Collaboration, with a 202%
increase in mAP for the short range and a 254% increase in mAP for the long
range, indicating the effectiveness of collaboration. In contrast to fully
connected baselines, our RS-DFM enhances object detection accuracy and recall
by 2–4% and reduces errors in localization, scale, and orientation by 6–30%,
while requiring only approximately half the transmission ratio. When compared
to the partially connected baselines, RS-DFM surpasses the previous state-of-
the-art method, Where2com, by a considerable margin, including a 4.4% increase
in mAP and decreases of 6% in mATE, 10% in mASE, and 9.5% in mAOE during the
long-range setting.

TABLE I: A comparison of collaboration baselines for 3D object detection.

Methods | mAP$\uparrow$ | mATE $\downarrow$ | mASE $\downarrow$ | mAOE $\downarrow$
---|---|---|---|---
Short | Long | Short | Long | Short | Long | Short | Long
No Collaboration | 0.219 | 0.125 | 0.570 | 0.631 | 0.195 | 0.204 | 0.182 | 0.190
Early Collaboration | 0.641 | 0.437 | 0.382 | 0.430 | 0.160 | 0.169 | 0.065 | 0.083
Late Collaboration | 0.506 | 0.303 | 0.416 | 0.423 | 0.212 | 0.213 | 0.101 | 0.103
Intermediate Collaboration (Fully Connected) | V2X-ViT [31] | 0.638 | 0.434 | 0.410 | 0.457 | 0.240 | 0.208 | 0.094 | 0.116
V2VNet [30] | 0.631 | 0.433 | 0.403 | 0.453 | 0.184 | 0.190 | 0.088 | 0.108
Intermediate Collaboration (Partially Connected) | Who2com [11] | 0.461 | 0.291 | 0.522 | 0.580 | 0.188 | 0.196 | 0.116 | 0.135
When2com [12] | 0.137 | 0.063 | 0.495 | 0.513 | 0.185 | 0.189 | 0.222 | 0.237
Where2comm [13] | 0.634 | 0.424 | 0.398 | 0.447 | 0.189 | 0.196 | 0.118 | 0.136
RS-DFM (Ours) | 0.662 | 0.443 | 0.374 | 0.413 | 0.170 | 0.176 | 0.109 | 0.123

####  IV-E2 Benchmark Comparison in BEV Instance Segmentation

As presented in Tab. II, our method demonstrates the top performance in
instance segmentation across all evaluation metrics among collaborative
approaches. Compared with the second-best early collaboration approach, which
processes multi-view raw observations in a centralized manner, our RS-DFM
improves the IoU by 2.8–3.1% and PQ by 2.4–5.1% through feature-level
interactions, facilitated by our designed GBG and HLFDC modules. Unlike the
intricate task of 3D object detection, which considers orientation, instance
segmentation focuses on recognizing and segmenting objects to reflect recall
and localization accuracy. Attributed to our GBG module improving feature
mapping from 2D to BEV, our RS-DFM surpasses prevalent fully connected
collaboration approaches by 3–4.5% in IoU and 1.8–5% in PQ. In contrast to
previous sparse collaborative perception methods that reduce transmission
costs at the expense of performance, our method outperforms these baselines by
35–75% in IoU and 42–63% in PQ.

TABLE II: A comparison of collaboration baselines for BEV instance
segmentation.

Models | IoU (%) $\uparrow$ | PQ (%) $\uparrow$ | SQ (%) $\uparrow$ | RQ (%) $\uparrow$
---|---|---|---|---
Short | Long | Short | Long | Short | Long | Short | Long
No Collaboration | 37.02 | 18.74 | 31.59 | 17.06 | 72.14 | 71.60 | 43.78 | 23.82
Early Collaboration | 62.15 | 29.66 | 49.23 | 26.67 | 74.13 | 73.63 | 66.41 | 36.22
Late Collaboration | 53.66 | 26.10 | 31.12 | 17.41 | 67.65 | 67.54 | 46.01 | 25.78
Intermediate Collaboration (Fully Connected) | V2X-ViT [31] | 62.10 | 29.84 | 49.74 | 26.49 | 73.87 | 73.19 | 67.33 | 36.20
V2VNet [30] | 61.68 | 30.25 | 48.60 | 26.83 | 73.47 | 72.98 | 66.15 | 36.77
Intermediate Collaboration (Partially Connected) | Who2com [11] | 47.10 | 21.73 | 35.89 | 18.76 | 70.54 | 70.41 | 50.88 | 26.65
When2com [12] | 36.45 | 18.40 | 31.17 | 16.85 | 71.34 | 70.73 | 43.69 | 23.83
Where2comm [13] | 51.63 | 25.69 | 35.60 | 19.68 | 67.75 | 67.50 | 52.54 | 29.15
RS-DFM (Ours) | 63.92 | 31.17 | 51.04 | 27.30 | 75.47 | 74.76 | 67.63 | 36.51

####  IV-E3 Benchmark Comparison in Trajectory Prediction

As presented in Tab. III, it is challenging to predict objects’ future
trajectories solely based on single-view observation, and the performance of
future instance segmentation and motion prediction significantly improves with
the help of any collaborative methods. Distinct from observation patterns of
single-frame instance segmentation, previous collaborative approaches, such as
early collaboration, V2XViT, and V2VNet, exhibit comparative performance with
our method in the short range. This suggests that temporal information can
mitigate the estimation error of feature mapping. However, the gap between our
method and previous collaborative methods becomes evident in the long-range
setting. Our RS-DFM outperforms early collaboration by 2.5% in IoU and 2% in
VPQ. Compared to fully connected collaborative approaches, our method
surpasses V2X-ViT by 3.8% in IoU and 5.7% in VPQ, and V2VNet by 17.8% in IoU
and 12.8% in VPQ. Notably, even with the same transmission ratio as the
partially connected baseline Where2comm, our method significantly outperforms
it by 10.8% in IoU and by 42.4% in VPQ, showcasing the effectiveness of our
HLDFC module.

TABLE III: A comparison of collaboration baselines for trajectory prediction.

Models | IoU (%) $\uparrow$ | VPQ (%) $\uparrow$
---|---|---
Short | Long | Short | Long
No Collaboration | 25.05 | 13.43 | 11.51 | 5.92
Early Collaboration | 60.70 | 34.77 | 50.07 | 29.70
Late Collaboration | 58.23 | 31.48 | 46.42 | 25.91
Intermediate Collaboration (Fully Connected) | V2X-ViT [31] | 59.74 | 34.32 | 49.44 | 28.64
V2VNet [30] | 60.68 | 30.25 | 48.60 | 26.83
Intermediate Collaboration (Partially Connected) | Who2com [11] | 50.26 | 27.65 | 40.75 | 27.65
When2com [12] | 45.41 | 25.66 | 36.26 | 16.85
Where2comm [13] | 58.18 | 32.18 | 26.20 | 21.26
RS-DFM (Ours) | 61.02 | 35.65 | 50.05 | 30.27

###  IV-F Ablation Studies

####  IV-F1 Effectiveness of Proposed Modules

Our RS-DFM collaborative framework introduces two innovative components: the
GBG and HLFDC modules. We evaluate these modules based on their ability to
enhance downstream tasks in the long-range setting, including 3D object
detection, BEV instance segmentation, and trajectory prediction, as well as
their ability to optimize the performance-transmission trade-off, as depicted
in Tab. IV. The GBG-only variant significantly enhances 3D object detection
accuracy, with improvements of approximately 4% in mAP and a reduction in
position deviation by about 7.2% in mATE. This is primarily attributable to
the more accurate BEV representations guided by the generalized feature
mapping. Notably, while the GBG module improves mAP, it may lead to
orientation and scale errors, as observed in the benchmark comparisons
mentioned previously. This phenomenon merits further investigation. For
instance segmentation, the GBG-only variant improves segmentation and
recognition quality by 1.5-5.4%. In trajectory prediction, this variant also
shows improvements of 3.5% in IoU and 2.6% in VPQ. The evaluation metrics for
these tasks are highly related to position precision, thereby demonstrating
the effectiveness of our designed GBG module. On the other hand, the HLFDC-
only variant reduces transmission costs by 43.8%, with only a marginal
performance decrease of less than 1% in mAP, IoU, and VPQ relative to the
baseline. This indicates the validity of the high-low frequency decoupled
collaboration, albeit with minor performance trade-offs. Overall, our RS-DFM,
integrated with both GBG and HLFDC modules, achieves a balance between
collaborative perception and transmission efficiency.

TABLE IV: The ablation study of proposed modules. ’Ratio’ in the table refers
to the transmission ratio during collaborative perception.

GBG | HLFDC | 3D Object Detection | BEV Instance Segmentation | Trajectory Prediction | Ratio
---|---|---|---|---|---
mAP $\uparrow$ | mATE $\downarrow$ | mASE $\downarrow$ | mAOE $\downarrow$ | IoU (%) $\uparrow$ | PQ (%) $\uparrow$ | SQ (%) $\uparrow$ | RQ (%) $\uparrow$ | IoU (%) $\uparrow$ | VPQ (%) $\uparrow$
$\times$ | $\times$ | 0.437 | 0.430 | 0.169 | 0.083 | 29.66 | 26.67 | 73.63 | 36.22 | 34.77 | 29.70 | 1
✓ | $\times$ | 0.455 | 0.399 | 0.172 | 0.135 | 31.24 | 27.65 | 75.24 | 36.74 | 35.97 | 30.46 | 1
$\times$ | ✓ | 0.433 | 0.443 | 0.189 | 0.105 | 29.64 | 26.05 | 72.81 | 35.77 | 34.59 | 29.35 | 0.563
✓ | ✓ | 0.443 | 0.413 | 0.176 | 0.123 | 31.17 | 27.30 | 74.76 | 36.51 | 35.65 | 30.27 | 0.563

####  IV-F2 Effect of window size in HLFDC module

TABLE V: A comparison of performance metrics for different window sizes
implemented in the HLFDC module.

Window Size | 1 | 2 | 4 | 8 | 16 | 32
---|---|---|---|---|---|---
Det-mAP | 0.458 | 0.447 | 0.443 | 0.432 | 0.425 | 0.402
Seg-IoU | 31.21 | 30.79 | 31.17 | 30.58 | 30.23 | 30.05
Pred-IoU | 35.78 | 35.60 | 35.65 | 35.34 | 34.90 | 34.73
Ratio | 1.000 | 0.625 | 0.531 | 0.508 | 0.502 | 0.500

The window size determines the contextual range in each local window and the
compression ratio for low-frequency information. Smaller windows are more
sensitive to capturing high-frequency local feature variations but may be
susceptible to noise, and result in a considerable transmission expense for
collaborative interactions. Conversely, larger windows encompass a broader
range of textual information within the local windows but may potentially lose
significant information for low-frequency global attention processing. After
investigating the effects of various window sizes across different downstream
tasks, as shown in Table V, we select a 4$\times$4 window size in the HLFDC
module for optimal performance.

###  IV-G Visualization

####  IV-G1 Collaborative 3D object detection

Fig.6 illustrates that collaborative perception significantly enhances the
detection of occluded and out-of-range objects through inter-platform
interactions compared with no collaboration. For instance, UAV1 hardly
recognizes any objects due to significant occlusion in its single
observations. However, collaborative approaches successfully recall these
occluded objects by leveraging complementary information from other platforms.
Additionally, this collaboration alleviates the challenge of detecting small
objects at high altitudes, as demonstrated by the row of UAV5. Intuitively,
our method exhibits superior detection performance compared to other
approaches. In particular, in the red circles in the rows corresponding to
UAV3, UAV4, and UAV5, early collaboration and Where2comm recall numerous
erroneous bounding boxes, whereas our method significantly reduces such
errors. This improvement is attributed to our specially designed GBG module,
which achieves better feature mapping by incorporating geometric priors.

####  IV-G2 Collaborative BEV instance segmentation

The task of BEV instance segmentation focuses on segmenting the BEV
representation into small object bricks, where accuracy highly relies on
precise mapping from 2D observations to the BEV. Our method is obviously
better at accurately segmenting distant objects, typically positioned at the
edges of the BEV, as depicted by the two small red circles in Figure 7. While
early collaboration can centralize the processing of raw observations, it
still performs less effectively than our approach, which leverages sparse
feature-level interactions. This suggests that precise feature mapping holds
more significance than transmitting a larger quantity of features.

####  IV-G3 Collaborative trajectory prediction

Trajectory prediction heavily depends on inter-frame continuity. Previous
sparse collaborative approaches often convey spatially discrete features,
potentially disrupting this continuity. However, our method maintains the
overall structure of intermediate BEV features and extracts critical
information through high and low-frequency decoupling. As a result, our method
achieves comparable performance to early collaboration and fully-connected
intermediate collaboration paradigms. Leveraging our specially designed GBG
and HLFDC modules, our method accurately forecasts the trajectories of
multiple objects in challenging intersections and achieves more precise
segmentation and prediction results than other well-known baselines. These
findings are consistent with our quantitative evaluations.

##  V Conclusion

We proposed a novel distributed remote sensing foundation model, RS-DFM. The
core idea is to utilize the geometric priors of oblique perspectives to
generalize and map features from different spatial observations into a unified
feature space, facilitating subsequent multi-task perception. Simultaneously,
we decouple high-frequency and low-frequency features in remote sensing and
perform selective information compression to minimize the loss of critical
information during compression. Comprehensive experiments covering multi-
altitude, multi-perspective inputs, and multi-task outputs demonstrate that
RS-DFM achieves an excellent balance between perception performance and
communication bandwidth in multi-platform collaborative online inference.

Figure 6: Visualization of collaborative 3D object detection. Red circles
highlight the areas where other baselines make mistaken predictions.
Figure 7: Visualization of collaborative instance segmentation. Each instance
is allocated a distinct color. Red circles highlight the areas where other
baselines make mistaken predictions.

Figure 8: Visualization of collaborative trajectory prediction. Each instance
is allocated a distinct color, and its predicted trajectory is represented
with the same color and slight transparency. Red circles highlight the areas
where other baselines make mistaken predictions.

## References

  * [1] Z. Yan, P. Wang, F. Xu, X. Sun, and W. Diao, “Air-pv: A benchmark dataset for photovoltaic panel extraction in optical remote sensing imagery,” 2023.
  * [2] X. Sun, Y. Tian, W. Lu, P. Wang, R. Niu, H. Yu, and K. Fu, “From single-to multi-modal remote sensing imagery interpretation: A survey and taxonomy,” Science China Information Sciences, vol. 66, no. 4, p. 140301, 2023.
  * [3] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
  * [4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  * [5] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000–16009, 2022.
  * [6] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, “Simmim: A simple framework for masked image modeling,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9653–9663, 2022.
  * [7] X. Sun, P. Wang, W. Lu, Z. Zhu, X. Lu, Q. He, J. Li, X. Rong, Z. Yang, H. Chang, et al., “Ringmo: A remote sensing foundation model with masked image modeling,” IEEE Transactions on Geoscience and Remote Sensing, 2022.
  * [8] K. Cha, J. Seo, and T. Lee, “A billion-scale foundation model for remote sensing images,” arXiv preprint arXiv:2304.05215, 2023.
  * [9] Z. Yan, J. Li, X. Li, R. Zhou, W. Zhang, Y. Feng, W. Diao, K. Fu, and X. Sun, “Ringmo-sam: A foundation model for segment anything in multimodal remote-sensing images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–16, 2023.
  * [10] Y. Wang, T. Zhang, L. Zhao, L. Hu, Z. Wang, Z. Niu, P. Cheng, K. Chen, X. Zeng, Z. Wang, et al., “Ringmo-lite: A remote sensing lightweight network with cnn-transformer hybrid framework,” IEEE Transactions on Geoscience and Remote Sensing, 2024.
  * [11] Y.-C. Liu, J. Tian, C.-Y. Ma, N. Glaser, C.-W. Kuo, and Z. Kira, “Who2com: Collaborative perception via learnable handshake communication,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 6876–6883, IEEE, 2020.
  * [12] Y.-C. Liu, J. Tian, N. Glaser, and Z. Kira, “When2com: Multi-agent perception via communication graph grouping,” in Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 4106–4115, 2020.
  * [13] Y. Hu, S. Fang, Z. Lei, Y. Zhong, and S. Chen, “Where2comm: Communication-efficient collaborative perception via spatial confidence maps,” Advances in neural information processing systems, vol. 35, pp. 4874–4886, 2022.
  * [14] Y. Hu, Y. Lu, R. Xu, W. Xie, S. Chen, and Y. Wang, “Collaboration helps camera overtake lidar in 3d detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9243–9252, 2023.
  * [15] Y. Lu, Y. Hu, Y. Zhong, D. Wang, S. Chen, and Y. Wang, “An extensible framework for open heterogeneous collaborative perception,” arXiv preprint arXiv:2401.13964, 2024.
  * [16] Y. Hu, J. Peng, S. Liu, J. Ge, S. Liu, and S. Chen, “Communication-efficient collaborative perception via information filling with codebook,” arXiv preprint arXiv:2405.04966, 2024.
  * [17] S. Duan, P. Cheng, Z. Wang, Z. Wang, K. Chen, X. Sun, and K. Fu, “Mdcnet: A multi-platform distributed collaborative network for object detection in remote sensing imagery,” IEEE Transactions on Geoscience and Remote Sensing, 2024.
  * [18] T. Zhang, Z. Wang, P. Cheng, G. Xu, and X. Sun, “Dcnnet: a distributed convolutional neural network for remote sensing image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–18, 2023\.
  * [19] J. W. Cooley, P. A. W. Lewis, and P. D. Welch, “The fast fourier transform and its applications,” IEEE Transactions on Education, p. 27–34.
  * [20] G. Deng and L. Cahill, “An adaptive gaussian filter for noise reduction and edge detection,” in 1993 IEEE Conference Record Nuclear Science Symposium and Medical Imaging Conference, Aug 2005.
  * [21] N. Park and S. Kim, “How do vision transformers work?,”
  * [22] E. Voigtman and J. D. Winefordner, “Low‐pass filters for signal averaging,” Review of Scientific Instruments, p. 957–966, May 1986.
  * [23] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in Conference on robot learning, pp. 1–16, PMLR, 2017.
  * [24] O. L. De Carvalho, O. A. de Carvalho Júnior, A. O. de Albuquerque, N. C. Santana, and D. L. Borges, “Rethinking panoptic segmentation in remote sensing: A hybrid approach using semantic segmentation and non-learning methods,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2022.
  * [25] T. Fernando, C. Fookes, H. Gammulle, S. Denman, and S. Sridharan, “Towards on-board panoptic segmentation of multispectral satellite images,” IEEE Transactions on Geoscience and Remote Sensing, 2023.
  * [26] P. Li, S. Ding, X. Chen, N. Hanselmann, M. Cordts, and J. Gall, “Powerbev: a powerful yet lightweight framework for instance prediction in bird’s-eye view,” arXiv preprint arXiv:2306.10761, 2023.
  * [27] J. Huang, G. Huang, Z. Zhu, Y. Ye, and D. Du, “Bevdet: High-performance multi-camera 3d object detection in bird-eye-view,” arXiv preprint arXiv:2112.11790, 2021.
  * [28] A. Hu, Z. Murez, N. Mohan, S. Dudas, J. Hawke, V. Badrinarayanan, R. Cipolla, and A. Kendall, “Fiery: Future instance prediction in bird’s-eye view from surround monocular cameras,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15273–15282, 2021.
  * [29] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012–10022, 2021.
  * [30] T.-H. Wang, S. Manivasagam, M. Liang, B. Yang, W. Zeng, and R. Urtasun, “V2vnet: Vehicle-to-vehicle communication for joint perception and prediction,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pp. 605–621, Springer, 2020.
  * [31] R. Xu, H. Xiang, Z. Tu, X. Xia, M.-H. Yang, and J. Ma, “V2x-vit: Vehicle-to-everything cooperative perception with vision transformer,” in European conference on computer vision, pp. 107–124, Springer, 2022.