# Single-Model and Any-Modality for Video Object Tracking

Zongwei Wu1 Jilai Zheng2 Xiangxuan Ren2 Florin-Alexandru Vasluianu 1
Chao Ma2 Danda Pani Paudel3,4 Luc Van Gool3,4 Radu Timofte1
1 Computer Vision Lab, CAIDAS & IFI, University of Wurzburg 2 AI Institute,
Shanghai Jiao Tong University
3 CVL, ETH Zurich 4 INSAIT, Sofia University  Corresponding Author

###### Abstract

In the realm of video object tracking, auxiliary modalities such as depth,
thermal, or event data have emerged as valuable assets to complement the RGB
trackers. In practice, most existing RGB trackers learn a single set of
parameters to use them across datasets and applications. However, a similar
single-model unification for multi-modality tracking presents several
challenges. These challenges stem from the inherent heterogeneity of inputs –
each with modality-specific representations, the scarcity of multi-modal
datasets, and the absence of all the modalities at all times. In this work, we
introduce Un-Track, a Unified Tracker of a single set of parameters for any
modality. To handle any modality, our method learns their common latent space
through low-rank factorization and reconstruction techniques. More
importantly, we use only the RGB-X pairs to learn the common latent space.
This unique shared representation seamlessly binds all modalities together,
enabling effective unification and accommodating any missing modality, all
within a single transformer-based architecture and without the need for
modality-specific fine-tuning. Our Un-Track achieves +8.1 absolute F-score
gain, on the DepthTrack dataset, by introducing only +2.14 (over 21.50) GFLOPs
with +6.6M (over 93M) parameters, through a simple yet efficient prompting
strategy. Extensive comparisons on five benchmark datasets with different
modalities show that Un-Track surpasses both SOTA unified trackers and
modality-specific finetuned counterparts, validating our effectiveness and
practicality. The source code is publicly available at
https://github.com/Zongwei97/UnTrack.

##  1 Introduction

Figure 1: Motivation: Traditional multimodal trackers [71, 93] use multiple
domain-specific parameter sets. Our unified tracker with a single parameter
set seamlessly integrates any modality (of RGB-X).

Video object tracking [88, 13, 10, 92] is a fundamental task in computer
vision with wide-ranging applications spanning from surveillance [91] to
augmented reality [24], where accuracy and robustness are paramount. While
traditional RGB trackers have shown promise in general settings, they often
struggle with challenging scenarios such as occlusions [48, 41], low
visibility [94, 85], or fast-moving objects [11, 49]. For more reliable
tracking under such challenging conditions, the integration of auxiliary
modalities (X) like depth [21, 70], thermal [85, 89], and event [81, 100] have
proven effective in multimodal tracking.

While the idea of fusing RGB with other modalities holds promise [86, 84], the
main challenge is the discrepancy in the representation of information across
different modalities. Despite the success of previous fusion works [16, 61]
tailored for each specific scenario to improve RGB trackers, their reliance on
modality-specific designs limits adaptability. Recent initiatives [71, 93]
towards a uniform architecture for various modalities show promise but
necessitate modality-specific fine-tuning. This approach leads to multiple
parameter sets, as shown in Fig. 1(a), thereby compromising practicality in
diverse real-world applications.

In this work, we aim to avoid such modality-specific fine-tuning to keep only
one model-parameter set at all times. Another practical constraint arises from
the differences in available auxiliary modalities across settings. Unifying
modalities by a common representation can handle any modality at its disposal,
addressing the mentioned problems by its virtue. However, two additional
multifaceted challenges emerge from the scarcity of multimodal datasets and
the absence of all paired data combinations. The former makes cross-modal
mappings through a large-scale data prior unfeasible [20, 40, 78], while the
latter leads to missing modalities and renders joint learning using all
possible combinations of paired data unfeasible [82, 42].

To achieve such a unification, in this paper, we present Un-Track, denoting
“one” in French, which learns a cohesive embedding across diverse input
modalities. Unlike previous approaches [82, 75, 59, 52], Un-Track relies
solely on RGB-X pairs for training, with X representing auxiliary modalities,
without the need for all modalities to co-occur. Our objective is to discover
a shared embedding seamlessly binding all auxiliary modalities (as depicted in
Fig. 1(b)). More specifically, we leverage the factorization prior, allowing
reasoning about a common embedding directly from the low-rank latent space.
Factorization is a simple composition prior with the assumption that the
global approximation can be constructed from a set of subset vectors. The
factorization prior, successfully utilized in previous studies [5, 25, 1], is
employed in our work to reconstruct a shared embedding. This process
transforms the heterogeneous modal representation into a uniform one, thereby
facilitating the emergent cross-modal alignment.

Moreover, to harness the full potential of auxiliary inputs while maintaining
efficiency, we leverage cross-modal features as prompts to enable RGB-X
interaction. Different from previous works [28, 26], our goal is to enhance
less reliable tokens, defined by a learnable score function, using multimodal
cues. We approach this as a token recovery problem and leverage low-rank
factorization to achieve the goal, which is first suggested in multimodal
fusion, to the best of our knowledge.

With its unified model architecture and prompting blocks, Un-Track is the
first to offer support for cross-modality alignment under a single
architecture with uniform parameters. In comparison to our RGB baseline with
21.50 GFLOPs and 92M parameters, Un-Track introduces only +2.14 GFLOPs with
+6.6M parameters, resulting in a significant +8.1 absolute F-score gain
demonstrated on the DepthTrack dataset. Extensive comparisons across five
datasets with different modalities validate Un-Track’s superiority over
specialized state-of-the-art (SOTA) models, surpassing both unified trackers
and modality-specific fine-tuned counterparts with a substantial margin.

##  2 Related Works

Multimodal tracking: Video object tracking [8, 73] aims to detect the bounding
box of objects in a video sequence based on their initial positions. Early
approaches treated tracking as a per-frame target-matching problem, with
Siamese networks [76, 33, 34, 51, 63, 87] being a notable example. More
recently, transformer-based methods [65, 7, 6, 55, 3, 12] have gained
popularity for feature extraction and per-frame correlation in tracking.
Large-scale training datasets [43, 15, 23, 46] have empowered RGB trackers to
uniformly apply parameters across various applications. While RGB trackers
deliver promising results, challenges such as occlusion, low illumination, and
fast-moving scenes have led to the exploration of additional modalities.
Several works have investigated the role of depth [97, 72], thermal [38, 39],
and event modalities [50, 56] in enhancing tracking performance. Specifically,
depth cues [69, 17] contribute to handling objects with certain
characteristics. The use of thermal cameras [57, 61] addresses specific
tracking challenges such as low illumination or transparent objects.
Integrating event cameras improves the temporal awareness [99, 80, 58].

Despite the plausible achievements, many rely on modality-specific blocks
designed for individual modalities [90, 95], limiting their adaptability.
Recent efforts have focused on achieving architectural unification [71, 93],
yet they still necessitate modality-specific fine-tuning, resulting in
distinct parameter sets for different modalities. The ideal scenario would
involve a large-scale dataset encompassing all possible modal combinations,
but current tracking datasets predominantly feature a single modality — depth
[96, 68], thermal [36, 37], or event [56] — posing challenges for a unified
model with a single parameter set.

Learning with Missing modalities: Recent research has addressed real-world
scenarios where models must cope with missing modalities [42, 45]. One common
strategy involves estimating missing values by learning joint multimodal
representations [32, 77], feasible when complete samples are available during
training. However, tracking datasets typically exhibit only one modality at a
time, complicating the learning of such joint representations. Other works
[40, 20] implicitly learn cross-modal alignment end-to-end using large-scale
datasets and deep networks, demanding substantial computational resources.
Extending such approaches to tracking is challenging due to limited downstream
datasets and real-world applicability constraints [66, 27]. In contrast to
existing methods, our approach investigates cross-modal relationships by
leveraging edge priors to learn a joint representation that unifies all
modalities. Our method does not require the simultaneous occurrence of all
modalities, offering a unique perspective on dealing with diverse and
individual modalities.

##  3 Methods

Figure 2: Our proposed framework, termed Un-Track, is composed of a shared
embedding, a modal prompting, and a LoRA-finetuned pretrained RGB tracker. The
shared embedding learns a joint representation that unifies all modalities
(Sec. 3.2). The modal prompting block enhances feature modeling with modal
awareness at each scale (Sec. 3.3). To track the target object, we finetune a
pretrained foundation model [74] using the LoRA technique (Sec. 3.4). Our
model achieves a unified model applicable across different modalities under a
single parameter set. During inference, Un-Track seamlessly integrates any
image-paired data, thanks to the emergent alignment.

###  3.1 Overall Framework

In this paper, our primary focus is on multimodal tracking, with the
constraint that only one modality is available at a time, as shown in Fig. 2.
We define our multimodal tracking dataset as $M=\\{M^{D},M^{T},M^{E}\\}$,
where $M^{X}$ represents the subset dataset with only a single modality $X$
available. Conventional methods tackling missing modalities often use dummy
inputs for the absent pixels [32, 54], simulating complete datasets with all
possible combinations of paired datasets. In contrast, our method transforms
any auxiliary input into a shared embedding, seamlessly binding all modalities
together and creating a complete and paired representation with the master RGB
feature.

To mitigate overfitting on sparse downstream multimodal datasets, we adopt a
transformer-based RGB tracker with frozen parameters and fine-tune it for
multimodal tracking. Leveraging a lightweight outer prompting method, we
identify uncertain tokens at each scale and enhance them with cross-modal
awareness. Simultaneously, an inner fine-tuning process is implemented using
the LoRA technique [22].

During training, our model learns the shared embedding from samples in the
mixed dataset $M$, effectively binding all modalities together. As for
inference, our model can accommodate any modal input $X$, thanks to the
emergent alignment. Our trainable parameters only include cross-modal binding,
outer prompting parameters, and inner LoRA parameters, ensuring a training-
friendly pipeline that can be efficiently employed end-to-end on a single 24G
GPU.

###  3.2 Shared Embedding

Explicit Edge Awareness: We observe that, as illustrated in Fig. 2(a), depth
data introduce 3D distance information, effectively delineating objects with
varying granularity and enhancing the sharpness of 3D boundaries; thermal
images generate a scene heat map, highlighting objects based on their
temperatures and providing clearer contours; event data capture intensity
changes, particularly around an object’s outbound region. Notably, a
consistent feature emerges across these modalities: the representation of the
“true” objects’ shape, often manifested through edges.

Motivated by this observation, our objective is to harness edge embedding to
unify all modalities. To achieve this, as shown in Fig. 3, we generate
gradient maps from auxiliary modalities by computing differences between
neighboring pixels along both the x- and y-axes. Simultaneously, without
compromising texture edge, we also generate RGB gradient maps. Subsequently,
all gradient maps are integrated with the visual feature, forming the gradient
feature $G$.

Implicit Low-Rank Reconstruction: While edges present a shared feature across
different modalities, exclusively transforming all modalities into edges may
risk overlooking modality-specific clues. Therefore, we introduce an implicit
learning pipeline to complement this by discovering the shared embedding,
guided by the previously generated explicit edge awareness. This combined
approach allows for a more effective identification of the shared embedding,
leveraging both data-driven learning and edge priors.

In technical terms, we redefine the challenge of learning the shared embedding
as a quest for the shared low-rank approximation. Both objectives share the
essence of distilling common features across all modalities. However, direct
estimation of the low-rank approximation becomes impractical due to the
distinct data domains and modal representations. In response, we propose a
pragmatic strategy: decomposing the shared low-rank vector into the low-rank
of each subset component. This alternative, more manageable and feasible
within a single domain, lays the groundwork for approximating the global
shared low-rank from these individual low-rank components.

The overall algorithm can be found in Algorithm 1. Specifically, let $M$ be
the input feature with mixed auxiliary modalities, decomposed into subset
features $D,T,E$ representing depth, thermal, and event samples from subset
datasets ${M^{D},M^{T},M^{E}}$. Their respective $k_{th}$ low-rank matrices
$D_{k},T_{k},E_{k}$, are approximated by:

$D_{k}=\sigma_{d}(D),\quad T_{k}=\sigma_{t}(T),\quad E_{k}=\sigma_{e}(E),$ (1)

where $\sigma_{x}$ is the modality-specific learning through a simple MLP
projecting features from input channel $c$ to the low-rank space $k$ ($k<c$).
Simultaneously, we compute the low-rank matrix $G_{k}$ from the gradient
feature $G$.

Algorithm 1 Implicit Shared Embedding

Input: $M$ \- Mix Modal Feature, $G$ \- Explicit Gradient Binding.

Output: $F$ \- Reconstructed feature

1\. Initialize: Separate $M$ into subsets, each representing a modality-
specific feature: $D,T,E$.

2\. In-Domain Approximation: Train individual approximators $\sigma_{x}$ for
each modality $x$ to derive domain-specific low-rank matrices
$D_{k},T_{k},E_{k}$. Idem for explicit low-rank gradient $G_{k}$.

3\. Fuse and Guide: Merge $D_{k},T_{k},E_{k}$ through a fusion function
$\varphi_{1}$. Incorporate the explicit gradient $G_{k}$ using addition, after
projecting it into a compatible latent space through $\varphi_{2}$.

4\. Reconstruct: Train a reconstruction function $\Phi_{R}$ to construct the
shared embedding $F$, considering the explicit guidance $G$ in the original
feature space.

Figure 3: Shared Embedding. We derive a joint representation through low-rank
factorization and reconstruction. Such an implicit learning is additionally
integrated with explicit edge awareness to enhance the embedding.

The global shared low-rank matrix $M_{k}$ is then approximated by fusing the
subset low-rank matrices $D_{k},T_{k},E_{k}$, along with the gradient
guidance. Technically, we concatenate $D_{k},T_{k},E_{k}$ and learn the joint
low-rank approximation, incorporating it with the gradient guidance. This
pipeline can be expressed as follows:

$M_{k}=\varphi_{R_{1}}([D_{k},T_{k},E_{k}])+\varphi_{R_{2}}(G_{k}),$ (2)

where $[.]$ is the channel concatenation and $\varphi_{R_{i}}$ are the MLP
projections to the low-rank latent space. Finally, we reconstruct the shared
embedding $F$ through:

$F=\Phi_{R}(M_{k})+G,$ (3)

where $\Phi_{R}$ is another MLP that projects the jointly-learned low-rank
matrix back to the departing feature space. Our ablation studies validate that
this subset low-rank approximation and regrouping efficiently unify all input
modalities, despite their heterogeneous representations.

Figure 4: Modal Prompting. For the visual feature $I$, we employ a score
function to categorize tokens into negative, uncertain, and positive segments.
Using a token exchange policy, we discard negative tokens, enhance uncertain
ones with corresponding tokens from $M$, and retain positive ones. Then, we
transform the feature fusion task into a token recovery problem, addressed by
low-rank factorization. Similarly, we extract the most informative low-rank
matrix from $M$ to fuse and reconstruct the shared output.

###  3.3 Outer Modal Prompting

RGB-tracker may fail to perform accurately in corner cases where auxiliary
clues can contribute. Drawing inspiration from the success of adapting large
pre-trained models to specific downstream tasks [22, 26], we introduce a modal
prompting method devised to enhance RGB token $I$ with modality-awareness $F$,
as shown in Fig. 4.

Specifically, our approach employs a shrinkage token fusion strategy. Taking
$I$ as an example, we categorize tokens into three groups—negative, uncertain,
and positive—based on a dynamic scoring function $s$. These regions are
defined with mask form, expressed as $m_{n},m_{u},m_{p}=s(I)$. To harness
multimodal clues effectively, we replace negative tokens with those from the
other modality, omit uncertain ones with dummy values, and retain the positive
tokens. Subsequently, these modified tokens undergo projection into the low-
rank space using the approximation function $\sigma_{c}$. Our objective is to
enhance robustness by completing uncertain tokens with information from
reliable neighboring tokens. Mathematically, we obtain the first low-rank
matrix $I_{l_{1}}$ by:

$I_{l_{1}}=\sigma_{c}(m_{n}\cdot F+m_{p}\cdot I).$ (4)

Next, we target the uncertain tokens by merging them with the paired tokens
from the other modality and approximate the low-rank matrix similarly. Here,
we aim to throw out the possible noise, resulting in a matrix that is more
informative than the original. Let $\sigma_{n}$ be another approximation
function, we obtain the second low-rank matrix $I_{l_{n}}$ by:

$I_{l_{2}}=\sigma_{n}(m_{u}\cdot F+m_{u}\cdot I).$ (5)

Then, we fuse these two low-rank matrices in low-rank space, forming the
shared low-rank matrix $I_{l}$ for input $I$. In such a manner, we improve the
image token modeling by fully benefiting from the auxiliary clues.
Mathematically, the whole process can be formulated as:

$I_{l}=\varphi_{P}([I_{l_{1}},I_{l_{2}}]),$ (6)

where $\varphi_{P}$ is learnable fusion. Similarly, from the input $F$, we
follow the same process to obtain the low-rank matrix $F_{l}$. For the cross-
modal fusion, we add these two low-rank matrices and then project back to the
original space. We can obtain the fused output $O$ by:

$O=\Phi_{P}(I_{l}+F_{l}),$ (7)

where $\Phi_{P}$ is another learnable MLP.

Our method can be regarded as a mixer of token exchange (for negative tokens)
and token fusion (for uncertain tokens), while retaining the most informative
modality-specific clues (for positive tokens). As the majority of fusion
operations occur in the low-rank feature space, our progressive cross-modal
shrinkage avoids imposing a significant additional computational burden, while
being able to excavate and accumulate the rich cues from each modality for
effective modal prompting.

###  3.4 Inner Finetuning

In addition to the outer modal prompting, we incorporate the LoRA technique
[22] for more efficient finetuning. For each transformer attention module with
the weight matrix $W_{0}\in\mathbb{R}^{d\times k}$, we introduce two learnable
matrices: $B\in\mathbb{R}^{d\times r}$ and $A\in\mathbb{R}^{r\times k}$. This
leads to the replacement of the frozen attention mechanism
$h=W_{0}\displaystyle{\bm{x}}$ with the new LoRA attention:

$h=W_{0}{\displaystyle{\bm{x}}}+BA{\displaystyle{\bm{x}}}.$ (8)

To train our network, we adopt the same loss functions as our baseline tracker
[74] for end-to-end learning.

| Finetuned Depth-Specific Parameters | Single Set of Parameters
---|---|---
|  | ATCAIS
---
[29]
| DDiMP
---
[29]
| DeT
---
[68]
| SPT
---
[96]
| ProTrack
---
[71]
| ViPT
---
[93]
| Un-Track
---
(ours)
| Stark
---
[64]
| AiATrack
---
[18]
| OSTrack
---
[74]
| UniNext
---
[67]
| SeqTrack
---
[9]
| ViPT
---
[93]
| Un-Track
---
(ours)
F-score($\uparrow$) | 0.476 | 0.485 | 0.532 | 0.538 | 0.578 | 0.594 | 0.612 | 0.397 | 0.515 | 0.569 | 0.422 | 0.590 | 0.561 | 0.610
Re($\uparrow$) | 0.455 | 0.469 | 0.506 | 0.549 | 0.573 | 0.596 | 0.610 | 0.406 | 0.526 | 0.582 | 0.432 | 0.600 | 0.562 | 0.608
Pr($\uparrow$) | 0.500 | 0.503 | 0.560 | 0.527 | 0.583 | 0.592 | 0.613 | 0.388 | 0.505 | 0.557 | 0.413 | 0.580 | 0.560 | 0.611

Table 1:  Overall performance on DepthTrack test set [68]. Red/Green/Blue
stands for the best/second/third performance. Our fine-tuned model sets the
SOTA record, while our uni-modal with single parameters set also outperforms
previous fine-tuned SOTA.

##  4 Experiments

###  4.1 Training Data

In the absence of a comprehensive multi-modal tracking dataset encompassing
all possible combinations (RGB-D-T-E), our training is conducted solely on
RGB+X paired data, comprising RGB-D, RGB-T, and RGB-E combinations. Different
from previous approaches that fine-tune separate sets of parameters for each
modality, Un-Track is trained on all modalities with a unified set of
parameters.

Specifically, our RGB-D samples are sourced from DepthTrack [68], a pioneering
RGB-D tracking benchmark with 150 training long-term sequences. RGB-T samples
are extracted from the extensive LasHeR [37] dataset, featuring 979 diverse
training sequences. RGB-E samples are obtained from VisEvent [56], which
boasts 500 real-world sequences. For each input depth, thermal, or event, we
transform them into an RGB-like form.

###  4.2 Within distribution Evaluation

Given that DepthTrack [68], Lasher [37], and VisEvent [56] provide domain-
specific testing sequences, our initial evaluation focuses on these within
distribution sequences. For each dataset, we adhere to the metrics specified
in the original papers and prior standards [71, 93] for evaluation.

Comparison on DepthTrack [68]: For evaluation, we use metrics such as
precision (Pr) and recall (Re), as well as the F-score, which are the primary
metrics. As shown in Tab. 1 When exclusively trained and fine-tuned on
DepthTrack, Un-Track achieves a +2.1% absolute precision improvement over the
current finetuned SOTA ViPT [93]. Notably, even when trained with mixed data
using a single parameter set, without domain-specific finetuning, Un-Track
outperforms the finetuned ViPT. Furthermore, when the current ViPT is jointly
trained on all datasets with a single set of parameters, its performance
significantly deteriorates.

Additionally, we observe that trackers excelling in other tracking datasets
[15, 23, 43] might struggle in RGB-D downstream settings. UniNext [67], a
leading tracker trained on various large-scale tracking datasets and related
image/ video tasks, exhibits poor performance. In contrast, our model achieves
cross-modal unification within a single set of parameters, surpassing all
finetuned counterparts and performing closely to our finetuned version. This
underscores the efficacy of our shared embedding in achieving global alignment
across diverse modalities.

| Finetuned Thermal-Specific Parameters | Single Set of Parameters
---|---|---
|  | SGT
---
[35]
| FANet
---
[98]
| mfDiMP
---
[83]
| DAFNet
---
[19]
| MaCNet
---
[79]
| ProTrack
---
[71]
| ViPT
---
[93]
| Un-Track
---
(ours)
| Stark
---
[64]
| AiATrack
---
[18]
| OSTrack
---
[74]
| SeqTrack
---
[9]
| ViPT
---
[93]
| Un-Track
---
(ours)
PR($\uparrow$) | 0.327 | 0.441 | 0.447 | 0.448 | 0.482 | 0.538 | 0.651 | 0.659 | 0.418 | 0.463 | 0.530 | 0.582 | 0.608 | 0.646
SR($\uparrow$) | 0.242 | 0.340 | 0.395 | 0.342 | 0.400 | 0.501 | 0.632 | 0.638 | 0.384 | 0.425 | 0.500 | 0.532 | 0.589 | 0.626

Table 2:  Overall performance on the Lasher thermal testing set [37].
Red/Green/Blue stands for the best/second/third performance. Our fine-tuned
model sets a new SOTA record. Our uni-modal variant competes strongly with the
previous SOTA fine-tuned model and significantly surpasses its unimodal
version.

| Finetuned Event-Specific Parameters | Single Set of Parameters
---|---|---
|  | SiamCar
---
[64]
| Stark_E
---
[64]
| VITAL_E
---
[47]
| PrDiMP_E
---
[14]
| TransT_E
---
[6]
| ProTrack
---
[71]
| ViPT
---
[93]
| Un-Track
---
(ours)
| Stark
---
[64]
| AiATrack
---
[18]
| OSTrack
---
[74]
| SeqTrack
---
[9]
| ViPT
---
[93]
| Un-Track
---
(ours)
Precision($\uparrow$) | 0.599 | 0.612 | 0.649 | 0.644 | 0.650 | 0.632 | 0.758 | 0.763 | 0.616 | 0.626 | 0.691 | 0.665 | 0.740 | 0.755
Success($\uparrow$) | 0.420 | 0.446 | 0.415 | 0.453 | 0.474 | 0.471 | 0.592 | 0.597 | 0.448 | 0.444 | 0.525 | 0.504 | 0.579 | 0.589

Table 3:  Overall performance on VisEvent dataset [56]. Red/Green/Blue denote
top three performances. Our fine-tuned model sets a new SOTA record. Our uni-
variant, with the same parameter set, consistently achieves competitive
performance across various modalities, leading to a significant margin over
the uni-variant of the SOTA fine-tuning model. Figure 5: More
precision/success comparisons on LasHer thermal dataset [37]. “Uni” stands for
models with a single parameter set.

Comparison on LasHer [37]: Similarly, we conduct evaluations on the LasHer
testing set for RGB-T tracker assessment as shown in Tab. 2. A more detailed
comparison can be found in Fig. 5. Precision (PR) and success rates (SR) are
reported following conventional standards [71, 37, 93]. Initial comparisons
under domain-specific finetuned settings reveal the challenges of achieving an
architecture-unified model across RGB-D and RGB-T domains, with ProTrack [71]
and ViPT [93] being the only works consistently leading in both settings. Our
Un-Track, following domain-specific finetuning, outperforms the leading ViPT
by a significant margin and sets a new SOTA record. In the cross-domain joint
learning with a single set of parameters, our uni-model achieves a +3.8%
absolute gain over ViPT. Remarkably, our model with a single set of parameters
already achieves very competitive performance compared to the finetuned ViPT
version.

Comparison on VisEvent [56]: We also evaluate tracker performance with RGB-
Event input. Event data, being inherently sparse compared to depth and thermal
information, presents challenges in extending existing RGB-D or RGB-T fusion
designs for effective integration, hence necessitating specific fusion designs
[100, 4, 44]. In contrast, we adopt a unified cross-modal prompting method
based on shrinkage fusion. Our approach, with gradual token exchanges between
RGB and event modalities, effectively preserves crucial modality-specific
clues to enhance feature modeling. Performance-wise, under the finetuned
setting, our Un-Track outperforms all counterparts, as shown in Tab. 3 and in
Fig. 6. In the single set of parameters setting, our uni-model achieves a
+1.1% absolute gain in precision over the current SOTA. This underscores the
effectiveness of our approach in handling the unique challenges posed by event
data integration, which can be mainly contributed to our shared binding that
learns the global RGB+X alignment.

Figure 6: More precision/success comparisons on VisEvent dataset [56]. “Uni”
stands for models with a single parameter set. “_E” stands for the extension
of RGB trackers with event fusion.

###  4.3 Generalization Across Datasets

In this section, we assess the versatility by evaluating performance on
datasets that differ from the training ones, aligning with the goal of
achieving a universal model checkpoint applicable to diverse scenarios.

VOT-RGBD2022 [31]: We first perform inference on the VOT-RGBD2022 dataset.
Notably, our uni-modal model achieves superior performance compared to the
depth-finetuned ViPT with a +0.5% absolute gain in accuracy.

RGBT234 [36]: We also test our model on the other thermal dataset RGBT234,
which encompasses sequences with different distributions. Our uni-modal model
surpasses the thermal-finetuned ViPT with a notable +0.7% absolute precision
gain, as shown in Tab. 5.

| Finetuned Depth-Specific Parameters | Single Set of Parameters
---|---|---
|  | DRefine
---
[30]
| Stark_D
---
[30]
| DMTracker
---
[31]
| DeT
---
[68]
| SBT_D
---
[31]
| SPT
---
[96]
| ProTrack
---
[71]
| ViPT
---
[93]
| Stark
---
[64]
| AiATrack
---
[18]
| OSTrack
---
[74]
| SeqTrack
---
[9]
| Un-Track
---
(ours)
EAO($\uparrow$) | 0.592 | 0.647 | 0.658 | 0.657 | 0.708 | 0.651 | 0.651 | 0.721 | 0.445 | 0.641 | 0.666 | 0.679 | 0.721
Accuracy($\uparrow$) | 0.775 | 0.803 | 0.758 | 0.760 | 0.809 | 0.798 | 0.801 | 0.815 | 0.714 | 0.769 | 0.808 | 0.802 | 0.820
Robustness($\uparrow$) | 0.760 | 0.798 | 0.851 | 0.845 | 0.864 | 0.851 | 0.802 | 0.871 | 0.598 | 0.832 | 0.814 | 0.846 | 0.869

Table 4:  Overall performance on VOT-RGBD2022 [31]. Our uni-model, trained on
a mix of all modalities, shows robust generalization and outperforms all
depth-finetuned models and other uni-modal counterparts.

| Finetuned Thermal-Specific Parameters | Single Set of Parameters
---|---|---
|  | mfDiMP
---
[83]
| SGT
---
[35]
| DAFNet
---
[19]
| FANet
---
[98]
| MaCNet
---
[79]
| CMPP
---
[53]
| APFNet
---
[60]
| ProTrack
---
[71]
| ViPT
---
[93]
| Stark
---
[64]
| AiATrack
---
[18]
| OSTrack
---
[74]
| SeqTrack
---
[9]
| Un-Track
---
(ours)
MPR($\uparrow$) | 0.646 | 0.720 | 0.796 | 0.787 | 0.790 | 0.823 | 0.827 | 0.795 | 0.835 | 0.677 | 0.711 | 0.755 | 0.806 | 0.842
MSR($\uparrow$) | 0.428 | 0.472 | 0.544 | 0.553 | 0.554 | 0.575 | 0.579 | 0.599 | 0.617 | 0.496 | 0.508 | 0.569 | 0.599 | 0.625

Table 5:  Overall performance on RGBT234 dataset [36]. Our uni-model sets new
SOTA records without specific thermal finetuning. Figure 7: Per-attribute
analysis on the thermal dataset RGBT234 [36]. Challenges related to motion and
geometry are generally better addressed by event and depth cameras,
respectively. Nevertheless, when inferring only with RGB-T data, our Un-Track
surpasses both the SOTA thermal-finetuned method and the current leading uni-
tracker. This success underscores our ability to learn emergent alignment
across diverse modalities.

Moreover, we present a detailed per-attribute comparison with the fine-tuned
ViPT in Fig. 7. We are particularly interested in sequences with motion blur-
fast motion-camera moving, as well as sequences with heavy occlusion-scale
variants-background clutter. The former motion-related challenges can
typically benefit from event clues, renowned for asynchronous computing, while
the latter geometry-related challenges can typically benefit from depth
cameras. However, these event/depth clues are not directly available in the
RGB-T setting. Nevertheless, our Un-Track, trained on all modalities,
outperforms both thermal fine-tuned ViPT [93] and the current leading uni-
tracker [9] with large margins. This underscores our capability in learning
event and depth priors through shared binding, without the need for the
presence of these modalities during inference.

RGB-only: In practical scenarios, challenges arise when there are no modal
clues available, a typical case is when the auxiliary sensor fails to work
properly. We address this demanding case in our study by substituting the
modal input with dummy values, akin to the configuration of our baseline RGB
tracker. Under a such challenging setting, as shown in Tab. 6, our uni-method
consistently outperforms both our RGB baseline fine-tuned counterparts
significantly. Notably, such an improvement is achieved with a very limited
increase in learning parameters, _i.e_., +6.65M with +2.14 GFLOPs.

|  |  RGB Baseline
---
[74]
|  ViPT
---
[93]
|  Un-Track
---
(ours)
GFLOPs |  21.50 |  21.80 |  23.64
Params (M) |  92.08 |  92.96 |  98.73
F-score($\uparrow$) |  0.529 |  0.542 |  0.558
Re($\uparrow$) |  0.522 |  0.538 |  0.557
Pr($\uparrow$) |  0.536 |  0.546 |  0.560

Table 6:  Overall performance on DepthTrack test set [68] with dummy depth
input.

|  | w/o
---
Shared Embed
|  [93] as
---
Modal Prompt
| w/o
---
LoRA Finetune
F-score($\uparrow$) | 0.599 | 0.579 | 0.594
Re($\uparrow$) | 0.602 | 0.575 | 0.598
Pr($\uparrow$) | 0.597 | 0.584 | 0.596

Table 7:  Key component analysis. Table 8: Low Rank Approximation Our plain
version is highlighted in gray.

| 2 | 4 | 8
---|---|---|---
F-score($\uparrow$) | 0.607 | 0.610 | 0.602
Re($\uparrow$) | 0.606 | 0.608 | 0.601
Pr($\uparrow$) | 0.608 | 0.611 | 0.604
(a)

4 | 8 | 16
---|---|---
0.596 | 0.610 | 0.606
0.593 | 0.608 | 0.609
0.599 | 0.611 | 0.604
(b)

2 | 4 | 8
---|---|---
0.601 | 0.610 | 0.600
0.599 | 0.608 | 0.598
0.602 | 0.611 | 0.602
(c)

1/8 | 1/4 | 1/3
---|---|---
0.604 | 0.610 | 0.595
0.606 | 0.608 | 0.593
0.602 | 0.611 | 0.596
(d)

##  5 Ablation Studies

In this section, we perform all experiments on the DepthTrack testing set [68]
under a single parameter set setting.

Key Component Analysis: We begin by studying the effectiveness of key
components, including the shared embedding, modal prompting, and LoRA
finetuning, as summarized in Tab. 7. We initially remove the shared embedding
by directly feeding mixed modalities into the learning diagram. It can be seen
that this equal treatment of all modalities harms network performance due to
the heterogeneous representation across modal domains. Secondly, we replace
our prompting block with a recent counterpart that computes fovea attention
from additional input [93]. Our designed gradual shrinkage fusion, allowing
token-wise interaction, proves to be more effective. We also report
performance when we remove the inner fine-tuning with Lora. It can be seen
that the performance deteriorates.

Low-Rank: Low-rank approximation plays a vital role in our model, influencing
our shared embedding, modal fusion, and LoRA-finetuning. Hence, the choice of
rank is crucial for both the efficiency and effectiveness of our approach. In
Tab. 8, we systematically explore the impact of low-rank choices within each
component.

Shared Embedding: Our objective is to identify an optimal low-rank latent
space for merging different modalities effectively. Tab. 8(a) presents our
ablation study, where we explore ranks of 2, 4, and 8. Lower ranks result in
poorer performance due to sparse representations. Conversely, higher ranks
capture too many modality-specific details, complicating the search for a
shared embedding.

Modal Prompting: As shown in Tab. 8(b), similar trends are observed when
investigating the low-rank choices for modal prompting with rank values of 4,
8, and 16, as low ranks struggle to capture essential information, while
higher ranks introduce an overload of modality-specific details.

LoRA-finetuning: We also vary the ranks between 2, 4, and 8 for the LoRA
finetuning technique, as shown in Tab. 8(c). Lower ranks exhibited
consistently poor performance, while higher ranks in this case tend to result
in poorer performance, likely due to overfitting.

Remarks: These experiments emphasize the importance of selecting the best low-
rank representations. Nevertheless, our model shows great resilience to the
choice of LoRA, showcasing consistent performance across different low-rank
configurations. Notably, all our low-rank variants outperform the current SOTA
ViPT under the uni-setting, validating our robustness and effectiveness.

Modal Prompting: During the prompting fusion, a learnable score function is
used to categorize tokens into three groups based on their confidence scores.
Here, we explore different percentiles for the number of positive, which is
the same as the number of negative tokens, leaving the rest as uncertain
tokens. As shown in Tab. 8(d), the choice of percentile can influence overall
performance. Lower percentiles result in poorer performance, as recovering
uncertain tokens from very few neighboring tokens is challenging. Higher
percentiles also lead to performance degradation since the focus is on token
exchange rather than token fusion. The choice of 1/4 leads to the best balance
between exchanger and fuser, leading to the best performance.

Shared Embedding: Here, we perform ablation studies on our shared embedding, a
foundational component of our uni-modal model. The quantitative results are
presented in Tab. 9. We begin by exploring a scenario where our shared
embedding is replaced with a variant lacking explicit edge guidance (w/o
Explicit Edge). In this configuration, the network learns the shared embedding
solely without any edge prior. The results highlight a substantial performance
drop, underscoring the pivotal role of explicit edge guidance — a natural and
static embedding that binds all modalities together — in facilitating this
implicit learning process.

We also conduct experiments using only the explicit edge as the shared
embedding, excluding any additional learning modules (w/o Implicit Learning).
This approach, too, yields suboptimal performance, primarily due to the
neglect of modality-specific clues within each domain.

Finally, we directly compute the low-rank approximation from mixed modalities,
bypassing the initial in-domain approximation and subsequent fusion steps (w/o
In-domain Approx.). The outcomes reveal the network’s struggle in effectively
learning a shared embedding in this direct low-rank approximation setup,
primarily due to the intricately mixed representation caused by the domain gap
between depth, thermal, and event modalities.

|  | w/o
---
Explicit Edge
| w/o
---
Implicit Learning
| w/o
---
In-domain Approx.
F-score($\uparrow$) | 0.600 | 0.604 | 0.581
Re($\uparrow$) | 0.602 | 0.609 | 0.583
Pr($\uparrow$) | 0.598 | 0.599 | 0.579

Table 9:  Ablation on shared embedding

##  6 Conclusion and Future Work

We present a successful cross-modal unified model for multi-modal video object
tracking, demonstrating the model’s ability to leverage auxiliary modalities
such as depth, thermal, or event data. The proposed method achieves a shared
embedding that binds all modalities together, overcoming their heterogeneous
representations. This unification is facilitated by lightweight modal
prompting and inner finetuning, inheriting benefits from large-scale pre-
trained trackers without introducing a substantial computational burden.
Exhaustive experiments validate the success of our approach, showcasing
improved tracking performance and robust generalization, with any modality
input.

Future work. First, although our method sets new leading performances across
different datasets with any modality, there is room for further improvement,
especially in cooperating with more recent trackers [62, 92]. A second
direction can be on lightweight tracking [66], with the help of techniques
such as pruning [2], to reduce the FLOPs.

## References

  * Arridge et al. [2022] Simon Arridge, Pascal Fernsel, and Andreas Hauptmann.  Joint reconstruction and low-rank decomposition for dynamic inverse problems.  _Inverse Problems and Imaging_ , 16(3):483–523, 2022.
  * Blalock et al. [2020] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag.  What is the state of neural network pruning?  _PMLR_ , 2020.
  * Chen et al. [2022a] Boyu Chen, Peixia Li, Lei Bai, Lei Qiao, Qiuhong Shen, Bo Li, Weihao Gan, Wei Wu, and Wanli Ouyang.  Backbone is all your need: A simplified architecture for visual object tracking.  In _ECCV_. Springer, 2022a.
  * Chen et al. [2023a] Qinyu Chen, Zuowen Wang, Shih-Chii Liu, and Chang Gao.  3et: Efficient event-based eye tracking using a change-based convlstm network.  _arXiv preprint arXiv:2308.11771_ , 2023a.
  * Chen et al. [2020a] Wanli Chen, Xinge Zhu, Ruoqi Sun, Junjun He, Ruiyu Li, Xiaoyong Shen, and Bei Yu.  Tensor low-rank reconstruction for semantic segmentation.  In _ECCV_. Springer, 2020a.
  * Chen et al. [2021a] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu.  Transformer tracking.  In _CVPR_ , 2021a.
  * Chen et al. [2021b] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu.  Transformer tracking.  In _CVPR_ , 2021b.
  * Chen et al. [2022b] Xin Chen, Bin Yan, Jiawen Zhu, Huchuan Lu, Xiang Ruan, and Dong Wang.  High-performance transformer tracking.  _TPAMI_ , 2022b.
  * Chen et al. [2023b] Xin Chen, Houwen Peng, Dong Wang, Huchuan Lu, and Han Hu.  Seqtrack: Sequence to sequence learning for visual object tracking.  In _CVPR_ , 2023b.
  * Chen et al. [2020b] Zedu Chen, Bineng Zhong, Guorong Li, Shengping Zhang, and Rongrong Ji.  Siamese box adaptive network for visual tracking.  In _CVPR_ , 2020b.
  * Cioppa et al. [2022] Anthony Cioppa, Silvio Giancola, Adrien Deliege, Le Kang, Xin Zhou, Zhiyu Cheng, Bernard Ghanem, and Marc Van Droogenbroeck.  Soccernet-tracking: Multiple object tracking dataset and benchmark in soccer videos.  In _CVPR_ , 2022.
  * Cui et al. [2022] Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu.  Mixformer: End-to-end tracking with iterative mixed attention.  In _CVPR_ , 2022.
  * Danelljan et al. [2019] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg.  ATOM: Accurate tracking by overlap maximization.  In _CVPR_ , 2019.
  * Danelljan et al. [2020] Martin Danelljan, Luc Van Gool, and Radu Timofte.  Probabilistic regression for visual tracking.  In _CVPR_ , 2020.
  * Fan et al. [2019] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling.  LaSOT: A high-quality benchmark for large-scale single object tracking.  In _CVPR_ , 2019.
  * Fu et al. [2023] Yingkai Fu, Meng Li, Wenxi Liu, Yuanchen Wang, Jiqing Zhang, Baocai Yin, Xiaopeng Wei, and Xin Yang.  Distractor-aware event-based tracking.  _TIP_ , 2023.
  * Gao et al. [2022a] Shang Gao, Jinyu Yang, Zhe Li, Feng Zheng, Aleš Leonardis, and Jingkuan Song.  Learning dual-fused modality-aware representations for rgbd tracking.  In _European Conference on Computer Vision_ , pages 478–494. Springer, 2022a.
  * Gao et al. [2022b] Shenyuan Gao, Chunluan Zhou, Chao Ma, Xinggang Wang, and Junsong Yuan.  Aiatrack: Attention in attention for transformer visual tracking.  In _ECCV_ , 2022b.
  * Gao et al. [2019] Yuan Gao, Chenglong Li, Yabin Zhu, Jin Tang, Tao He, and Futian Wang.  Deep adaptive fusion network for high performance RGBT tracking.  In _ICCVW_ , pages 0–0, 2019.
  * Girdhar et al. [2023] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  Imagebind: One embedding space to bind them all.  In _CVPR_ , 2023.
  * He et al. [2021] Botao He, Haojia Li, Siyuan Wu, Dong Wang, Zhiwei Zhang, Qianli Dong, Chao Xu, and Fei Gao.  Fast-dynamic-vision: Detection and tracking dynamic objects with event and depth sensing.  In _IROS_ , 2021.
  * Hu et al. [2022] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  In _ICLR_ , 2022.
  * Huang et al. [2019] Lianghua Huang, Xin Zhao, and Kaiqi Huang.  Got-10k: A large high-diversity benchmark for generic object tracking in the wild.  _TPAMI_ , 2019.
  * Javed et al. [2022] Sajid Javed, Martin Danelljan, Fahad Shahbaz Khan, Muhammad Haris Khan, Michael Felsberg, and Jiri Matas.  Visual object tracking with discriminative filters and siamese networks: a survey and outlook.  _TPAMI_ , 45(5):6552–6574, 2022.
  * Jhuo et al. [2012] I-Hong Jhuo, Dong Liu, DT Lee, and Shih-Fu Chang.  Robust visual domain adaptation with low-rank reconstruction.  In _CVPR_. IEEE, 2012.
  * Jia et al. [2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In _ECCV_. Springer, 2022.
  * Kang et al. [2023] Ben Kang, Xin Chen, Dong Wang, Houwen Peng, and Huchuan Lu.  Exploring lightweight hierarchical vision transformers for efficient visual tracking.  In _ICCV_ , 2023.
  * Khattak et al. [2023] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Maple: Multi-modal prompt learning.  In _CVPR_ , 2023.
  * Kristan et al. [2020] Matej Kristan, Aleš Leonardis, Jiří Matas, Michael Felsberg, Roman Pflugfelder, Joni-Kristian Kämäräinen, Martin Danelljan, Luka Čehovin Zajc, Alan Lukežič, Ondrej Drbohlav, et al.  The eighth visual object tracking vot2020 challenge results.  In _ECCVW_ , pages 547–601. Springer, 2020.
  * Kristan et al. [2021] Matej Kristan, Jiří Matas, Aleš Leonardis, Michael Felsberg, Roman Pflugfelder, Joni-Kristian Kämäräinen, Hyung Jin Chang, Martin Danelljan, Luka Cehovin, Alan Lukežič, et al.  The ninth visual object tracking vot2021 challenge results.  In _ICCVW_ , pages 2711–2738, 2021.
  * Kristan et al. [2023] Matej Kristan, Aleš Leonardis, Jiří Matas, Michael Felsberg, Roman Pflugfelder, Joni-Kristian Kämäräinen, Hyung Jin Chang, Martin Danelljan, Luka Čehovin Zajc, Alan Lukežič, et al.  The tenth visual object tracking vot2022 challenge results.  In _ECCVW_ , pages 431–460. Springer, 2023.
  * Lee et al. [2023] Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen-Yu Lee.  Multimodal prompting with missing modalities for visual recognition.  In _CVPR_ , 2023.
  * Li et al. [2018] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu.  High performance visual tracking with siamese region proposal network.  In _CVPR_ , 2018.
  * Li et al. [2019a] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan.  SiamRPN++: Evolution of siamese visual tracking with very deep networks.  In _CVPR_ , 2019a.
  * Li et al. [2017] Chenglong Li, Nan Zhao, Yijuan Lu, Chengli Zhu, and Jin Tang.  Weighted sparse representation regularized graph learning for RGB-T object tracking.  In _ACMMM_ , pages 1856–1864, 2017.
  * Li et al. [2019b] Chenglong Li, Xinyan Liang, Yijuan Lu, Nan Zhao, and Jin Tang.  RGB-T object tracking: Benchmark and baseline.  _Pattern Recognition_ , 96:106977, 2019b.
  * Li et al. [2021] Chenglong Li, Wanlin Xue, Yaqing Jia, Zhichen Qu, Bin Luo, Jin Tang, and Dengdi Sun.  Lasher: A large-scale high-diversity benchmark for RGBT tracking.  _TIP_ , 31:392–404, 2021.
  * Liu et al. [2020] Qiao Liu, Xin Li, Zhenyu He, Nana Fan, Di Yuan, and Hongpeng Wang.  Learning deep multi-level similarity for thermal infrared object tracking.  _TMM_ , 23:2114–2126, 2020.
  * Long Li et al. [2019] Cheng Long Li, Andong Lu, Ai Hua Zheng, Zhengzheng Tu, and Jin Tang.  Multi-adapter rgbt tracking.  In _ICCVW_ , 2019.
  * Lu et al. [2023] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi.  Unified-io: A unified model for vision, language, and multi-modal tasks.  In _ICLR_ , 2023.
  * Lukezic et al. [2019] Alan Lukezic, Ugur Kart, Jani Kapyla, Ahmed Durmush, Joni-Kristian Kamarainen, Jiri Matas, and Matej Kristan.  Cdtb: A color and depth visual object tracking dataset and benchmark.  In _ICCV_ , 2019.
  * Ma et al. [2021] Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng.  Smil: Multimodal learning with severely missing modality.  In _AAAI_ , 2021.
  * Muller et al. [2018] Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem.  TrackingNet: A large-scale dataset and benchmark for object tracking in the wild.  In _ECCV_ , 2018.
  * Peng et al. [2023] Yansong Peng, Yueyi Zhang, Zhiwei Xiong, Xiaoyan Sun, and Feng Wu.  Get: Group event transformer for event-based vision.  In _ICCV_ , 2023.
  * Qiu et al. [2023] Yansheng Qiu, Ziyuan Zhao, Hongdou Yao, Delin Chen, and Zheng Wang.  Modal-aware visual prompting for incomplete multi-modal brain tumor segmentation.  In _ACM MM_ , 2023.
  * Real et al. [2017] Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke.  Youtube-boundingboxes: A large high-precision human-annotated data set for object detection in video.  In _CVPR_ , 2017.
  * Song et al. [2018] Yibing Song, Chao Ma, Xiaohe Wu, Lijun Gong, Linchao Bao, Wangmeng Zuo, Chunhua Shen, Rynson WH Lau, and Ming-Hsuan Yang.  Vital: Visual tracking via adversarial learning.  In _CVPR_ , pages 8990–8999, 2018.
  * Stadler and Beyerer [2021] Daniel Stadler and Jurgen Beyerer.  Improving multiple pedestrian tracking by track management and occlusion handling.  In _CVPR_ , 2021.
  * Tang et al. [2022a] Chuanming Tang, Xiao Wang, Ju Huang, Bo Jiang, Lin Zhu, Jianlin Zhang, Yaowei Wang, and Yonghong Tian.  Revisiting color-event based tracking: A unified network, dataset, and metric.  _arXiv preprint arXiv:2211.11010_ , 2022a.
  * Tang et al. [2022b] Chuanming Tang, Xiao Wang, Ju Huang, Bo Jiang, Lin Zhu, Jianlin Zhang, Yaowei Wang, and Yonghong Tian.  Revisiting color-event based tracking: A unified network, dataset, and metric.  _arXiv preprint arXiv:2211.11010_ , 2022b.
  * Voigtlaender et al. [2020] Paul Voigtlaender, Jonathon Luiten, Philip H. S. Torr, and Bastian Leibe.  Siam R-CNN: Visual tracking by re-detection.  In _CVPR_ , 2020.
  * Wan et al. [2023] Zhexiong Wan, Yuxin Mao, Jing Zhang, and Yuchao Dai.  Rpeflow: Multimodal fusion of rgb-pointcloud-event for joint optical flow and scene flow estimation.  In _ICCV_ , 2023.
  * Wang et al. [2020] Chaoqun Wang, Chunyan Xu, Zhen Cui, Ling Zhou, Tong Zhang, Xiaoya Zhang, and Jian Yang.  Cross-modal pattern-propagation for RGB-T tracking.  In _CVPR_ , pages 7064–7073, 2020.
  * Wang et al. [2023] Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carneiro.  Multi-modal learning with missing modality via shared-specific feature modelling.  In _CVPR_ , 2023.
  * Wang et al. [2021a] Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li.  Transformer meets tracker: Exploiting temporal context for robust visual tracking.  In _CVPR_ , pages 1571–1580, 2021a.
  * Wang et al. [2021b] Xiao Wang, Jianing Li, Lin Zhu, Zhipeng Zhang, Zhe Chen, Xin Li, Yaowei Wang, Yonghong Tian, and Feng Wu.  Visevent: Reliable object tracking via collaboration of frame and event flows.  _arXiv preprint arXiv:2108.05015_ , 2021b.
  * Wang et al. [2022a] Xiao Wang, Xiujun Shu, Shilliang Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu.  Mfgnet: Dynamic modality-aware filter generation for rgb-t tracking.  _TMM_ , 2022a.
  * Wang et al. [2022b] Zuowen Wang, Yuhuang Hu, and Shih-Chii Liu.  Exploiting spatial sparsity for event cameras with visual transformers.  In _ICIP_ , 2022b.
  * Wisth et al. [2021] David Wisth, Marco Camurri, Sandipan Das, and Maurice Fallon.  Unified multi-modal landmark tracking for tightly coupled lidar-visual-inertial odometry.  _RA-L_ , 6(2):1004–1011, 2021.
  * Xiao et al. [2022a] Yun Xiao, Mengmeng Yang, Chenglong Li, Lei Liu, and Jin Tang.  Attribute-based progressive fusion network for RGBT tracking.  In _AAAI_ , 2022a.
  * Xiao et al. [2022b] Yun Xiao, Mengmeng Yang, Chenglong Li, Lei Liu, and Jin Tang.  Attribute-based progressive fusion network for rgbt tracking.  In _AAAI_ , 2022b.
  * Xie et al. [2023] Fei Xie, Lei Chu, Jiahao Li, Yan Lu, and Chao Ma.  Videotrack: Learning to track objects via video transformer.  In _CVPR_ , 2023.
  * Xu et al. [2020] Yinda Xu, Zeyu Wang, Zuoxin Li, Ye Yuan, and Gang Yu.  SiamFC++: Towards robust and accurate visual tracking with target estimation guidelines.  In _AAAI_ , 2020.
  * Yan et al. [2021a] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu.  Learning spatio-temporal transformer for visual tracking.  In _ICCV_ , 2021a.
  * Yan et al. [2021b] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu.  Learning spatio-temporal transformer for visual tracking.  In _ICCV_ , 2021b.
  * Yan et al. [2021c] Bin Yan, Houwen Peng, Kan Wu, Dong Wang, Jianlong Fu, and Huchuan Lu.  Lighttrack: Finding lightweight neural networks for object tracking via one-shot architecture search.  In _CVPR_ , 2021c.
  * Yan et al. [2023] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu.  Universal instance perception as object discovery and retrieval.  In _CVPR_ , 2023.
  * Yan et al. [2021d] Song Yan, Jinyu Yang, Jani Käpylä, Feng Zheng, Aleš Leonardis, and Joni-Kristian Kämäräinen.  Depthtrack: Unveiling the power of RGBD tracking.  In _ICCV_ , pages 10725–10733, 2021d.
  * Yan et al. [2021e] Song Yan, Jinyu Yang, Ales Leonardis, and Joni-Kristian Kamarainen.  Depth-only object tracking.  In _BMVC_ , 2021e.
  * Yang et al. [2022a] Jinyu Yang, Zhe Li, Song Yan, Feng Zheng, Aleš Leonardis, Joni-Kristian Kämäräinen, and Ling Shao.  Rgbd object tracking: An in-depth review.  _arXiv preprint arXiv:2203.14134_ , 2022a.
  * Yang et al. [2022b] Jinyu Yang, Zhe Li, Feng Zheng, Ales Leonardis, and Jingkuan Song.  Prompting for multi-modal tracking.  In _ACMMM_ , pages 3492–3500, 2022b.
  * Yang et al. [2023] Jinyu Yang, Shang Gao, Zhe Li, Feng Zheng, and Aleš Leonardis.  Resource-efficient rgbd aerial tracking.  In _CVPR_ , 2023.
  * Yao et al. [2020] Rui Yao, Guosheng Lin, Shixiong Xia, Jiaqi Zhao, and Yong Zhou.  Video object segmentation and tracking: A survey.  _ACM TIST_ , 11(4):1–47, 2020.
  * Ye et al. [2022] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen.  Joint feature learning and relation modeling for tracking: A one-stream framework.  In _ECCV_ , pages 341–357. Springer, 2022.
  * Yin et al. [2021] Jie Yin, Ang Li, Tao Li, Wenxian Yu, and Danping Zou.  M2dgr: A multi-sensor and multi-scenario slam dataset for ground robots.  _RA-L_ , 7(2):2266–2273, 2021.
  * Yu et al. [2020] Yuechen Yu, Yilei Xiong, Weilin Huang, and Matthew R Scott.  Deformable siamese attention networks for visual object tracking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6728–6737, 2020.
  * Zeng et al. [2022] Jiandian Zeng, Tianyi Liu, and Jiantao Zhou.  Tag-assisted multimodal sentiment analysis under uncertain missing modalities.  In _ACM SIGIR_ , 2022.
  * Zhang et al. [2023a] Chunhui Zhang, Xin Sun, Yiqian Yang, Li Liu, Qiong Liu, Xi Zhou, and Yanfeng Wang.  All in one: Exploring unified vision-language tracking with multi-modal alignment.  In _ACM MM_ , 2023a.
  * Zhang et al. [2020a] Hui Zhang, Lei Zhang, Li Zhuo, and Jing Zhang.  Object tracking in RGB-T videos using modal-aware attention network and competitive learning.  _Sensors_ , 20(2):393, 2020a.
  * Zhang et al. [2021] Jiqing Zhang, Xin Yang, Yingkai Fu, Xiaopeng Wei, Baocai Yin, and Bo Dong.  Object tracking by jointly exploiting frame and event domain.  In _ICCV_ , 2021.
  * Zhang et al. [2022a] Jiqing Zhang, Bo Dong, Haiwei Zhang, Jianchuan Ding, Felix Heide, Baocai Yin, and Xin Yang.  Spiking transformers for event-based single object tracking.  In _CVPR_ , 2022a.
  * Zhang et al. [2023b] Jiaming Zhang, Ruiping Liu, Hao Shi, Kailun Yang, Simon Reiß, Kunyu Peng, Haodong Fu, Kaiwei Wang, and Rainer Stiefelhagen.  Delivering arbitrary-modal semantic segmentation.  In _CVPR_ , 2023b.
  * Zhang et al. [2019a] Lichao Zhang, Martin Danelljan, Abel Gonzalez-Garcia, Joost van de Weijer, and Fahad Shahbaz Khan.  Multi-modal fusion for end-to-end RGB-T tracking.  In _ICCVW_ , pages 0–0, 2019a.
  * Zhang et al. [2020b] Pengyu Zhang, Dong Wang, and Huchuan Lu.  Multi-modal visual tracking: Review and experimental comparison.  _arXiv preprint arXiv:2012.04176_ , 2020b.
  * Zhang et al. [2022b] Pengyu Zhang, Jie Zhao, Dong Wang, Huchuan Lu, and Xiang Ruan.  Visible-thermal uav tracking: A large-scale benchmark and new baseline.  In _CVPR_ , 2022b.
  * Zhang et al. [2019b] Wenwei Zhang, Hui Zhou, Shuyang Sun, Zhe Wang, Jianping Shi, and Chen Change Loy.  Robust multi-modality multi-object tracking.  In _ICCV_ , 2019b.
  * Zhang and Peng [2019] Zhipeng Zhang and Houwen Peng.  Deeper and wider siamese networks for real-time visual tracking.  In _CVPR_ , 2019.
  * Zhao et al. [2023] Haojie Zhao, Dong Wang, and Huchuan Lu.  Representation learning for visual object tracking by masked appearance transfer.  In _CVPR_ , 2023.
  * Zhao et al. [2021a] Jinjian Zhao, Xiaohan Zhang, and Pengyu Zhang.  A unified approach for tracking uavs in infrared.  In _ICCV_ , 2021a.
  * Zhao et al. [2021b] Shaochuan Zhao, Tianyang Xu, Xiao-Jun Wu, and Xue-Feng Zhu.  Adaptive feature fusion for visual object tracking.  _PR_ , 111:107679, 2021b.
  * Zheng et al. [2021] Aihua Zheng, Zi Wang, Zihan Chen, Chenglong Li, and Jin Tang.  Robust multi-modality person re-identification.  In _AAAI_ , 2021.
  * Zhu et al. [2023a] Jiawen Zhu, Zhenyu Chen, Zeqi Hao, Shijie Chang, Lu Zhang, Dong Wang, Huchuan Lu, Bin Luo, Jun-Yan He, Jin-Peng Lan, et al.  Tracking anything in high quality.  _arXiv preprint arXiv:2307.13974_ , 2023a.
  * Zhu et al. [2023b] Jiawen Zhu, Simiao Lai, Xin Chen, Dong Wang, and Huchuan Lu.  Visual prompt multi-modal tracking.  In _CVPR_ , 2023b.
  * Zhu et al. [2023c] Jiawen Zhu, Huayi Tang, Zhi-Qi Cheng, Jun-Yan He, Bin Luo, Shihao Qiu, Shengming Li, and Huchuan Lu.  Dcpt: Darkness clue-prompted tracking in nighttime uavs.  _arXiv preprint arXiv:2309.10491_ , 2023c.
  * Zhu et al. [2021] Xue-Feng Zhu, Xiao-Jun Wu, Tianyang Xu, Zhen-Hua Feng, and Josef Kittler.  Robust visual object tracking via adaptive attribute-aware discriminative correlation filters.  _TMM_ , 24:301–312, 2021.
  * Zhu et al. [2023d] Xue-Feng Zhu, Tianyang Xu, Zhangyong Tang, Zucheng Wu, Haodong Liu, Xiao Yang, Xiao-Jun Wu, and Josef Kittler.  RGBD1K: A large-scale dataset and benchmark for RGB-D object tracking.  _AAAI_ , 2023d.
  * Zhu et al. [2023e] Xue-Feng Zhu, Tianyang Xu, Zhangyong Tang, Zucheng Wu, Haodong Liu, Xiao Yang, Xiao-Jun Wu, and Josef Kittler.  Rgbd1k: A large-scale dataset and benchmark for rgb-d object tracking.  In _AAAI_ , 2023e.
  * Zhu et al. [2020] Yabin Zhu, Chenglong Li, Jin Tang, and Bin Luo.  Quality-aware feature aggregation network for robust RGBT tracking.  _IEEE Transactions on Intelligent Vehicles_ , 6(1):121–130, 2020.
  * Zhu et al. [2022] Zhiyu Zhu, Junhui Hou, and Xianqiang Lyu.  Learning graph-embedded key-event back-tracing for object tracking in event clouds.  _NeurIPS_ , 2022.
  * Zhu et al. [2023f] Zhiyu Zhu, Junhui Hou, and Dapeng Oliver Wu.  Cross-modal orthogonal high-rank augmentation for rgb-event transformer-trackers.  In _ICCV_ , 2023f.


††thanks: The first two authors contribute equally to this work

# Generation of subcycle isolated attosecond pulses by pumping ionizing gating

Zhaohui Wu  Science and Technology on Plasma Physics Laboratory, Research
Center of Laser Fusion, China Academy of Engineering Physics, Mianyang,
Sichuan, China, 621900.  Hao Peng  College of Physics and Optoelectronic
Engineering, Shenzhen University, Shenzhen 518060, China  Xiaoming Zeng
Zhaoli Li  Zhimeng Zhang  Science and Technology on Plasma Physics Laboratory,
Research Center of Laser Fusion, China Academy of Engineering Physics,
Mianyang, Sichuan, China, 621900.  Huabao Cao  Yuxi Fu  Center for Attosecond
Science and Technology, Xi’an Institute of Optics and Precision Mechanics,
Chinese Academy of Sciences, Xi’an 710119, Shaanxi, China  Xiaodong Wang  Xiao
Wang  Jie Mu  Yanlei Zuo<EMAIL_ADDRESS>Science and Technology on
Plasma Physics Laboratory, Research Center of Laser Fusion, China Academy of
Engineering Physics, Mianyang, Sichuan, China, 621900.  C. Riconda  LULI,
Sorbonne Université, CNRS, École Polytechnique, CEA, F-75005, Paris, France
S. Weber  ELI Beamlines facility, Extreme Light Infrastructure ERIC, 25241
Dolni Brezany, Czech Republic  Jingqin Su  Science and Technology on Plasma
Physics Laboratory, Research Center of Laser Fusion, China Academy of
Engineering Physics, Mianyang, Sichuan, China, 621900.

###### Abstract

We present a novel approach named as pumping ionizing gating (PIG) for the
generation of isolated attosecond pulses (IAPs). In this regime, a short laser
is used to ionize a pre-existing gas grating, creating a fast-extending plasma
grating(FEPG) having an ionization front propagating with the velocity of
light. A low-intensity long counterpropagating pump pulse is then reflected by
a very narrow region of the ionization front, only where the Bragg conditions
for resonant reflection is satisfied. Consequently, the pump reflection is
confined within a sub-cycle region called PIG, and forms a wide-band coherent
IAP in combination with the frequency up-conversion effect due to the plasma
gradient. This approach results in a new scheme to generate IAPs from long
picosecond pump pulses. Three-dimensional (3D) simulations show that a 1.6-ps,
1-$\mu$m pump pulse can be used to generate a 330 as laser pulse with a peak
intensity approximately 33 times that of the pump and a conversion efficiency
of around 0.1$\%$.These results highlight the potential of the PIG method for
generating IAPs with high conversion efficiency and peak intensity.

isolated attosecond pulse, frequency up-conversion, ionizing gating, plasma
grating

††preprint: APS/123-QED

Ultrafast techniques have enabled the generation of laser pulses with
durations on the attosecond scale, allowing for the direct observation of
electron motion in various systems such as atoms, molecules, and solids.
Isolated attosecond pulses (IAPs) are typically generated through gas high
harmonics generation(GHHG) with gating techniquesCalegari  _et al._ (2016);
Chini, Zhao, and Chang (2014); Hentschel  _et al._ (2001); Sansone  _et al._
(2006); Goulielmakis  _et al._ (2004); Gaumnitz  _et al._ (2017); Witting  _et
al._ (2012); Corkum, Burnett, and Ivanov (1994); Li  _et al._ (2017); Z (2004,
2005); Sola  _et al._ (2006); Zhao  _et al._ (2012); Chang (2007); Mashiko
_et al._ (2008); Vincenti and Quéré (2012); Kim  _et al._ (2013); Heyl  _et
al._ (2014); Zhong  _et al._ (2016); Tzallas  _et al._ (2007); Feng  _et al._
(2009); Skantzakis  _et al._ (2009); H, K, and T (2013); Tzallas  _et al._
(2007), which have spectral coverage from hard X-rays to extreme ultraviolet
(EUV,$<120$ nm) and durations as short as a few tens of attoseconds. However,
the pulse energies of these IAPs are limited to the nanojoule range due to the
low efficiency of high harmonics generation. Alternative methods for
generating IAPs include coherent field synthesisHuang  _et al._ (2011); Hassan
_et al._ (2016); Rossi  _et al._ (2020), which has been demonstrated to
produce pulses with durations of as short as 380 attoseconds and spectra
covering the deep ultraviolet to near infrared rangeHassan  _et al._ (2016).
It leads to longer pulse durations while allowing for higher pulse energies
(tens of microjoules). Further increasing of the IAPs energy opens new
possibilities for the nonlinear attosecond optics. However, it is limited by
the energy of the few-cycle driving pulses.

The key principle behind GHHG for generating IAPs lies in the use of gating
techniques to confine the resonant generation zone to a narrow temporal
region. However, this inherently results in low energy transfer efficiency
since a significant portion of the driving pulse is not utilized. On the other
hand, plasma has been shown to be an effective media for laser compression by
exploiting the plasma wavesMalkin, Shvets, and Fisch (1999, 2000); Trines  _et
al._ (2011); Ping  _et al._ (2004); Cheng  _et al._ (2005); Ren  _et al._
(2007); Turnbull  _et al._ (2018); Wu  _et al._ (2019); Andreev  _et al._
(2006); Weber  _et al._ (2013); Lancia  _et al._ (2010, 2016); Marqués  _et
al._ (2019) or fast-extending plasma grating(FEPG)Wu  _et al._ (2022); Li  _et
al._ (2022). This technique enables laser compression with high peak power and
efficiency. However, the shortest duration of the output pulse is limited to
several optical cycles.

Figure 1: (a) Illustration of the pumping ionizing gating(PIG) formed by the
gradient FEPG. (b) Scheme of IAP generation due to pump reflection and
frequency upconversion in the resonant region(PIG).

In this letter, we propose a novel approach making joule-level IAPs possible
by incorporating ionization gating techniques into the plasma compression
using a gradient FEPG. The FEPG is formed by utilizing a short laser pulse to
ionize a background gas grating, which can be created using established
methods such as stimulated Brillouin scattering (SBS) Wu  _et al._ (2022); Li
_et al._ (2022); Damzen  _et al._ (2003); Gorbunov  _et al._ (1983); Yang,
Gygerand, and Thévenaz (2020); Carr and Hanna (1985); Meng and H.J.Eichler
(1991) or interference ionization Zhang  _et al._ (2021) in the gas medium.
Following the ionization pulse, the plasma grating has a boundary extending at
the light speed. In previous studies, the pump beam is resonantly reflected in
the bulk plasma grating region to ensure pump depletion Wu  _et al._ (2022);
Li  _et al._ (2022). However, in our proposed scheme, as illustrated in
Fig.1(a), the resonant reflection region of the pump is confined to a narrow
region at the ionization front by selecting a proper grating period. Within
this ionization front, the gas grating is gradually ionized by the driving
pulse, resulting in an increasing average plasma density $n_{e0}\rho(\xi)$,
where $\xi=z-tv_{g}$, $v_{g}$ is the group velocity of the driving pulse,
$\rho$ is the ionization rate, and ${n_{e}}_{0}$ is average electron density
of the fully-ionized gas. For a grating with a period $\Lambda$, the Bragg
condition for resonant reflection of the pump can only be fulfilled in a very
narrow region around $\xi\approx\xi_{0}$, given by $k_{0}N_{0}\Lambda=\pi$,
where $k_{0}=2\pi/\lambda_{0}$ and $\lambda_{0}$ are the pump wavevector and
wavelength, and $N_{0}=(1+\sqrt{1-{n_{e}}_{0}\rho(\xi_{0})/n_{c}})/2$
represents the refractive index at the resonant region of the FEPGLi  _et al._
(2022). Moreover, as the laser propagates through the gradient plasma, a
frequency-upconversion effect occurs Qu  _et al._ (2018); K.Qu and Fisch
(2019); Howard  _et al._ (2019); Esarey, G.Joyce, and Sprangle (1991); H.Peng
_et al._ (2021). Consequently, the reflected pump beam is confined and up-
converted in the narrow resonant region, resulting in a wide-band coherent
IAP, as shown Fig.1(b). Since the reflected pulse extracts energy from the
entire long pump pulse, we call the approach pumping ionizing gating(PIG)
which enables the IAP energy several orders of magnification higher than the
present level.

To explain the physical model of PIG, we firstly consider the energy transfer
from the pump to driving pulse by the FEPG, which is given by the coupled-
waves equation in the variable of $\xi$Wu  _et al._ (2022):

$\displaystyle\begin{array}[]{rcl}&&\frac{\partial E}{\partial
t}=-i\frac{\omega^{2}_{pe}}{2\omega_{0}}E_{0}fe^{2i\omega_{0}\delta Nt},\\\
&&\frac{\partial E_{0}}{\partial t}-2v_{g}\frac{\partial
E_{0}}{\partial\xi}=-i\frac{\omega^{2}_{pe}}{2\omega_{0}}Efe^{-2i\omega_{0}\delta
Nt},\end{array}$ (3)

where $E$ and $E_{0}$ are the electric filed of driving pulse and pump pulse
respectively, $\omega_{pe}=\sqrt{n_{e}e^{2}/m_{e}\varepsilon_{0}}$ is the
plasma frequency, $n_{e}$ is the electron density, $e$ and $m_{e}$ are the
electron charge and mass, $\varepsilon_{0}$ is the
permittivity,$f\equiv\frac{\delta{n_{e}}_{0}}{{n_{e}}_{0}}$ is the normalized
FEPG amplitude, $\delta{n_{e}}_{0}$ is the amplitude of the fully-ionized
plasma grating, and $\omega_{0}$ is the pump frequency.

The electron density $n_{e}$ in the gradient FEPG can be written as

$\displaystyle\begin{array}[]{rcl}n_{e}=\left\\{\begin{array}[]{rcl}{n_{e}}_{0}+\delta{n_{e}}_{0}sin(2\pi\xi/\Lambda),&&\xi<-L,\\\
\rho[{n_{e}}_{0}+\delta{n_{e}}_{0}sin(2\pi\xi/\Lambda)],&&-L\leq\xi\leq 0,\\\
0,&&\xi>0,\end{array}\right.\end{array}$ (8)

where $L$ is the length of the ionization front.

As most regions of the gradient FEPG can not satisfy the resonant condition,
there is a detuning factor of $e^{2i\omega_{0}\delta Nt}$ in the Eq.3, where
$\delta N=(N-N_{0})$, $N=(1+\sqrt{1-\rho(\xi){n_{e}}_{0}/n_{c}})/2$ is the
refraction index of the pump in FEPGLi  _et al._ (2022),
$n_{c}=\omega^{2}_{0}m_{e}\varepsilon_{0}/{n_{e}}_{0}e^{2}$is the critical
plasma density for the pump. One has $2k_{0}N_{0}\Lambda=2\pi$, therefore

$\displaystyle\begin{array}[]{rcl}\Lambda=\lambda_{0}/(1+\sqrt{1-\rho(\xi_{0}){n_{e}}_{0}/n_{c}}).\end{array}$
(10)

From Eq.3 and Eq.10 it can be seen that the growth rate of the driving pulse
decreases exponentially at the position away from the resonant point,
therefore a narrow gating is created, and it moves to the position a higher
$\rho$ as $\Lambda$ increases. For $\rho$=1, Eq.10 becomes the standard Bragg
condition of FEPGLi  _et al._ (2022).

In a medium with a time-varying refractive index, the laser experiences a
frequency shift while maintaining a constant wavevector in order to satisfy
the dispersion relation Esarey, G.Joyce, and Sprangle (1991). Theoretical
models have been developed to describe this phenomenon in the ionizing plasma
Qu  _et al._ (2018); K.Qu and Fisch (2019); Howard  _et al._ (2019). In the
case of the ionizing plasma grating, although the periodic structure also
provides a time-varying refractive index, the integration over one period is
close to zero. Consequently, the frequency upconversion effect resulting from
the grating structure can be neglected, allowing us to extend the existing
theoretical models to the gradient FEPG. To validate this assumption, we
performed particle-in-cell (PIC) simulations using the same laser pulse to
ionize the gas and the gas grating. The up-converted spectra were found to be
nearly identical(see Fig.S1 in the supplement materials).

For a laser induced plasma, a roughly linear density gradient is formed in the
pulse front. The up-converted light has an analytical solution of
$\frac{\omega_{0}}{\omega}Ee^{i\omega t(\xi-\xi_{0})}$ when
$\omega_{pe}\ll\omega$, where
$\omega=\omega_{0}(1+ct\omega^{2}_{pe}/L\omega_{0}^{2})^{1/2}$ is the shifted
laser frequency. However, the up-conversion process in PIG is more complex due
to the reflected pump. As the reflected pump is mainly affected by the plasma
gradient and the propagation time, it can be approximately divided into a
series of independent infinitesimal pulses. For $dE$ at time $t$, the up-
converted electric field at time $t+\delta t$ can be described by

$\displaystyle\begin{array}[]{rcl}dE(t+\delta
t)=\frac{\omega_{0}}{\omega(\delta t)}dEe^{i\omega(\delta
t)(\xi-\xi_{0})}.\end{array}$ (12)

As the resonant region in the gradient FEPG is very narrow, the pump
reflectivity is small and $E_{0}$ is approximate to a constant in Eq.3, so the
driving pulse has an analytical solution by combing Eq.3 and Eq.12.

$\displaystyle\begin{array}[]{rcl}E=\int^{t}_{0}i\frac{\omega^{2}_{pe}}{2\omega}E_{0}fe^{2i\omega_{0}\delta
Nt+i\omega(\xi-\xi_{0})}dt.\end{array}$ (14)

The detailed derivation of Eq.14 is provided in the supplement material.

In order to further visualize the mechanism, PIC simulations were carried out
using the code EPOCHArber  _et al._ (2015). A picosecond flat-top pulse with
intensity $I_{0}=4\times 10^{13}~{}\rm{W/cm^{2}}$ and wavelength
$\lambda_{0}=1~{}\mu$m was used as the pump, while a femtosecond Gaussian
pulse with intensity $10^{14}~{}\rm{W/cm^{2}}$ and an arbitrary wavelength
served as the driving(ionizing) pulse. The background gas grating was set to
hydrogen with a modulation period near $\sim\lambda_{0}/2$. The ionization
process was modeled using the Ammosov-Delone-Krainov tunnel ionization
modelAmmosov, Delone, and Krainov (1986).

Figure 2:  (a)Profiles of the 3D IAPs.(b)Distribution of FWHM pulse
duration.(c)Temporal intensity of the IAP at the beam center.

We firstly show 3D PIC simulation with high efficiency energy transfer and
generation of intense IAPs. Parametric scans of the plasma densities and
grating amplitudes allowed to identify the optimal result of IAPs(see the
supplement materials), and the optimal parameters are given in the Tab.S1 in
the supplement materials. In the simulation, a static window of
$200\lambda_{0}\times 200\lambda_{0}$ for the transverse x and y directions
and $250\lambda_{0}$ for the longitudinal z direction was applied. Each cell
had a size of $\lambda_{0}\times\lambda_{0}\times\lambda_{0}/100$ and
contained 16 particles. Both the pump and driving pulse beams had a 6-order
super Gaussian transverse distribution with a beam waist of 50 $\mu$m. The
output driving pulse has a FWHM duration of 330 as, and a peak intensity of
$\sim 33I_{0}$, as shown in Fig.2(c). The total conversion efficiency from the
pump to the IAP is about $0.1\%$, which is only 1/7 of the 1D case. This is
mainly due to the imperfect formation of IAPs at the beam edge where both the
intensity of the input driving pulse and the pump decrease, as shown in
Fig.2(b). The simulation results suggest that the efficiency could be enhanced
by employing a larger beam with a more uniform intensity distribution.

Varying the FEPG periods $\Lambda$, some typical features of PIG are observed
as described below: Firstly, the output driving pulse shifts towards regions
of higher average density as $\Lambda$ increases, with positions agreeing well
with those of the resonant points predicted by Eq.10, as demonstrated in
Fig.3(b) where several profiles of the driving pulses at different $\Lambda$
are gathered together. Secondly, the pulse amplitude rises with $\Lambda$,
mainly due to a stronger reflection of the pump at resonant regions with
higher average density, as depicted in Fig.3(b). Thirdly, the pulse duration
firstly swiftly decreases to attosecond and then increases again. As $\Lambda$
increases, the PIG gradually forms, enabling the generation of IAPs within the
range of $0.504\lambda_{0}\leq\Lambda\leq 0.507\lambda_{0}$. However, when the
resonant region is further shifted towards the fully ionized zone as $\Lambda$
increases, the range becomes wider due to a smaller density gradient.
Consequently, the pulse duration exhibits a rebound effect after achieving its
shortest value, as depicted in Fig.3(c). In experimental setups, the precise
adjustment of $\Lambda$ can be achieved by varying the incident angle of the
pump.

The frequency upconversion effect plays a significant role in the IAPs
generation since it can provide a much wider spectrum for the output laser
pulse. The PIC simulation shows that the central frequency of the IAP is
shifted to around 2$\omega_{0}$[see Fig.3(d)], and the pulse duration changes
to more than 100 fs with the frequency over $1.8\omega_{0}$ absent, indicating
the up-converted light is the key ingredient for IAPs. Moreover, we utilized
the analytical model given by Eq.14 to investigate the influence of the
upcoversion effect while it cannot be explicitly removed in the PIC
simulation. The analytical results with different $\Lambda$ using Eq.14 with
$L=8\lambda_{0}$ is presented in Fig.3(d) and Fig.3(e). They exhibit good
agreement with the PIC simulation in terms of waveform profiles and pulse
positions, and the central frequency of the IAP is also shifted to around
2$\omega_{0}$. Note that the pulse spectra remains constant in the analytical
model, whereas they vary in the PIC simulation with changing $\Lambda$. This
discrepancy can be attributed to the assumption of a linear plasma gradient in
the analytical model, whereas the PIC simulation incorporates a nonlinear
gradient. Despite this difference, the analytical result can provide a
qualitative investigation of the frequency upconversion effect. By disabling
the upconversion effect in the analytical model, the pulse duration increases
from hundreds of attosecond to over 1 fs, implying that it is indispensable
for IPAs.

Figure 3:  (a)Distribution of the electron density in the gradient FEPG.
(b)Electric field of the driving-pulses at different $\Lambda$ obtained by the
PIC simulation. (c)The shortest durations(FWHM) of the driving pulse with
various $\Lambda$.(d)Spectra of the driving-pulses at different grating
periods, and the analytical solution. (e)Wavefroms of the driving pulses at
different $\Lambda$ obtained by the analytical solution.

PIG is available in a wide parameter range as it can be formed whenever any
region along the gradient FEPG satisfies the resonant condition. To confirm
this, we summarize the simulation results with different average densities and
modulation depths in Tab.1(more details are given by Fig.S2 in the
supplemental material). The results show that IAPs can be obtained at a wide
range of average plasma densities(0.02$n_{c}$-0.5$n_{c}$) and modulation
depths($60\%{n_{e}}_{0}$-$10\%{n_{e}}_{0}$). As shown in Tab.1, the required
interaction length for IAP decrease with increasing average plasma density,
indicating that higher plasma densities are preferred for higher conversion
efficiency. However, the simulations reveal that pulse durations become
unstable and peak intensities decrease when ${n_{e}}_{0}>0.1n_{c}$. Thus,
${n_{e}}_{0}=0.1n_{c}$ is identified as the optimal plasma density.
Furthermore, IAPs can still be generated at a modulation depth as small as
$10\%{n_{e}}_{0}$, indicating that the creation of the required gas grating
for IAPs can be significantly simplified.

Table 1: Summary of 1D PIC simulation results with various average plasma density(${n_{e}}_{0}$) and modulation depths($\delta{n_{e}}_{0}/{n_{e}}_{0}$), where $L_{i}$ is the interaction length, $\tau$ is the FWHM duration of the driving pulse Case | I | II | III | IV | V | VI | VII
---|---|---|---|---|---|---|---
${n_{e}}_{0}(n_{c})$ | 0.02 | 0.05 | 0.1 | 0.5 | 0.1 | 0.1 | 0.1
$\delta{n_{e}}_{0}/{n_{e}}_{0}$ | 0.5 | 0.5 | 0.5 | 0.5 | 0.1 | 0.2 | 0.6
$\Lambda$ | 0.5016 | 0.5035 | 0.507 | 0.535 | 0.506 | 0.506 | 0.507
$L_{i}(mm)$ | 0.9 | 0.44 | 0.24 | 0.24 | 0.26 | 0.24 | 0.22
Peak $E(E_{0})$ | 4.7 | 5.4 | 5.75 | 4.4 | 2.6 | 4.4 | 5.5
$\tau$(fs) | 0.42 | 0.38 | 0.33 | 0.32 | 0.42 | 0.4 | 0.35
efficiency($\%$) | 0.15 | 0.39 | 0.7 | 1.3 | 0.14 | 0.48 | 0.72

A sufficiently short input driving pulse is required to form a narrow resonant
region. However, the pulse cannot be excessively short, or the resonant region
would be less than one period, rendering it inefficient for forming a Bragg
grating. In the PIC simulation, IAPs are obtained with the input pulse
durations $\tau_{in}$ ranging from 20 fs to 90 fs, as depicted in Fig.4(a).
The peak $E$ firstly increases and then drops as $\tau_{in}$ increases, with
the maximum value of 5.75$E_{0}$ at $\tau_{in}=30$ fs. However, as shown in
Eq.14 and Fig.3(b), the peak $E$ is also influenced by the grating period
$\Lambda$ which slightly varies with $\tau_{in}$ to obtain optimal IAPs.
Consequently, there is a dip of the peak $E$ at $\tau_{in}$ of 50-60 fs when
$\Lambda$ is relative smaller, as shown in Fig.4(b). In addition, a high
contrast is not required, given that the peak intensity of the input driving
pulse only slightly surpasses the ionization threshold of the background gas
(approximately $10^{14}~{}\rm{W/cm^{2}}$). The requisite driving pulse can be
readily provided by commercial Ti:sapphire CPA laser facilities.

Figure 4: Simulation results of the output FWHM duration $\tau$ (a) and the
peak $E$(b) of the driving pulse with various input FWHM durations
$\tau_{in}$.

In summary, a method for generating IAPs, referred to as the PIG mechanism,
has been proposed. This approach involves the reflection of a picosecond pump
pulse by a gradient FEPG where a sub-cycle gating is formed in the narrow
resonant region. In corporation with the frequency up-conversion effect of the
laser in the gradient plasma, it allows for the generation of sub-cycle IAPs.
The feasibility of the approach has been demonstrated through PIC simulations,
showing that IAPs can be generated across a wide range of average plasma
densities (0.02$n_{c}$-0.5$n_{c}$) and modulation depths
(60$\%{n_{e}}_{0}$-10$\%{n_{e}}_{0}$). The 3D simulation results obtains a
330-as IAP in a 240 $\mu$m-long, $0.1n_{c}$-density FEPG, with a conversion
efficiency of $\sim 0.1\%$. Using a 1-kJ, 1.6-ps, 1053-nm laser pulse produced
by chirped pulse amplificationStrickland and Mourou (1985); Mourou  _et al._
(2012) as the pump, would allow to generate joule-level IAPs.

This work was partly supported by National Key Program for S$\&$T Research and
Development (Grant No. 2018YFA0404804), and the Science and Technology on
Plasma Physics Laboratory Fund(Grant No. 6142A04220203 and 6142A04220204)

## References

  * Calegari  _et al._ (2016) F. Calegari, G. Sansone, S. Stagira, C. Vozzi, and M. Nisoli, Journal of Physics B: Atomic, Molecular and Optical Physics 49, 062001 (2016).
  * Chini, Zhao, and Chang (2014) M. Chini, K. Zhao, and Z. Chang, Nat. Photonics 8, 178 (2014).
  * Hentschel  _et al._ (2001) M. Hentschel, R. Kienberger, C. Spielmann, G. A. Reider, N. Milosevic, T. Brabec, P. Corkum, U. Heinzmann, M. Drescher, and F. Krausz, Nature 414, 508 (2001).
  * Sansone  _et al._ (2006) G. Sansone, E. Benedetti, F. Calegari, C. Vozzi, L. Avaldi, R. Flammini, L. Poletto, P. Villoresi, C. Altucci, R. Velotta, S. Stagira, S. D. Silvestri, and M. Nisoli, Science 314, 443 (2006).
  * Goulielmakis  _et al._ (2004) E. Goulielmakis, M. Schultze, M. Hofstetter, V. S. Yakovlev, J. Gagnon, M. Uiberacker, A. L. Aquila, E. M. Gullikson, D. T. Attwood, R. Kienberger, F. Krausz, and U. Kleineberg, Nature 427, 817 (2004).
  * Gaumnitz  _et al._ (2017) T. Gaumnitz, A. Jain, Y. Pertot, M. Huppert, I. Jordan, F. Ardana-Lamas, and H. Worner, Opt. Express 25, 27506 (2017).
  * Witting  _et al._ (2012) T. Witting, F. Frank, W. A. Okell, C. A. Arrell, J. P. Marangos, and J. W. G. Tisch, Journal of Physics B: Atomic, Molecular and Optical Physics 45, 074014 (2012).
  * Corkum, Burnett, and Ivanov (1994) P. B. Corkum, N. H. Burnett, and M. Y. Ivanov, Opt. Lett. 19, 1870 (1994).
  * Li  _et al._ (2017) J. Li, X. Ren, Y. Yin, K. Zhao, A. Chew, Y. Cheng, E. Cunningham, Y. Wang, S. Hu, Y. Wu, M. Chini, and Z. Chang, Nat. Commun. 8, 186 (2017).
  * Z (2004) C. Z, Phys. Rev. A 70, 043802 (2004).
  * Z (2005) C. Z, Phys. Rev. A 71, 023813 (2005).
  * Sola  _et al._ (2006) I. J. Sola, E. Mével, L. Elouga, E. Constant, V. Strelkov, L. Poletto, P. Villoresi, E. Benedetti, J.-P. Caumes, S. Stagira, C. Vozzi, G. Sansone, and M. Nisoli, Nat. Phys. 9, 319 (2006).
  * Zhao  _et al._ (2012) K. Zhao, Q. Zhang, M. Chini, Y. Wu, X. Wang, and Z. Chang, Opt. Lett. 37, 3891 (2012).
  * Chang (2007) Z. Chang, Phys. Rev. A. 76, 051403(R) (2007).
  * Mashiko  _et al._ (2008) H. Mashiko, S. Gilbertson, C. Li, S. D. Khan, M. M. Shakya, E. Moon, and Z. Chang, Phys Rev Lett 100, 103906 (2008).
  * Vincenti and Quéré (2012) H. Vincenti and F. Quéré, Phys Rev Lett 108, 113904 (2012).
  * Kim  _et al._ (2013) K. T. Kim, C. Zhang, T. Ruchon, J.-F. Hergott, T. Auguste, D. M. Villeneuve, P. B. Corkum, and F. Quóró, Nat. Photonics 7, 651 (2013).
  * Heyl  _et al._ (2014) C. M. Heyl, S. N. Bengtsson, S. Carlstroöm, J. Mauritsson, C. L. Arnold, and A. LH̀uillier;, New J Phys 16, 052001 (2014).
  * Zhong  _et al._ (2016) S. Zhong, X. He, Y. Jiang, H. Teng, P. He, Y. Liu, K. Zhao, and Z. Wei, Phys. Rev. A 93, 033854 (2016).
  * Tzallas  _et al._ (2007) P. Tzallas, E. Skantzakis, C. Kalpouzos, E. P. Benis, G. D. Tsakiris, and D. Charalambidis, Nat. Phys. 3, 846 (2007).
  * Feng  _et al._ (2009) X. Feng, S. Gilbertson, H. Mashiko, H. Wang, S. D. Khan, M. Chini, Y. Wu, K. Zhao, and Z. Chang, Phys. Rev. Lett. 103, 183901 (2009).
  * Skantzakis  _et al._ (2009) E. Skantzakis, P. Tzallas, J. Kruse, C. Kalpouzos, and D. Charalambidis, Opt. Lett. 34, 1732 (2009).
  * H, K, and T (2013) M. H, O. K, and S. T, Appl Phys Lett, 102, 171111 (2013).
  * Huang  _et al._ (2011) S.-W. Huang, G. Cirmi, J. Moses, K.-H. Hong, S. Bhardwaj, J. R. Birge, L.-J. Chen, E. Li, B. J. Eggleton, G. Cerullo, and F. X. Körtner, Nat. Photonics 5, 475 (2011).
  * Hassan  _et al._ (2016) M. T. Hassan, T. T. Luu1, A. Moulet, O. Raskazovskaya, P. Zhokhov, M. Garg, N. Karpowicz, V. P. A. M. Zheltikov3, 4, F. Krausz, and . E. Goulielmakis, Nature 530, 66 (2016).
  * Rossi  _et al._ (2020) G. M. Rossi, R. E. Mainz, Y. Yang, F. Scheiba, M. A. Silva-Toledo, S.-H. Chia, P. D. Keathley, S. Fang, O. D. Mücke, C. Manzoni, G. Cerullo, G. Cirmi, and F. X. Körtner, Nat. Photonics 14, 629 (2020).
  * Malkin, Shvets, and Fisch (1999) V. M. Malkin, G. Shvets, and N. J. Fisch, Phys. Rev. Lett 82, 4448 (1999).
  * Malkin, Shvets, and Fisch (2000) V. M. Malkin, G. Shvets, and N. J. Fisch, Phys. Rev. Lett 84, 1208 (2000).
  * Trines  _et al._ (2011) R. M. G. M. Trines, F. Fiuza, R. Bingham, R. A. Fonseca, L. O. Silva, R. A. Cairns, and P. A. Norreys1, Nat. Phys. 7, 87 (2011).
  * Ping  _et al._ (2004) Y. Ping, W. Cheng, S. Suckewer, D. S. Clark, and N. J. Fisch, Phys. Rev. Lett 92, 175007 (2004).
  * Cheng  _et al._ (2005) W. Cheng, Y. Avitzour, Y. Ping, S. Suckewer, N. J. Fisch, M. S. Hur, and J. S. Wurtele, Phys. Rev. Lett 94, 045003 (2005).
  * Ren  _et al._ (2007) J. Ren, W. Cheng, S. Li, and S. Suckewer, Nat. Phys. 3, 732 (2007).
  * Turnbull  _et al._ (2018) D. Turnbull, P. Franke, J. Katz, J. P. Palastro, I. A. Begishev, R. Boni, J. Bromage, A. L. Milder, J. L. Shaw, and D. H. Froula, Phys. Rev. Lett. 120, 225001 (2018).
  * Wu  _et al._ (2019) Z. Wu, Q. Chen, A. Morozov, and S. Suckewer, Phys. Plasmas 26, 103111 (2019).
  * Andreev  _et al._ (2006) A. A. Andreev, C. Riconda, V. T. Tikhonchuk, and S. Weber, Phys. Plasma 13, 053110 (2006).
  * Weber  _et al._ (2013) S. Weber, C. Riconda, L. Lancia, J.-R. Marques̀, G. A. Mourou, and J. Fuchs, Phys. Rev. Lett. 111, 055004 (2013).
  * Lancia  _et al._ (2010) L. Lancia, J. R. Marquès, M. Nakatsutsumi, C. Riconda, S. Weber, S. Hüller, A.Mančić, P. Antici, V. T. Tikhonchuk, A. Héron, P. Audebert, and J. Fuchs, Phys. Rev. Lett. 104, 025001 (2010).
  * Lancia  _et al._ (2016) L. Lancia, A. Giribono, L. Vassura, M. Chiaramello, C. Riconda, S. Weber, A. Castan, A. Chatelain, A. Frank, T. Gangolf, M. N. Quinn, J. Fuchs, and J.-R. Marquès, Phys. Rev. Lett. 116, 075001 (2016).
  * Marqués  _et al._ (2019) J.-R. Marqués, L. Lancia, T. Gangolf, M. Blecher, S. Bolanos, J.Fuchs, O. Willi, F. Amiranoff, R. L. Berger, M. Chiaramello, S. Weber, and C. Riconda, Phys. Rev. X 9, 021008 (2019).
  * Wu  _et al._ (2022) Z. Wu, X. Zeng, Z. Li, Z. Zhang, X. Wang, B. Hu, X. Wang, J. Mu, J. Su, Q. Zhu, X. Wei, , and Y. Zuo, Matter Radiate Extreme 7, 064402 (2022).
  * Li  _et al._ (2022) Z. Li, Y. Zuo, X. Zeng, Z. Wu, X. Wang, X. Wang, J. Mu, and B. Hu, Matter Radiate Extreme 8, 014401 (2022).
  * Damzen  _et al._ (2003) M. Damzen, V. Vlad, V. Babin, and A. Mocofanescu,  _Stimulated Brillouin Scattering Fundamentals and Applications_ (Institute of Physics Publishing, London, 2003).
  * Gorbunov  _et al._ (1983) V. A. Gorbunov, S. B. Papernyi, V. F. Petrov, and V. R. Startsev, Sov. Quantum Electronics 13, 900 (1983).
  * Yang, Gygerand, and Thévenaz (2020) F. Yang, F. Gygerand, and L. Thévenaz, Nat. Photonics 14, 700 (2020).
  * Carr and Hanna (1985) I. D. Carr and D. C. Hanna, Applied Physics B: Lasers and Optics 36, 83 (1985).
  * Meng and H.J.Eichler (1991) H. Meng and H.J.Eichler, Opt. Lett. 16, 569 (1991).
  * Zhang  _et al._ (2021) C. Zhang, Z. Nie, Y. Wu, M. Sinclair, C.-K. Huang, K. A. Marsh, and C. Joshi, Plasma Physics and Controlled Fusion 63, 095011 (2021).
  * Qu  _et al._ (2018) K. Qu, Q. Jia, M. R. Edwards, and N. J. Fisch, Phys. Rev. E 98, 023202 (2018).
  * K.Qu and Fisch (2019) K.Qu and N. J. Fisch, Phys. Rev. E 99, 063201 (2019).
  * Howard  _et al._ (2019) A. J. Howard, D. Turnbull, A. S. Davies, P. Franke, D. H. Froula, and J. P. Palastro, Phys. Rev. Lett. 123, 124801 (2019).
  * Esarey, G.Joyce, and Sprangle (1991) E. Esarey, G.Joyce, and P. Sprangle, Phys. Rev. A 44, 3908 (1991).
  * H.Peng  _et al._ (2021) H.Peng, C. Riconda, S.Weber, C. T. Zhou, and S. C. Ruan, Phys. Rev. Applied 15, 054053 (2021).
  * Arber  _et al._ (2015) T. D. Arber, K. Bennett, C. S. Brady, A. Lawrence-Douglas, M. G. Ramsay, N. J. Sircombe, P. Gillies, R. G. Evans, H. Schmitz, A. R. Bell, and C. P. Ridgers, Nsma Phys. Controlled Fusion 57, 113001 (2015).
  * Ammosov, Delone, and Krainov (1986) M. Ammosov, N. Delone, and V. Krainov, Sov. Phys. JETP 64, 1991 (1986).
  * Strickland and Mourou (1985) D. Strickland and G. Mourou, Opt. Commun. 56, 219 (1985).
  * Mourou  _et al._ (2012) G. A. Mourou, N. J. Fisch, V. M. Malkin, Z. Toroker, E. A. Khazanov, A. M. Sergeev, T. Tajima, and B. L. Garrec, Opt. Commun. 285, 720 (2012).