[affiliation=]Van TungPham [affiliation=]YistLin [affiliation=]TaoHan
[affiliation=]WeiLi [affiliation=]JunZhang [affiliation=]LuLu
[affiliation=]YuxuanWang

# A Comprehensive Solution to Connect Speech Encoder and Large Language Model
for ASR

###### Abstract

Recent works have shown promising results in connecting speech encoders to
large language models (LLMs) for speech recognition. However, several
limitations persist, including limited fine-tuning options, a lack of
mechanisms to enforce speech-text alignment, and high insertion errors
especially in domain mismatch conditions. This paper presents a comprehensive
solution to address these issues. We begin by investigating more thoughtful
fine-tuning schemes. Next, we propose a matching loss to enhance alignment
between modalities. Finally, we explore training and inference methods to
mitigate high insertion errors. Experimental results on the Librispeech corpus
demonstrate that partially fine-tuning the encoder and LLM using parameter-
efficient methods, such as LoRA, is the most cost-effective approach.
Additionally, the matching loss improves modality alignment, enhancing
performance. The proposed training and inference methods significantly reduce
insertion errors.

###### keywords:

speech recognition, pretrained speech encoder, parameter efficient,
hallucination, modalities matching

##  1 Introduction

Recently, large language models (LLM), characterised by their billions of
parameters and training on massive datasets, have demonstrated emergent
abilities to address various tasks in natural language processing field.
Meanwhile, foundation models also advanced state-of-the-art in other research
fields such as speech processing [1, 2] and computer vision [3, 4]. Hence, it
is natural to develop effective approaches that unify foundation models from
different modalities to build strong speech and visual understanding ability.
In this work, we aim to connect a speech foundation model to an LLM for speech
recognition.

The architecture of LLM-based ASR generally consists of three components: a
speech encoder, an adapter, and an LLM. Recent works in LLM-based ASR examined
techniques to compress the output of speech encoder [5, 6, 7], design
different adapters [7, 8, 9], fine-tune the LLM partially [6] or keep it
unchanged [7]. Despite these efforts, several issues still persist. Firstly,
existing works only fine-tuned certain modules with specific configurations,
which might exclude many efficient settings. Secondly, there is no method to
explicitly force the representations generated by the adapter to be in a
similar space, i.e. aligned, to those of LLM embeddings, although [6] shown
that these representations are aligned to some extent after ASR fine-tuning.
Finally, we observe very high insertion errors when the model is trained with
limited training data or when test sets are mismatched with the training set -
a problem that has not been studied before.

In this work, we present a comprehensive solution to address aforementioned
issues. We first perform an empirical study about more thoughtful fine-tuning
settings, each corresponding to different fine-tune options for each module,
to identify efficient settings. We then propose a matching loss on top of a
cross attention to explicitly force the two modalities to align with each
other, leading to better ASR performance. Finally, we propose several training
and inference strategies to address the high insertion problem. Specifically,
for inference, we apply n-gram non-repetition and length penalty constraints
during beam search decoding. In terms of training, we propose using a non-
speech corpus with empty transcript for fine-tuning and data augmentations
based on speed and volume perturbations.

The paper is organized as follows. Section 2 presents related works. In
Section 3, we describe our proposed solution, followed by experimental setup
in Section 4. Section 5 presents experimental results and analysis, then we
conclude our work in Section 6.

##  2 Related works

Recently published works have extended LLMs with the ability to ingest other
modalities, such as audio [5, 6, 7, 8, 10, 11] and vision [4, 12]. Regarding
audio processing, LLM-based models have been studied for diverse tasks such as
automatic speech recognition (ASR) [5, 6, 7, 8, 13], speech translation [14,
15], audio event detection and understanding [16], etc. There are two
paradigms for these models. The first one [10, 11, 15] converts continuous
audio into discrete tokens and then merges text and audio tokens into a shared
vocabulary while the second one [6, 7, 8] directly connects and adapts
continuous representation of pretrained audio encoder into LLM. Although the
first strategy is able to generate speech outputs from text or speech inputs
by using codec-based discrete tokens, it may suffer from the information loss
caused by discretization process hence might not be optimal for the speech-to-
text task. In this work, we focus on the ASR task and we follow the second
paradism.

Due to the immense number of parameters in LLM models, it can be
computationally impractical to adapt the whole LLM to a target task. Several
approaches have been proposed to address this issue include: inserting adapter
layers [17] or prefix embeddings [18] which are trained on target tasks. While
these approaches are parameter-efficient, they increase the inference costs.
Low-rank Adaptation (LoRa) [19] solves these issues by using low-rank matrices
which are memory efficient during training and does not impact inference time.

##  3 Proposed solution

We first investigate more comprehensive fine-tuning schemes in Section 3.1. In
Section 3.2, we introduce the proposed matching loss to improve alignment
between modalities, followed by training and inference techniques to address
the insertion issue in Section 3.3.

###  3.1 Fine-tuning schemes for LLM-based ASR

Existing works only fine-tuned certain modules of LLM-based ASR models with
limited configurations. For example, SLM [7] only trained an adapter while
keeping the other modules unchanged; whereas in [6], LoRa was used in the LLM
and fine-tuned together with the adapter and entire encoder. It is also worth
noting that existing works either fully fine-tuned the encoder or kept it
frozen, ignoring the option for partial fine-tuning, which is more practical
given the immense number of parameters in many current speech encoders.

In this work, we investigate more thoughtful fine-tuning schemes with more
options for each module. Specifically,

  * •

The LLM module typically has a very large size, thus it is impossible to fully
fine-tune it using popular deep learning frameworks. Therefore, we consider
freezing the LLM or fine-tuning it using a popular parameter-efficient method,
e.g. LoRa [19].

  * •

Foundation encoders such as HuBert [1] or wav2vec2 [20] can be fully fine-
tuned on recent advanced GPUs, such as the A100, which have high memory
capacity. Hence, we consider three fine-tuning options for the encoder:
frozen, partially fine-tuned with LoRa, and fully fine-tuned.

  * •

For the adapter, we use a fully connected layer adapter, which is similar to
[8]. Specifically, it consists of a 1D convolution layer for subsampling,
followed by the GeLu operation then a linear projection to have the same
dimension as LLM embedding space. We denote this adapter as Conv1dMLP. Besides
Conv1dMLP, we also consider different architectures for the adapter. Each of
them requires a different number of parameters and has different expressive
capacity. The first variant, denoted as DwsMLP, replaces a standard 1D
convolution with a simple depthwise separable CNN, which requires fewer
parameters. The second variant, denoted as Conv1dTransformer, uses Transformer
[21] layers after subsampling (similar to [7]), which generally have high
expressive capacity but require substantially more parameters.

###  3.2 Improving modalities alignment by matching loss

Past work [6] has shown that the representation generated by the adapter is
aligned, to some extent, with the LLM embeddings of the text. This alignment
is implicitly achieved by fine-tuning the model using the ASR task, and there
is no mechanism to explicitly force the two modalities to be aligned.

In this work, we propose a matching loss to explicitly align these modalities,
which is not a trivial problem since the two sequences have different lengths.
To overcome this challenge, we first apply cross attention, such as dot-
product attention [21], between the text embedding and acoustic embedding
sequences. This generates a sequence of acoustic representations that has the
same length as the text embedding sequence. After that, we can easily apply
standard loss functions such as mean square error (MSE) or Cosine distance
between the text embedding and the newly generated acoustic representations.

Formally, let us first denote $<$X, Y$>$ as a training utterance, where X is a
sequence of acoustic features and
$\textbf{Y}=\\{y_{1},y_{2},...,y_{|\textbf{Y}|}\\}$ is a sequence of output
text. Then, the matching loss, denoted as $L_{m}$, can be described as follows

$\displaystyle\textbf{E}_{\textbf{Y}}$ $\displaystyle=Emb(y_{1})\
Emb(y_{2})...Emb(y_{|\textbf{Y}|})$ (1) $\displaystyle\textbf{X}_{1}$
$\displaystyle=Adapt(Enc(\textbf{X}))$ (2) H
$\displaystyle=CrossAtt(\textbf{E}_{\textbf{Y}},\textbf{X}_{1})=softmax(\frac{\textbf{E}_{\textbf{Y}}\textbf{X}_{1}^{T}}{\sqrt{d_{llm}}})\textbf{X}_{1}$
(3) $\displaystyle L_{m}$ $\displaystyle=a\
MSE(\textbf{E}_{\textbf{Y}},\textbf{H})+b\
Cosine(\textbf{E}_{\textbf{Y}},\textbf{H})\vspace{-0.3cm}$ (4)

where $Emb$, $Enc$, $Adapt$ are LLM embedding, encoder and adapter functions
respectively; $d_{llm}$ is the LLM hidden dimension; $a$ and $b$ are tunable
hyper-parameters. The model is then trained using a combination of cross-
entropy and $L_{m}$ losses.

###  3.3 Inference and training methods to reduce insertion errors

When analyzing model inference using the standard beam search algorithm, we
found that models trained with a small amount of data, i.e. 10h, produced
relatively high insertion errors. More importantly, when testing the
robustness of our model on several out-of-domain test sets, we observed more
severe insertion problems. Specifically, in these cases, models repeat an
n-gram until reaching the output length limit.

To alleviate insertion errors, one straightforward approach is to apply
following constraints during beam search inference:

  * •

Apply n-gram non-repetition constraint (denoted as NRNS): This constraint
ensures that n-gram tokens of a specified length, e.g., 5, do not repeat
during decoding.

  * •

Apply length penalty (denoted as LP): This method imposes a greater penalty on
long decoding transcripts, thereby reducing insertion errors.

Note that we utilize a publicly available LLM model from Hugging Face, where
the above constraints are readily provided as options, namely,
'no_repeat_ngram_size' and 'length_penalty'
111https://huggingface.co/docs/transformers/en/main_classes/text_generation.

We observed that after applying inference constraints, insertion errors still
persist substantially on out-of-domain test sets. We also noticed that the
repetition problem frequently occurs in audio trunks containing only non-
speech signals such as music or noise. We propose to apply the following
training methods to address the insertion issue.

  * •

Data augmentation (referred to as DA): To enhance the system's robustness
against variations in acoustic conditions and speaking styles, we perform data
augmentation to introduce diverse acoustic conditions during training. Our
data augmentation includes volume perturbation and speed perturbation, which
are applied with certain probabilities. We refrain from adding random noise or
music to training utterances, as we observed that it negatively impacts
performance.

  * •

Fine-tuning a pre-trained model using non-speech segments with empty
transcripts (referred to as NSET): Ideally, a model should not generate any
output for audio trunks containing non-speech signals. We achieve this by
augmenting the ASR training data with a non-speech corpus containing audio
segments with empty transcripts. Subsequently, we fine-tune already trained
models for a short duration.

##  4 Experimental setup

###  4.1 Dataset

We conducted experiments using the LibriSpeech corpus [22]. In addition to
training on the full 960h dataset, we trained models on two subsets: train-
clean 100h and 10h. We report Word Error Rate (WER) results on the LibriSpeech
dev-clean, dev-other, test-clean, and test-other sets. To assess model
robustness, we also report results on two out-of-domain test sets: CoVoST2
[23] and GigaSpeech [24]. For NSET training, we utilized the noise and music
subsets of the Musan [25] corpus to construct the non-speech corpus.

###  4.2 Model settings

We utilized the hubert-large-ll60k model [1] (300M parameters) and the Vicuna
vicuna-7b-v1.5 model [26] (7B parameters) from the Hugging Face website as the
speech encoder and LLM, respectively. The encoder and LLM have hidden
dimensions of 1024 and 4096, respectively. For the adapter, the 1D convolution
has input and output channels of 1024 and 4096, respectively, and performs 8
times subsampling. The linear transformations in Conv1dMLP and DwsMLP have
input and output dimensions of 4096. Consequently, Conv1dMLP has 48M
parameters, while DwsMLP employs depthwise separable convolution, resulting in
only 20M parameters. Conv1dTransformer, however, employs 2 layers of
Transformer with a hidden dimension of 4096 and FFN dimension 2.5x larger,
i.e., 10240, similar to [7]. Consequently, Conv1dTransformer has 320M
parameters, significantly more than the other two adapter types.

For LoRa of the encoder, we implement $\\{r=8,\alpha=16\\}$ on the query and
value matrices of the self-attention module at each layer, resulting in 0.65M
parameters. As for the LLM, we employ $\\{r=16,\alpha=16\\}$ across all query,
key, and value matrices of the self-attention, yielding 16M parameters.

Table 1 presents all fine-tuning schemes, each corresponding to a specific
configuration of each module. Due to the large number of configurations, we
only explore adapter variants under two conditions: (1) with both encoder and
decoder frozen, similar to [7]; (2) with both encoder and decoder fine-tuned
using the LoRa method.

Table 1: Different fine-tuning configurations SID | Encoder | Adapter | LLM | Params
---|---|---|---|---
S1 | Frozen | Conv1dMLP | Frozen | 48M
S2 | Frozen | Conv1dMLP | LoRa | 64M
S3 | LoRa | Conv1dMLP | Frozen | 49M
S4 | LoRa | Conv1dMLP | LoRa | 65M
S5 | Full | Conv1dMLP | Frozen | 345M
S6 | Full | Conv1dMLP | LoRa | 361M
S7 | Frozen | DwsMLP | Frozen | 20M
S8 | Frozen | Conv1dTransformer | Frozen | 320M
S9 | LoRa | DwsMLP | LoRa | 37M
S10 | LoRa | Conv1dTransformer | LoRa | 337M

###  4.3 Training and inference setting

We trained our models on A100 GPUs for 50k, 20k, and 10k steps for 960h, 100h,
and 10h datasets, respectively. Checkpoints were saved at every 1k steps for
the 960h and 100h data and every 500 steps for the 10h data. Following
training, we selected five consecutive checkpoints with the best averaged
validation loss and averaged them for evaluation.

In our initial experiments on the matching loss, we found that the best
results were achieved with {a = 0.01, b = 0.04}, hence we use this setting for
all experiments in Section 3.2. For inference, we employed beam search with
default settings from Hugging Face, i.e. {beam_size = 5, max_length = 256,
NRNS = 0, LP = 1.0}

Table 2: WER results of different fine-tuning configurations (a) and matching
loss (b)

| 960h training data | 100h training data | 10h training data
---|---|---|---
SID | dev-clean | dev-other | test-clean | test-other | dev-clean | dev-other | test-clean | test-other | dev-clean | dev-other | test-clean | test-other
S1 | 2.38 | 5.99 | 2.29 | 5.67 | 3.55 | 9.35 | 4.23 | 8.69 | 11.96 | 22.55 | 14.51 | 20.60
S2 | 2.01 | 4.71 | 2.02 | 4.31 | 3.42 | 6.34 | 3.45 | 7.19 | 13.52 | 21.45 | 13.30 | 18.75
S3 | 1.87 | 4.61 | 1.79 | 3.91 | 2.86 | 5.33 | 2.70 | 5.99 | 7.36 | 12.49 | 6.74 | 11.72
S4 | 1.70 | 3.59 | 1.78 | 3.62 | 2.68 | 5.64 | 2.57 | 5.27 | 7.36 | 11.92 | 7.44 | 11.49
S5 | 1.72 | 3.96 | 1.70 | 3.58 | 2.51 | 4.87 | 2.86 | 5.14 | 6.27 | 10.25 | 5.75 | 10.40
S6 | 1.50 | 3.21 | 1.74 | 3.16 | 2.40 | 4.55 | 2.28 | 4.92 | 6.43 | 8.08 | 4.97 | 9.11
S7 | 3.34 | 7.98 | 4.26 | 7.24 | 3.65 | 7.22 | 4.19 | 9.09 | 12.12 | 23.51 | 12.30 | 19.96
S8 | 1.96 | 4.13 | 2.02 | 4.41 | 3.91 | 7.31 | 4.34 | 8.25 | 21.63 | 31.06 | 25.96 | 34.72
S9 | 1.61 | 4.34 | 1.59 | 3.89 | 2.72 | 5.31 | 2.66 | 5.65 | 7.23 | 13.56 | 8.12 | 12.22
S10 | 1.88 | 3.83 | 1.85 | 3.77 | 3.62 | 5.88 | 3.80 | 5.90 | 13.30 | 19.22 | 13.61 | 18.55

(a) (a) Results of different fine-tuning configurations

| 960h training data | 100h training data | 10h training data
---|---|---|---
SID | dev-clean | dev-other | test-clean | test-other | dev-clean | dev-other | test-clean | test-other | dev-clean | dev-other | test-clean | test-other
S1 | 2.38 | 5.99 | 2.29 | 5.67 | 3.55 | 9.35 | 4.23 | 8.69 | 11.96 | 22.55 | 14.51 | 20.60
T1 | 2.38 | 5.33 | 2.36 | 4.81 | 4.32 | 7.31 | 3.93 | 6.93 | 9.67 | 15.93 | 12.40 | 15.02
S4 | 1.70 | 3.59 | 1.78 | 3.62 | 2.68 | 5.64 | 2.57 | 5.27 | 7.36 | 11.92 | 7.44 | 11.49
T4 | 1.63 | 3.47 | 1.63 | 3.48 | 2.58 | 5.62 | 2.59 | 5.26 | 6.83 | 10.34 | 7.29 | 10.60

(b) (b) Results with and without matching loss. S1 and S4 are systems without
matching loss while T1 and T4 are systems with matching loss

##  5 Experimental results and analysis

###  5.1 Results of different fine-tuning scheme

Table 2 (a) shows the results of different fine-tuning configurations which
can be summarized as follows:

  * •

Applying LoRa to LLM significantly enhances performance across most test sets,
especially on dev/test-other. For instance, on the 960h condition, S2
surpasses S1 by 21.4%/24.0% relative WER reduction (WER reduced from
5.99%/5.67% to 4.71%/4.31%) on dev/test-other and by 15.6%/11.8% (WER reduced
from 2.38%/2.29% to 2.01%/2.02%) on dev/test-clean subsets. We posit that the
dev/test-other subsets present challenging cases in terms of both acoustic and
linguistic conditions, and fine-tuning the LLM enables better adaptation to
the LibriSpeech domain.

  * •

In the encoder module, full fine-tuning yields the best outcomes, followed by
partial fine-tuning with LoRa across all conditions. However, partial fine-
tuning proves to be the most cost-effective. For instance, on 960h condition,
partial fine-tuning S4 setting significantly outperforms the frozen setting S2
(WER reduced from 2.01%/4.71% to 1.70%/3.59% on dev-clean/other) while only
adding 0.65M extra parameters.

  * •

Regarding adapter achitectures, the Conv1dTransformer clearly outperforms
Conv1dMLP when both encoder and LLM are frozen and trained with 960h data, but
does not exhibit benefit in other conditions. The other variant, DwsMLP, is
generally worse than Conv1dMLP in most of conditions.

In summary, fine-tuning both the encoder and LLM with LoRa and using Conv1dMLP
as the adapter (S4 setting) achieves the optimal balance between performance
and additional parameters. Thus, we employ S4 for subsequent experiments.
Additionally, we utilize S1 for further experiments due to its crucial
advantage: preserving all functionalities of the foundation encoder and LLM,
which may be critical for certain applications.

###  5.2 Results of matching loss

Table 2(b) presents the results of systems with and without matching loss. S1
and S4 are systems without matching loss taken from Table 2 (a), while T1 and
T4 are corresponding systems with the matching loss. The matching loss
enhances performance across most conditions, with a more pronounced
improvement observed in S1 compared to S4. This discrepancy may stem from the
LoRa modules in both encoder and LLM components, which already aid in aligning
acoustic and text representations, thereby diminishing the benefit from the
matching loss. Notably, the improvement on dev/test-other subsets exceeds that
of dev/test-clean. This discrepancy can be attributed to the presence of
challenging cases in dev/test-other, characterized by higher modality
mismatch, which the matching loss mitigates, resulting in improved
performance. We then take the best model, i.e. T4, for experiments in the next
section.

###  5.3 Results of inference and training techniques

We analyze the impact of NRNS and LP constraints on T4 models trained with
960h (full data) and 10h (low resource) on dev sets. We observe a consistent
trend in the results of both dev-clean and dev-other, so we present only the
results for dev-other in Table 3. We also report the insertion error rate
(IER) in some cases for analysis. Each constraint helps to reduce IER in the
10h conditions. For instance, NRNS = 10 reduces IER from 1.88% to 0.89% on
dev-other, resulting in a WER improvement from 10.35% to 9.35%. However,
combining these constraints does not yield additional benefits, suggesting
that LP and NRNS may not be complementary. Overall, the optimal setting is
NRNS = 10, LP = 0.

Table 3: Effect of NRNS and LP on WER and IER of dev-other. IER are showed in
parentheses

Data | 0 | 5 | 10 | 15
---|---|---|---|---
960h | 3.47 (0.33) | 3.46 | 3.44 (0.33) | 3.44
10h | 10.34 (1.88) | 9.36 | 9.35 (0.89) | 9.46

(a) (a) Effect of NRNS constraint on WER and IER

Data | NRNS | 1.0 | 0.5 | 0 | -0.5
---|---|---|---|---|---
960h | 0 | 3.47 (0.33) | 3.49 (0.32) | 3.50 | 3.50
10 | 3.44 (0.33) | 3.46 (0.33) | 3.47 | 3.49
10h | 0 | 10.34 (1.88) | 9.44 (0.94) | 9.61 | 9.71
10 | 9.35 (0.89) | 9.35 (0.89) | 9.51 | 9.61

(b) (b) Effect of the LP constraint on WER and IER

We assess the above constraints on test-clean, test-other, and two out-of-
domain test sets. Results are summarized in Table 4. Applying constraints
reduces IER and consequently enhances performance on in-domain test sets under
the 10h training condition. Importantly, applying constraints significantly
reduces IER on out-of-domain test sets for both 960h and 10h conditions. For
instance, for the model trained with 960h, IER decreases from 33.06% to 14.52%
in GigaSpeech, leading to a WER reduction from 46.42% to 27.83%.

Table 4: WER and IER (showed in parentheses) on different test sets without and with contraints during inference. Data | Constraint | test-clean | test-other | CoVoST2 | GigaSpeech
---|---|---|---|---|---
960h | No | 1.63 (0.15) | 3.48 (0.40) | 31.47 (16.94) | 46.42 (33.06)
Yes | 1.63 (0.15) | 3.49 (0.40) | 19.73 (5.21) | 27.83 (14.52)
10h | No | 7.29 (1.51) | 10.60 (1.83) | 47.89 (24.80) | 59.67 (39.02)
Yes | 6.37 (0.61) | 9.73 (1.00) | 33.09 (10.07) | 44.39 (23.86)

We examine the effects of training methods: DA and NSET, summarized in Table
5. DA significantly improves performance across all conditions, notably in
low-resource training and out-of-domain evaluation scenarios. Notably, in
scenarios like 960h training and testing on GigaSpeech, the reduction in IER
is close to that of WER, indicating that most of the WER benefit stems from
reduced insertion errors. Moreover, DA not only reduces insertion errors but
also leads to fewer deletion/substitution errors in test-clean and test-other,
particularly in the 10h condition. NSET training remarkably reduces insertion
errors in out-of-domain test sets. However, this method sometime increases
deletion errors resulting in higher WER, particularly in the 10h training
condition. Although the combination of DA and NSET does not always yield the
lowest insertion errors, it generally results in better WER due to fewer
deletion errors. Overall, DA+NSET demonstrates increased robustness while
effectively suppressing insertion errors, as demonstrated in an example from
CoVoST2 corpus in Table 6.

Table 5: WER and IER (showed in parentheses) on different test sets with different training methods Data | Methods | test-clean | Test-other | CoVoST2 | GigaSpeech
---|---|---|---|---|---
960h | default | 1.63 (0.15) | 3.49 (0.40) | 19.73 (5.21) | 27.83 (14.52)
DA | 1.61 (0.15) | 3.37 (0.39) | 18.93 (4.67) | 23.90 (10.81)
NSET | 1.70 (0.16) | 3.53 (0.39) | 19.05 (3.77) | 16.78 (3.22)
DA+NSET | 1.67 (0.15) | 3.48 (0.41) | 18.40 (3.79) | 16.58 (3.33)
10h | default | 6.37 (0.61) | 9.73 (1.00) | 33.09 (10.07) | 44.39 (23.86)
DA | 5.36 (0.46) | 8.65 (0.83) | 31.25 (9.20) | 40.73 (21.51)
NSET | 7.05 (0.60) | 10.57 (1.06) | 32.84 (8.44) | 26.36 (5.56)
DA+NSET | 5.80 (0.48) | 9.13 (0.94) | 31.40 (8.02) | 26.87 (6.84)
Table 6: Decoding hypotheses generated by models trained with different methods for the utterance common_voice_en_37177 ref / hyps | Text
---|---
reference | ican't stand the sight of blood
hyp - default |  | or ican't stand the sight of blood or ican't
---
stand the site of blood … or ican't stand
the sight of blood
hyp - DA | ican't stand the sight of blood i
hyp - NSET | <empty>
hyp - DA+NSET | ican't stand the sight of blood

##  6 Conclusion

We presented a systematic approach to integrate a speech encoder with an LLM
for the ASR task. Our findings indicate that employing the parameter-efficient
LoRa method on both the encoder and LLM represents the most cost-effective
fine-tuning strategy. Additionally, we introduced a matching loss to enhance
alignment between modalities, thereby improving ASR performance. Moreover, we
explored various inference methods alongside training techniques to mitigate
insertion errors. Results demonstrate a substantial reduction in insertion
errors using these methods, resulting in significantly improved ASR
performance, especially for out-of-domain test sets.

## References

  * [1] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Trans. Audio, Speech and Lang. Proc._ , vol. 29, p. 3451–3460, Oct 2021.
  * [2] Y. Zhang, W. Han, J. Qin, Y. Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V. Axelrod, G. Wang, Z. Meng, K. Hu, A. Rosenberg, R. Prabhavalkar, D. S. Park, P. Haghani, J. Riesa, G. Perng, H. Soltau, T. Strohman, B. Ramabhadran, T. Sainath, P. Moreno, C.-C. Chiu, J. Schalkwyk, F. Beaufays, and Y. Wu, ``Google usm: Scaling automatic speech recognition beyond 100 languages,'' _arXiv preprint arXiv:2306.09093_ , 2023.
  * [3] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, ``An image is worth 16x16 words: Transformers for image recognition at scale,'' in _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_ , 2021.
  * [4] J. Li, D. Li, S. Savarese, and S. Hoi, ``Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models,'' in _ICML_ , ser. ICML'23, 2023.
  * [5] J. Wu, Y. Gaur, Z. Chen, L. Zhou, Y. Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu, and Y. Wu, ``On decoder-only architecture for speech-to-text and large language model integration,'' _2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , pp. 1–8, 2023.
  * [6] Y. Fathullah, C. Wu, E. Lakomkin, J. Jia, Y. Shangguan, K. Li, J. Guo, W. Xiong, J. Mahadeokar, O. Kalinli, C. Fuegen, and M. Seltzer, ``Prompting large language models with speech recognition abilities,'' _CoRR_ , vol. abs/2307.11795, 2023.
  * [7] M. Wang, W. Han, I. Shafran, Z. Wu, C.-C. Chiu, Y. Cao, N. Chen, Y. Zhang, H. Soltau, P. K. Rubenstein _et al._ , ``Slm: Bridge the thin gap between speech and text foundation models,'' in _2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2023, pp. 1–8.
  * [8] W. Yu, C. Tang, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, ``Connecting speech encoder and large language model for asr,'' _ArXiv_ , vol. abs/2309.13963, 2023.
  * [9] S. Ling, Y. Hu, S. Qian, G. Ye, Y. Qian, Y. Gong, E. Lin, and M. Zeng, ``Adapting large language model with speech for fully formatted end-to-end speech recognition,'' _ArXiv_ , vol. abs/2307.08234, 2023.
  * [10] D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu, ``SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities,'' in _EMNLP_ , H. Bouamor, J. Pino, and K. Bali, Eds., Dec. 2023.
  * [11] P. Rubenstein, C. Asawaroengchai, D. Nguyen, A. Bapna, Z. Borsos, F. Quitry, P. Chen, D. Badawy, W. Han, E. Kharitonov, H. Muckenhirn, D. Padfield, J. Qin, D. Rozenberg, T. Sainath, J. Schalkwyk, M. Sharifi _et al._ , ``Audiopalm: A large language model that can speak and listen,'' _CoRR_ , vol. abs/2306.12925, 06 2023.
  * [12] C. Lyu, M. Wu, L. Wang, X. Huang, B. Liu, Z. Du, S. Shi, and Z. Tu, ``Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration,'' _arXiv preprint arXiv:2306.09093_ , 2023.
  * [13] Y. Li, Y. Wu, J. Li, and S. Liu, ``Prompting large language models for zero-shot domain adaptation in speech recognition,'' in _IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2023, Taipei, Taiwan, December 16-20, 2023_. IEEE, 2023, pp. 1–8.
  * [14] L. Barrault _et al._ , ``Seamless: Multilingual expressive and streaming speech translation,'' _ArXiv_ , vol. abs/2312.05187, 2023.
  * [15] T. Wang, L. Zhou, Z. Zhang, Y. Wu, S. Liu, Y. Gaur, Z. Chen, J. Li, and F. Wei, ``Viola: Unified codec language models for speech recognition, synthesis, and translation,'' _CoRR_ , vol. abs/2305.16107, 2023.
  * [16] Y. Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. Glass, ``Listen, think, and understand,'' _arXiv preprint arXiv:2305.10790_ , 2023.
  * [17] S.-A. Rebuffi, H. Bilen, and A. Vedaldi, ``Learning multiple visual domains with residual adapters,'' in _NIPS_ , 2017.
  * [18] X. L. Li and P. Liang, ``Prefix-tuning: Optimizing continuous prompts for generation,'' in _ACL-IJCNLP_ , Aug. 2021, pp. 4582–4597.
  * [19] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, ``Lora: Low-rank adaptation of large language models,'' _arXiv preprint arXiv:2106.09685_ , 2021.
  * [20] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' in _34th Conference on Neural Information Processing Systems (NeurIPS)_ , 2020.
  * [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, ``Attention is all you need,'' in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS'17, 2017, p. 6000–6010.
  * [22] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: An asr corpus based on public domain audio books,'' in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 04 2015, pp. 5206–5210.
  * [23] C. Wang, A. Wu, and J. Pino, ``Covost 2 and massively multilingual speech-to-text translation,'' _arXiv:2007.10310_ , 2020.
  * [24] G. Chen, S. Chai, G.-B. Wang, J. Du, W. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhang, M. Jin, S. Khudanpur, S. Watanabe, S. Zhao, W. Zou, X. Li, X. Yao, Y. Wang, Y. Wang, Z. You, and Z. Yan, ``Gigaspeech: An evolving, multi-domain asr corpus with 10, 000 hours of transcribed audio,'' _ArXiv_ , vol. abs/2106.06909, 2021.
  * [25] D. Snyder, G. Chen, and D. Povey, ``Musan: A music, speech, and noise corpus,'' _ArXiv_ , vol. abs/1510.08484, 2015.
  * [26] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, ``Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,'' March 2023. [Online]. Available: https://lmsys.org/blog/2023-03-30-vicuna/


# CMT: Cross Modulation Transformer with Hybrid Loss for Pansharpening

Wen-Jie Shu, Hong-Xia Dou, Rui Wen, Xiao Wu and Liang-Jian Deng The research
is supported by NSFC (No. 12271083), and National Key Research and Development
Program of China (No. 2020YFA0714001).Wen-Jie Shu is with the School of
Optoelectronic Science and Engineering, University of Electronic Science and
Technology of China, Chengdu 611731, China (e-mail: wenjieshu2003@gmail.com).
Hong-Xia Dou is with the School of Science, Xihua University, Chengdu 610039,
China (e-mail: hongxia.dou@mail.xhu.edu.cn). Rui Wen, Xiao Wu and Liang-Jian
Deng are with the School of Mathematical Sciences, University of Electronic
Science and Technology of China, Chengdu 611731, China (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>liangjian.deng@uestc.edu.cn).

###### Abstract

Pansharpening aims to enhance remote sensing image (RSI) quality by merging
high-resolution panchromatic (PAN) with multispectral (MS) images. However,
prior techniques struggled to optimally fuse PAN and MS images for enhanced
spatial and spectral information, due to a lack of a systematic framework
capable of effectively coordinating their individual strengths. In response,
we present the Cross Modulation Transformer (CMT), a pioneering method that
modifies the attention mechanism. This approach utilizes a robust modulation
technique from signal processing, integrating it into the attention
mechanism’s calculations. It dynamically tunes the weights of the carrier’s
value (V) matrix according to the modulator’s features, thus resolving
historical challenges and achieving a seamless integration of spatial and
spectral attributes. Furthermore, considering that RSI exhibit large-scale
features and edge details along with local textures, we crafted a hybrid loss
function that combines Fourier and wavelet transforms to effectively capture
these characteristics, thereby enhancing both spatial and spectral accuracy in
pansharpening. Extensive experiments demonstrate our framework’s superior
performance over existing state-of-the-art methods. The code will be publicly
available to encourage further research.

###### Index Terms:

Pansharpening, Cross Modulation Transformer, Fourier and Wavelet Transforms

##  I Introduction

Given the inherent constraints of remote sensing technology, obtaining MS
images with high spatial resolution directly from satellites is a significant
challenge. As a solution, pansharpening has become a crucial technique. It
merges low-resolution MS (LRMS) images with high-resolution PAN images to
produce high-spatial-resolution MS (HRMS) images with superior spatial detail.
This fusion technique effectively navigates around the limitations of sensor
technology, offering invaluable data for RSI analysis.

Figure 1:  (a) The modulation process in the field of communication. (b) The
modulation process in coded aperture snapshot spectral imaging (CASSI). (c)
Our Cross Modulation Multi-head Self-Attention (CM-MSA) modulation process.

Deep learning breakthroughs, led by Convolutional Neural Networks (CNNs), have
significantly advanced the field of pansharpening [4], [12], [29]. CNNs have
shown exceptional prowess in pansharpening, skillfully extracting and
combining intricate features from various images to enhance both spatial and
spectral quality. Furthermore, since their introduction, Transformers [21]
have revolutionized numerous fields, including pansharpening [11], [28], [17]
by their unparalleled ability to model long-range dependencies using self-
attention mechanisms. This capability gives them a significant edge in
effectively blending spatial and spectral features, outperforming traditional
CNN-based approaches by capturing the complex dynamics between different types
of images.

However, the field of pansharpening still faces several challenges. Firstly,
Hyperspectral Images (HSI) show spatial sparsity and spectral self-similarity,
complicating spatial dependency modeling and highlighting the importance of
prioritizing inter-spectral over spatial correlations in Transformers.
Secondly, RSI are marked by their rich spectral content, complex surface
textures and edge details, posing challenges for fusion and necessitating
tailored approaches for effective integration. Lastly, the potential of
attention structures, specifically tailored for pansharpening within the
Transformer framework, warrants further exploration. Current integration
methods of PAN and LRMS images within frameworks like PanFormer [30] and
Hyperformer [1] mainly involve linear projection of images into tokens for
merging via attention computations. While these methods are efficient, they
don’t fully exploit the intricate relationship between the spatial resolution
of PAN and the spectral diversity of LRMS images.

Figure 2:  Overall structure of the proposed method.

In response to these challenges, we have developed innovative solutions,
notably the CMT and a hybrid loss function, to elevate the pansharpening
workflow. Firstly, as illustrated in Fig. 1 (c), to harness spectral features
for LRMS images and spatial correlation for PAN images, we independently
compute the attention block for each spectral and spatial channel ensuring a
targeted processing of the distinct characteristics inherent. Secondly, to
address the nuanced complexities of RSI, characterized by their rich spectral
content and complex surface textures, we’ve implemented a hybrid loss function
that combines Fourier and wavelet transforms. This approach utilizes Fourier
transforms for identifying widespread features and wavelet transforms for
enhancing local texture details. Together, they effectively improve spatial
detail and maintain spectral fidelity in the pansharpening process. Lastly,
our approach innovatively blends PAN’s spatial details with LRMS’s spectral
data by applying advanced modulation techniques to the pansharpening process.
Specifically, rather than merely concatenating features, our approach
leverages the features of the modulator to dynamically modulate the carrier,
altering the weights of the Transformer’s value (V) matrix, which achieves a
more sophisticated fusion of features. Our approach, inspired by signal
processing techniques depicted in Fig. 1 (a), uses modulation to seamlessly
integrate high and low-frequency signals, significantly improving signal
fidelity and richness. In parallel, CASSI systems, illustrated in Fig. 1 (b),
compress images by modulating high frequencies with masks, akin to our deep
learning methods for image enhancement like MST by Cai et al [3], which also
employ masks for recovery.

In conclusion, the CMT framework significantly advances pansharpening through
the following contributions:

  1. 1.

Our Cross Modulation module within the CMT framework significantly enhances
the fusion of PAN and LRMS images through a novel modulation technique.

  2. 2.

We introduce a pioneering hybrid loss function that combines Fourier and
wavelet transforms, which is the first attempt in the field of pansharpening
to the best of our knowledge.

  3. 3.

The CMT framework delivers outstanding results on benchmark datasets,
establishing a new benchmark for pansharpening performance.

Figure 3:  (a) The CMAB module consists of a Double Feed-Forward Network
(DFFN), a CM-MSA module, and two layers of normalization. (b) Components of
the DFFN. (c) The hybrid loss between Predicted Images ($PI$) and ground truth
($GT$), which employs both the 2D Discrete Fourier Transform (DFT) and the 2D
Discrete Wavelet Transform (DWT).

##  II Method

###  II-A Overall Architecture

The CMT architecture, depicted in Fig. 2, is structured into three primary
phases: feature extraction, modulation, and feature aggregation. In the
feature extraction phase, distinct extractors are designed to capture the
unique characteristics of PAN and LRMS images. During the modulation phase,
the Cross Modulation Attention Block (CMAB) modulator separately modulates PAN
images to enrich LRMS images and vice versa, allowing for an effective blend
of spatial and spectral information. Finally, to aggregate feature, we use a
$3\times 3$ convolutional kernel and four ResNet blocks simply.

Initially, LRMS images are upscaled to dimensions $H\times W\times c$. Then,
PAN and LRMS images are processed through convolutional and ResNet blocks to
extract local and global spatial details, resulting in feature sets
$F_{\text{pan}}$ and $F_{\text{ms}}$, respectively, with channel expansion to
$C=32$ in implementation.

The extracted features $F_{\text{pan}}$ and $F_{\text{ms}}$ are then modulated
within the CMAB module, as shown in Fig. 3 (a). This module comprises a CM-
MSA, two layers of normalization and a DFFN, employing varied activation
functions for enhanced modulation. Fig. 3 (b) illustrates the specifics of the
DFFN. Post-modulation, the features are concatenated and merged through a
$3\times 3$ convolution and four ResNet blocks, allowing for further
integration of spatial and spectral information.

Ultimately, the aggregated features are combined with the upsampled LRMS
images to produce the HRMS.

###  II-B Modulation Approach

In signal processing, Double Side-Band modulation (DSB) [9], [19] stands as a
classic and effective technique wherein the amplitude of a carrier wave is
varied in accordance with the instantaneous value of the message signal, thus
encoding information within the carrier. The mathematical expression for an
DSB signal can be accurately given by:

$s(t)=m(t)\cdot A_{c}\cdot\cos(2\pi f_{c}t),$ (1)

where $s(t)$ denotes the modulated signal, $A_{c}$ represents the constant
amplitude of the carrier wave, $f_{c}$ is the carrier frequency, and $m(t)$
embodies the message signal intended for transmission. This foundational
concept of modulation underscores its robust capability to encode and transmit
complex information efficiently and accurately across different mediums. In a
similar vein, depicted in Fig. 1 (c), our pansharpening framework leverages a
cross modulation paradigm where high-resolution spatial features and spectral
details are mutually modulated. This bilateral modulation mechanism
considerably amplifies the model’s proficiency in capturing and amalgamating
multi-dimensional information, marking a significant leap from traditional
methods.

Firstly, for the carrier, the input feature $X_{\text{in}}$
$\in\mathbb{R}^{H\times W\times C}$ is reshaped into tokens
$X\in\mathbb{R}^{HW\times C}$. A multi-head attention mechanism is employed to
improve generalization and capture multi-dimensional information, splitting
$X$ into $k$ heads:

$X=[X_{1},X_{2},\ldots,X_{k}],$ (2)

where $X_{i}\in\mathbb{R}^{HW\times d_{k}}$, $d_{k}=\frac{C}{k}$, and
$i=1,2,\ldots,k$.

Each $X_{i}$ is then linearly projected into queries $Q_{i}$, keys $K_{i}$,
and values $V_{i}$ using the following equations:

$Q_{i}=X_{i}W^{Q}_{i},\quad K_{i}=X_{i}W^{K}_{i},\quad V_{i}=X_{i}W^{V}_{i},$
(3)

where $W^{Q}_{i}$, $W^{K}_{i}$, and $W^{V}_{i}\in\mathbb{R}^{d_{k}\times C}$
are learnable parameters, and $i=1,2,\ldots,k$.

For the modulator, aligned with the carrier, the input feature is reshaped
into tokens $M\in\mathbb{R}^{HW\times C}$ and split into $k$ heads. Then, the
modulation is integrated into the self-attention calculation by element-wise
multiplying $M_{i}$ with $V_{i}$:

$M=[M_{1},M_{2},\ldots,M_{k}],\quad V^{\prime}_{i}=M_{i}\odot V_{i}$ (4)

where $M_{i}\in\mathbb{R}^{HW\times d_{k}}$, $i=1,2,\ldots,k$.

For a single head, the modulation-attention computation is:

$\begin{split}\text{MA}(Q_{i},K_{i},V^{\prime}_{i})&=V^{\prime}_{i}\cdot\text{softmax}\left(\frac{K_{i}^{T}Q_{i}}{\alpha_{i}}\right),\end{split}$
(5)

where MA is modulation attention and $\alpha_{i}\in\mathbb{R}$ is a learnable
parameter that adaptively scales the matrix multiplication, enhancing the
model’s ability to adjust attention weights dynamically.

The results from multi-heads are concatenated together, and with the addition
of position encoding, the final output $X_{\text{out}}$ is derived as follows:

$X_{\text{out}}=\text{Fc}(\text{Concatenate}(\text{MA}_{1},\text{MA}_{2},\ldots,\text{MA}_{k}))+F_{p},$
(6)

where Fc denotes a fully connected layer, and $F_{p}$ represents the position
encoding.

TABLE I: Quantitative results on 20 reduced-resolution and 20 full-resolution samples of GF2. (Red: best; Blue: second best). Method | Reduced-Resolution Metrics | Full-Resolution Metrics
---|---|---
SAM↓ | ERGAS↓ | Q4↑ |  $D_{\lambda}$↓ |  $D_{s}$↓ | HQNR↑
PNN[16] | 1.048 $\pm$ 0.226 | 1.057 $\pm$ 0.235 | 0.960 $\pm$ 0.010 | 0.0317 $\pm$ 0.0286 | 0.0943 $\pm$ 0.0224 | 0.877 $\pm$ 0.036
PanNet[27] | 0.997 $\pm$ 0.212 | 0.919 $\pm$ 0.191 | 0.967 $\pm$ 0.010 |  0.0179 $\pm$ 0.0110 | 0.0799 $\pm$ 0.0178 | 0.904 $\pm$ 0.020
DiCNN[10] | 1.052 $\pm$ 0.231 | 1.081 $\pm$ 0.254 | 0.959 $\pm$ 0.010 | 0.0369 $\pm$ 0.0132 | 0.0992 $\pm$ 0.0131 | 0.868 $\pm$ 0.016
FusionNet[5] | 0.973 $\pm$ 0.212 | 0.988 $\pm$ 0.222 | 0.964 $\pm$ 0.009 | 0.0350 $\pm$ 0.0124 | 0.1013 $\pm$ 0.0134 | 0.867 $\pm$ 0.018
DCFNet[25] | 0.872 $\pm$ 0.169 | 0.784 $\pm$ 0.146 | 0.974 $\pm$ 0.009 | 0.0240 $\pm$ 0.0115 |  0.0659 $\pm$ 0.0096 |  0.912 $\pm$ 0.012
MMNet[26] | 0.993 $\pm$ 0.141 | 0.777 $\pm$ 0.134 | 0.969 $\pm$ 0.020 | 0.0443 $\pm$ 0.0298 | 0.1033 $\pm$ 0.0129 | 0.857 $\pm$ 0.027
LAGConv[13] |  0.786 $\pm$ 0.148 | 0.687 $\pm$ 0.113 | 0.980 $\pm$ 0.009 | 0.0284 $\pm$ 0.0130 | 0.0792 $\pm$ 0.0136 | 0.895 $\pm$ 0.020
HMPNet[20] | 0.803 $\pm$ 0.156 |  0.564 $\pm$ 0.099 |  0.981 $\pm$ 0.030 | 0.0819 $\pm$ 0.0499 | 0.1146 $\pm$ 0.0126 | 0.813 $\pm$ 0.049
Proposed |  0.722 $\pm$ 0.136 |  0.624 $\pm$ 0.107 |  0.992 $\pm$ 0.001 |  0.0202 $\pm$ 0.0103 |  0.0338 $\pm$ 0.0086 |  0.947 $\pm$ 0.012
Figure 4:  Qualitative result comparison between representative methods on the
GF2 reduced-resolution dataset. The first row presents RGB outputs, while the
second row gives the corresponding QNR maps.

###  II-C Loss Function

To enhance the resolution and quality of RSI, as shown in Fig. 3 (c), our
method improves upon traditional fusion techniques by leveraging both Fourier
and wavelet transforms. Fourier transforms [18] are essential for mapping
images into the frequency domain, capturing widespread environmental features.
Besides, wavelet transforms [14] excel in delineating images across multiple
scales, adeptly enhancing local textures and detailing.

The Fourier transform loss function is defined as:

$L_{\text{Fourier}}=\frac{1}{M}\sum_{i=1}^{M}\|\mathcal{F}(PI_{i})-\mathcal{F}(GT_{i})\|_{1},$
(7)

which applies the L1 loss to the difference in Fourier-transformed Predicted
Images ($PI_{i}$) and ground truth images ($GT_{i}$), averaged over all $M$
training samples.

The loss function for the wavelet transform is defined as:

$L_{\text{Wavelet}}=\frac{1}{M}\sum_{i=1}^{M}D_{j,c}(i),$ (8)

where $D_{j,c}(i)=\|\mathcal{W}{j,c}(PI_{i})-\mathcal{W}{j,c}(GT_{i})\|_{1}$
and $\mathcal{W}{j,c}$ captures wavelet coefficients at scale $j$ and
orientation $c$, reflecting local variations and textures, averaged across all
training instances.

Combining these components, our comprehensive loss function integrates
spatial, frequency, and wavelet domain losses:

$L_{\text{total}}=L_{\text{spa}}+\lambda_{1}L_{\text{Fourier}}+\lambda_{2}L_{\text{Wavelet}}$
(9)

where $L_{\text{spa}}$ is the L1 loss in the spatial domain, $\lambda_{1}$ and
$\lambda_{2}$ are set to 0.7 and 0.2 to balance the contributions of the
different loss components as weighting coefficients in implementation.

##  III Experiment

###  III-A Datasets and Implementation Details

To validate our approach, we construct datasets following Wald’s protocol [6],
[24] on data collected from the WorldView-3 (WV3) and GaoFen-2 (GF2)
satellites. Our datasets and data processing methods are downloaded from the
PanCollection repository [7]. The datasets consist of images cropped from
entire remote sensing images, divided into training and testing sets. The
training set comprises PAN/LRMS/GT image pairs obtained by downsampling
simulation, with dimensions of 64×64, 16×16×C and 64×64×C. Besides, we
evaluated our method on the commonly used metrics in the field of
pansharpening, including SAM [2], ERGAS [23] and Q8 [8] for reduced-resolution
dataset, $Ds$, $D_{\lambda}$ and HQNR [22] for full-resolution dataset. The
CMT was trained with an initial learning rate of 0.001, for 400 epochs, and a
batch size of 32, using Adam optimizer [15] with learning rate halved every
100 epochs. As for other DL-based methods, we utilize the default settings in
related papers or codes to train the networks.

TABLE II: Quantitative results of DL-based methods on 20 reduced-resolution samples acquired by WV3. (Red: best; Blue: second best). Method | Q8↑ | SAM↓ | ERGAS↓
---|---|---|---
PNN | 0.893 $\pm$ 0.092 | 3.677 $\pm$ 0.762 | 2.680 $\pm$ 0.647
PanNet | 0.891 $\pm$ 0.093 | 3.613 $\pm$ 0.766 | 2.664 $\pm$ 0.688
DiCNN | 0.900 $\pm$ 0.087 | 3.592 $\pm$ 0.762 | 2.672 $\pm$ 0.662
FusionNet | 0.904 $\pm$ 0.090 | 3.324 $\pm$ 0.698 | 2.465 $\pm$ 0.644
MMNet | 0.915 $\pm$ 0.086 | 3.084 $\pm$ 0.640 | 2.343 $\pm$ 0.626
LAGConv | 0.910 $\pm$ 0.091 | 3.103 $\pm$ 0.558 | 2.292 $\pm$ 0.607
HMPNet |  0.916 $\pm$ 0.087 |  3.063 $\pm$ 0.577 |  2.229 $\pm$ 0.545
Proposed |  0.917 $\pm$ 0.086 |  3.001 $\pm$ 0.610 |  2.201 $\pm$ 0.522
TABLE III: Ablation study on loss components. (Red: best; Blue: second best). Loss Components | Metrics
---|---
L${}_{\text{spa}}$ | L${}_{\text{wave}}$ | L${}_{\text{fft}}$ | Q8↑ | SAM↓ | ERGAS↓
✓ | ✗ | ✗ | 0.912±0.086 | 3.043±0.618 | 2.238±0.530
✓ | ✓ | ✗ | 0.913±0.087 | 3.033±0.612 | 2.223±0.523
✓ | ✗ | ✓ |  0.916±0.086 |  3.006±0.609 |  2.208±0.521
✓ | ✓ | ✓ |  0.917±0.086 |  3.001±0.610 |  2.201±0.522
TABLE IV: Ablation study on Modulation Approach. (Red: best; Blue: second best). Method |  $D_{\lambda}$↓ |  $Ds$↓ | HQNR↑
---|---|---|---
V1 |  0.0210±0.0074 | 0.0364±0.0125 |  0.9435±0.0180
V2 | 0.0249±0.0123 |  0.0355±0.0132 | 0.9406±0.0187
V3 | 0.0234±0.0079 | 0.0388±0.0155 | 0.9389±0.0210
CMT |  0.0201±0.0074 |  0.0344±0.0135 |  0.9463±0.0188

###  III-B Results

The performance of the proposed CMT method is show cased through extensive
evaluations on GF2 datasets. TABLE I present a comprehensive comparison of CMT
with various state-of-the-art methods on the GF2 dataset. The quantitative
results show our method consistently surpasses existing approaches in all
metrics. The visual comparison results are provided in Fig. 3. TABLE II
present the results on the WV3 datasets, and the proposed method obtains the
best average results on all quality indexes.

###  III-C Ablation Experiment

Ablation on Hybrid Loss. We compare the outcomes after training with different
loss functions. The results are displayed in TABLE III. This comparison aims
to ascertain the impact of each loss component on the overall performance of
the model. We perform experiments on 20 reduced-resolution samples acquired by
WV3 satellite.

Ablation on Modulation Approach. To validate the effectiveness of our method,
we create three variants of the CMT. In the first variant (V1), modulation is
omitted, taining only the transformer to evaluate its baseline feature
integration performance. The second variant (V2) involves exclusively using
the PAN image to modulate the MS features. In the third variant (V3), we
solely use LRMS modulation of the PAN image. We perform experiments on 20
full-resolution samples acquired by the WV3 satellite. The results in TABLE IV
show CMT has the best overall performance, proving our method’s efficacy.

##  IV Conclusion

In this study, we introduce the CMT, a novel pansharpening method that
synergistically merges PAN and LRMS images. Central to CMT is the application
of signal modulation, innovatively incorporated into a Transformer-based
architecture. This allows for precise modulation of the Transformer’s value
(V) matrix, facilitating a superior integration of spatial detail and spectral
depth. Our method is characterized by its unique CM-SMA modulation technique
and a bespoke hybrid loss function that blends Fourier and wavelet transforms.
This loss function adeptly captures both global patterns and local textures,
thereby enhancing spatial resolution while maintaining spectral fidelity. The
versatility of CMT suggests its applicability beyond pansharpening, offering
promising enhancements in various fields that require intricate and spectrally
accurate image fusion.

## References

  * [1] Wele Gedara Chaminda Bandara and Vishal M Patel.  Hypertransformer: A textural and spectral feature fusion transformer for pansharpening.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1767–1777, 2022.
  * [2] Joseph W Boardman.  Automating spectral unmixing of aviris data using convex geometry concepts.  1993\.
  * [3] Yuanhao Cai, Jing Lin, Xiaowan Hu, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu Timofte, and Luc Van Gool.  Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17502–17511, 2022.
  * [4] Zhixuan Chen, Cheng Jin, Tianjing Zhang, Xiao Wu, and Liangjian Deng.  Spanconv: A new convolution via spanning kernel space for lightweight pansharpening.  International Joint Conference on Artificial Intelligence, pages 841–847, 2022.
  * [5] Liangjian Deng, Gemine Vivone, Cheng Jin, and Jocelyn Chanussot.  Detail injection-based deep convolutional neural networks for pansharpening.  IEEE Transactions on Geoscience and Remote Sensing, pages 6995–7010, 2020.
  * [6] Liangjian Deng, Gemine Vivone, Cheng Jin, and Jocelyn Chanussot.  Detail injection-based deep convolutional neural networks for pansharpening.  IEEE Transactions on Geoscience and Remote Sensing, pages 6995–7010, 2020.
  * [7] Liangjian Deng, Gemine Vivone, Mercedes E Paoletti, Giuseppe Scarpa, Jiang He, Yongjun Zhang, Jocelyn Chanussot, and Antonio Plaza.  Machine learning in pansharpening: A benchmark, from shallow to deep networks.  IEEE Geoscience and Remote Sensing Magazine, pages 279–315, 2022.
  * [8] Renwei Dian, Shutao Li, and Leyuan Fang.  Learning a low tensor-train rank representation for hyperspectral image super-resolution.  IEEE Transactions on Neural Networks and Learning Systems, pages 2672–2683, 2019.
  * [9] Ralph VL Hartley.  Transmission of information 1.  Bell System technical journal, pages 535–563, 1928.
  * [10] Lin He, Yizhou Rao, Jun Li, Jocelyn Chanussot, Antonio Plaza, Jiawei Zhu, and Bo Li.  Pansharpening via detail injection based convolutional neural networks.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, pages 1188–1204, 2019.
  * [11] Jinfan Hu, Tingzhu Huang, Liangjian Deng, Hongxia Dou, Danfeng Hong, and Gemine Vivone.  Fusformer: A transformer-based fusion network for hyperspectral image super-resolution.  IEEE Geoscience and Remote Sensing Letters, pages 1–5, 2022.
  * [12] Zirong Jin, Liangjian Deng, Tianjing Zhang, and Xiao Xu Jin.  Bam: Bilateral activation mechanism for image fusion.  Proceedings of the 29th ACM international conference on multimedia, pages 4315–4323, 2021.
  * [13] Zirong Jin, Tianjing Zhang, Taixiang Jiang, Gemine Vivone, and Liangjian Deng.  Lagconv: Local-context adaptive convolution kernels with global harmonic bias for pansharpening.  AAAI Conference on Artificial Intelligence, 2022.
  * [14] Er Saiqa Khan and Er Arun Kulkarni.  An efficient method for detection of copy-move forgery using discrete wavelet transform.  International Journal on Computer Science and Engineering, page 2010, 1801.
  * [15] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [16] Giuseppe Masi, Davide Cozzolino, Luisa Verdoliva, and Giuseppe Scarpa.  Pansharpening by convolutional neural networks.  Remote Sensing, page 594, 2016.
  * [17] Xiangchao Meng, Nan Wang, Feng Shao, and Shutao Li.  Vision transformer for pansharpening.  IEEE Transactions on Geoscience and Remote Sensing, pages 1–11, 2022.
  * [18] Raymond Edward Alan Christopher Paley and Norbert Wiener.  Fourier transforms in the complex domain.  American Mathematical Soc., 1934.
  * [19] Hans Roder.  Amplitude, phase, and frequency modulation.  Proceedings of the Institute of Radio Engineers, pages 2145–2176, 1931.
  * [20] Xin Tian, Kun Li, Wei Zhang, Zhongyuan Wang, and Jiayi Ma.  Interpretable model-driven deep network for hyperspectral, multispectral, and panchromatic image fusion.  IEEE Transactions on Neural Networks and Learning Systems, 2023.
  * [21] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 2017.
  * [22] Gemine Vivone, Luciano Alparone, Jocelyn Chanussot, Mauro Dalla Mura, Andrea Garzelli, Giorgio A. Licciardi, Rocco Restaino, and Lucien Wald.  A critical comparison among pansharpening algorithms.  IEEE Transactions on Geoscience and Remote Sensing, pages 2565–2586, 2015.
  * [23] Gemine Vivone, Rocco Restaino, Mauro Dalla Mura, Giorgio Licciardi, and Jocelyn Chanussot.  Contrast and error-based fusion schemes for multispectral image pansharpening.  IEEE Geoscience and Remote Sensing Letters, pages 930–934, 2014.
  * [24] Lucien Wald, Thierry Ranchin, and Marc Mangolini.  Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images.  Photogrammetric engineering and remote sensing, pages 691–699, 1997.
  * [25] Xiao Wu, Tingzhu Huang, Liangjian Deng, and Tianjing Zhang.  Dynamic cross feature fusion for remote sensing pansharpening.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14687–14696, October 2021.
  * [26] Keyu Yan, Man Zhou, Li Zhang, and Chengjun Xie.  Memory-augmented model-driven network for pansharpening.  European Conference on Computer Vision, pages 306–322, 2022.
  * [27] Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, and John Paisley.  Pannet: A deep network architecture for pan-sharpening.  In Proceedings of the IEEE international conference on computer vision, pages 5449–5457, 2017.
  * [28] Hao Zhang, Hebaixu Wang, Xin Tian, and Jiayi Ma.  P2sharpen: A progressive pansharpening network with deep spectral transformation.  Information Fusion, pages 103–122, 2023.
  * [29] Tianjiang Zhang, Liangjian Deng, Tingzhu Huang, Jocelyn Chanussot, and Gemine Vivone.  A triple-double convolutional neural network for panchromatic sharpening.  IEEE Transactions on Neural Networks and Learning Systems, 2022.
  * [30] Huanyu Zhou, Qingjie Liu, and Yunhong Wang.  Panformer: A transformer based model for pan-sharpening.  IEEE International Conference on Multimedia and Expo, pages 1–6, 2022.