# S4Fusion: Saliency-aware Selective State Space Model for Infrared Visible
Image Fusion

Haolong Ma1 Hui Li1 Chunyang Cheng1 Gaoang Wang2 Xiaoning Song1 Xiaojun Wu1
1Jiangnan University  2Zhejiang University
{lihui.cv, wu_xiaojun<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

As one of the tasks in Image Fusion, Infrared and Visible Image Fusion aims to
integrate complementary information captured by sensors of different
modalities into a single image. The Selective State Space Model (SSSM), known
for its ability to capture long-range dependencies, has demonstrated its
potential in the field of computer vision. However, in image fusion, current
methods underestimate the potential of SSSM in capturing the global spatial
information of both modalities. This limitation prevents the simultaneous
consideration of the global spatial information from both modalities during
interaction, leading to a lack of comprehensive perception of salient targets.
Consequently, the fusion results tend to bias towards one modality instead of
adaptively preserving salient targets. To address this issue, we propose the
Saliency-aware Selective State Space Fusion Model (S4Fusion). In our S4Fusion,
the designed Cross-Modal Spatial Awareness Module (CMSA) can simultaneously
focus on global spatial information from both modalities while facilitating
their interaction, thereby comprehensively capturing complementary
information. Additionally, S4Fusion leverages a pre-trained network to
perceive uncertainty in the fused images. By minimizing this uncertainty,
S4Fusion adaptively highlights salient targets from both images. Extensive
experiments demonstrate that our approach produces high-quality images and
enhances performance in downstream tasks.

##  1 Introduction

As an image processing technique, Infrared and Visible Image Fusion aims to
integrate information from different modalities captured by various sensors in
the same scene, generating a comprehensive and informative image [18; 51; 52].
This technique has been widely used in various scenarios such as image
restoration[5], UAV reconnaissance[16; 32], image registration[15], and can be
combined with downstream tasks like image segmentation[35], detection[47; 38],
and object tracking[25]. Visible images possess rich texture details and high
resolution but are susceptible to illumination and occlusion, leading to loss
of salient information[46]. Infrared images, on the other hand, rely on
thermal radiation information to capture salient details under extreme
conditions (e.g., low light) but are prone to noise[43]. Thus, integrating
complementary information from both modalities comprehensively is essential
for better reconstructing real-world scenes.

Originating from control systems, Selective State Space Models (SSSM) are a
type of State Space Model (SSM)[8; 9]. For IVIF tasks, selectively
highlighting information from both source images is crucial[18]. Previous
works overlook the importance of considering global spatial information during
the interaction between the two modalities[36; 17; 26]. For example, while
Cross Attention[45] effectively captures inter-modal information, it can only
focus on the spatial information of the opposite modality during interactions.
This means that it neglects the spatial information within its own modality
during the interaction, which leads to a failure in comprehensively perceiving
complementary information based on its own global space. This limitation is
particularly critical for tasks like infrared and visible image fusion (IVIF),
where the two modalities have highly complementary information.

Therefore, it is essential to design a module that can simultaneously focus on
the spatial information of the images while perceiving and selecting the most
beneficial complementary information between the two modalities. To address
this, we designed a new module called Cross-Modal Spatial Awareness Module
(CMSA). This module can simultaneously facilitate interactions between
modalities and within the spatial domains of each modality, thereby enhancing
the perception of salient targets.

To drive the CMSA to adaptively highlight salient information, we use a pre-
trained ResNet50 network as a supervisory signal. In real-world scenes, the
information should be deterministic and structured rather than random noise.
Building on this observation, we designed a novel loss function called
perception-enhanced loss at the decision layer of the pre-trained network.
This function adaptively highlights salient targets and removes redundant
information by minimizing the uncertainty perception of the pre-trained neural
network on the fused image. Finally, we applied the designed CMSA module to a
multi-scale network structure and proposed the Saliency-aware Selective State
Space Model (S4Fusion) for IVIF task. The contributions of this paper can be
summarized as follows:

(I) We designed a module called Cross-Modal Spatial Awareness Module (CMSA).
This module captures information from both modalities spaces and their
interrelations simultaneously, with linear complexity. It selectively
preserves salient information while eliminating redundant details.

(II) To drive the CMSA to adaptively highlight salient information, we
developed a novel loss function at the decision layer to simulate human visual
perception. This function aims to highlight salient information adaptively,
resulting in images that better align with human visual perception.

(III) We conducted extensive experiments to validate the effectiveness of our
method, including qualitative evaluations, quantitative evaluations, and
evaluations involving integration with downstream tasks.

##  2 Related Work

###  2.1 Infrared and Visible Image Fusion

With the rise of deep learning, an increasing number of researchers are
adopting deep learning methods to study IVIF[12; 13; 14]. These methods can
generally be classified into three categories: generative models[22; 23],
autoencoder-based models[12; 13], and end-to-end models[44; 48]. Generative
models often suffer from unstable training, autoencoder-based methods offer
more stable training but still rely to some extent on handcrafted fusion
strategies[43]. End-to-end methods, on the other hand, often design loss
functions at the shallow pixel level, which can still be considered as
handcrafted fusion rules, limiting the network’s adaptability.

To enable models to adaptively integrate information from two source images,
many end-to-end models make attempts[18; 17; 33]. Among them, U2Fusion[41]
achieves excellent results by using features from a pre-trained VGG16 network
as a supervisory signal. However, due to the lack of consideration for
highlighting salient targets adaptively, it weakened the preservation of
saliency information. Therefore, we design a new perception-enhanced loss,
simulating human visual perception at the decision layer of the pre-trained
model, to highlight salient targets.

###  2.2 Selective State Space Model

Recently, State Space Models attract increasing interest from researchers
across various fields, demonstrating significant potential in NLP and CV[21;
30; 24]. Early works like LSSL[9], when combined with HiPPO initialization[7],
exhibited powerful capabilities in capturing long-range dependencies. With the
emergence of Mamba[37; 6], the Selective State Space Model was introduced and
achieved remarkable success in NLP. The introduction of VMamba[20] brought
Mamba into the field of computer vision (CV). Its powerful global receptive
field, rapid inference speed, and lack of need for positional embedding have
inspired many related works in CV.

In Image Fusion, some works start to combine with Mamba[26; 2]. For example,
FusionMamba[26] designed FSSM, inspired by cross-attention, which achieved
notable success in the hyperspectral pansharpening task. However, these
methods focus primarily on inter-modal interactions, overlooking the
importance of simultaneously considering information from both modality
spaces. To address this, we propose a new module called Cross-Modal Spatial
Awareness Module (CMSA). CMSA can perform inter-modal interactions and
interactions within each modality space in a single stage, enabling
comprehensive perception of salient information in both spatial and modal
dimensions.

Figure 1: The architecture of S4Fusion with the number of layers $N=3$. (a)
depicts the overall architecture of S4Fusion. $G_{i}$ represents the number of
$i$-th layer VSS Blocks. (b) shows the internal workflow of the VSS Block. (c)
illustrates the structure of the CMSA in the $l$-th layer, consisting of K
Cross-Modal Spatial Awareness Module Blocks. (d) provides notations and
abbreviations. Figure 2: Figures of Patch Mark and Cross SS2D. (a) and (b)
demonstrate the computation process of Patch Mark. Panel (c) illustrates the
Cross SS2D scanning trajectory in the Horizontal Forward direction. (d) shows
the four scanning trajectories of SS2D as described in the original VMamba
paper.

##  3 Proposed Method

In this section, we delve into the detailed process of our approach. As shown
in Fig.1, following the form of UNet[1; 30], our model adopts a multi-scale
structure to capture image information at different scales. Our model
primarily consists of two types of modules: VSS Block[20] and CMSA. Overlap
Patch Embedding and Fold & Output Projection are utilized at the input and
output ends of the model to encode overlapping patches and integrate outputs
across spatial and channel dimensions. Infrared and visible images are
processed in two separate streams, each with its own independent encoder to
extract features specific to their respective modalities. For simplicity, we
assume the encoder and decoder each have $N$ layers.

###  3.1 Preliminary

####  3.1.1 SSM

The State Space Model (SSM) initially served as a time-series processing
model, mapping input sequences $x(t)$ to output sequences $y(t)$ through
hidden states $h(t)$ to memorize and capture temporal relationships. Formally,
SSM can be expressed by the following two equations:
$h^{\prime}(t)=Ah(t)+Bx(t);\ y(t)=Ch(t)$, where $A$, $B$, and $C$ are
parameters. $A$ controls the update of hidden states, while $B$ and $C$ govern
how data flows into and out of hidden states. Subsequently, to better handle
discrete data in real-world scenarios (such as text), the time step parameter
$\Delta$ is introduced to control the time interval. Through zero-order hold,
SSM is further discretized into the following form[8]:
$h_{t}=\bar{A}h_{t-1}+\bar{B}x_{t};\ y_{t}=Ch_{t}$, where $\bar{A}$ and
$\bar{B}$ are counterparts of $A$ and $B$ after zero-order hold, with
$\bar{A}=\exp(\Delta A)$ and $\bar{B}=\Delta B$.

####  3.1.2 Mamba and VMamba

Mamba transforms the original parameters $A,B$, and $C$ into context-dependent
parameters, making SSM selective to inputs[37]. Specifically, at time $t$, the
computation process of SSM from input to output utilizes parameters $A$, $B$,
and $C$ derived from the input $x_{t}$.

Subsequently, VMamba[30] introduced Mamba into the realm of CV for the first
time and designed the SS2D module to address the inability of Mamba to handle
image data. As shown in Fig.2(d), VMamba non-overlappingly divides the image
into patches and flattens them into sequences in four different directions.
These sequences are then individually inputted into Mamba Blocks for
computation and finally aggregated to extract features.

###  3.2 Encoder and Decoder

We use VSS Blocks[30] for feature extraction and image reconstruction for each
modality. As shown in Fig.1(b), the VSS Block adopts a design similar to Mamba
Block, resembling the structure of ResNet. Within it, SS2D has a global
receptive field. We employ VSS Blocks as encoders for each modality to extract
shallow features within their respective modalities. These features will be
further utilized for deeper interactions. Multiple VSS Blocks serve as
encoders at each scale. Let $I$ and $V$ denote the two original images. After
applying Overlap Patch Embedding, we obtain
$I^{(0)},V^{(0)}\in\mathbb{R}^{H\times W\times C}$, which represent the
infrared and visible features, respectively.

$I^{(l)}=Down^{(l)}(E_{\mathrm{IR}}^{(l)}(I^{(l-1)})),V^{(l)}=Down^{(l)}(E_{\mathrm{VI}}^{(l)}(V^{(l-1)})),\
\mathrm{where}\ l\in\\{1,2,...N\\}$ (1)

where $E_{\mathrm{IR}}^{(l)}(\cdot)$ and $E_{\mathrm{VI}}^{(l)}(\cdot)$
represent the encoder for the infrared and visible modalities at the $l$-th
layer. $Down^{(l)}(\cdot)$ denotes Patch Merging or $Identity$ operation. As
depicted in Fig.1(a), $Down^{(l)}(\cdot)=Identity(\cdot)$ only when $l=N$.

For the decoder at the $l$-th layer, it takes the fused feature $R^{(l)}$ and
the output from the previous scale $F^{(l)}$ as input to reconstruct the image
feature at that scale.

$F^{(l-1)}=D^{(l)}(UP^{(l)}(F^{(l)})+R^{(l)}),\ \mathrm{where}\
F^{(N)}=I^{(N)}+V^{(N)},l\in\\{1,2,..N\\}$ (2)

where $UP^{(l)}$ represents Patch Expanding or $Identity$ operation. Similar
to $Down$, $UP^{(l)}(\cdot)=Identity(\cdot)$ only at the $N$-th layer.

Finally, the output $F^{(0)}$ undergoes folding to remove overlaps and is
linearly mapped to integrate channels, resulting in the final fused image $F$.

Figure 3: Demonstration of the Interleaving and Recovering processes. The
numbers in different colors represent patches from different modalities, and
identical numbers denote patches from the same position in the feature maps.

###  3.3 Cross-Modal Spatial Awareness Module

The Cross-Modal Spatial Awareness Module (CMSA) is employed to interact with
features from the infrared and visible modalities at various scales, thereby
efficiently perceiving salient information. Specifically, CMSA consists of $K$
CMSA Blocks (CMSAB).

For ease of discussion, in this section, we will use the $k$-th CMSAB in the
$l$-th layer as an example. And, we also denote $V^{(l,0)}=V^{(l)}$ and
$I^{(l,0)}=I^{(l)},k\in\\{1,2..,K\\},l\in\\{1,2..,N\\}$.

As depicted in Fig.1(c), features from the two modalities (i.e.
$I^{(l,k)},V^{(l,k)}$) are first split into two branches separately using
Layer Norm and a linear mapping. One branch is used for gating mechanism,
while the other passes through a shared 3×3 depth-wise convolution to extract
features, denoted as $P^{(l,k)}_{\mathrm{IR}}$ and $P^{(l,k)}_{\mathrm{VI}}$
respectively. These features are then utilized for subsequent deep-level
feature interaction. This process consists of four steps: Patch Mark,
Interleaving, Cross SS2D, and Recovering. Subsequently, the two obtained
outputs go through a layer norm, a linear layer and a residual connection,
yielding the outputs $I^{(l,k+1)}$ and $V^{(l,k+1)}$ of the $k$-th CMSAB. At
the end of the $K$-th CMSAB, the two deeply interacted features are element-
wise summed: $R^{(l)}=I^{(l,K)}+V^{(l,K)}$, resulting in the output $R^{(l)}$
of the $l$-th CMSA.

####  3.3.1 Patch Mark

Patch Mark, similar to positional encoding, is used to group patches from the
same positions in the two modalities. Specifically, as shown in Fig.2(a, b),
Patch Mark concatenates the two inputs along the channel dimension and then
applies a linear layer to generate a mark for each patch position.
Consequently, patches from the same position in each modality are labeled with
the same mark, resulting in $P^{(l,k)}_{\mathrm{MIF}}$ and
$P^{(l,k)}_{\mathrm{MVF}}$:

$M^{(l,k)}=linear^{(l,k)}([P^{(l,k)}_{\mathrm{IR}};P^{(l,k)}_{\mathrm{VI}}])$
(3)
$P^{(l,k)}_{\mathrm{MIR}}=P^{(l,k)}_{\mathrm{IR}}+M^{(l,k)},P^{(l,k)}_{\mathrm{MVI}}=P^{(l,k)}_{\mathrm{VI}}+M^{(l,k)}$
(4)

where $[\cdot;\cdot]$ denotes concatenation along the channel dimension, and
$M^{(l,k)}$ represents the mark at $l$-th layer and $k$-th CMSAB. Patch Mark
effectively groups patches from the same positions in both modalities,
enabling a clear distinction between inter-modal interactions and spatial
interactions.

####  3.3.2 Interleaving

The Interleaving operation flattens the features of two modalities into
sequences in four directions. Subsequently, sequences from the same unfolding
direction in the two modalities are merged alternately into a new sequence. As
depicted in Fig.3, the features are initially unfolded into eight sequences
based on combinations of vertical, horizontal, forward, and backward
directions. Then, sequences with the same flattening direction are interleaved
to form four long sequences:
$\Phi^{(l,k)}_{\mathrm{HF}},\Phi^{(l,k)}_{\mathrm{HB}},\Phi^{(l,k)}_{\mathrm{VF}},\Phi^{(l,k)}_{\mathrm{VB}}=\mathrm{inter}(P^{(l,k)}_{\mathrm{MIR}},P^{(l,k)}_{\mathrm{MVI}})$
where $\mathrm{inter}$ denotes the interleaving operation. Subscripts
$\mathrm{HF}$ and $\mathrm{VB}$ represent Horizontal Forward and Vertical
Backward flattening directions, respectively, and similarly for others. Taking
$\Phi^{(l,k)}_{\mathrm{HF}}$ in Fig.3 as an example, due to Patch Mark
grouping patches, patches with the same number can be considered as a whole.
Each grouped patches is considered as a single entity, representing the
spatial dimension from left to right across the entire sequence.

####  3.3.3 Cross SS2D

Cross SS2D is employed to capture contextual relationships on interleaved long
sequences. Cross SS2D first computes $\Delta$, $B$, and $C$ using different
parameters for each modality. Subsequently, these computed parameters are
interleaved so that each patch on every sequence corresponds to its own
modality-specific parameters. While this approach may seem to hinder
interactions between modalities, both modalities share a common hidden state.
Using modality-specific parameters allows selective compression of relevant
information within each modality based on information from the other modality,
or selective extraction of relevant information from the hidden state. As
illustrated in Fig.2(c), from the modality perspective, Cross SS2D iteratively
traverses between the two modalities to accomplish inter-modality
interactions. Moreover, thanks to Patch Mark, Cross SS2D can consider grouped
patches as a single entity, enabling the consideration of saliency information
from a global spatial perspective. This enables comprehensive perception and
integration of saliency information within both modalities:
$F^{(l,k)}_{\mathrm{HF}},F^{(l,k)}_{\mathrm{HB}},F^{(l,k)}_{\mathrm{VF}},F^{(l,k)}_{\mathrm{VB}}=CSS2D(\Phi^{(l,k)}_{\mathrm{HF}},\Phi^{(l,k)}_{\mathrm{HB}},\Phi^{(l,k)}_{\mathrm{VF}},\Phi^{(l,k)}_{\mathrm{VB}})$,
where $CSS2D$ denotes Cross SS2D.

Algorithm 1 describes the Cross SS2D algorithm in a manner similar to
Mamba[6]. Assuming the input sequence length is $2L$, where $L$ is the number
of patches in the feature map, $H$ and $B$ represent the size of the hidden
state in SSM and the batch size, respectively. $S_{\cdot}(\cdot)$ denotes a
linear layer, where the first letter in the subscript indicates the perceived
parameter(e.g. $B$ and $C$), and the second letter denotes the modality (i.e.
I and V). $\tau_{\Delta}$ represents softplus function.
$\mathrm{Inter}(\cdot,\cdot)$ represents the interleaving arrangement of
linear layers from different modalities, corresponding to the input sequence
interleaved with modalities. This means that parameters $\Delta$, $B$, $C$,
and $A$ have different perceptions for different modalities.
$\mathrm{discretize}(\cdot,\cdot,\cdot)$ represents the process of
discretization, calculated according to section 3.1.1. We provide a more
detailed explanation of these parameters in the appendixA.3.

Algorithm 1 Cross SS2D

0: x: (B, 2L, C)

0: y: (B, 2L, C)

$\textbf{A}:(B,L,H)\leftarrow Parameter$

$\textbf{B}:(B,2L,H)\leftarrow\mathrm{Inter}(S_{\mathrm{BI}},S_{\mathrm{BV}})(\textbf{x})$

$\textbf{C}:(B,2L,H)\leftarrow\mathrm{Inter}(S_{\mathrm{CI}},S_{\mathrm{CV}})(\textbf{x})$

$\Delta:(B,2L,H)\leftarrow\tau_{\Delta}(Parameter+\mathrm{Inter}(S_{\mathrm{\Delta
I}},S_{\mathrm{\Delta V}})(\textbf{x}))$

$\bar{\textbf{A}},\bar{\textbf{B}}:(B,2L,C,H)\leftarrow\mathrm{discretize}(\Delta,\textbf{A},\textbf{B})$

$\textbf{y}\leftarrow
SSM(\bar{\textbf{A}},\bar{\textbf{B}},\textbf{C})(\textbf{x})$

return y

####  3.3.4 Recovering

Recovering is responsible for disassembling the features after Cross SS2D
interaction and restoring them to their original feature map shapes. This
process is illustrated in Fig.2. Initially, the features are disassembled into
sequences organized in their original four directions. Subsequently, patches
at corresponding positions within each modality are added together and
combined to reconstruct the original shape of the feature map:
$T_{\mathrm{IR}}^{(l,k)},T_{\mathrm{VI}}^{(l,k)}=\mathrm{Recover}(F^{(l,k)}_{\mathrm{HF}},F^{(l,k)}_{\mathrm{HB}},F^{(l,k)}_{\mathrm{VF}},F^{(l,k)}_{\mathrm{VB}})$.
The recovered features, as shown in Fig.1(c), are passed through layer
normalization, a gating mechanism, and residual connections to obtain the
output $V^{(l,k+1)},I^{(l,k+1)}$ of the $k$-th CMSAB layer.

###  3.4 Loss

The loss function comprises four components: a L1 loss
$\mathcal{L}_{\mathrm{L1}}$, a structural similarity loss
$\mathcal{L}_{\mathrm{ssim}}$, a gradient loss $\mathcal{L}_{\mathrm{grad}}$,
and our proposed perceptual enhancement loss $\mathcal{L}_{\mathrm{per}}$. The
$\mathcal{L}_{\mathrm{total}}$ is

$\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{per}}+\alpha_{1}\mathcal{L}_{\mathrm{L1}}+\alpha_{2}\mathcal{L}_{\mathrm{ssim}}+\alpha_{3}\mathcal{L}_{\mathrm{grad}}$
(5)

where $\alpha_{1}$, $\alpha_{2}$, and $\alpha_{3}$ are hyperparameters used to
balance each loss.

$\mathcal{L}_{\mathrm{ssim}}$ and $\mathcal{L}_{\mathrm{L1}}$ are inspired by
U2Fusion[41]. We use a pretrained ResNet50 to measure the amount of
information of the two original images, obtaining estimates $\omega_{1}$ and
$\omega_{2}$ to compute $\mathcal{L}_{\mathrm{L1}}$ and
$\mathcal{L}_{\mathrm{ssim}}$.

To simulate human visual perception and highlight saliency, we design
$\mathcal{L}_{\mathrm{per}}$ loss at the decision layer of the pretrained
ResNet50[10]. For the fusion result $F$, if its salient targets are more
prominent, it should contain more definitive information rather than noise. We
assume $y$ as the probability distribution of ResNet50’s estimates for $F$
over 1000 categories. The Entropy of this distribution estimates the
certainty, reflecting the saliency target perception:
$\mathcal{L}_{\mathrm{per}}=-\sum\limits_{i=1}^{1000}y_{i}log(y_{i})$, where
$y=\mathrm{softmax}(ResNet50(F))$. $\mathcal{L}_{\mathrm{per}}$ minimizes this
uncertainty to highlight salient objects in the two original images. However,
focusing solely on salient targets may lead to loss of texture details and
degrade image quality. Moreover, excessive convergence of
$\mathcal{L}_{\mathrm{per}}$ can result in image quality deterioration.
Therefore, we introduce $\mathcal{L}_{\mathrm{grad}}$ to preserve texture
details while restraining the perceptual enhancement loss:
$\mathcal{L}_{\mathrm{grad}}=|||\nabla F|-\max(|\nabla I|,|\nabla V|)||_{1}$.
Where $\nabla$ represents the Sobel operator used to compute the edge texture
information of the image.

Table 1: Quantitative Comparison on MSRS[34] and RoadScene[42] datasets |  | MSRS[34] | RoadScene[42]
---|---|---|---
Method | year | MI | SF | AG | SSIM | VIF | Qabf | MI | SF | AG | SSIM | VIF | Qabf
DeFusion | 2022 | 3.35 | 8.61 | 2.78 | 0.46 | 0.76 | 0.53 | 2.96 | 8.82 | 3.55 | 0.43 | 0.51 | 0.37
U2Fusion | 2022 | 2.28 | 6.21 | 1.98 | 0.32 | 0.52 | 0.29 | 2.68 | 12.41 | 5.05 | 0.50 | 0.54 | 0.49
ReCoNet | 2022 | 2.28 | 9.98 | 3.00 | 0.20 | 0.49 | 0.40 | 3.13 | 9.01 | 3.78 | 0.45 | 0.54 | 0.38
SemLA | 2023 | 2.45 | 6.34 | 2.24 | 0.37 | 0.61 | 0.29 | 2.75 | 13.86 | 4.26 | 0.39 | 0.49 | 0.30
DDFM | 2023 | 2.73 | 7.39 | 2.52 | 0.45 | 0.74 | 0.47 | 1.85 | 11.41 | 4.66 | 0.04 | 0.10 | 0.15
IRFS | 2023 | 2.15 | 9.89 | 3.16 | 0.43 | 0.74 | 0.48 | 2.82 | 10.21 | 3.93 | 0.49 | 0.57 | 0.42
DATFuse | 2023 | 3.89 | 10.93 | 3.57 | 0.45 | 0.91 | 0.64 | 3.70 | 11.34 | 4.07 | 0.46 | 0.59 | 0.47
EMMA | 2024 | 4.24 | 11.56 | 3.79 | 0.48 | 0.97 | 0.64 | 3.27 | 15.82 | 6.14 | 0.46 | 0.62 | 0.44
S4Fusion | - | 3.81 | 11.68 | 3.80 | 0.49 | 1.00 | 0.68 | 3.75 | 16.16 | 6.24 | 0.50 | 0.79 | 0.61

Table 2: Quantitative Comparison on M3FD[19] Dataset Dataset | MI | SF | AG | SSIM | VIF | Qabf
---|---|---|---|---|---|---
DeFusion | 2.89 | 7.75 | 2.79 | 0.42 | 0.53 | 0.34
U2Fusion | 2.62 | 11.21 | 4.26 | 0.49 | 0.61 | 0.53
ReCoNet | 2.99 | 11.06 | 4.23 | 0.43 | 0.57 | 0.49
SemLA | 2.61 | 7.37 | 2.75 | 0.28 | 0.38 | 0.18
DDFM | 2.68 | 9.58 | 3.38 | 0.31 | 0.49 | 0.38
IRFS | 2.76 | 10.92 | 3.68 | 0.48 | 0.62 | 0.50
DATFuse | 4.09 | 10.97 | 3.70 | 0.46 | 0.63 | 0.50
EMMA | 3.81 | 15.88 | 5.72 | 0.46 | 0.75 | 0.59
S4Fusion | 4.43 | 15.36 | 5.31 | 0.48 | 0.91 | 0.66
Table 3: Various ablation experiments conducted on MSRS[34]. "Number" represents the experiment ID, and "$\to$" indicates module replacement. Number | Method | MI | AG | VIF | Qabf | SSIM | SF
---|---|---|---|---|---|---|---
Exp.1 | CMSA$\to$ ADD | 3.24 | 3.78 | 0.84 | 0.61 | 0.46 | 11.52
Exp.2 | CMSA$\to$ CNN | 3.61 | 3.70 | 0.91 | 0.63 | 0.47 | 11.66
Exp.3 | CMSA$\to$ CA | 2.83 | 3.53 | 0.66 | 0.47 | 0.39 | 11.62
Exp.4 | CMSA$\to$ FSSM | 3.5 | 3.77 | 0.98 | 0.64 | 0.48 | 11.50
Exp.5 | w/o PM | 3.27 | 3.78 | 0.85 | 0.63 | 0.46 | 11.49
Exp.6 | w/o Per | 3.71 | 3.75 | 0.97 | 0.65 | 0.48 | 11.03
Exp.7 | S4Fusion | 3.81 | 3.80 | 1.00 | 0.68 | 0.49 | 11.68

Table 4: AP@50 values for MM Detection on M3FD[19] Dataset | Car | People | Truck | Lamp | Motor | Bus | mAP@50
---|---|---|---|---|---|---|---
IR | 86.1 | 77.0 | 74.9 | 51.5 | 62.1 | 91.0 | 73.8
VI | 89.4 | 66.8 | 79.3 | 80.4 | 68.6 | 91.5 | 79.3
DeFusion | 89.6 | 75.0 | 80.4 | 77.8 | 68.2 | 91.9 | 80.5
DDFM | 90.0 | 73.8 | 77.5 | 77.6 | 71.7 | 92.0 | 80.4
ReCoNet | 89.9 | 72.8 | 78.3 | 78.6 | 69.9 | 90.8 | 80.1
SemLA | 87.4 | 71.4 | 75 | 73.2 | 65.9 | 90.2 | 77.2
U2Fusion | 90.1 | 75.1 | 78.4 | 77.8 | 65.4 | 90.6 | 79.6
IRFS | 90.4 | 73.9 | 79 | 78.6 | 73.3 | 90.9 | 81.0
DATFuse | 88.8 | 74.4 | 79.6 | 73.1 | 69.8 | 91.6 | 79.6
EMMA | 89.2 | 72.2 | 78.2 | 71.9 | 67.1 | 90.5 | 78.2
S4Fusion | 90.1 | 72.8 | 78.4 | 80.7 | 72.4 | 93.4 | 81.3
Figure 4: Qualitative comparison of the original images and the results of all
methods on "03989" and "00338" from M3FD[19]. Figure 5: Visualization of the
feature maps before and after the second-layer CMSA in M3FD[19]for "00388",
along with the original images. AF and BF represent "After" and "Before",
indicating the feature maps post- and pre-CMSA, respectively.

##  4 Experiment

Existing datasets suffer from distributional biases, yet the objectives of
significance are diverse. To address this, we amalgamate the training sets of
MSRS[34], 70% of M3FD[19], and 70% of RoadScene[42] to form a sufficiently
diversified dataset for training, with the remaining portions reserved for
testing and validation. We set the hyperparameters $\alpha_{1},\alpha_{2}$,
and $\alpha_{3}$ to 15, the number of layers $N$ to 3, the number of CMSAB $K$
to 3, and the number of VSS Blocks in both Encoders and Decoder to
$G=[1,2,1]$. For further training details, please refer to the appendixA.1. We
compare our approach with eight state-of-the-art methods, namely U2Fusion[41],
DeFusion[17], DDFM[50], DATFusion[36], SemLA[40], ReCoNet[11], IRFS[3], and
EMMA[49]. For quantitative evaluation, we employ six metrics according to
previous works[12; 18; 49]: Mutual Information (MI)[28], Spatial Frequency
(SF)[4], Structural Similarity Index Measure[39] (SSIM), Visual Information
Fidelity [31](VIF), Average Gradient (AG), and Qabf[27]. For downstream tasks,
we utilize AP@50 on M3FD for Multi-Modality Object detection task evaluation.
For specific details, please refer to the appendixA.1.

##  5 Qualitative Comparison

As illustrated in Fig.4, our approach remains effective in highlighting
salient objects clearly while preserving as much texture detail as possible.
Other methods, such as ReCoNet and DeFusion, fail to achieve comparable
clarity. Conversely, when faced with excessive noise in infrared images, our
method avoids excessive bias towards the infrared modality. In contrast,
methods like DeFusion, IRFS, and U2Fusion suffer from a lack of adaptability,
leading to blurred text on road signs due to over-reliance on the infrared
modality. More qualitative results are provided in the appendixA.2.

##  6 Quantitative Comparison

As shown in Table.1 and 3.4, we conducted quantitative testing on three
datasets. Compared to other methods, ours yielded outstanding results. Our
method demonstrated higher SSIM, AG, and SF, indicating that it preserves more
texture details and structure information and aligns better with human
perception. Additionally, our method also achieved good results in Qabf, VIF,
and MI, indicating that it retains more original image information.

As shown in Table. 4, our approach achieves excellent results in multimodal
object detection. Our mAP@50 surpasses other methods, indicating that overall,
our approach effectively highlights salient objects. While our method also
shows some effectiveness in the People and Truck categories, slight deviations
due to inherent biases in ResNet50 lead to slightly inferior performance
compared to other methods.

##  7 Ablation Study

To demonstrate the validity of our approach, we conducted ablation experiments
as shown in Table 3.4. In Exp.6, we removed the proposed perceptual
enhancement loss function while maintaining the remaining experimental
configurations to verify its effectiveness. The results confirmed the efficacy
of our proposed perceptual enhancement loss. Regarding module design, in
Exp.1-4, we replaced the proposed CDM with ADD, Cross Attention[45], CNN[10],
and FSSM[26], respectively, while adjusting parameters to maintain consistency
of model size with our proposed method. Among these, ADD, which directly adds
features, yielded inferior results due to the lack of interaction between
features. Cross Attention(CA) performed poorly due to its lack of spatial
interaction and certain incompatibilities with SSM. CNN suffered from
information loss to some extent due to the lack of a global receptive field,
while FSSM lost significant saliency information as it couldn’t simultaneously
interact within and between modalities. In Exp.5, we removed Patch Mark to
assess its role in distinguishing spatial and inter-modality interactions in
Cross SS2D. The results indicated a significant decrease in information
retention and fidelity when Patch Mark was absent.

##  8 Visualization

To investigate how CMSA operates, we analyzed the features pre- and post-
processing. Fig.5 illustrates the changes observed. Prior to CMSA processing,
visible features displayed a blurry human figure and unclear background,
whereas infrared features showed the opposite characteristics. Following CMSA
processing, infrared features highlighted the human figure against the
background, while both human figures and backgrounds in the visible spectrum
were improved. This showcases the CMSA module’s capacity to capture and
integrate information from different modalities, facilitating the enhancement
of significant elements through their synergistic interaction.

##  9 Conclusion

This paper tackles the challenge of adapting reinforcement saliency objectives
within pre-trained neural networks’ decision layers for perceiving the
uncertainty of fused images. Furthermore, to enhance the integration of
complementary information across modalities at a global level, we introduce a
one-stage fusion module, named CMSA. This module addresses the lack of global
space information in previous methods during the intermodal interaction.
Finally, we conduct extensive experiments to validate the effectiveness of our
proposed approach.

## References

  * [1] H. Cao, Y. Wang, J. Chen, D. Jiang, X. Zhang, Q. Tian, and M. Wang.  Swin-unet: Unet-like pure transformer for medical image segmentation.  In European conference on computer vision, pages 205–218. Springer, 2022.
  * [2] Z. Cao, X. Wu, L.-J. Deng, and Y. Zhong.  A novel state space model with local enhancement and state sharing for image fusion.  arXiv preprint arXiv:2404.09293, 2024.
  * [3] W. Di, L. Jinyuan, R. Liu, and F. Xin.  An interactively reinforced paradigm for joint infrared-visible image fusion and saliency object detection.  In Information Fusion, 2023.
  * [4] A. M. Eskicioglu and P. S. Fisher.  Image quality measures and their performance.  IEEE Transactions on communications, 43(12):2959–2965, 1995.
  * [5] K. Falahkheirkhah, K. Yeh, M. P. Confer, and R. Bhargava.  Drb-net: Dilated residual block network for infrared image restoration.  In International Symposium on Visual Computing, pages 104–115. Springer, 2022.
  * [6] A. Gu and T. Dao.  Mamba: Linear-time sequence modeling with selective state spaces.  arXiv preprint arXiv:2312.00752, 2023.
  * [7] A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré.  Hippo: Recurrent memory with optimal polynomial projections.  Advances in neural information processing systems, 33:1474–1487, 2020.
  * [8] A. Gu, K. Goel, and C. Ré.  Efficiently modeling long sequences with structured state spaces.  arXiv preprint arXiv:2111.00396, 2021.
  * [9] A. Gu, I. Johnson, K. Goel, K. Saab, T. Dao, A. Rudra, and C. Ré.  Combining recurrent, convolutional, and continuous-time models with linear state space layers.  Advances in neural information processing systems, 34:572–585, 2021.
  * [10] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [11] Z. Huang, J. Liu, X. Fan, R. Liu, W. Zhong, and Z. Luo.  Reconet: Recurrent correction network for fast and efficient multi-modality image fusion.  In European conference on computer Vision, pages 539–555. Springer, 2022.
  * [12] H. Li and X.-J. Wu.  Densefuse: A fusion approach to infrared and visible images.  IEEE Transactions on Image Processing, 28(5):2614–2623, 2018.
  * [13] H. Li, X.-J. Wu, and T. Durrani.  Nestfuse: An infrared and visible image fusion architecture based on nest connection and spatial/channel attention models.  IEEE Transactions on Instrumentation and Measurement, 69(12):9645–9656, 2020.
  * [14] H. Li, X.-J. Wu, and J. Kittler.  Rfn-nest: An end-to-end residual fusion network for infrared and visible images.  Information Fusion, 73:72–86, 2021.
  * [15] H. Li, J. Zhao, J. Li, Z. Yu, and G. Lu.  Feature dynamic alignment and refinement for infrared–visible image fusion: Translation robust fusion.  Information Fusion, 95:26–41, 2023.
  * [16] J. Li, Y. Peng, and T. Jiang.  Embedded real-time infrared and visible image fusion for uav surveillance.  Journal of Real-Time Image Processing, 18(6):2331–2345, 2021.
  * [17] P. Liang, J. Jiang, X. Liu, and J. Ma.  Fusion from decomposition: A self-supervised decomposition approach for image fusion.  In European Conference on Computer Vision, pages 719–735. Springer, 2022.
  * [18] J. Liu, R. Dian, S. Li, and H. Liu.  Sgfusion: A saliency guided deep-learning framework for pixel-level image fusion.  Information Fusion, 91:205–214, 2023.
  * [19] J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong, and Z. Luo.  Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5802–5811, 2022.
  * [20] Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang, Q. Ye, and Y. Liu.  Vmamba: Visual state space model.  In International Conference on Machine Learning, 2024.
  * [21] C. Lu, Y. Schroecker, A. Gu, E. Parisotto, J. Foerster, S. Singh, and F. Behbahani.  Structured state space models for in-context reinforcement learning.  Advances in Neural Information Processing Systems, 36, 2024.
  * [22] J. Ma, W. Yu, P. Liang, C. Li, and J. Jiang.  Fusiongan: A generative adversarial network for infrared and visible image fusion.  Information fusion, 48:11–26, 2019.
  * [23] J. Ma, H. Zhang, Z. Shao, P. Liang, and H. Xu.  Ganmcc: A generative adversarial network with multiclassification constraints for infrared and visible image fusion.  IEEE Transactions on Instrumentation and Measurement, 70:1–14, 2020.
  * [24] E. Nguyen, K. Goel, A. Gu, G. Downs, P. Shah, T. Dao, S. Baccus, and C. Ré.  S4nd: Modeling images and videos as multidimensional signals with state spaces.  Advances in neural information processing systems, 35:2846–2861, 2022.
  * [25] J. Peng, H. Zhao, Z. Hu, Y. Zhuang, and B. Wang.  Siamese infrared and visible light fusion network for rgb-t tracking.  International Journal of Machine Learning and Cybernetics, 14(9):3281–3293, 2023.
  * [26] S. Peng, X. Zhu, H. Deng, Z. Lei, and L.-J. Deng.  Fusionmamba: Efficient image fusion with state space model.  arXiv preprint arXiv:2404.07932, 2024.
  * [27] G. Piella and H. Heijmans.  A new quality metric for image fusion.  In Proceedings 2003 international conference on image processing (Cat. No. 03CH37429), volume 3, pages III–173. IEEE, 2003.
  * [28] G. Qu, D. Zhang, and P. Yan.  Information measure for performance of image fusion.  Electronics letters, 38(7):1, 2002.
  * [29] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [30] J. Ruan and S. Xiang.  Vm-unet: Vision mamba unet for medical image segmentation.  arXiv preprint arXiv:2402.02491, 2024.
  * [31] H. R. Sheikh and A. C. Bovik.  Image information and visual quality.  IEEE Transactions on image processing, 15(2):430–444, 2006.
  * [32] H. Sun, Q. Liu, J. Wang, J. Ren, Y. Wu, H. Zhao, and H. Li.  Fusion of infrared and visible images for remote detection of low-altitude slow-speed small targets.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:2971–2983, 2021.
  * [33] L. Tang, J. Yuan, and J. Ma.  Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network.  Information Fusion, 82:28–42, 2022.
  * [34] L. Tang, J. Yuan, H. Zhang, X. Jiang, and J. Ma.  Piafusion: A progressive infrared and visible image fusion network based on illumination aware.  Information Fusion, 83:79–92, 2022.
  * [35] L. Tang, H. Zhang, H. Xu, and J. Ma.  Rethinking the necessity of image fusion in high-level vision tasks: A practical infrared and visible image fusion network based on progressive semantic injection and scene fidelity.  Information Fusion, 99:101870, 2023.
  * [36] W. Tang, F. He, Y. Liu, Y. Duan, and T. Si.  Datfuse: Infrared and visible image fusion via dual attention transformer.  IEEE Transactions on Circuits and Systems for Video Technology, 33(7):3159–3172, 2023.
  * [37] A. G. Tri Dao.  Transformers are ssms: Generalized models and efficient algorithms with structured state space duality.  In International Conference on Machine Learning, 2024.
  * [38] D. Wang, J. Liu, R. Liu, and X. Fan.  An interactively reinforced paradigm for joint infrared-visible image fusion and saliency object detection.  Information Fusion, 98:101828, 2023.
  * [39] Z. Wang and A. C. Bovik.  A universal image quality index.  IEEE signal processing letters, 9(3):81–84, 2002.
  * [40] H. Xie, Y. Zhang, J. Qiu, X. Zhai, X. Liu, Y. Yang, S. Zhao, Y. Luo, and J. Zhong.  Semantics lead all: Towards unified image registration and fusion from a semantic perspective.  Information Fusion, page 101835, 2023.
  * [41] H. Xu, J. Ma, J. Jiang, X. Guo, and H. Ling.  U2fusion: A unified unsupervised image fusion network.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
  * [42] H. Xu, J. Ma, Z. Le, J. Jiang, and X. Guo.  Fusiondn: A unified densely connected network for image fusion.  In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI), pages 12484–12491, 2020.
  * [43] H. Zhang, H. Xu, X. Tian, J. Jiang, and J. Ma.  Image fusion meets deep learning: A survey and perspective.  Information Fusion, 76:323–336, 2021.
  * [44] H. Zhang, H. Xu, Y. Xiao, X. Guo, and J. Ma.  Rethinking the image fusion: A fast unified image fusion network based on proportional maintenance of gradient and intensity.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 12797–12804, 2020.
  * [45] J. Zhang, H. Liu, K. Yang, X. Hu, R. Liu, and R. Stiefelhagen.  Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers.  IEEE Transactions on Intelligent Transportation Systems, 2023.
  * [46] X. Zhang and Y. Demiris.  Visible and infrared image fusion using deep learning.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  * [47] W. Zhao, S. Xie, F. Zhao, Y. He, and H. Lu.  Metafusion: Infrared and visible image fusion via meta-feature embedding from object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13955–13965, 2023.
  * [48] Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte, and L. Van Gool.  Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5906–5916, 2023.
  * [49] Z. Zhao, H. Bai, J. Zhang, Y. Zhang, K. Zhang, S. Xu, D. Chen, R. Timofte, and L. Van Gool.  Equivariant multi-modality image fusion.  CVPR, 2024.
  * [50] Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte, and L. Van Gool.  Ddfm: denoising diffusion model for multi-modality image fusion.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8082–8093, 2023.
  * [51] Z. Zhao, S. Xu, C. Zhang, J. Liu, and J. Zhang.  Bayesian fusion for infrared and visible images.  Signal Processing, 177:107734, 2020.
  * [52] Z. Zhao, S. Xu, C. Zhang, J. Liu, J. Zhang, and P. Li.  Didfuse: Deep image decomposition for infrared and visible image fusion.  In IJCAI, pages 970–976. ijcai.org, 2020.

##  Appendix A Appendix / supplemental material

###  A.1 Detailed Setup

Our model has three layers with channel numbers of [48, 96, 192], patch size
of 4×4, and adjacent patches overlapping by one pixel. All experiments were
conducted on an RTX 3090 Ti using AdamW as the optimizer and WarmupCosine for
learning rate scheduling, with an initial learning rate of 0.0003, peaking at
0.0015, and decaying to 0.0006. The batch size was set to 20, and training was
performed for 800 epochs. Input images were randomly cropped to 229×229 and
augmented with random flipping.

RoadScene and M3FD datasets were randomly split, with 70% for training and the
remainder for validation and testing, detailed in the code. For multi-modality
object detection, we used Yolov5 with a pre-trained YOLOv5s model. The fusion
results on M3FD were split 7:1:2 for training, validation, and testing,
respectively. All methods were evaluated under identical experimental
conditions, including the same random seeds, learning rates, and dataset
splits. Each method was trained for 40 epochs on its fusion images using
Yolov5, with the best model selected based on validation performance and
evaluated on the test set.

For ablation studies, we replaced the fusion module and adjusted its layer
count and channel numbers to match the parameter count of our method. For the
ADD replacement method, we adjusted the encoder and decoder channel numbers to
modulate model size. All ablation experiments were conducted under the same
experimental conditions to ensure fairness.

###  A.2 More Qualitative Comparison

In Fig.6,7,8,9,10,11, we provide additional qualitative comparisons to
demonstrate the effectiveness of our method. Our approach does not rely on
favoring a specific modality for fusion but adaptively integrates information.
For instance, in Fig.8, the visible light image shows a blurry person, while
the infrared image highlights the person clearly. Our method adaptively
perceives this, unlike other methods, except DeFusion, which fail to highlight
the salient target effectively. However, DeFusion performs poorly on edge
details, introducing artifacts not present in the original image, such as the
utility pole in the red box. In Fig.6, the person in the visible image stands
out against the background, whereas the infrared image is blurrier. Other
methods favor the infrared, retaining its information and causing the
originally clear person to become blurred. Our method preserves infrared
details on the face and visible information for the rest, making the person
more prominent than in the visible image. Additionally, our method retains
more complete details, aligning better with human perception. These examples
demonstrate that our method does not favor one modality over another but truly
achieves adaptive integration to highlight salient information.

###  A.3 Cross SS2D

In this section, we further explain the parameters of Cross SS2D in detail.
First, we elaborate on the operation of $\mathrm{Inter}$ in Cross SS2D. Taking
$\mathrm{Inter}(S_{\mathrm{BI}},S_{\mathrm{BV}})(\textbf{x})$ as an example,
we assume the input sequence is
$\textbf{x}=[I_{1},V_{1},I_{2},V_{2},...,I_{L},V_{L}]$, where $V_{i}$ and
$I_{i}$ represent the $i$-th patches of the visible and infrared modalities,
respectively, for $i\in\\{1,2,...,L\\}$. The $\mathrm{Inter}$ operation
arranges the parameters into a sequence of length 2L:
$\mathrm{Inter}(S_{\mathrm{BI}},S_{\mathrm{BV}})=[S_{\mathrm{BI}},S_{\mathrm{BV}},S_{\mathrm{BI}},S_{\mathrm{BV}},...,S_{\mathrm{BI}},S_{\mathrm{BV}}]$,
where the $S_{BI}$ and $S_{BV}$ are linear layers. For the linear layers, the
$\mathrm{Inter}$ operation simply aligns the same modality’s linear layer with
the sequence positions. Finally,
$\mathrm{Inter}(S_{\mathrm{BI}},S_{\mathrm{BV}})(\textbf{x})=[S_{\mathrm{BI}}(I_{1}),S_{\mathrm{BV}}(V_{1}),S_{\mathrm{BI}}(I_{2}),S_{\mathrm{BV}}(V_{2}),...,S_{\mathrm{BI}}(I_{L}),S_{\mathrm{BV}}(V_{L})]$,
meaning that the corresponding patches are input into their respective linear
layers to compute the corresponding $B$ parameters.

We denote $B$ as
$B=[B_{\mathrm{I}1},B_{\mathrm{V}1},B_{\mathrm{I}2},B_{\mathrm{V}2},...,B_{\mathrm{I}L},B_{\mathrm{V}L}]$,
where $B_{\mathrm{I}i}$ and $B_{\mathrm{V}i}$ represent the $B$ parameters for
the $i$-th patches in the sequence, with $i\in\\{1,2,...,N\\}$. Where,
subscripts V and I denote the visible and infrared modalities, respectively.
It’s worth noting that the Inter parameter merely arranges the linear layers
without duplicating parameters, indicating the presence of only two sets of
linear layer parameters.

####  A.3.1 Interpretation of A

According to the theory of parameter A in Mamba[6], parameter A can also
become selective. However, following Mamba’s design philosophy, we make
parameter A independent of the input and shared between the two modalities.
Based on the discretization formula $\bar{\textbf{A}}=\exp(\Delta\textbf{A})$,
making $\Delta$ selective is sufficient for $\bar{\textbf{A}}$ to be
selective, thus making A selective is unnecessary. Similarly, for two
different modalities, it is not required to set separate parameter A for each;
the respective parameters $\Delta$ are sufficient to distinguish their
respective $\bar{\textbf{A}}$ for different modalities.

Table 5: Comparison of the average $\Delta$ for the two modalities across different layers. | $\bar{\Delta}_{\mathrm{I}}$ | $\bar{\Delta}_{\mathrm{V}}$
---|---|---
Layer1 | 0.0014 | -0.0056
Layer2 | -3.999 | -5.885
Layer3 | -6.308 | -19.811

####  A.3.2 Interpretation of $\Delta$

Similar to parameter B, $\Delta$ can be expressed as
$\Delta=[\Delta_{\mathrm{I}1},\Delta_{\mathrm{V}1},\Delta_{\mathrm{I}2},\Delta_{\mathrm{V}2},...,\Delta_{\mathrm{I}L},\Delta_{\mathrm{V}L}]$.
Generally, $\Delta$ controls the relationship between the current and previous
inputs. In SSM, $\Delta$ represents the interval after discretizing a
continuous signal. Larger values of $\Delta$ indicate a greater emphasis on
the current input, while smaller values indicate the opposite. Notably, the
infrared parameter $\Delta_{\mathrm{I}}$ controls the interval for different
spatial positions of different modalities, whereas $\Delta_{\mathrm{V}}$
controls the interval for the same spatial position of different modalities.

For the $l$-th layer of CMSA, we denote $\bar{\Delta}_{\mathrm{I}}^{(l)}$ and
$\bar{\Delta}_{\mathrm{V}}^{(l)},l\in\\{1,2,3\\}$ as the average sizes of the
$\Delta$ parameters for infrared and visible in $l$-th layer. As shown in
Table.5, we conducted experiments on the MSRS test set, calculating the
average $\Delta$ parameter values for infrared and visible across all CMSABs
in each layer to obtain $\bar{\Delta}_{\mathrm{I}}^{(l)}$ and
$\bar{\Delta}_{\mathrm{V}}^{(l)},l\in\\{1,2,3\\}$. Due to the frequent inter-
modal and spatial interactions in Cross SS2D, this interleaving $\Delta$
effectively controls the transmission of information between modalities and
spatial locations.

Most of the time, $\bar{\Delta}_{\mathrm{I}}^{(l)}$, where $l\in\\{1,2,3\\}$,
represents the interval between adjacent patches of different modalities in
space. As shown in Table.5, in all layers, $\bar{\Delta}_{\mathrm{I}}^{(l)}$
is greater than $\bar{\Delta}_{\mathrm{V}}^{(l)}$. Larger values indicate
greater differences between inputs over time, emphasizing current position
information and compressing past information to capture spatial information.
This is because the information difference between spatially adjacent patches
is greater than the patches of different modalities in the same position. For
example, infrared and visible images at the same spatial position may capture
the same object, exhibiting strong complementarity.

Conversely, $\bar{\Delta}_{\mathrm{V}}^{(l)}$, where $l\in\\{1,2,3\\}$,
represents the interval between patches of the same position but different
modalities. These values are significantly smaller than
$\bar{\Delta}_{\mathrm{I}}^{(l)}$, indicating that the model prefers previous
information and selects a smaller portion of complementary information from
the current position. SSM reconstructs signals better when they are closer,
suggesting that $\bar{\Delta}_{\mathrm{V}}^{(l)}$ controls the acquisition of
complementary information between patches of different modalities at the same
position.

We also observe that the value of both $\bar{\Delta}_{\mathrm{I}}$ and
$\bar{\Delta}_{\mathrm{V}}$ decreases with increasing layers. This indicates
that as the semantic level increases, the model tends to consider each patch’s
information more deeply, forgetting the current hidden state less and
remembering past information more.

####  A.3.3 Interpretation of B and C

B and C regulate the input-to-hidden state and hidden state-to-output
information flow, respectively. We employ distinct parameters for each
modality to compute B and C. This facilitates the precise extraction and
retention of complementary modality and spatial information from the hidden
state, tailored to the specific characteristics of each modality’s
information.

##  Appendix B Limitation

Thanks to the pre-trained ResNet50 model, we can dynamically integrate useful
information from two original images. However, relying solely on a single
ResNet50 network to mimic human vision introduces inherent biases from the
pre-trained model. Additionally, as most classification networks are trained
on datasets with fixed categories (e.g., ImageNet-1K), their decision layers
are limited by these categories. Yet, real-world salient objects exhibit
diversity beyond these limitations. Using multiple pre-trained networks for
integration as a form of supervision and employing networks with "almost
infinite categorization capabilities" like CLIP[29] as supervision signals
will be left for future exploration. Nonetheless, we provide a new loss
function design idea and a new single-stage fusion module, which may offer
more insights for the image fusion community.

Figure 6: Qualitative comparison of the original images and the results of all
methods on "01416" from M3FD[19]. Figure 7: Qualitative comparison of the
original images and the results of all methods on "01432" from M3FD[19].
Figure 8: Qualitative comparison of the original images and the results of all
methods on "FLIR_06621" from RoadScene[42]. Figure 9: Qualitative comparison
of the original images and the results of all methods on "02119" from
M3FD[19]. Figure 10: Qualitative comparison of the original images and the
results of all methods on "00037N" from MSRS[34] Figure 11: Qualitative
comparison of the original images and the results of all methods on
"FLIR_08954" from RoadScene[42].


# Building Legal Datasets

Jerrold Soh Tsin Howe
Centre for Computational Law, School of Law, Singapore Management University
<EMAIL_ADDRESS>

###### Abstract

Data-centric AI calls for better, not just bigger, datasets. As data
protection laws with extra-territorial reach proliferate worldwide, ensuring
datasets are legal is an increasingly crucial yet overlooked component of
“better”. To help dataset builders become more willing and able to navigate
this complex legal space, this paper reviews key legal obligations surrounding
ML datasets, examines the practical impact of data laws on ML pipelines, and
offers a framework for building legal datasets.

##  1 Introduction

Data-centric AI is about making better datasets. But what does “better” mean?
Conventionally it has meant cheaper. That is, easier to crowdsource [14],
generate [4], augment [5], or broadly to collect [22]. Bigger is also often
better, as the rise of large language models suggest [12, 33]. To
statisticians, better typically means unbiased, though “bias” is used
differently from in the bias-variance tradeoff [8], or in algorithmic bias
[7]. The growing “responsible AI” literature emphasizes that datasets are
better when they are ethically and fairly sourced [22, 13, 24]. This paper
underscores legality as one desideratum for “better”. To this end, it reviews
key legal obligations on data collection and use, examines the practical
impact of data laws on ML pipelines, and offers a framework for thinking about
data legality.

##  2 When are datasets legal?

Legal datasets may be understood broadly as datasets which are legally
collected, retained, processed, and disseminated. This fourfold categorization
builds off Solove’s classic taxonomy of privacy [27], and finds expression in
a range of relatively new legislation worldwide. This notably includes the
European Union’s (EU’s) General Data Protection Regulation (GDPR) which came
into force in 2018. Parallel to the GDPR are national data laws, such as South
Korea’s Personal Information Protection Act (passed in 2011), Singapore’s
Personal Data Protection Act (passed in 2014) and, most recently, China’s
Personal Information Protection Law (PIPL, August 2021). While the US does not
presently have data legislation at the federal level, states like California,
New York, and Massachusetts have passed data privacy acts. Further, legal
scholars and courts have increasingly considered how pre-existing laws, such
as copyright and anti-discrimination law, affect ML datasets [25, 19, 10]. As
there are too many countries and variations to cover, I use the GDPR, PIPL,
and California’s Consumer Privacy Act (CCPA, 2018) as case studies.

Although one jurisdiction’s laws generally do not apply in another, modern
data laws tend to have extra-territorial effect. Both the GDPR and PIPL apply
as long as any personal data about persons in the EU/China is processed for
any commercial or behavioral monitoring purposes (GDPR, Art 3; PIPL, Art 3).
Likewise, Art 2 of the EU’s proposed Artificial Intelligence Act (AIA)
expressly covers AI systems deployed in, or whose outputs are used, in the EU,
regardless of where the providers and users of the system are. By contrast,
the CCPA applies primarily to large businesses which “do business in” the
state (CCPA, §1798.140). Thus, ML researchers and practitioners worldwide are
now subject to foreign, and increasingly complex [17], data laws. Below I non-
exhaustively review key legal obligations they impose. Note that “legal” here
refers only to formal law. This distinguishes my scope from (no less
important) work on “ethical”, “fair” or “responsible” AI [22, 13, 24]. Despite
clear overlaps, neither is a subset of the other. To illustrate, for some in
certain states abortion is ethical yet illegal, for others elsewhere it is
unethical yet legal.

###  2.1 Collection

Most centrally, data protection laws require informed consent before
“personal” data may be obtained (GDPR, Arts 6–11; PIPL, Arts 13–17). The CCPA
does not expressly require “consent”, but businesses must inform consumers of
the scope and purposes of data collected before collection (CCPA, §1798.100).
The legality of numerous facial recognition datasets has been challenged for
lack of consent [1, 22, 21]. Facial recognition clearly involves personal data
because the task is to identify. “Personal data” is, however, wider. Article 4
GDPR defines it as “any information relating to an identified or identifiable
natural person”. One is “identifiable” when they may be identified directly
(i.e. by name) or indirectly. Names are not necessary; zip codes, gender,
race, etc, could collectively identify. Indeed, Wong suggests that the EU’s
definition of personal data “appears to be capable of encompassing all
information in its ambit”, as EU courts have taken “personal” to include not
only data about a person, but also data which affects them [31].

The breadth of data laws explains why although most of ImageNet’s [6] label
classes do not target persons, its caretakers recently blurred out all human
faces in the data, citing privacy concerns [16]. A similar fate appears to
have befallen the new Meta’s facial recognition systems [21]. While most legal
scrutiny has been on images, text, sound, and other modalities can also be
“personal”. One’s forum posts, even if pseudonymous, could reveal much of
their background. As such, dataset builders should be deliberate about
obtaining consent even (or especially) when it is not obvious if the data is
“personal”.

###  2.2 Retention

A standard feature of legal consent is that consent may be withdrawn at any
time. Data subjects may request to correct or erase their data (GDPR, Arts
16–17; PIPL, Arts 15, 16, 44–47; CCPA, §1798.105–106). Beyond consent, data
controllers are also obliged to keep data in personally-identifiable form for
no longer than necessary for its stated purposes (GDPR, Art 5(1)(e); CCPA,
§1798.100(3)). Data that has served these purposes (say, the model has been
trained) must be deleted or anonymized. However, given that data previously
collected for one purpose can turn out useful for another, deletion may be
quite undesirable for ML engineers. Anonymization is not much better, since
preventing re-identification may require destroying most of a dataset’s
informative signals [23, 32].

As such, prior thought should be given to delineating, and communicating, what
the data will be used for. Conveying a specific purpose such as “training ML
models” may not cover maintaining or updating the model post-deployment. Too
general a purpose, such as “for ML processing” invites user suspicion and may
fall outside the legal requirement that consent must be given in respect of
“specific” purposes (GDPR, Art 6(1)(a); PIPL, Art 6).

###  2.3 Processing

Consent obligations surrounding data collection apply equally to data use.
Further, data subjects have a right to be informed of and object to decisions
“based solely on automated processing” (GDPR, Art 22; see also PIPL, Art 24).
This legally advantages human-in-the-loop systems. Beyond data protection
laws, anti-discrimination laws in certain jurisdictions (e.g. US disparate
treatment/impact laws; UK’s Equality Act 2010) may prohibit the use of
protected attributes like race and gender for profiling [11]. This restricts
the feature set which can legally be used for training ML models. Features
highly co-linear with protected attributes may be indirectly prohibited as
well.

While the obligations above may be more relevant to models than to datasets,
laws can target the latter directly. Most prominently, the proposed Art 10 AIA
stipulates that “[t]raining, validation and testing data sets” to be used in
what the Act identifies as “high-risk AI systems” shall be “relevant,
representative, free of errors and complete”. This extends to having
“appropriate statistical properties” regarding the system’s target persons,
and considering characteristics “particular to the … setting within which the
high-risk AI system is intended to be used”. The draft AIA is in early stages
and may take years and numerous amendments to come into force (if it does).
Should it become law as is, it may effectively render data-centricity legally
mandatory for “high-risk” AI. Examples of high-risk AI enumerated in Annex III
AIA non-exhaustively include systems for biometric identification, educational
assessments, recruitment, credit scoring, law enforcement, and judicial
decisions.

###  2.4 Sharing and disclosure

As data sharing or disclosure also constitutes processing, unauthorized
disclosure is also a breach (GPDR, Arts 4(2); PIPL, Art 25). Thus, datasets
with potentially personal information cannot be open-sourced without proper
anonymization, even for research purposes. Another concern particular to large
neural networks is the possibility that the network may memorize and leak
personal information in the training data [20, 3]. Personal information in
datasets used for training large neural networks may therefore need to be
removed before training.

###  2.5 Research exemptions

The above obligations may be subject to limited research exemptions whose
scope differs across jurisdictions [18]. For instance, both the GDPR and the
CCPA regard subsequent scientific or statistical research as compatible with
the initial purposes of the data collection for which consent was presumably
obtained. This allows research to proceed without needing to ask for
additional consent, subject to appropriate safeguards (GDPR, Arts 1(b) &
89(1); CCPA, §1798.140(s)). This may have been sufficient to cover research
applications of the ImageNet data (discussed above) without requiring
anonymization. China’s PIPL, however, does not have such an exemption.

##  3 Implications on ML pipelines

There is, in short, an expanding range of legal constraints on when and how
data may be used. This has obvious implications for the ML community. Since
legal data is necessarily a subset of all data, prioritizing legality seems to
require sacrificing model performance. But less data is not always worse,
especially if it also means less noise. More formally, if we think of ML
broadly as seeking $argmax_{H}g(H,y)$, where the hypothesis $H(\theta;X,D)$
takes weights $\theta$ learned from features $X$ in dataset $D$, and $g(H,y)$
is a performance metric measured against (holdout) truth labels $y$, then
legality constraints might be understood as follows:

$argmax_{H}g(H,y)\quad\mathbf{s.t.}\quad X\subseteq X_{legal};D\subseteq
D_{legal}$

with $X_{legal}$ and $D_{legal}$ respectively denoting the legally-permissible
set of features and datasets.

Formally framing the problem as such identifies three situations where legal
constraints may not necessarily limit model performance. For brevity, we
illustrate this with $D$, though similar logic applies with $X$. First, if
$D_{legal}\\!=\\!D_{\Omega}$ (all data relevant to a task). That is, data laws
have no practical effect on datasets in that area. For example, the task
involves only cat detection and never implicates personal data. Second, if
$D^{*}$, the theoretically optimal dataset, happens to be perfectly legal so
that $D^{*}=D_{legal}$. Third, and least obviously, if the legally-constrained
optimization problem produces better performance than its unconstrained
variant. This counter-intuitive result may occur in the real world because the
law may force one to exclude noisy data (or features) that would otherwise
have been included. In this sense, data laws, like other optimization
constraints, may turn out to have a useful regularizing effect.

Moreover, in practice $g(H,y)$ is not the only metric to be optimized. Even
assuming we care solely about economic value, profits, while correlated with
(F1 score) performance, turn also on variables like user adoption and trust
[28, 9, 15]. A perfectly accurate classifier that is never used generates no
revenue. Fines also reduce profit. Ignoring legality provides another source
of “hidden debt” in ML pipelines [26]. Thus. early investments in processes
and practices for making legal datasets could yield better real world
performance, particularly in the long-run (where legal enforcement becomes
more feasible). Apart from any obligation to follow the law just because it is
law, there are practical reasons why the ML community should do so.

##  4 Building legal datasets

Complying with the intricate and growing web of data laws is non-trivial. The
challenge is how we might turn motherhood calls for “multi-disciplinary
collaboration” into actionable steps for ML researchers. The rise of ethics
guidelines and responsible AI checklists [34, 24] offers one solution. In a
sense, this involves, ethicists, sociologists, etc pre-computing complex,
open-ended obligations into simpler, close-ended compliance heuristics for
computer scientists. Following this trend, Figure 1 offers a framework for
thinking about dataset legality. This builds on existing work that already
incorporates some legal principles (e.g. [24]) but differs in two ways. First,
the framework focuses more on legality and thus complements
responsible/ethical AI work. Second, while checklists and impact statements
are generally backward-looking, encouraging researchers to justify choices
already made, these considerations are forward-looking, encouraging
researchers to think about legality at each stage of the ML process. This is
crucial because legal errors, especially the need to obtain informed consent
for processing, are expensive to rectify post facto if not avoided ex ante.

Before Collection — Which Law(s) Apply? 1\. Will directly or indirectly
personally-identifying data (“personal data”) be collected? Note the wide
scope of “personal data” (see Part 2.1). 2\. If personal data is collected,
which countries are we (the data processors) in? What data laws do these
countries have? 3\. What countries are the data subjects in? Do these
countries have extra-territorial data laws? 4\. What is the data for? Are
these purposes (e.g. research) subject to any exemptions? During Collection —
Obtaining Consent 1\. Are subjects adequately informed of the purposes of and
consent to data collection? Adequate “consent” varies by country, but refer
generally to GDPR, Arts 6(1)(a) & 7. 2\. Ensure that communicated purposes
truly cover all intended data uses. Purposes may be stated more broadly to
leave room for future uses, but should not be framed too widely either. After
Collection — Data Use and Model Training  1\. Ensure that subjects who
withdraw consent are removed from the dataset. Data pipelines should be robust
to consent withdrawals. Already-generated artefacts should be re-generated.
2\. Check that protected attributes (and strong correlates) are excluded from
the feature set. After Training —- Deployment 1\. If the system is entirely
end-to-end with no humans-in-the-loop, subjects must be informed. 2\. If the
communicated purposes of data collection have been spent, the data may need to
be anonymized or deleted. 3\. Does the trained model leak personal data? If
so, either tweak the model and its outputs, or obtain consent for data
disclosure.

Figure 1: Suggested framework for dataset legality

##  5 Limitations

All heuristics are wrong, but some are useful [2]. This paper does not cover
all the legal obligations, duties, and exemptions affecting ML datasets. Nor
can following the proposed framework completely guarantee legality (nor
fairness or morality). Indeed, the corpus of legislation affecting ML datasets
is set to grow, amid concerns that current laws offer insufficient safeguards
[30, 29]. The draft AIA, if and when passed, would significantly alter the AI
and data governance landscape; other jurisdictions may follow suit with their
own Acts. The minutiae of dataset legality should be fleshed out in future,
lengthier work. The primary aim here is to spark discussion on when and why
legal data is better data. Data-centric AI presents an opportunity for the ML
community to build better datasets — in all the technical, statistical,
ethical, and legal senses of the word.

## Acknowledgments and Disclosure of Funding

The authors disclose no funding sources nor competing interests.

## References

  * noa [2021] Legality of collecting faces online challenged, May 2021.  URL https://www.bbc.com/news/technology-57268121.
  * Box [1976] G. Box.  Science and Statistics.  _Journal of the American Statistical Association_ , 71(356):791–799, 1976.  URL http://www-sop.inria.fr/members/Ian.Jermyn/philosophy/writings/Boxonmaths.pdf.
  * Chen et al. [2020] C. Chen, B. Wu, M. Qiu, L. Wang, and J. Zhou.  A Comprehensive Analysis of Information Leakage in Deep Transfer Learning.  _arXiv:2009.01989 [cs]_ , Sept. 2020.  URL http://arxiv.org/abs/2009.01989.  arXiv: 2009.01989.
  * Collins et al. [2008] B. Collins, J. Deng, K. Li, and L. Fei-Fei.  Towards Scalable Dataset Construction: An Active Learning Approach.  In D. Forsyth, P. Torr, and A. Zisserman, editors, _Computer Vision – ECCV 2008_ , volume 5302, pages 86–98. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008.  ISBN 978-3-540-88681-5 978-3-540-88682-2.  doi: 10.1007/978-3-540-88682-2_8.  URL http://link.springer.com/10.1007/978-3-540-88682-2_8.  Series Title: Lecture Notes in Computer Science.
  * Dao et al. [2019] T. Dao, A. Gu, A. J. Ratner, V. Smith, C. D. Sa, and C. Re.  A Kernel Theory of Modern Data Augmentation.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97, pages 1528–1537, 2019.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 248–255, 2009.
  * Friedman and Nissenbaum [1996] B. Friedman and H. Nissenbaum.  Bias in Computer Systems.  _ACM Transactions on Information Systems_ , 14(3):330–347, July 1996.
  * Geman et al. [1992] S. Geman, E. Bienenstock, and R. Doursat.  Neural Networks and the Bias/Variance Dilemma.  _Neural Computation_ , 4:1–58, 1992.
  * Gillath et al. [2021] O. Gillath, T. Ai, M. S. Branicky, S. Keshmiri, R. B. Davison, and R. Spaulding.  Attachment and trust in artificial intelligence.  _Computers in Human Behavior_ , 115:106607, Feb. 2021.  ISSN 07475632.  doi: 10.1016/j.chb.2020.106607.  URL https://linkinghub.elsevier.com/retrieve/pii/S074756322030354X.
  * Gillis and Spiess [2019] T. B. Gillis and J. L. Spiess.  Big Data and Discrimination.  _The University of Chicago Law Review_ , 86(2):459, 2019.
  * Hellman [2020] D. Hellman.  Measuring Algorithmic Fairness.  _Virginia Law Review_ , 106(4):811, 2020.
  * Hendrycks et al. [2020] D. Hendrycks, X. Liu, E. Wallace, A. Dziedzic, R. Krishnan, and D. Song.  Pretrained Transformers Improve Out-of-Distribution Robustness.  pages 2744–2751, July 2020.  doi: 10.18653/v1/2020.acl-main.244.  URL https://aclanthology.org/2020.acl-main.244.
  * Hutchinson et al. [2021] B. Hutchinson, A. Smart, A. Hanna, E. Denton, C. Greer, O. Kjartansson, P. Barnes, and M. Mitchell.  Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pages 560–575, Virtual Event Canada, Mar. 2021. ACM.  ISBN 978-1-4503-8309-7.  doi: 10.1145/3442188.3445918.  URL https://dl.acm.org/doi/10.1145/3442188.3445918.
  * Irani and Silberman [2013] L. C. Irani and M. S. Silberman.  Turkopticon: interrupting worker invisibility in amazon mechanical turk.  In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , pages 611–620, Paris France, Apr. 2013. ACM.  ISBN 978-1-4503-1899-0.  doi: 10.1145/2470654.2470742.  URL https://dl.acm.org/doi/10.1145/2470654.2470742.
  * Kerasidou et al. [2021] C. X. Kerasidou, A. Kerasidou, M. Buscher, and S. Wilkinson.  Before and beyond trust: reliance in medical AI.  _Journal of Medical Ethics_ , Aug. 2021.  doi: 10.1136/medethics-2020-107095.
  * Knight [2021] W. Knight.  Researchers Blur Faces That Launched a Thousand Algorithms.  _Wired_ , Mar. 2021.  ISSN 1059-1028.  URL https://www.wired.com/story/researchers-blur-faces-launched-thousand-algorithms/.  Section: tags.
  * Koops [2014] B.-J. Koops.  The trouble with European data protection law.  _International Data Privacy Law_ , 4(4):250–261, Nov. 2014.  ISSN 2044-3994, 2044-4001.  doi: 10.1093/idpl/ipu023.  URL https://academic.oup.com/idpl/article-lookup/doi/10.1093/idpl/ipu023.
  * Mabel and Tara [2019] C. Mabel and S. Tara.  The Research Exemption Carve Out: Understanding Research Participants Rights Under Gdpr and U.s. Data Privacy Laws.  _Jurimetrics_ , 60:125, 2019.
  * Mayson [2019] S. G. Mayson.  Bias In, Bias Out.  _Yale Law Journal_ , 128:2218, 2019.  URL https://www.yalelawjournal.org/pdf/Mayson_p5g2tz2m.pdf.
  * Nasr et al. [2019] M. Nasr, R. Shokri, and A. Houmansadr.  Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning.  _2019 IEEE Symposium on Security and Privacy (SP)_ , pages 739–753, May 2019.  doi: 10.1109/SP.2019.00065.  arXiv: 1812.00910.
  * O’Brien and Ortutay [2021] M. O’Brien and B. Ortutay.  Facebook to shut down face-recognition system, delete data.  _Washington Post_ , Nov. 2021.  ISSN 0190-8286.  URL https://www.washingtonpost.com/business/facebook-to-shut-down-face-recognition-system-delete-data/2021/11/02/79c0b782-3c02-11ec-bd6f-da376f47304e_story.html.
  * Paullada et al. [2020] A. Paullada, I. D. Raji, E. M. Bender, E. Denton, and A. Hanna.  Data and its (dis)contents: A survey of dataset development and use in machine learning research.  In _Proceedings of the Workshop on ML-Retrospectives, Surveys & Meta-Analyses @ NeurIPS 2020_, Dec. 2020.
  * Rocher et al. [2019] L. Rocher, J. M. Hendrickx, and Y.-A. de Montjoye.  Estimating the success of re-identifications in incomplete datasets using generative models.  _Nature Communications_ , 10(1), Dec. 2019.  ISSN 2041-1723.  doi: 10.1038/s41467-019-10933-3.  URL http://www.nature.com/articles/s41467-019-10933-3.
  * Rogers et al. [2021] A. Rogers, T. Baldwin, and K. Leins.  Just What do You Think You’re Doing, Dave?’ A Checklist for Responsible Data Use in NLP.  Sept. 2021.  URL http://arxiv.org/abs/2109.06598.
  * Sag [2019] M. Sag.  The New Legal Landscape for Text Mining and Machine Learning.  _Journal of the Copyright Society of the USA_ , 66:291, 2019\.  doi: 10.2139/ssrn.3331606.
  * Sculley et al. [2015] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison.  Hidden Technical Debt in Machine Learning Systems.  In _Proceedings of the 28th International Conference on Neural Information Processing Systems_ , volume 2, pages 2503–2511, 2015\.
  * Solove [2006] D. J. Solove.  A Taxonomy of Privacy.  _University of Pennsylvania Law Review_ , 154(3):477, Jan. 2006.  ISSN 00419907.  doi: 10.2307/40041279.  URL https://www.jstor.org/stable/10.2307/40041279?origin=crossref.
  * Toreini et al. [2020] E. Toreini, M. Aitken, K. Coopamootoo, K. Elliott, C. G. Zelaya, and A. van Moorsel.  The relationship between trust in ai and trustworthy machine learning technologies.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , FAT* ’20, page 272–283, New York, NY, USA, 2020. Association for Computing Machinery.  ISBN 9781450369367.  doi: 10.1145/3351095.3372834.  URL https://doi.org/10.1145/3351095.3372834.
  * United Nations High Commissioner for Human Rights [2021] United Nations High Commissioner for Human Rights.  The right to privacy in the digital age.  Technical Report A/HRC/48/31, Sept. 2021.
  * Wachter [2019] S. Wachter.  Data protection in the age of big data.  _Nature Electronics_ , 2(1):6–7, Jan. 2019.  ISSN 2520-1131.  doi: 10.1038/s41928-018-0193-y.  URL https://www.nature.com/articles/s41928-018-0193-y.  Institution: Nature Publishing Group Number: 1.
  * Wong [2019] B. Wong.  Delimiting the concept of personal data after the GDPR.  _Legal Studies_ , 39(3):517–532, Sept. 2019.  ISSN 0261-3875, 1748-121X.  doi: 10.1017/lst.2018.52.  URL https://www.cambridge.org/core/product/identifier/S0261387518000521/type/journal_article.
  * Xu and Zhang [2021] H. Xu and N. Zhang.  Implications of Data Anonymization on the Statistical Evidence of Disparity.  _Management Science_ , pages 1–19, 2021.  ISSN 1556-5068.  doi: 10.2139/ssrn.3662612.
  * Zhong et al. [2021] R. Zhong, D. Ghosh, D. Klein, and J. Steinhardt.  Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3813–3827, Aug. 2021.  doi: 10.18653/v1/2021.findings-acl.334.  URL https://aclanthology.org/2021.findings-acl.334.
  * Zook et al. [2017] M. Zook, S. Barocas, D. Boyd, K. Crawford, E. Keller, S. P. Gangadharan, A. Goodman, R. Hollander, B. A. Koenig, J. Metcalf, A. Narayanan, A. Nelson, and F. Pasquale.  Ten simple rules for responsible big data research.  _PLOS Computational Biology_ , 13(3):e1005399, Mar. 2017.  ISSN 1553-7358.  doi: 10.1371/journal.pcbi.1005399.  URL https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005399.