# Array Camera Image Fusion using Physics-Aware Transformers

Qian Huang
Wyant College of Optical Sciences
University of Arizona
Tucson, AZ 85721
<EMAIL_ADDRESS>
Minghao Hu
Wyant College of Optical Sciences
University of Arizona
Tucson, AZ 85721
<EMAIL_ADDRESS>
David J. Brady
Wyant College of Optical Sciences
University of Arizona
Tucson, AZ 85721
<EMAIL_ADDRESS>Qian Huang and Minghao Hu are students at Duke University.
This work was finished when they were doing internship at the University of
Arizona.

###### Abstract

We demonstrate a physics-aware transformer for feature-based data fusion from
cameras with diverse resolution, color spaces, focal planes, focal lengths,
and exposure. We also demonstrate a scalable solution for synthetic training
data generation for the transformer using open-source computer graphics
software. We demonstrate image synthesis on arrays with diverse spectral
responses, instantaneous field of view and frame rate.

##  1 Introduction

In contrast with systems that use physical optics to form images,
computational imaging uses physical processing to code measurements but relies
on electronic processing for image formation [1]. In optical systems,
computational imaging enables "light field cameras [2]" that capture high
dimensional spatio-spectral-temporal data cubes. The primary challenge of
light field camera design is that, while the light field is 3, 4 or 5
dimensional, measurements still rely on 2D photodetector arrays. High
dimensional light field capture on 2D detectors can be achieved using three
sampling approaches: interleaved coding, temporal coding and multiaperture
coding. Interleaved coding, as is famously done with color filter arrays [3],
consists of enabling adjacent pixels on the 2D plane to access different parts
of the light field. Mathematically similar sampling strategies for depth of
field are implemented in integral imaging [4] and plenoptic cameras [5]. This
interleaved approach generalizes to arbitrary high dimensional data cubes in
the context of snapshot compressive imaging [6]. Temporal coding consists of
scanning the spectral [7] or focal [8] response of the camera during
recording.

Array cameras [9] offer many potential advantages over interleaved and
temporal coding. The advantage over temporal scanning is obvious, a camera
array can capture snapshot light fields and does not therefore sacrifice
temporal resolution. Additionally, development of cameras with dynamic
spectral, spatial and focal sampling is more challenging than development of
array components that sample slices of the data cube. The advantage of
multiaperture cameras relative to interleaved sampling is more subtle,
although implementation of interleaved sampling is also physically
challenging. On a deeper level, however, interleaved sampling makes the
physically implausible assumption that temporal sampling rates and exposure
should be the same for different regions of the light field. In practice,
photon flux in the blue is often very different from in the red and setting
these channels to a common exposure level is unfortunate. The design of lenses
and sensors optimized for specific spectral and focal ranges leads to higher
quality data.

With these advantages in mind, many studies have previously considered array
cameras for computational imaging [10, 11, 12]. More recently, artificial
neural networks have found extensive application in array camera control and
image processing [13, 14]. Of course, biological imaging systems rely heavily
on array imaging solutions. While conventional array cameras originally relied
on image-based registration [15] for "stitching", biological systems integrate
multiaperture data deep in the visual cortex. In analogy with the biological
system approach, here we demonstrate that array camera image fusion from deep
layer features, rather than pixel maps, is effective in data fusion from
diverse camera arrays. Our approach is based on transformers [16, 17]
networks, which excel at establishing long-range connections and integrating
related features.

Since transformer networks are more densely connected than convolutional
networks, high computational costs have inhibited their use in common computer
vision tasks. Non-local neural connections drastically increase the receptive
field for each voxel. As shown here, however, when the transformer is
integrated with the physics of the system the connections outside of the
physical receptive fields can be trimmed to the extent that the complexity of
transformers is comparable to convolutional networks.

As with many physical image capture and processing tasks, the forward model
for array camera imaging is easily simulated but the inverse problem is
difficult to describe analytically. This class of problems can be addressed by
training neural processors on synthetic data. Recent achievements in render
engines like GPU-aided ray tracing allow us to realistically, accurately and
efficiently model the world and render the modeled world to sensors of ideal
virtual cameras. The rendered frames can be regarded as the ground truth.
Synthetic sensor data that is degraded can be generated from the ground truth
via the forward model of the camera array. Surprisingly, it is unnecessary to
build photorealistic real-world scenes for some computer vision problems,
including image fusion; scenes with abstract objects native to computer
graphics software with diverse colors and textures can span the problem domain
of image fusion. Along with the programming interface provided by Blender
[18], this data synthesis pipeline can be easily deployed and automatically
scaled up to fit diverse data demands.

Here we use this approach to build a physics-aware transformer (PAT) network
that can fuse data from the array cameras of the diverse resolution, color
spaces, focal planes, focal lengths, and exposure. The purpose of this system
is to combine data from array cameras to return a computed image superior to
the image available to any single camera. Array cameras are designed in these
systems to exploit differences in the spatial and temporal resolution needed
to capture color, texture and motion.

We demonstrate four example systems. The first system combines data from wide
field color cameras with narrow field monochrome cameras, the second combines
color information from visible cameras with the textural information from near
infrared cameras, the third combines short exposure monochrome imagery with
long exposure narrow spectral band data and the fourth combines high frame
rate monochrome data with low frame rate color images. As a group, these
designs combine with PAT processing to show that array cameras optimized for
effective data capture can create virtual cameras with radically improved
dynamic range, color fidelity and spatial/temporal resolution.

##  2 Related Work

There are three main branches of combining physical models with neural
algorithms. First, plugging a learned model as a prior into a physical model,
which is also known as "plug and play". RED [19] is an example that applied a
denoiser as its prior. The second way initiated by Deep image priors [20] is
using a network architecture as a prior, thus removing the requirement of
pretraining. The methods above, however, are optimized for a scene in a loop,
restricting real-time applications. The third method is integrating the
physics of the system with the neural algorithm. The physics of the system can
take the form of, for example, a parameterized input to the algorithm (e.g.,
the noise level in a denoising system [21]) or a sub-module in the algorithm
architecture (e.g., the spatial transformer [22]). In a camera array, the
intrinsics and extrinsics are usually exploited. Using them to characterize an
array have several advantages: 1) they are not likely to change once the
cameras are encapsulated; 2) their derivatives like the epipolar geometry
naturally build connections of sensor pixels; 3) they can be achieved by
mature calibration techniques like [23] with efficiency. The epipolar
transformer [24] is an example that leverages the epipolar geometry to
estimate the human pose. Within the field of image fusion, the parallax-aware
attention model [25] was introduced to derive a high-resolution image from two
rectified low-resolution images. The model has been extended to process
unrectified pairs [26] and to solve general image restoration tasks from
homogeneous views [27]. Our algorithm is also inspired by this architecture
but focuses on general fusion tasks in the camera array.

Synthetic data is highly effective in training imaging systems with well-
characterized forward models. Array cameras, in particular, are easily
simulated in the forward process. The forward model accounts for both optical
components (e.g., lenses) and electronics (e.g., sensors) scene
transformations. Datasets that include synthetic data can be semi-virtual,
containing synthetic labels from real media of high quality such as the DND
denoising benchmark [28], the Flickr1024 stereo super-resolution dataset [29],
and the KITTI2012 multitask vision benchmark [30]. On the other hand, datasets
can be completely virtual from source to sensors, among those the MPI-Sintel
Dataset [31] is one of the milestones that use CG software to generate data.
It contains rich scenes and incredible labeling accuracy of the optical flow,
depth, and segmentation, which are either not achievable or expensive to
generate in real scenes. Its successors include FlyingChairs [32] and Scene
Flow Datasets [33]. As CG software keeps evolving, we see more fancy features
being developed and integrated into handy packages, like BlenderProc [34], to
create and render scenes that most resemble the real world. With the
assistance of better synthetic data, we can expect networks of better
performance to be easily deployed.

##  3 Our Method

The goal of PAT is to fuse data from cameras in an array. The fusion result
reflects the viewpoint of one selected camera. The selected viewpoint is the
alpha viewpoint $\alpha$ while others are alternative beta viewpoints
$\beta_{1}\sim\beta_{m}$, where $m$ is the number of alternative viewpoints.
To represent images, features, or parameters from a certain viewpoint, the
viewpoint symbol is marked as the left superscript.

The architecture of PAT is illustrated in Figure 1. The workflow of PAT is 1)
each sensor image goes through Figure 1(a) to generate its corresponding
feature; 2) features are fed into Figure 1(b) to generate the final output.

Figure 1: The architecture of PAT. "$a\times a$ conv" (blue blocks) are
convolutional layers with kernels of $a\times a$ size. The stride of
convolutional layers is 1 and the padding is $(a-1)/2$. The dilation of
convolutional layers $d$ is 1 unless specified. The depth dimension of
convolutional layers and residual blocks (orange and green blocks) are notated
after commas. The parameter each gray block may take is inside the
parenthesis. (a) Image representation module. It takes sensor image $I$ and
produces the associate feature $F$. $\oplus$ denotes addition. (b) The
attention engine and post-fusion module. ${}^{\alpha}F$ is the feature of
${}^{\alpha}I$ from alpha viewpoint and ${}^{\beta}F$s are features of
${}^{\beta}I$s from alternative viewpoints. $s$ is the upscaling factor. The
output is the fusion result ${}^{\alpha}\tilde{I}$. (c) The architecture of
"$3\times 3$ resblock". (d) The architecture of "$3\times 3$ resASPPblock".

###  3.1 Attention Engine

The attention engine densely aligns the features with regard to the input
physical receptive fields. The attention engine starts from image
representations (features) of sensor images. As images may differ in
resolution, color spaces, focal planes, etc., proper representations are
learned to facilitate correspondence matching. We adopt the Residual ASPP
Module [25] to fulfill this task, which demonstrates effectiveness to generate
multiscale features. For each camera, the feature produced by the image
representation module shares the spatial dimensions with its sensor image,
thus the position of each pixel is inherently encoded to the indices of
voxels. Note here we do not require the resolution of all the cameras in the
array to be the same, thus features may have diverse spatial dimensions but
share the third dimension $D$. The representation modules of different input
frames share the weights.

We apply dot-product attention to compare and transfer the alternative
features. From the feature
${}^{\alpha}F\in\mathbb{R}^{{}^{\alpha}H\times{}^{\alpha}W\times D}$ of the
alpha camera, we produce a query feature
${}^{\alpha}Q\in\mathbb{R}^{{}^{\alpha}H\times{}^{\alpha}W\times D}$ through a
residual block and a convolutional layer. Similarly, each alternative feature
${}^{\beta}F\in\mathbb{R}^{{}^{\beta}H\times{}^{\beta}W\times D}$ produces a
key feature ${}^{\beta}K\in\mathbb{R}^{{}^{\beta}H\times{}^{\beta}W\times D}$
and a value feature
${}^{\beta}V\in\mathbb{R}^{{}^{\beta}H\times{}^{\beta}W\times D}$. We perform
$\mathbf{C^{3}}$ operations: Collect, Correlate, and Combine to generate the
feature output. For simplicity, we use a system with two viewpoints to
illustrate $\mathbf{C^{3}}$ operations, as shown in Figure 3. Collect:
${}^{\alpha}q_{j}\in\mathbb{R}^{D}$ is $j$-th voxel in ${}^{\alpha}Q$, where a
"voxel" denotes a $D$-length vector along the feature dimension, as
illustrated in Figure 2. The range of $j$ is from 1 to
${}^{\alpha}H{}^{\alpha}W$. Voxels
$\\{{}^{\beta}k_{j_{1}},{}^{\beta}k_{j_{2}},{}^{\beta}k_{j_{3}},\dots,{}^{\beta}k_{j_{n}}\\},{}^{\beta}k\in\mathbb{R}^{D}$
in ${}^{\beta}K$ and voxels
$\\{{}^{\beta}v_{j_{1}},{}^{\beta}v_{j_{2}},{}^{\beta}v_{j_{3}},\dots,{}^{\beta}v_{j_{n}}\\},{}^{\beta}v\in\mathbb{R}^{D}$
are selected according to the receptive field of ${}^{\alpha}q_{j}$, where
$j_{n}$ depends on $j$, ranging from 1 to ${}^{\beta}H{}^{\beta}W$.

Figure 2: The diagram of a feature (blue cube) and one of its voxels (orange
stick).

Correlate: we calculate the score ${}^{\beta\rightarrow\alpha}s_{j}$ between
${}^{\alpha}q_{j}$ and ${}^{\beta}k$s to find the correspondence.
${}^{\beta\rightarrow\alpha}s_{j}$ is equal to the dot product of
$\left[{}^{\beta}k_{j_{1}}^{T};{}^{\beta}k_{j_{2}}^{T};{}^{\beta}k_{j_{3}}^{T};\dots;{}^{\beta}k_{j_{n}}^{T}\right]$
and ${}^{\alpha}q_{j}$. The $i$-th entry of ${}^{\beta\rightarrow\alpha}s_{j}$
is the score of $j_{i}$-th voxel.

Combine: we combine voxels
$\\{{}^{\beta}v_{j_{1}},{}^{\beta}v_{j_{2}},{}^{\beta}v_{j_{3}},\dots,{}^{\beta}v_{j_{n}}\\}$
with regard to ${}^{\beta\rightarrow\alpha}s_{j}$ by calculating the dot
product of
$\left[{}^{\beta}v_{j_{1}},{}^{\beta}v_{j_{2}},{}^{\beta}v_{j_{3}},\dots,{}^{\beta}v_{j_{n}}\right]$
and $\texttt{softmax}({}^{\beta\rightarrow\alpha}s_{j})$, where
softmax($\cdot$) is the softmax function. We denote the combined voxel as
${}^{\beta\rightarrow\alpha}v_{j}$. The concatenation of ${}^{\alpha}f_{j}$
and ${}^{\beta\rightarrow\alpha}v_{j}$ is the $j$-th voxel $u_{j}$ of the
output feature $U\in\mathbb{R}^{{}^{\alpha}H\times{}^{\alpha}W\times 2D}$.

For more than one alternative viewpoints, the $j$-th output vector $u_{j}$ is
equal to
$\texttt{concat}({}^{\alpha}f_{j},{}^{\beta_{1}\rightarrow\alpha}v_{j},{}^{\beta_{2}\rightarrow\alpha}v_{j},\dots,{}^{\beta_{m}\rightarrow\alpha}v_{j})$,
for concat($\cdot$) is the concatenate function. $\mathbf{C^{3}}$ operations
are fully vectorized, thus deployment of the attention engine on trending deep
learning platforms is of convenience.

Figure 3: Workflow of the attention engine and $\mathbf{C^{3}}$ operations in
a dual-camera array. We use color bars to represent vectors and matrices.
"$\cdot$" is the symbol of matrix multiplication.

The output feature
$U\in\mathbb{R}^{{}^{\alpha}H\times{}^{\alpha}W\times(m+1)D}$ goes through
several convolutional layers and residual blocks to produce the image output.

One may notice the attention engine reduces to PAM [25] when
$m=1,{}^{\alpha}H={}^{\beta}H,{}^{\alpha}W={}^{\beta}W,C=3,D=64$, and each
receptive field follows the epipolar geometry in a rectified stereo pair,
except that the intercorrelated validation mask is not incorporated into $U$.
In comparison, PAT can integrate other physical clues, like maximum disparity
or homography-based approximation, to optimize computation. Moreover, PAT can
process images of different characteristics from three or more cameras, where
rectification cannot be performed.

###  3.2 Complexity Analysis

It is essential to ensure the above operations is achievable and efficient
with regard to time complexity. For $j$ is the voxel index of the output
feature $U$, let us assume
$L=\max_{j}|\\{~{}^{\beta}k_{j_{1}},~{}^{\beta}k_{j_{2}},...,~{}^{\beta}k_{j_{n}}\\}|$,
where $|\cdot|$ returns the size of the set. The complexity of Collect is
$O(L)$, of Correlate is $O(D\times L)$, and of Combine is $O(D\times L)$.
Hence overall to compute the entire output feature for $m$ alternative
viewpoints we have the complexity $O(m\times H\times W\times D\times L)$.

For example, we assume we know the intrinsics and extrinsics in a dual-camera
array, where $m=1$ and the resolution of the cameras is $H\times W$. We can
specify the physical receptive field in the attention engine to be the indices
of beta feature voxels along the epipolar line of each alpha feature voxel,
assuming the feature representations of images also follow the epipolar
geometry in the spatial dimensions. If the baseline in a dual-camera system
intersects both image planes at infinity, the epipolar line is always
horizontal, i.e., $L=W$, bringing the total time complexity to $O(H\times
W\times D\times W)$, which agrees with the complexity of PAM in [25]. In
systems of which the layout does not meet the condition above, $L$ is still
linear to $H+W$, thus no extra time expense is introduced under the big $O$
notation. Furthermore, we can incorporate homography-based approximation,
where we roughly locate associate voxels of each alpha voxel via perspective
transformation. Knowing the depth range of interest in a scene, we can set the
maximum pixel displacement $l$ between the rough estimates and exact
correspondences to truncate the epipolar lines. Thus the complexity can be
further reduced to $O(H\times W\times D\times l)$, as typically $l\ll W$.

Here we can see another merit the attention engine carries in terms of time
complexity. We can chop the alpha feature into patches with spatial resolution
$H_{p}\times W_{p}$. The attention engine infers on those patches in parallel,
bringing the time complexity down to $O(m\times H_{p}\times W_{p}\times
D\times L)$.

###  3.3 Data synthesis

As we stated in the introduction, the pipeline of data simulation is fully
automatic. We use the Python API of Blender to scale up the generation of
scenes. Blender provides a variety of meshes, from which we select several
representative meshes including ’plane’, ’cube’, and ’uv-sphere’, and enrich
the database by perturbing the surface to create diverse ridges and valleys.
We can specify the dimensions, locations, and rotations of the meshes to
diversify their distribution in a scene. Upon the creation of each shape, we
can attach the ’material’ attribute to customize its interaction with the
light source. There are dozens of knobs to adjust the base color, diffusion,
or specularity; apart from those, we can apply vectorized textures, e.g.,
brick texture and checkerboard texture, to add varieties to color distribution
on the mesh. Occlusion and shadows are naturally introduced while stacking up
the meshes.

Blender provides camera objects to render the scene. Just as in real cameras,
parameters like the focal length, sensor size, pixel pitch, and resolution can
be easily set. If the ’Depth of field’ feature is on, parameters like the
focal plane and F-stop allow realistic modeling of the defocus blur. Blender
allows common picture formats as outputs, including lossy JPEG, lossless PNG,
or even RAW with full float accuracy. The color space of the output can be BW,
RGB, or RGBA. Figure 4 shows an example of rendered views of a dual-camera
system. We can see rich features, colors, and interactions of the objects in
the frames, and also parallax between two frames.

Figure 4: An example of the views of a dual camera system. The resolution of
the images is $2048\times 1536$.

We also implement other functions, among which we would like to emphasize the
function of animation generation. We can assign random trajectories and
transformations to mashes, and stream the data with regard to a given frame
rate. It can benefit array camera research on temporal connections.

###  3.4 Implementation Details

The following details are shared by PATs for the experiments. The unique
settings for each application are specified in the next section.

#### Dataset

We rendered 900 scenes of the resolution $2048\times 1536$ to two cameras
using the EEVEE render engine in Blender 2.92. 800 scenes were for the
training and 100 scenes for the validation. One of the virtual cameras was
selected to have the alpha viewpoint and the other had the beta viewpoint.The
objects in the scene distribute within a 20-meter range. Rendered frames were
in RGB color. We selected 49 patches of the resolution $384\times 128$ across
each scene, and then cubically downsampled the patches to $96\times 32$. Each
sample in the dataset has a pair of patches, where the patch from the alpha
view is regarded as the ground truth. The degraded inputs, instead, were
generated from the patch from the alpha view (alpha patch) and beta view (beta
patch) while training via the forward model with regard to the array setting.
We exported the extrinsics and intrinsics of two cameras and constructed the
receptive fields according to the epipolar geometry, i.e., a dense map from
each voxel index $j$ in the alpha view to the associate voxel indices
$j_{1}\sim j_{n}$ in the beta views. $n$ for all $j$ was set to 96 in our
dataset. The physical receptive fields for each sample were stored as arrays
along with two patches.

#### Training

PATs were trained on the NVIDIA Tesla V100S GPU. Hyperparameters below were
shared by the experimental systems:

$\mathbf{D}$ | 64 |  | Epoch | 80
---|---|---|---|---
$\mathbf{s}$ | 1 |  | Loss | Mean Square Error
$\mathbf{\tilde{C}}$ | 3 |  | Optimizer | Adam [35]
Learning Rate | 0.0002, decays by half per 30 epochs

The parameters that were specific to the application are clarified in the
following subsections. The model with the best peak signal-to-noise-ratio
(PSNR) performance on the validation set was selected for inference.

#### Inference

The intrinsics and extrinsics of the camera array were calibrated through
MATLAB Stereo Vision Toolbox. We combined epipolar geometry and homography-
based approximation to construct the physical receptive fields. The max
displacement $l$ is set to 80.

##  4 Experiments

Here we demonstrate four experimental systems with diverse sampling designs
and PAT processing for image fusion, following the order mentioned in the
introduction. We have posted the training and evaluation code for all of these
experiments online [36].

###  4.1 Wide Field - Narrow Field System

(a) Narrow field camera view

(b) Wide field camera view

(c) Wide field camera view (patch)

Figure 5: The views of the wide field - narrow field array. The pixel count of
(a) is around $10\times$ the pixel count of (c). Images are rescaled for
display purposes.

It is observed that chroma can be substantially compressed compared to
luminance before the decompression error is perceived by humans. Inspired by
that, we demonstrate a wide field color - narrow field monochrome system that
compressed the color of the narrow field of view (FoV) by up to $40\times$.
The configurations of the array were:

Narrow field camera
---
Body | Allied Vision Alvium 1800 U-1240m
Sensor | CMOS Monochrome
Lens | 25 mm TECHSPEC HR Series
Resolution | $4024\times 3036$
Wide field camera
Body | iDS UI-3590LE-C-HQ
Sensor | CMOS Color
Lens | 5 mm Kowa LM5JCM
Resolution | $4912\times 3684$

The focal plane of the wide field camera was set to its hyperfocal distance.
The narrow field camera focused on the black optical table around 7 meters
away.

Figure 5 shows the camera views in an example scene. Considering the color
filters on the wide field camera and $10\times$ resolution gap in the narrow
field, the red and blue raw signals were subsampled by $10\times 4=40\times$
and the green raw signals were subsampled by $10\times 2=20\times$ compared to
the luminance.

PAT acted as a color decompressor on this system that upsampled the colors in
the narrow field. PAT was trained with two inputs. We converted the alpha
patch to grayscale as one input and had the beta patch unchanged as the other
input. To model possible resolution gaps and defocus blur, we augmented the
training data by 1) adding box blur to the alpha input; 2) adding box blur to
the beta input; 3) $2\times$ bicubically downsampling the beta input; 4)
combining 2) and 3). These augmentation techniques were selected at random
with equivalent probabilities during training and validation. The batch size
of training was set to 32 and $C$ was set to 3. In the training and inference
phases, the alpha input was repeated along the feature dimension three times
and the beta input was bicubically upsampled to its original dimensions if it
had been downsampled.

Before implementing the trained PAT on the system, we evaluated the algorithm
on Flickr1024 [29] and KITTI2012 [30] (20 frames) test sets. For each testing
sample, we used whole frames instead of patches to generate inputs. The alpha
frame was converted to grayscale as the alpha input and the beta frame was
$2\times$ or $4\times$ bicubically downsampled as the beta input. Based on the
characteristics of the test sets, the physical receptive fields indicated
truncated horizontal epipolar lines of the length 120 divided by the
downsampling rate. In Table 1, we listed average PSNR and SSIM [37] scores
between 1) the ground truth alpha frames and the grayscale alpha inputs in the
"Alpha Input" column; 2) the beta frames and the beta inputs bicubically-
upsampled to the original size in the "Beta Input" column; 3) the ground truth
alpha frames and the fused results of PAT in the "Fusion" column. We can see
PAT improved the test system by maintaining the structures of the alpha input
and improving the color upsampling results compared to the beta input solely.

Table 1: Comparison between inputs and fused results of PAT (monochrome - color inputs) Dataset | Scale | Alpha Input | Beta Input | Fusion
---|---|---|---|---
Flickr2014 | x2 | 21.42/0.8800 | 24.95/0.8161 | 27.26/0.8992
x4 | 21.84/0.6265 | 25.85/0.8840
KITTI2012 | x2 | 26.40/0.9178 | 28.48/0.8845 | 29.55/0.9097
x4 | 24.56/0.7376 | 28.40/0.8957

We assigned the alpha viewpoint to the narrow field camera while inferencing.
The result is shown in Figure 6. In comparison, colors were upsampled by up to
$40\times$ to the narrow view without scarifying the sampling rate of the
luminance. Although the color bleeding artifacts caused by a large upsampling
rate can be observed in certain regions, we reduced the artifacts to the
minimum by providing accurate physical information to the system. As
illustrated in Figure 7, the correct receptive field yielded the result in
Figure 7(a) with correct colors (pink stickers in the orange window) and less
artifacts (storage box in the green window).

Figure 6: (a) is the fused frame on the wide field - narrow field array. (b)
and (c) are associate details of the fused frame and the view of the wide
field camera, respectively.

Figure 7: The fused frame with different receptive fields. (a) is generated
with the calibrated receptive fields that accurately reflect the physics of
the system. (b) is generated with the receptive fields assuming the inputs are
rectified.

###  4.2 Visible - Near Infrared Systems

As a result of reduced atmospheric scatter and absorption, near infrared
cameras achieve higher contrast in landscape photography. However, IR signals
are typically recorded as monochromatic data, thus are not visual friendly.
Here we show PAT acted as a visualization tool to fuse color and NIR views
while retaining the texture of remote objects on visible - NIR camera arrays.
We used the data from two visible-NIR arrays, one was from a public database
PittsStereo-RGBNIR [38] and the other was built by us111Thanks to Emily Chau
in Infrared Imaging Group at the University of Arizona for the array setup and
data collection.. The configurations of the camera array from the online
dataset are available in [38]. We used rectified images of the resolution
$582\times 429$ from the database. Our visible-NIR system was composed of two
35 mm EO-4010 cameras, one with a color filter and the other with a NIR
filter. The resolution of both cameras was $2048\times 2048$.

We applied the pretrained PAT from the wide field - narrow field system to
this fusion task to highlight the ability of domain adaptation of our
algorithm. The attention engine of PAT operates on the features, thus is
robust to the data that differs in appearance, brightness, etc.

The alpha viewpoint was assigned to the NIR camera while inferencing. Figure 8
and 9 demonstrate the fusion results with zoomed-in details on the given data.
The color was well transferred to the fusion results in the presence of
complicated occlusion and parallax. Also different appearances of distant
objects in the visible and NIR frames were fused nicely, as demonstrated in
the green boxes.

(a) NIR frame (b) Visible frame (c) The fused frame

Figure 8: Data from PittsStereo-RGBNIR dataset [38] and the fused result. The
orange, blue, and green windows contain the details in the scene from near to
far. The brightness of details is adjusted to enhance contrast.

(a) NIR frame

(b) Visible frame

(c) The fused frame

Figure 9: Data from our visible-NIR camera array and the fused result. The
orange, blue and green windows contain the details in the scene from near to
far. The brightness of details is adjusted to enhance contrast.

###  4.3 Short Exposure - Long Exposure System

For visible color imaging, multiaperture sampling allows independent exposure
and focus control for each band. We demonstrate this capability using a
$2\times 2$ camera array based on the Arducam 1MP$\times 4$ Quadrascopic
OV9281. The cameras were monochromatic and had $1280\times 800$ resolution.
One camera with a 12 mm lens had no filter, while the others with 8 mm lenses
were equipped with three filters. The central wavelengths of the filters were
450 nm, 550 nm, and 600 nm respectively. The filters shared 80 nm full width
at half maximum. The exposure time of each camera was controlled independently
to optimize the dynamic range of the signal. Figure 10 shows the views of four
cameras. Compared to uni-exposure systems like cameras with the Bayer filter,
our system allowed up to $5\times$ differences in exposure, thus having higher
overall throughput of spectral data.

PAT was trained with 4 inputs. The alpha patch was converted to be grayscale
as the alpha input. The red, green, and blue channels were unpacked from the
beta patch as three alternative inputs. Note the color filters of our
synthetic training data did not exactly resemble the filters we used with
regard to the spectral curves. The batch size of training was set to 16 and
$C$ was set to 1 as inputs were monochromatic.

Most of the test settings agreed with those in the wide field - narrow field
system, except that the beta frame was unpacked into three frames of a single
color channel and downsampled to generate three beta inputs. In Table 2, PSNR
and SSIM scores in the "Beta Inputs" column were first averaged between three
spectral bands of the beta frame and corresponding beta inputs, and then
averaged across all beta frames. The meanings of other columns are the same as
in Table 1. Similarly we can also see PAT improved overall system performance.

Table 2: Comparison between inputs and fused results of PAT (monochrome - spectral inputs) Dataset | Scale | Alpha Input | Beta Inputs | PAT
---|---|---|---|---
Flickr2014 | x2 | 21.42/0.8800 | 24.95/0.8161 | 25.55/0.8714
x4 | 21.84/0.6265 | 24.00/0.8493
KITTI2012 | x2 | 26.40/0.9178 | 28.48/0.8845 | 28.77/0.8955
x4 | 24.56/0.7376 | 28.13/0.8903

While inferencing, the alpha viewpoint was assigned to the camera without the
filter. Figure 11 shows the fused result. The result preserved the geometry of
the alpha camera view and displayed the correct color, which indicates the
algorithm effectively adapted to data with different filter functions. We can
expect the result generated with the optimized spectral throughput to have a
higher dynamic range. Note that PAT is physical-based rather than perception-
based, therefore the network does not "guess" the color beyond physical clues.
As shown in Figure 12, the color channels of the fused frame were permuted
with regard to the way that the inputs were permuted.

(a) unfiltered camera view, 150 units exposure time

(b) 450 nm camera view, 800 units exposure time

(c) 550 nm camera view, 300 units exposure time

(d) 600 nm camera view, 500 units exposure time

Figure 10: The views of the short exposure - long exposure array under diverse
exposures in relative scales. 1 unit is equal to approximately 12 micro
seconds.

Figure 11: (a) is the fused frame on the short exposure - long exposure array.
Zoomed-in views on the side highlight objects of different spectral responses.
(b) are the associate details of the original frame from the short exposure
camera. Details in (c) are captured by a cellphone camera to provide the
readers with color references. The color balance of the fused frame and the
brightness of details are adjusted for visual purposes. Figure 12: The fused
results of the permutated input sequence, where the 450 nm view and 650 nm
view were switched. The color balance is adjusted for visual purposes.

###  4.4 High Frame Rate - Low Frame Rate System

Sensors with color filters sacrifice quantum efficiency compared to monochrome
sensors, thus requiring a longer exposure time to achieve a comparable signal-
to-noise ratio (SNR). This prevents standalone spectral cameras from achieving
a higher frame rate. Here we demonstrate an imaging system that combines one
high frame rate (HFR) monochrome camera with three low frame rate (LFR)
spectral cameras is a better solution to sample the light field temporally.
This system enables PAT to reconstruct the light field at a high frame rate.

We applied one Basler acA1440-220um camera with a 12 mm len as the HFR camera,
which can reach 227 frames per second (fps) at the $1456\times 1088$
resolution. Three Arducam cameras with 8 mm lenses and spectral filters in the
short exposure - long exposure system were applied as the LFR cameras. The LFR
cameras are synchronized, operating at 30 fps. Figure 13 shows the views of
four cameras in a scene where the moderate motion of the pillow occurred. We
can see the LFR frames deteriorated in the region that has motions, while the
associate region in the HFR frame remained sharp.

(a) HFR camera view

(b) 450 nm LFR camera view

(c) 550 nm LFR camera view

(d) 600 nm LFR camera view

Figure 13: The views of the HFR - LFR array. The exposure time of (a) was 4.3
ms. The exposure times of (b) - (d) were the same, around 12 ms. Since the HFR
camera was not synchronized with LFR cameras, (a) was captured $\pm 2.15$ ms
away from the moment that (b) - (d) were captured. The orange windows
highlight the moving pillow. The brightness and contrast of the patches in the
orange windows were adjusted for visual purposes.

The alpha and beta viewpoints were assigned to the HFR camera and LFR cameras,
respectively. For one LFR frame captured at a certain moment, the HFR frames
captured $\pm 15$ ms from that moment correspond to that LFR frame. We assumed
the epipolar constraint was valid in general between the LFR frame and
associate HFR frames and built physical receptive fields accordingly. We
applied the pretrained PAT from the short exposure-long exposure system while
inferencing. Figure 14 shows the fused result, which fused the HFR camera view
with colors from three spectral cameras. Because the epipolar geometry does
not strictly hold for unsynchronized frames, slight color jittering of letters
in the pillow was observed. However, the majority of colors of the pillow were
effectively fused and the motion boundary was well preserved.

Figure 14: The fused results on the HFR - LFR array. The color balance is
adjusted for display purposes.

Given two sets of LFR frames with three filters (6 frames in total) captured
at 0 and 26 ms, PAT fused seven HFR frames with color in between. Figure 15
shows the patches of the moving pillow in the fused results. The pillow in the
fused patches was in color with sharp motion boundaries, compared to LFR
patches.

(a) $0$ ms

(b) 0.1 ms

(c) 4.4 ms

(d) 8.7 ms

(e) 13.0 ms

(f) $26$ ms

(g) 17.3 ms

(h) 21.6 ms

(i) 25.9 ms

Figure 15: The patches of the moving pillow. (a) and (f) are two consecutive
frames from a 600 nm LFR camera. The left images in (b) - (e) and (g) - (i)
are the patches from the HFR camera while the right images are from the fused
results. The labels are the estimated time elapsed from the moment that (a)
was captured. Three LFR frames captured at 0 ms were used to generate the
results in (b) - (e), while three LFR frames captured at 26 ms were used to
generate the results in (g) - (h). The color balance is adjusted for visual
purposes.

##  5 Conclusion

In this paper, we proposed a physics-aware transformer for image fusion on
array cameras. This network architecture incorporates tailored receptive
fields to reflect the physics of the imaging system. The proposed pipeline of
data synthesis effectively provides training data for transformers and has the
potential to benefit other learning algorithms. We demonstrated the
versatility of PAT on four different camera arrays. We envision PAT being a
standard processing tool for array cameras of the next generation, and
inspiring designs, combinations and applications of array cameras for better
light field sampling.

## References

  * [1] Joseph N Mait, Gary W Euliss, and Ravindra A Athale.  Computational imaging.  Advances in Optics and Photonics, 10(2):409–483, 2018.
  * [2] Ivo Ihrke, John Restrepo, and Lois Mignard-Debise.  Principles of light field imaging: Briefly revisiting 25 years of research.  IEEE Signal Processing Magazine, 33(5):59–69, 2016.
  * [3] Rastislav Lukac and Konstantinos N Plataniotis.  Color filter arrays: Design and performance analysis.  IEEE Transactions on Consumer electronics, 51(4):1260–1267, 2005\.
  * [4] Xiao Xiao, Bahram Javidi, Manuel Martinez-Corral, and Adrian Stern.  Advances in three-dimensional integral imaging: sensing, display, and applications.  Applied optics, 52(4):546–560, 2013.
  * [5] Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz, and Pat Hanrahan.  Light field photography with a hand-held plenoptic camera.  PhD thesis, Stanford University, 2005.
  * [6] Xin Yuan, David J Brady, and Aggelos K Katsaggelos.  Snapshot compressive imaging: Theory, algorithms, and applications.  IEEE Signal Processing Magazine, 38(2):65–88, 2021.
  * [7] Xuemei Hu, Xing Lin, Tao Yue, and Qionghai Dai.  Multispectral video acquisition using spectral sweep camera.  Optics Express, 27(19):27088–27102, 2019.
  * [8] Chengyu Wang, Qian Huang, Ming Cheng, Zhan Ma, and David J Brady.  Deep learning for camera autofocus.  IEEE Transactions on Computational Imaging, 7:258–271, 2021.
  * [9] David J Brady, Wubin Pang, Han Li, Zhan Ma, Yue Tao, and Xun Cao.  Parallel cameras.  Optica, 5(2):127–137, 2018.
  * [10] Jun Tanida.  Multi-aperture optics as a universal platform for computational imaging.  Optical Review, 23(5):859–864, 2016.
  * [11] Robert Plemmons, Sudhakar Prasad, Scott Mathews, Mark Mirotznik, Ryan Barnard, Brian Gray, Paul Pauca, Todd Torgersen, Joe Van Der Gracht, and Greg Behrmann.  Periodic: integrated computational array imaging technology.  Computational Optical Sensing and Imaging, page CMA1, 2007.
  * [12] Premchandra M. Shankar, William C. Hasenplaugh, Rick L. Morrison, Ronald A. Stack, and Mark A. Neifeld.  Multiaperture imaging.  Appl. Opt., 45(13):2871–2883, 2006.
  * [13] David J Brady, Lu Fang, and Zhan Ma.  Deep learning for camera data acquisition, control, and image estimation.  Advances in Optics and Photonics, 12(4):787–846, 2020.
  * [14] Xiaoyun Yuan, Mengqi Ji, Jiamin Wu, David J Brady, Qionghai Dai, and Lu Fang.  A modular hierarchical array camera.  Light: Science & Applications, 10(1):1–9, 2021.
  * [15] Luo Juan and Gwun Oubong.  Surf applied in panorama image stitching.  2010 2nd international conference on image processing theory, tools and applications, pages 495–499, 2010.
  * [16] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [18] http://www.blender.org.
  * [19] Yaniv Romano, Michael Elad, and Peyman Milanfar.  The little engine that could: Regularization by denoising (red).  SIAM Journal on Imaging Sciences, 10(4):1804–1844, 2017.
  * [20] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Deep image prior.  Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9446–9454, 2018.
  * [21] Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand.  Deep joint demosaicking and denoising.  ACM Transactions on Graphics (ToG), 35(6):1–12, 2016.
  * [22] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  Advances in neural information processing systems, 28, 2015.
  * [23] Zhengyou Zhang.  A flexible new technique for camera calibration.  IEEE Transactions on pattern analysis and machine intelligence, 22(11):1330–1334, 2000.
  * [24] Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu.  Epipolar transformers.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7779–7788, 2020.
  * [25] Longguang Wang, Yingqian Wang, Zhengfa Liang, Zaiping Lin, Jungang Yang, Wei An, and Yulan Guo.  Learning parallax attention for stereo image super-resolution.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12250–12259, 2019.
  * [26] Canqiang Chen, Chunmei Qing, Xiangmin Xu, and Patrick Dickinson.  Cross parallax attention network for stereo image super-resolution.  IEEE Transactions on Multimedia, 2021.
  * [27] Bo Yan, Chenxi Ma, Bahetiyaer Bare, Weimin Tan, and Steven CH Hoi.  Disparity-aware domain adaptation in stereo image restoration.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13179–13187, 2020.
  * [28] Tobias Plotz and Stefan Roth.  Benchmarking denoising algorithms with real photographs.  Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1586–1595, 2017.
  * [29] Yingqian Wang, Longguang Wang, Jungang Yang, Wei An, and Yulan Guo.  Flickr1024: A large-scale dataset for stereo image super-resolution.  International Conference on Computer Vision Workshops, pages 3852–3857, 2019.
  * [30] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  Conference on Computer Vision and Pattern Recognition (CVPR), 2012\.
  * [31] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black.  A naturalistic open source movie for optical flow evaluation.  European Conf. on Computer Vision (ECCV), pages 611–625, 2012.
  * [32] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  Proceedings of the IEEE international conference on computer vision, pages 2758–2766, 2015.
  * [33] Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox.  A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation.  Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4040–4048, 2016.
  * [34] Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Dmitry Olefir, Tomas Hodan, Youssef Zidan, Mohamad Elbadrawy, Markus Knauer, Harinandan Katam, and Ahsan Lodhi.  Blenderproc: Reducing the reality gap with photorealistic rendering.  International Conference on Robotics: Sciene and Systems, RSS 2020, 2020.
  * [35] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [36] https://github.com/djbradyAtOpticalSciencesArizona/physicsAwareTransformer.
  * [37] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [38] Tiancheng Zhi, Bernardo R Pires, Martial Hebert, and Srinivasa G Narasimhan.  Deep material-aware cross-spectral stereo matching.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1916–1925, 2018.