# DeTurb: Atmospheric Turbulence Mitigation with Deformable 3D Convolutions
and 3D Swin Transformers

Zhicheng Zou and Nantheera Anantrasirichai
Visual Information Laboratory
University of Bristol
This work was supported by the UKRI MyWorld Strength in Places Programme
(SIPF00006/1).

###### Abstract

Atmospheric turbulence in long-range imaging significantly degrades the
quality and fidelity of captured scenes due to random variations in both
spatial and temporal dimensions. These distortions present a formidable
challenge across various applications, from surveillance to astronomy,
necessitating robust mitigation strategies. While model-based approaches
achieve good results, they are very slow. Deep learning approaches show
promise in image and video restoration but have struggled to address these
spatiotemporal variant distortions effectively. This paper proposes a new
framework that combines geometric restoration with an enhancement module.
Random perturbations and geometric distortion are removed using a pyramid
architecture with deformable 3D convolutions, resulting in aligned frames.
These frames are then used to reconstruct a sharp, clear image via a multi-
scale architecture of 3D Swin Transformers. The proposed framework
demonstrates superior performance over the state of the art for both synthetic
and real atmospheric turbulence effects, with reasonable speed and model size.

##  1 Introduction

Light propagation through the various layers of the atmosphere, which differ
in temperature, pressure, humidity, and wind speed, introduces diffraction-
related blurring and random refractions. These variations cause fluctuation in
intensity and random phase distortions in the wavefront of light waves,
significantly degrading the performance of imaging systems. Most image and
video enhancement and restoration techniques have been proposed, addressing
specific problems like denoising and deblurring. However, these methods may
not be directly used to solve the problem of atmospheric turbulence, as it
involves multiple types of distortion, making it challenging to model its
degradation accurately. Successful approaches will be invaluable in many
applications, including air-to-ground imaging, long-range terrestrial video
surveillance, creative industries such as natural history filmmaking, and
other computer vision applications, including object recognition and tracking.

Traditional methods for turbulence video restoration typically involve: (i)
removing pixel offset caused by tilt, often using the optical flow method;
(ii) employing lucky image fusion to select and combine the clearest pixel
blocks within a specific interval; and/or (iii) applying blind deconvolution
algorithms to remove residual fuzziness (Cheng et al. (2023); Zhang et al.
(2011); Çaliskan and Arica (2014); Zhu and Milanfar (2013); Chen et al.
(2020)). However, these methods have notable limitations. They rely heavily on
a large quantity of measured data for fusion, use general and non-optimized
point spread function (PSF) priors in blind deconvolution, and struggle to
effectively address the complex statistical behavior of atmospheric turbulence
(Cheng et al. (2023); Hunt et al. (2018); Fried (1978)). Additionally, they
are prone to artifacts from inaccurate flow estimates and are very slow.

In recent years, deep learning approaches have emerged to tackle the challenge
of atmospheric turbulence mitigation. These methods include deep-stacked
autoencoder neural network models and convolutional UNet-like architectures
(Gao et al. (2019); Cheng et al. (2023); Anantrasirichai (2023)). Although
these models have shown promising results in simulations, they often rely on
simplified assumptions about atmospheric turbulence and lack sufficient real-
world datasets, limiting their generalization to diverse scene
reconstructions. The effectiveness of these models on actual measured data
remains an area of active research.

In this work, we propose a novel framework called DeTurb that integrates two
modules: (i) a non-rigid registration module to reduce wavy effects and
temporal distortion caused by atmospheric turbulence, and (ii) a feature
fusion module to select and fuse useful features for enhanced visualization.
Several existing video restoration methods include alignment and feature
fusion modules (Wang et al. (2019); Jiang et al. (2023); Lin et al. (2024a,
b)), which we found to be more effective when separated rather than combined
into a single process. The two-step approach has also proven effective in
dealing with atmospheric turbulence issues. Model-based methods, such as
space-invariant deconvolution (SID) (Zhu and Milanfar (2013)) and complex
wavelet-based fusion (CLEAR) (Anantrasirichai et al. (2013)), employ non-rigid
image registration to reduce random perturbations. SID then applies deblurring
to the near-diffraction-limited image, while CLEAR produces a sharp image
through wavelet-based image fusion. With the advent of deep learning, similar
strategies have continued (sometimes referred to as tilt-blur models), such as
the CNN-based method AT-Net (Yasarla and Patel (2021)) and the transformer-
based method TMT (Zhang et al. (2024)).

In our DeTurb framework, both modules employ UNet-like architectures. The
first module learns different levels of turbulence distortion similar to TMT,
while the second module extracts features from different scales,
reconstructing local details related to semantic meanings. To deal with
displacement among frames due to turbulence and moving objects, deformable
convolutions have been applied in (Hu and Anantrasirichai (2023)). However,
our architecture is distinct in that it processes data in a 3D manner,
handling features in both spatial and temporal dimensions simultaneously.
Specifically, non-rigid registration is achieved using deformable 3D
convolutions (Ying et al. (2020)). After the local spatial alignment among
frames, the aligned features from each layer of the pyramid are then enhanced
using 3D Swin Transformers (Yang et al. (2023)).

Our main contributions can be summarised as follows:

  * •

We propose a novel framework, DeTurb, for restoring long-range videos affected
by spatiotemporal distortions due to atmospheric turbulence. With comparable
inference speed, DeTurb significantly outperforms the state of the art in
terms of video quality.

  * •

DeTurb mitigates geometric distortion using a non-rigid registration module,
and then enhances edges and texture details with a feature fusion module.

  * •

The non-rigid registration module estimates the flow of random perturbations
and moving objects via a UNet-like architecture, in which each scale performs
deformable 3D convolutions.

  * •

The feature fusion module combines features of registered frames with 3D Swin
transformers arranged in a UNet-like architecture. This aims to enhance both
local and global details for better visualization.

##  2 Existing methods

Atmospheric turbulence, usually resulting from temporal variations occurring
near the ground, is typically anisoplanatic for large field-of-view objects,
exhibiting spatial variations that complicate the correction process. Given
these complexities, learning-based techniques, specifically those involving
deep learning, have become increasingly effective in mitigating these effects.
Unlike traditional methods (e.g., (Anantrasirichai et al. (2013); Zhu and
Milanfar (2013); Kelmelis et al. (2017); Boehrer et al. (2021)), these
techniques rely on the processing power of neural networks to predict and
correct distortions, leading to more adaptable and robust solutions.

Convolutional Neural Networks (CNNs) have been at the forefront of this
effort, as they are well-suited for image processing tasks due to their
ability to extract hierarchical features from images. CNNs can effectively
learn to recognize and mitigate the effects of turbulence, thereby
significantly enhancing image clarity (Gao et al. (2019); Li et al. (2018);
Yasarla and Patel (2022)). Another innovative approach is Generative
Adversarial Networks (GANs), which employ a dual-network architecture
comprising a generator and a discriminator that work in tandem to produce
highly refined outputs from severely degraded inputs. GANs are capable of
generating clear, high-resolution images from those distorted by atmospheric
conditions. For example, ATVR-GAN (Ettedgui and Yitzhaky (2023)) integrates a
Recurrent Neural Network (RNN) into the GAN’s generator, while in (Leihong et
al. (2021)), phase disturbance reduction is performed in the Fourier domain,
which is then used as a condition for the GAN. LTT-GAN (Mei and Patel (2023))
applies style transfer via GAN to restore faces degraded by atmospheric
turbulence.

More recently, transformer-based methods have shown promising results in
handling atmospheric turbulence by employing different attention mechanisms to
process image patches. This enables precise correction of atmospheric
distortions across various scales within an image. These methods have
discovered long-term dependencies in data and demonstrated great potential in
this field (Mao et al. (2022); Zhang et al. (2023a, 2024, b)). TurbNet (Mao et
al. (2022)) extracts features through a transformer UNet-like architecture and
utilizes physics-inspired downstream methods to reconstruct clean images. The
Swin transformer is employed for estimating turbulent flow in (Zhang et al.
(2023a)). TMT (Zhang et al. (2024)) utilizes vision transformers to remove
blur, while ASF-Transformer (Zhang et al. (2023b)) integrates spatial-aware
and frequency-aware transformer blocks into a UNet framework.

Diffusion models (DMs) are also of interest in these applications. Two recent
methods, (Nair et al. (2023)) and (Suin et al. (2024)), focus on restoring
faces degraded by atmospheric turbulence using the Denoising Diffusion
Probabilistic Model (DDPM), initially proposed by Ho et al. (2020). In
(Jaiswal et al. (2023)), a physics-based simulator is directly integrated into
the training process of a restoration model. Furthermore, Wang et al. (2023)
proposes a conditional diffusion model under a variational inference framework
for more generic images. DMs serve as generative priors for blind restoration
in (Chung et al. (2023)), which exploits a degradation model involving tilt
and blur, akin to the concept in TMT (Zhang et al. (2024)) that utilizes two
concatenated modules for tilt removal and deblurring.

##  3 Proposed method

The proposed framework is shown in Fig. 1, where the non-rigid registration
module (described in Section 3.1) is concatenated with the feature fusion
module (described in Section 3.2).

Figure 1: Diagram of the proposed DeTurb. Top-row: end-to-end framework
comprising the non-rigid registration and the feature fusion modules. Bottom-
row: block diagrams of 3D Swin transformer encoder and decoder blocks.

###  3.1 Non-rigid registration module

As mentioned earlier, several methods address multiple frame inputs of the
video using frame or feature alignment processes. Among these, we were
inspired by (Zhang et al. (2024)), where a UNet architecture with depth-wise
3D convolutions is used to estimate flow across multiple frames. We improve
this process with deformable 3D convolutions. A key contribution is that, in
atmospheric turbulent environments, objects exhibit visual distortions within
small ranges of pixel displacement, appearing randomly in all directions. The
use of deformable 3D convolutions provides flexibility in capturing the shapes
of the distorted objects across different scales of the UNet in space and
time, enabling the extraction of appropriate features from the distorted
scenes. After obtaining multi-scale flows each frame in the input group is
warped to the current frame from coarse to fine displacements, as shown in
Fig. 1.

#### Deformable 3D convolutions.

The transition from the deformable 2D convolutions (Dai et al. (2017)) to 3D
ones involves extending the adaptability of convolution operations to the
third dimension. Deformable 3D convolutions are defined as Eq. 1,

$y(p_{0})=\sum_{p_{n}\in\mathcal{G}}w(p_{n})\cdot x(p_{0}+p_{n}+\Delta
p_{n}),$ (1)

where $p_{0}$ denotes a location in the output feature map $y$, $p_{n}$
represents the $n$-th location in the convolution sampling grid
$\mathcal{G}\in\\{(-1,-1,-1),(-1,-1,0),...,(1,1,\\\ 0),(1,1,1)\\}$ for a grid
size of 3$\times$3$\times$3, $w$ is the convolution weight, and $x$ is the
input feature map. $\Delta p_{n}$ is the learnable offset for the $n$-th
location, introducing adaptability to the convolution operation (Ying et al.
(2020)). Consequently, the convolutional kernel’s receptive field can adjust
in response to alterations in the input feature map’s shape, thereby
accommodating changes in the dimensions and scales of the identified patterns.

#### Architecture settings.

The basic architecture is a 3D-UNet with a depth of 4. The number of depths of
the non-rigid registration relies on the local displacement among frames due
to atmospheric turbulence. The deformable 3D convolutions (Def3DConv) are used
in depth 2${}^{\text{nd}}$-4${}^{\text{th}}$ as listed in Table 1 (left). Each
convolution is integrated with ReLU activation functions. Deformable
convolutions still rely on a predefined kernel size, but it adds the ability
to dynamically adjust the sampling locations within that predefined kernel
grid during the convolution process. This adjustment allows them to adapt
better to local variations in the input data. Following (Zhang et al. (2024))
to further enhance the perceptual field and improve the detection of features
across varying scales, the kernel sizes are strategically set in the encoder.
The first level uses a larger kernel size of 7, maintained for the second
level to capture broader features and anomalies caused by turbulence before
reducing to size 5 and 3 in the third and fourth levels, respectively. This
arrangement allows the network to capture and process various distortion
patterns, from broad to more localised disturbances. For decoder, we use 3D
transposed convolutions (3DTranspConv) and the kernel size of 3 for all
levels.

Table 1: Configuration of proposed DeTurb network with the input size of
$H\times W\times N$. 3DConv is a 3D convolution block, Def3DConv is a
deformable 3D convolution block, and 3DTranspConv is a transposed convolution
block. 3DSwinBlock_Enc is a 3D Swin Transformer Encoder block and
3DSwinBlock_Dec is a 3D Swin Transformer Decoder block.

Non-rigid registration module | Feature fusion module
---|---
Layer (kernel size) | Out dimension | Layer (kernel size) | Out dimension
3DConv+MaxPool (7$\times$7) |  $\frac{H}{2}\times\frac{W}{2}\times$64 | 2$\times$3DConv (4$\times$4) |  $\frac{H}{2}\times\frac{W}{2}\times$32
Def3DConv+MaxPool (7$\times$7) | $\frac{H}{4}\times\frac{W}{4}\times 256$ | 3DSwinBlock_Enc |  $\frac{H}{4}\times\frac{W}{4}\times$64
Def3DConv+MaxPool (5$\times$5) | $\frac{H}{8}\times\frac{W}{8}\times 256$ | 3DSwinBlock_Enc | $\frac{H}{8}\times\frac{W}{8}\times 128$
Def3DConv (3$\times$3) | $\frac{H}{8}\times\frac{W}{8}\times 512$ | 3DSwinBlock_Enc | $\frac{H}{16}\times\frac{W}{16}\times 256$
3DTranspConv+3DConv (3$\times$3) | $\frac{H}{4}\times\frac{W}{4}\times(256\rightarrow 2N)$ | 3DSwinBlock_Enc | $\frac{H}{32}\times\frac{W}{32}\times 512$
3DTranspConv+3DConv (3$\times$3) | $\frac{H}{2}\times\frac{W}{2}\times(128\rightarrow 2N)$ | 3DSwinBlock_Dec | $\frac{H}{16}\times\frac{W}{16}\times 256$
3DTranspConv+3DConv (3$\times$3) | $H\times W\times(64\rightarrow 2N)$ | 3DSwinBlock_Dec | $\frac{H}{8}\times\frac{W}{8}\times 128$
2DWarpk, $k\in\\{0,1,2\\}$ | $\frac{H}{2^{k}}\times\frac{W}{2^{k}}\times 3N$ | 3DSwinBlock_Dec | $\frac{H}{4}\times\frac{W}{4}\times 64$
3DConv (7$\times$7) | $H\times W\times 3N$ | 3DTranspConv | $H\times W\times 32$
|  | 3DConv (1$\times$1) | $H\times W\times 3$

###  3.2 Feature fusion module

Similar to many methods proposed for video processing (Anantrasirichai (2023);
Shang et al. (2023); Lin et al. (2024b)), we process data through multiscale
feature learning using a UNet-like architecture, as shown in Fig. 1 and
parameters are listed in Table 1 (right). The process begins with input
initial feature extraction through two blocks of 3D convolutions, which
prepare the data by highlighting essential features for subsequent layers. The
3D Swin Transformer (Yang et al. (2023)) is used as its efficiency in
modelling complex dependencies. It utilises shifted window mechanisms to
handle the input data’s non-uniformity. The processed data is then channelled
through a UNet-like architecture with a depth of 4. This approach ensures that
the enhancement of local areas is related to the semantic information of those
areas and their surroundings. This multi-scale processing is crucial for
restoring fine textures and edges in distorted frames, enhancing the model’s
capability to effectively address a range of distortion scales introduced by
atmospheric conditions. The last 3D convolution layer converts features to RGB
output.

#### 3D Swin Transformer.

The 3D Swin transformers (Tang et al. (2022); Yang et al. (2023); Cai et al.
(2023)) extend the Swin transformer architecture to three dimensions, adapting
it to better understand volumetric or sequential data by incorporating the
temporal dimension. This offers significant advancements over traditional
models in handling the complexities of video reconstruction. The model
introduces a hierarchical structure to process data at multiple scales,
capturing detailed spatial-temporal features. Our 3D Swin transformer uses a
shifted window-based self-attention mechanism across three dimensions as used
in (Yang et al. (2023); Cai et al. (2023)), effectively capturing dynamic
changes over time.

As shown in Fig. 1 bottom-row, the 3D Swin Transformer Encoder block uses 3D
convolutions to merge and downsample features across multiple channels. These
features are then split to undergo processing by one 3D ResNet block and one
3D Swin transformer block, with the final output being the sum of these two
blocks’ outputs. In each 3D Swin Transformer Decoder block, a 3D transposed
convolution operator is used to combine and upsample feature maps.

Another benefit of the 3D Swin Transformer blocks is that in both the encoder
and decoder, cyclic shifts are applied to the input of multi-head self-
attention to improve interaction between adjacent and non-adjacent tokens in
three directions. It works by shifting the input tokens cyclically by a
specified number of units $s$ in each dimension, where $s$ is typically set to
half the window size. This shifting rearranges the tokens so that those from
adjacent windows in the original configuration may end up in the same window
post-shift. This process enables the model to compute similarities and
interactions between tokens that were initially in neighboring windows, which
helps overcome the limitation of the original window-based self-attention that
only computes interactions within the same window. The cyclic shifting thus
facilitates the capture of broader contextual information across adjacent
windows, enhancing the network’s ability to learn long-distance dependency
information. However, it also leads to an increase in the number of windows
and variability in window sizes, which is managed through a window-masking
mechanism to ensure that only relevant token similarities are considered. More
details about the 3D Swin transformer block can be found in (Cai et al.
(2023)).

###  3.3 Loss Functions

Two loss functions are employed similar to (Zhang et al. (2024)): Charbonnier
Loss and Edge Loss. In the distortion mitigating module, atmospheric
turbulence effects can create outliers in the pixel-wise loss. To address
this, Charbonnier loss is used, as it combines the benefits of both $\ell_{1}$
and $\ell_{2}$ losses, effectively handling outliers better (Charbonnier et
al. (1997)). Defined by Eq. 2,

$L_{\text{Char}}(x,y)=\sqrt{(x-y)^{2}+\epsilon^{2}},$ (2)

where $x$ and $y$ represent the predicted and true values, and $\epsilon$ is a
small constant (e.g., $1e-3$) to ensure numerical stability, this loss
function effectively balances the error distribution. It provides a smooth
gradient even when small errors are present, which is particularly beneficial
for handling the subtle but critical differences in turbulence-affected
images, where precision in error correction is essential. To ensure sharp
results, the loss $L_{\text{Edge}}$ is added after training for a certain
number of iterations (300k in this paper). The edge loss $L_{\text{Edge}}$ is
defined as described in Eq. 3,

$L_{\text{Edge}}(x,y)=\lambda L_{\text{Char}}(x-g((g(x)^{\downarrow
2})^{\uparrow 2}),y-g((g(y)^{\downarrow 2})^{\uparrow 2})),$ (3)

where $g$ is a Gaussian filter, $\lambda$ represents a small gain (set to 0.05
in this paper), and $(\cdot)^{\downarrow 2}$ and $(\cdot)^{\uparrow 2}$ denote
downsampling and upsampling by 2, respectively

##  4 Datasets

###  4.1 Synthetic dataset

As ground truth for atmospheric turbulence mitigation is not available, we
generate synthetic distorted sequences from clean ones. We employed the method
based on P2S transform, proposed by Mao et al. (2021), to generate atmospheric
turbulence distortions, as it has proved to be efficient for use as training
data.

For static scenes, the dataset is sourced from the Place Dataset (Zhou et al.
(2018)), where 9,017 images were randomly selected. These images serve as the
basis for simulation, where each generates 50 corresponding turbulence-
impacted images and their distortion-free counterparts, resulting in a total
of 9,017 pairs of static scene sequences (total 450,850 image pairs).

For dynamic scenes, the dataset is enriched with video content from multiple
sources to ensure variability and complexity, mimicking real-world conditions
more accurately. The primary sources for these dynamic scenes are the Sports
Video in the Wild (SVW) (Safdarnejad et al. (2015)) dataset, the ground truth
videos from the TSRWGAN (Jin et al. (2021)) project, and the Video Dataset of
Perceived Visual Enhancements (VDPVE) (Gao et al. (2023)). This integration
forms a comprehensive collection of 6,495 video pairs (total 2,749,582 image
pairs).

The dataset is split into training and testing subsets to facilitate practical
training and evaluation. The static images are divided into 7,499 pairs for
training and 1,518 pairs for validation. The dynamic videos are similarly
split, allocating 4,700 videos for training and 1,795 for testing, maintaining
a cap of 120 frames per video in the testing set to ensure uniformity in
evaluation conditions.

###  4.2 Real datasets

Videos with real atmospheric turbulence are used to evaluate the performance
and generalization capabilities of the proposed method. This includes two
datasets: the OTIS (Gilles and Ferrante (2017)) dataset and the CLEAR
(Anantrasirichai (2023)) dataset. The OTIS dataset contains 16 static scenes
with ground truth. The CLEAR dataset includes 11 dynamic scenes with
significant motion and 8 static scenes with minimal or no motion. This dataset
comes with pseudo ground truth generated using complex wavelet-based image
fusion. Collectively, these datasets encompass a broad spectrum of turbulence
conditions.

##  5 Results and discussions

###  5.1 Experiment settings

We trained the two modules separately. Initially, the non-rigid registration
module was trained for 400K iterations using 12 frames of input video, a patch
size of 128, and a batch size of 2. This phase focused on establishing a
robust base for angular correction before proceeding to more complex tasks.
Once this module’s training was solidified, the feature fusion module was
trained for 600K iterations under the same batch size, patch size, and
learning rate conditions, ensuring consistency in training dynamics. Both
models employ the Adam optimizer, known for its efficiency in handling sparse
gradients on noisy problems, in conjunction with a Cosine Annealing scheduler.
This scheduler adjusts the learning rate from an initial value of $2\times
10^{-4}$ to $1\times 10^{-6}$, promoting a gradual and controlled optimization
process.

We evaluated and compared our method with four state-of-the-art models
designed to mitigate atmospheric turbulence in long-range imaging: AT-Net
(Yasarla and Patel (2021)), TurbNet (Mao et al. (2022)), TSRWGAN (Jin et al.
(2021)), and TMT (Zhang et al. (2024)). Additionally, we included the state-
of-the-art video restoration models, BasicVSR++ (Chan et al. (2022)), STA-
SUNet (Lin et al. (2024b)), and VRT (Liang et al. (2024)). These models were
retrained with our synthetic datasets and tested on both synthetic and real-
world datasets. All comparative models were trained using similar data
augmentation strategies as employed for the TMT, ensuring comparable
conditions and fair performance evaluation.

###  5.2 Performance on synthetic datasets

With ground truth available, we perform an objective assessment using PSNR,
SSIM, and LPIPS measurements. Table 2 presents the average results, calculated
by first averaging the results of all frames within each scene, and then
averaging these scene results. This approach ensures that the results are not
biased toward longer videos. The results clearly show that our method
outperforms other existing methods for all metrics, highlighting that the
deformable property facilitates a more flexible convolution operation,
effectively modeling and correcting geometric distortions caused by
atmospheric variations. Additionally, 3D Swin transformers demonstrate their
effectiveness in deblurring and visual quality enhancement. Interestingly,
BasicVSR++ delivers good results in PSNR and SSIM, but not in LPIPS.
BasicVSR++ also exploits deformable convolutions, but applies optical flows
for large motion beforehand. This should benefit dynamic scenes, which we plan
to investigate further in future work. Example subjective results of the
synthetic videos are shown in Fig. 2, where our visual results are sharper,
and the zoomed-in areas of the straight lines reveal cleaner and clearer
restored lines than other methods. This comparison obviously aligns with the
objective results.

Table 2: Performance comparison on static and dynamic scenes using a synthetic dataset. Bold and underline indicate the best and second-best results, respectively. Methods | Static Scenes | Dynamic Scenes
---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
AT-Net (Yasarla and Patel (2021)) | 20.86 | 0.603 | 0.518 | 20.21 | 0.587 | 0.488
TurbNet (Mao et al. (2022)) | 21.40 | 0.637 | 0.431 | 21.35 | 0.635 | 0.433
STA-SUNet (Lin et al. (2024b)) | 23.58 | 0.712 | 0.352 | 23.45 | 0.706 | 0.348
BasicVSR++ (Chan et al. (2022)) | 26.10 | 0.792 | 0.279 | 26.14 | 0.790 | 0.280
TSRWGAN (Jin et al. (2021)) | 24.99 | 0.763 | 0.249 | 25.05 | 0.760 | 0.251
VRT (Liang et al. (2024)) | 25.85 | 0.782 | 0.218 | 25.78 | 0.766 | 0.266
TMT (Zhang et al. (2024)) | 25.94 | 0.795 | 0.202 | 26.09 | 0.767 | 0.264
DeTurb (ours) | 27.17 | 0.827 | 0.170 | 27.44 | 0.815 | 0.242
Figure 2: Subjective results of a synthetic scene. The bottom of each picture
shows a magnified version of the straight lines.

###  5.3 Performance on real dataset

For real atmospheric turbulence, we evaluated our proposed DeTurb and compared
it with existing methods using both reference and no-reference metrics. The
static scenes of the OTIS dataset come with ground truth, while the scenes of
the CLEAR dataset come with pseudo ground truth; hence, objective assessment
is possible. We employed NIQE scores (Mittal et al. (2013)) as a no-reference
evaluation. NIQE is based on a quality-aware set of statistical features
derived from a straightforward yet effective natural scene statistic (NSS)
model in the spatial domain. These features are extracted from a collection of
natural, undistorted images. Lower scores indicate better perceptual quality.

Table 3 shows the average results of these scenes. Comparing the two existing
methods that performed best on the synthetic data, BasicVSR++ and TMT, our
method achieves better scores across all metrics, particularly in LPIPS. For
the no-reference metric NIQE, our method achieves approximately 9% and 16%
higher scores, respectively. Fig. 3 illustrates examples of the restored
scenes, with the last column showing the pseudo ground truth from CLEAR. Our
method produces results with clearer edges and more readable text compared to
BasicVSR++ and TMT. Although it cannot restore textures or text as well as
CLEAR, our method achieves smoother edges than CLEAR. Overall, DeTurb
outperforms other learning-based methods.

We found that when the atmospheric turbulence is low, DeTurb produces very
sharp edges, achieving straight lines and clear curves, particularly at high
contrast, such as text on road signs. However, strong distortions are more
difficult to recover, as demonstrated in the last row where light rays
propagate through the medium from a distance. Although the results appear
sharper, random geometric distortion is still present in the results of all
methods.

Table 3: Performance comparison on real atmospheric turbulence scenes. Bold
and underline indicate the best and second-best results, respectively.

Methods | Static Scenes | Dynamic Scenes
---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | NIQE $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | NIQE $\downarrow$
Distorted inputs | - | - | - | 26.48 | - | - | - | 28.07
AT-Net (Yasarla and Patel (2021)) | 15.65 | 0.593 | 0.483 | 16.37 | 18.04 | 0.773 | 0.472 | 27.56
TurbNet (Mao et al. (2022)) | 15.08 | 0.684 | 0.453 | 26.11 | 19.38 | 0.736 | 0.411 | 27.23
STA-SUNet (Lin et al. (2024b)) | 20.74 | 0.698 | 0.433 | 26.94 | 22.66 | 0.784 | 0.358 | 26.68
BasicVSR++ (Chan et al. (2022)) | 21.66 | 0.754 | 0.232 | 26.72 | 26.84 | 0.845 | 0.206 | 25.03
TSRWGAN (Jin et al. (2021)) | 16.87 | 0.710 | 0.228 | 26.72 | 24.73 | 0.809 | 0.130 | 24.88
VRT (Liang et al. (2024)) | 15.25 | 0.620 | 0.338 | 28\. 46 | 25.26 | 0.818 | 0.133 | 25.63
TMT (Zhang et al. (2024)) | 15.17 | 0.611 | 0.329 | 29.21 | 25.59 | 0.822 | 0.122 | 27.98
DeTurb (ours) | 24.53 | 0.841 | 0.128 | 24.34 | 26.99 | 0.847 | 0.088 | 22.77

Figure 3: Subjective results of a real distorted scene. The left column shows
a static scene, while the other columns depict dynamic scenes. From top to
bottom, the rows display the distorted input, BasicVSR++ results, TMT results,
our DeTurb results, and CLEAR results.

###  5.4 Ablation study

The results of the ablation study are shown in Table 4. First, we investigated
the influence of deformable 3D convolutions by replacing them with depth-wise
3D convolutions, as used in TMT (Zhang et al. (2024)). All metrics indicated a
performance reduction, particularly for LPIPS. Next, we tested the impact of
the 3D Swin transformation by replacing it with 2D Swin transformers (Liu et
al. (2021)). Although the overall performance was reduced, it was not as
significant as the decrease observed when replacing deformable 3D convolutions
with depth-wise 3D convolutions. This highlights the importance of addressing
spatiotemporal distortions caused by atmospheric turbulence. This significance
is further confirmed by the substantial reduction in restoration quality when
the non-rigid registration module is removed, which has a greater impact than
removing the feature fusion module from the pipeline.

Table 4: Ablation study showing the effects of replacing or removing certain
modules in the proposed framework. The results are averaged from both
synthetic and real scenes, except for NIQE, which is computed only on the real
scenes.

Methods | Static Scenes | Dynamic Scenes
---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | NIQE $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | NIQE $\downarrow$
Deform3D $\rightarrow$ DW Conv 3D | 24.25 | 0.772 | 0.249 | 25.73 | 27.05 | 0.789 | 0.231 | 25.08
3D Swin $\rightarrow$ 2D Swin | 25.15 | 0.800 | 0.188 | 25.75 | 27.13 | 0.816 | 0.193 | 23.72
wo Non-rigid | 23.88 | 0.761 | 0.261 | 26.02 | 26.87 | 0.795 | 0.246 | 25.12
wo Fusion | 24.07 | 0.778 | 0.258 | 25.70 | 26.53 | 0.800 | 0.228 | 24.89
DeTurb | 25.85 | 0.834 | 0.149 | 24.34 | 27.22 | 0.831 | 0.165 | 22.77

For clearer visualization, Fig. 4 shows the $y$-$t$ plane images, illustrating
how a specific line or point at a particular $x$ position within an image
evolves over time across video frames. This provides a view of the
stabilization achieved by turbulence mitigation models. We selected the middle
point of the width. The left image is from the original distorted scene,
showing clear vertical streaks and inconsistencies. The feature fusion model
alone shows improvement, with noticeably reduced streaking and clearer
continuity of lines, but some random geometric distortions are still present,
as seen in the middle figure. The entire pipeline, including the non-rigid
registration, exhibits the highest level of clarity and consistency,
effectively mirroring the original structure of the scene with superior
stabilization of vertical elements.

Figure 4: Example $y$-$t$ planes of static ‘Man’ scene restored without and
with the non-rigid registration module

###  5.5 Computing Budget

We include an assessment of the inference-time computing budget for each
model. This analysis is conducted on a single NVIDIA 4090 GPU, providing a
standardized basis for comparing the computational demands of each model.
Table 5 summarizes the computing resources and time required for each model.
Although none of the learning-based methods currently meet real-time
requirements, they are significantly faster than conventional model-based
methods such as SID and CLEAR.

Table 5: Inference time computing budget, measured per frame on a 256$\times$256 resolution image (average speed calculated from five test trials). Methods |  $\\#$ parameters (M) | FLOPs/frame (G) | speed (s)
---|---|---|---
SID (Zhu and Milanfar (2013)) | - | - | 132.32
CLEAR (Anantrasirichai (2023)) | - | - | 25.37
TurbNet (Mao et al. (2022)) | 26.60 | 190.69 | 5.79
STA-SUNet (Lin et al. (2024b)) | 21.82 | - | 2.04
BasicVSR++ (Chan et al. (2022)) | 9.76 | 127.30 | 1.20
TSRWGAN (Jin et al. (2021)) | 46.28 | 2,836 | 2.58
VRT (Liang et al. (2024)) | 18.32 | 7,759 | 7.35
TMT (Zhang et al. (2024)) | 23.92 | 1,304 | 2.37
DeTurb (ours) | 58.79 | 1,975 | 2.55

##  6 Conclusion

This paper introduces DeTurb, a novel framework for atmospheric turbulence
reduction. The proposed framework consists of two modules: non-rigid
registration and feature fusion. The first module uses deformable 3D
convolutions to estimate flow and mitigate random geometric distortion across
several distorted frames. The second module, utilizing a UNet-like
architecture of 3D Swin transformers, further sharpens and enhances the
details of the current frame. Experimental results demonstrate that DeTurb
outperforms existing methods specifically designed for atmospheric turbulence
problems as well as methods proposed for video restoration and enhancement.
However, there is still room for improvement in situations of strong
atmospheric turbulence.

## References

  * Anantrasirichai et al. [2013] N. Anantrasirichai, A. Achim, N.G. Kingsbury, and D.R. Bull.  Atmospheric turbulence mitigation using complex wavelet-based fusion.  _Image Processing, IEEE Transactions on_ , 22(6):2398–2408, 2013.
  * Anantrasirichai [2023] Nantheera Anantrasirichai.  Atmospheric turbulence removal with complex-valued convolutional neural network.  _Pattern Recognition Letters_ , 171:69–75, 2023.  ISSN 0167-8655.  doi: https://doi.org/10.1016/j.patrec.2023.05.017.
  * Boehrer et al. [2021] Nicolas Boehrer, Robert P. J. Nieuwenhuizen, and Judith Dijk.  Turbulence mitigation in imagery including moving objects from a static event camera.  _Optical Engineering_ , 60(5):1 – 19, 2021.  doi: 10.1117/1.OE.60.5.053101.  URL https://doi.org/10.1117/1.OE.60.5.053101.
  * Cai et al. [2023] Y. Cai, Y. Long, Z. Han, et al.  Swin Unet3D: a three-dimensional medical image segmentation network combining vision transformer and convolution.  _BMC Medical Informatics and Decision Making_ , 23:33, 2023.  doi: 10.1186/s12911-023-02129-z.
  * Chan et al. [2022] Kelvin C.K. Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy.  Basicvsr++: Improving video super-resolution with enhanced propagation and alignment.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5972–5981, June 2022.
  * Charbonnier et al. [1997] P. Charbonnier, L. Blanc-Feraud, G. Aubert, and M. Barlaud.  Deterministic edge-preserving regularization in computed imaging.  _IEEE Transactions on Image Processing_ , 6(2):298–311, 1997.  doi: 10.1109/83.551699.
  * Chen et al. [2020] Gongping Chen, Zhisheng Gao, Qiaolu Wang, and Qingqing Luo.  Blind de-convolution of images degraded by atmospheric turbulence.  _Applied Soft Computing_ , 89:106131, 2020.  ISSN 1568-4946.  doi: https://doi.org/10.1016/j.asoc.2020.106131.  URL https://www.sciencedirect.com/science/article/pii/S1568494620300715.
  * Cheng et al. [2023] Jiuming Cheng, Wenyue Zhu, Jianyu Li, Gang Xu, Xiaowei Chen, and Cao Yao.  Restoration of atmospheric turbulence-degraded short-exposure image based on convolution neural network.  _Photonics_ , 10(6), 2023.  ISSN 2304-6732.  doi: 10.3390/photonics10060666.  URL https://www.mdpi.com/2304-6732/10/6/666.
  * Chung et al. [2023] H. Chung, J. Kim, S. Kim, and J. Ye.  Parallel diffusion models of operator and image for blind inverse problems.  In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6059–6069, jun 2023.  doi: 10.1109/CVPR52729.2023.00587.
  * Dai et al. [2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * Ettedgui and Yitzhaky [2023] Bar Ettedgui and Yitzhak Yitzhaky.  Atmospheric turbulence degraded video restoration with recurrent GAN (ATVR-GAN).  _Sensors_ , 23(21), 2023.  ISSN 1424-8220.  doi: 10.3390/s23218815.  URL https://www.mdpi.com/1424-8220/23/21/8815.
  * Fried [1978] David L. Fried.  Probability of getting a lucky short-exposure image through turbulence$\ast$.  _J. Opt. Soc. Am._ , 68(12):1651–1658, Dec 1978.  doi: 10.1364/JOSA.68.001651.  URL https://opg.optica.org/abstract.cfm?URI=josa-68-12-1651.
  * Gao et al. [2019] Jing Gao, N. Anantrasirichai, and David Bull.  Atmospheric turbulence removal using convolutional neural network.  _arXiv preprint arXiv:1912.11350_ , 2019.
  * Gao et al. [2023] Yixuan Gao, Yuqin Cao, Tengchuan Kou, Wei Sun, Yunlong Dong, Xiaohong Liu, Xiongkuo Min, and Guangtao Zhai.  Vdpve: Vqa dataset for perceptual video enhancement.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 1474–1483, June 2023.
  * Gilles and Ferrante [2017] Jérôme Gilles and Nicholas B. Ferrante.  Open turbulent image set (otis).  _Pattern Recognition Letters_ , 86:38–41, 2017.  ISSN 0167-8655.  URL https://www.sciencedirect.com/science/article/pii/S0167865516303750.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 6840–6851, 2020.
  * Hu and Anantrasirichai [2023] D Hu and N Anantrasirichai.  Object recognition in atmospheric turbulence scenes.  In _2023 31st European Signal Processing Conference (EUSIPCO)_ , pages 561–565, 2023.
  * Hunt et al. [2018] Bobby R. Hunt, Amber L. Iler, Christopher A. Bailey, and Michael A. Rucci.  Synthesis of atmospheric turbulence point spread functions by sparse and redundant representations.  _Optical Engineering_ , 57(2):024101, 2018.  doi: 10.1117/1.OE.57.2.024101.  URL https://doi.org/10.1117/1.OE.57.2.024101.
  * Jaiswal et al. [2023] Ajay Jaiswal, Xingguang Zhang, Stanley H. Chan, and Zhangyang Wang.  Physics-driven turbulence image restoration with stochastic refinement.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 12170–12181, October 2023.
  * Jiang et al. [2023] Weiyun Jiang, Vivek Boominathan, and Ashok Veeraraghavan.  Nert: Implicit neural representations for unsupervised atmospheric turbulence mitigation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 4236–4243, June 2023.
  * Jin et al. [2021] Darui Jin, Ying Chen, Yi Lu, Junzhang Chen, Peng Wang, Zichao Liu, Sheng Guo, and Xiangzhi Bai.  Neutralizing the impact of atmospheric turbulence on complex scene imaging via deep learning.  _Nature Machine Intelligence_ , 3(10):876–884, Oct 2021.  doi: https://doi.org/10.1038/s42256-021-00392-1.
  * Kelmelis et al. [2017] Eric J. Kelmelis, Stephen T. Kozacik, and Aaron L. Paolini.  Practical considerations for real-time turbulence mitigation in long-range imagery.  _Optical Engineering_ , 56:1 – 12, 2017.  doi: 10.1117/1.OE.56.7.071506.  URL http://dx.doi.org/10.1117/1.OE.56.7.071506.
  * Leihong et al. [2021] Zhang Leihong, Bian Zhixiang, Ye Hualong, Wang Zhaorui, Wang Kaimin, and Zhang Dawei.  Restoration of single pixel imaging in atmospheric turbulence by fourier filter and cgan.  _Applied Physics B_ , 127(3), Mar 2021.  doi: https://doi.org/10.1007/s00340-021-07596-8.
  * Li et al. [2018] Jin Li, Min Zhang, Danshi Wang, Shaojun Wu, and Yueying Zhan.  Joint atmospheric turbulence detection and adaptive demodulation technique using the cnn for the oam-fso communication.  _Opt. Express_ , 26(8):10494–10508, Apr 2018.  doi: 10.1364/OE.26.010494.  URL https://opg.optica.org/oe/abstract.cfm?URI=oe-26-8-10494.
  * Liang et al. [2024] Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, and Luc Van Gool.  VRT: A video restoration transformer.  _IEEE Transactions on Image Processing_ , 33:2171–2182, 2024.  doi: 10.1109/TIP.2024.3372454.
  * Lin et al. [2024a] R Lin, N Anantrasirichai, G Huang, J Lin, Q Sun, A Malyugina, and DR Bull.  BVI-RLV: A fully registered dataset and benchmarks for low-light video enhancement.  _arXiv preprint arXiv:2407.03535_ , 2024a.
  * Lin et al. [2024b] Ruirui Lin, Nantheera Anantrasirichai, Alexandra Malyugina, and David Bull.  A spatio-temporal aligned sunet model for low-light video enhancement.  In _IEEE International Conference on Image Processing_ , 2024b.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 10012–10022, October 2021.
  * Mao et al. [2021] Zhiyuan Mao, Nicholas Chimitt, and Stanley H. Chan.  Accelerating atmospheric turbulence simulation via learned phase-to-space transform.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 14759–14768, October 2021.
  * Mao et al. [2022] Zhiyuan Mao, Ajay Jaiswal, Zhangyang Wang, and Stanley H. Chan.  Single frame atmospheric turbulence mitigation: A benchmark study and a new physics-inspired transformer model.  In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, _Computer Vision – ECCV 2022_ , pages 430–446, Cham, 2022. Springer Nature Switzerland.  ISBN 978-3-031-19800-7.
  * Mei and Patel [2023] Kangfu Mei and Vishal M. Patel.  LTT-GAN: Looking through turbulence by inverting gans.  _IEEE Journal of Selected Topics in Signal Processing_ , 17(3):587–598, 2023.  doi: 10.1109/JSTSP.2023.3238552.
  * Mittal et al. [2013] Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik.  Making a “completely blind” image quality analyzer.  _IEEE Signal Processing Letters_ , 20(3):209–212, 2013.  doi: 10.1109/LSP.2012.2227726.
  * Nair et al. [2023] Nithin Gopalakrishnan Nair, Kangfu Mei, and Vishal M. Patel.  At-ddpm: Restoring faces degraded by atmospheric turbulence using denoising diffusion probabilistic models.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 3434–3443, January 2023.
  * Safdarnejad et al. [2015] Seyed Morteza Safdarnejad, Xiaoming Liu, Lalita Udpa, Brooks Andrus, John Wood, and Dean Craven.  Sports videos in the wild (svw): A video dataset for sports analysis.  In _2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)_ , volume 1, pages 1–7, 2015.  doi: 10.1109/FG.2015.7163105.
  * Shang et al. [2023] Wei Shang, Dongwei Ren, Yi Yang, Hongzhi Zhang, Kede Ma, and Wangmeng Zuo.  Joint video multi-frame interpolation and deblurring under unknown exposure time.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 13935–13944, June 2023.
  * Suin et al. [2024] Maitreya Suin, Nithin Gopalakrishnan Nair, Chun Pong Lau, Vishal M. Patel, and Rama Chellappa.  Diffuse and restore: A region-adaptive diffusion model for identity-preserving blind face restoration.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 6343–6352, January 2024.
  * Tang et al. [2022] Yucheng Tang, Dong Yang, Wenqi Li, Holger R. Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh.  Self-supervised pre-training of swin transformers for 3d medical image analysis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 20730–20740, June 2022.
  * Wang et al. [2023] Xijun Wang, Santiago Lopez-Tapia, and Aggelos K. Katsaggelos.  Atmospheric turbulence correction via variational deep diffusion.  In _2023 IEEE 6th International Conference on Multimedia Information Processing and Retrieval (MIPR)_ , pages 1–4, 2023.  doi: 10.1109/MIPR59079.2023.00022.
  * Wang et al. [2019] Xintao Wang, Kelvin C.K. Chan, Ke Yu, Chao Dong, and Chen Change Loy.  EDVR: Video restoration with enhanced deformable convolutional networks.  In _IEEE/CVF CVPR_ , June 2019.
  * Yang et al. [2023] Yu-Qi Yang, Yu-Xiao Guo, Jian-Yu Xiong, Yang Liu, Hao Pan, Peng-Shuai Wang, Xin Tong, and Baining Guo.  Swin3d: A pretrained transformer backbone for 3d indoor scene understanding, 2023.
  * Yasarla and Patel [2021] Rajeev Yasarla and Vishal M. Patel.  Learning to restore images degraded by atmospheric turbulence using uncertainty.  In _2021 IEEE International Conference on Image Processing (ICIP)_ , pages 1694–1698, 2021.  doi: 10.1109/ICIP42928.2021.9506614.
  * Yasarla and Patel [2022] Rajeev Yasarla and Vishal M. Patel.  CNN-based restoration of a single face image degraded by atmospheric turbulence.  _IEEE Transactions on Biometrics, Behavior, and Identity Science_ , 4(2):222–233, 2022.  doi: 10.1109/TBIOM.2022.3169697.
  * Ying et al. [2020] Xinyi Ying, Longguang Wang, Yingqian Wang, Weidong Sheng, Wei An, and Yulan Guo.  Deformable 3d convolution for video super-resolution.  _IEEE Signal Processing Letters_ , 27:1500–1504, 2020.  doi: 10.1109/LSP.2020.3013518.
  * Zhang et al. [2023a] Meng Zhang, Mustafa Z. Yousif, Linqi Yu, and Hee-Chang Lim.  A Swin-transformer-based model for efficient compression of turbulent flow data.  _Physics of Fluids_ , 35(8):085108, 08 2023a.  ISSN 1070-6631.  doi: 10.1063/5.0160755.  URL https://doi.org/10.1063/5.0160755.
  * Zhang et al. [2011] Shixue Zhang, Yuanhao Wu, Jinyu Zhao, and Jianli Wang.  Astronomical image restoration through atmosphere turbulence by lucky imaging.  In Ting Zhang, editor, _Third International Conference on Digital Image Processing (ICDIP 2011)_ , volume 8009, page 80090B. International Society for Optics and Photonics, SPIE, 2011.  doi: 10.1117/12.896183.  URL https://doi.org/10.1117/12.896183.
  * Zhang et al. [2024] Xingguang Zhang, Zhiyuan Mao, Nicholas Chimitt, and Stanley H. Chan.  Imaging through the atmosphere using turbulence mitigation transformer.  _IEEE Transactions on Computational Imaging_ , 10:115–128, 2024.  doi: 10.1109/TCI.2024.3354421.
  * Zhang et al. [2023b] Ziran Zhang, Bin Zhao, Yueting Chen, Zhigang Wang, Dong Wang, Jiawei Sun, Jie Zhang, Zhihai Xu, and Xuelong Li.  ASF-Transformer: neutralizing the impact of atmospheric turbulence on optical imaging through alternating learning in the spatial and frequency domains.  _Opt. Express_ , 31(22):37128–37141, Oct 2023b.  doi: 10.1364/OE.503131.  URL https://opg.optica.org/oe/abstract.cfm?URI=oe-31-22-37128.
  * Zhou et al. [2018] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 40(6):1452–1464, 2018.  doi: 10.1109/TPAMI.2017.2723009.
  * Zhu and Milanfar [2013] Xiang Zhu and Peyman Milanfar.  Removing atmospheric turbulence via space-invariant deconvolution.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 35(1):157–170, 2013.  doi: 10.1109/TPAMI.2012.82.
  * Çaliskan and Arica [2014] Tufan Çaliskan and Nafiz Arica.  Atmospheric turbulence mitigation using optical flow.  In _2014 22nd International Conference on Pattern Recognition_ , pages 883–888, 2014.  doi: 10.1109/ICPR.2014.162.


# A Whole Cosmology View of the Hubble Constant

Eric V. Linder1,2 1Berkeley Center for Cosmological Physics & Berkeley Lab,
University of California, Berkeley, CA 94720, USA
2Energetic Cosmos Laboratory, Nazarbayev University, Astana, Qazaqstan 010000

###### Abstract

The Hubble constant $H_{0}$ is the value of the cosmic expansion rate at one
time (the present), and cannot be adjusted successfully without taking into
account the entire expansion history and cosmology. We outline some
conditions, that if not quite “no go” are “no thanks”, showing that changing
the expansion history, e.g. employing dynamical dark energy, cannot reconcile
disparate deductions of $H_{0}$ without upsetting some other cosmological
measurement.

##  I Introduction

Cosmology is the study of the universe as a whole. This is usually interpreted
as Cosmology is the study of the (universe as a whole), where universe as a
whole means large spatial scales and large volumes. However one should also be
aware that Cosmology is the study of the universe (as a whole), where as a
whole means both over a wide range of history and over the full panoply of
observations. This has two essential implications for the Hubble constant
discussion:

  1. 1.

$H_{0}\equiv H(z=0)$ is merely a value at one instant of a dynamical
description of cosmic expansion, and it is very little use getting it “right”
without accounting properly for the whole dynamics.

  2. 2.

$H_{0}$ is merely one number from the foundational functions, e.g. cosmic
expansion history, cosmic growth history, cosmic gravity history, that
describe cosmology, and cannot be examined in isolation.

This chapter focuses on these two aspects of the Hubble constant issue,
specifically whether changing the expansion history through using a dynamical
(dark energy) density can reconcile different deductions of $H_{0}$.

We will give arguments that are as model independent as possible, and hence
not list the quite numerous models that have been considered (see, for
example, [1, 2, 3, 4, 5] and references therein for many examples). Rather we
will seek to come as close to “no go” strictures as possible, though in the
vast arena of trade offs between standard and created model parameters these
are closer to “no thanks”. The philosophy follows that of Occam, not to
multiply entities unless necessary.

One of the simplest reconcilings of the Hubble constant values is that some
measurement is incorrect. Indeed, in the nearly 100 year history of the Hubble
constant this has always been the answer, with much of that history having
discrepancies at nominal statistical $\sigma$ levels in excess of the present
situation. We will say no more about this resolution beyond Figure 1, instead
assuming that the discrepant deductions are all valid and seeking for a
cosmological paradigm to reconcile them.

Figure 1: Ladders require care in use and are most stable when anchored firmly
at both ends.

##  II Expansion History

It is relatively easy to shift the value of $H_{0}$ by changing the
cosmological model, but again $H_{0}$ is not the whole of cosmology. To
repeat: cosmology is the study of the universe as a whole, or alternately, all
cosmology all the time [6, 7]. We need to take into account the effects on the
full expansion history $H(z)$, not just $H(z=0)$.

Let us being by saying that the cold dark matter plus cosmological constant,
i.e. $\Lambda$CDM, cosmology is a good description of the expansion history.
That is, we would like to match some expansion history model to observations
that are adequately described within the (flat) cosmology

$H^{2}(z)=H^{2}_{2}\Omega_{m,2}(1+z)^{3}+H^{2}_{2}(1-\Omega_{m,2})\,.$ (1)

Here $H_{2}=H(z=0)$ is the Hubble constant for this cosmology and we specify
that the model has a matter density fraction $\Omega_{m,2}$. However, when
making local universe measurements we find the Hubble constant is $H_{1}$ and
try to reconcile the situation by allowing deviations from $\Lambda$CDM in the
form of dynamical dark energy. So we take a model

$H^{2}(z)=H^{2}_{1}\Omega_{m,1}(1+z)^{3}+\frac{8\pi G}{3}\rho_{\rm de}(z)\ .$
(2)

We only relate these models over a finite range of redshift, where
observations indicate the $\Lambda$CDM model matches the observations well.
(One could add spatial curvature as well but this does not qualitatively
change the results.) Thus the necessary dark energy density is

$\frac{8\pi G}{3}\rho_{\rm
de}(z)=(H^{2}_{2}\Omega_{m,2}-H^{2}_{1}\Omega_{m,1})(1+z)^{3}+H^{2}_{2}(1-\Omega_{m,2})\
.$ (3)

We can immediately see why it is so crucial to do cosmology as a whole, rather
than only think about the Hubble constant. If we have observations determining
$\Omega_{m}h^{2}$ (the cosmic microwave background (CMB) value for what should
be a universal constant $\Omega_{m}h^{2}$ should be much less model dependent
than the CMB value for $h$ itself) then we are forced to conclude that our
attempt to change the Hubble constant through dynamical dark energy fails: one
must have $H_{2}=H_{1}$. It is only when we ignore the rest of cosmology that
we can obtain a different Hubble constant.

Suppose that we do not have $\Omega_{m}h^{2}$ determined by the CMB. In that
case then Eq. (3) tells us the form of dark energy that can shift the Hubble
constant – it looks like the sum of a cosmological constant and a matter-like
term. In particular, if the value $H_{2}$ coming from the $\Lambda$CDM fit to
the observations (e.g. the “CMB” value) is less than the value $H_{1}$ coming
from the local measurements then the effective energy density will generally
be negative (one has to compensate for the higher expansion rate). In any
case, dark energy looking like the sum of matter and $\Lambda$ has been
studied for decades, for example in mocker and polytropic dark energy [8, 9];
very generically it causes growth of structure to deviate significantly from
the $\Lambda$CDM cosmology. Once again, one must use cosmology as a whole when
assessing the viability of shifting the Hubble constant.

##  III Distance-redshift relation

One could attempt to sidestep the “no thanks” scenarios of the previous
section by saying we do not know the expansion history that well, and are
willing to allow $H(z)$ to deviate significantly from the $\Lambda$CDM form.
This is tenable in 2022, but will shortly be severely constrained by radial
baryon acoustic oscillation (BAO) measurements from the Dark Energy
Spectroscopic Instrument [10, 11]. For this approach, rather than matching
$H(z)$ over some redshift range one only matches the integral quantity of the
distance-redshift relation,

$d(z)=\int_{0}^{z}\frac{dz^{\prime}}{H(z^{\prime})}\ .$ (4)

One can have $H(z)$ dip below the $\Lambda$CDM expectation at some redshifts
and rise above it at others to obtain $d(z)$ near $\Lambda$CDM over some
finite redshift range.

Since distances have been measured fairly accurately, and since a deviation in
$H(z^{\prime})$ at some redshift affects all $d(z>z^{\prime})$, this is not
that easy to accomplish, especially when taking into account CMB distances as
well. Furthermore, changing $H(z)$ also affects growth of structure and
generally the large swings needed to alter the Hubble constant throw off the
growth history (including the large scale integrated Sachs-Wolfe effect in the
CMB, see e.g. [12]). In fact, growth of structure is not only tightly bound to
the expansion history within general relativity but nearly determined by the
well measured distance to CMB last scattering.

Figure 2 shows how tightly the growth of structure is bound to the distance to
CMB last scattering, not just for $\Lambda$CDM but for quite disparate models
along the dynamical “mirage of $\Lambda$” line. Conversely, at low redshift a
cartoon relation is that the growth rate

$f\approx\Omega_{m}^{0.55}\sim\left(\frac{\Omega_{m}h^{2}}{h^{2}}\right)^{0.55}\
;$ (5)

so for preserved $\Omega_{m}h^{2}$ one has $f\sim h^{-1.1}$, or if
$f\sigma_{8}$ is measured then $\sigma_{8}\sim h^{1.1}(f\sigma_{8})$.

Figure 2: Constraints on the distance to CMB last scattering $d_{\rm lss}$
tightly constrain structure growth as well, even for highly dynamical matching
dark energy cosmologies. [From [13]]

Thus there is a difficult balancing act in changing the Hubble constant, due
to the impact of the altered expansion history on growth probes (and the CMB
through the Sachs-Wolfe effect), as well as calibration of supernova distances
to higher redshift (while preserving CMB and BAO distances). All these
elements must be considered together. See [14, 15, 16, 17, 18] for some
specific analyses.

##  IV Early Universe Deviations

One might think of moving alterations of the expansion history to high
redshift, above where distance-redshift constraints apply. This still affects
growth however, and generally more severely because of the long lever arm. One
way to get partly around this is to push modifications to before the
recombination epoch. Such an approach is often called early dark energy.

In fact, early dark energy was “discovered” in CMB data in 2013 [19], along
with its concomitant shift in the Hubble constant, as shown in Figure 3.
However the shift was much smaller than the current discrepancy, and
increasing the dark energy density – raising the early expansion rate (and
secondarily decreasing the sound horizon) yet reducing the clustering matter
density fraction – would have a severe effect on both the matter growth
history and the CMB perturbation power spectrum. Indeed, the effect of early
dark energy on the sound horizon has been familiar for decades [20, 21, 22],
but the impact of the reduced perturbation potentials is more deleterious,
forcing a disfavored greater amplitude of mass perturbations ($\sigma_{8}$) to
compensate and obtain consistency with power spectrum observations. It is
extremely difficult to obtain $H_{0}>70$ km/s/Mpc and viable CMB and viable
distances and viable growth. See [23, 24, 25] for some specific examples
regarding this. Models that push even further back, changing the primordial
power spectrum beyond the power form, for example with oscillations or
features, also have difficulty in substantially shifting $H_{0}$, e.g. [26,
27] (and will be tightly constrained by tomographic redshift surveys and
future CMB experiments, e.g. [28]).

Figure 3: Model independent reconstruction of the expansion history from CMB
data in 2013. The expansion deviation $\delta\equiv\delta
H^{2}/H^{2}_{\Lambda{\rm CDM}}$ shows a preference for precombination early
dark energy. This is turn shifts the Hubble constant $H_{0}$ from the value
derived assuming $\Lambda$CDM – but only slightly, even for 8% early dark
energy. [From [19]]

For postrecombination alterations of the expansion history, one again must
emphasize the importance of accounting for the impact on growth, including the
Sachs-Wolfe effect. Recall that the Poisson equation imposes

$\displaystyle(k/a)^{2}\Phi$ $\displaystyle=$ $\displaystyle 4\pi\delta\rho$
(6) $\displaystyle\Rightarrow$ $\displaystyle\Phi\sim a^{2}\delta\rho\sim
a^{2}\frac{\delta\rho}{\rho}\rho$ (7) $\displaystyle\Rightarrow$
$\displaystyle\Phi\sim(a^{3}\rho)\left(a^{-1}\,\frac{\delta\rho}{\rho}\right)\
.$ (8)

In the standard scenario the gravitational potential is thus time independent
and no large Sachs-Wolfe effect is generated. However, if we change the energy
density of clustering matter, e.g. through adding an unclustering early dark
energy that influences the recombination era or through some interaction, then
we affect the growth of structure $\delta\rho/\rho\sim a$ (and possibly
$\rho\sim a^{-3}$) and give a time dependence to the gravitational potential
$\Phi$, resulting in a large Sachs-Wolfe effect contrary to observations. More
specifically, $\delta\rho/\rho\sim a^{1-(3/5)\Omega_{e}}$ so only a very small
fraction of early dark energy $\Omega_{e}$ can be tolerated, hence enabling
only a small shift in the Hubble constant.

A rough rule of thumb is that

$\frac{\delta H_{0}}{H_{0}}\sim\frac{1}{2}\,\frac{\delta
H^{2}}{H^{2}}\sim{\Omega_{e}}\ ,$ (9)

(naively this would be $\sim\Omega_{e}/2$ but the shifting of other parameters
to fit the CMB power spectrum pushes it closer to $\Omega_{e}$ [29]) so a
change in $H_{0}$ to reconcile a 10% discrepancy in values would require
$\Omega_{e}\sim 0.1$ and cause a $10^{3\times(3/5)\Omega_{e}}\sim 50\%$ change
in the growth amplitude today (modulo other shifts). This gives a heuristic
explanation for why it is nearly impossible to shift $H_{0}$ from, say, $\sim
67$ km/s/Mpc to above 70 (see, for example, [30, 4]) and still satisfy
distance, growth, and CMB measurements – i.e. cosmology as a whole.

##  V Extensions

Within the framework used here – matter plus (an effective) dark energy with
background and perturbation equations governed by general relativity in a
homogeneous and isotropic universe – reconciliation of significantly differing
values of the Hubble constant are, if not quite “no go”, then substantially
“no thanks”. Due to the impacts on the many other cosmological
characteristics, mechanisms for shifting $H_{0}$ cause other alterations away
from the observed quantities. One could add further physics to compensate,
e.g. interactions, changes in initial conditions, etc., but these in turn
change a variety of observables and this runs the risk of becoming epicyclic.
Basically it requires multiplying entities beyond necessity – unless one’s
necessity is to obtain a certain value of $H_{0}$ at any cost. Such eschewing
of Occam’s Razor we regard as “no thanks”.

Furthermore, models that seek reconciliation through a mechanism taking place
just before recombination or just before the present add new fine tuning
issues, not simply in the model parameters but in why then questions. If one
seeks a dynamical solution ideally it would arise naturally out of the
cosmology history.

Gravity modified from general relativity offers one avenue of pursuit: it
could compensate for the impact of the altered expansion history on the growth
of structure, or change light propagation in a redshift dependent manner to
better align the values of the Hubble constant derived from strong lens time
delay systems. Modified gravity that achieves a whole cosmology perspective –
simultaneously providing a basis for the current accelerated expansion, growth
history measurements, and reconciliation of Hubble constant values – would be
a theory to explore in depth. However it is difficult to see how such a theory
would work: we generally want gravity to restore to general relativity at
early times such as CMB recombination and on small scales such as the Cepheid
distance ladder measurements so there should be no shift between the $H_{0}$
values.

##  VI Conclusions

The history of the Hubble constant across a century has always been one of
disparate values. In all cases those values have shifted as measurement and
modeling uncertainties have become better studied. There seems little reason
to expect the current situation to be drastically different.

If a compelling, succinct physical explanation for differing values when
derived from distinct probes awaited, then perhaps one might give more weight
to new physics relative to systematics. However as we have reviewed here, it
is remarkably difficult to shift the Hubble constant from a given value, even
in the presence of quite dramatic dynamics at any point in cosmic history –
when taking into account cosmology as a whole. While this does not rise to the
level of a no go theorem, it is definite enough to weigh heavily as a no
thanks likelihood.

A desideratum for a model with new physics would be one that gives an
advantageous likelihood for the full panoply of cosmic observations –
distances, growth, and CMB – without putting a thumb on the scale by imposing
a prior on the Hubble constant, the very issue one is examining. That is
postdiction not prediction. Plainly put, $H_{0}$ data or prior should not be
input into the analysis whose aim is examine the value of $H_{0}$. If the
posterior for distances plus growth plus CMB observations over cosmic history
is not advantaged over a standard $\Lambda$CDM model, then it does not matter
what value of $H_{0}$ can be achieved with the model (the same way that if one
was trying to reconcile the matter density $\Omega_{m}$ then one would not put
a prior on that).

If an alternate model does succeed in fitting all cosmology, all the time,
then one proceeds with Occam’s Razor and physical naturalness examinations. In
the end, more data – with more understanding of the subtleties, selections,
systematics (see for example [31, 32] for steps in this direction) – from more
probes will be the path to resolution, and a firm foundation for cosmology as
a study of the universe as a whole.

## References

  * [1] P. Shah, P. Lemos, O. Lahav, A Buyer’s Guide to the Hubble Constant, Astron. Astroph. Rev. 29, 9 (2021) [arXiv:2109.01161]
  * [2] N. Schönberg, G.F. Abellán, A.P. Sánchez, S.J. Witte, V. Poulin, J. Lesgourgues, The $H_{0}$ Olympics: A Fair Ranking of Proposed Models, Phys. Rep. 984, 1 (2022) [arXiv:2107.10291]
  * [3] E. Di Valentino et al., In the Realm of the Hubble Tension – a Review of Solutions, Class. Quantum Grav. 38, 153001 (2021) [arXiv:2103.01183]
  * [4] E. Di Valentino et al., Cosmology Intertwined II: The Hubble Constant Tension, Astropart. Phys. 131, 102605 (2021) [arXiv:2008.11284]
  * [5] L. Knox, M. Millea, The Hubble Hunter’s Guide, Phys. Rev. D 101, 043533 (2020) [arXiv:1908.03663]
  * [6] E.V. Linder, What is the Standard Cosmological Model?, arXiv:2105.02903
  * [7] K.L. Greene, F-Y. Cyr-Racine, Hubble Distancing: Focusing on Distance Measurements in Cosmology, JCAP 2206, 002 (2022) [arXiv:2112.11567]
  * [8] K. Freese, Generalized Cardassian Expansion: Models in Which the Universe is Flat, Matter Dominated, and Accelerating, Nucl. Phys. B Proc. Suppl. 124, 50 (2003) [arXiv:hep-ph/0208264]
  * [9] E.V. Linder, The Paths of Quintessence, Phys. Rev. D 73, 063010 (2006) [arXiv:astro-ph/0601052]
  * [10] DESI Collaboration, The DESI Experiment Part I: Science, Targeting, and Survey Design, arXiv:1611.00036
  * [11] DESI Collaboration, Overview of the Instrumentation for the Dark Energy Spectroscopic Instrument, Astron. J. 164, 207 (2022) [arXiv:2205.10939]
  * [12] S. Das, A. Shafieloo, T. Souradeep, ISW Effect as Probe of Features in the Expansion History of the Universe, JCAP 1310, 016 (2013) [arXiv:1305.4530]
  * [13] E.V. Linder, The Mirage of $w=-1$, arXiv:0708.0024
  * [14] G. Benevento, W. Hu, M. Raveri, Can Late Dark Energy Transitions Raise the Hubble Constant?, Phys. Rev. D 101, 103517 (2020) [arXiv:2002.11707]
  * [15] D. Camarena, V. Marra, On the Use of the Local Prior on the Absolute Magnitude of Type Ia Supernovae in Cosmological Inference, MNRAS 504, 5164 (2021) [arXiv:2101.08641]
  * [16] G. Alestas, L. Perivolaropoulos, Late Time Approaches to the Hubble Tension Deforming H(z) Worsen the Growth Tension, MNRAS 504, 3956 (2021) [arXiv:2103.04045]
  * [17] G. Efstathiou, To $H_{0}$ or not to $H_{0}$?, MNRAS 505, 3866 (2021) [arXiv:2103.08723]
  * [18] R.E. Keeley, A. Shafieloo, Ruling Out New Physics at Low Redshift as a Solution to the $H_{0}$ Tension, arXiv:2206.08440
  * [19] A. Hojjati, E.V. Linder, J. Samsing, New Constraints on the Early Expansion History, Phys. Rev. Lett. 111 041301 (2013) [arXiv:1304.3724]
  * [20] G. Efstathiou, J. Bond, Cosmic Confusion: Degeneracies among Cosmological Parameters Derived from Measurements of Microwave Background Anisotropies, MNRAS 304, 75 (1999) [arXiv:astro-ph/9807103]
  * [21] D.J. Eisenstein, M.J. White, Theoretical uncertainty in baryon oscillations, Phys. Rev. D 70, 103523 (2004) [arXiv:astro-ph/0407539]
  * [22] M. Doran, S. Stern, E. Thommes, Baryon Acoustic Oscillations and Dynamical Dark Energy, JCAP 0704, 015 (2007) [arXiv:astro-ph/0609075]
  * [23] V.I. Sabla, R.R. Caldwell, No $H_{0}$ Assistance from Assisted Quintessence, Phys. Rev. D 103, 103506 (2021) [arXiv:2103.04999]
  * [24] K. Jedamzik, L. Pogosian, G-B. Zhao, Why Reducing the Cosmic Sound Horizon Alone Can Not Fully Resolve the Hubble Tension, Commun. Phys. 4, 123 (2021) [arXiv:2010.01458]
  * [25] J.C. Hill, E. McDonough, M.W. Toomey, S. Alexander, Early Dark Energy Does Not Restore Cosmological Concordance, Phys. Rev. D 102, 043507 (2020) [arXiv:2003.07355]
  * [26] M. Liu, Z. Huang, Band-limited Features in the Primordial Power Spectrum Do Not Resolve the Hubble Tension, ApJ 897, 166 (2020) [arXiv:1910.05670]
  * [27] A. Antony, F. Finelli, D.K. Hazra, A. Shafieloo, Discordances in Cosmology and the Violation of Slow-roll Inflationary Dynamics, arXiv:2202.14028
  * [28] G. Brando, E.V. Linder, Exploring Early and Late Cosmology with Next Generation Surveys, Phys. Rev. D 101, 103510 (2020) [arXiv:2001.07738]
  * [29] E.V. Linder, G. Robbers, Shifting the Universe: Early Dark Energy and Standard Rulers, JCAP 0806, 004 (2008) [arXiv:0803.2877]
  * [30] E. DiValentino, A. Melchiorri, E.V. Linder, J. Silk, Constraining Dark Energy Dynamics in Extended Parameter Space, Phys. Rev. D 96, 023523 (2017) [arXiv:1704.02153]
  * [31] E. Mörtsell, A. Goobar, J. Johansson, S. Dhawan, The Hubble Tension Revisited: Additional Local Distance Ladder Uncertainties, ApJ 935, 58 (2022) [arXiv:2106.09400]
  * [32] R. Wojtak, J. Hjorth, Intrinsic Tension in the Supernova Sector of the Local Hubble Constant Measurement and its Implications, MNRAS 515, 2790 (2022) [arXiv:2206.08160]