[table]capposition=top capbtabboxtable[][]

# A Unified Framework for Microscopy Defocus Deblur with Multi-Pyramid
Transformer and Contrastive Learning 111Accepted in CVPR 2024. This is not the
final camera-ready version.

Yuelin Zhang1 Pengyu Zheng1 Wanquan Yan1 Chengyu Fang2 Shing Shin
Cheng1222Corresponding author.
1 The Chinese University of Hong Kong  2 Shenzhen International Graduate
School, Tsinghua University

###### Abstract

Defocus blur is a persistent problem in microscope imaging that poses harm to
pathology interpretation and medical intervention in cell microscopy and
microscope surgery. To address this problem, a unified framework including
multi-pyramid transformer (MPT) and extended frequency contrastive
regularization (EFCR) is proposed to tackle two outstanding challenges in
microscopy deblur: longer attention span and feature deficiency. The MPT
employs an explicit pyramid structure at each network stage that integrates
the cross-scale window attention (CSWA), the intra-scale channel attention
(ISCA), and the feature-enhancing feed-forward network (FEFN) to capture long-
range cross-scale spatial interaction and global channel context. The EFCR
addresses the feature deficiency problem by exploring latent deblur signals
from different frequency bands. It also enables deblur knowledge transfer to
learn cross-domain information from extra data, improving deblur performance
for labeled and unlabeled data. Extensive experiments and downstream task
validation show the framework achieves state-of-the-art performance across
multiple datasets. Project page: https://github.com/PieceZhang/MPT-CataBlur.

##  1 Introduction

Microscope offers observers enhanced resolution and magnification [48, 9, 49],
which greatly promotes the advancement of cell microscopy [9] and surgical
microscopy [48]. Cell microscopy employs various optical techniques to reveal
the structure and function of cells [9]. Surgical microscopy assists surgeons
in performing delicate operations [48] including neurosurgery [81],
ophthalmology [4], dentistry [15], etc. In microscopy, out-of-focus, or
defocus, is one of the most common visual impairments caused by inferior
optical quality, lens aperture, or object magnification [48, 9], resulting in
blurred or distorted imaging. It can be disastrous for the downstream tasks
[88], including segmentation [30, 29], detection [61], and classification [6].
While various microscopes with auto-focusing [79, 55, 35], assisted-focusing
[67], or multi-focus [82, 39] capabilities have been developed to mitigate the
defocus effect on-site, image degradation remains if the objects are
distributed non-uniformly and not co-planar [50], or the cavities are too deep
to be aligned with the focal plane [48]. Microscopy defocus deblur methods
have thus been introduced as an offsite restoration approach.

Figure 1: Normalized average attention distance of different datasets. The
distance of real-world datasets (shown in blue) is significantly smaller than
that of microscopy datasets (shown in red), showing the inter-domain feature
difference.

Recent advances in deep learning have led to the development of various deep
defocus deblur methods [60, 34, 56, 57], including those designed for
microscopy [17, 50, 70, 85, 28, 18, 93, 72, 91, 32, 41]. Microscopy deblurring
poses different challenges from real-world deblur tasks, due to the
significant discrepancy between the features in the microscope images and
natural scene images [88]. This difference can be quantified by calculating
the normalized average attention distance for different datasets (attention
intensity multiplied by pixel distance then normalize by image size). The
evaluation involves real-world datasets (DPDD [1], LFDOF [59], DED [47]), cell
microscopy datasets (3DHistech [17], WNLO [17], BBBC006 [44]), and surgical
microscopy datasets for cataract surgery (CaDISv2 [19], Cataract-101 [62]),
retinal microsurgery (RFMiD [54]), neurosurgery (NeuroSurgicalTools [5]). As
shown in Fig. 1, all of the cell microscopy datasets have normalized average
attention distance around 0.35, and surgical microscope datasets around 0.3,
indicating a much longer attention span than the real-world datasets at under
0.25. This result reveals the substantial discrepancy between the two domains,
suggesting that modeling attention in wider areas with larger receptive fields
would benefit microscopy tasks. Motivated by this analysis, we introduce a
multi-pyramid transformer (MPT) with cross-scale window attention (CSWA),
intra-scale channel attention (ISCA) and feature-enhancing feed-forward
network (FEFN), to construct multiple pyramids explicitly on each stage of the
network, fully exploiting latent cross-scale features in every projection
space. CSWA captures the interaction between local $query$ and cross-scale
$key$-$value$ pairs for long-range attention modeling with a quadratically
enlarged receptive field while keeping computational efficiency. ISCA builds
channel-wise attention on a local scale to provide global channel context,
which is then integrated with the spatially correlated feature from CSWA by
the proposed FEFN through an asymmetric activation mechanism.

Another problem in microscopy deblur is the insufficient data for training a
robust model. Different from natural scene defocus deblur methods that use
datasets captured with varying aperture sizes [1] or light field camera [60,
47], the high-quality training data for microscopy deblur can be much harder
to obtain[17, 48]. For cell microscopy, insufficient training feature leads to
a generalizability problem caused by different staining and imaging methods
[17, 88]. The situation is worse for surgical microscopy because the imaging
principle of microscope makes it impossible to simultaneously acquire blur-
sharp pairs for model training [18, 48]. To alleviate the feature deficiency
problem, there exist some training diagrams to learn rich information from
extra data, such as extra data pretraining followed by testing data fine-
tuning [60]. This paradigm, however, may not be applicable to microscopy
deblurring, as there is an inter-domain gap between natural scenes and
microscopy images, and intra-domain gaps among different microscopy datasets
that are highly task-specific. The extended frequency contrastive
regularization (EFCR) is proposed to address the feature deficiency problem by
encouraging the model to learn representations from decoupled frequency bands
in the wavelet domain [86, 23], and further exploiting latent information
leveraging the fact that model trained with synthetic reblurring images can
deblur its naturally blurred counterpart [18]. It also enables cross-domain
deblur knowledge transfer, facilitating multiple scenarios including extra
data training and unlabeled data deblur.

This paper presents a unified deblur framework with MPT and EFCR to address
the aforementioned two challenges in microscopy deblur. As the most common
surgery worldwide [66, 13, 21], cataract surgery is chosen to illustrate
surgical microscopy deblur. Extensive experiments are conducted on various
open-source cell and surgical microscopy datasets, along with downstream tasks
validation on cell detection and surgery scene semantic segmentation. For
surgical microscopy deblur, we present a realistic blur synthesizing method,
and collect a new defocus microscope surgery dataset in cataract microscopic
surgery, which is the first dataset for surgical microscopy deblur. The method
achieves state-of-the-art performance on not only microscope datasets but also
real-world datasets, showing the universality of the proposed framework. The
deblur result on unlabeled dataset also proves the effectiveness of the
proposed EFCR as a knowledge transfer tool. The main contributions are as
follows, 1) The multi-pyramid transformer (MPT) is for the first time proposed
for microscopy defocus deblur. It models the long-range cross-scale spatial
attention in each explicit pyramid using the proposed cross-scale window
attention (CSWA) with quadratically enlarged receptive field to adapt to the
longer attention span of microscopy datasets that hinders the existing
methods. 2) The intra-scale channel attention (ISCA) is presented to model the
global channel attention, incorporating channel context in the CSWA spatial
information via the proposed feature-enhancing feed-forward network (FEFN),
providing additional channel feature to the pyramid. 3) A training strategy
with extended frequency contrastive regularization (EFCR) is presented to
alleviate feature deficiency by exploiting latent deblur signal beyond the
pixel-wise constraint through synthetic reblurring, which is the first
implementation of contrastive learning in microscopy deblur. It also enables
cross-domain deblur knowledge transfer, facilitating extra data training and
enhancing unlabeled image deblur.

##  2 Related Work

Figure 2: Overview of MPT. MPT constructs an explicit pyramid block at each
stage. Inside the pyramid block, CSWAs constitute a coarse-to-fine pyramid,
exploring cross-scale spatial interaction for each scale. The ISCA is built
beside each CSWA to provide global channel context. Information from CSWA and
ISCA is aggregated by FEFN using the asymmetric activation mechanism.

#### Single image defocus deblurring

For learning-based deblur model, end-to-end method is widely applied [34, 65,
60, 57, 56] for its better performance and robustness [56] than methods based
on defocus map estimation [33, 47, 59, 34]. Among them, many deblurring works
have been done on cell microscopy to alleviate defocus brought by mechanical
axial shift [70] and non-coplanar cells [50], and enhance the imaging quality
of human cell [50, 72], pathology [17, 28], parasite [85], etc. Compared with
cell microscopy, surgical microscopy deblur has not been well-explored due to
the difficult acquisition of blur datasets with ground truth [18], although
defocus blur is commonly encountered in microscopic surgery [18, 93].

#### Multi-scale methods

The image pyramid in existing methods is usually built in two ways, i.e.,
explicitly stacking multi-scale feature maps in a single pyramid [51, 16, 80],
or implicitly applying multi-stage structure [7, 31, 60, 8, 56, 71]. Explicit
pyramid methods face single-level feature deficiency since the explicit
pyramid is built on downscaling feature in a single latent space [51, 16, 80].
Most of the existing microscope deblur methods [70, 40, 18, 17] adopt implicit
multi-stage design to perform aggregation on separated latent space but suffer
from inter-level feature discrepancy. The proposed MPT in this work addresses
these drawbacks by building multiple explicit pyramids with CSWA, ISCA, and
FEFN on each feature level, achieving cross-scale feature aggregation with
enlarged receptive field.

#### Contrastive learning

Contrastive learning has been widely applied in low-level tasks [27, 92, 75,
14, 3, 94, 25]. They construct contrastive pairs on the feature space that
take clean and corrupted images as positive and negative pairs, respectively
[76, 37]. Some works have leveraged contrastive frequency information [94] and
extracted frequency representations by wavelet transformation [86, 3, 75].
Following the fact that sharp and blurry images have similar low-frequency
components but differ significantly in the high-frequency part [75], the idea
of comparing different frequency bands separately is adopted in [90] by
applying $\mathcal{L}_{1}$ loss directly to the frequency bands in contrastive
regularization. In this paper, the proposed EFCR adopts basic CR and extended
CR to encourage learning of latent deblur signals and transferring of cross-
domain deblur information, thus addressing the feature deficiency problem.

##  3 Method

The proposed framework consists of MPT and EFCR to address the two outstanding
problems in microscopy defocus deblur, namely longer attention span and
feature deficiency.

###  3.1 Multi-pyramid Transformer (MPT)

MPT builds multiple explicit pyramids on each feature level, thus avoiding
single-level feature deficiency and inter-level feature discrepancy [51, 80,
70, 18]. As shown in Fig. 2, the proposed MPT follows a U-shaped structure
[58, 74, 22]. The blur image $I_{in}$ with the size of $H\times W$ first gets
input feature projection $I_{1}\in\mathbb{R}^{H\times W\times C}$ through a
convolution. Then, the feature goes through the network with seven pyramid
blocks followed by re-sampling using pixel shuffles [64], and finally projects
back to the image with a convolution. The shortcut connection is built between
each encoder and decoder stage by element-wise addition.

#### Pyramid block

The pyramid block $i$ receives
$I_{i}\in\mathbb{R}^{\frac{H}{2^{i}}\times\frac{W}{2^{i}}\times 2^{i}C}$
($i\in\\{1,2,3,4\\}$), and outputs the map $\hat{I_{i}}$ in the same size.
There are $N_{i}$ sub-blocks in pyramid block $i$. Each of them handles a
local scale $S_{i}^{j}$ ($j\in\\{1,2,...,N_{i}\\}$,
$S_{i}^{j}\in\\{\frac{1}{8},\frac{1}{4},\frac{1}{2},1\\}$). Multiple stacked
sub-blocks build the pyramid in a coarse-to-fine manner, i.e., $S_{i}^{k}\leq
S_{i}^{k+1}$, $1\leq k\textless N_{i}$. This design achieves progressive
multi-scale feature aggregation and ensures the full exploration of each
scale. In practice, $N_{i}$ is set to be an even number. The sub-block in the
even level adopts common cyclic window shifting strategy [43] to gain cross-
window interaction.

#### Cross-scale window attention (CSWA)

CSWA captures the long-range interaction by modeling the attention between
windows from the local-scale map and the down-scale map. The layer normalized
[2] feature $x\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times C^{\prime}}$
first passes through a downsampling module to get $x_{S_{i}^{j}}$ with the
size of ${H^{\prime}S_{i}^{j}\times{W^{\prime}}{S_{i}^{j}}\times C^{\prime}}$.
A strided average pooling followed by a linear projection with a shortcut is
adopted in the downsampling module. The $3\times 3$ neighboring padding
convolution (NPConv) is proposed to generate $Q,K,V$ projection
($Q\in\frac{H^{\prime}W^{\prime}}{M^{2}}\times M^{2}\times C^{\prime}$,
$K,V\in\frac{H^{\prime}W^{\prime}({S_{i}^{j}})^{2}}{M^{2}}\times M^{2}\times
C^{\prime}$) with inductive bias[77, 20], where $M$ is the patch width. It
pads a patch with its neighborhood pixels, providing the isolated edge pixels
with neighboring information [69] (all $3\times 3$ convolutions adopt bias-
free grouped convolution by default with group size equal to the feature
dimension). To get the cross-scale spatial attention map, the cross-scale
multiplication ($\circledast$) is introduced as a one-to-many strategy, where
each patch in $K$ is multiplied with
$\frac{1}{S_{i}^{j}}\times\frac{1}{S_{i}^{j}}$ patches in the corresponding
location of $Q$, as illustrated in Fig. 2. This operation provides an $M\times
M$ local patch in $Q$ with interaction with a $M\times M$ patch in $K$ whose
information comes from a $\frac{M}{S_{i}^{j}}\times\frac{M}{S_{i}^{j}}$ region
in the original map. Although the downscaled map loses information, it can
still be highly instructive, since attention distributions of different scales
are highly consistent [36]. It also leverages the pool of sharper patches
generated by downscaling which serve as priors for deblurring [95, 53]. It
makes receptive field quadratically enlarged by $(S_{i}^{j})^{2}$ times while
keeping the computational complexity $O(M^{2}H^{\prime}W^{\prime}C^{\prime})$
unchanged as the vanilla local window attention [43]. A similar strategy is
adopted in the multiplication between attention map and $V$. The self-
attention in CSWA for a local window in the size of $M^{2}\times C$ can be
defined as:

$Attention_{1}(q,k,v)=Softmax(qk^{T}/\sqrt{d}+B)v,\vspace{-2mm}$ (1)

where $q,k,v\in\mathbb{R}^{M^{2}\times d}$, and $B$ is the relative positional
encoding [43]. In practice, we implement the multi-head self-attention [68] by
concatenating the result of $h$ parallelly calculated attention. The output
$x_{1}$ is then obtained by a linear projection with a shortcut.

#### Intra-scale channel attention (ISCA)

ISCA handles a single local scale feature $x\in\mathbb{R}^{H^{\prime}\times
W^{\prime}\times C^{\prime}}$ and generates cross-channel interaction with
encoded global context [84]. By applying $1\times 1$ convolutions followed by
$3\times 3$ convolutions, projections
$Q,K,V\in\mathbb{R}^{{H^{\prime}W^{\prime}\times C^{\prime}}}$ are generated
from layer normalized $x$, introducing convolutional inductive bias and
extracting cross-channel information in both point-wise and spatial-wise
manners. The intra-scale channel attention map is then calculated by
multiplying $Q$ with $K$. The self-attention in ISCA can be formulated as:

$Attention_{2}(Q,K,V)=Softmax(QK^{T})V\vspace{-2mm}$ (2)

Similar to CSWA, ISCA implements the multi-head self-attention [68]. The self-
attention result finally goes through a linear projection with a shortcut and
gets $x_{2}$, similar to CSWA.

#### Feature-enhancing feed-forward network (FEFN)

The FEFN aggregates the spatial-wise feature $x_{1}$ with the channel-wise
context $x_{2}$. The input features are first projected to $x_{1}^{\prime}$
and $x_{2}^{\prime}$ in the size of $H^{\prime}\times W^{\prime}\times\alpha
C^{\prime}$, where $\alpha$ is the expansion ratio. Instead of combining
$x_{1}^{\prime}$ and $x_{2}^{\prime}$ by simply adding them together [36],
FEFN adopts an asymmetrical activation mechanism with GELU [26], where $x_{1}$
is element-wisely multiplied by GELU activated $x_{2}$. The FEFN can be
formulated as

$\hat{x}=W_{p}(GELU(x_{2}^{\prime})\odot x_{1}^{\prime})+x_{1},\vspace{-2mm}$
(3)

where $\hat{x}\in H^{\prime}\times W^{\prime}\times C^{\prime}$, and $W_{p}$
refers to linear projection. Compared with the regular FN [12], this
asymmetrical operation allows the spatial information from $x_{1}$ to be
guided by the non-linearly activated signal from the channel context in
$x_{2}$, offering $x_{1}$ an extra global view in terms of feature channels.

###  3.2 Extended Frequency Contrastive Regularization (EFCR)

The proposed EFCR contains basic CR and extended CR to explore latent deblur
guidance beyond the pixel-wise constraints.

#### Constructions of contrastive pairs

Given a training pair $i$ with ground truth $I_{i}^{gt}$, blur input
$I_{i}^{in}$, and deblurred output $I_{i}^{out}$, the Haar wavelet
transformation [86] decouples the samples into low-low (LL), low-high (LH),
high-low (HL), and high-high (HH) bands. For simplicity, here we define
$f^{h}(\cdot)$ as the operator decoupling and concatenating high-frequency
bands (LH, HL, HH), and $f^{l}(\cdot)$ as the operator for low-frequency band
(LL). For basic CR, the frequency bands are directly taken as contrastive
pairs. The positive and negative basic CR $\mathcal{L}_{i}^{+}$ and
$\mathcal{L}_{i}^{-}$ are given by:

(4) (5)

Both bands are included in $\mathcal{L}_{i}^{+}$, since both high and low
frequencies of $I_{i}^{out}$ are expected to be pulled closer to $I_{i}^{gt}$.
Only high frequency is taken for $\mathcal{L}_{i}^{-}$ to push the
$f^{h}(I_{i}^{out})$ away from $f^{h}(I_{i}^{in})$ as blur degradation mainly
happens in the high-frequency parts [8, 42].

The extended CR enforces the model to learn latent information from degraded
high-frequency components beyond the pixel-wise constraint. Based on the idea
that a model trained with synthetic blurred images can deblur natural blurry
images in the dataset [18], the blurred image $B_{i}^{in}$ is generated by
applying random Gaussian blur (kernel size in $\\{3,5,7\\}$) on $I_{i}^{in}$,
followed by calculating its deblurred result $B_{i}^{out}$. The extended CR
$\mathcal{L}_{i}^{ext}$ based on extended training pair
($B_{i}^{in},B_{i}^{out}$) is then formulated as:

$\mathcal{L}_{i}^{ext}=\frac{{\parallel}f^{h}(B_{i}^{out})-f^{h}(B_{i}^{in}){\parallel}_{1}}{{\parallel}f^{h}(I_{i}^{in})-f^{h}(B_{i}^{in}){\parallel}_{1}}.\vspace{-2mm}$
(6)

The $\mathcal{L}_{i}^{ext}$ is derived as a relative loss term by normalizing
with high-frequency $\mathcal{L}_{1}$ distance between $I_{i}^{in}$ and its
blurred counterpart $B_{i}^{in}$ to alleviate the disturbance from the
gradient distribution of reblurred image [11].

The overall optimization objective $\mathcal{L}$ with the proposed EFCR
$\mathcal{L}_{CR}$ is given by:

$\mathcal{L}=\mathcal{L}_{1}+\beta\mathcal{L}_{CR}=\mathcal{L}_{1}+\beta\frac{1}{n}\sum_{i=1}^{n}\frac{\mathcal{L}_{i}^{+}}{\mathcal{L}_{i}^{-}+\mathcal{L}_{i}^{ext}},\vspace{-2mm}$
(7)

where $\mathcal{L}_{1}$ is the supervised pixel loss, $n$ is the number of
samples, and $\beta$ is the scaling factor.

#### Knowledge transfer from extra data

Defocus blur mainly causes high-frequency degradation [8, 42], implying that
the high-frequency part can provide informative cross-domain deblur guidance.
EFCR with extra data (denoted by EFCRex) constructs contrastive pairs on high-
frequency components. Given an extra training pair
$\\{I_{i}^{gt^{\prime}},I_{i}^{in^{\prime}},I_{i}^{out^{\prime}}\\}$ from
external dataset and corresponding extended samples
$\\{B_{i}^{in^{\prime}},B_{i}^{out^{\prime}}\\}$, EFCRex with
$\\{\mathcal{L}_{i}^{+^{\prime}},\mathcal{L}_{i}^{-^{\prime}},\mathcal{L}_{i}^{ext^{\prime}}\\}$
can be formulated as:

$\displaystyle\mathcal{L}_{i}^{+^{\prime}}={\parallel}f^{h}(I_{i}^{out^{\prime}})-f^{h}(I_{i}^{gt^{\prime}}){\parallel}_{1},$
(8)
$\displaystyle\mathcal{L}_{i}^{-^{\prime}}={\parallel}f^{h}(I_{i}^{out^{\prime}})-f^{h}(I_{i}^{in^{\prime}}){\parallel}_{1},$
(9)
$\displaystyle\mathcal{L}_{i}^{ext^{\prime}}=\frac{{\parallel}f^{h}(B_{i}^{out^{\prime}})-f^{h}(B_{i}^{in^{\prime}}){\parallel}_{1}}{{\parallel}f^{h}(I_{i}^{in^{\prime}})-f^{h}(B_{i}^{in^{\prime}}){\parallel}_{1}}.\vspace{-2mm}$
(10)

The overall optimization objective follows a similar pattern with Eq. 7, where
the supervised training on the testing dataset ($\mathcal{L}_{1}$) proceeds
simultaneously with EFCRex ($\mathcal{L}_{CR}$) to transfer knowledge without
interfering with supervised training.

EFCRex facilitates two important applications. One is to transfer rich deblur
signals from a real-world blur dataset to microscopy deblur tasks, in which
EFCRex is composed by
$\\{\mathcal{L}_{i}^{+^{\prime}},\mathcal{L}_{i}^{-^{\prime}},\mathcal{L}_{i}^{ext^{\prime}}\\}$.
Another is to learn latent deblur direction from an unlabeled microscopy
dataset thus enhancing the deblur performance, where the model is trained on a
labeled dataset with an unlabeled microscopy dataset as the extra data. EFCRex
here is reformulated as
$\\{\mathcal{L}_{i}^{+},\mathcal{L}_{i}^{-},\mathcal{L}_{i}^{ext^{\prime}}\\}$.

##  4 Experiments

Method | BBBC006w1 [44] | BBBC006w2 [44] | 3DHistech [17] | CaDISBlur
---|---|---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM $\uparrow$ | LPIPS$\downarrow$
DRBNet [60] | 32.83 | 0.737 | 0.381 | 26.66 | 0.589 | 0.458 | 32.83 | 0.853 | 0.131 | 42.54 | 0.776 | 0.243
GKMNet [56] | 34.41 | 0.887 | 0.218 | 29.32 | 0.721 | 0.296 | 33.42 | 0.852 | 0.130 | 44.27 | 0.860 | 0.178
MIMO-UNet [7] | 32.73 | 0.725 | 0.412 | 26.90 | 0.601 | 0.457 | 32.40 | 0.837 | 0.169 | 43.36 | 0.823 | 0.197
MSSNet [31] | 34.01 | 0.790 | 0.289 | 28.68 | 0.736 | 0.361 | 33.09 | 0.870 | 0.126 | 44.09 | 0.871 | 0.160
SwinIR [38] | 33.90 | 0.801 | 0.274 | 27.61 | 0.696 | 0.403 | 32.57 | 0.841 | 0.136 | 41.83 | 0.710 | 0.349
PANet [51] | 34.45 | 0.890 | 0.230 | 29.07 | 0.743 | 0.290 | 33.24 | 0.869 | 0.129 | 44.49 | 0.917 | 0.134
GRL [36] | 34.76 | 0.907 | 0.129 | 29.39 | 0.786 | 0.249 | 33.49 | 0.878 | 0.120 | 44.86 | 0.960 | 0.087
Restormer [84] | 34.79 | 0.904 | 0.135 | 29.78 | 0.801 | 0.241 | 33.46 | 0.880 | 0.125 | 44.85 | 0.941 | 0.101
MPT | 34.89 | 0.912 | 0.127 | 29.85 | 0.813 | 0.237 | 33.55 | 0.881 | 0.121 | 44.98 | 0.962 | 0.087
MPT+EFCR | 34.96 | 0.917 | 0.119 | 29.89 | 0.820 | 0.230 | 33.58 | 0.887 | 0.119 | 45.09 | 0.969 | 0.082
MPT+EFCRex | 35.16 | 0.935 | 0.083 | 30.11 | 0.829 | 0.205 | 33.63 | 0.892 | 0.116 | 45.25 | 0.971 | 0.077

Table 1: Quantitative evaluation on microscopy deblur. The experiments on the sub-set $w1$ (stained by Hoechst to show nuclei structure) and $w2$ (stained by phalloidin to show cell structure) of BBBC006 [44] are conducted separately. Except for the methods using EFCR, the methods with the best and second best performance are noted in red and blue colors. Figure 3: Qualitative evaluation on microscopy deblur. Our method achieves the best restoration of different types of defocus blur.  Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---
KPAC [65] | 25.24 | 0.774 | 0.226
IFAN [34] | 25.37 | 0.789 | 0.217
DRBNet [60] | 25.47 | 0.787 | 0.246
GKMNet [56] | 25.47 | 0.789 | 0.219
Restormer [84] | 25.98 | 0.811 | 0.178
NRKNet [57] | 26.11 | 0.810 | 0.210
GRL [36] | 26.18 | 0.822 | 0.168
MPT | 26.21 | 0.826 | 0.175
MPT+EFCR | 26.23 | 0.829 | 0.172
MPT+EFCRex | 26.27 | 0.831 | 0.161
Table 2: Quantitative evaluation on real-world single-image deblur with DPDD
[1].

###  4.1 Datasets and Implementation

Extensive experiments are carried out on various real-world and microscopy
datasets, including five labeled datasets: DPDD [1], LFDOF [59], BBBC006 [44],
3DHistech [17], CaDISBlur, and three unlabeled datasets: CUHK [63], WNLO [17],
CataBlur. The LFDOF [59] is adopted as an extra training dataset for knowledge
transfer using EFCR, since LFDOF has substantial samples with rich information
and good cross correlation between defocused and ground truth pairs [60]. For
surgical microscopy, two new surgical microscopy deblur datasets are
presented, which are CaDISBlur and CataBlur. CaDISBlur is synthesized using
images from a high-quality dataset CaDIS [19] by a novel realistic blur
simulation method, in which the instruments and anatomies in the surgery scene
are blurred respectively leveraging the object segmentation mask in CaDIS to
simulate different focal planes. CataBlur is a new surgical microscope defocus
blur dataset, including 4614 defocus images collected from 12 cataract
surgeries, for evaluation on real surgical defocus blur. More details about
the datasets, training settings, and proposed blur synthesizing method are
provided in the supplementary material.

The proposed framework employs the same structure in all tests as follows. The
MPT adopts a 4-stage design as shown in Fig. 2, with $[6,6,6,6]$ sub-blocks,
$[40,80,160,320]$ feature dimensions, and $[1,2,4,8]$ attention heads. The
scale set of each pyramid block is set as
$S_{1}=[\frac{1}{8},\frac{1}{8},\frac{1}{4},\frac{1}{4},1,1]$,
$S_{2}=[\frac{1}{4},\frac{1}{4},\frac{1}{2},\frac{1}{2},1,1]$,
$S_{3}=S_{4}=[\frac{1}{2},\frac{1}{2},\frac{1}{2},\frac{1}{2},1,1]$. The
expansion ratio $\alpha$ in FEFN is set to 2.6, and the scaling factor $\beta$
in EFCR is set to 1e-5. The method is implemented using PyTorch and trained
with AdamW optimizer [46] ($\beta_{1}=0.9$, $\beta_{2}=0.999$, weight decay is
1e-4) for $3\times 10^{5}$ iterations on NVIDIA A800 GPUs. The initial
learning rate is set to 1e-4 and gradually decreases to 1e-6 by cosine
annealing [45]. The batch size is set to 8 with training patches in the size
of 256$\times$256 augmented with random scaling, and horizontal and vertical
flips. Three implementations are included, which are MPT, MPT with EFCR, and
MPT with EFCR using LFDOF as extra data (noted as EFCRex). Then the result is
reported in three metrics, including Peak Signal-to-Noise Ratio (PSNR),
Structural Similarity (SSIM) [73], and Learned Perceptual Image Patch
Similarity (LPIPS) [87].

###  4.2 Comparison and Analysis

#### Evaluation on supervised deblur

The evaluation on cell microscopy deblur and surgical microscopy deblur is
conducted on three microscopy datasets covering a wide range of state-of-the-
art defocus deblur methods and image restoration methods. Real-world deblur
evaluation is also conducted on DPDD [1] to further prove the generalizability
and universality. The result is shown in Tab. 1 and Tab. 2. The proposed
framework demonstrates remarkable performance on all microscopy datasets and
real-world dataset, showing the superiority of the proposed MPT structure and
EFCR training strategy. Compared with Restormer [84], which achieves the
second-best performance on BBBC006 [44], MPT (76 FLOPs, 19.80 M) outperforms
Restormer (141 FLOPs, 26.12 M) by 0.10 dB and 0.07 dB regarding PSNR while
saving 46% FLOPs (for a 256$\times$256 input) and 24% parameters. GRL [36]
achieves the second-best performance on 3DHistech [17] and CaDISBlur, but it
takes 1230 FLOPs for a 256$\times$256 input that is 17$\times$ larger than
ours. Compared to those methods that adopt multi-scale or pyramid design,
including MSSNet [31], MIMO-UNet [7], and PANet [51], our method surpasses
them significantly in all tests, proving the superiority of our multi-pyramid
structure. SwinIR [38] adopts the original local window attention [43], which
is hindered by the limited receptive field and fails to build long-range
interaction. The visualizations shown in Fig. 3 prove that our method achieves
the best restoration of fine details against strong defocus blur, especially
for the miniature cell shape and complex cell structure, as well as precise
features of surgical anatomies. For real-world deblur on DPDD [1], our method
achieves state-of-the-art performance in terms of SSIM and PSNR. Although the
LPIPS is slightly lower than the second-best method, it still surpasses the
second-best method NRKNet [57] by 17%. It shows that our model is universally
applicable to different types of images. Visualization of deblurring on DPDD
is provided in supplementary materials.

For MPT trained with EFCR, the performance is improved by learning latent
deblur information. By further applying EFCRex to learn cross-domain deblur
guidance, the deblur performance is significantly enhanced in all four
microscopy datasets, proving that deblurring benefits from cross-domain
knowledge, despite the significant feature discrepancy between real-world
extra data and the microscope images. Improvements are also observed in SSIM
and LPIPS, showing that EFCRex enhances deblurring from the perspective of
human visual system, which is of great significance for clinical application.
Visualizations in Fig. 3 draw a similar conclusion that the model with EFCRex
can restore the fine details more precisely. Real-world deblur can also
benefit from EFCR as all three metrics improve with integration of EFCR or
EFCRex. Further discussion in Sec. 4.4 shows the superiority of this proposed
training diagram against simply pretraining and fine-tuning.

#### Evaluation on unsupervised deblur

The deblur experiments on unlabeled datasets are conducted to qualitatively
evaluate the generalizability of the model, and also to prove the
effectiveness of knowledge transfer based on the proposed EFCRex. Two
unlabeled microscopy blur datasets are involved, including WNLO [17] and
CataBlur. All methods are trained on LFDOF since the rich information in real-
world dataset benefits microscopy deblur (see proof in supplementary
materials). Unlabeled datasets are adopted as the extra data to learn cross-
domain latent information using EFCRex. The qualitative comparison is shown in
Fig. 4. Even without EFCRex, our method still shows the best generalizability
with fewer artifacts and successfully restores the detail from strong defocus
degradation. With the help of EFCRex, the artifacts are further reduced,
resulting in clearer deblurred images. Further experiments on the real-world
dataset CUHK [63] are conducted (see supplementary materials), which proves
the universality of our method.

Figure 4: Qualitative evaluation on unsupervised deblur with WNLO (top) and
CataBlur (bottom).

floatrowsep=qquad,captionskip=10pt

Table 3: Ablation studies on attention blocks with four variants, where WA
refers to the original version of window attention [38, 43]. Performance
degradation occurs in all variants.

Configuration | PSNRD | PSNRB | PSNRC
---|---|---|---
$V_{1}$ (CSWA$\times$2) | 26.10 | 34.76 | 44.83
$V_{2}$ (WA$\times$2) | 25.92 | 33.98 | 44.02
$V_{3}$ (ISCA$\times$2) | 26.01 | 34.60 | 44.71
$V_{4}$ (WA+ISCA) | 26.13 | 34.10 | 44.78
MPT (CSWA+ISCA) | 26.21 | 34.89 | 44.98

Table 4: Ablation studies on FEFN with three variants, which are symmetric
structures: concatenation ($V_{1}$) and adding ($V_{2}$) followed by GELU
activation, and reversed structure that uses the feature from CSWA for
activation ($V_{3}$)

Configuration | PSNRD | PSNRB | PSNRC
---|---|---|---
$V_{1}$ (Cat+GELU) | 26.18 | 34.86 | 44.84
$V_{2}$ (Add+GELU) | 26.03 | 34.75 | 44.87
$V_{3}$ (reversed) | 25.98 | 34.72 | 44.50
MPT (FEFN) | 26.21 | 34.89 | 44.98

Table 5: Ablation studies on EFCR. $\Delta$PSNR refers to the changes in PSNR
compared to the baseline.

Configuration |  $\Delta$PSNRD |  $\Delta$PSNRB |  $\Delta$PSNRC
---|---|---|---
MPT+$V_{1}$ | +0.01 | +0.04 | +0.05
MPT+EFCR | +0.02 | +0.07 | +0.11
Restormer+EFCR | +0.04 | +0.07 | +0.09
MPT+pretrain | +0.05 | -0.02 | +0.01
MPT+$V_{ex1}$ | +0.05 | +0.21 | +0.18
MPT+EFCRex | +0.06 | +0.27 | +0.27
Restormer+EFCRex | +0.07 | +0.19 | +0.21

###  4.3 Validation on downstream tasks

#### Cell detection on BBBC006

Defocus blur can cause failure in cell detection and segmentation [17, 6] that
essential for many biological tasks [52]. The cell segmentation is performed
using StarDist [61] on images in BBBC006 before and after deblur. The result
is reported in Tab. 6 regarding the average precision (AP) over different
intersection-of-union (IoU) thresholds, where higher AP means more cells are
successfully detected. Our deblur framework significantly improves the cell
detection performance by 19.57% (with extra data) and 18.54% (without extra
data) compared with blurry input, surpassing the improvement brought by
Restormer (16.05%) and GRL (13.50%). The visualization shown in Fig. 5 also
proves that our method achieves better restoration of the cell shape and
structure.

IoU | 0.5 | 0.7 | 0.9 | Mean AP
---|---|---|---|---
blur | 0.7010 | 0.5623 | 0.2194 | 0.4942
GRL [36] | 0.7702 | 0.6433 | 0.2691 | 0.5609
Restormer [84] | 0.7789 | 0.6703 | 0.2714 | 0.5735
MPT+EFCR | 0.7814 | 0.6791 | 0.2970 | 0.5858
MPT+EFCRex | 0.7865 | 0.6843 | 0.3019 | 0.5909
sharp | 0.8021 | 0.7192 | 0.3518 | 0.6244

Table 6: Cell detection result on deblurred BBBC006.

#### Semantic segmentation on CaDISBlur

Semantic segmentation plays an important role in surgical scene understanding
[19]. The semantic segmentation of the cataract surgical scene is conducted
using OCRNet [83] on CaDISBlur based on CaDIS [19]. Results of images with
different focal plane positions (blurry instruments or blurry anatomies, noted
as ins or ana) are reported separately in Tab. 7 regarding mean IoU (mIoU) and
pixel accuracy (PA). The deblurred images from our method lead to the best
performance in most metrics. Visualizations shown in Fig. 5 also prove the
superiority of our method.

Method | Blurry instrument | Blurry anatomies
---|---|---
mIoUins | PAins | mIoUana | PAana
blur | 0.7577 | 0.8677 | 0.7092 | 0.8194
Restormer [84] | 0.7558 | 0.8803 | 0.8149 | 0.8674
GRL [36] | 0.7606 | 0.8849 | 0.8135 | 0.8689
MPT+EFCR | 0.7610 | 0.8842 | 0.8295 | 0.8830
MPT+EFCRex | 0.7667 | 0.8859 | 0.8361 | 0.8896
sharp | 0.7733 | 0.8886 | 0.8582 | 0.9305

Table 7: Semantic segmentation result on deblurred CaDISBlur. Figure 5:
Downstream tasks result on BBBC006 (top) and CaDISBlur (bottom). Our method
leads to less false segmentation than others.

###  4.4 Ablation Studies

For ablation studies, the model variants are evaluated regarding PSNR on DPDD
[1], BBBC006w1 [44] and CaDISBlur datasets, which are denoted by PSNRD, PSNRB
and PSNRC, respectively. More ablation experiments and analyses are provided
in supplementary materials.

#### Configurations of pyramid block

Ablation studies on CSWA and ISCA are first carried out. As shown in
LABEL:tab:ablationcswa, a performance drop is observed when changing CSWA to
WA ($V_{4}$) since WA only models attention within a local window. Although
hierarchical network structure in MPT may provide WA with larger receptive
field at low-resolution stages, it still causes inferior performance than CSWA
since cross-scale interaction are not built. The situation gets worse if the
two attention blocks are all changed to WA ($V_{2}$) since the model totally
lost long-range modeling ability in both channel and spatial means.
Experiments are then carried out on variants of FEFN, as shown in
LABEL:tab:ablationfefn, which proves the superiority of the proposed
asymmetrical activation.

#### Improvements in EFCR

The result is shown in LABEL:tab:ablationefcr. $V_{1}$ and $V_{ex1}$ refer to
$\\{\mathcal{L}_{i}^{+},\mathcal{L}_{i}^{-}\\}$ and
$\\{\mathcal{L}_{i}^{+^{\prime}},\mathcal{L}_{i}^{-^{\prime}}\\}$. The
proposed EFCR and EFCRex yield significant improvements over baseline, not
only on our method but also on Restormer [84], proving the effectiveness of
the proposed training diagram. Following a similar approach in [60] to
pretrain and fine-tune (denoted as MPT+pretrain), the trained model leads to a
trivial improvement or even degradation. It further demonstrates the
superiority of our EFCRex.

##  5 Conclusion

This paper presents a unified framework to address the outstanding problems in
microscopy defocus deblur. The MPT outperforms existing multi-scale networks
by incorporating spatial-channel context from CSWA and ISCA using FEFN. The
proposed EFCR enforces the model to explore latent deblur guidance and further
learn cross-domain knowledge from the extra data, yielding significant
performance gain in both supervised and unsupervised image deblur. In the
future, larger-scale datasets, e.g. ImageNet [10], will be adopted for
knowledge transfer using EFCR, along with experiments on weakly supervised or
unsupervised learning [24] and domain adaptation [89]. Experiments on MPT
variants incorporating varied window mechanisms [78] will be carried out.

#### Acknowledgement

We would like to thank Dr. Danny Siu-Chun Ng and the Department of
Ophthalmology and Visual Sciences at the Chinese University of Hong Kong for
providing us with the cataract surgery dataset for research.

## References

  * Abuolaim and Brown [2020] Abdullah Abuolaim and Michael S. Brown.  Defocus Deblurring Using Dual-Pixel Data.  In _Computer Vision – ECCV 2020_ , pages 111–126, Cham, 2020. Springer International Publishing.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Bai and Yuan [2022] Yunpeng Bai and Chun Yuan.  Contrastive Learning in Wavelet Domain for Image Dehazing.  In _2022 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–7, 2022.  ISSN: 2161-4407.
  * Bille [2019] Josef F Bille.  High resolution imaging in microscopy and ophthalmology: new frontiers in biomedical optics.  2019\.
  * Bouget et al. [2015] David Bouget, Rodrigo Benenson, Mohamed Omran, Laurent Riffaud, Bernt Schiele, and Pierre Jannin.  Detecting Surgical Tools by Modelling Local Appearance and Global Shape.  _IEEE Transactions on Medical Imaging_ , 34(12):2603–2617, 2015.  Conference Name: IEEE Transactions on Medical Imaging.
  * Chen and Ju [2022] Xingyu Chen and Fujiao Ju.  Automatic Classification of Pollen Grain Microscope Images Using a Multi-Scale Classifier with SRGAN Deblurring.  _Applied Sciences_ , 12(14):7126, 2022.  Number: 14 Publisher: Multidisciplinary Digital Publishing Institute.
  * Cho et al. [2021] Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko.  Rethinking Coarse-To-Fine Approach in Single Image Deblurring.  pages 4641–4650, 2021.
  * Cui et al. [2023] Yuning Cui, Yi Tao, Wenqi Ren, and Alois Knoll.  Dual-Domain Attention for Image Deblurring.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 37(1):479–487, 2023.  Number: 1.
  * Cuny et al. [2022] Andreas P. Cuny, Fabian P. Schlottmann, Jennifer C. Ewald, Serge Pelet, and Kurt M. Schmoller.  Live cell microscopy: From image to insight.  _Biophysics Reviews_ , 3(2):021302, 2022.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Dong et al. [2019] Han Dong, Aodong Shen, Youyong Kong, Yu Shen, and Huazhong Shu.  No-Reference Defocused Image Quality Assessment Based on Human Visual System.  In _2019 IEEE International Conference on Signal, Information and Data Processing (ICSIDP)_ , pages 1–6, 2019.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2021.  arXiv:2010.11929 [cs].
  * Fang et al. [2022] Rui Fang, Yang-Fan Yu, En-Jie Li, Ning-Xin Lv, Zhao-Chuan Liu, Hong-Gang Zhou, and Xu-Dong Song.  Global, regional, national burden and gender disparity of cataract: findings from the global burden of disease study 2019.  _BMC Public Health_ , 22(1):2068, 2022.
  * Feng et al. [2023] Xin Feng, Yifeng Xu, Guangming Lu, and Wenjie Pei.  Hierarchical Contrastive Learning for Pattern-Generalizable Image Corruption Detection.  pages 12076–12085, 2023.
  * García Calderón et al. [2007] Manuel García Calderón, Daniel Torres Lagares, Carmen Calles Vázquez, Jesús Usón Gargallo, and José Luis Gutiérrez Pérez.  The application of microscopic surgery in dentistry.  _Medicina Oral, Patología Oral y Cirugía Bucal (Internet)_ , 12(4):311–316, 2007.
  * Gendy et al. [2022] Garas Gendy, Nabil Sabor, Jingchao Hou, and Guanghui He.  Balanced Spatial Feature Distillation and Pyramid Attention Network for Lightweight Image Super-resolution.  _Neurocomputing_ , 509:157–166, 2022.
  * Geng et al. [2022] Xiebo Geng, Xiuli Liu, Shenghua Cheng, and Shaoqun Zeng.  Cervical cytopathology image refocusing via multi-scale attention features and domain normalization.  _Medical Image Analysis_ , 81:102566, 2022.
  * Ghamsarian et al. [2020] Negin Ghamsarian, Mario Taschwer, and Klaus Schoeffmann.  Deblurring Cataract Surgery Videos Using a Multi-Scale Deconvolutional Neural Network.  In _2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI)_ , pages 872–876, 2020.  out of focus blurry ISSN: 1945-8452.
  * Grammatikopoulou et al. [2022] Maria Grammatikopoulou, Evangello Flouty, Abdolrahim Kadkhodamohammadi, Gwenol’e Quellec, Andre Chow, Jean Nehme, Imanol Luengo, and Danail Stoyanov.  CaDIS: Cataract Dataset for Image Segmentation, 2022.  arXiv:1906.11586 [cs].
  * Guo et al. [2022] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu.  CMT: Convolutional Neural Networks Meet Vision Transformers.  pages 12175–12185, 2022.
  * Han et al. [2023] Xiaotong Han, Jiaqing Zhang, Zhenzhen Liu, Xuhua Tan, Guangming Jin, Mingguang He, Lixia Luo, and Yizhi Liu.  Real-world visual outcomes of cataract surgery based on population-based studies: a systematic review.  _British Journal of Ophthalmology_ , 107(8):1056–1065, 2023.
  * He et al. [2023a] Chunming He, Chengyu Fang, Yulun Zhang, Kai Li, Longxiang Tang, Chenyu You, Fengyang Xiao, Zhenhua Guo, and Xiu Li.  Reti-diff: Illumination degradation image restoration with retinex-based latent diffusion model.  2023a.
  * He et al. [2023b] Chunming He, Kai Li, Yachao Zhang, Longxiang Tang, Yulun Zhang, Zhenhua Guo, and Xiu Li.  Camouflaged object detection with feature decomposition and edge reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22046–22055, 2023b.
  * He et al. [2024a] Chunming He, Kai Li, Yachao Zhang, Guoxia Xu, Longxiang Tang, Yulun Zhang, Zhenhua Guo, and Xiu Li.  Weakly-supervised concealed object segmentation with sam-based pseudo labeling and multi-scale feature grouping.  _Advances in Neural Information Processing Systems_ , 36, 2024a.
  * He et al. [2024b] Chunming He, Kai Li, Yachao Zhang, Yulun Zhang, Zhenhua Guo, Xiu Li, Martin Danelljan, and Fisher Yu.  Strategic preys make acute predators: Enhancing camouflaged object detectors by generating camouflaged objects.  2024b.
  * Hendrycks and Gimpel [2016] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Jan and Tang [2022] Feng-Kai Jan and Chih-Wei Tang.  Contrastive Learning Aided Single Image Deblurring.  In _2022 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia)_ , pages 1–3, 2022.
  * Jiang et al. [2020] Cheng Jiang, Jun Liao, Pei Dong, Zhaoxuan Ma, De Cai, Guoan Zheng, Yueping Liu, Hong Bu, and Jianhua Yao.  Blind deblurring for microscopic pathology images using deep learning networks, 2020.  arXiv:2011.11879 [cs, eess].
  * Kalavakonda et al. [2019] Niveditha Kalavakonda, Blake Hannaford, Zeeshan Qazi, and Laligam Sekhar.  Autonomous neurosurgical instrument segmentation using end-to-end learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 0–0, 2019.
  * Keaton et al. [2023] Matthew R Keaton, Ram J Zaveri, and Gianfranco Doretto.  Celltranspose: Few-shot domain adaptation for cellular instance segmentation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 455–466, 2023.
  * Kim et al. [2023] Kiyeon Kim, Seungyong Lee, and Sunghyun Cho.  MSSNet: Multi-Scale-Stage Network for Single Image Deblurring.  In _Computer Vision – ECCV 2022 Workshops_ , pages 524–539, Cham, 2023. Springer Nature Switzerland.
  * Kornilova et al. [2021] Anatasiia Kornilova, Mikhail Salnikov, Olga Novitskaya, Maria Begicheva, Egor Sevriugov, Kirill Shcherbakov, Valeriya Pronina, and Dmitry V. Dylov.  Deep Learning Framework For Mobile Microscopy.  In _2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)_ , pages 324–328, 2021.  ISSN: 1945-8452.
  * Lee et al. [2019] Junyong Lee, Sungkil Lee, Sunghyun Cho, and Seungyong Lee.  Deep Defocus Map Estimation Using Domain Adaptation.  pages 12222–12230, 2019.  DMENet+SYNDOF.
  * Lee et al. [2021] Junyong Lee, Hyeongseok Son, Jaesung Rim, Sunghyun Cho, and Seungyong Lee.  Iterative Filter Adaptive Network for Single Image Defocus Deblurring.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2034–2042, Nashville, TN, USA, 2021. IEEE.
  * Li et al. [2021] Chen Li, Adele Moatti, Xuying Zhang, H Troy Ghashghaei, and Alon Greenbaum.  Deep learning-based autofocus method enhances image quality in light-sheet fluorescence microscopy.  _Biomedical Optics Express_ , 12(8):5214–5226, 2021.
  * Li et al. [2023] Yawei Li, Yuchen Fan, Xiaoyu Xiang, Denis Demandolx, Rakesh Ranjan, Radu Timofte, and Luc Van Gool.  Efficient and Explicit Modelling of Image Hierarchies for Image Restoration.  pages 18278–18289, 2023.
  * Liang et al. [2022] Dong Liang, Ling Li, Mingqiang Wei, Shuo Yang, Liyan Zhang, Wenhan Yang, Yun Du, and Huiyu Zhou.  Semantically contrastive learning for low-light image enhancement.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 1555–1563, 2022.
  * Liang et al. [2021] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.  SwinIR: Image Restoration Using Swin Transformer, 2021.  arXiv:2108.10257 [cs, eess].
  * Lin et al. [2019] Wei Lin, Dongping Wang, Yunlong Meng, and Shih-Chi Chen.  Multi-focus microscope with hilo algorithm for fast 3-d fluorescent imaging.  _PloS one_ , 14(9):e0222729, 2019.
  * Liu et al. [2022] Gaosheng Liu, Huanjing Yue, and Jingyu Yang.  A coarse-to-fine convolutional neural network for light field angular super-resolution.  In _CAAI International Conference on Artificial Intelligence_ , pages 268–279. Springer, 2022.
  * Liu et al. [2020a] Jiahao Liu, Xiaoshuai Huang, Liangyi Chen, and Shan Tan.  Deep learning–enhanced fluorescence microscopy via degeneration decoupling.  _Optics Express_ , 28(10):14859–14873, 2020a.  Publisher: Optica Publishing Group.
  * Liu et al. [2020b] Keng-Hao Liu, Chia-Hung Yeh, Juh-Wei Chung, and Chuan-Yu Chang.  A Motion Deblur Method Based on Multi-Scale High Frequency Residual Image Learning.  _IEEE Access_ , 8:66025–66036, 2020b.  Conference Name: IEEE Access.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, 2021.  arXiv:2103.14030 [cs].
  * Ljosa et al. [2012] Vebjorn Ljosa, Katherine L. Sokolnicki, and Anne E. Carpenter.  Annotated high-throughput microscopy image sets for validation.  _Nature Methods_ , 9(7):637–637, 2012.  Number: 7 Publisher: Nature Publishing Group.
  * Loshchilov and Hutter [2016] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ , 2016.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Ma et al. [2022] Haoyu Ma, Shaojun Liu, Qingmin Liao, Juncheng Zhang, and Jing-Hao Xue.  Defocus Image Deblurring Network With Defocus Map Estimation as Auxiliary Task.  _IEEE Transactions on Image Processing_ , 31:216–226, 2022.  Conference Name: IEEE Transactions on Image Processing.
  * Ma and Fei [2021] Ling Ma and Baowei Fei.  Comprehensive review of surgical microscopes: technology development and medical applications.  _Journal of Biomedical Optics_ , 26(1):010901, 2021.  Publisher: SPIE.
  * Masters [2008] Barry R Masters.  History of the optical microscope in cell biology and medicine.  _eLS_ , 2008.
  * Mazilu et al. [2023] Ioana Mazilu, Shunxin Wang, Sven Dummer, Raymond Veldhuis, Christoph Brune, and Nicola Strisciuglio.  Defocus Blur Synthesis and Deblurring via Interpolation and Extrapolation in Latent Space.  In _Computer Analysis of Images and Patterns_ , pages 201–211, Cham, 2023. Springer Nature Switzerland.
  * Mei et al. [2023] Yiqun Mei, Yuchen Fan, Yulun Zhang, Jiahui Yu, Yuqian Zhou, Ding Liu, Yun Fu, Thomas S. Huang, and Humphrey Shi.  Pyramid Attention Network for Image Restoration.  _International Journal of Computer Vision_ , 2023.
  * Meijering [2012] Erik Meijering.  Cell segmentation: 50 years down the road [life sciences].  _IEEE signal processing magazine_ , 29(5):140–145, 2012.
  * Michaeli and Irani [2014] Tomer Michaeli and Michal Irani.  Blind Deblurring Using Internal Patch Recurrence.  In _Computer Vision – ECCV 2014_ , pages 783–798, Cham, 2014. Springer International Publishing.
  * Pachade et al. [2021] Samiksha Pachade, Prasanna Porwal, Dhanshree Thulkar, Manesh Kokare, Girish Deshmukh, Vivek Sahasrabuddhe, Luca Giancardo, Gwenolé Quellec, and Fabrice Mériaudeau.  Retinal Fundus Multi-Disease Image Dataset (RFMiD): A Dataset for Multi-Disease Detection Research.  _Data_ , 6(2):14, 2021.  Number: 2 Publisher: Multidisciplinary Digital Publishing Institute.
  * Pinkard et al. [2019] Henry Pinkard, Zachary Phillips, Arman Babakhani, Daniel A Fletcher, and Laura Waller.  Deep learning for single-shot autofocus microscopy.  _Optica_ , 6(6):794–797, 2019.
  * Quan et al. [2021] Yuhui Quan, Zicong Wu, and Hui Ji.  Gaussian Kernel Mixture Network for Single Image Defocus Deblurring.  In _Advances in Neural Information Processing Systems_ , pages 20812–20824. Curran Associates, Inc., 2021.
  * Quan et al. [2023] Yuhui Quan, Zicong Wu, and Hui Ji.  Neumann Network With Recursive Kernels for Single Image Defocus Deblurring.  pages 5754–5763, 2023.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  In _Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015_ , pages 234–241, Cham, 2015. Springer International Publishing.
  * Ruan et al. [2021] Lingyan Ruan, Bin Chen, Jizhou Li, and Miu-Ling Lam.  AIFNet: All-in-Focus Image Restoration Network Using a Light Field-Based Dataset.  _IEEE Transactions on Computational Imaging_ , 7:675–688, 2021.  Conference Name: IEEE Transactions on Computational Imaging.
  * Ruan et al. [2022] Lingyan Ruan, Bin Chen, Jizhou Li, and Miuling Lam.  Learning to Deblur Using Light Field Generated and Real Defocus Images.  pages 16304–16313, 2022.
  * Schmidt et al. [2018] Uwe Schmidt, Martin Weigert, Coleman Broaddus, and Gene Myers.  Cell Detection with Star-Convex Polygons.  In _Medical Image Computing and Computer Assisted Intervention – MICCAI 2018_ , pages 265–273, Cham, 2018. Springer International Publishing.
  * Schoeffmann et al. [2018] Klaus Schoeffmann, Mario Taschwer, Stephanie Sarny, Bernd Münzer, Manfred Jürgen Primus, and Doris Putzgruber.  Cataract-101: video dataset of 101 cataract surgeries.  In _Proceedings of the 9th ACM Multimedia Systems Conference_ , pages 421–425, New York, NY, USA, 2018. Association for Computing Machinery.
  * Shi et al. [2014] Jianping Shi, Li Xu, and Jiaya Jia.  Discriminative Blur Detection Features.  pages 2965–2972, 2014.
  * Shi et al. [2016] Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network.  pages 1874–1883, 2016.
  * Son et al. [2021] Hyeongseok Son, Junyong Lee, Sunghyun Cho, and Seungyong Lee.  Single Image Defocus Deblurring Using Kernel-Sharing Parallel Atrous Convolutions.  pages 2642–2650, 2021.
  * Trikha et al. [2013] Sameer Trikha, Andrew Michael John Turnbull, RJ Morris, David F Anderson, and Parwez Hossain.  The journey to femtosecond laser-assisted cataract surgery: new beginnings or a false dawn?  _Eye_ , 27(4):461–473, 2013.
  * Trukhova et al. [2022] Anna Trukhova, Marina Pavlova, Olga Sinitsyna, and Igor Yaminsky.  Microlens-assisted microscopy for biology and medicine.  _Journal of Biophotonics_ , 15(9):e202200078, 2022.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is All you Need.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2017.
  * Vaswani et al. [2021] Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens.  Scaling Local Self-Attention for Parameter Efficient Visual Backbones.  pages 12894–12904, 2021.
  * Wang and Han [2023] Jiahe Wang and Boran Han.  Defocus Deblur Microscopy via Head-to-Tail Cross-scale Fusion, 2023.  arXiv:2201.02876 [cs, eess].
  * Wang et al. [2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction Without Convolutions.  pages 568–578, 2021.
  * Wang et al. [2023] Yanqi Wang, Zheng Xu, Yifan Yang, Xiaodong Wang, Jiaheng He, Tongqun Ren, and Junshan Liu.  Deblurring microscopic image by integrated convolutional neural network.  _Precision Engineering_ , 82:44–51, 2023.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * Wang et al. [2022] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li.  Uformer: A General U-Shaped Transformer for Image Restoration.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 17662–17672, New Orleans, LA, USA, 2022. IEEE.
  * Wu et al. [2023] Gang Wu, Junjun Jiang, and Xianming Liu.  A Practical Contrastive Learning Framework for Single-Image Super-Resolution.  _IEEE Transactions on Neural Networks and Learning Systems_ , pages 1–12, 2023.  Conference Name: IEEE Transactions on Neural Networks and Learning Systems.
  * Wu et al. [2021a] Haiyan Wu, Yanyun Qu, Shaohui Lin, Jian Zhou, Ruizhi Qiao, Zhizhong Zhang, Yuan Xie, and Lizhuang Ma.  Contrastive learning for compact single image dehazing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10551–10560, 2021a.
  * Wu et al. [2021b] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang.  CvT: Introducing Convolutions to Vision Transformers.  pages 22–31, 2021b.
  * Xia et al. [2022] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang.  Vision Transformer With Deformable Attention.  pages 4794–4803, 2022.
  * Xu et al. [2017] Jing Xu, Xiaolin Tian, Xin Meng, Yan Kong, Shumei Gao, Haoyang Cui, Fei Liu, Liang Xue, Cheng Liu, and Shouyu Wang.  Wavefront-sensing-based autofocusing in microscopy.  _Journal of Biomedical Optics_ , 22(8):086012–086012, 2017.
  * Xu et al. [2021] Ruikang Xu, Zeyu Xiao, Jie Huang, Yueyi Zhang, and Zhiwei Xiong.  EDPN: Enhanced Deep Pyramid Network for Blurry Image Restoration.  pages 414–423, 2021.
  * Yaşargil [2013] Mahmut Gazi Yaşargil.  _Microsurgery: applied to neurosurgery_.  Elsevier, 2013.
  * Yoo et al. [2018] Seunghwan Yoo, Pablo Ruiz, Xiang Huang, Kuan He, Nicola J Ferrier, Mark Hereld, Alan Selewa, Matthew Daddysman, Norbert Scherer, Oliver Cossairt, et al.  3d image reconstruction from multi-focus microscope: axial super-resolution and multiple-frame processing.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1453–1457. IEEE, 2018.
  * Yuan et al. [2020] Yuhui Yuan, Xilin Chen, and Jingdong Wang.  Object-Contextual Representations for Semantic Segmentation.  In _Computer Vision – ECCV 2020_ , pages 173–190, Cham, 2020. Springer International Publishing.
  * Zamir et al. [2022] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang.  Restormer: Efficient Transformer for High-Resolution Image Restoration.  pages 5728–5739, 2022.
  * Zhang et al. [2022a] Chi Zhang, Hao Jiang, Weihuang Liu, Junyi Li, Shiming Tang, Mario Juhas, and Yang Zhang.  Correction of out-of-focus microscopic images by deep learning.  _Computational and Structural Biotechnology Journal_ , 20:1957–1966, 2022a.
  * Zhang and Zhang [2019] Dengsheng Zhang and Dengsheng Zhang.  Wavelet transform.  _Fundamentals of image data mining: Analysis, Features, Classification and Retrieval_ , pages 35–44, 2019.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018.
  * Zhang et al. [2022b] Yunlong Zhang, Yuxuan Sun, Honglin Li, Sunyi Zheng, Chenglu Zhu, and Lin Yang.  Benchmarking the Robustness of Deep Neural Networks to Common Corruptions in Digital Pathology.  In _Medical Image Computing and Computer Assisted Intervention – MICCAI 2022_ , pages 242–252, Cham, 2022b. Springer Nature Switzerland.
  * Zhang et al. [2022c] Yuelin Zhang, Sihao Xiang, Zehuan Wang, Xiaoyan Peng, Yutong Tian, Shukai Duan, and Jia Yan.  Tdacnn: Target-domain-free domain adaptation convolutional neural network for drift compensation in gas sensors.  _Sensors and Actuators B: Chemical_ , 361:131739, 2022c.
  * Zhang et al. [2023a] Yanni Zhang, Qiang Li, Miao Qi, Di Liu, Jun Kong, and Jianzhong Wang.  Multi-scale frequency separation network for image deblurring.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 2023a.
  * Zhang et al. [2023b] Yulun Zhang, Donglai Wei, Richard Schalek, Yuelong Wu, Stephen Turney, Jeff Lichtman, Hanspeter Pfister, and Yun Fu.  High-Throughput Microscopy Image Deblurring with Graph Reasoning Attention Network.  In _2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI)_ , pages 1–5, 2023b.  ISSN: 1945-8452.
  * Zhao et al. [2023] Bingxin Zhao, Weihong Li, and Weiguo Gong.  Real-aware motion deblurring using multi-attention CycleGAN with contrastive guidance.  _Digital Signal Processing_ , 135:103953, 2023.
  * Zhao et al. [2020] Huangxuan Zhao, Ziwen Ke, Ningbo Chen, Songjian Wang, Ke Li, Lidai Wang, Xiaojing Gong, Wei Zheng, Liang Song, Zhicheng Liu, Dong Liang, and Chengbo Liu.  A new deep learning method for image deblurring in optical microscopic systems.  _Journal of Biophotonics_ , 13(3):e201960147, 2020.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/jbio.201960147.
  * Zhao et al. [2022] Suiyi Zhao, Zhao Zhang, Richang Hong, Mingliang Xu, Yi Yang, and Meng Wang.  FCL-GAN: A Lightweight and Real-Time Baseline for Unsupervised Blind Image Deblurring.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 6220–6229, New York, NY, USA, 2022. Association for Computing Machinery.
  * Zontak et al. [2013] Maria Zontak, Inbar Mosseri, and Michal Irani.  Separating signal from noise using patch recurrence across scales.  In _proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1195–1202, 2013.

Supplementary Material

##  6 Dataset Description

This section provides supplementary information about the datasets involved in
the evaluation. The details are shown in Tab. 8.

For real-world scenarios, three datasets are included. For DPDD [1], the
sharp-blur training pairs are collected sequentially by a DSLR camera with
different aperture sizes. A labeled defocus deblur dataset LFDOF [59]
collected by a light field camera is adopted in this paper as extra data for
knowledge transfer since the training pairs captured by the light field camera
have strict pixel-wise consistency [60]. LFDOF contains many more images than
DPDD, which also benefits the microscopy deblur by transferring rich cross-
domain information. For unsupervised deblur, an unlabeled blur dataset CUHK
[63] is adopted, which is collected from the internet.

Three datasets are involved in cell microscopy deblur. BBBC006 comes from the
Broad Bioimage Benchmark Collection [44], which contains images in two sub-
sets stained by Hoechst and phalloidin captured by fluorescence microscope. It
contains images with different focal planes (denoted as different z-stacks).
Following dataset description [44], images collected on the optimal focal
length (z-stack = 16) are set as the ground truth, and images above the
optimal focal plane are used for training. To avoid redundancy, images with
z-stack = [2, 6, 10] are set as blurry input for training. Since the images in
BBBC006 only contain a single grayscale channel, for EFCRex in BBBC006, the
images in LFDOF are converted into grayscale with one channel. 3DHistech and
WNLO [17] are two cell imaging datasets for cytopathology scanned by digital
scanners. The labeled dataset 3DHistech is scanned using different focal
planes, where the focal plane with the most cells in focus is set as the
ground truth. WNLO is an unlabeled dataset with defocus images only.

Regarding the surgical microscopy deblur, two new datasets are presented,
which are the labeled synthesized dataset CaDISBlur and the unlabeled cataract
surgery defocus blur dataset CataBlur. CaDISBlur is synthesized based on CaDIS
[19], which is a dataset for surgical scene semantic segmentation. Leveraging
segmentation masks, the instruments and anatomies are blurred separately to
simulate different focal planes. The original images in CaDIS are of high
quality thus they can be treated as the sharp ground truth. CataBlur is an
unlabeled real defocus blur dataset acquired during cataract surgeries, from
which the severity of defocus blur in microscope surgery can be observed.

Scenario | Dataset | #Image | Resolution | Label
---|---|---|---|---
Real world |  DPDD [1] | 500 |  1680$\times$1120 | Labeled
LFDOF [59] | 12,826 |  1008$\times$688 | Labeled
CUHK [63] | 704 |  $\sim$ 610$\times$470 | Unlabeled
Cell microscopy |  BBBC006 [44] | 6144 |  696$\times$520 | Labeled
3DHistech [17] | 94,973 |  256$\times$256 | Labeled
WNLO [17] | 108,065 |  256$\times$256 | Unlabeled
Surgical microscopy | CaDISBlur | 9340 |  960$\times$540 | Labeled
CataBlur | 4614 |  1280$\times$720 | Unlabeled

Table 8: Dataset description.

##  7 Supplementary Experiments

#### Feature deficiency

As addressed in Sec. 1, the feature deficiency in microscopy datasets can pose
harm to generalizability. To further demonstrate the drawbacks brought by
feature deficiency, experiments are conducted on unlabeled microscopy datasets
WNLO [17] and CataBlur regarding two settings. For the first setting $S_{1}$,
the MPT is trained on intra-domain microscopy images and tested on unlabeled
microscopy datasets, i.e., trained on 3DHistech [17] then tested on WNLO [17],
and trained on CaDISBlur then tested on CataBlur. For $S_{2}$, the MPT is
trained on cross-domain real-world images (LFDOF [59]) and tested on unlabeled
microscopy datasets.

The deblur results are shown in Fig. 6, from which it can be observed that the
model trained with cross-domain real-world dataset ($S_{2}$) leads to fewer
artifacts and more fine details in its deblurred results than the model
trained with intra-domain microscopy dataset ($S_{1}$). This phenomenon proves
the existence of feature deficiency in microscopy dataset, i.e., the model
trained with microscopy dataset suffers from poor generalizability caused by
the insufficient features contained in microscopy dataset. From visualization
of $S_{2}$, the necessity of learning cross-domain rich deblur guidance is
also proved.

Figure 6: Illustration of models trained with feature deficient microscopy
dataset ($S_{1}$) and feature sufficient real-world dataset ($S_{2}$),
including unlabeled dataset WNLO [17] (left) and CataBlur (right). Even if the
model is trained with intra-domain microscopy data ($S_{1}$), the deblur
restoration turns out to be trivial for CataBlur, or even brings strong
artifacts for WNLO. As for $S_{2}$, the deblur result tends to have fewer
artifacts, and more fine details are restored. It proves the existence of
feature deficiency problem and the necessity of learning cross-domain
knowledge.

#### Ablation studies on CSWA and FEFN

Figure 7: Downsampling module adopted in CSWA.  Figure 8: Baseline and variant
design of ablation studies for FEFN. VFEFN1 is the reversed structure (V3)
shown in LABEL:tab:ablationfefn

Configuration |  $\Delta$PSNRD |  $\Delta$PSNRB |  $\Delta$PSNRC
---|---|---|---
$V_{ds1}$ (w/o shortcut) | -0.14 | -0.02 | -0.15
$V_{ds2}$ (w/o linear) | -0.05 | -0.04 | -0.07
$V_{ds3}$ (max pooling) | -0.37 | -0.49 | -0.63
$V_{ds4}$ (convolution) | -0.03 | -0.09 | -0.06
$V_{ds5}$ (interpolate) | -0.01 | +0.02 | -0.04
$V_{w/o~{}NPC}$ | -0.09 | -0.07 | -0.12
$V_{w/o~{}sw}$ | -0.06 | -0.13 | -0.09
$V_{FEFN1}$ | -0.23 | -0.17 | -0.48
$V_{FEFN2}$ | -0.09 | -0.12 | -0.15
$V_{FEFN3}$ | -0.60 | -0.74 | -0.91
$V_{FEFN4}$ | -0.24 | -0.31 | -0.20

Table 9: Ablation studies on CSWA and FEFN. $\Delta$PSNR refers to the change
in PSNR compared with the MPT baseline.

Additional ablation studies regarding the structure of the proposed CSWA and
FEFN are conducted. The results are shown in Tab. 9. For the downsampling
module adopted in CSWA (Fig. 7), a series of variants are evaluated, including
variant without shortcut connection ($V_{ds1}$), variant without linear
projection and shortcut connection ($V_{ds2}$), and variants changing the
downsampling operation from strided average pooling to strided maximum pooling
($V_{ds3}$), strided convolution ($V_{ds4}$), and bicubic interpolation
($V_{ds5}$). The results in Tab. 9 show that the current baseline outperforms
the variants in most of the situations. Although $V_{ds5}$, which adopts
bicubic interpolation, achieves almost the same performance as the baseline or
even trivial improvement, it leads to complex computation that hinders the
parallel training and inference. Ablation studies on NPConv and shifting
window mechanism in CSWA are also carried out, which are denoted by
$V_{w/o~{}NPC}$ and $V_{w/o~{}sw}$, respectively. The results prove the
superiority of our design.

Going further from the experiments in Sec. 4.4, more ablation studies
regarding the asymmetric activation mechanism and shortcut connection in FEFN
are conducted, including four variants as shown in Fig. 8. Based on the result
reported in Tab. 9, we can conclude that the baseline structure achieves the
best performance among all the variants.

#### Ablation studies on pyramid scales

Following the description in Sec. 4.1 to keep the sub-block number, feature
dimensions, and attention heads unchanged, ablation studies on pyramid scales
($S_{i}$) are carried out regarding variants with similar FLOPs and
parameters:

1) V1: $S_{1}=S_{2}=S_{3}=S_{4}=[1,1,1,1,1,1]$. In this variant, CSWA actually
downgrades to the original WA (the variant with WA+ISCA shown in
LABEL:tab:ablationcswa), and the image pyramid is not constructed.

2) V2: $S_{1}=[\frac{1}{16},\frac{1}{16},\frac{1}{8},\frac{1}{8},1,1]$,
$S_{2}=[\frac{1}{8},\frac{1}{8},\frac{1}{4},\frac{1}{4},1,1]$,
$S_{3}=S_{4}=[\frac{1}{4},\frac{1}{4},\frac{1}{2},\frac{1}{2},1,1]$. This
variant explores the pyramid structure with smaller scales than the baseline.

3) V3: $S_{1}=[\frac{1}{4},\frac{1}{4},\frac{1}{2},\frac{1}{2},1,1]$,
$S_{2}=[\frac{1}{2},\frac{1}{2},\frac{1}{2},\frac{1}{2},1,1]$,
$S_{3}=S_{4}=[\frac{1}{2},\frac{1}{2},\frac{1}{2},\frac{1}{2},1,1]$. This
variant adopts larger-scale pyramids than the baseline.

As shown by the results in Tab. 10, all the variants lead to performance drop.
For V1 with no pyramid structure, significant performance degradation is
observed on BBBC006, which is the dataset with one of the longest attention
spans. A similar phenomenon is observed in SwinIR [38] that does not feature a
multi-scale pyramid. It achieves inferior performance on BBBC006 as shown in
Tab. 1. These together prove the effectiveness of our multi-pyramid design,
especially for the microscopy deblur tasks. The performance degradation in V2
and V3 shows the superiority of the pyramid scales in the baseline model.

Pyramid |  $\Delta$PSNRD |  $\Delta$PSNRB |  $\Delta$PSNRC
---|---|---|---
V1 | -0.08 | -0.79 | -0.20
V2 | -0.05 | -0.02 | -0.07
V3 | -0.01 | -0.04 | -0.03
Table 10: Ablation studies on pyramid scales.

#### Additional visualization

Demonstrations of supervised real-world deblur are shown in Fig. 10 for
labeled datasets DPDD [1]. Illustrations of unsupervised deblur on microscopy
dataset and real-world dataset are provided in Fig. 11(a) and Fig. 11(b),
respectively. The visualization proves that our method achieves the best
performance on microscopy datasets and real-world datasets regarding various
patterns in both supervised and unsupervised scenarios. More visualizations of
the results of cell detection on BBBC006 [44] and surgical scene semantic
segmentation on CaDISBlur are provided in Fig. 12, from which it can be
concluded that the downstream tasks results on deblurred images from our
method achieves more satisfactory outcomes.

(a) Examples for CaDISBlur dataset, where the top row is the original sharp
images (ground truth), the middle row and bottom row are the corresponding
blurry images with the focal plane on instruments (blurry anatomies) or
anatomies (blurry instruments), respectively.

(b) Examples for CataBlur dataset, which is an unlabeled dataset containing
only defocus images.

Figure 9: Examples of samples in CaDISBlur and CataBlur. Figure 10:
Visualization of deblur on DPDD [1] dataset. As the comparison shows, our MPT
achieves the best deblur in the real-world scenes, restoring most of the
small-scale fine details and large-scale patterns. With the help of EFCRex,
the performance is further enhanced. It proves the superiority and
generalizability of our framework.

(a) Visualization of deblur on WNLO [17] (top) and CataBlur (bottom) datasets.

(b) Visualization of deblur on CUHK [63] dataset.

Figure 11: Visualization of unsupervised deblur on CataBlur, WNLO [17], and
CUHK [63]. The proposed framework achieves the best unsupervised deblur
performance on both microscopy datasets and real-world dataset, showing high
generalizability. By further applying EFCRex, the deblur performance is
significantly enhanced, proving the effectiveness of the proposed EFCR for
knowledge transfer. Figure 12: Results of downstream tasks on BBBC006 [44]
(top) and CaDISBlur (bottom). For BBBC006, our method can restore precise cell
shape with sharper outline, achieving more accurate segmentation and
detection. For CaDISBlur, the deblurred images by our framework contain more
differentiable features, leading to more accurate semantic segmentation in
both anatomies and instruments. (Colormap:  Pupil,  Iris,  surgeon’s hand, ccc
Cornea,  Skin,  Surgical tape,  Eye retractors,  Instruments)


Partial $\gamma$-Positivity for Quasi-Stirling Permutations of Multisets

Sherry H.F. Yan, Yunwei Huang, Lihong Yang

Department of Mathematics

Zhejiang Normal University

Jinhua 321004, P.R. China

<EMAIL_ADDRESS>
Abstract. We prove that the enumerative polynomials of quasi-Stirling
permutations of multisets with respect to the statistics of plateaux, descents
and ascents are partial $\gamma$-positive, thereby confirming a recent
conjecture posed by Lin, Ma and Zhang. This is accomplished by proving the
partial $\gamma$-positivity of the enumerative polynomials of certain ordered
labeled trees, which are in bijection with quasi-Stirling permutations of
multisets. As an application, we provide an alternative proof of the partial
$\gamma$-positivity of the enumerative polynomials on Stirling permutations of
multisets.

Keywords: quasi-Stirling permutation, Stirling permutation, partial
$\gamma$-positivity.

AMS Subject Classifications: 05A05, 05C30

##  1 Introduction

Gamma-positive polynomials arise frequently in combinatorics; the reader is
referred to the survey of Athanasiadis [2] for more information. A univariate
polynomial $f_{n}(x)$ is said to be $\gamma$-positive if it can be expanded as

$f_{n}(x)=\sum_{k=0}^{\lfloor{n\over 2}\rfloor}\gamma_{k}x^{k}(1+x)^{n-2k}$

with $\gamma_{k}\geq 0$. A bivariate polynomial $h_{n}(x,y)$ is said to be
homogeneous $\gamma$-positive, if $h_{n}(x,y)$ can be expressed as

$h_{n}(x,y)=\sum_{k=0}^{\lfloor{n\over
2}\rfloor}\gamma_{k}(xy)^{k}(x+y)^{n-2k}$

with $\gamma_{k}\geq 0$. Foata and Schüzenberger [8] proved that the bivariate
Eulerian polynomial
$A_{n}(x,y)=\sum_{\pi\in\mathcal{S}_{n}}x^{asc(\pi)}y^{des(\pi)}$ is
homogeneous $\gamma$-positive, where $\mathcal{S}_{n}$ is the set of
permutations of $[n]=\\{1,2,\ldots,n\\}$. A trivariate polynomial
$p(x,y,z)=\sum_{i}s_{i}(x,y)z^{i}$ is said to be partial $\gamma$-positive if
every $s_{i}(x,y)$ is homogeneous $\gamma$-positive.

Let $\mathcal{M}=\\{1^{k_{1}},2^{k_{2}},\ldots,n^{k_{n}}\\}$ be a multiset
where $k_{i}$ is the number of occurrences of $i$ in $\mathcal{M}$ and
$k_{i}\geq 1$. A permutation $\pi=\pi_{1}\pi_{2}\ldots\pi_{n}$ of a multiset
$\mathcal{M}$ is said to be a Stirling  permutation if $i<j<k$ and
$\pi_{i}=\pi_{k}$, then $\pi_{j}>\pi_{i}$. For a multiset $\mathcal{M}$, let
$\mathcal{Q}_{\mathcal{M}}$ denote the set of Stirling permutations of
$\mathcal{M}$. For example, if $\mathcal{M}=\\{1^{2},2^{2}\\}$, we have
$\mathcal{Q}_{\mathcal{M}}=\\{1221,2211,1122\\}.$ Stirling permutations were
originally introduced by Gessel and Stanley [9] in the case of the multiset
$\mathcal{M}=\\{1^{2},2^{2},\ldots,n^{2}\\}$. There is an extensive literature
on Stirling permutations, see [3, 4, 6, 12, 14, 15] and references therein. In
analogy to Stirling permutations, Archer et al. [1] introduced quasi-Stirling
permutations. A permutation $\pi=\pi_{1}\pi_{2}\ldots\pi_{n}$ of a multiset is
said to be a quasi-Stirling  permutation if there does not exist four indices
$i<j<k<\ell$ such that $\pi_{i}=\pi_{k}$ and $\pi_{j}=\pi_{\ell}$. For a
multiset $\mathcal{M}$, denote by $\overline{\mathcal{Q}}_{\mathcal{M}}$ the
set of quasi-Stirling permutations of $\mathcal{M}$. For example, if
$\mathcal{M}=\\{1^{2},2^{2}\\}$, we have
$\overline{\mathcal{Q}}_{\mathcal{M}}=\\{1221,2112,1122,2211\\}.$ Clearly, we
have $\mathcal{Q}_{\mathcal{M}}\subseteq\overline{\mathcal{Q}}_{\mathcal{M}}$
for a fixed multiset $\mathcal{M}$.

For a permutation $\pi=\pi_{1}\pi_{2}\ldots\pi_{n}$, an index $i$, $0\leq
i\leq n$, is called an ascent (resp. a descent, a plateau) of $\pi$ if
$\pi_{i}<\pi_{i+1}$ (resp. $\pi_{i}>\pi_{i+1}$, $\pi_{i}=\pi_{i+1}$) with the
convention $\pi_{0}=\pi_{n+1}=0$. Let $asc(\pi)$ (resp. $des(\pi)$,
$plat(\pi)$) denote the number of ascents (resp. descents, plateaux) of $\pi$.
An index $i$, $1\leq i\leq n$, is called a double descent of $\pi$ if
$\pi_{i-1}>\pi_{i}>\pi_{i+1}$ with the convention $\pi_{0}=\pi_{n+1}=0$. Let
$ddes(\pi)$ denote the number of double descents of $\pi$. In [5], Bona proved
that the three statistics $asc$, $plat$ and $des$ are equidistributed over
Stirling permutations on $\mathcal{M}=\\{1^{2},2^{2},\ldots,n^{2}\\}$. In [7],
Elizalde investigated the distribution of the statistics of plateaux, descents
and ascents on quasi-Stirling permutations of the multiset
$\mathcal{M}=\\{1^{k},2^{k},\ldots,n^{k}\\}$.

For a multiset $\mathcal{M}$, we define

$\overline{Q}_{\mathcal{M}}(x,y,z)=\sum_{\pi\in\overline{\mathcal{Q}}_{\mathcal{M}}}x^{asc(\pi)}y^{des(\pi)}z^{plat(\pi)},$

and

$Q_{\mathcal{M}}(x,y,z)=\sum_{\pi\in\mathcal{Q}_{\mathcal{M}}}x^{asc(\pi)}y^{des(\pi)}z^{plat(\pi)}.$

In [10], Lin, Ma and Zhang proved that the polynomial $Q_{\mathcal{M}}(x,y,z)$
is partial $\gamma$-positive via the machine of context-free grammars and a
group action on Stirling multipermutations. Specializing their result to the
so-called Jacobi-Stirling permutations introduced in [11] confirms a recent
partial $\gamma$-positivity conjecture due to Ma, Ma and Yeh [13]. Lin, Ma and
Zhang [10] also posed the following conjecture.

###### Conjecture 1.1

For a multiset $\mathcal{M}$, the polynomial
$\overline{Q}_{\mathcal{M}}(x,y,z)$ is partial $\gamma$-positive.

In this paper, we confirm Conjecture 1.1 by proving the partial
$\gamma$-positivity of the enumerative polynomials of certain ordered labeled
trees, which are shown to be in bijection with quasi-Stirling permutations in
[16]. As an application, we provide an alternative proof of the partial
$\gamma$-positivity of the polynomial $Q_{\mathcal{M}}(x,y,z)$.

The rest of this paper is organized as follows. In Section 2, we prove the
partial $\gamma$-positivity of the enumerative polynomials of certain ordered
labeled trees by introducing a group action in the sprit of the Foata-Strehl
action on permutations [8]. Section 3 is devoted to the proofs of the partial
$\gamma$-positivity of the polynomial $\overline{Q}_{\mathcal{M}}(x,y,z)$ and
$Q_{\mathcal{M}}(x,y,z)$.

##  2 Partial $\gamma$-positivity for Trees

The objective of this section is to prove the partial $\gamma$-positivity of
the enumerative polynomials of certain ordered labeled trees, which were shown
to be in bijection with quasi-Stirling permutations in [16]. Recall that an
ordered tree is a tree with one designated vertex, which is called the root,
and the subtrees of each vertex are linearly ordered. In an ordered tree $T$,
the level of a vertex $v$ in $T$ is defined to be the length of the unique
path from the root to $v$. A vertex $v$ is said to be odd  (resp. even) if the
level of $v$ is odd (resp. even).

Yan and Zhu [16] introduced a new class of ordered labeled trees $T$ verifying
the following properties:

  * (i)

the vertices are labeled by the elements of the multiset
$\\{0\\}\cup\mathcal{M}$, where
$\mathcal{M}=\\{1^{k_{1}},2^{k_{2}},\ldots,n^{k_{n}}\\}$ with $k_{i}\geq 1$;

  * (ii)

the root is labeled by $0$;

  * (iii)

for an odd vertex $v$, if $v$ is labeled by $i$, then $v$ has exactly
$k_{i}-1$ children and the children of $v$ have the same label as that of $v$
in $T$.

Let $\mathcal{T}_{\mathcal{M}}$ denote the set of such ordered labeled trees.
For example, a tree $T\in\mathcal{T}_{\mathcal{M}}$ with
$\mathcal{M}=\\{1,2,3,4,5^{3},6,7^{2}\\}$ is illustrated in Figure 1.

Figure 1: A tree $\mathcal{T}\in\mathcal{T}_{\mathcal{M}}$ with
$\mathcal{M}=\\{1,2,3,4,5^{3},6,7^{2}\\}$

For a sequence $\pi=\pi_{1}\pi_{2}\ldots\pi_{n}$, the entry $\pi_{i}$, $1\leq
i\leq n$, is said to be a cyclic descent (resp. a cyclic ascent) if
$\pi_{i}>\pi_{i+1}$ (resp. $\pi_{i}<\pi_{i+1}$) with the convention
$\pi_{n+1}=\pi_{1}$. An entry $\pi_{i}$, $1\leq i\leq n$, is said to be a
double cyclic descent (resp. a double cyclic ascent, a cyclic peak, a cyclic
valley ) if $\pi_{i-1}>\pi_{i}>\pi_{i+1}$ (resp.
$\pi_{i-1}<\pi_{i}<\pi_{i+1}$, $\pi_{i-1}<\pi_{i}>\pi_{i+1}$,
$\pi_{i-1}>\pi_{i}<\pi_{i+1}$ ) with the convention $\pi_{n+1}=\pi_{1}$ and
$\pi_{0}=\pi_{n}$. Let $cdes(\pi)$ (resp. $casc(\pi)$, $dcdes(\pi)$,
$dcasc(\pi)$, $cpeak(\pi)$, $cval(\pi)$ ) denote the number of cyclic descents
(resp. cyclic ascents, double cyclic descents, double cyclic ascents, cyclic
peaks, cyclic valleys) of $\pi$. Denote by $CDES(\pi)$ the set of cyclic
descents of $\pi$. For any sequence $\pi=\pi_{1}\pi_{2}\ldots\pi_{n}$, if
$\pi_{i}$ is a double cyclic ascent or a double cyclic descent, the cyclic
$\pi_{i}$-factorization of $\pi$ is defined to be $W_{1}\pi_{i}W_{2}W_{3}$,
where $W_{1}$ (resp. $W_{2}$) is the maximal contiguous subsequence of $\pi$
immediately to the left (resp. right) of $\pi_{i}$ whose entries are all
smaller than $\pi_{i}$ in the sense that the entries of $\pi$ are arranged in
a cycle clockwisely. For instance, let $\pi=15324$, then the cyclic
$3$-factorization of $\pi$ is given by $W_{1}3W_{2}W_{3}$ where $W_{1}$ is
empty, $W_{2}=2$ and $W_{3}=415$.

In an ordered labeled tree $T$, let $u$ be a vertex of $T$ which is labeled by
$v_{0}$. Suppose that the children of $u$ are labeled by
$v_{1},v_{2},\ldots,v_{\ell}$ from left to right. The number of cyclic
descents (resp. cyclic ascents) of $u$, denoted by $cdes(u)$ (resp.
$casc(u)$), is defined to be the number $cdes(v_{0}v_{1}v_{2}\ldots v_{\ell})$
(resp. $casc(v_{0}v_{1}v_{2}\ldots v_{\ell})$). The number of cyclic descents
and cyclic ascents of $T$ are defined to be

$cdes(T)=\sum_{u\in V(T)}cdes(u)$

and

$casc(T)=\sum_{u\in V(T)}casc(u)$

where $V(T)$ denotes the vertex set of $T$. Denote by $eleaf(T)$ the number of
leaves at even level.

Let $v$ be an odd vertex of a tree $T\in\mathcal{T}_{n}$ and let $u$ be its
parent labeled by $v_{0}$. Suppose that the children of the vertex $u$ are
labeled by $v_{1},v_{2},\ldots,v_{\ell}$ from left to right. Assume that the
vertex $v$ is labeled by $v_{i}$ for some $1\leq i\leq\ell$. Then the odd
vertex $v$ is said to be a double cyclic descent  (resp. a double cyclic
ascent , a cyclic peak , a cyclic valley) of $T$ if and only if $v_{i}$ is a
double cyclic descent (resp. a double cyclic ascent, a cyclic peak, a cyclic
valley) of the sequence $v_{0}v_{1}v_{2}\ldots v_{\ell}$.

Let $u$ be an even vertex of a tree $T\in\mathcal{T}_{n}$ which is labeled by
$v_{0}$. Suppose that the children of $u$ are labeled by
$v_{1},v_{2},\ldots,v_{\ell}$ from left to right. Then the even vertex $u$ is
said to be a double cyclic descent  (resp. a double cyclic ascent , a cyclic
peak, a cyclic valley) of $T$ if and only if $v_{0}$ is a double cyclic
descent (resp. a double cyclic ascent, a cyclic peak, a cyclic valley) of the
sequence $v_{0}v_{1}v_{2}\ldots v_{\ell}$. Denote by $dcdes(T)$ (resp.
$dcasc(T)$, $cpeak(T)$, $cval(T)$) the number of double cyclic descents (resp.
double cyclic ascents, cyclic peaks, cyclic valleys) of $T$.

For example, if we let $T$ be a tree as shown in Figure 1, we have
$casc(T)=5$, $cdes(T)=4$, $eleaf(T)=2$, $dcdes(T)=0$, $dcasc(T)=1$,
$cpeak(T)=4$ and $cval(T)=4$.

Now we are ready to state our main results of this section.

###### Theorem 2.1

Let

$T_{\mathcal{M}}(x,y,z)=\sum_{T\in\mathcal{T}_{\mathcal{M}}}x^{casc(T)}y^{cdes(T)}z^{eleaf(T)},$

where $\mathcal{M}=\\{1^{k_{1}},2^{k_{2}},\ldots,n^{k_{n}}\\}$ and
$K=k_{1}+k_{2}+\ldots+k_{n}$ with $k_{i}\geq 1$. The polynomial
$T_{\mathcal{M}}(x,y,z)$ is partial $\gamma$-positive and has the expansion

$T_{\mathcal{M}}(x,y,z)=\sum_{i=0}^{K-n}z^{i}\sum_{j=1}^{\lfloor{K+1-i\over
2}\rfloor}\gamma^{\mathcal{T}}_{\mathcal{M},i,j}(xy)^{j}(x+y)^{K+1-i-2j},$

where
$\gamma^{\mathcal{T}}_{\mathcal{M},i,j}=|\\{T\in\mathcal{T}_{\mathcal{M}}\mid
cdes(T)=j,eleaf(T)=i,dcdes(T)=0\\}|$.

A tree $T\in\mathcal{T}_{\mathcal{M}}$ is said to be weakly increasing  if the
labels of the vertices on the path from the root to any leaf is weakly
increasing from top to bottom. See Figure 2 for an example. Denote by
$\mathcal{IT}_{\mathcal{M}}$ the set of weakly increasing ordered labeled
trees $T\in\mathcal{T}_{\mathcal{M}}$.

Figure 2: A tree $\mathcal{T}\in\mathcal{IT}_{\mathcal{M}}$ with
$\mathcal{M}=\\{1^{2},2,3,4,5^{3},6,7\\}$.

###### Theorem 2.2

Let

$IT_{\mathcal{M}}(x,y,z)=\sum_{T\in\mathcal{IT}_{\mathcal{M}}}x^{casc(T)}y^{cdes(T)}z^{eleaf(T)},$

where $\mathcal{M}=\\{1^{k_{1}},2^{k_{2}},\ldots,n^{k_{n}}\\}$ and
$K=k_{1}+k_{2}+\ldots+k_{n}$ with $k_{i}\geq 1$. The polynomial
$IT_{\mathcal{M}}(x,y,z)$ is partial $\gamma$-positive and has the expansion

$IT_{\mathcal{M}}(x,y,z)=\sum_{i=0}^{K-n}z^{i}\sum_{j=1}^{\lfloor{K+1-i\over
2}\rfloor}\gamma^{\mathcal{IT}}_{\mathcal{M},i,j}(xy)^{j}(x+y)^{K+1-i-2j},$

where
$\gamma^{\mathcal{IT}}_{\mathcal{M},i,j}=|\\{T\in\mathcal{IT}_{\mathcal{M}}\mid
cdes(T)=j,eleaf(T)=i,dcdes(T)=0\\}|$.

In the following, we shall define a group action on ordered labeled trees in
the sprit of the Foata-Strehl action on permutations [8].

Let $T\in\mathcal{T}_{\mathcal{M}}$. For any vertex $u$ of $T$, we define the
the tree FS-action $\psi_{u}$ on the ordered labeled trees $T$ in the
following way.

Case 1. If the vertex $u$ is an even leaf, a cyclic valley or a cyclic peak,
let $\psi_{u}(T)=T$. Case 2. The vertex $u$ is a double cyclic ascent or a
double cyclic descent.
Subcase 2.1. The vertex $u$ is odd. Suppose that the vertex $v$ labeled by
$v_{0}$ is the parent of $u$ and the children of $v$ are labeled by
$v_{1},v_{2},\ldots,v_{\ell}$ from left to right. Assume that the vertex $u$
is labeled by $v_{k}$ for some $1\leq k\leq\ell$. Let $T_{i}$, $(1\leq
i\leq\ell)$, be the subtree rooted at the odd vertex labeled by $v_{i}$.
Suppose that the cyclic $v_{k}$-factorization of the sequence
$v_{0}v_{1}v_{2}\ldots v_{\ell}$ is given by $W_{1}v_{k}W_{2}W_{3}$. Then let
$\psi_{u}(T)$ be the tree obtained from $T$ by rearranging the subtrees
$T_{1},T_{2},\ldots,T_{\ell}$ such that
$CDES(W_{2}v_{k}W_{1}W_{3})=CDES(v_{0}v^{\prime}_{1}v^{\prime}_{2}\ldots
v^{\prime}_{\ell})$, where the children of the vertex $v$ in $\psi_{u}(T)$ are
labeled by $v^{\prime}_{1},v^{\prime}_{2},\ldots,v^{\prime}_{\ell}$ from left
to right. See Figure 3 for an example.

Figure 3: An example of the action $\psi_{u}$ where the vertex $u$ is circled.

Subcase 2.2. The vertex $u$ is even. Suppose that the vertex $u$ is labeled by
$v_{0}$ and the children of $u$ are labeled by $v_{1},v_{2},\ldots,v_{\ell}$
from left to right. Let $T_{i}$, $(1\leq i\leq\ell)$, be the subtree rooted at
the odd vertex labeled by $v_{i}$. Suppose that the cyclic
$v_{0}$-factorization of the sequence $v_{0}v_{1}v_{2}\ldots v_{\ell}$ is
given by $W_{1}v_{0}W_{2}W_{3}$. Then let $\psi_{u}(T)$ be the tree obtained
from $T$ by rearranging the subtrees $T_{1},T_{2},\ldots,T_{\ell}$ such that
$CDES(W_{2}v_{0}W_{1}W_{3})=CDES(v_{0}v^{\prime}_{1}v^{\prime}_{2}\ldots
v^{\prime}_{\ell})$, where the children of the vertex $u$ in $\psi_{u}(T)$ are
labeled by $v^{\prime}_{1},v^{\prime}_{2},\ldots,v^{\prime}_{\ell}$ from left
to right. See Figure 4 for an example.

Figure 4: An example of the action $\psi_{u}$ where the vertex $u$ is circled.

The following observation will play an essential role in the proof of Theorem
2.1.

###### Observation 2.3

Let $\mathcal{M}=\\{1^{k_{1}},2^{k_{2}},\ldots,n^{k_{n}}\\}$ and
$K=k_{1}+k_{2}+\ldots+k_{n}$ with $k_{i}\geq 1$. Then we have

$eleaf(T)+dcdes(T)+dcasc(T)+cpeak(T)+cval(T)=K+1,$ (2.1) $cpeak(T)=cval(T),$
(2.2) $cdes(T)=cval(T)+dcdes(T)$ (2.3)

and

$casc(T)=cpeak(T)+dcasc(T)$ (2.4)

for any $T\in\mathcal{T}_{\mathcal{M}}$.

We are now ready for the proof of Theorem 2.1.

Proof of Theorem 2.1. Define $\mathcal{T}_{\mathcal{M},i}$ be the set of
ordered labeled trees $T\in\mathcal{T}_{\mathcal{M}}$ with $eleaf(T)=i$. Then
Theorem 2.1 is equivalent to

$\begin{array}[]{lll}\sum_{T\in\mathcal{T}_{\mathcal{M},i}}x^{casc(T)}y^{cdes(T)}&=&\sum_{T\in\widetilde{\mathcal{T}_{\mathcal{M},i}}}(xy)^{cdes(T)}(x+y)^{K+1-i-2cdes(T)}.\end{array}$
(2.5)

where $\widetilde{\mathcal{T}_{\mathcal{M},i}}$ denotes the set of ordered
labeled trees $T\in\mathcal{T}_{\mathcal{M},i}$ with $dcdes(T)=0$.

Clearly, the tree FS-actions $\psi_{u}$’s are involutions and commute. Thus,
for any $S\subseteq V(T)$, we can define the function
$\psi_{S}:\mathcal{T}_{\mathcal{M}}\rightarrow\mathcal{T}_{\mathcal{M}}$ by
$\psi_{S}=\prod_{u\in S}\psi_{u}$, where the product is the functional
compositions. Hence the group $\mathbb{Z}_{2}^{K+1}$ acts on
$\mathcal{T}_{\mathcal{M}}$ via the function $\psi_{S}$. Since the statistic
“eleaf” is invariant under this group action, it divides
$\mathcal{T}_{\mathcal{M},i}$ into some disjoint orbits. For each
$T\in\mathcal{T}_{\mathcal{M},i}$, let
$Orb(T)=\\{g(T):T\in\mathbb{Z}_{2}^{K+1}\\}$ be the orbit of $T$ under the
tree FS-action. Notice that the vertex $u$ is a double cyclic descent of $T$
if and only if the vertex $u$ is a double cyclic ascent of $\psi_{u}(T)$. This
yields that there is a unique tree $\widetilde{T}\in Orb(T)$ such that
$dcdes(T)=0$. Therefore, we have

$\begin{array}[]{lll}\sum_{T\in
Orb(\widetilde{T})}x^{casc(T)}y^{cdes(T)}&=&x^{cpeak(\widetilde{T})}y^{cval(\widetilde{T})}(x+y)^{cdasc(\widetilde{T})}\\\
&=&x^{cpeak(\widetilde{T})}y^{cval(\widetilde{T})}(x+y)^{K+1-i-cpeak(\widetilde{T})-cval(\widetilde{T})}\\\
&=&(xy)^{cdes(\widetilde{T})}(x+y)^{K+1-i-2cdes(\widetilde{T})},\\\
\end{array}$

where the second equality follows from (2.1) and the third equality follows
from (2.2)-(2.4). Summing over all orbits of $\mathcal{T}_{\mathcal{M},i}$
under the tree FS-action then gives (2.5), completing the proof.

Proof of Theorem 2.2. From the construction of the cyclic FS-action
$\psi_{u}$, it is easily seen that $\psi_{u}(T)\in\mathcal{Q}_{\mathcal{M}}$
for any $T\in\mathcal{IT}_{\mathcal{M}}$. Therefore, Theorem 2.2 can be proved
by similar reasoning as in the proof of Theorem 2.1.

##  3 Partial $\gamma$-positivity for Quasi-Stirling Permutations

Let $\pi$ be a permutation, the leftmost entry of $\pi$ is denoted by
$first(\pi)$. Similarly, for an ordered labeled tree $T$, we denote by
$first(T)$ the label of the leftmost child of the root.

In [16], Yan and Zhu established a bijection $\phi$ between
$\mathcal{T}_{\mathcal{M}}$ and $\overline{\mathcal{Q}}_{\mathcal{M}}$. First
we give an overview of the bijection $\phi$. If $T$ has only one vertex, let
$\phi(T)=\epsilon$, where $\epsilon$ denotes the empty permutation. Otherwise,
suppose that $first(T)=r$.

Case 1. The leftmost child of the root is a leaf. Let $T_{0}$ be the tree
obtained from $T$ by removing the leftmost child of the root together with the
edge incident to it. Define $\phi(T)=r\phi(T_{0})$.

Case 2. The leftmost child of the root has $k$ children. For $1\leq i\leq k$,
let $T_{i}$ be the subtree rooted at the $i$-th child of the leftmost child of
the root. Denote by $T^{\prime}_{i}$ the tree obtained from $T_{i}$ by
relabeling its root by $0$. Let $T_{0}$ be the tree obtained from $T$ by
removing all the subtrees $T_{1}$, $T_{2}$, $\ldots,$ $T_{k}$ and all the
vertices labeled by $r$ together with the edges incident to them. Define
$\phi(T)=r\phi(T^{\prime}_{1})r\phi(T^{\prime}_{2})\ldots
r\phi(T^{\prime}_{k})r\ \phi(T_{0})$.

For instance, let $T$ be a tree illustrated in Figure 1. By applying the map
$\phi$, we have $\phi(T)=2773516455$.

In [17], Yan et al. proved that the bijection $\phi$ has the following
property.

###### Lemma 3.1

( [17])  For any nonempty multiset $\mathcal{M}$, the map $\phi$ induces a
bijection between $\mathcal{T}_{\mathcal{M}}$ and
$\overline{\mathcal{Q}}_{\mathcal{M}}$ such that

$(cdes,casc,eleaf)T=(des,asc,plat)\phi(T)$

for any $T\in\mathcal{T}_{\mathcal{M}}$.

Combining Lemma 3.1 and Theorem 2.1, we immediately derive the following
partial $\gamma$-positivity for $\overline{Q}_{\mathcal{M}}(x,y,z)$, thereby
confirming Conjecture 1.1.

###### Theorem 3.2

Let $\mathcal{M}=\\{1^{k_{1}},2^{k_{2}},\ldots,n^{k_{n}}\\}$ and
$K=k_{1}+k_{2}+\ldots+k_{n}$ with $k_{i}\geq 1$. The polynomial
$\overline{Q}_{\mathcal{M}}(x,y,z)$ is partial $\gamma$-positive and has the
expansion

$\overline{Q}_{\mathcal{M}}(x,y,z)=\sum_{i=0}^{K-n}z^{i}\sum_{j=1}^{\lfloor{K+1-i\over
2}\rfloor}\gamma^{\mathcal{T}}_{\mathcal{M},i,j}(xy)^{j}(x+y)^{K+1-i-2j},$

where
$\gamma^{\mathcal{T}}_{\mathcal{M},i,j}=|\\{T\in\mathcal{T}_{\mathcal{M}}\mid
cdes(T)=j,eleaf(T)=i,dcdes(T)=0\\}|$.

From the construction of the map $\phi$, it is easily seen that
$\phi(T)\in\mathcal{Q}_{\mathcal{M}}$ for any
$T\in\mathcal{IT}_{\mathcal{M}}$. Hence, the following result follows
immediately from Lemma 3.1.

###### Lemma 3.3

For any nonempty multiset $\mathcal{M}$, the map $\phi$ induces a bijection
between $\mathcal{IT}_{\mathcal{M}}$ and $\mathcal{Q}_{\mathcal{M}}$ such that

$(cdes,casc,eleaf)T=(des,asc,plat)\phi(T)$

for any $T\in\mathcal{IT}_{\mathcal{M}}$.

Combining Lemma 3.3 and Theorem 2.2, we are led to an alternative proof of the
partial $\gamma$-positivity of the polynomial $Q_{\mathcal{M}}(x,y,z)$.

###### Theorem 3.4

Let $\mathcal{M}=\\{1^{k_{1}},2^{k_{2}},\ldots,n^{k_{n}}\\}$ and
$K=k_{1}+k_{2}+\ldots+k_{n}$ with $k_{i}\geq 1$. The polynomial
$Q_{\mathcal{M}}(x,y,z)$ is partial $\gamma$-positive and has the expansion

$Q_{\mathcal{M}}(x,y,z)=\sum_{i=0}^{K-n}z^{i}\sum_{j=1}^{\lfloor{K+1-i\over
2}\rfloor}\gamma^{\mathcal{IT}}_{\mathcal{M},i,j}(xy)^{j}(x+y)^{K+1-i-2j},$

where
$\gamma^{\mathcal{IT}}_{\mathcal{M},i,j}=|\\{T\in\mathcal{IT}_{\mathcal{M}}\mid
cdes(T)=j,eleaf(T)=i,dcdes(T)=0\\}|$.

###### Remark 3.5

In [10], Lin, Ma and Zhang provided three combinatorial interpretations for
the $\gamma$-coefficients of the partial $\gamma$-positivity expansion of
$Q_{\mathcal{M}}(x,y,z)$. Theorem 3.4 gives a new combinatorial
interpretation.

We conclude this section with an alternative proof of the partial
$\gamma$-positivity of the polynomial $\overline{Q}_{\mathcal{M}}(x,y,z)$ by
employing the result obtained by Yan et al. [17] and the partial
$\gamma$-positivity of the polynomial $Q_{\mathcal{M}}(x,y,z)$ proved by Lin,
Ma and Zhang [10]. In [17], Yan et al. obtained the following result
concerning the polynomial $\overline{Q}_{\mathcal{M}}(x,y,z)$.

###### Theorem 3.6

( [17])  Let $\mathcal{M}=\\{1^{k_{1}},2^{k_{2}},\ldots,n^{k_{n}}\\}$,
$\mathcal{M^{\prime}}=\\{1^{K-n+1},2,3,\ldots,n\\}$ and
$K=k_{1}+k_{2}+\ldots+k_{n}$ with $k_{i}\geq 1$. We have
$\overline{Q}_{\mathcal{M}}(x,y,z)=\overline{Q}_{\mathcal{M^{\prime}}}(x,y,z)$.

Notice that quasi-Stirling permutations of
$\mathcal{M^{\prime}}=\\{1^{K-n+1},2,3,\ldots,n\\}$ are equivalent to Stirling
permutations of $\mathcal{M^{\prime}}=\\{1^{K-n+1},2,3,\ldots,n\\}$. Thus, by
Theorem 3.6, we have
$\overline{Q}_{\mathcal{M}}(x,y,z)=Q_{\mathcal{M^{\prime}}}(x,y,z)$. Thus, the
partial $\gamma$-positivity of the polynomial $Q_{\mathcal{M}}(x,y,z)$ implies
the the partial $\gamma$-positivity of the polynomial
$\overline{Q}_{\mathcal{M}}(x,y,z)$.

Acknowledgments. This work was supported by the National Natural Science
Foundation of China (12071440).

## References

  * [1] K. Archer, A. Gregory, B. Pennington, S. Slayden, Pattern restricted quasi-Stirling permutations, Australas. J. Comb., 74 (2019), 389–407.
  * [2] C.A. Athanasiadis, Gamma-positivity in combinatorics and geometry, Sém. Lothar. Combin., 77 (2018), Article B77i, 64pp (electronic).
  * [3] F. Brenti, Unimodal, log-concave and Pólya frequency sequences in combinatorics, Mem. Amer. Math. Soc., 81(413) (1989).
  * [4] F. Brenti, Hilbert polynomials in combinatorics, J. Algebraic Combin., 7 (1998), 127–156.
  * [5] M. Bóna, Real zeros and normal distribution for statistics on Stirling permutations defined by Gessel and Stanley, SIAM J. Discrete Math., 23 (2008), 401–406.
  * [6] A. Dzhumadil’daev, D. Yeliussizov, Stirling permutations on multisets, European J. Combin., 36 (2014), 377–392.
  * [7] S. Elizalde, Descents on quasi-Stirling permutations, J. Combin. Theory Ser. A, 180 (2021), 105429.
  * [8] D. Foata and M.-P. Schüzenberger, Théorie Géométrique des Polynômes Eulériens, Lecture Notes in Mathematics, Vol. 138, Springer-Verlag, Berlin, 1970.
  * [9] I.M. Gessel, R.P. Stanley, Stirling polynomials, J. Combin. Theory Ser. A, 24(1978), 24–33.
  * [10] Z. Lin, J. Ma, P.B. Zhang, Statistics on multipermutations and partial $\gamma$-positivity, J. Combin. Theory Ser. A, 183 (2021), 105488.
  * [11] I.M. Gessel, Z. Lin, J. Zeng, Jacobi-Stirling polynomials and P-partitions, European J. Combin., 33 (2012), 1987–2000.
  * [12] S. Liu, MacMahon’s equidistribution theorem for $k$-Stirling permutations, Adv. Appl. Math., 128(2021), 102193.
  * [13] S. Ma, J. Ma, Y. Yeh, $\gamma$-positivity and partial $\gamma$-positivity of descent-type polynomials, J. Combin. Theory Ser. A, 167 (2019), 257–293.
  * [14] S. Park, Inverse descents of $r$-multipermutations, Discrete Math., 132(1994), 215–229.
  * [15] S. Park, The r-multipermutations, J. Combin. Theory Ser. A, 67(1994), 44–71.
  * [16] S.H.F. Yan, X. Zhu, Quasi-Stirling polynomials on multisets, arXiv: math.CO 2106.04347.
  * [17] S.H.F. Yan, L. Yang, Y. Huang, X. Zhu, Quasi-Stirling permutations on multisets, arXiv: math.CO 2106.04348.