# Simple proofs of certain inequalities with logarithmic coefficients of
univalent functions

Milutin Obradović  Department of Mathematics, Faculty of Civil Engineering,
University of Belgrade, Bulevar Kralja Aleksandra 73, 11000, Belgrade, Serbia.
<EMAIL_ADDRESS>and  Nikola Tuneski  Department of Mathematics and
Informatics, Faculty of Mechanical Engineering, Ss. Cyril and Methodius
University in Skopje, Karpoš II b.b., 1000 Skopje, Republic of North
Macedonia<EMAIL_ADDRESS>
###### Abstract.

In this paper, we give simple proofs for the bounds (some of them sharp) of
the difference of the moduli of the second and the first logarithmic
coefficient for the general class of univalent functions and for the class of
convex univalent functions.

###### Key words and phrases:

univalent functions, convex functions, logarithmic coefficient, moduli,
estimate

###### 2020 Mathematics Subject Classification:

30C45, 30C50

##  1\. Introduction and definitions

Let ${\mathcal{A}}$ be the class of functions $f$ analytic in the open unit
disk ${\mathbb{D}}=\\{z:|z|<1\\}$ and normalized such that
$f(z)=z+a_{2}z^{2}+a_{3}z^{3}+\cdots$. If additionally, $f$ is one-on-one and
onto, we say that it is univalent and denote with ${\mathcal{S}}$ the
corresponding subclass of ${\mathcal{A}}$ containing all such functions.

One of the most important results of the twentieth century, the Bieberbach
conjecture is related with this class, and says that for all univalent
functions, $|a_{n}|\leq n$ for all positive integers $n$. It was formulated in
1916 ([1]) and proven in 1985 by de Branges [2]. Huge part of the theory of
univalent functions is oriented towards finding estimates (preferably sharp)
of expressions involving moduli of coefficients $a_{n}$. More details can be
found in [4]. In parallel, significant attention is given to expressions
involving so-called logarithmic coefficients, $\gamma_{n}$, $n=1,2,\ldots$,
defined by

(1) $\log\frac{f(z)}{z}=2\sum_{n=1}^{\infty}\gamma_{n}z^{n}.$

The expressions are studied over the general class ${\mathcal{S}}$ of
univalent functions, or over its subclasses (starlike, convex, close-to-
convex, et c.).

In this paper we give estimates of the difference of the moduli of the second
and the first logarithmic coefficient,

$|\gamma_{2}|-|\gamma_{1}|,$

for the general class of univalent functions and for the class of convex
functions. In the first case the estimate is sharp, while in the second is
partly sharp. The class of convex functions, ${\mathcal{K}}$, consists of
functions $f$ from ${\mathcal{A}}$ that map the open unit disk ${\mathbb{D}}$
onto a convex domain. These functions are univalent.

The result for the general class of univalent functions is sharp and was
previously obtained in [3, Theorem 3.1] and is rather complicated. Here we
give a simple proof. The result for the class of convex functions is partly
sharp.

Before continuing, let note that by equating the coefficients on both sides of
(1) we receive

(2)
$\gamma_{1}=\frac{a_{2}}{2}\quad\text{and}\quad\gamma_{2}=\frac{1}{2}\left(a_{3}-\frac{a_{2}^{2}}{2}\right).$

##  2\. The result for the general class ${\mathcal{S}}$

In [3] the authors used robust technique to obtain sharp estimate of
$|\gamma_{2}|-|\gamma_{1}|$ for the general class of univalent functions. Here
we will prove the same using elementary technique and the well known sharp
inequality holding for all $f$ from ${\mathcal{S}}$:

(3) $|a_{3}-a_{2}^{2}|\leq 1.$

For its proof see [4, p.5].

######  Theorem 1.

For every function $f\in{\mathcal{S}}$,
$-\frac{\sqrt{2}}{2}\leq|\gamma_{2}|-|\gamma_{1}|\leq\frac{1}{2}$ holds
sharply.

###### Proof.

From the Bieberbach conjecture, for all $f\in{\mathcal{S}}$, we have
$\frac{1}{2}|a_{2}|\leq 1$, and further, applying (3) in the final step, we
have

$\begin{split}|\gamma_{2}|-|\gamma_{1}|&=\frac{1}{2}\left|a_{3}-\frac{a_{2}^{2}}{2}\right|-\frac{1}{2}|a_{2}|\\\
&\leq\frac{1}{2}\left|a_{3}-\frac{a_{2}^{2}}{2}\right|-\frac{1}{2}|a_{2}|\left(\frac{1}{2}|a_{2}|\right)\\\
&\leq\frac{1}{2}\left(\left|a_{3}-\frac{a_{2}^{2}}{2}\right|-\frac{1}{2}|a_{2}|^{2}\right)\\\
&\leq\frac{1}{2}\left|\left(a_{3}-\frac{a_{2}^{2}}{2}\right)-\frac{1}{2}a_{2}^{2}\right|\\\
&=\frac{1}{2}\left|a_{3}-a_{2}^{2}\right|\leq\frac{1}{2}.\end{split}$

The estimate is sharp since equality is attained for the function

$f_{1}(z)=\frac{z}{1+e^{i\theta}z^{2}}=z-e^{i\theta}z^{3}+e^{2i\theta}z^{5}+\cdots.$

The second inequality that is to be proven,
$|\gamma_{2}|-|\gamma_{1}|\geq-\frac{\sqrt{2}}{2}$, is equivalent to
$\frac{1}{2}\left|a_{3}-\frac{a_{2}^{2}}{2}\right|-\frac{1}{2}|a_{2}|\geq-\frac{\sqrt{2}}{2}$,
i.e., to

(4) $\left|a_{3}-\frac{a_{2}^{2}}{2}\right|\geq|a_{2}|-\sqrt{2}.$

If $|a_{2}|<\sqrt{2}$, then (4) is obviously true. For the remaining case,
when $\sqrt{2}\leq|a_{2}|\leq 2$, we have

$\begin{split}\left|a_{3}-\frac{a_{2}^{2}}{2}\right|&=\left|\left(a_{3}-a_{2}^{2}\right)+\frac{a_{2}^{2}}{2}\right|\geq\frac{|a_{2}|^{2}}{2}-\left|a_{3}-a_{2}^{2}\right|\geq\frac{|a_{2}|^{2}}{2}-1\geq|a_{2}|-\sqrt{2}.\end{split}$

The last inequality holds since it is equivalent to
$(|a_{2}|-\sqrt{2})(|a_{2}|+\sqrt{2}-2)\geq 0$, and also $|a_{2}|-\sqrt{2}\geq
0$ and $|a_{2}|+\sqrt{2}-2>0$. The estimate is sharp with the equality
attained for the function

$f(z)=\frac{z}{1-\sqrt{2}e^{i\theta}z+e^{2i\theta}z^{2}}=z+\sqrt{2}e^{i\theta}z^{2}+e^{2i\theta}z^{3}+\cdots$

for which $|a_{2}|=\sqrt{2}$ and $|a_{3}-a_{2}^{2}|=1$. ∎

##  3\. The result for the class ${\mathcal{K}}$

Now we will prove a partly sharp estimate of $|\gamma_{2}|-|\gamma_{1}|$ over
the class ${\mathcal{K}}$, and for that we will make use of the following
result due to [5].

######  Lemma 1.

For all functions $f(z)=z+a_{2}z^{2}+a_{3}z^{3}+\cdots$ from ${\mathcal{K}}$,

$\left|a_{3}-a_{2}^{2}\right|\leq\frac{1}{3}\left(1-|a_{2}|^{2}\right).$

The inequality is sharp with extremal function

$f_{\lambda}(z)=\int_{0}^{z}\left(\frac{1-t}{1+t}\right)^{\lambda}\frac{1}{1-t^{2}}\,dt=z+\lambda
z^{2}+\frac{1}{3}\left(2\lambda^{2}+1\right)z^{3}+\cdots,$

with $0\leq\lambda\leq 1$.

Note that the range of $\lambda$ for the extremal function exploits the fact
that all coefficients in the expansion of convex univalent functions have
modulus less or equal to 1.

######  Theorem 2.

For every function $f\in{\mathcal{K}}$,
$-\frac{1}{\sqrt{10}}\leq|\gamma_{2}|-|\gamma_{1}|\leq\frac{1}{6}$. The second
inequality is sharp.

###### Proof.

From Lemma 1 we have

$\begin{split}|\gamma_{2}|-|\gamma_{1}|&=\frac{1}{2}\left|a_{3}-\frac{a_{2}^{2}}{2}\right|-\frac{1}{2}|a_{2}|\\\
&=\frac{1}{2}\left|\left(a_{3}-a_{2}^{2}\right)+\frac{a_{2}^{2}}{2}\right|-\frac{1}{2}|a_{2}|\\\
&\leq\frac{1}{2}\left|a_{3}-a_{2}^{2}\right|+\frac{1}{4}|a_{2}|^{2}-\frac{1}{2}|a_{2}|\\\
&\leq\frac{1}{6}\left(1-|a_{2}|^{2}\right)+\frac{1}{4}|a_{2}|^{2}-\frac{1}{2}|a_{2}|\\\
&\leq\frac{1}{12}\left(|a_{2}|^{2}-6|a_{2}|+2\right)\\\
&\leq\frac{1}{6},\end{split}$

since $|a_{2}|\leq 1$ for $f\in{\mathcal{K}}$. Equality holds for the function
$f_{0}(z)=z+\frac{1}{3}z^{3}+\cdots$, where $f_{0}$ is given in Lemma 1 for
$\lambda=0$.

For the remaining case we need to prove

$|\gamma_{2}|-|\gamma_{1}|=\frac{1}{2}\left|a_{3}-\frac{a_{2}^{2}}{2}\right|-\frac{1}{2}|a_{2}|\geq-\frac{1}{\sqrt{10}},$

i.e.,

(5) $\left|a_{3}-\frac{a_{2}^{2}}{2}\right|\geq|a_{2}|-\frac{2}{\sqrt{10}}.$

If $|a_{2}|<\frac{2}{\sqrt{10}}$, the above obviously holds, while for
$\frac{2}{\sqrt{10}}\leq|a_{2}|\leq 1$, we have

$\begin{split}\left|a_{3}-\frac{a_{2}^{2}}{2}\right|&=\left|\left(a_{3}-a_{2}^{2}\right)+\frac{a_{2}^{2}}{2}\right|\\\
&\geq\frac{|a_{2}|^{2}}{2}-\left|a_{3}-a_{2}^{2}\right|\\\
&\geq\frac{|a_{2}|^{2}}{2}-\frac{1}{3}\left(1-|a_{2}|^{2}\right)\\\
&=\frac{5}{6}|a_{2}|^{2}-\frac{1}{3}\\\
&\geq|a_{2}|-\frac{2}{\sqrt{10}}.\end{split}$

The last inequality holds since it is equivalent to
$5|a_{2}|^{2}-6|a_{2}|+\frac{12}{\sqrt{10}}-2\geq 0$ which is easily verified
to be true. ∎

######  Remark 1.

We were not able to prove sharpness of the left estimate in the previous
theorem and it remains an open problem.

## References

  * [1] L. Bieberbach. Über die Koeffizienten derjenigen Potenzreihen, welche eine schlichte Abbildung des Einheitskreises vermitteln. Sitzungsber. Preuss. Akad. Wiss. Phys-Math., 138:940– 955, 1916.
  * [2] L. de Branges, A proof of the Bieberbach conjecture. Acta Math., 154(1-2):137–152, 1985.
  * [3] A. Lecko, D. Partyka, Successive Logarithmic Coefficients of Univalent Functions, Computational Methods and Function Theory, (2023). https://doi.org/10.1007/s40315-023-00500-9
  * [4] D.K. Thomas, N. Tuneski, and A. Vasudevarao, Univalent Functions: A Primer, De Gruyter Studies in Mathematics 69, De Gruyter, Berlin, Boston, 2018.
  * [5] S.Y. Trimble, A Coefficient Inequality for Convex Univalent Functions, Proceedings of the American Mathematical Society, Vol. 48, No. 1 (1975), 266-267.


# HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network
that Achieves over 0.9 Mean Dice and 86 FPS

Chien-Hsiang Huang  Hung-Yu Wu  Youn-Long Lin

###### Abstract

We propose a new convolution neural network called HarDNet-MSEG for polyp
segmentation. It achieves the SOTA in both accuracy and inference speed on
five popular datasets (Kvasir-SEG[22], CVC-ColonDB[19], EndoScene[31], ETIS-
Larib Polyp DB[30] and CVC-Clinic DB[5]). For Kvasir-SEG, HarDNet-MSEG
delivers 0.904 mean Dice running at 86.7 FPS on a GeForce RTX 2080 Ti GPU
(showing in Figure 1). It consists of a backbone and a decoder. The backbone
is a low memory traffic CNN called HarDNet68[7], which has been successfully
applied to various CV tasks including image classification, object detection,
multi-object tracking ,and semantic segmentation, etc. The decoder part is
inspired by the Cascaded Partial Decoder[32], known for fast and accurate
salient object detection. We have evaluated HarDNet-MSEG using those five
popular datasets. The code and all experiment details are available at Github.
https://github.com/james128333/HarDNet-MSEG

Figure 1: Mean Dice accuracy vs frame rate running on a GeForce RTX 2080 Ti
GPU as reported in [20](blue) and [13](orange). HarDNet-MSEG is faster and
more accurate than the SOTA (U-Net[ResNet34] and PraNet).

##  1 Introduction

The incidence of colorectal cancer (CRC) has been ranked third in the world
for many years. Therefore, how to prevent CRC is an important global issue.
Studies have pointed out that 95% of CRC is due to a colorectal adenomatous
polyp. The resection of colorectal adenomatous polyps can greatly reduce the
incidence of CRC. Therefore, it is very important to have a colonoscopy on a
regular basis as well as early invention and treatment.

At present, the best way to prevent CRC is by taking regular colonoscopy and
undergo a polyp removal resection. With the emergence and popularization of
painless colonoscopy, people’s acceptance of the examination is getting
higher. However, the detection of polyps was performed manually by
endoscopists in the past, which is a consuming task for human beings and
greatly depends on the doctor’s experience and ability. Early segmentation
methods[2, 19, 4] are based on extracting features such as color, patterns,
etc., and then using a classifier to distinguish polyps from their
surroundings. However, this method still has a high rate of missed detection.
The position, size, color, etc. of each polyp are different, so it is very
difficult to segment them automatically and accurately.

In recent years, CNN has grown rapidly with breakthrough growth in the
application of various imaging tasks. The segmentation of polyp have also
benefited[6, 1]. For this task, FCN[27, 6, 1], U-Net[28, 29], U-Net++[24, 36],
DoubleU-Net[21] and ResUNet[23, 33, 20] series, etc., have good results
compared to the early methods. Most polyp blocks can be segmented out well,
but there are still many problems, such as the cutting of boundary areas and
the lack of smaller blocks, as well as broken images in large areas. Moreover,
the inference time of these networks is usually long, and the training time is
relatively time-consuming.

We propose HarDNet-MSEG based on the backbone of HarDNet68[7]. With a simple
encoder-decoder[3] architecture, it achieves excellent accuracy and efficient
inference time for related benchmarks such as Kvasir-SEG, CVC-ColonDB, etc.

##  2 Related work

Since the emergence of LeNet[25] in 1998, CNN has grown rapidly and has been
used in different computer vision fields. Among them, the task of image
segmentation is widely used in medical imaging.

In 2015, Long et al. first introduced fully convolutional networks (FCN)[27]
for the task of image segmentation. An end-to-end trained convolutional neural
network is used to classify each pixel in an image. Since then, the
convolutional neural network has flourished in the field of image
segmentation. In the same year, U-Net[29] introduced at MICCAI has been widely
used in the field of medical imaging. Through a fairly symmetrical U-shaped
encoder-decoder architecture, combined with skip connections at different
scales to integrate deep and shallow features, it has now become a baseline
network architecture for most medical imaging semantic segmentation. Then, the
emergence of U-Net++[24, 36] expands the original U-shaped architecture. With
more skip connections and convolutions to achieve the effect of deep layer
aggregation[34]. It solves the problem that edge information and small objects
are easily lost due to deep network down-sampling and up-sampling.

In recent years, the use of a better CNN backbone, or the introduction of
additional modules like spatial pyramid pooling[16], attention modules[8, 15],
etc., have achieved very good results in medical imaging semantic
segmentation. Examples of the former include ResUNet[33, 20], ResUNet++[23],
and DoubleU-Net[21]. By integrating a better CNN backbone with a U-shaped
structure, the entire network has a stronger recognition capability, a larger
receiving domain, and multi-scale information integration. The second is to
insert additional modules, such as DoubleU-Net[21] uses ASPP[9] between the
encoder and the decoder, which helps to deal with different object scales and
improve accuracy; PraNet[13] adds an RFB[26] module to skip connection to
capture more visual information for features of different scales. In recent
years, attention has also been widely used in the field of computer vision,
especially for semantic segmentation which requires detailed edge information
at the pixel level. Examples include PraNet[13], PolypSeg[35] and ABC-Net[14].
After adding different context modules, they all get good results in medical
imaging segmentation. Context modules such as Spatial Attention Module[10] and
Channel Attention Module[10] will reduce the inference speed, but on the other
hand, they are very efficient in improving accuracy and making edge cutting
more precise.

The HarDNet-MSEG we proposed uses HarDNet[7] as the backbone and is designed
with an encoder-decoder architecture. It has achieved the high accuracy of the
current state of the art in CVC-ColonDB, EndoScene, ETIS-Larib Polyp DB,
CVCClinic DB, and Kvasir-SEG, and at the same time has an efficient inference
speed. In addition, we have also tried to add additional modules such as RFB,
ASPP, Attention, etc. to our network architecture to further improve the
accuracy.

Figure 2:  HarDNet-MSEG overview. The encoder part consists of HarDNet68, and
the decoder part is using partial decoder. Figure 3: HarDNet Block overview.

##  3 HarDNet-MSEG

Figure 2 depicts the architecture of our proposed HarDNet-MSEG. It consists of
an encoder backbone and a decoder.

###  3.1 Backbone : HarDNet

HarDNet[7], improved the original dense block of Densenet[18] are illustrated
in Figure 3. Considering the impact of memory traffic on model design, it
reduces shortcuts to increase the inference speed, and at the same time
increases its channels’ width for the key layer to make up for the loss of
accuracy. It also uses a small amount of Conv1x1 to increase computational
density.

Through this design, it not only achieves 30% inference time reduction
compared with DenseNet[18] and ResNet[17], also having higher accuracy on
ImageNet[12]. On the other hand, FC-HarDNet70[18] also reaches the state of
the art in image segmentation on Cityscapes Dataset[11]. Therefore, we use
HarDNet68 as the model backbone for Colorectal Polyps image semantic
segmentation.

###  3.2 Cascaded Partial Decoder

Many well-known medical image segmentation networks are often modified based
on the U-Net. Our design also went in this direction at the beginning. But
based on the balance of the inference time and performance, we did not use
HarDBlock (HarDBlk) in the Decoder part, which is different from FC-HarDNet.

We reference the Cascaded partial decoder [32]. It found out that the shallow
features have high resolution and occupy computing resources, and the deep
information can also represent the spatial details of the shallow information
relatively well. So we decide to discard the shallow features and do more
computing on the deeper layers’ features. At the same time, the aggregation of
feature maps at different scales can be achieved by adding appropriate
convolution and skip connections.

Figure 4: RFB Module overview.

####  3.2.1 RFB Module

Figure 4 shows a Receptive Field Block[26]. It can strengthen the deep
features learned from a lightweight CNN backbone. By using multi-branch with
different kernel size convolution and dilated convolution layers, it generates
features with the different receptive fields. Afterwards, it applies a 1x1
convolution to merge these features and generate the final representation.

We add this module to the skip connection according to [32], so that we could
enlarge our receptive fields from each different resolutions’ feature maps.

####  3.2.2 Dense Aggregation

We perform aggregation by element-wise multiplication shown in Figure 5. After
up-sampling to the same scale, the feature is multiplied with another input
feature of the corresponding scale.

Figure 5: Aggregation Module overview.

##  4 Experiments

We used the training data from [13] and [20] for training because they have
excellent performance in polyp segmentation. The training data and training
methods used in the two articles are different. In order to reduce the
variable factors, the training methods we use will refer to the methods
proposed in the two articles respectively, and then compare the accuracy and
inference speed with other models.

| mIoU | mDice | F2-score | Precision | Recall | Overall Acc. | FPS
---|---|---|---|---|---|---|---
U-Net | 0.471 | 0.597 | 0.598 | 0.672 | 0.617 | 0.894 | 11
ResUNet | 0.572 | 0.690 | 0.699 | 0.745 | 0.725 | 0.917 | 15
ResUNet++ | 0.613 | 0.714 | 0.720 | 0.784 | 0.742 | 0.917 | 7
FCN8 | 0.737 | 0.831 | 0.825 | 0.882 | 0.835 | 0.952 | 25
HRNet | 0.759 | 0.845 | 0.847 | 0.878 | 0.859 | 0.952 | 12
DoubleUNet | 0.733 | 0.813 | 0.820 | 0.861 | 0.840 | 0.949 | 7.5
PSPNet | 0.744 | 0.841 | 0.831 | 0.890 | 0.836 | 0.953 | 17
DeepLabv3+[ResNet50] | 0.776 | 0.857 | 0.855 | 0.891 | 0.8616 | 0.961 | 28
DeepLabv3+[ResNet101] | 0.786 | 0.864 | 0.857 | 0.906 | 0.859 | 0.961 | 17
U-Net[ResNet34] | 0.810 | 0.876 | 0.862 | 0.944 | 0.860 | 0.968 | 35
HarDNet-MSEG | 0.848 | 0.904 | 0.915 | 0.907 | 0.923 | 0.969 | 86.7

Table 1: Quantitative results on Kvasir dataset (training/testing
split:880/120). Showing the performance of different metrics and inference
speed evaluating on GeForce RTX 2080 Ti GPU. Others evaluation scores are
refer from [20].

###  4.1 Dataset

We used the datasets proposed in the two papers mentioned earlier, namely
Kvasir-SEG, CVC-ColonDB, EndoScene, ETIS-Larib Polyp DB, and CVC-Clinic DB.
And we will make a detailed comparison with other SOTA models on these
datasets.

###  4.2 Training setting and policy

The two articles are based on different splitting method of training data, so
we made two different experiments on each training setting to compare, and the
details of the experiments will be explained below.

For [20], 880 images of Kvasir-SEG is used for training, and the other 120
images are used for testing. It does use augmentations like random rotation,
horizontal flip, vertical flip. Our training input size is 512x512. We train
our model with SGD optimizer for 100 epochs and the learning rate is set to
1e-2. The results comparing to [20] is in Table 1. HarDNet-MSEG shows the
greatest accuracy on most metrics, and the inference speed is much faster than
others.

In [13], 1450 training images without any augmentation is used, including 900
images in Kvasir-SEG and 550 images in CVC-ClinicDB. And the testing set has 5
datasets mentioned above. Our training input size is 312x312, We train our
model with Adam optimizer for 100 epochs and the learning rate is set to 1e-4.
The quantitative results of each 5 datasets are shown in Table 2 (Kvasir-SEG)
and Table 3 (ETIS, CVC-ClinicDB, CVC-ColonDB and EndoScene). We achieve the
best performance in mean Dice and mIoU on each dataset, with the fastest
inference speed (88 FPS).

| mDice | mIoU | wfm | Sm | MAE | maxEm | FPS
---|---|---|---|---|---|---|---
U-Net | 0.818 | 0.746 | 0.794 | 0.858 | 0.055 | 0.893 | 53
U-Net++ | 0.821 | 0.743 | 0.808 | 0.862 | 0.048 | 0.910 | 25
ResUNet-mod | 0.791 | n/a | n/a | n/a | n/a | n/a | n/a
ResUNet++ | 0.813 | 0.793 | n/a | n/a | n/a | n/a | n/a
SFA | 0.723 | 0.611 | 0.67 | 0.782 | 0.075 | 0.849 | 40
PraNet | 0.898 | 0.840 | 0.885 | 0.915 | 0.030 | 0.948 | 66
HarDNet-MSEG | 0.912 | 0.857 | 0.903 | 0.923 | 0.025 |  0.958 | 88

Table 2: Quantitative results on Kvasir, comparing with the SOTA. Using the
same training script with the release code of PraNet. The inference speed is
testing under 312x312 resolution on GeForce RTX 2080 Ti GPU.

| ClinicDB | ColonDB | ETIS | CVC-T
---|---|---|---|---
| mDice | mIoU | mDice | mIoU | mDice | mIoU | mDice | mIoU
U-Net | 0.823 | 0.755 | 0.512 | 0.444 | 0.398 | 0.335 | 0.71 | 0.627
U-Net++ | 0.794 | 0.729 | 0.483 | 0.410 | 0.401 | 0.344 | 0.707 | 0.624
ResUNet-mod | 0.779 | n/a | n/a | n/a | n/a | n/a | n/a | n/a
ResUNet++ | 0.796 | 0.796 | n/a | n/a | n/a | n/a | n/a | n/a
SFA | 0.700 | 0.607 | 0.469 | 0.347 | 0.297 | 0.217 | 0.467 | 0.329
PraNet | 0.899 | 0.849 | 0.709 | 0.640 | 0.628 | 0.567 | 0.871 | 0.797
HarDNet-MSEG | 0.932 | 0.882 | 0.731 | 0.660 | 0.677 | 0.613 | 0.887 | 0.821

Table 3: More results on CVC-ClinicDB, CVC-ColonDB, ETIS, and CVC-T, comparing
with the SOTA. Among them, CVC-T is the testing data for EndoScene.

###  4.3 Metrics

Mean Dice $=\frac{2*tp}{2*tp+fp+fn}$ mIoU $=\frac{tp}{tp+fp+fn}$
Recall $=\frac{tp}{tp+fn}$ Precision $=\frac{tp}{tp+fp}$
F2 $=\frac{5p*r}{4p+r}$ Acc. $=\frac{tp+tn}{tp+tn+fp+fn}$

We will mainly use Kvasir’s official website as the basis for comparison,
namely mean Dice and Mean IoU, but we will still use other metrics mentioned
in these two articles for comparison so that we can show our advantages more
clearly.

###  4.4 Training and Inference Environment setting:

In order to show our advantage in speed, we respectively compare with other
famous models. The platforms we use for evaluation is written below.

Intel i9-9900k CPU, GeForce RTX 2080 Ti, Pytorch: 1.6 and CUDA: 10.2

##  5 Conculsion

HarDNet-MSEG achieved the SOTA in all five challenging datasets. It is the
only network that has achieved over 0.90 mean Dice (0.912 comparing with [13]
and 0.904 comparing with [20]) on Kvasir-SEG. And it is 1.3 times faster than
PraNet and more than 2 times faster than other models. We achieve this with a
simple encoder-decoder architecture without any attention module used in [13]
and [14]. See Figure 6 for some inference results of Kvasir-SEG. It shows that
our model outputs better boundary and the prediction is more accurate.

Again, it shows that HarDNet[7] is a great and efficient backbone in not only
classification and detection, but also medical imaging segmentation. We hope
this study can help pushing the frontier of medical imaging and contribute to
the application of CNN in this field.

Figure 6: Inference results of Kvasir-SEG.

## Acknowledgements

This research is supported in part by a grant from the Ministry of Science and
Technology (MOST) of Taiwan. We thank National Center for High-performance
Computing (NCHC) for providing computational and storage resources. Without it
this research is impossible. We would also like to thank Mr.Ping Chao for many
fruitful discussions.

## References

  * [1] Mojtaba Akbari, Majid Mohrekesh, Ebrahim Nasr-Esfahani, SM Reza Soroushmehr, Nader Karimi, Shadrokh Samavi, and Kayvan Najarian.  Polyp segmentation in colonoscopy images using fully convolutional network.  In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 69–72. IEEE, 2018\.
  * [2] Mamonov A.V., Figueiredo I.N.and Figueiredo P.N., and Tsai Y.H.R.  Automated polyp detection in colon capsule endoscopy.  IEEE TMI 33(7), 1488–1502, 2014.
  * [3] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE transactions on pattern analysis and machine intelligence, 39(12):2481–2495, 2017.
  * [4] Seung-Hwan Bae and Kuk-Jin Yoon.  Polyp detection via imbalanced learning and discriminative feature learning.  IEEE TMI 34(11), 2379–2393, 2015.
  * [5] Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparrach, Debora Gil, Cristina Rodríguez, and Fernando Vilariño.  Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians.  Computerized Medical Imaging and Graphics, 43:99–111, 2015.
  * [6] Patrick Brandao, Evangelos Mazomenos, Gastone Ciuti, Renato Caliò, Federico Bianchi, Arianna Menciassi, Paolo Dario, Anastasios Koulaouzidis, Alberto Arezzo, and Danail Stoyanov.  Fully convolutional neural networks for polyp segmentation in colonoscopy.  In Medical Imaging 2017: Computer-Aided Diagnosis, volume 10134, page 101340F. International Society for Optics and Photonics, 2017.
  * [7] Ping Chao, Chao-Yang Kao, Yu-Shan Ruan, Chien-Hsiang Huang, and Youn-Long Lin.  Hardnet: A low memory traffic network.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3552–3561, 2019.
  * [8] Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, and Alan L Yuille.  Attention to scale: Scale-aware semantic image segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3640–3649, 2016.
  * [9] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018.
  * [10] Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua.  Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5659–5667, 2017.
  * [11] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [13] Deng-Ping Fan, Ge-Peng Ji, Tao Zhou, Geng Chen, Huazhu Fu, Jianbing Shen, and Ling Shao.  Pranet: Parallel reverse attention network for polyp segmentation.  MICCAI, 2020.
  * [14] Yuqi Fang, Delong Zhu, Jianhua Yao, Yixuan Yuan, and Kai-yu Tong.  Abc-net: Area-boundary constraint network with dynamical feature selection for colorectal polyp segmentation.  IEEE Sensors Journal, 2020.
  * [15] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3146–3154, 2019.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Spatial pyramid pooling in deep convolutional networks for visual recognition.  IEEE transactions on pattern analysis and machine intelligence, 37(9):1904–1916, 2015.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [18] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [19] Tajbakhsh N.and Gurudu S.R.and Liang J.  Automated polyp detection in colonoscopy videos using shape and context information.  IEEE TMI 35(2), 630–644, 2015.
  * [20] Debesh Jha, Sharib Ali, Håvard D Johansen, Dag D Johansen, Jens Rittscher, Michael A Riegler, and Pål Halvorsen.  Real-time polyp detection, localisation and segmentation in colonoscopy using deep learning.  arXiv preprint arXiv:2011.07631, 2020.
  * [21] Debesh Jha, Michael A Riegler, Dag Johansen, Pål Halvorsen, and Håvard D Johansen.  Doubleu-net: A deep convolutional neural network for medical image segmentation.  arXiv preprint arXiv:2006.04868, 2020.
  * [22] Debesh Jha, Pia H Smedsrud, Michael A Riegler, Pål Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D Johansen.  Kvasir-seg: A segmented polyp dataset.  In International Conference on Multimedia Modeling, pages 451–462. Springer, 2020.
  * [23] Debesh Jha, Pia H Smedsrud, Michael A Riegler, Dag Johansen, Thomas De Lange, Pål Halvorsen, and Håvard D Johansen.  Resunet++: An advanced architecture for medical image segmentation.  In 2019 IEEE International Symposium on Multimedia (ISM), pages 225–2255. IEEE, 2019.
  * [24] Nicolas Boutry Le Duy Huynh.  A u-net++ with pre-trained efficientnet backbone for segmentation of diseases and artifacts in endoscopy images and videos.  ISBI 2020, 2020.
  * [25] Yann LeCun et al.  Lenet-5, convolutional neural networks.  URL: http://yann. lecun. com/exdb/lenet, 20(5):14, 2015.
  * [26] Songtao Liu, Di Huang, et al.  Receptive field block net for accurate and fast object detection.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 385–400, 2018.
  * [27] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [28] Ahmed Mohammed, Sule Yildirim, Ivar Farup, Marius Pedersen, and Øistein Hovde.  Y-net: A deep convolutional neural network for polyp detection.  arXiv preprint arXiv:1806.01907, 2018.
  * [29] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [30] Juan Silva, Aymeric Histace, Olivier Romain, Xavier Dray, and Bertrand Granado.  Toward embedded detection of polyps in wce images for early diagnosis of colorectal cancer.  International Journal of Computer Assisted Radiology and Surgery, 9(2):283–293, 2014.
  * [31] David Vázquez, Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparrach, Antonio M López, Adriana Romero, Michal Drozdzal, and Aaron Courville.  A benchmark for endoluminal scene segmentation of colonoscopy images.  Journal of healthcare engineering, 2017, 2017.
  * [32] Zhe Wu, Li Su, and Qingming Huang.  Cascaded partial decoder for fast and accurate salient object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3907–3916, 2019.
  * [33] Xiaofei Yang, Xutao Li, Yunming Ye, Raymond YK Lau, Xiaofeng Zhang, and Xiaohui Huang.  Road detection and centerline extraction via deep recurrent convolutional neural network u-net.  IEEE Transactions on Geoscience and Remote Sensing, 57(9):7209–7220, 2019.
  * [34] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell.  Deep layer aggregation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2403–2412, 2018.
  * [35] Jiafu Zhong, Wei Wang, Huisi Wu, Zhenkun Wen, and Jing Qin.  Polypseg: An efficient context-aware network for polyp segmentation from colonoscopy videos.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 285–294. Springer, 2020.
  * [36] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang.  Unet++: A nested u-net architecture for medical image segmentation.  In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pages 3–11. Springer, 2018.


textures D103 and D111, selecting just 3 features for the MSM
($f_{3}({\mathbf{x}})$, $f_{4}({\mathbf{x}})$ and
$f_{5}({\mathbf{x}})$121212Note that $f_{3}({\mathbf{x}})$ and
$f_{4}({\mathbf{x}})$ are equal to the classical definition of sample skewness
and kurtosis, respectively.) and the proposed $\text{DF}_{\text{MSM}}$
($\hat{f}_{3}({\mathbf{x}})$, $\hat{f}_{4}({\mathbf{x}})$ and
$\hat{f}_{5}({\mathbf{x}})$) sets. We include the projections onto the three
orthogonal planes. The decoupling between every pair of features is apparent,
specially for the (3,5)-th and (4-5)-th order cases.

Figure 11: Illustrating the decoupling effect in the feature space on two
classes of texture patches. Left: Using classical MSM features; Right: Using
proposed $\text{DF}_{\text{MSM}}$ features.

We used two classifiers: a naïve (univariate) Gaussian and a parameter-
optimized Support Vector Machine (SVM) using Radial Basis Functions131313We
thank the authors of the Pattern Recognition Toolbox
(http://covartech.github.io), which we used in preliminary experiments.. We
applied cross validation with 4 folds, and averaged 8 runs for each result.
Figure 12 shows the test classification results as a function of the order of
the moments included in the feature’s set for the three compared sets and the
two classifiers (see legend). We observe a totally different behavior between
using classical MSM and modified [16] moments, vs. the marginally decoupled
ones $\text{DF}_{\text{MSM}}$: whereas the former achieve their optima when
using only variance ($f_{2}({\mathbf{x}})$), skewness ($f_{3}({\mathbf{x}})$)
and kurtosis ($f_{4}({\mathbf{x}})$), roughly achieving 3% and 2% error ratios
for naïve and SVM, respectively, the latter keep on decreasing the error when
adding higher-order features, reaching $1.34\%$ and $0.86\%$, respectively,
for $n=10$. We see how [16] produces just a marginal improvement. It is also
very significant how, for MSM, the naïve method behaves very differently from
SVM, whereas, for $\text{DF}_{\text{MSM}}$, results of SVM and naïve
classifiers run in close parallel. We believe this is due to the strong
assumption made by the naïve method (namely, that features are mutually
independent), which holds approximately true after the feature decoupling (as
shown in Table 3), but not before.

Figure 12: Texture patch classification, 1st experiment. Test classification
error (%) as a function of the order of the moments included in the feature’s
set.

####  6.4.2 Texture classification based on jointly decoupled moments in a
filter bank

In these experiments we compared the performance of a classifier trained using
three different sets of features, namely: (i) MSM measured at the output of a
filter bank; (ii) the decoupled standardized moments $\text{DF}_{\text{MSM}}$
(same as in the previous experiment), without considering the coupling caused
by the filter bank; and (iii) fully decoupled features,
$\text{DF}_{\text{MF}}$, using as original features marginal moments at the
outputs of the filter bank. The three sets were obtained from order 2 to 6.
The first 20 textures from the Broadtz database [35] were taken. The upper
left quarter of every texture was normalized and divided into 25 disjoint,
$64\times 64$ pixel, patches. The problem consisted of classifying these
patches into one of the 20 texture classes. To extract the features that
characterized every patch we first applied the TILs representation, using the
9 subbands (thus including the low-pass residual, not used in previous
section). Then, for every subband, the above described sets of 45 features (5
moments $\times$ 9 subbands) were calculated. We used two classifiers: a Naïve
Bayes and a parameter-optimized Support Vector Machine (SVM) using Gaussian
kernels [42]. We applied cross validation with 10 folds, and averaged 200 runs
for each result. Table V shows the mean $\pm$ standard deviation test
classification error across runs when using the MSM (1st column), the
partially decoupled $\text{DF}_{\text{MSM}}$ [6] (2nd column), and the fully
decoupled features $\text{DF}_{\text{MF}}$ (3rd column). First row shows
results using Naïve Bayes, and second using SVM.

TABLE V:  Texture patch classification, 2nd experiment. Test classification
error (%).

| MSM |  $\text{DF}_{\text{MSM}}$ [6] | $\text{DF}_{\text{MF}}$
---|---|---|---
Naïve Bayes | 9.1 $\pm$ 0.4 | 7.4 $\pm$ 0.3 | 4.0 $\pm$ 0.3
SVM | 5.2 $\pm$ 0.5 | 4.5 $\pm$ 0.4 | 3.1 $\pm$ 0.4

We observed that the probability of error is greatly reduced when using
$\text{DF}_{\text{MSM}}$ in comparison with MSM. In addition, fully decoupled
features $\text{DF}_{\text{MF}}$ also significantly improved the results of
the partially decoupled set [6]. Using the Naïve Bayes classifier, we see that
the error obtained by the fully decoupled set $\text{DF}_{\text{MF}}$
decreased the error in a factor of 0.54 and 0.44 w.r.t.
$\text{DF}_{\text{MSM}}$ and original MSM sets, respectively. We also did the
experiment of incrementally increasing the highest order in the set of
features, from just the variance to including all orders from 2 to 6.
Significantly, the error was minimized when including high order
$\text{DF}_{\text{MF}}$, up to 6, while for the MSM case the minimum error was
obtained when including features up to the fourth order (kurtosis). This
result is consistent with the one reported in Fig.12.

It is especially relevant, at this point, to recall that the MF set (for
orders above 2) does not fulfill Frobenius condition (as mentioned in Section
4). This implies that decoupling solutions do not even exist within the
theoretical framework presented here. Nevertheless, Algorithm 3 (NeN, narrow
path) provides a well-defined transformation that, as empirically shown in
Section 6.2.2 for the case of the second-order moments at the output of a
filter bank, greatly reduces the amount of coupling between the features’
gradients. This translates here into a substantial performance boost when
applied to texture classification.

##  7 Conclusions and Future Work

We have presented a new mathematical and algorithmic framework for, given a
set of differentiable functions acting as global data descriptors, obtaining a
closely related set such that its gradients are mutually orthogonal. We have
set the conditions under which this can be done hierarchically and
progressively, adding a new feature at a time, and devised a new family of
algorithms based on nested normalization operations. We have also studied the
need of adding small perturbations in some cases, and devised some specific
methods for that.

We have shown, first, that the proposed method allows for locally
decorrelating features of statistical distributions, and why this has a
positive impact in discriminating close values of statistical parameters. We
have also tested empirically (with both real textured image patches and
pseudo-random numbers, for some distributions) the degree of accomplishment of
mutual decoupling (gradient orthogonality) for different global features,
obtaining results that are both practically interesting and consistent with
the theory.

We have applied our decoupling methods to marginal moments, both in the pixel
domain and at the output of a filter bank. Using the new decoupled features as
descriptors with state-of-the-art machine learning methods we have obtained a
dramatic decrease in error in regression problems (over simulated random
variables, under three different distributions) and classification (over real
textured image patches), as compared to using classical standardized moments.
It is worth noting that we obtained substantial improvements in classification
accuracy even when theoretical conditions for perfect decoupling were not met.

It is noteworthy that, applying our decoupling method to the first three raw
moments results into their standardized counterparts. In addition, we obtained
for the first time, a (quasi-analytic) normalized fourth-order moment that is
decoupled from mean, variance, and skewness, which we have termed
orthokurtosis. For higher orders, we obtained other new well-defined features,
numerically computable, which present a much-decreased amount of coupling.
Finally, although not fully developed here, natural application fields for our
method go beyond analysis and include promising style transfer and synthesis
techniques [2, 28].

In future work we want to explore the extension of the deterministic
decoupling methods for non-hierarchical sets, as well as its applicability for
improving economy and robustness in ANNs, by decoupling their features, either
at training phase or a posteriori. To conclude, a promising research area is
that of exploring actual biological mechanisms of adaptation involving
decoupling/cross-invariance, in perceptual neural science.

##  Appendix A Equilibrium points of gradient systems

In this section we study equilibrium points for the gradient system (4).

If $f$ is continuous we must have $f(\overline{\Omega})=[a,b]$ (here $a,b$
maybe $\pm\infty$ if the set is not bounded, but the analysis can be performed
similarly). We would like to see if, given $x_{0}\in\Omega$, we can reach all
values in the range of $f$ by moving in the direction of the gradient. We look
at stationary points in $\overline{\Omega}$ that are not a global maximum or
minimum points. The basin (or domain) of attraction of an equilibrium point
$\bar{\mathbf{x}}$ is the set of all initial conditions with solutions that
tend to it. Let us consider all possible cases according to the behavior of
the linearized equation ${\mathbf{x}}=-D^{2}f(\bar{\mathbf{x}}){\mathbf{x}}$
(see, for instance, the introduction in [43]). Assume that $D^{2}f$ has no
zero eigenvalues:

  * •

If the eigenvalues of the Hessian $D^{2}f$ are all strictly positive then
$\bar{\mathbf{x}}$ is a sink.

  * •

Similarly for max.

  * •

If the Hessian has a negative real eigenvalue then the equilibrium is
unstable. If all eigenvalues are nonzero, then the dimension of the unstable
manifold is equal to the number of negative eigenvalues counting multiplicity.
The dimension of the unstable manifold is the number of negative eigenvalues
counting multiplicity. The tangent spaces of these manifolds are the spans of
the corresponding eigenspaces so are orthogonal at the equilibrium point.

If some eigenvalue is zero, then the picture is much more complicated and we
will make additional assumptions in order to avoid technicalities:

  * •

If $\bar{x}$ is a local minimum of $f$ (not necessarily strict), then
$\bar{x}$ is a Lyapunov-stable equilibrium point of the gradient flow of a
real-analytic function $f$ [44, Section 3]. Thus we pose the additional
constraint that all local maxima and minima are also global.

  * •

If ${\mathbf{x}}$ is a degenerate saddle, we assume that its basin of
attraction has a lower dimension and thus, saddles can be avoided by
introducing a suitable perturbation. We will actually assume that the union of
all basins of attraction, denoted by $\Lambda$, is also lower dimensional.

##  Appendix B Frobenius Theorem

Let ${\mathcal{S}}=\\{f_{i}:\Omega\rightarrow{\mathbb{R}},i=1\dots M\\}$ be a
set of features. Fix $p$ a positive integer, $p\leq M$. A $p$-dimensional
_distribution_ $\mathcal{D}$ in $\Omega$ is a (smooth) choice of a
$p$-dimensional subspace of the linear span of the $\\{\nabla
f_{i}({\mathbf{x}})\\}_{i=1}^{M}$ for every point in ${\mathbf{x}}\in\Omega$.
We denote the plane at ${\mathbf{x}}$ by $\mathcal{D}({\mathbf{x}})$. We say
that the set of features $\mathcal{S}$ is of _maximal rank_ in $\Omega$ if the
gradients are linearly independent at every point ${\mathbf{x}}$ so that the
distribution spanned by the gradients has $p=M$. This implies, in particular,
that we must choose $\Omega$ not containing critical points of the $f_{i}$.

We say that $\mathcal{D}$ satisfies the _Frobenius condition_ if for every
$f,g\in\mathcal{S}$, writing the gradient vectors as

$X=\sum_{\ell}\partial_{x_{\ell}}f\frac{\partial}{\partial x_{\ell}},\quad
Y=\sum_{j}\partial_{x_{j}}g\frac{\partial}{\partial x_{j}},$ (34)

we have that the combination

$XY-
YX=\sum_{j,\ell}\left(\partial_{x_{\ell}}f\partial_{x_{\ell}x_{j}}g-\partial_{x_{\ell}}g\partial_{x_{j}x_{\ell}}f\right)\frac{\partial}{\partial
x_{j}}$ (35)

also belongs to $\mathcal{D}$.

The classical Frobenius theorem [45, Chapter 6] states that a distribution
$\mathcal{D}$ that satisfies the Frobenius condition in $\Omega$ is
integrable. That is, one can find a submanifold
${\mathcal{I}}({\mathbf{x}}_{0},{\mathcal{S}})$ passing by ${\mathbf{x}}_{0}$
whose tangent hyperplane at each location ${\mathbf{x}}$ is the linear span of
$\\{\nabla f_{i}({\mathbf{x}}),i=1\dots M\\}$. In addition, this submanifold
is $p$-dimensional, and $\Omega$ is foliated by these submanifolds. A
different presentation of Frobenius theorem from the dynamical systems point
of view can be found in [46, Chapter VI].

###  B.1 Back to Proposition 2.2

We give now the proof of the only if statement in Proposition 2.2. Assume
that, given set of features ${\mathcal{S}}$, there exists an invariant mapping
$\hat{\mathbf{x}}_{\mathcal{S}}$ exists, and denote by
$J:=J_{\hat{\mathbf{x}}_{\mathcal{S}}}$ its Jacobian matrix. We will show that
the gradients of those features in ${\mathcal{S}}$ satisfy the Frobenius
condition C2.

For this, it is enough to pick two any two features in ${\mathcal{S}}$, say
$f$ and $g$. By the definition of invariant mapping, we must have

$\begin{split}&J\cdot\nabla f=\textbf{0},\\\ &J\cdot\nabla
g=\textbf{0},\end{split}$

at each point in the domain (which is not written in to simplify the
notation). Differentiating both equations w.r.t. the $\ell$-th coordinate,
component by component, we obtain

$\begin{split}&\partial_{x_{\ell}}J\cdot\nabla
f+J\cdot\partial_{x_{\ell}}\nabla f=\textbf{0},\\\
&\partial_{x_{\ell}}J\cdot\nabla g+J\cdot\partial_{x_{\ell}}\nabla
g=\textbf{0},\end{split}$

from where we can isolate the second derivatives of each feature

$\begin{split}&\partial_{x_{\ell}}\nabla
f=-J^{-1}\cdot\partial_{x_{\ell}}J\cdot\nabla f,\\\ &\partial_{x_{\ell}}\nabla
g=-J^{-1}\cdot\partial_{x_{\ell}}J\cdot\nabla g.\end{split}$

Now we extract the $j$-th component of both vectors above, to have a formula
for $\partial_{x_{\ell}x_{j}}f$ and $\partial_{x_{\ell}x_{j}}g$. This is,

$\begin{split}&\partial_{x_{\ell}x_{j}}f=-\sum_{k,s}(J^{-1})_{jk}\cdot(\partial_{x_{\ell}}J)_{ks}\cdot\partial_{x_{s}}f,\\\
&\partial_{x_{\ell}x_{j}}g=-\sum_{k,s}(J^{-1})_{jk}\cdot(\partial_{x_{\ell}}J)_{ks}\cdot\partial_{x_{s}}g.\end{split}$

Next, in order to check Frobenius condition (35) we need to calculate

$\begin{split}\sum_{\ell}&(\partial_{x_{\ell}}f\partial_{x_{\ell}x_{j}}g-\partial_{x_{\ell}}g\partial_{x_{j}x_{\ell}}f)\\\
&=-\sum_{\ell,k,s}\Big{[}\partial_{x_{\ell}}f\cdot(J^{-1})_{jk}\cdot(\partial_{x_{\ell}}J)_{ks}\cdot\partial_{x_{s}}g\\\
&\qquad\quad\quad-\partial_{x_{\ell}}g\cdot(J^{-1})_{jk}\cdot(\partial_{x_{\ell}}J)_{ks}\cdot\partial_{x_{s}}f\Big{]}.\end{split}$

If one is able to interchange the indexes $s$ and $\ell$, the above quantity
is identically zero. And this is possible since, for a Jacobian matrix,

$\partial_{x_{\ell}}J_{ks}=\partial_{x_{\ell}}\partial_{x_{s}}(\hat{\mathbf{x}}_{\mathcal{S}})_{k}=\partial_{x_{s}}\partial_{x_{\ell}}(\hat{\mathbf{x}}_{\mathcal{S}})_{k}=\partial_{x_{s}}J_{k\ell}.$

We have shown that Frobenius condition (35) holds for any two features in the
set ${\mathcal{S}}$, which completes the proof.

###  B.2 Examples

####  B.2.1 Average of scalar functions

If the set $\mathcal{S}$ consists on features of the form

$f_{j}({\mathbf{x}})=\frac{1}{N}\sum_{n=1}^{N}{m_{j}(x_{n})},\quad j=1..M,$

for scalar functions $m_{j}:\mathbb{R}\to\mathbb{R}$, then it clearly
satisfies the Frobenius condition. This follows by simple inspection of (35),
because the Hessians $\partial_{x_{\ell}x_{j}}f_{i}$ are multiples of the
identity.

####  B.2.2 Second-order moments at the output of a set of filters

Another example of features that satisfy Frobenius is a set of second-order
moments measured at the output of a set of filters. To fix notation, we take
two functions

$f({\mathbf{x}})=\frac{1}{N}({\mathbf{x}}*h)^{\odot 2},\quad
g({\mathbf{x}})=\frac{1}{N}({\mathbf{x}}*h^{\prime})^{\odot 2}.$

Calculate

$\begin{split}&\partial_{x_{i}}f=\sum_{k}({\mathbf{x}}*h)_{k}h_{k-i}=\frac{2}{N}({\mathbf{x}}*(h*\tilde{h}))_{i}=\frac{2}{N}({\mathbf{x}}*\hat{h})_{i},\\\
&\partial_{x_{i}x_{j}}f=\hat{h}_{i-j},\end{split}$

where we have denoted $(\tilde{h})_{s}=h_{i-s}$, $\hat{h}=h*\tilde{h}$, and
similarly for $h^{\prime}$. Then, in the notation of (34), dropping the
multiplicative constants,

$\begin{split}XY-
YX&=\sum_{i,j,s}x_{s}\hat{h}_{i-s}\hat{h}^{\prime}_{i-j}\frac{\partial}{\partial
x_{j}}-\sum_{i,j,s}x_{s}\hat{h}^{\prime}_{i-s}\hat{h}_{i-j}\frac{\partial}{\partial
x_{j}}\\\
&=\sum_{j}\left\\{[({\mathbf{x}}*\hat{h})*\hat{h}^{\prime}]_{j}-[({\mathbf{x}}*\hat{h}^{\prime})*\hat{h}]_{j}\right\\}\frac{\partial}{\partial
x_{j}}=0,\end{split}$

as desired.

##  Appendix C Adding perturbations

###  C.1 Marginal moments

####  C.1.1 Critical points of the decoupled moments and their basins of
attraction

In order to apply the methods proposed in Section 3, conditions B1 (no local
non-global extrema) and B2 (lower dimensional basins for saddles) must hold.
Here we test this, and search for explicit formulas for all critical points of
the new decoupled features. In particular, this is essential to understand the
structure of basins of attraction and discuss the effectiveness of
perturbations.

From the method in Subsection 3.3.1, obtaining the critical points of the
decoupled features is sort of dual to finding the decoupled gradients within
their corresponding reference manifolds. Whereas for the latter we imposed
orthogonality on each new gradient with respect to the previous ones (see
Subsection 4.1.2), for finding the critical points we impose co-linearity on
each gradient with respect to the previous ones, see Eq. (18). A particular
solution corresponds to finding common critical points (through order $k$) of
the original features, where all gradients for $j\leq k$ vanish. In this case
there are no local (non-global) extrema, as gradients are made of monomials,
which either have a single minimum at zero, for even orders, or a saddle point
at zero and no extrema in ${\mathbb{R}}$ (odd orders). Therefore, the only
solution coming from the original features’ gradients vanishing corresponds to
${\mathbf{x}}_{0}^{*}={\mathbf{0}}$, i.e., at zero all decoupled moments have
a critical point, same as the original moments.

For the rest of solutions, we start by solving for the critical points of
$\hat{f}_{2}$, denoted by ${\mathbf{x}}_{1}^{*}$ 141414Although for notation
simplicity critical points here are denoted as a vector
${\mathbf{x}}_{j}^{*}$, they actually represent sets of vectors., in the
equation $\nabla
f_{2}({\mathbf{x}}_{1}^{*})=\lambda_{1,2}f_{1}({\mathbf{x}}_{1}^{*})$, which,
substituting its corresponding expressions, gives us
${\mathbf{x}}_{1}^{*}=c{\mathbf{1}},\,c\in{\mathbb{R}}$. No surprisingly,
constant signals provide the minimal (zero) variance, the only extreme value
of this feature. Moreover, as explained in Subsection 3.3.1, decoupled
features are not defined at ${\mathbf{x}}_{1}^{*}$ (being the sample variance
null it can not be normalized to one). This manifold only intersects
${\mathcal{R}}_{1}$ in ${\mathbf{x}}={\bf 0}$, and has no intersections with
${\mathcal{R}}_{2}$ and subsequent. Next, the calculation of the critical
points of $\hat{f}_{3}$ comes from solving for ${\mathbf{x}}_{2}^{*}$ in
$f_{3}({\mathbf{x}}_{2}^{*})=\lambda_{1,3}f_{1}({\mathbf{x}}_{2}^{*})+\lambda_{2,3}f_{2}({\mathbf{x}}_{2}^{*})$.
This is a quadratic equation, whose solution is a vector
${\mathbf{x}}_{2}^{*}$ made of only two arbitrary values, repeated in
arbitrary proportions ($p$ and $1-p$), for all the coefficients. If we impose,
in addition, $f_{1}({\mathbf{x}}_{2}^{*})=0$ and
$f_{2}({\mathbf{x}}_{2}^{*})=1$ (conditions of
${\mathbf{x}}_{2}^{*}\in{\mathcal{R}}_{2}$), we obtain, after some operations,
that the only possible values of the coefficients, for ${\mathbf{x}}_{2}^{*}$
in ${\mathcal{R}}_{2}$, are:

$\displaystyle x_{2,1}^{*}$ $\displaystyle=$ $\displaystyle\sqrt{p}/(1-p)$
$\displaystyle x_{2,2}^{*}$ $\displaystyle=$ $\displaystyle-\sqrt{1-p}/p,$
(36)

where $0<p<1$ denotes the proportion of the first value. From this we can
easily compute the skewness of ${\mathbf{x}}_{2}^{*}$, that only depends on
$p$, not on the particular values:
$\hat{f}_{3}({\mathbf{x}}_{2}^{*})=(2p-1)/\sqrt{p(1-p)}$. Although the
previous expression has no extrema for a continuous $p$, given the discrete
nature of $p$ for discrete signals (from $1/N$ to $(N-1)/N$) we obtain that
the maximal and minimal skewness are produced when $p$ is either $1/N$ or
$(N-1)/N$, that is, when the vector is constant except for one coefficient.
The rest of the values of $p$ produce the vast majority of the critical points
of $\hat{f}_{3}$, which are saddles. There are no local (non-global) extrema
(condition B1). These critical points have subsequent undefined decoupled
moments (see Subsection 3.3.1), with the only exception of the case $p=1/2$
(possible just when $N$ is even), the only intersection of
${\mathbf{x}}_{2}^{*}$ with ${\mathcal{R}}_{3}$, which, having already zero
skewness, its (minimal) kurtosis is also its orthokurtosis. One can see that
the skewness of a bivaluated vector can not be adjusted by applying a
coefficient-wise reversible (monotonous) non-linearity, because the skewness
only depends on $p$.

For $k>3$ we apply the same procedure for obtaining the critical points of
$\hat{f}_{k}$ in ${\mathcal{R}}_{k-1}$, namely, solving an algebraic equation
of degree $k-1$, and obtaining any combination of distinct $k-1$ solutions for
the values in the vector. Same as for the previous case only the $k-2$
distinct proportions ($k-1$ in total, adding up to 1) of each of these $k-1$
different values matter for the computation of $\hat{f}_{k}$. And, again, the
vast majority of these critical points are saddles, presenting no local (non-
global) extrema (condition B1). For instancce, in the case of $k=4$, critical
points have three different distinct values or less, the minimal orthokurtosis
point being a degenerate case (having two single values with $p=1/2$, as
mentioned above), whereas the maximal orthokurtosis is produced for
$p_{1}=(N-2)/N$, $p_{2}=1/N$, corresponding to all pixels having the same
value, except for two pixels, now having each of these two a different value
from the dominant and from each other. These vectors can be adjusted to have
the desired mean, variance and skewness, but, having only three distinct
values, we run out of degrees of freedom to adjust their orthokurtosis too.
Therefore, except for the case when the orthokurthosis is already three (its
reference value), $\hat{{\mathbf{x}}}_{4}({\mathbf{x}}_{3}^{*})$ does not
exist, and, as a consequence, the fifth order and subsequent decoupled moment
are not defined at these points. Note also that the range of the decoupled
features generally change with respect to their original counterparts. Whereas
the second-order moment range does not change when decoupled (it is still
${\mathbb{R}}^{+}$), the fourth-order (skewness) is constrained from
${\mathbb{R}}$ to $[-\frac{N-2}{\sqrt{N-1}},\frac{N-2}{\sqrt{N-1}}]$, and the
fourth order (orthokurtosis) from ${\mathbb{R}}^{+}$ to $[1,N/2]$. Finally, we
note that the existence of higher-order decoupled moments also depends on
having a large enough number of samples, $N$, In particular, the last result
implies that, for $N<6$, there are no decoupled moment of higher-than-four
orders, as the reference value for normalizing the orthokurtosis, 3, is not
reachable within its valid range. This limitation also comes from the already
explained requirement of having enough distinct values for the samples.

Whereas a typical vector will have more than a few distinct quantization
values, and, thus, it will not produce critical points for the first few
decoupled moments, a different situation is created by the basins of
attraction of the saddles, which, as pointed out above, constitute the vast
majority of the critical points. It is easy to informally check that any
vector having coefficients with repeated maximal (or minimal, for odd orders)
values lies within the basin of attraction of a saddle. To illustrate this,
let us consider a gray-level image in the range $[0,255]$, with two pixels
having the $255$ value. As we increase its fourth-order decoupled moment (the
skewness) staying in ${\mathcal{R}}_{2}$, these two values are going to grow
at exactly the same pace, relatively to the rest of the coefficients, that,
due to the normalization, will get relatively lower and progressively closer
to each other. If we keep on increasing the skewness we would approach, in the
limit, to an image made of all pixels sharing the same value, except for two
pixels, both sharing another value. This mental experiment shows how our
original image, having more than one pixel with the maximal value, lies in
$\Lambda$, i.e., in the basin of attraction of a saddle. This situation,
unless avoided by adding a proper perturbation, provokes the algorithm to
eventually get stuck in the saddle, thus not letting the vector to be adjusted
along its full range (as shown before, the maximal/minimal skewness is
achieved when all samples except for one have the same value). The desired
effect of a perturbation is to “break the tie”, thus allowing the gradient to
further advance towards the absolute extrema of the feature. An analogous
reasoning can be applied for higher-order moments.

Then, how likely is that a vector belongs to $\Lambda$? The answer, dealing
with vectors having regular density distributions in ${\mathbb{R}}^{N}$ is:
zero probability (as the basins of attraction of these saddles are lower-
than-$N$ dimensional, condition B1). However, the answer for digital (discrete
quantized samples) is totally different: potentially very likely. Let us
assume a vector having $N$ samples quantized in $Q$ levels. Then, the
probability of that a particular value $v$ (e.g., the highest) is repeated,
assuming a uniform and independent distribution for each of the vector
coefficients is given by a binomial distribution:151515This does not pretend
to be an accurate estimation of the probability of the repetition of the
maximal/minimal value $v$ in a typical signal (e.g., an image). However, it
does provide a useful reference value.
$P(n(v)>1)=1-(1-1/Q)^{N}-N/Q(1-1/Q)^{N-1}$, $n(v)$ being the number of
occurrences of $v$. For instance, for a very small size image of $64\times 64$
pixels ($N=4096$ \- for larger images it gets worse) with pixels ranging from
0 to 255 ($Q=256$), the probability of a given quantization level appearing
more than once is extremely high: $1-1.86\times 10^{-6}$. This example shows
the importance of adding a proper perturbation to our digital signal
${\mathbf{x}}$, as the one proposed in Subsection 3.3.3.

####  C.1.2 Adding a perturbation

In this subsection we propose a perturbation method that not only ensures that
all perturbed coefficient values $x^{\prime}_{i}=x_{i}+\epsilon_{i},\,i=1\dots
N$ are different (high entropy) but it also maximizes the minimal possible
difference between them. The latter feature is achieved by using for the
perturbation $N$ uniform extra levels within each single quantizing step and
assigning a different level to each $\epsilon_{i}$. On the other hand,
noticeable local oscillations are avoided by choosing a very low frequency
pattern for the perturbation, which minimizes its perceptual impact.

Algorithm 6 explains the process. First, a random angle tangent is chosen for
generating a ramp for the image grid (same method can be straightforwardly
generalized to $n$-dimensional grids). The tangent value must not be a
rational $p/q$ number with small $p,q\in{\mathbb{N}}$, because that would
create repeated values on the ramp. Then, after checking that there are no
repeated values on the generated ramp, its cells are sorted in a ranking
according to their value, corresponding the number $1$ to the lowest value and
$N$ the highest. Finally, these integers are normalized to the interval
$[-1/2,1/2)$ and returned as the perturbation. The result is a (slightly
curved, sigmoid) ramp made of all different values, having an exactly uniform
distribution.

0: An empty array of $N_{x}\times N_{y}=N$ pixels

1: repeat

2: Generate a pseudo-random number $r\in[0,1]$

3: Compute a ramp $v(n_{x},n_{y})=n_{x}+r*n_{y}$

4: Check for repeated values in $v$

5: until there are no repeated values in $v$

6: Compute $o(n_{x},n_{y})=rank(v(n_{x},n_{y}))\in\\{1\dots N\\}$

7: $\epsilon(n_{x},n_{y})=(o(n_{x},n_{y})-1-N/2)/N\in[-1/2,1/2)$

8: return $\epsilon$ ($N_{x}\times N_{y}$ array)

Algorithm 6 High-entropy, low-impact perturbation

###  C.2 Second-order moments at the output of a filter bank

####  C.2.1 Critical points, active frequencies and perturbations

Being the sample second-order moment at the output of a filter a positive
definite quadratic function, it has no saddle points. Therefore, in this case,
in contrast with Section C.1.1, we do not face the problem of their basins of
attraction.

The critical points of the decoupled features are those vectors where the
gradients are either zero (critical points inherited from the original
features) or co-linear. Substituting Eq. (25) into Eq. (18) it yields:

$|H_{k}({\mathbf{\xi}})|^{2}X_{k}^{*}({\mathbf{\xi}})=\sum_{j=1}^{k-1}\lambda_{j,k}|H_{j}({\mathbf{\xi}})|^{2}X_{k}^{*}({\mathbf{\xi}}),\quad\\{\lambda_{j,k}\neq
0\\}.$

This equation always admits the solution ${\mathbf{x}}_{0}^{*}={\mathbf{0}}$.
In addition, in the case there are regions of the signal spectrum that are not
covered by any filter (e.g., signals made of a constant value, for band-pass
or high-pass filters), the corresponding vectors having only those frequencies
will also be critical points. Apart from the previous solutions, all
corresponding to the “null space” of the filters’ output (which provide the
absolute minima of the original features), the equation may only hold if the
squared filters are themselves co-linear. Thus, the latter possibility must be
prevented - otherwise all points in $\bar{\Omega}$ would be critical!

We see that, although in this case the decoupled features do not introduce
additional critical points, zeros in the frequency domain, both of the signal
and of the filters, act as partial “gradient killers”: signal will not change
at those frequencies where kernels are zero, and, similarly, the signal will
neither change at those frequencies where the signal itself vanishes.

Whereas in some applications it is normal to ignore some regions of the
spectrum that are not useful for a given task, and thus they are left
uncovered by the filters bank, it seems advisable, nevertheless: i) to filter
out those frequencies (having a support $D$) also in ${\mathbf{x}}$ (otherwise
those spectral components will remain unchanged in ${\mathbf{x}}$), and ii) to
introduce a small perturbation $\epsilon$ in ${\mathbf{x}}$, such that
${\mathbf{x}}+\epsilon$ will not be zero or too small at any frequency in the
above defined support $D$. Then, a first reasonable concrete choice for the
perturbation may be

$\epsilon=\arg\min_{{\mathbf{z}}}\|{\mathbf{z}}\|\,s.t.\,|X({\mathbf{\xi}})+Z({\mathbf{\xi}})|\geq\theta,\forall{\mathbf{\xi}}\in
D$

which yields, in the Fourier domain, for ${\mathbf{\xi}}\in D$:

$E({\mathbf{\xi}})=\left\\{\begin{tabular}[]{ll}$0$,&\text{if}
$|X({\mathbf{\xi}})|>\theta$\\\
$(\theta-|X({\mathbf{\xi}})|)e^{i2\pi\arg(X({\mathbf{\xi}}))}$,&\text{if}
$\theta>|X({\mathbf{\xi}})|>0$\\\ $\theta e^{i2\pi r}$,&\text{if}
$|X({\mathbf{\xi}})|=0$,\end{tabular}\right.$ (37)

and 0 for ${\mathbf{\xi}}\not\in D$, where $E({\mathbf{\xi}})={\cal
F}(\epsilon)$. Here $r$ is a uniform random value in $[0,1]$. The so defined
$\epsilon$ is a perturbation of maximal entropy amongst all minimal Euclidean
norm ensuring a spectral content above a threshold $\theta$ in $D$.161616A
perceptually-based perturbation may be easily obtained from here by
considering perceptual metrics/threshold instead. The threshold $\theta$ can
be chosen as the supreme of the set
$\\{\theta:q\left({\mathbf{x}}+\epsilon({\mathbf{x}},\theta)\right)=q({\mathbf{x}})\\}$,
or a similar criterion. Furthermore, depending on the complete set of
features, different types of perturbations (see Subsection 3.3.3) can be fused
into a single one fulfilling all the requirements.

##  Appendix D Regression experiments

In this section we give some specific details about the regression
experiments. The probability density function of the Generalized Gaussian
Distribution (GGD) is given by:

$f(x)=\frac{\beta}{2\alpha\Gamma(1/\beta)}e^{-(\left|x-\mu\right|/\alpha)^{\beta}},$

where $\Gamma$ denotes the gamma function, $\beta$ is the shape parameter,
$\alpha$ the scale parameter and $\mu$ the location parameter. The probability
density function of the Gamma Distribution (GMD) is given by:

$f(x)=\frac{1}{\Gamma(\beta)\theta^{\beta}}x^{\beta-1}e^{-x/\theta},$

where $\beta$ is the shape parameter and $\theta$ the scale parameter.

The probability density function of the absolute value of a Normal
distribution raised to $\beta$ (GND), $X=|T|^{\beta}$,
$T\sim\mathcal{N}(0,\,1)$, is given by, for every positive $\beta$:

$f(x)=\frac{2}{\beta\sqrt{2\pi}}x^{(1/\beta)-1}e^{-\frac{1}{2}x^{2/\beta}},$

where $\beta$ is the shape parameter. Note that, for $\beta=2$, this
distribution leads to the chi-squared distribution of one degree of freedom,
$\chi^{2}(1)$.

The RMSE results for the different regression methods and sample sizes $N$ are
shown in Tables VI (Generalized Gaussian Distribution distribution, GGD), VII
(Gamma distribution, GMD) and VIII (absolute value of a Normal distribution
raised to $\beta$, GND). The regression methods compared are: linear
regression models (LRM), regression trees (RT), support vector regression
(SVR), Gaussian process regression (GPR), ensembles of trees (ET) and neural
networks (NNR). MSM stands for the set of classical marginal standardized
moments, and $\text{DF}_{\text{MSM}}$ for its corresponding decoupled set.
Highlighted in bold, the regression method that minimizes the RMSE for each
$N$ value.

TABLE VI:  RMSE Results for the different regression methods, Generalized Gaussian distribution (GGD). | $N$ | LRM | RT | SVR | ET | GPR | NNR
---|---|---|---|---|---|---|---
MSM | 64 | 0.71 | 0.59 | 0.54 | 0.55 | 0.52 | 0.51
128 | 0.75 | 0.46 | 0.46 | 0.44 | 0.41 | 0.41
256 | 0.85 | 0.36 | 0.44 | 0.35 | 0.34 | 0.32
512 | 0.95 | 0.30 | 0.47 | 0.29 | 0.29 | 0.27
1024 | 1.02 | 0.26 | 0.55 | 0.25 | 0.25 | 0.24
2048 | 1.13 | 0.22 | 0.63 | 0.22 | 0.22 | 0.21
$\text{DF}_{\text{MSM}}$ | 64 | 0.79 | 0.48 | 0.51 | 0.45 | 0.47 | 0.44
128 | 0.67 | 0.34 | 0.44 | 0.32 | 0.38 | 0.31
256 | 0.85 | 0.25 | 0.38 | 0.23 | 0.31 | 0.22
512 | 0.51 | 0.18 | 0.31 | 0.17 | 0.26 | 0.17
1024 | 0.45 | 0.13 | 0.29 | 0.13 | 0.26 | 0.13
2048 | 0.41 | 0.10 | 0.32 | 0.10 | 0.31 | 0.10
TABLE VII:  RMSE Results for the different regression methods, Gamma distribution (GMD). | $N$ | LRM | RT | SVR | ET | GPR | NNR
---|---|---|---|---|---|---|---
MSM | 64 | 0.58 | 0.76 | 0.59 | 0.71 | 0.50 | 0.52
128 | 0.45 | 0.59 | 0.46 | 0.56 | 0.39 | 0.40
256 | 0.37 | 0.48 | 0.38 | 0.46 | 0.31 | 0.31
512 | 0.27 | 0.39 | 0.30 | 0.36 | 0.23 | 0.24
1024 | 0.21 | 0.29 | 0.22 | 0.28 | 0.21 | 0.19
2048 | 0.18 | 0.22 | 0.18 | 0.21 | 0.15 | 0.13
$\text{DF}_{\text{MSM}}$ | 64 | 0.45 | 0.45 | 0.39 | 0.40 | 0.39 | 0.38
128 | 0.32 | 0.32 | 0.28 | 0.29 | 0.27 | 0.28
256 | 0.25 | 0.24 | 0.21 | 0.22 | 0.20 | 0.20
512 | 0.19 | 0.18 | 0.16 | 0.15 | 0.14 | 0.14
1024 | 0.15 | 0.14 | 0.12 | 0.12 | 0.10 | 0.10
2048 | 0.12 | 0.10 | 0.11 | 0.08 | 0.07 | 0.07
TABLE VIII:  RMSE Results for the different regression methods, absolute value of a Normal raised to $\beta$ (GND). | $N$ | LRM | RT | SVR | ET | GPR | NNR
---|---|---|---|---|---|---|---
MSM | 64 | 0.64 | 0.77 | 0.68 | 0.73 | 0.55 | 0.59
128 | 0.62 | 0.74 | 0.65 | 0.71 | 0.53 | 0.56
256 | 0.53 | 0.65 | 0.56 | 0.61 | 0.43 | 0.47
512 | 0.46 | 0.56 | 0.47 | 0.52 | 0.36 | 0.40
1024 | 0.40 | 0.49 | 0.41 | 0.45 | 0.31 | 0.34
2048 | 0.36 | 0.41 | 0.35 | 0.38 | 0.34 | 0.29
$\text{DF}_{\text{MSM}}$ | 64 | 0.41 | 0.45 | 0.38 | 0.39 | 0.37 | 0.38
128 | 0.37 | 0.40 | 0.34 | 0.35 | 0.34 | 0.34
256 | 0.27 | 0.31 | 0.25 | 0.26 | 0.25 | 0.25
512 | 0.20 | 0.21 | 0.18 | 0.18 | 0.17 | 0.18
1024 | 0.16 | 0.16 | 0.14 | 0.14 | 0.13 | 0.13
2048 | 0.13 | 0.12 | 0.11 | 0.10 | 0.10 | 0.10

## Acknowledgments

The authors would like to thank Matteo Bonforte, Rafael Molina, Ivan
Selesnick, Gustau Camps-Valls and Eero Simoncelli for fruitful discussions.

## References

  * [1] J. Portilla and E. Simoncelli, “A parametric texture model based on joint statistics of complex wavelet coefficients,” _International Journal of Computer Vision_ , vol. 40(1), pp. 49–71, 2000, ®Matlab code publicly available at http://www.cns.nyu.edu/ lcv/texture/.
  * [2] E. Martínez-Enríquez and J. Portilla, “Controlled feature adjustment for image processing and synthesis,” in _2020 IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP)_ , 2020, pp. 1–6.
  * [3] J. Balas, “Texture synthesis and perception: Using computational models to study texture representations in the human visual system,” _Vision Research_ , vol. 46, no. 3, pp. 299 – 309, 2006. [Online]. Available: http://www.sciencedirect.com/science/article/pii/ S0042698905002336
  * [4] J. A. Lee and M. Verleysen, _Nonlinear Dimensionality Reduction_ , 1st ed. Springer Publishing Company, Incorporated, 2007.
  * [5] S. J. Pan and Q. Yang, “A survey on transfer learning,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 22, no. 10, pp. 1345–1359, 2010.
  * [6] J. Portilla and E. Martinez-Enriquez, “Nested normalizations for decoupling global features,” in _2018 25th IEEE International Conference on Image Processing (ICIP)_ , Oct 2018, pp. 2112–2116.
  * [7] E. Martinez-Enriquez and J. Portilla, “Deterministic feature decoupling by surfing invariance manifolds,” in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 6049–6053.
  * [8] J. R. M. Hosking and J. R. Wallis, “Parameter and quantile estimation for the generalized pareto distribution,” _Technometrics_ , vol. 29, no. 3, pp. 339–349, 1987. [Online]. Available: http://www.jstor.org/stable/1269343
  * [9] A. Azzalini, “A class of distributions which includes the normal ones,” _Scandinavian Journal of Statistics_ , vol. 12, no. 2, pp. 171–178, 1985\. [Online]. Available: http://www.jstor.org/stable/4615982
  * [10] C. Flecher, P. Naveau, and D. Allard, “Estimating the closed skew-normal distribution parameters using weighted moments,” _Statistics & Probability Letters_, vol. 79, no. 19, pp. 1977–1984, 2009.
  * [11] C. G. Justus, W. R. Hargraves, A. Mikhail, and D. Graber, “Methods for estimating wind speed frequency distributions,” _Journal of Applied Meteorology and Climatology_ , vol. 17, no. 3, pp. 350 – 353, 1978.
  * [12] S. A. Akdağ and A. Dinler, “A new method to estimate Weibull parameters for wind energy applications,” _Energy Conversion and Management_ , vol. 50, no. 7, pp. 1761–1766, 2009.
  * [13] A. Bovik, M. Clark, and W. Geisler, “Multichannel texture analysis using localized spatial filters,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 12, no. 1, pp. 55–73, 1990.
  * [14] D. J. Heeger and J. R. Bergen, “Pyramid-based texture analysis/synthesis,” in _Proceedings., International Conference on Image Processing_ , vol. 3, Oct 1995, pp. 648–651 vol.3.
  * [15] J. Portilla, R. Navarro, O. Nestares, and A. Tabernero, “Texture synthesis-by-analysis based on a multiscale early-vision model,” _Optical Engineering_ , vol. 35, no. 8, pp. 2403–2417, 1996.
  * [16] D. C. Blest, “A new measure of kurtosis adjusted for skewness,” _Australian and New Zealand Journal of Statistics_ , vol. 45, no. 2, pp. 175–179, 2003. [Online]. Available: http://dx.doi.org/10.1111/1467-842X.00273
  * [17] K. Pearson, “IX. Mathematical contributions to the theory of evolution.—XIX. Second supplement to a memoir on skew variation,” _Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences_ , vol. 216, no. 538-548, pp. 429–457, 1916. [Online]. Available: http://rsta.royalsocietypublishing.org/content/216/538-548/429
  * [18] R. Sharma and R. Bhandari, “Skewness, kurtosis and Newton’s inequality,” _ArXiv e-prints_ , Sep. 2013.
  * [19] K. Pearson, “Contributions to the mathematical theory of evolution,” _Philosophical Transactions of the Royal Society of London. A_ , vol. 185, pp. 71–110, 1894. [Online]. Available: http://www.jstor.org/stable/90667
  * [20] A. T. Kalai, A. Moitra, and G. Valiant, “Efficiently learning mixtures of two Gaussians,” in _Proceedings of the forty-first annual ACM symposium on the Theory of Computing (STOC), 2010_. ACM Press, June 2010, pp. 553–562. [Online]. Available: https://www.microsoft.com/en-us/research/publication/efficiently-learning-mixtures-two-Gaussians/
  * [21] K. Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,” _Biological Cybernetics_ , vol. 36, pp. 193–202, 1980.
  * [22] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the Institute of Radio Engineers_ , vol. 86, no. 11, pp. 2278–2323, 1998.
  * [23] D. H. Hubel and T. N. Wiesel, “Receptive fields of single neurones in the cat’s striate cortex,” _The Journal of Physiology_ , vol. 148, no. 3, pp. 574–591, 1959. [Online]. Available: https://physoc.onlinelibrary.wiley.com/doi/abs/10.1113/ jphysiol.1959.sp006308
  * [24] J. G. Daugman, “Complete discrete 2-d Gabor transforms by neural networks for image analysis and compression,” _IEEE Trans. Acoust. Speech Signal Process._ , vol. 36, pp. 1169–1179, 1988.
  * [25] R. F. Navarro, A. Tabernero, and G. Cristóbal, “Image representation with Gabor wavelets and its applications,” _Advances in Imaging and Electron Physics_ , vol. 99, p. 329, 1997.
  * [26] T. Dau, B. Kollmeier, and A. A. Kohlrausch, “Modeling auditory processing of amplitude modulation. i. detection and masking with narrow-band carriers.” _The Journal of the Acoustical Society of America_ , vol. 102 5 Pt 1, pp. 2892–905, 1997.
  * [27] I. Daubechies, “Ten lectures on wavelets,” _Computers in Physics_ , vol. 6, pp. 697–697, 1992.
  * [28] T. Canham, A. Martín, M. Bertalmío, and J. Portilla, “Using decoupled features for photo-realistic style transfer,” _SIAM Journal on Imaging Sciences, Submitted_ , 2022.
  * [29] S. Robert L. Devaney, M. Hirsch, S. Smale, and R. Devaney, _Differential Equations, Dynamical Systems, and an Introduction to Chaos_ , ser. Pure and Applied Mathematics - Academic Press. Elsevier Science, 2004. [Online]. Available: https://books.google.es/books?id=GNOmchErrMgC
  * [30] J. Kovacević and A. Chebira, “An introduction to frames,” _Foundations and Trends in Signal Processing_ , vol. 2, no. 1, pp. 1–94, 2008.
  * [31] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” _Mach. Learn._ , vol. 8, no. 3–4, p. 229–256, may 1992. [Online]. Available: https://doi.org/10.1007/BF00992696
  * [32] D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” 2013. [Online]. Available: https://arxiv.org/abs/1312.6114
  * [33] J. Portilla, A. Tristan-Vega, and I. Selesnick, “Efficient and robust image restoration using multiple-feature L2-relaxed sparse analysis priors,” _IEEE Transactions on Image Processing_ , vol. 24, no. 12, pp. 5046–5059, Dec 2015, ®Matlab code publicly available at DOI: 10.13140/RG.2.1.3531.5280.
  * [34] A. Winkelbauer, “Moments and Absolute Moments of the Normal Distribution,” _arXiv e-prints_ , p. arXiv:1209.4340, Sep. 2012.
  * [35] P. Brodatz, _Textures: a photographic album for artists and designers_ , ser. Dover pictorial archives. Dover Publications, 1966, available at https://multibandtexture. recherche.usherbrooke.ca/original_brodatz.html. [Online]. Available: http://books.google.es/books?id=4rBk_SemB2QC
  * [36] “https://in.mathworks.com/help/stats/regression-learner-app.html.”
  * [37] M. Nardon and P. Pianca, “Simulation techniques for generalized Gaussian densities,” _Journal of Statistical Computation and Simulation_ , vol. 79, no. 11, pp. 1317–1329, 2009.
  * [38] K. Kokkinakis and A. K. Nandi, “Exponent parameter estimation for generalized Gaussian probability density functions with application to speech modeling,” _Signal Processing_ , vol. 85, no. 9, pp. 1852–1858, 2005.
  * [39] G. Marsaglia and W. W. Tsang, “A simple method for generating gamma variables,” _ACM Trans. Math. Softw._ , vol. 26, no. 3, p. 363–372, sep 2000. [Online]. Available: https://doi.org/10.1145/358407.358414
  * [40] M. Mandal, T. Aboulnasr, and S. Panchanathan, “Image indexing using moments and wavelets,” _IEEE Transactions on Consumer Electronics_ , vol. 42, pp. 557–565, 1996.
  * [41] H. Peng, F. Long, and C. Ding, “Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 27, no. 8, pp. 1226–1238, Aug 2005.
  * [42] C. J. Burges, “A tutorial on support vector machines for pattern recognition,” _Data Mining and Knowledge Discovery_ , vol. 2, no. 2, pp. 121–167, Jun 1998. [Online]. Available: https://doi.org/10.1023/A:1009715923555
  * [43] G. Schneider and H. Uecker, _Nonlinear PDEs_ , ser. Graduate Studies in Mathematics. American Mathematical Society, Providence, RI, 2017, vol. 182, a dynamical systems approach. [Online]. Available: https://doi.org/10.1090/gsm/182
  * [44] L. Simon, “Asymptotics for a class of non-linear evolution equations, with applications to geometric problems,” _Annals of Mathematics_ , vol. 118, no. 3, pp. 525–571, 1983. [Online]. Available: http://www.jstor.org/stable/2006981
  * [45] M. Spivak, _A Comprehensive Introduction to Differential Geometry_ , ser. A Comprehensive Introduction to Differential Geometry. Publish or Perish, Inc., Wilmington, Del., 1979, no. v. 1.
  * [46] P. Hartman, _Ordinary Differential Equations_ , ser. Classics in Applied Mathematics, 38. Society for Industrial and Applied Mathematics, 2002. [Online]. Available: https://books.google.es/books?id=CENAPMUEpfoC