# LaplaceNet: A Hybrid Energy-Neural Model
for Deep Semi-Supervised Classification

Philip Sellars1, Angelica I. Aviles-Rivero1 and Carola-Bibiane Schönlieb1 P.
Sellars, Angelica I. Aviles-Rivero and Carola-Bibiane Schönlieb are with the
Department of Theoretical Physics and Applied Mathematics, Univeristy of
Cambridge, Cambridge, UK<EMAIL_ADDRESS>.

###### Abstract

Semi-supervised learning has received a lot of recent attention as it
alleviates the need for large amounts of labelled data which can often be
expensive, requires expert knowledge and be time consuming to collect. Recent
developments in deep semi-supervised classification have reached unprecedented
performance and the gap between supervised and semi-supervised learning is
ever-decreasing. This improvement in performance has been based on the
inclusion of numerous technical tricks, strong augmentation techniques and
costly optimisation schemes with multi-term loss functions. We propose a new
framework, LaplaceNet, for deep semi-supervised classification that has a
greatly reduced model complexity. We utilise a hybrid energy-neural network
where graph based pseudo-labels, generated by minimising the graphical
Laplacian, are used to iteratively improve a neural-network backbone. Our
model outperforms state-of-the-art methods for deep semi-supervised
classification, over several benchmark datasets. Furthermore, we consider the
application of strong-augmentations to neural networks theoretically and
justify the use of a multi-sampling approach for semi-supervised learning. We
demonstrate, through rigorous experimentation, that a multi-sampling
augmentation approach improves generalisation and reduces the sensitivity of
the network to augmentation. Code available at
https://github.com/psellcam/LaplaceNet.

##  I Introduction

The advent of deep learning has been key in achieving outstanding performance
in several computer vision tasks including image classification [1, 2, 3, 4,
5], object detection e.g. [6, 7, 8] and image segmentation [9, 10, 11].
Training deep learning models often relies upon access to large amounts of
labelled training data. In real-world scenarios we often find that labels are
scarce, expensive to collect, prone to errors (high uncertainty) and might
require expert knowledge. Therefore, relying on a well-representative dataset
to achieve good performance is a major limitation for the practical deployment
of machine learnt methods. These issues have motivated the development of
techniques which are less reliant on labelled data.

Semi-supervised learning aims to extract information from unlabelled data, in
combination with a small amount of label data, and produce results comparable
to fully supervised approaches. In recent years, the developments in deep
learning have motivated new directions in semi-supervised learning (SSL) for
image classification. The major benefit of these new deep approaches being the
ability to learn feature representations rather than rely upon hand-crafted
features. In the last few years, deep SSL papers have reached unprecedented
performance e.g. [12, 13], and the gap between supervised and semi-supervised
models is much smaller now that it was even five years ago, with semi-
supervised methods surpassing certain supervised techniques.

What techniques have been crucial to the improved performance of deep semi-
supervised methods? Although, there is no universal answer, there are several
shared commonalities between the current SOTA. The works of [13, 12, 14]
demonstrated that a key factor for improving performance is the use of strong
augmentations strategies such as AutoAugment [15], RandAugment [16], Cutout
[17] and CTAugment [14]. Additionally, the use of confidence thresholding [12,
18] and temperature sharpening [13, 19] are thought to be vital in improving
performance for pseudo-labeling methods. Other papers [20, 21, 19] have shown
great improvement from using interpolating techniques such as MixUp [22].
Several SOTA have also promoted large batch sizes [12] with a large ratio of
unlabelled to labelled data per batch.

Recent approaches in SSL have proposed costly optimisation schemes involving
multi-term loss functions to improve the generalisation of their models [14,
20]. Some approaches [12] use separate loss terms for unlabelled and labelled
data, whilst consistency regularisation approaches such as [13] use a standard
supervised loss in combination with a specialised consistency loss. Other
approaches go even further [20, 14] and use three or more loss terms which
promote entropy minimisation, class balancing or simultaneously minimise
several consistency losses.

Over-costly computational approaches and unnecessary complexity, make it hard
to directly say what tools or approaches are important for improved
generalisation and make it difficult to use SSL methods in realistic settings.
Furthermore, despite the significant improvements found in using
augmentations, there has been little effort in the field of SSL to investigate
how best to include strong augmentations techniques in the learning framework.
With these points in mind, in this work, we introduce a new deep SSL framework
for image classification which offers state-of-the-art performance with
massively reduced model complexity. Our main contributions are:

  * –

We propose a graph based pseudo-label approach for semi-supervised image
classification which we name LaplaceNet. We demonstrate through extensive
testing, that our approach produces state-of-the-art results on benchmark
datasets CIFAR-10, CIFAR-100 and Mini-ImageNet. We do so with vastly reduced
model complexity compared to the current state-of-the-art. We show that a
single loss, the classic supervised loss, is all that is required for
fantastic performance in the SSL domain.

  * –

We show that using an energy-based graphical model for pseudo-label generation
produces more accurate pseudo-labels, with a small computational overhead,
than using the network’s predictions directly. Furthermore, we demonstrate
that energy-based pseudo-label approaches can produce state-of-the-art results
without the techniques (temperature sharpening, confidence thresholding, soft
labels) that are currently thought to be essential for pseudo-label methods.

  * –

Instead, we offer further evidence that strong augmentation is by far and away
the most important tool for improving the performance of semi-supervised
models in the natural image domain. With this in mind, we propose,
theoretically justify and experimentally demonstrate that a multi-sample
averaging approach to strong augmenation not only improves generalisation but
reduces the sensitivity of the model’s output to data augmentation.

##  II Related Work

The problem of improving image classification performance using SSL has been
extensively investigated from the classic perspective e.g. [23, 24, 25, 26,
27, 28], in which one seeks to minimise a given energy functional that
exploits the assumed relationship between labelled and unlabelled data [29].
However, classical approaches tended to rely on hand-crafted features that
limited their performance and generalisation capabilities. With the
popularisation of deep learning and its ability to learn generalisable feature
representations, many techniques have incorporated neural networks to mitigate
problems of generalisation. These modern state-of-the-art methods are
dominated by two approaches, consistency regularisation and pseudo-labelling,
which differ in how they incorporate unlabelled data into the loss function.

###  II-A Consistency Regularisation Techniques

One of the fundamental assumptions that allows semi-supervised learning to
help performance is the cluster assumption, which states that points in the
same cluster are likely to be in the same class. This can be seen to be
equivalent to the low-density assumption which states that the decision
boundaries of the model should lie in low-density regions of the data
distribution. Following from the above assumptions, if we have access to some
labelled data $Z_{l}=\\{x_{i},y_{i}\\}_{i=1}^{n_{l}}$ and a large amount of
unlabelled data $Z_{u}=\\{x_{i}\\}_{i=n_{l}+1}^{n_{l}+n_{u}}$, we should seek
to move our decisions boundaries to be in low density regions of the joint
labelled and unlabelled data distributions.

Consistency regularisation seeks to implement the low-density assumption by
encouraging the model $f$ to be invariant to perturbations $\delta$ to the
data $x$. As a result the decision boundaries are pushed to low-density
regions. Mathematically, given some data perturbing function
$u:\mathcal{X}\rightarrow\mathcal{X}$, such that $u(x)=x+\delta$, consistency
based approaches seek to minimise some consistency loss $L_{\text{con}}$ in
the general form of

$L_{\text{con}}=||f(u(x))-f(x)||_{2}^{2}.$ (1)

A large number of papers have applied this idea to SSL including the
$\prod-$Model and temporal ensembling [30], Virtual Adversarial Training (VAT)
[31], Mean Teacher [32], the Interpolation Consistency Training (ICT) [21]
RemixMatch [14] and MixMatch [19]. The downside of consistency regularisation
techniques is the vagueness in choosing an appropriate $\delta$. This
vagueness is reflected in the wide range of perturbations which have been used
in the field. Virtual Adversarial Training uses adversarial training to learn
an effective $\delta$ for each point. Mean Teacher [32] decided to apply a
perturbation to the model itself, and replaces $f(u(x))$ with an exponential
moving average of the model $f_{\text{EMA}}(x)$. Interpolation Consistency
Training [21] seeks to train the model to provide consistent predictions at
interpolations of unlabelled points. The authors of [13] demonstrated that by
replacing simple noising perturbations with stronger augmentation
perturbations (eg, RandAugment [16] or CTAugment [14]) leads to a substantial
performance improvements.

Although these techniques have demonstrated great performance, it is unclear
how best to set the perturbations $\delta$ and how best to incorporated them
in learning frameworks. In our work, we avoid using model based perturbations
and instead focus on the the application of strong data augmentation.

###  II-B Pseudo-Labelling Techniques

Another family of methods, termed pseudo-label approaches, focus on estimating
labels for the unlabelled data points and then using them in a modified loss
function. Forcing the network to make predictions on unlabelled points
minimises the entropy of the unlabelled predictions [29] and moves the
decision boundaries to low-density regions. Additional, dependent on the
accuracy of the pseudo-labels, we increase the amount of labelled data the
model has access to and reduce overfitting to the initally small label set.
There are many ways to incorporate unlabelled data / pseudo-label pairs into
the loss function but the most common ways are to either create a specific
loss term for the unlabelled data pseudo-label pairs [12, 18] or by using
composite batches containing both labelled and unlabelled data and keeping the
standard supervised classification loss [33, 20].

The first application of this idea to the deep learning setting was presented
by Lee [34]. Viewing the output of the neural network $f(x)$ as a discrete
probability distribution, Lee assigned a hard pseudo-label $\hat{y}$ for each
unlabelled data point according to its most likely prediction
$\hat{y}_{i}=\operatorname*{arg\,max}f(x_{i})$. These pseudo-labels were then
used in a two termed loss function of the form

labelled loss unlabelled loss (2)
$\displaystyle\hat{L}_{ssl}=\color[rgb]{0,0,0}\frac{1}{n_{l}}\sum_{i=1}^{n_{l}}l_{s}(f(x),y)+\color[rgb]{0,0,0}\eta\color[rgb]{0,0,0}\frac{1}{n_{u}}\sum_{i=1}^{n_{u}}l_{s}(f(x),\hat{y}),$

where $l_{s}$ is some loss function and $\eta$ is a weighting parameter. The
pseudo-labels are recalculated every-time the unlabeled data is passed through
the network. As an alternative to hard labels, [19] used the full output
probability distribution of the network as a soft label for each point.
However, it was found that sharpening this distribution helped ensured the
model’s prediction entropy was minimised.

As pointed out by Arazo et al [20] there is a potential pitfall in this style
of approach. Networks are often wrong and the neural network can overfit to
its own incorrectly guessed pseudo-labels in a process termed confirmation
bias. Arazo et al proposed using MixUp [22], soft labels and a minimum ratio
of labeled to unlabeled data to reduce confirmation bias. An alternative
method to reduce confirmation bias is to use uncertainty quantification for
the produced pseudolabels. These methods calculate a confidence score $r_{i}$
for each pseudo-label $\hat{y}_{i}$. The works of [12, 33] used the entropy of
the probability distribution to give $r_{i}$ whilst [35] used the distance
between unlabelled points and labelled points in feature space. One can then
either weight the loss terms by $r_{i}$ or exclude pseudo-labels whose $r_{i}$
is below some threshold $\tau$ in an attempt to prevent the network learning
from low confidence predictions. This style of approach is based upon the idea
that the neural network is well calibrated, i.e that the model’s softmax score
is a good indicator of the actually likelihood of a correct prediction.
However, recent research has suggested that modern neural networks are not as
well calibrated as may be intuitively thought [36]. In our work we demonstrate
that, whilst a intuitive solution, uncertainty quantification is not needed
for our pseudo-label approach.

In a completely different direction to network predictions, it has been shown
from a classical perspective [25] that energy based models such as graphs are
well suited to the task of label propagation. Therefore, several works [33,
37, 38] have shown good performance by iteratively feeding the feature
representation of a neural network to a graph, performing pseudo-label
generation on the graph and then using those labels to train the network.
However, graphical approaches have yet to show that they can produce state-of-
the-art results compared to model based approaches such as [12, 13]. In our
work, we present a graphical approach which surpasses the performance of model
based approaches, demonstrating that graphical approaches have a lot of
promise for practical applications.

##  III Proposed Technique

This section details our proposed semi-supervised method. We cover the
generation of pseudo-labels, the optimisation of the model alongside a full
algorithm and we explore our multi-sample augmentation approach.

###  III-A Problem Statement:

From a joint distribution $\mathcal{Z}=(\mathcal{X},\mathcal{Y})$ we have a
dataset $Z$ of size $n=n_{l}+n_{u}$ comprised of a labelled part of joint
samples $Z_{l}=\\{x_{i},y_{i}\\}_{i=1}^{n_{l}}$ and a unlabelled part
$Z_{u}=\\{x_{i}\\}_{i=n_{l}+1}^{n}$ of single samples on $\mathcal{X}$. The
labels come from a discrete set of size C $y\in\\{1,2,..,C\\}$. Our task is to
train a classifier $f$, modelled by a neural network with parameter vector
$\theta$, which can accurately predict the labels of unseen data samples from
the same distribution $\mathcal{X}$. The classifier $f$ can be viewed as the
composition of two functions $z$ and $g$ such that $f(x)=g(z(x))$.
$z:\mathcal{X}\rightarrow\mathbb{R}^{d_{p}}$ is the embedding function mapping
our data input to some $d_{p}$ dimensional feature space and
$g:\mathbb{R}^{d_{p}}\rightarrow\mathbb{R}^{C}$ projects from the feature
space to the classification space.

###  III-B Pseudo-labels Generation

As a pseudo-label based approach, we iteratively assign a pseudo-label
$\hat{y}$ to all data points in $Z_{u}$ once per epoch. In this work, we
generate hard pseudo-labels using a graph based approach first proposed by
Zhou et al [26] and first adapted to deep networks by Iscen et al [33] which
has been thoroughly studied in the classical machine learning literature.

We first extract the feature representation of the dataset $V$ by using the
embedding function of the neural network $z$ so that
$V=\\{z(x_{1}),..,z(x_{n})\\}$. Unlike other works we do not apply
augmentation to the data whilst producing the pseudo-labels. Using $V$ and a
similarity metric $d$, we use $d(v_{i},v_{j})=\langle v_{i},v_{j}\rangle$, we
construct a symmetric weighted adjacency matrix $W\in\mathbb{R}^{n\times n}$.
The elements $w_{ij}\in W$ are given by $W_{ij}=d(v_{i},v_{j})$ and represent
the pairwise similarities between data points. We then sparsify $W$ using the
following nearest neighbour approach, which reads:

$W_{ij}=\begin{cases}d(v_{i},v_{j}),&\text{if $i$ is one of the $k$ nearest
neighbor of $j$,}\\\ &\text{or vice versa.}\\\ 0&\text{otherwise}.\end{cases}$
(3)

We then construct the degree matrix $D:=\text{diag}(W\mathbbm{1}_{n})$ and use
this to normalise the affinity matrix $\mathcal{W}=D^{-1/2}WD^{-1/2}$, which
prevent nodes with high degree having a large global impact. Finally, we use
the initial label information to create the labelled matrix
$Y\in\mathbb{R}^{n\times C}$

$Y_{ij}=\begin{cases}1,&\text{if $y_{i}=j$,}\\\
0&\text{otherwise}.\end{cases}$ (4)

We can then propagate the information contained in $Y$ across the graph
structure $\mathcal{W}$ by minimising the graphical Laplacian of the
prediction matrix $F\in\mathbb{R}^{n\times C}$ plus a fidelity term to the
supplied labelled data:

$\mathcal{Q}(F)=\frac{1}{2}\sum_{i,j=1}^{n}\mathcal{W}_{ij}\left|\left|\frac{F_{i}}{\sqrt{D_{ii}}}-\frac{F_{j}}{\sqrt{D_{jj}}}\right|\right|^{2}+\frac{\mu}{2}\sum_{i=1}^{n}||F_{i}-Y_{i}||^{2},$
(5)

where $\mu$ is a scalar weight. The first term enforces points which are close
according to the metric $d$ to share a similar label whilst the second term
encourages initially labelled points to keep their label. To side-step the
computationally infeasible closed form solution, we use the conjugate gradient
approach to solve the linear system $\left(I-\gamma\mathcal{W}\right)F=Y$,
where $\gamma(1+\mu)=1$. Using $F$ the pseudo-labels $\hat{y}_{i}$ are given
by

$\hat{y}_{i}=\operatorname*{arg\,max}_{j}F_{ij}.$ (6)

A common problem in label propogation is that the psuedo-labels produced by
the graph may be unbalanced over the classes and Iscen et al [33] attempted to
weight the optimisation problem to avoid this possibility. We found that the
weighting approach of Iscen et al actually made the performance of the model
worse than leaving the predictions as is. An alternate approach to counter
class in-balances is distribution alignment [14], which enforces the
distribution of the pseudo-label predictions to match some given prior
distribution. The implementation of this idea by ReMixMatch focused on
applying this idea to the network predictions and wasn’t optimal for a graph
based framework.

Instead we propose a novel smoother version of distribution alignment which
can be applied during or just after the conjugate gradient approach. We give a
full algorithm for this in Algorithm 1. The algorithm is an iterative approach
which smoothly deforms the pseudo-label predictions $F$ by the ratio $R$
between the prior distribution $D$ and the pseudo-label distribution of the
unlabelled points $D_{U}$. Thereby promoting the prediction of
underrepresented classes and vice versa. To ensure the deformation is smooth
we clip the range of $R$ values to be close to one. We show in the
experimental section that this approach improves the performance of the model.

Algorithm 1 Smooth Distribution Alignment

1: Input: Pseudo-label Prediction $F\in\mathbb{R}^{n\times C}$, Prior
Distribution $D\in\mathbb{R}^{C}$, labelled and unlabelled indexes
$L=\\{l_{i}\\}_{i=1}^{n_{l}}$ and $U=\\{u_{i}\\}_{i=1}^{n_{u}}$ and max
iteration $T$

2: Output: Adjusted Pseudo-label Prediction $F\in\mathbb{R}^{n\times C}$

3: for $t_{i}=1$, $t_{i}{+}{+}$, while $t_{i}<T$ do

4: $D_{U}\in\mathbb{R}^{C}\leftarrow\text{Initialise with zeros}$ Get the
pseudo-label distribution:

5: for $u_{i}\in U$ do

6: $D_{U}[\operatorname*{arg\,max}_{j}F[u_{i}]]\mathrel{+}=\frac{1}{n_{u}}$

7: end for

8: $R=D/D_{u}$ Clip values for smooth deformation:

9: $R[R>1.01]=1.01$ and $R[R<0.99]=0.99$ # Deform the current predictions:

10: for $c_{i}=1$, $c_{i}{+}{+}$, while $c_{i}<C$ do

11: $F[U,c_{i}]\mathrel{*}=R[c_{i}]$

12: end for

13: Row normalise $F$ to give valid distributions.

14: end for

###  III-C Semi-Supervised Loss

In the deep semi-supervising setting, particularly in the current SOTA [12]
[19], several works seek to minimise a semi-supervised loss $\hat{L}_{ssl}$
composed of two or more terms, one each for the labelled and unlabelled data
points and potentially others covering entropy minimisation etc., which has
the following form:

labelled loss unlabelled loss other terms (7)
$\displaystyle{L}_{\text{gen}}=\color[rgb]{0,0,0}\frac{1}{n_{l}}\sum_{i=1}^{n_{l}}l_{s}(f(x),y)+\color[rgb]{0,0,0}\eta\color[rgb]{0,0,0}\frac{1}{n_{u}}\sum_{i=1}^{n_{u}}l_{s}(f(x),\hat{y})+.....,$

where $\eta$ is a balancing parameter. For our approach we wanted to strip
away as much complexity from the loss function as possible in an effort to see
what elements are required for good performance. We move away from using a
multi-term loss and instead only use the standard supervised loss which has
worked so well in supervised image classification. To include our unlabelled
data we use mixed batches of size $b$ which are made up of $b_{l}$ labelled
samples and $b_{u}$ unlabelled samples to which we have assigned a pseudo-
label $\hat{y}$. Our semi-supervised loss, $L_{ssl}$, is given by:

$L_{ssl}=\frac{1}{b}\sum_{i=1}^{b}l_{s}(f(x_{i}),y_{i}).$ (8)

Note that in (8) $y_{i}$ may be a ground truth label or a pseudo-label. What
is remarkable about this loss is its simplicity. There is no confidence
thresholding of the pseudo-labels, additional weighting parameters, no
consistency based terms or other regularisations. Instead we rely upon the
strength of the combination of an a energy based graphical approach to pseudo-
labels estimation and the clever use of strong augmentation to increase
generalisation.

###  III-D Training the model

For initialisation purposes, we quickly extract some baseline knowledge from
the dataset by minimising a supervised loss $L_{sup}$, for one hundred passes
through $Z_{l}$. This supervised loss reads:

$L_{\text{sup}}=\frac{1}{b}\sum_{i=1}^{b}l_{s}(f(x_{i}),y_{i}),$ (9)

where $b$ is the batch size and $l_{s}$ is the cross entropy loss. We emphasis
that (9) uses only the tiny labelled set $Z_{l}$, and is performed once before
the main semi-supervised optimisation. We then begin our main learning loop
which alternates between updating the pseudo-label predictions and minimising
the semi-supervised loss $L_{ssl}$ for one epoch, where we define one epoch to
be one pass through the unlabelled data $Z_{u}$. This cycle then runs for a
total of $S$ optimisation steps and the fully trained model is then tested on
the relevant testing set. Note that we do use Mixup [22] on both $L_{sup}$ and
$L_{ssl}$ with a beta distribution parameters $\alpha$. In Algorithm 2, we
give an overview of training our model for $S$ optimisation steps.

Algorithm 2 Training Scheme for LaplaceNet

1: Input: labelled data $Z_{l}=\\{x_{i},y_{i}\\}_{i=1}^{n_{l}}$, unlabelled
data $Z_{u}=\\{x_{i}\\}_{i=n_{l}+1}^{n}$, untrained model $f$ with trainable
parameters $\theta$ and embedding function $z$. Hyper-parameters: Number of
optimisation steps $S$ # Initialisation:

2: for $i=1$ to $100$ do

3: optimise $L_{sup}$ over $Z_{l}$

4: end for

5: Set current step to zero $s_{i}=0$ # Main Optimisation Process:

6: while $s_{i}<S$ do

7: Extract features: $V=\\{z(x_{i})\\}_{i=1}^{n}$

8: Construct Graph Matrix $W$

9: Degree Normalisation $\mathcal{W}=D^{\frac{-1}{2}}WD^{\frac{-1}{2}}$

10: Propagate Information via $\mathcal{Q}(F)$

11: Distributed Alignment on $F$

12: $\hat{y_{i}}=\operatorname*{arg\,max}F_{i}\text{ }\forall\text{
}n_{l}+1\leq i\leq n$

13: for $i=1$ to $\lfloor\frac{n_{u}}{b_{u}}\rfloor$ do

14: $B_{L}=\\{x_{i},y_{i}\\}_{i=1}^{b_{l}}\subset Z_{l}$ ,
$B_{u}=\\{x_{i},\hat{y}_{i}\\}_{i=1}^{b_{u}}\subset Z_{u}$

15: Composite Batch $B=B_{L}\cup B_{u}$

16: Optimise $L_{ssl}$ , $s_{i}++$

17: end for

18: end while

###  III-E Multi-Sampling Augmentation

Since the work of [13], several approaches have implemented the use of strong
augmentations [18, 14, 12] to the problem of semi-supervised learning, with
each work having a different way of including augmentation to their framework.
Very recent works [18, 14] have begun using multiple augmented versions of the
same unlabelled image. As yet there is no motivation for why this multiple
sampling idea is preferable to alternatives such as larger batch sizes or
running the code for more steps. In this section we offer a theoretical
motivation for why multi-sampling improves generalization along with a
mathematically bound on its performance gain. With this knowledge in mind we
provide a simple method for including augmentation averaging into our SSL
framework and demonstrate this approach increases accuracy and reduces the
sensitivity of the model to data augmentation.

We view an augmentation strategy $A$ as a set $T$ of transformations
$t:\mathcal{X}\rightarrow\mathcal{X}$ and denote it as
$T=\\{t_{1},t_{2},..,t_{\delta}\\}$. The current standard approach, that the
majority of existing techniques follow, is to simply sample $t\sim T$ once for
each data point and compute some augmented loss $L_{Aug}$:

$L_{\text{Aug}}=\frac{1}{n}\sum_{i=1}^{n}l_{s}(f(\color[rgb]{0,0,0}t(x_{i})\color[rgb]{0,0,0}),y_{i}).$
(10)

However, we argue that such a simple implementation, might not extract the
full information present in the augmentation. If we want to encourage our
model output to be more resistant to data augmentations from $T$, and as a
result produce a more generalisable model, we need to perform a multi-sample
approach. To justify this, we consider the following loss $L_{T}$:

$L_{T}=\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}_{t\sim T}[l(f(t(x_{i})),y)],$ (11)

which measures the risk of the model over the entire augmentation set. If we
want to minimise (11) then we must minimise the expected augmentation error
over the entire transformation set for each data point $\mathbb{E}_{t\sim
T}[l(f(t(x_{i})),y)]$. To see how a multi-sample approach helps us do just
that we use Hoeffding’s inequality which provides us with a probability bound
that the sum of bounded independent random variables deviates from its
expected value by more than a certain amount.

Let $Z_{1},..,,Z_{n_{a}}$ be a sequence of i.i.d random variables. Assume that
$\mathbb{E}[Z]=\mu$ and $\mathbb{P}[a\leq Z_{i}\leq b]=1$ for every $i$. Then,
by Hoeffding’s inequality, for any $\epsilon>0$, one has:

$\mathbb{P}\left[\left|\frac{1}{n_{a}}\sum_{i=1}^{n_{a}}Z_{i}-\mu\right|>\epsilon\right]\leq
2\text{exp}(-2m\epsilon^{2}/(b-a)^{2}).$ (12)

We can rewrite (12) in context of the previously defined augmentation loss
where we replace $Z_{1},..,,Z_{n_{a}}$ with $n_{a}$ samples from $T$ :
$f(t_{1}(x)),f(t_{2}(x)),..,f(t_{n_{a}}(x))$

$\displaystyle\mathbb{P}\left[\left|\frac{1}{n_{a}}\sum_{j=1}^{n_{a}}l(f(t(x_{i})),y_{i})-\mathbb{E}_{t\sim
T}[l(f(t(x_{i})),y_{i})]\right|>\epsilon\right]$ (13) $\displaystyle\leq
2\text{exp}(-2n_{a}\epsilon^{2}/b^{2}).$

As we increase $n_{a}$, we converge in probability to the desired loss
$\mathbb{E}_{t\sim T}[l(f(t(x_{i})),y)]$ for each data point. Subsequently, we
should optimise to a lower of $L_{T}$ meaning that the model output will
fluctuate less over the augmentation set $T$ for the used training data, and
in the process make our model more generalisable. Furthermore, we can see that
the probability is bounded by an exponent whose power is $\propto-n_{a}$.
Therefore, as we increase $n_{a}$ the rate of decrease for the bound also
decreases, making the first few samples far more important than later ones.
This result explains prior behaviours reported but not reasoned in past papers
such as [14]. When using a $n_{a}$ sample the computational complexity
increases as $O(n_{a})$ but as there should be dimishing returns for
increasing $n_{a}$ it should only be necessary to use $n_{a}$ values slightly
above one. As we have shown that a multi-sample approach should offer generic
performance increases for suitable $T$ we change (8) and (9) to a multi-sample
version. For (8) this becomes

$L_{ssl}=\frac{1}{b}\frac{1}{n_{a}}\sum_{i=1}^{b}\sum_{j=1}^{n_{a}}l_{s}(f(t_{j}(x_{i})),y_{i}).$
(14)

where the index $j$ represents repeated samples from $T$. In the ablation
section, we perform a thorough experimental evaluation to test the theoretical
predictions we have made in this section.

Augmentation Implementation Similarly to other approaches we use two different
augmentation strategies: one for labelled data and another for unlabelled
data. However, we apply strong augmentations to both labelled and unlabelled
data, unlike past approaches [12] which reported divergences using this
approach. For strong augmentations we make use of RandAugment [16], and CutOut
augmentation [17]. For completeness we list the full data transformations for
labelled and unlabeld data in Table I and the implementation of RandAugment
and CutOut in the supplementary material.

TABLE I: The augmentation transformations used for labelled and unlabelled data. For normalisation we use the official channel labelled Transform | Unlabelled Transform
---|---
Random Horizontal Flip
Random Crop and Pad
RandAugment Sample | RandAugment Sample
- | RandAugment Sample
CutOut
Normalisation

##  IV Implementation and Evaluation

In this section we detail the implementation of LaplaceNet, including hyper
parameter values and training schemes, and the evaluation protocol we used to
measure our model’s performance and compare against the current state-of-the-
art.

###  IV-A Dataset Description

We use three image classification datasets: CIFAR-10 and CIFAR-100 [39] and
Mini-ImageNet [40]. Following standard protocol, we evaluate our method’s
performance on differing amounts of labelled data for each datasets.

(a) CIFAR-10,CIFAR-100 Containing 50k training images and 10k test images,
these datasets contain 10 and 100 classes respectively. The image size is
small at $32$ by $32$ pixels. We perform experiments using 500,1k,2k and 4k
labels for CIFAR-10 and 4k and 10k labels for CIFAR-100.

(b) Mini-ImageNet A subset of the popular ImageNet dataset, containing 100
classes each with 500 training and 100 test images. The resolution of the
images is $84\times 84$ pixels and represents a much harder challenge than the
CIFAR-10/CIFAR-100 datasets. We use 4k and 10k labels in our experiment.

###  IV-B Implementation Details

Architectures For a fair comparison to older works we use the ”13-CNN”
architecture [32] and for comparison to recent state-of-the-art works we use a
WideResNet (WRN) 28-2 and a WRN-28-8 [41] architecture. We additional use a
ResNet-18 [5] for Mini-Imagenet. For all models we set the drop-out rate to
$0$. For the ”13-CNN” we add a $l_{2}$-normalisation layer to the embedding
function. Infrastructure For all experiments, we use 1-2 Nvidia P100 GPUs.
Training Details: We train with stochastic gradient descent (SGD) using
Nesterov momentum $n_{m}$ with value $0.9$ and weight decay $\omega$ with
value $0.0005$. We use an initial learning rate of $l_{r}=0.3$ and use
$S=250000$ optimisation steps in total. We utilise a cosine learning rate
decay such that the learning rate decays to zero after 255000 steps. We do not
make use of any $EMA$ model averaging. Parameters We list the parameter values
used in Table II. Most parameter values are common parameter settings from the
deep learning field and are not fine-tuned to our application. Being able to
work with reasonably generic parameters is well suitable to the task of SSL
where using fine-tuning over validation sets is often impossible in practical
applications.

TABLE II: List of hyperparameters used in the paper across the CIFAR-10/100
and Mini-Imagenet datasets.

Parameter | CIFAR-10 | CIFAR-100 | Mini-ImageNet
---|---|---|---
$\alpha$ | 1.0 | 0.5 | 0.5
$\mu$ | 0.01 | 0.01 | 0.01
$k$ | 50 | 50 | 50
$S$ | $2.5\times 10^{5}$ | $2.5\times 10^{5}$ | $2.5\times 10^{5}$
$b$ | 300 | 100 | 100
$b_{l}$ | 48 | 50 | 50
$l_{r}$ | 0.03 | 0.03 | 0.1
$n_{m}$ | 0.9 | 0.9 | 0.9
$\omega$ | $5\times 10^{-4}$ | $5\times 10^{-4}$ | $5\times 10^{-4}$
$n_{a}$ | 3 | 3 | 3

TABLE III: Top-1 error rate on the CIFAR-10/100 datasets for our method and other methods using the 13-CNN architecture. We denote with $\dagger$ experiments we have ran. Dataset | CIFAR-10 | CIFAR-100
---|---|---
Method | 500 | 1000 | 2000 | 4000 | 4000 | 10000
Supervised Baseline | 37.12 $\pm$ 0.89 | 26.60 $\pm$ 0.22 | 19.53 $\pm$ 0.12 | 14.02 $\pm$ 0.10 | 53.10 $\pm$ 0.34 | 36.59 $\pm$ 0.47
Consistency Based Approaches
$\Pi$-Model | - | - | - | 12.36 $\pm$ 0.31 | - | 39.19 $\pm$ 0.36
MT$\dagger$ | 27.45 $\pm$ 2.64 | 21.55 $\pm$ 1.48 | 15.73 $\pm$ 0.31 | 12.31 $\pm$ 0.20 | 45.36 $\pm$ 0.49 | 36.08 $\pm$ 0.51
VAT | - | - | - | 11.36 $\pm$ 0.34 | - | -
MT-LP | 24.02 $\pm$ 2.44 | 16.93 $\pm$ 0.70 | 13.22 $\pm$ 0.29 | 10.61 $\pm$ 0.28 | 43.73 $\pm$ 0.20 | 35.92 $\pm$ 0.47
SNTG | – | 18.41$\pm$0.52 | 13.64$\pm$0.32 | 9.89$\pm$0.34 | – | 37.97$\pm$0.29
MT-fast-SWA | - | 15.58 $\pm$ 0.12 | 11.02 $\pm$ 0.12 | 9.05 $\pm$ 0.21 | - | 34.10 $\pm$ 0.31
MT-ICT | - | 15.48 $\pm$ 0.78 | 9.26 $\pm$ 0.09 | 7.29 $\pm$ 0.02 | - | -
Dual Student | – | 14.17$\pm$0.38 | 10.72$\pm$0.19 | 8.89$\pm$0.09 | – | 32.77$\pm$0.24
Pseudo-Labelling Approaches
TSSDL$\dagger$ | - | 21.13 $\pm$ 1.17 | 14.65 $\pm$ 0.33 | 10.90 $\pm$ 0.23 | - | -
LP$\dagger$ | 32.40 $\pm$ 1.80 | 22.02 $\pm$ 0.88 | 15.66 $\pm$ 0.35 | 12.69 $\pm$ 0.29 | 46.20 $\pm$ 0.76 | 38.43 $\pm$ 1.88
DAG | 9.30 $\pm$ 0.73 | 7.42 $\pm$ 0.41 | 7.16 $\pm$ 0.38 | 6.13 $\pm$ 0.15 | 37.38 $\pm$ 0.64 | 32.50 $\pm$ 0.21
Pseudo-Label Mixup | 8.80 $\pm$ 0.45 | 6.85 $\pm$ 0.15 | - | 5.97 $\pm$ 0.15 | 37.55 $\pm$ 1.09 | 32.15 $\pm$ 0.50
LaplaceNet $\dagger$ |  5.68 $\pm$ 0.08 |  5.33 $\pm$ 0.02 |  4.99 $\pm$ 0.12 |  4.64 $\pm$ 0.07 |  31.64 $\pm$ 0.02 |  26.60 $\pm$ 0.23
TABLE IV: Top-1 error rate for CIFAR-10/100. All methods, except MixMatch, are tested using the same code-base and use the same model code, the same optimiser (SGD) with the same optimisation parameters, the same number of optimisation steps and the same RandAugment implementation. We denote with $\dagger$ experiments we have ran. Dataset | CIFAR-10 | CIFAR-100
---|---|---
Method | 500 | 2000 | 4000 | 4000 | 10000
Other Methods
MixMatch | 9.65 $\pm$ 0.94 | 7.03 $\pm$ 0.15 | 6.34 $\pm$ 0.06 | — | —
Same Codebase
UDA $\dagger$ | 6.88 $\pm$ 0.74 | 5.61 $\pm$ 0.16 | 5.40 $\pm$ 0.19 | 36.19 $\pm$ 0.39 | 31.49 $\pm$ 0.19
FixMatch(RA) $\dagger$ | 5.92 $\pm$ 0.11 | 5.42 $\pm$ 0.11 | 5.30 $\pm$ 0.08 | 34.87 $\pm$ 0.17 | 30.89 $\pm$ 0.18
LaplaceNet $\dagger$ |  5.57 $\pm$ 0.60 |  4.71 $\pm$ 0.05 |  4.35 $\pm$ 0.10 |  33.16 $\pm$ 0.22 |  27.49 $\pm$ 0.22
TABLE V: Top-1 error rate for Mini-ImageNet. We compare against methods which have used an identical ResNet-18 architecture. Method | 4000 | 10000
---|---|---
Supervised Baseline | 66.04 $\pm$ 0.32 | 52.89 $\pm$ 0.33
Consistency Regularisation Methods
MT | 72.51 $\pm$ 0.22 | 57.55 $\pm$ 1.11
MT-LP | 72.78 $\pm$ 0.15 | 57.35 $\pm$ 1.66
Pseudo-Label Methods
LP | 70.29 $\pm$ 0.81 | 57.58 $\pm$ 1.47
Pseudo-Label Mixup | 56.49 $\pm$ 0.51 | 46.08 $\pm$ 0.11
LaplaceNet |  46.32 $\pm$ 0.27 |  39.43 $\pm$ 0.09

###  IV-C Evaluation Protocol

We evaluate the performance of LaplaceNet on the CIFAR-10/CIFAR-100 and Mini-
Imagenet datasets and compare against the current SOTA models for semi-
supervised learning. For ease of comparison, we split the current SOTA into
two groups.

  1. 1.

Methods which used the $13$-CNN architecture [32]: $\Pi$-Model [30], Mean
Teacher(MT) [32], Virtual Adversarial Training (VAT) [31], Label Propogation
for Deep Semi-Supervised Learning (LP) [33], Smooth Neighbors on Teacher
Graphs (SNTG) [42], Stochastic Weight Averaging(SWA) [43], Interpolation
Consistency Training (ICT) [21], Dual Student [44], Transductive Semi-
Supervised Deep Learning(TSSDL) [35], Density-Aware Graphs (DAG) [38] and
Pseudo-Label Mixup [20]. Unfortunately, due to the natural progress in the
field, each paper has different implementation choices which are not
standardised. Despite this, comparisons to this group are still useful as a
barometer for model performance.

  2. 2.

Recent methods which used the WRN [41] (MixMatch [19], FixMatch (RandAugment
variant) [12] and UDA [13]). To guarantee a fair comparison to these
techniques, and as suggested by [45], we used a shared code-base for UDA and
FixMatch which reimplemented the original baselines. Additionally we then
ensured UDA and FixMatch used the same model code, the same optimiser with the
same parameters, the same number of optimisation steps and the same
RandAugment implementation as our approach.

Evaluation Protocol For each dataset we use the official train/test partition
and use the Top-1 error rate as the evaluation metric. For each result we give
the mean and standard deviation over five label splits.

##  V Results and Discussion

In this section, we discuss the experiments we performed to evaluate and
compare our model against the state-of-the-art (SOTA). Additionally, we detail
several ablation experiments which explore the benefits of graph-based pseudo-
labels, the effect of augmentation averaging and evaluating the importance of
individual components.

###  V-A Comparison to SOTA

Firstly, we test our model on the less complex CIFAR-10 and CIFAR-100
datasets. In Table III, we compare LaplaceNet against the first group of
methods using the $13$-CNN network. Our approach, by some margin, produces the
best results on CIFAR-10 and CIFAR-100 and represents a new SOTA for pseudo-
labels methods. We obtain a lower error rate using 500 labels than the recent
work of Arazo et al [20] obtain using 4000 labels. For CIFAR-100 LaplaceNet is
a full $6\%$ more accurate than any other approach and the first method to
achieve an error rate below $30\%$ on CIFAR-100 using 10k labels. In Table IV
we compare against the second group of methods using the WRN-28-2 network.
LaplaceNet is again the best performing method, outperforming the recent works
of UDA [13] and FixMatch [12]. In particular we find a significant increase in
performance on the more complex CIFAR-100 dataset and beat the other
considered methods by more than $3\%$ with 10k labels.

To test the performance of LaplaceNet on a more complex dataset, we evaluate
our model on the Mini-ImageNet dataset, which is a subset of the well known
ImageNet dataset and in Table V we compare our results against all others
methods which have used this dataset. Once again, we find our method performs
very well, producing an error rate a $10\%$ and $7\%$ better than any other
method on 4k and 10k labelled images respectively. Demonstrating our approach
can be applied to complex problems in the field. Additionally, we are more
than $20\%$ more accurate that the nearest graphical approach (LP).

To test the effect of increasing network size on our performance we also ran
our model on CIFAR-10/100 using an WRN-28-8(26 million parameters)
architecture and and compared that to the WRN-28-2(1.6 million parameters)
architecture in Table VI. Unsurprisingly, we achieved a large performance
improvement using a WRN-28-8 on both CIFAR-10 and CIFAR-100, with an $2.87$
error rate on CIFAR-10 using 4k labels and an $22.11\%$ error rate on
CIFAR-100 using 10k labels.

TABLE VI: The effect on Top-1 error rate by scaling up the neural network in
size from a WRN-28-2 to a WRN-28-8 on the CIFAR-10/100 datasets.

Dataset | CIFAR-10 | CIFAR-100
---|---|---
Model | 500 | 4000 | 4000 | 10000
WRN-28-2 | 5.57 $\pm$ 0.60 | 4.35 $\pm$ 0.10 | 33.16 $\pm$ 0.22 | 27.49 $\pm$ 0.22
WRN-28-8 | 3.81 $\pm$ 0.37 | 2.87 $\pm$ 0.18 | 26.61 $\pm$ 0.10 | 22.11 $\pm$ 0.23

(a)

(b)

Figure 1: Experimental comparison of the effect of using pseudo-labels
produced in a graphical framework versus pseudo-labels generated by the neural
network on the Top-1 error rate on the CIFAR-100 dataset ((a) 4k and (b) 10k
labelled images) with the 13-CNN network. Using graphically produced pseudo-
labels we achieve a much higher accuracy than using the network predictions.

###  V-B Graph Based Pseudo-Labels

Many pseudo-label based techniques [12] [20] have produced state-of-the-art
results using pseudo-labels generated directly by the network rather than
using an energy based approach such as label propogation on a constructed
graph, which is computationally more complex. Therefore, in this section we
examine whether there is any advantage in using a graph based approach? To
test the importance of graph based pseudo-labels, we created two variants of
LaplaceNet, both without distribution alignment and with $n_{a}=1$.

  1. 1.

The pseudo-labels are generated directly from the network predictions:
$\hat{y}_{i}=\text{argmax }f(x_{i})\text{ }\forall\text{ }i>l$

  2. 2.

The pseudo-labels are generated from the graph, as in Equation 6,
$\hat{y}_{i}=\text{argmax}_{j}\text{ }F_{ij}\text{ }\forall\text{ }i>l$

We then compared the Top-1 error rate of these two variants on the CIFAR-100
dataset, see Fig 1. The graph-variant greatly outperformed the direct
prediction variant, emphasising the clear advantage that graphically produced
pseudo-labels have. What is contributing to this advantage? As an energy-based
approach, propogation on the graph incorporates information on the global
structure of the data, whilst the network is making a purely local decision at
each point. Arazo et al [20] showed that naive network based pseudo-label
approach could not generate an accurate solution for the ”two moons” toy
dataset, despite the fact that this problem has been solved by graphical
methods for some time [25]. Thus demonstrating that purely local decisions are
detrimental to accuracy when the global structure of data isn’t taken into
account.

###  V-C Augmentation Averaging

In this paper we justify a multiple augmentation approach to further improve
semi-supervised models. In this section, we present the experimental
verification of our theoretical predictions about augmentation averaging as
well as comparing its effect to potential alternative techniques. To test the
effect of augmentation averaging we ran our approach on the CIFAR-100 dataset
using the 13-CNN network for a range of values $n_{a}=[1,3,5]$. Additionally
we compared the changed caused by augmentation averaging to the more common
approaches of scaling the batch size $b$ and labelled batch size $b_{l}$ by
$[1,3,5]$ and scaling the number of optimisation steps $S$ by $[1,3,5]$

To quantify the effect of a given change we use two measures: the augmentation
invariance of the classifier, which we define in this paper, and Top-1 error.
Augmentation invariance measures the extent to which the classifier’s
performance changes under data augmentation. Given an augmenation function
$u:\mathcal{X}\rightarrow\mathcal{X}$ and a classifier $f_{\theta}$ the
augmentation invariance $V$ with respect to a dataset $Z$ made up of $n$
point-label pairs $Z=\\{x_{i},y_{i}\\}_{i=1}^{n}$ is given by

$V_{Z}=\frac{\frac{1}{n}\sum_{i=1}^{n}\mathbbm{1}_{\operatorname*{arg\,max}f_{\theta}(u(x_{i}))=y_{i}}}{\frac{1}{n}\sum_{i=1}^{n}\mathbbm{1}_{\operatorname*{arg\,max}f_{\theta}(x_{i})=y_{i}}},$
(15)

which can be viewed as the performance ratio with and without data
augmentation. We consider both the augmentation invariance of our model with
respect to the fully labelled training and test data in order to give a full
picture of the model’s invariance, but we still only use a subset of the
labelled data for training.

TABLE VII: The effect of removing individual components from the baseline model on Top-1 error rate for CIFAR-100 on the 13-CNN network. | CIFAR-100
---|---
Model | 4k | 10k
Baseline | 32.41 $\pm$ 0.25 | 27.37 $\pm$ 0.20
Component Removed |  |
RandAugment | 44.43 $\pm$ 0.66 | 34.75 $\pm$ 0.23
Distribution Alignment | 33.26 $\pm$ 0.24 | 29.07 $\pm$ 0.07
MixUp | 33.74 $\pm$ 0.26 | 28.02 $\pm$ 0.20

Figure 2: A comparison on the effect of increasing batch size versus
increasing the number of augmentation samples on Top-1 error rate, test data
augmentation invariance and training data augmentation invariance for the
CIFAR-100 dataset. Increasing the amount of augmentation averaging decreased
the error rate whilst also decreasing the sensitivity of the model’s output
predictions to augmented data. Increasing the batch size had a similar effect
on the model’s sensitivity, but it offered no improvement to model accuracy.

In Fig 2 we present our findings. We found that naively scaling the number of
optimisation steps without changing the hyperparameters led to the model
diverging as we spent too many epochs at a high learning rate. Therefore, we
provide results for the other two considered techniques which can be directly
compared. As theorised in Section III we find that increasing the number of
augmentation samples decreased the sensitivity of the model’s predictions to
augmentation on both the training and test data.An almost identical effect was
found by scaling the batch size. However, the major difference between the two
is their effect on Top-1 error rate. We found scaling the batch size offered
no improvement to Top-1 error, in-fact the largest batch size offered the
worst outcome, whilst increasing the number of augmentation samples noticeably
improved the model’s accuracy. Additionally as theorised in Section III, we
see that the gain in performance from $n_{a}=1\rightarrow 3$ is much greater
than $n_{a}=3\rightarrow 5$, supporting our statements regarding the
exponential bound in probability. These results suggests that scaling the
number of augmentation samples could be a great option for semi-supervised
models using suitable strong augmentations.

###  V-D Component Evaluation

As LaplaceNet combines several different techniques, we tested the importance
of strong augmentation, distribution alignment and MixUp to the overall
accuracy of the model. We created a baseline model ($n_{a}=1$) and then remove
each component one at a time and tested the performance on the CIFAR-100
dataset, see Table VII. Whilst the removal of each component decreased the
performance of the model, it is clear the most crucial component to model
performance is strong augmentation and removing it drastically reduces model
accuracy. However, unlike other works [20] we find that whilst MixUp [22]
offers a small advantage is it not critical for composite batch pseudo-label
approaches. This may be due to the advantages of graph-based approaches
overcoming the flaws of naive neural network predictions.

##  VI Conclusion

We propose a new graph based pseudo-label approach for semi-supervised image
classification, LaplaceNet, that outperforms the current state-of-the-art on
several datasets whilst having a much lower model complexity. Our model
utilises a simple single term loss function without the need for additionally
complexity whilst additionally avoiding the need for confidence thresholding
or temperature sharpening which was thought to be essential for state-of-the-
art performance. We instead generate accurate pseudo-labels through a graph
based technique with distribution alignment. We also explore the role that
augmentation plays in semi-supervised learning and justify a multi-sampling
approach to augmentation which we demonstrate through rigorous experimentation
improves both the generalisation of the network as well as the model’s
sensitivity to augmented data.

## Acknowledgment

PS thanks the UK Engineering and Physical Sciences Research Council (EPSRC)
and the National Physical Laboratory (NPL) for supporting this work. AIAR
gratefully acknowledges the financial support of the CMIH and CCIMI University
of Cambridge. CBS acknowledges support from the Philip Leverhulme Prize, the
Royal Society Wolfson Fellowship, the EPSRC grants EP/S026045/1 and
EP/T003553/1, EP/N014588/1, EP/T017961/1, the Wellcome Innovator Award
RG98755, the Leverhulme Trust project Unveiling the invisible, the European
Union Horizon 2020 research and innovation programme under the Marie
Skodowska-Curie grant agreement No. 777826 NoMADS, the Cantab Capital
Institute for the Mathematics of Information and the Alan Turing Institute.

##  Appendix A Augmentation Pool

In this work we use RandAugment [16] rather than a learnt augmentation
strategy such as AutoAugment [15] which has a large computational cost. In
Table IX we detail the augmentation pool used. Additionally, we apply CutOut
[17] augmentation after RandAugment sampling.

We use two different augmentation strategies in our work: one for labelled
data and one for unlabelled data. We use ”strong” augmentations, RandAugment
and CutOut, on both labelled and unlabelled data with the only difference
being that we sample once from RandAugment for labelled data and twice for
unlabelled data. Given a pre-selected list of transformations, RandAugment
randomly samples from the list with each transformation having a magnitude
parameter. Rather than optimising this parameter on a validation set, which
may not exist in typical semi-supervised applications, we sample a random
magnitude from a pre-set range. This is same as is done in FixMatch [12] and
UDA [13]. We list the transformation pool for RandAugment and the
implementation of CutOut in Table IX.

TABLE VIII: Computational time taken for our approach using 4k labelled images on the CIFAR-100 dataset using the 13-CNN architecture. We provide the time taken for a number of different settings used in the results section. All experiments were performed using one NVIDIA P100 GPU. Model | Computational Time (Hours)
---|---
Baseline | 7.52 $\pm$ 0.04
Component Removal
No Distribution Alignment | 6.18 $\pm$ 0.01
No Strong Augmentation | 5.84 $\pm$ 0.03
No Graphical Propogation | 6.32 $\pm$ 0.01
Model Scaling
$3\times$-Batch-size | 12.28 $\pm$ 0.03
$5\times$-Batch-size | 17.23 $\pm$ 0.06
$3\times$-Samples | 12.88 $\pm$ 0.01
$5\times$-Samples | 18.14 $\pm$ 0.11

##  Appendix B Computational Time

To give clarity on the how long our code takes to run we provide the
computational run times of LaplaceNet on the CIFAR-100 dataset using the
13-CNN model for a variety of settings, see Table VIII. Each experiment was
run on one P100 NVIDIA GPU. From Table VIII, we see that the time increased
caused by increasing the batch size or increasing the number of samples is
very similar. Component-wise, removing strong augmentation gives the largest
decrease in computational time whilst removing the graphical propogation saved
just over an hour on CIFAR-100. This represent a very small time trade off
given the advantages present in using graphical pseudo-labels.

TABLE IX: List of Transformations used in our application of RandAugment as
well their description and magnitude range. Additionally, we list the CutOut
transformation used at the end of RandAugment sampling.

Transformation | Description | Range
---|---|---
RandAugment Transformations
Autocontrast | Maximises the image contrast by setting the darkest (lightest) pixel to black (white) | —–
Brightness | Adjusts the brightness of the image. where $B=0$ returns a black image. image | $B\in[0.05,0.95]$
Color | Adjusts the colour balance of the image. $C_{l}=0$ returns a black and white image. | $C_{l}\in[0.05,0.95]$
Contrast | Controls the contrast of the image. $C_{o}=0$ returns a gray image. | $C_{o}\in[0.05,0.95]$
Equalise | Equalises the image histogram. | —–
Identity | Returns the original image. | —–
Posterise | Reduces each pixel to $B$ bits. | $B\in[4,8]$
Rotate | Rotates the image by $\theta$ degrees. | $\theta\in[-30,30]$
Sharpness | Adjusts the sharpness of the image, where $S=0$ returns a blurred image | $S\in[0.05,0.95]$
Shear X | Shears the image along the horizontal axis with rate $R$. | $R\in[-0.3,0.3]$
Shear Y | Shears the image along the vertical axis with rate $R$ | $R\in[-0.3,0.3]$
Solarize | Inverts all pixels above a threshold value of $T$ | $T\in[0,1]$
Translate X | Translates the image horizontally by ($\lambda$×image width) pixels. | $\lambda\in[-0.3,0.3]$
Translate Y | Translates the image vertically by ($\lambda$×image height) pixels | $\lambda\in[-0.3,0.3]$
CutOut Augmentation
CutOut | Sets a random square patch of side-length ($L$×image width) pixels to grey | $L\in[0,0.5]$

## References

  * [1] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in Neural Information Processing Systems (NIPS)_ , 2012, pp. 1097–1105.
  * [3] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [4] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 7132–7141.
  * [5] F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, and X. Tang, “Residual attention network for image classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3156–3164.
  * [6] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in _Advances in neural information processing systems_ , 2015, pp. 91–99.
  * [7] R. Girshick, “Fast r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1440–1448.
  * [8] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016.
  * [9] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [10] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)_ , 2015, pp. 234–241.
  * [11] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)_ , vol. 40, no. 4, pp. 834–848, 2017.
  * [12] K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” _Advances in neural information processing systems_ , 2020.
  * [13] Q. Xie, Z. Dai, E. Hovy, M.-T. Luong, and Q. V. Le, “Unsupervised data augmentation for consistency training,” _Neural Information Processing Systems_ , 2020.
  * [14] D. Berthelot, N. Carlini, E. Cubuk, A. Kurakin, H. Zhang, and C. Raffel, “Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring,” in _Eighth International Conference on Learning Representations_ , 2020.
  * [15] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, “Autoaugment: Learning augmentation policies from data,” _arXiv preprint arXiv:1805.09501_ , 2018.
  * [16] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, “Randaugment: Practical automated data augmentation with a reduced search space,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 702–703.
  * [17] T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” _arXiv preprint arXiv:1708.04552_ , 2017.
  * [18] Z. Hu, Z. Yang, X. Hu, and N. R, “Simple: Similar pseudo label exploitation for semi-supervised classification,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [19] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” in _Advances in Neural Information Processing Systems_ , 2019.
  * [20] E. Arazo, D. Ortego, P. Albert, N. E. O’Connor, and K. McGuinness, “Pseudo-labeling and confirmation bias in deep semi-supervised learning,” _arXiv preprint arXiv:1908.02983_ , 2019.
  * [21] V. Verma, A. Lamb, J. Kannala, Y. Bengio, and D. Lopez-Paz, “Interpolation consistency training for semi-supervised learning,” _International Joint Conference on Artificial Intelligence (IJCAI)_ , 2019.
  * [22] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” _arXiv preprint arXiv:1710.09412_ , 2017.
  * [23] M. Belkin, P. Niyogi, and V. Sindhwani, “Manifold regularization: A geometric framework for learning from labeled and unlabeled examples,” _Journal of machine learning research_ , vol. 7, 2006.
  * [24] M. Belkin, I. Matveeva, and P. Niyogi, “Regularization and semi-supervised learning on large graphs,” in _International Conference on Computational Learning Theory_. Springer, 2004, pp. 624–638.
  * [25] X. Zhu, Z. Ghahramani, and J. D. Lafferty, “Semi-supervised learning using gaussian fields and harmonic functions,” in _P International conference on Machine learning (ICML)_ , 2003, pp. 912–919.
  * [26] D. Zhou, O. Bousquet, T. N. Lal, J. Weston, and B. Schölkopf, “Learning with local and global consistency,” in _Advances in Neural Information Processing Systems (NIPS)_ , 2004, pp. 321–328.
  * [27] Y. Grandvalet and Y. Bengio, “Semi-supervised learning by entropy minimization,” in _Advances in neural information processing systems_ , 2005, pp. 529–536.
  * [28] K. I. Kim, F. Steinke, and M. Hein, “Semi-supervised regression using hessian energy with an application to semi-supervised dimensionality reduction,” in _Advances in Neural Information Processing Systems (NIPS)_ , 2009, pp. 979–987.
  * [29] O. Chapelle, A. Zien, and B. Schölkopf, _Semisupervised learning_. MIT Press, 2006.
  * [30] S. Laine and T. Aila, “Temporal ensembling for semi-supervised learning,” _International Conference on Learning Representations (ICLR)_ , 2017.
  * [31] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii, “Virtual adversarial training: a regularization method for supervised and semi-supervised learning,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)_ , vol. 41, no. 8, pp. 1979–1993, 2018.
  * [32] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” in _Advances in neural information processing systems (NIPS)_ , 2017, pp. 1195–1204.
  * [33] A. Iscen, G. Tolias, Y. Avrithis, and O. Chum, “Label propagation for deep semi-supervised learning,” _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [34] D.-H. Lee, “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,” in _Workshop on Challenges in Representation Learning, ICML_ , 2013.
  * [35] W. Shi, Y. Gong, C. Ding, Z. MaXiaoyu Tao, and N. Zheng, “Transductive semi-supervised deep learning using min-max features,” in _European Conference on Computer Vision (ECCV)_ , 2018, pp. 299–315.
  * [36] S. Thulasidasan, G. Chennupati, J. Bilmes, T. Bhattacharya, and S. Michalak, “On mixup training: Improved calibration and predictive uncertainty for deep neural networks.” in _In 33rd Conference on Neural Information Processing Systems._ , 2019.
  * [37] A. I. Aviles-Rivero, N. Papadakis, R. Li, S. M. Alsaleh, R. T. Tan, and C.-B. Schonlieb, “Beyond supervised classification: Extreme minimal supervision with the graph 1-laplacian,” _arXiv:1906.08635_ , 2019.
  * [38] S. Li, B. Liu, D. Chen, Q. Chu, L. Yuan, and N. Yu, “Density-aware graph for deep semi-supervised visual recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 13 400–13 409.
  * [39] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images.” 2009.
  * [40] O. Vinyals, C. Blundell, T. Lillicrap, and D. e. a. Wierstra, “Matching networks for one shot learning.” NIPS, 2016.
  * [41] S. Zagoruyko and N. Komodakis, “Wide residual networks,” in _Proceedings of the British Machine Vision Conference_ , 2016.
  * [42] Y. Luo, J. Zhu, M. Li, Y. Ren, and B. Zhang, “Smooth neighbors on teacher graphs for semi-supervised learning,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 8896–8905.
  * [43] B. Athiwaratkun, M. Finzi, P. Izmailov, and A. G. Wilson, “There are many consistent explanations of unlabeled data: Why you should average,” _International Conference on Learning Representations (ICLR)_ , 2019.
  * [44] Z. Ke, D. Wang, Q. Yan, J. Ren, and R. W. Lau, “Dual student: Breaking the limits of the teacher in semi-supervised learning,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019.
  * [45] A. Oliver, A. Odena, C. Raffel, E. Cubuk, and I. Goodfellow, “Realistic evaluation of deep semi-supervised learning algorithms,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018.


# Injectivity of sampled Gabor phase retrieval in spaces with general
integrability conditions

Matthias Wellershoff University of Maryland, Department of Mathematics,
William E. Kirwan Hall, 4176 Campus Drive, College Park, MD 20742,
<EMAIL_ADDRESS>
###### Abstract

It was recently shown that functions in $L^{4}([-B,B])$ can be uniquely
recovered up to a global phase factor from the absolute values of their Gabor
transforms sampled on a rectangular lattice. We prove that this remains true
if one replaces $L^{4}([-B,B])$ by $L^{p}([-B,B])$ with $p\in[1,\infty]$. To
do so, we adapt the original proof by Grohs and Liehr and use a classical
sampling result due to Beurling. Furthermore, we present a minor modification
of a result of Müntz–Szász type by Zalik. Finally, we consider the
implications of our results for more general function spaces obtained by
applying the fractional Fourier transform to $L^{p}([-B,B])$ and for more
general nonuniform sampling sets.

Keywords Phase retrieval, Gabor transform, Sampling theory, Time-frequency
analysis

Mathematics Subject Classification (2010) 94A12, 94A20

##  1 Introduction

In this paper, we consider the _Gabor transform_ of functions $f\in
L^{p}(\mathbb{R})$, $p\in[1,\infty]$, given by

$\mathcal{G}f(x,\omega):=2^{1/4}\int_{\mathbb{R}}f(t)\mathrm{e}^{-\pi(t-x)^{2}}\mathrm{e}^{-2\pi\mathrm{i}t\omega}\,\mathrm{d}t,\qquad(x,\omega)\in\mathbb{R}^{2},$

and try to understand if one can recover $f$ from measurements of the absolute
value $\left\lvert\mathcal{G}f\right\rvert$ on discrete sets
$S\subset\mathbb{R}^{2}$. This so-called sampled Gabor phase retrieval problem
has recently been studied extensively [2, 3, 10, 11]. It is an elegant
mathematical problem in the sense that it is rather easy to state while, at
the same time, being less easy to solve. Moreover, it is connected to certain
audio processing applications such as the phase vocoder [8, 18].

A hallmark of all phase retrieval problems is that signals cannot be fully
recovered from phaseless measurements. For the Gabor phase retrieval problem,
we can see that the functions $f$ and $\mathrm{e}^{\mathrm{i}\alpha}f$, where
$\alpha\in\mathbb{R}$, generate the same measurements

$\left\lvert\mathcal{G}(\mathrm{e}^{\mathrm{i}\alpha}f)\right\rvert=\left\lvert\mathrm{e}^{\mathrm{i}\alpha}\mathcal{G}f\right\rvert=\left\lvert\mathcal{G}f\right\rvert.$

Hence, we are not able to distinguish between $f$ and
$\mathrm{e}^{\mathrm{i}\alpha}f$ on the basis of their Gabor transform
magnitude samples. We will therefore consider the equivalence relation $\sim$
on $L^{p}(\mathbb{R})$ defined by

$f\sim
g:\iff\operatorname{\exists}\alpha\in\mathbb{R}:f=\mathrm{e}^{\mathrm{i}\alpha}g.$
(1)

With the help of this relation, we can introduce the phase retrieval operator
$\mathcal{A}:V/{\sim}\to[0,\infty)^{S}$, where $V$ is a subspace of
$L^{p}(\mathbb{R})$, by

$\mathcal{A}(f)(x,\omega):=\left\lvert\mathcal{G}f(x,\omega)\right\rvert,\qquad(x,\omega)\in
S,$

for $f\in V/{\sim}$. The _sampled Gabor phase retrieval problem_ is the
problem of inverting $\mathcal{A}$ when $S\subset\mathbb{R}^{2}$ is discrete.

We note that it has long been known that one can invert $\mathcal{A}$ for
$V=L^{2}(\mathbb{R})$ and $S=\mathbb{R}^{2}$. In applications, one does
typically not have access to measurements of the Gabor transform magnitude on
the entire time-frequency plane, however, and we thus believe that the sampled
Gabor phase retrieval problem is a natural first step towards a better
understanding of settings encountered in practice.

Relatively little was known about the inversion of $\mathcal{A}$ for discrete
sets $S$ until recently when a series of breakthroughs was presented in the
papers [2, 3, 10, 11]. For the genesis of this paper, the work in [11] was
most important. The authors of that paper show that sampled Gabor phase
retrieval is unique when $V=L^{4}([-B,B])$ and
$S=\mathbb{Z}\times(4B)^{-1}\mathbb{Z}$. The assumption that signals lie in
$L^{4}([-B,B])$ does not seem very natural, however, such that we are
interested in extending the result to spaces with more general integrability
conditions, and notably to $L^{2}([-B,B])$ as well as $L^{1}([-B,B])$. We will
do so here and prove the following main result.

######  Theorem 1.1.

Let $p\in[1,\infty]$, $B>0$ and $b\in(0,\tfrac{1}{4B})$. Then, the following
are equivalent for $f,g\in L^{p}([-B,B])$:

  1. 1.

$f=\mathrm{e}^{\mathrm{i}\alpha}g$ for some $\alpha\in\mathbb{R}$,

  2. 2.

$\left\lvert\mathcal{G}f\right\rvert=\left\lvert\mathcal{G}g\right\rvert$ on
$\mathbb{N}\times b\mathbb{Z}$.

We observe that this theorem is almost optimal in view of the results
presented in [2]: there, for any lattice $S\subset\mathbb{R}^{2}$ in the time-
frequency plane, explicit examples $f,g\in L^{2}(\mathbb{R})$ were constructed
which do not agree up to global phase but which satisfy that

$\left\lvert\mathcal{G}f(x,\omega)\right\rvert=\left\lvert\mathcal{G}g(x,\omega)\right\rvert,\qquad(x,\omega)\in
S.$

In particular, it is necessary to restrict the Gabor phase retrieval problem
to a proper subspace $V$ of $L^{2}(\mathbb{R})$ in order to obtain a
uniqueness result from samples. It may not surprise the reader that one may
further generalise Theorem 1.1 in multiple ways to include more general
function spaces obtained by taking fractional Fourier transforms of elements
in $L^{p}([-B,B])$ or more general nonuniform sampling sets. Both of these
generalisation have already been suggested in [11] and we adapt them here. We
remark that during the review process for this paper, other manuscripts
addressing aspects of sampled short-time Fourier transform phase retrieval
have emerged, such as [1, 12, 13, 19].

#### Outline

In Section 2, we introduce some basic concepts needed for the further
understanding of this paper. Specifically, we introduce the fractional Fourier
transform and the Paley–Wiener spaces along with some of their most relevant
properties. Additionally, we provide two core insights: the first of those
being that the Gabor transform of a bandlimited function is bandlimited in the
first argument (cf. Lemma 2.4) and the second of those being a result of
Müntz–Szász type by Zalik (cf. Theorem 2.6).

In Section 3, we prove our main result, which states that for sufficiently
dense uniformly discrete sets $X$ and certain countable sets $\Omega$,
functions in $\mathrm{PW}_{B}^{p}$, $p\in[1,\infty]$, are uniquely determined
(up to global phase) from their Gabor magnitudes on $X\times\Omega$. We
finally extend this result by applying the fractional Fourier transform to
rotate the time-frequency plane.

#### Notation

We use the convention

$\mathcal{F}f(\xi)=\int_{\mathbb{R}^{d}}f(t)\mathrm{e}^{-2\pi\mathrm{i}(t,\xi)}\,\mathrm{d}t,\qquad\xi\in\mathbb{R}^{d},$

for the Fourier transform on the Schwartz space $\mathcal{S}(\mathbb{R}^{d})$.
It is well-known that the Fourier transform can be extended to the space of
tempered distributions $\mathcal{S}^{\prime}(\mathbb{R}^{d})$ and thereby to
all $L^{p}(\mathbb{R}^{d})$, $p\in[1,\infty]$. The inverse of the Fourier
transform is given by

$\mathcal{F}^{-1}g(t)=\int_{\mathbb{R}^{d}}g(\xi)\mathrm{e}^{2\pi\mathrm{i}(\xi,t)}\,\mathrm{d}\xi,\qquad
t\in\mathbb{R}^{d},$

for $g\in\mathcal{S}(\mathbb{R}^{d})$ and can likewise be extended to
$\mathcal{S}^{\prime}(\mathbb{R}^{d})$. We use the translation operators
$\\{\operatorname{T}_{x}\\}_{x\in\mathbb{R}}$ given by

$\operatorname{T}_{x}f(t):=f(t-x),\qquad t\in\mathbb{R},$

for $x\in\mathbb{R}$, as well as the modulation operators
$\\{\operatorname{M}_{\omega}\\}_{\omega\in\mathbb{R}}$ given by

$\operatorname{M}_{\omega}f(t):=\mathrm{e}^{2\pi\mathrm{i}t\omega}f(t),\qquad
t\in\mathbb{R},$

for $\omega\in\mathbb{R}$. We denote the normalised Gaussian by
$\phi(t)=2^{1/4}\mathrm{e}^{-\pi t^{2}}$ where $t\in\mathbb{R}$. The
cardinality of a finite set $X\subset\mathbb{R}$ is denoted by $\left\lvert
X\right\rvert$. Finally, we denote the reflection and complex conjugation of
functions by $f^{\\#}(t):=\overline{f(-t)}$ for $t\in\mathbb{R}$, and the
duality pairing between the space of tempered distributions
$\mathcal{S}^{\prime}(\mathbb{R}^{d})$ and the Schwartz space
$\mathcal{S}(\mathbb{R}^{d})$ by $\langle f,\psi\rangle$, where
$f\in\mathcal{S}^{\prime}(\mathbb{R}^{d})$ and
$\psi\in\mathcal{S}(\mathbb{R}^{d})$. With this, we can define the Gabor
transform of a tempered distribution $f\in\mathcal{S}^{\prime}(\mathbb{R})$
via $\mathcal{G}f(x,\omega):=\langle
f,\operatorname{M}_{\omega}\operatorname{T}_{x}\phi\rangle$.

##  2 Preliminaries

###  2.1 The fractional Fourier transform

The _fractional Fourier transform_ of a function $f\in\mathcal{S}(\mathbb{R})$
is defined by

$\mathcal{F}_{\theta}f(\xi):=c_{\theta}\mathrm{e}^{\pi\mathrm{i}\xi^{2}\cot\theta}\int_{\mathbb{R}}f(t)\mathrm{e}^{\pi\mathrm{i}t^{2}\cot\theta}\mathrm{e}^{-2\pi\mathrm{i}\frac{t\xi}{\sin\theta}}\,\mathrm{d}t,\qquad\xi\in\mathbb{R},$

for $\theta\in\mathbb{R}\setminus\pi\mathbb{Z}$, where
$c_{\theta}\in\mathbb{C}$ is the square root of $1-\mathrm{i}\cot\theta$ with
positive real part, and by $\mathcal{F}_{2k\pi}f:=f$ as well as
$\mathcal{F}_{(2k+1)\pi}f(\xi):=f(-\xi)$, for $\xi\in\mathbb{R}$, where
$k\in\mathbb{Z}$. One can show that the fractional Fourier transform satisfies
[14, Theorem 2.1 on p. 406]

$\int_{\mathbb{R}^{d}}\mathcal{F}_{\theta}f(\xi)g(\xi)\,\mathrm{d}\xi=\int_{\mathbb{R}^{d}}f(t)\mathcal{F}_{\theta}g(t)\,\mathrm{d}t,\qquad
f,g\in\mathcal{S}(\mathbb{R})$

and that it maps a Schwartz function to a Schwartz function [17, Theorem 5.3
on p. 170]. Therefore, we can extend the fractional Fourier transform to the
tempered distributions via

$\langle\mathcal{F}_{\theta}f,\psi\rangle:=\langle
f,\mathcal{F}_{\theta}\psi\rangle,\qquad\psi\in\mathcal{S}(\mathbb{R}),$

for $f\in\mathcal{S}^{\prime}(\mathbb{R})$.

The fractional Fourier transform is a powerful tool in time-frequency
analysis. One of its most crucial properties is that it corresponds to a
rotation of the time-frequency plane [4]. To describe this property, we
introduce the rotation operator
$\operatorname{R}_{\theta}:\mathbb{R}^{2}\to\mathbb{R}^{2}$, defined by
$\operatorname{R}_{\theta}(x,\omega):=(x\cos\theta-\omega\sin\theta,x\sin\theta+\omega\cos\theta)$,
where $\theta\in\mathbb{R}$ and $x,\omega\in\mathbb{R}$.

######  Lemma 2.1.

Let $\theta\in\mathbb{R}$ and $f\in\mathcal{S}^{\prime}(\mathbb{R})$. It holds
that

$\mathcal{G}\mathcal{F}_{\theta}f(x,\omega)=\mathrm{e}^{\pi\mathrm{i}\left(x^{2}-\omega^{2}\right)\sin\theta\cos\theta+2\pi\mathrm{i}x\omega\sin^{2}\theta}\cdot\mathcal{G}f(\operatorname{R}_{-\theta}(x,\omega)),\qquad(x,\omega)\in\mathbb{R}^{2}.$

###### Proof.

Let $(x,\omega)\in\mathbb{R}^{2}$ be arbitrary but fixed. We consider

$\mathcal{G}\mathcal{F}_{\theta}f(x,\omega)=\langle\mathcal{F}_{\theta}f,\operatorname{M}_{\omega}\operatorname{T}_{x}\phi\rangle=\langle
f,\mathcal{F}_{\theta}\operatorname{M}_{\omega}\operatorname{T}_{x}\phi\rangle$

such that the lemma boils down to understanding the commutative properties of
the fractional Fourier transform and the modulation and translation operators.
According to [4, Table I on p. 3086], it holds that

$\displaystyle\mathcal{F}_{\theta}\operatorname{T}_{x}=\mathrm{e}^{\pi\mathrm{i}x^{2}\sin\theta\cos\theta}\cdot\operatorname{M}_{-x\sin\theta}\operatorname{T}_{x\cos\theta}\mathcal{F}_{\theta},$
$\displaystyle\mathcal{F}_{\theta}\operatorname{M}_{\omega}=\mathrm{e}^{-\pi\mathrm{i}\omega^{2}\sin\theta\cos\theta}\cdot\operatorname{M}_{\omega\cos\theta}\operatorname{T}_{\omega\sin\theta}\mathcal{F}_{\theta}.$

Using that the fractional Fourier transform of the Gaussian is the Gaussian,
we therefore find

$\displaystyle\mathcal{F}_{\theta}\operatorname{M}_{\omega}\operatorname{T}_{x}\phi$
$\displaystyle=\mathrm{e}^{\pi\mathrm{i}\left(x^{2}-\omega^{2}\right)\sin\theta\cos\theta}\cdot\operatorname{M}_{\omega\cos\theta}\operatorname{T}_{\omega\sin\theta}\operatorname{M}_{-x\sin\theta}\operatorname{T}_{x\cos\theta}\phi$
$\displaystyle=\mathrm{e}^{\pi\mathrm{i}\left(x^{2}-\omega^{2}\right)\sin\theta\cos\theta+2\pi\mathrm{i}x\omega\sin^{2}\theta}\cdot\operatorname{M}_{\omega\cos\theta-x\sin\theta}\operatorname{T}_{x\cos\theta+\omega\sin\theta}\phi$

which proves the lemma. ∎

###  2.2 The Paley–Wiener spaces

In the following, we will mostly work with bandlimited functions. Precisely,
we consider the _Paley–Wiener spaces_ defined by

$\mathrm{PW}^{p}_{B}:=\left\\{f\in\mathcal{S}^{\prime}(\mathbb{R})\,\middle|\,f=\mathcal{F}F\mbox{
for some }F\in L^{p}([-B,B])\right\\},$

for $B>0$ and $p\in[1,\infty]$. One may see that the Paley–Wiener spaces are
nested — $\mathrm{PW}_{B}^{q}\subset\mathrm{PW}_{B}^{p}$ for $1\leq p\leq
q\leq\infty$ — which is due to the nestedness of $L^{p}$-spaces over closed
intervals — $L^{q}([-B,B])\subset L^{p}([-B,B])$ for $1\leq p\leq
q\leq\infty$. It therefore follows that
$\mathrm{PW}_{B}^{p}\subset\mathrm{PW}_{B}^{2}\subset L^{2}(\mathbb{R})$ for
$p\in[2,\infty]$ since the Fourier transform is unitary on
$L^{2}(\mathbb{R})$. Additionally, $\mathrm{PW}_{B}^{p}\subset
L^{q}(\mathbb{R})$ for $p\in[1,2]$, where $q\in[2,\infty]$ is the Hölder
conjugate of $p$, by the Hausdorff–Young inequality. Hence, the elements of
Paley–Wiener spaces are $L^{p}$-functions. In fact, one can show that the
elements of Paley–Wiener spaces extend to entire functions of exponential-
type. This is one direction of the well-known Paley–Wiener the- orem.

An important property of bandlimited functions is that one may recover them
from samples. Let us call a subset $X\subset\mathbb{R}$ _uniformly discrete_
if there exists an $\epsilon>0$ such that, for all $x,y\in X$ with $x\neq y$,
it holds that $\left\lvert x-y\right\rvert>\epsilon$. We say that $X$ is a
_set of uniqueness_ for $\mathrm{PW}_{B}^{p}$ if

$f(x)=0\mbox{ for all }x\in X\implies f=0,$

for $f\in\mathrm{PW}_{B}^{p}$. Similarly, we say that $X$ is a _set of
sampling_ for $\mathrm{PW}_{B}^{1}$ if there exists a constant $K>0$ such that

$\sup_{t\in\mathbb{R}}\left\lvert f(t)\right\rvert\leq K\sup_{x\in
X}\left\lvert f(x)\right\rvert,\qquad f\in\mathrm{PW}_{B}^{1}.$

Clearly, every set of sampling for $\mathrm{PW}_{B}^{1}$ is a set of
uniqueness for $\mathrm{PW}_{B}^{p}$ by the nestedness of the Paley–Wiener
spaces. Hence, $f\in\mathrm{PW}_{B}^{p}$ is uniquely determined by
$(f(x))_{x\in X}$ if $X$ is a set of sampling for $\mathrm{PW}_{B}^{1}$. Sets
of sampling for $\mathrm{PW}_{B}^{1}\subset C_{0}(\mathbb{R})$ were
characterised by Beurling in a series of seminar lectures given at Princeton
[6]. Specifically, Beurling introduces

$\underline{n}(r):=\inf_{t\in\mathbb{R}}\left\lvert
X\cap[t,t+r]\right\rvert,\qquad r>0,$

along with the _lower uniform density_

$\mathrm{l.u.d.}(X):=\lim_{r\to\infty}\frac{\underline{n}(r)}{r}$

and proves the following result.

######  Theorem 2.2 ([6, Theorem 5 on p. 346]).

A uniformly discrete set $X\subset\mathbb{R}$ is a set of sampling for
$\mathrm{PW}_{B}^{1}$ if and only if

$\mathrm{l.u.d.}(X)>2B.$

######  Remark 2.3.

Notably, there are uniformly discrete sets $X$ that form a set of uniqueness
for $\mathrm{PW}_{B}^{2}$ with $\mathrm{l.u.d.}(X)=0$ [15]. This illustrates
that $\mathrm{l.u.d.}(X)>2B$ is sufficient but not necessary for $X$ being a
set of uniqueness for $\mathrm{PW}_{B}^{p}$ in general.

Finally, we note that the Gabor transform of a bandlimited function is
bandlimited itself in the time variable (after modulation) and that therefore
the square of the Gabor transform magnitudes is bandlimited.

######  Lemma 2.4.

Let $p\in[1,\infty]$, $B>0$, $\omega\in\mathbb{R}$ and
$f\in\mathrm{PW}_{B}^{p}$. Then,
$x\mapsto\mathrm{e}^{2\pi\mathrm{i}x\omega}\mathcal{G}f(x,\omega)\in\mathrm{PW}_{B}^{p}$
and therefore
$x\mapsto\lvert\mathcal{G}f(x,\omega)\rvert^{2}\in\mathrm{PW}_{2B}^{q}$, where
$q\in[1,\infty]$ is such that

$1+\frac{1}{q}=\frac{2}{p}.$

###### Proof.

First, we write $\mathcal{G}f(x,\omega)$ in terms of the inverse Fourier
transform of the signal $\mathcal{F}^{-1}f$ and the Gaussian $\phi$ (cf. [9,
Lemma 3.1.1 on p. 39]):

$\mathrm{e}^{2\pi\mathrm{i}x\omega}\mathcal{G}f(x,\omega)=\mathcal{F}\left(\mathcal{F}f\cdot\operatorname{T}_{\omega}\phi\right)(-x)=\mathcal{F}\left(\mathcal{F}^{-1}f\cdot\operatorname{T}_{-\omega}\phi\right)(x).$

Next, we use the facts that $\mathcal{F}^{-1}f\in L^{p}([-B,B])$ and $\phi\in
L^{\infty}(\mathbb{R})$ to see that
$\mathcal{F}^{-1}f\cdot\operatorname{T}_{-\omega}\phi$ is also in
$L^{p}([-B,B])$. This implies that
$x\mapsto\mathrm{e}^{2\pi\mathrm{i}x\omega}\mathcal{G}f(x,\omega)$ is in the
Paley-Wiener space $\mathrm{PW}_{B}^{p}$.

We then compute the squared modulus of $\mathcal{G}f(x,\omega)$ as a
convolution of two functions in Fourier space: specifically, let
$F_{\omega}:=\mathcal{F}^{-1}f\cdot\operatorname{T}_{-\omega}\phi$. Then,

$\lvert\mathcal{G}f(x,\omega)\rvert^{2}=\mathcal{F}F_{\omega}(x)\cdot\overline{\mathcal{F}F_{\omega}(x)}=\mathcal{F}F_{\omega}(x)\cdot\mathcal{F}F_{\omega}^{\\#}(x)=\mathcal{F}\left(F_{\omega}\ast
F_{\omega}^{\\#}\right)(x).$

Using Young’s convolution inequality, we can show that $F\ast
F_{\omega}^{\\#}\in L^{q}(\mathbb{R})$, where $q\in[1,\infty]$ satisfies

$1+\frac{1}{q}=\frac{2}{p}.$

Finally, we can bound the support of $F\ast F_{\omega}^{\\#}$ by $[-2B,2B]$.
Putting everything together, we conclude that
$x\mapsto\lvert\mathcal{G}f(x,\omega)\rvert^{2}$ is in the Paley-Wiener space
$\mathrm{PW}_{2B}^{q}$. ∎

We have shown that $x\mapsto\left\lvert\mathcal{G}f(x,\omega)\right\rvert^{2}$
belongs to the Paley–Wiener space
$\mathrm{PW}_{2B}^{q}\subset\mathrm{PW}_{2B}^{1}$. Therefore, we can apply
Theorem 2.2 to conclude that
$(\left\lvert\mathcal{G}f(x,\omega)\right\rvert)_{x\in X}$ completely
determines $x\mapsto\left\lvert\mathcal{G}f(x,\omega)\right\rvert$, for
$\omega\in\mathbb{R}$ fixed, provided that $X\subset\mathbb{R}$ is a uniformly
discrete set such that $\mathrm{l.u.d.}(\Lambda)>4B$.

######  Remark 2.5.

A similar result can be proven for the short-time Fourier transform with
window $\psi\in\mathcal{F}L^{q}(\mathbb{R})$, where $q\in[1,\infty]$, by
following the same proof strategy. In this way, one may arrive at a partial
sampling result for short-time Fourier transform phase retrieval in which one
assumes knowledge of measurements on $X\times\mathbb{R}$, where $X$ is a set
of uniqueness for the appropriate Paley–Wiener space.

###  2.3 Zalik’s theorem

For the proof of our main result, we need the following result of Müntz–Szász
type due to Zalik. It asserts that certain translates of Gaussians are
complete in the spaces $L^{p}([a,b])$ and $C([a,b])$.

######  Theorem 2.6 (Zalik’s theorem; [20, Theorem 4 on p. 302]).

Let $p\in[1,\infty)$, $-\infty<a<b<\infty$, $c>0$, and let
$\Omega\subset\mathbb{R}$ be a countable set. Then,

$\left\\{t\mapsto\mathrm{e}^{-c(t-\omega)^{2}}\,\middle|\,\omega\in\Omega\right\\}$

is complete in $L^{p}([a,b])$ and $C([a,b])$ if and only if

$\sum_{\omega\in\Omega\setminus\\{0\\}}\left\lvert\omega\right\rvert^{-1}$ (2)

diverges.

###### Proof.

The result for $L^{p}([a,b])$ follows from a small modification of the
original proof in [20]. For $C([a,b])$ consider the following argument which
is also inspired by the original proof: suppose that the sum in equation (2)
diverges, and let $f\in\mathcal{C}([a,b])$ as well as $\epsilon>0$ be
arbitrary but fixed. Define

$g(x):=f\left(\frac{\log x}{2c}\right)\mathrm{e}^{\frac{\log^{2}x}{4c}},\qquad
x\in\left[\mathrm{e}^{2ca},\mathrm{e}^{2cb}\right],$

and note that $g\in C([\mathrm{e}^{2ca},\mathrm{e}^{2cb}])$. According to [16,
Theorem 6.1 on p. 30], $\\{x\mapsto x^{\omega}\,|\,\omega\in\Omega\\}$ is
complete in $C([\mathrm{e}^{2ca},\mathrm{e}^{2cb}])$. Therefore, there exist
$\Omega_{0}\subset\Omega$ finite and
$(\lambda_{\omega})_{\omega\in\Omega_{0}}\in\mathbb{C}$ such that

$\sup_{x\in[\mathrm{e}^{2ca},\mathrm{e}^{2cb}]}\left\lvert\sum_{\omega\in\Omega_{0}}\lambda_{\omega}x^{\omega}-g(x)\right\rvert<\epsilon.$

By the change of variable $x=\mathrm{e}^{2ct}$, it follows that

$\sup_{t\in[a,b]}\left\lvert\sum_{\omega\in\Omega_{0}}\lambda_{\omega}\mathrm{e}^{2c\omega
t}-f(t)\mathrm{e}^{ct^{2}}\right\rvert<\epsilon$

and therefore

$\sup_{t\in[a,b]}\left\lvert\sum_{\omega\in\Omega_{0}}\lambda_{\omega}\mathrm{e}^{c\omega^{2}}\cdot\mathrm{e}^{-c(t-\omega)^{2}}-f(t)\right\rvert\leq\sup_{t\in[a,b]}\left\lvert\sum_{\omega\in\Omega_{0}}\lambda_{\omega}\mathrm{e}^{2c\omega
t}-f(t)\mathrm{e}^{ct^{2}}\right\rvert<\epsilon$

showing that $\\{t\mapsto\mathrm{e}^{-c(t-\omega)^{2}}\,|\,\omega\in\Omega\\}$
is complete in $C([a,b])$.

Now, suppose that
$\\{t\mapsto\mathrm{e}^{-c(t-\omega)^{2}}\,|\,\omega\in\Omega\\}$ is complete
in $C([a,b])$. Then, a modification of the argument above shows that
$\\{x\mapsto x^{\omega}\,|\,\omega\in\Omega\\}$ is complete in
$C([\mathrm{e}^{2ca},\mathrm{e}^{2cb}])$. Therefore, [16, Theorem 6.1 on p.
30] implies that the sum in equation (2) diverges. ∎

######  Remark 2.7.

By the same proof, we can extend Zalik’s theorem to a countable subset
$\Omega$ of $\mathbb{C}$ under the condition that there exists a $\delta>0$
and a finite subset $\Omega_{0}\subset\Omega$ such that

$\left\lvert\operatorname{Re}\left(\omega-\frac{1}{2}\right)\right\rvert\geq\delta\left\lvert\omega-\frac{1}{2}\right\rvert,\qquad\omega\in\Omega\setminus\Omega_{0}.$

##  3 Main results

After laying out the groundwork, we present our main theorem, which
generalises a result previously established by Grohs and Liehr for
$\mathrm{PW}_{B}^{4}$ [11]. Specifically, we extend their result to hold for
the entire family of Paley–Wiener spaces $\mathrm{PW}_{B}^{p}$ for
$p\in[1,\infty]$. This encompasses the case $p=2$ which is commonly studied in
signal processing, as well as the challenging case $p=1$. Our result shows
that for sufficiently dense uniformly discrete sets $X$ and certain countable
sets $\Omega$, functions in $\mathrm{PW}_{B}^{p}$ are uniquely determined (up
to global phase) from their Gabor magnitudes on $X\times\Omega$.

######  Theorem 3.1 (Main result).

Let $p\in[1,\infty]$ and $B>0$. Let $X\subset\mathbb{R}$ be a uniformly
discrete set with $l.u.d.(X)>4B$ and let $\Omega\subset\mathbb{R}$ be a
countable set such that
$\sum_{\omega\in\Omega\setminus\\{0\\}}\left\lvert\omega\right\rvert^{-1}$
diverges. Then, the following are equivalent for $f,g\in\mathrm{PW}_{B}^{p}$:

  1. 1.

$f=\mathrm{e}^{\mathrm{i}\alpha}g$ for some $\alpha\in\mathbb{R}$,

  2. 2.

$\left\lvert\mathcal{G}f\right\rvert=\left\lvert\mathcal{G}g\right\rvert$ on
$X\times\Omega$.

###### Proof of Theorem 3.1.

The following arguments are inspired by [11]. It is obvious that item 1
implies item 2. Therefore, we assume that $f,g\in\mathrm{PW}_{B}^{p}$ are such
that $\left\lvert\mathcal{G}f\right\rvert=\left\lvert\mathcal{G}g\right\rvert$
on $X\times\Omega$. Let us now fix an arbitrary $\omega\in\Omega$ and note
that Lemma 2.4 implies that
$x\mapsto\lvert\mathcal{G}f(x,\omega)\rvert^{2},x\mapsto\lvert\mathcal{G}g(x,\omega)\rvert^{2}\in\mathrm{PW}_{2B}^{1}$.
It follows by Theorem 2.2 that

$\left\lvert\mathcal{G}f(x,\omega)\right\rvert=\left\lvert\mathcal{G}g(x,\omega)\right\rvert,\qquad
x\in\mathbb{R}.$

Following the argument in the proof of Lemma 2.4, it is readily shown that

$\left(\mathcal{F}^{-1}f\cdot\operatorname{T}_{-\omega}\phi\right)\ast\left(\mathcal{F}^{-1}f\cdot\operatorname{T}_{-\omega}\phi\right)^{\\#}=\left(\mathcal{F}^{-1}g\cdot\operatorname{T}_{-\omega}\phi\right)\ast\left(\mathcal{F}^{-1}g\cdot\operatorname{T}_{-\omega}\phi\right)^{\\#}$

in $L^{1}([-2B,2B])$. We reformulate the above to

$\int_{-B}^{B}\mathcal{F}^{-1}f(\eta)\overline{\mathcal{F}^{-1}f(\eta-\xi)}\phi(\eta+\omega)\phi(\eta-\xi+\omega)\,\mathrm{d}\eta\\\
=\int_{-B}^{B}\mathcal{F}^{-1}g(\eta)\overline{\mathcal{F}^{-1}g(\eta-\xi)}\phi(\eta+\omega)\phi(\eta-\xi+\omega)\,\mathrm{d}\eta$

for almost all $\xi\in\mathbb{R}$. By completing the square, we see that

$\phi(\eta+\omega)\phi(\eta-\xi+\omega)=\sqrt{2}\cdot\mathrm{e}^{-2\pi(\eta+\omega-\xi/2)^{2}}\cdot\mathrm{e}^{-\frac{\pi\xi^{2}}{2}}$

and therefore

$\int_{-B}^{B}\left(\mathcal{F}^{-1}f(\eta)\overline{\mathcal{F}^{-1}f(\eta-\xi)}-\mathcal{F}^{-1}g(\eta)\overline{\mathcal{F}^{-1}g(\eta-\xi)}\right)\mathrm{e}^{-2\pi(\eta+\omega-\xi/2)^{2}}\,\mathrm{d}\eta=0,$
(3)

for almost all $\xi\in\mathbb{R}$. According to Fubini’s theorem,

$\eta\mapsto
H_{\xi}(\eta):=\mathcal{F}^{-1}f(\eta)\overline{\mathcal{F}^{-1}f(\eta-\xi)}-\mathcal{F}^{-1}g(\eta)\overline{\mathcal{F}^{-1}g(\eta-\xi)}\in
L^{1}([-B,B])$

for almost all $\xi\in\mathbb{R}$. Let us now fix an arbitrary
$\xi\in\mathbb{R}$ such that the above two assertions hold and consider the
set $\Omega^{\xi}:=\tfrac{\xi}{2}-\Omega$. Then,
$\Omega^{\xi}\subset\mathbb{R}$ is countable and

$\sum_{\omega\in\Omega^{\xi}\setminus\\{0\\}}\left\lvert\omega\right\rvert^{-1}$

diverges. Therefore, Theorem 2.6 implies that the set

$\left\\{\eta\mapsto\mathrm{e}^{-2\pi(\eta-\omega)^{2}}\,\middle|\,\omega\in\Omega^{\xi}\right\\}$

is complete in $C([-B,B])$. It follows that, for all $t\in\mathbb{R}$ and all
$\epsilon>0$, there exist a finite set $\Omega_{0}^{\xi}\subset\Omega^{\xi}$
and a sequence
$(\lambda_{\omega}^{\xi})_{\omega\in\Omega_{0}^{\xi}}\in\mathbb{C}$ such that

$\sup_{\eta\in[-B,B]}\left\lvert\sum_{\omega\in\Omega_{0}^{\xi}}\lambda_{\omega}^{\xi}\mathrm{e}^{-2\pi(\eta-\omega)^{2}}-\mathrm{e}^{2\pi\mathrm{i}\eta
t}\right\rvert<\epsilon.$

According to equation (3), we have

$\displaystyle\left\lvert\int_{-B}^{B}H_{\xi}(\eta)\mathrm{e}^{2\pi\mathrm{i}\eta
t}\,\mathrm{d}\eta\right\rvert$ $\displaystyle\leq\int_{-B}^{B}\left\lvert
H_{\xi}(\eta)\right\rvert\left\lvert\sum_{\omega\in\Omega_{0}^{\xi}}\lambda_{\omega}^{\xi}\mathrm{e}^{-2\pi(\eta-\omega)^{2}}-\mathrm{e}^{2\pi\mathrm{i}\eta
t}\right\rvert\,\mathrm{d}\eta$ $\displaystyle\leq\epsilon\cdot\left\lVert
H_{\xi}\right\rVert_{1}.$

Since the above holds for all $\epsilon>0$, we conclude

$\mathcal{F}^{-1}H_{\xi}(t)=\int_{-B}^{B}H_{\xi}(\eta)\mathrm{e}^{2\pi\mathrm{i}\eta
t}\,\mathrm{d}\eta=0,\qquad t\in\mathbb{R}.$

Therefore, $H_{\xi}=0$ in $L^{1}(\mathbb{R})$ which implies that

$\mathcal{F}^{-1}f(\eta)\overline{\mathcal{F}^{-1}f(\eta-\xi)}=\mathcal{F}^{-1}g(\eta)\overline{\mathcal{F}^{-1}g(\eta-\xi)}$
(4)

for almost all $\eta\in\mathbb{R}$.

The rest of the proof is classical. We present the following argument inspired
by [5, Theorem 2.5 on p. 588] for the convenience of the reader: another
application of Fubini’s theorem implies that

$\xi\mapsto\mathcal{F}^{-1}f(\eta)\overline{\mathcal{F}^{-1}f(\eta-\xi)}\in
L^{1}(\mathbb{R})$

for almost all $\eta\in\mathbb{R}$ (and the same is true for $g$). If we fix
such an $\eta$, then we can apply the Fourier transform to equation (4) and
obtain

$\mathcal{F}^{-1}f(\eta)\overline{f(t)}=\mathcal{F}^{-1}g(\eta)\overline{g(t)},\qquad
t\in\mathbb{R}.$

If $f=0$, then $\mathcal{F}^{-1}g(\eta)\overline{g(t)}=0$ such that either
$g=0$ or $\mathcal{F}^{-1}g=0$ almost everywhere in which case $g=0$. Hence,
the theorem is proven if $f=0$. So let us assume that $f\neq 0$. Then, there
exists $t_{0}\in\mathbb{R}$ such that $f(t_{0})\neq 0$. Therefore,

$\mathcal{F}^{-1}f=\tau\cdot\mathcal{F}^{-1}g,\qquad\tau:=\frac{\overline{g(t_{0})}}{\overline{f(t_{0})}}$

in $L^{1}([-B,B])$ and thus $f=\tau g$. This proves the theorem since it
implies $\left\lvert f(t_{0})\right\rvert=\left\lvert g(t_{0})\right\rvert$
which shows that $\left\lvert\tau\right\rvert=1$. ∎

We can use the fractional Fourier transform to rotate our result in the time-
frequency plane. To this end, we introduce the spaces

$\mathcal{F}_{\theta}L^{p}([-B,B]):=\left\\{f\in\mathcal{S}^{\prime}(\mathbb{R})\,\middle|\,f=\mathcal{F}_{\theta}F\mbox{
for some }F\in L^{p}([-B,B])\right\\},$

for $B>0$ and $p\in[1,\infty]$. These spaces are nested similarly to the
Paley-Wiener spaces, with
$\mathcal{F}_{\theta}L^{q}([-B,B])\subset\mathcal{F}_{\theta}L^{p}([-B,B])$
for $1\leq p\leq q\leq\infty$. Thus, for $p\geq 2$, we have
$\mathcal{F}_{\theta}L^{p}([-B,B])\subset\mathcal{F}_{\theta}L^{2}([-B,B])\subset
L^{2}(\mathbb{R})$ since the fractional Fourier transform is unitary on
$L^{2}(\mathbb{R})$. Moreover, for $1\leq p\leq 2$, we have
$\mathcal{F}_{\theta}L^{p}([-B,B])\subset L^{q}(\mathbb{R})$, where $q$ is the
Hölder conjugate of $p$, since the Hausdorff-Young inequality extends to the
fractional Fourier transform [7, Theorem 4.2 on p. 88].

With these spaces in hand, we can now state and prove a generalisation of our
main result.

######  Theorem 3.2 (Generalised main result).

Let $p\in[1,\infty]$, $B>0$ and $\theta\in\mathbb{R}$. Let
$X\subset\mathbb{R}$ be a set of uniqueness for $\mathrm{PW}_{2B}^{1}$ and let
$\Omega\subset\mathbb{R}$ be a countable set such that
$\sum_{\omega\in\Omega\setminus\\{0\\}}\left\lvert\omega\right\rvert^{-1}$
diverges. Then, the following are equivalent for
$f,g\in\mathcal{F}_{\theta}L^{p}([-B,B])$:

  1. 1.

$f=\mathrm{e}^{\mathrm{i}\alpha}g$ for some $\alpha\in\mathbb{R}$,

  2. 2.

$\left\lvert\mathcal{G}f\right\rvert=\left\lvert\mathcal{G}g\right\rvert$ on
$\operatorname{R}_{\theta}(\Omega\times X)$.

###### Proof.

Clearly, item 1 implies item 2. We will therefore assume that
$\left\lvert\mathcal{G}f\right\rvert=\left\lvert\mathcal{G}g\right\rvert$ on
$\operatorname{R}_{\theta}(\Omega\times X)$. According to Lemma 2.1, we have
$\left\lvert\mathcal{G}\mathcal{F}_{-\theta}f\right\rvert=\left\lvert\mathcal{G}\mathcal{F}_{-\theta}g\right\rvert$
on $\Omega\times X$. Note that
$\mathcal{F}_{-\theta}f,\mathcal{F}_{-\theta}g\in L^{p}([-B,B])$ such that
$\mathcal{F}\mathcal{F}_{-\theta}f,\mathcal{F}\mathcal{F}_{-\theta}g\in\mathrm{PW}_{B}^{p}$.
Another application of Lemma 2.1 yields
$\left\lvert\mathcal{G}\mathcal{F}\mathcal{F}_{-\theta}f\right\rvert=\left\lvert\mathcal{G}\mathcal{F}\mathcal{F}_{-\theta}g\right\rvert$
on $X\times\Omega$. Therefore, Theorem 3.1 implies that
$\mathcal{F}\mathcal{F}_{-\theta}f=\mathrm{e}^{\mathrm{i}\alpha}\cdot\mathcal{F}\mathcal{F}_{-\theta}g$
for some $\alpha\in\mathbb{R}$. Finally, $f=\mathrm{e}^{\mathrm{i}\alpha}g$
follows from Fourier inversion. ∎

#### Acknowledgements

The author would like to thank Rima Alaifari for her comments which have
improved the presentation of the paper as well as the first reviewer whose
remarks have inspired a generalisation of the main results. Additionally, the
author acknowledges funding through the SNSF Grant 200021_184698.

#### Declaration of generative AI and AI-assisted technologies in the writing
process

During the review process for this work, the author used ChatGPT to correct
punctuation and orthography. After using this tool, the author reviewed and
edited the content as needed and takes full responsibility for the content of
the publication.

## References

  * [1] Rima Alaifari, Francesca Bartolucci, Stefan Steinerberger, and Matthias Wellershoff.  On the connection between uniqueness from samples and stability in Gabor phase retrieval.  https://doi.org/10.48550/arXiv.2205.07013, May 2022.
  * [2] Rima Alaifari and Matthias Wellershoff.  Phase retrieval from sampled Gabor transform magnitudes: counterexamples.  Journal of Fourier Analysis and Applications, 28(9):1–8, 2021.
  * [3] Rima Alaifari and Matthias Wellershoff.  Uniqueness of STFT phase retrieval for bandlimited functions.  Applied and Computational Harmonic Analysis, 50:34–48, 2021.
  * [4] Luis B Almeida.  The fractional Fourier transform and time-frequency representations.  IEEE Transactions on Signal Processing, 42(11):3084–3091, 1994\.
  * [5] Louis Auslander and R Tolimieri.  Radar ambiguity functions and group theory.  SIAM Journal on Mathematical Analysis, 16(3):577–601, 1985.
  * [6] Lennart Carleson, Paul Malliavin, John Neuberger, and John Wermer, editors.  Harmonic Analysis, volume 2 of Collected works of Arne Beurling.  Birkhäuser, Boston, 1989.
  * [7] Wei Chen, Zunwei Fu, Loukas Grafakos, and Yue Wu.  Fractional Fourier transforms on $L^{p}$ and applications.  Applied and Computational Harmonic Analysis, 55:71–96, November 2021.  https://doi.org/10.1016/j.acha.2021.04.004.
  * [8] James L Flanagan and Roger M Golden.  Phase vocoder.  Bell System Technical Journal, 45(9):1493–1509, 1966.
  * [9] Karlheinz Gröchenig.  Foundations of time-frequency analysis.  Birkhäuser Boston, MA, 2001.
  * [10] Philipp Grohs and Lukas Liehr.  On foundational discretisation barriers in STFT phase retrieval.  Journal of Fourier Analysis and Applications, 28:1–21, 2022.
  * [11] Philipp Grohs and Lukas Liehr.  Injectivity of Gabor phase retrieval from lattice measurements.  Applied and Computational Harmonic Analysis, 62:173–193, 2023.
  * [12] Philipp Grohs, Lukas Liehr, and Irina Shafkulovska.  From completeness of discrete translates to phaseless sampling of the short-time Fourier transform.  https://doi.org/10.48550/arXiv.2211.05687, November 2022.
  * [13] Mark Iwen, Michael Perlmutter, Nada Sissouno, and Aditya Viswanathan.  Phase retrieval for ${L}^{2}([-\pi,\pi])$ via the provably accurate and noise robust numerical inversion of spectrogram measurements.  Journal of Fourier Analysis and Applications, 29(8), 2023.  https://doi.org/10.1007/s00041-022-09990-y.
  * [14] Fiona H. Kerr.  Namias’ fractional Fourier transforms on $L^{2}$ and applications to differential equations.  Journal of Mathematical Analysis and Applications, 136(2):404–418, December 1988.  https://doi.org/10.1016/0022-247X(88)90094-7.
  * [15] Paul Koosis.  Sur la totalité des systèmes d’exponentielles imaginaires.  Comptes Rendus de l’Académie des Sciences, 250(2):2102–2103, March–April 1960.
  * [16] W. A. J. Luxemburg and J. Korevaar.  Entire functions and Müntz–Szász type approximation.  Transactions of the American Mathematical Society, 157:23–37, June 1971.  https://doi.org/10.2307/1995828.
  * [17] A. C. McBride and F. H. Kerr.  On Namias’s fractional Fourier transforms.  IMA Journal of Applied Mathematics, 38(2):159–175, May 1987.  https://doi.org/10.1093/imamat/39.2.159.
  * [18] Zdeněk Prŭša and Nicki Holighaus.  Phase vocoder done right.  In 2017 25th European Signal Processing Conference (EUSIPCO), pages 976–980. IEEE, 2017.
  * [19] Matthias Wellershoff.  Sampling at twice the Nyquist rate in two frequency bins guarantees uniqueness in Gabor phase retrieval.  Journal of Fourier Analysis and Applications, 29(7), 2023.  https://doi.org/10.1007/s00041-022-09990-y.
  * [20] RA Zalik.  On approximation by shifts and a theorem of Wiener.  Transactions of the American Mathematical Society, 243:299–308, 1978.